At the beginning of this year, after Tesla officially launched the FSD V12 version in North America, many CEOs and executives in China's autonomous driving industry went to experience it.
Xiaopeng Motors CEO He Xiaopeng is also one of them. After several experiences, He Xiaopeng was very excited. He took the initiative to talk to the Vice President of Autonomous Driving, Li Liyun, about his feelings, saying, "The silky feeling is significant, the personification is improved, and it can be clearly felt that FSD is thinking." He also hoped that the key members of the team would go to the United States to experience it as soon as possible.
The rapid iteration of FSD has made the Xiaopeng autonomous driving team more confident in the end-to-end big model route.
Xiaopeng Motors is a 'smart driving veteran'. In September 2017, Xiaopeng began to independently develop intelligent driving software algorithms, leading Huawei and Ideal by 1 year and 8 months, and 3 years and 5 months respectively. Afterwards, Xiaopeng went through the stages of high-speed assisted driving and urban assisted driving, and was the first to land in 200 cities in the opening race at the beginning of this year.
The end-to-end layout and pre research can be traced back to 2022. Li Liyun told 21st Century Business Herald reporters that the Xiaopeng autonomous driving team had made several explorations: initially, they used various small models. At that time, Xiaopeng "piled up" dozens of excellent algorithm engineers, hoping to solve the problem through rule traction, but ultimately could not escape the traditional rule limitations.
In March 2023, OpenAI released GPT 4, and soon after, new models such as Sora and o1 were born, leading to a massive AI explosion. These important events inspired Xiaopeng. At the beginning of 2023, Xiaopeng began exploring how to apply end-to-end big models to the field of autonomous driving, and then began to move towards cloud based big models.
The vast majority of Chinese car companies, on the other hand, firmly embraced end-to-end big models only after the Tesla FSD V12 version.
Since the beginning of this year, car companies such as NIO, Ideal, and Leapmotor have established research and development teams around end-to-end, hoping to gain new opportunities for overtaking on curves. When entering a new technology cycle led by Tesla, we cannot estimate the time it takes for new technologies to emerge based on traditional time. Don't think that we will spend as long as others spend, "a smart driving expert with over 10 years of experience told 21st Century Business Herald.
In order to achieve quick results, some car companies have chosen the One Piece end-to-end model. Xiaopeng, who has accumulated 7 years of experience in intelligent driving, has been questioned for adopting a segmented end-to-end approach and a "conservative route".
Li Liyun denied that Xiaopeng is segmented end-to-end. "We are similar to Huawei, where XNet, XBrain, and XPlanner play the roles of human eyes, brain, and cerebellum, respectively. The three overlap and couple with each other
In his opinion, a one piece big model on the car side has certain side effects - in the future, as the amount of data increases, the limited computing power on the car will not be able to handle so much data. And Xiaopeng's solution is a cloud based big model. "The parameters of the cloud based big model will be 80 to 100 times higher than those of the current car end, which is a complete one piece intelligent agent," said Li Liyun.
In the process of developing TuTu, with the increase of end-to-end penetration rate, the Xiaopeng autonomous driving team also adjusted its organizational structure: newly established three departments: AI model development, AI application delivery, and AI efficiency. Xiaopeng did not lay off algorithm engineers, but helped them complete end-to-end transformation. Xiaopeng's intelligent driving team has been stable at around 2000 people, following the orderly growth of the business, "said Li Liyun.
Li Liyun regards end-to-end as the "era of hot weapons", while the past era of assisted driving was the "era of cold weapons". In the era of cold weapons, as long as you gather all the martial arts masters, you can fight. But in the era of hot weapons, greater computing power, more data, mechanisms for the flow of computing power and data (data flywheel system), and engineering capabilities are needed.
Enterprises that keep up with the trend and transform may succeed, but overall, the era of hot weapons will systematically widen the gap between the first and second tiers, making overtaking even more difficult, "said Li Liyun.
The following is a conversation between 21st Century Business Herald and Li Liyun, Vice President of Xiaopeng Autonomous Driving, and Yuan Tingting, Senior Director of Autonomous Driving Products:
21st Century Business Herald: You previously had a L4 background and were a founding core member of Baidu's US unmanned vehicle development team, serving as an architect for the X laboratory at JD Silicon Valley R&D center. Why didn't you continue with L4 development and choose to join passenger car company Xiaopeng in 2019?
Li Liyun: I joined Xiaopeng in June 2019. Although I used to specialize in L4 autonomous driving, I am actually a firm believer in gradual progress. I agree that the ultimate form of autonomous driving must be true unmanned, but it is difficult to achieve it in one step and directly become unmanned.
I really like cars and I am a very enthusiastic person about products. I used to drive Xiaopeng myself. I used to drive P7, but now I drive G6 Max. I can see my code running on my own product and buy it back to drive every day. Watching it evolve, I think it's really cool.
21st Century Business Herald: When you joined Xiaopeng, what did He Xiaopeng say that moved you?
Li Liyun: I first met Wu Xinzhou (then the person in charge of Xiaopeng's autonomous driving) in the United States. At that time, he had been in Xiaopeng for half a year and there were already some members in the team. Then I went back to China and met He Xiaopeng. He Xiaopeng said, 'We must do autonomous driving.' He is very confident and determined that intelligent driving can bring about change, which deeply moved me.
In order to experience the product on the front line without flying around, I moved back to Guangzhou from the United States in 2020.
21st Century Business Herald: In terms of assisted driving, Tesla launched the FSD V12 version earlier this year, leading the end-to-end direction. Is Xiaopeng influenced by Tesla?
Li Liyun: As early as 2021 and 2022, we began actively laying out and pre researching end-to-end solutions. Adhering to the data-driven concept, we used light radar and light maps. Nowadays, the industry is more accustomed to using the terms "lidar" and "no high-precision map".
Tesla has always been committed to the concept of data-driven approach. We have great respect for Tesla. Currently, only Xiaopeng and Tesla can achieve the ability to adapt advanced assisted driving models with a single software without relying on high-definition maps or LiDAR.
In the era of hot weapons, overtaking on curves has become even more difficult
21st Century Business Herald: In September 2017, Xiaopeng began developing its own intelligent driving software algorithm, leading Huawei and Ideal by 1 year and 8 months, 3 years and 5 months respectively. It has gone through the complete stages of high-speed NOA, assisted driving in urban areas with maps, assisted driving in urban areas without maps, and the current end-to-end stage. What is the biggest difference between end-to-end and previous stages?
Li Liyun: In the past, assisted driving seemed like the era of cold weapons. We needed many martial arts masters, who could take the lead among all the generals - they understood driving scenarios, business, mathematics, and one or two small modules. They could be invincible. But in fact, it is very difficult to find many martial arts masters. Even if we find it, the complex scenes we face are ever-changing, equivalent to a greater number of enemies.
The end-to-end era is like transitioning from the era of cold weapons to the era of hot weapons, not relying on manpower, but winning through "gunpowder" and troop deployment. Gunpowder "is equivalent to data, computing power, and algorithms, which are transformed into models in factories and then trained to solve problems.
21st Century Business Herald: Where does Xiaopeng's massive end-to-end data come from?
Li Liyun: Compared with autonomous driving L4 enterprises, as a host factory, Xiaopeng has its own car, and we have better definition ability in data collection.
Compared with car companies that started late, Xiaopeng's excellent engineering skills accumulated before can help us collect data more efficiently. The original rules can provide some guidance for AI and become a teacher.
Finally, Xiaopeng offers a wide range of car models, from sedans and SUVs to MPVs, from A-class, B-class to C-Class, ensuring the diversity and richness of our data.
21st Century Business Herald: Is accumulating data an end-to-end challenge? Does having data and computing power mean that car companies can implement end-to-end large-scale models?
Li Liyun: In the original rule era, the system was connected to more than ten cameras. After entering the end-to-end era, the data volume of these sensors has not changed compared to before.
In the era of rules, before solving a problem, we first look at whether the problem is caused by perception, prediction, or a combination of two sets of problems. We will use these two sets of algorithm engineers to design scenarios, mathematical models, and rules to solve problems and regress scenarios. There are just too many details like this, and it will involve more modules.
After becoming end-to-end, the playing style changed and the entire chain became very long. Car companies need to collect a large amount of data to solve scenario problems, and even label and clean unsupervised data as models for themselves. This model can be pre trained and then jointly trained, or it can be trained as a large model. After training, the entire chain is very long to see if the quality of the trained model can be quantified, deployed, simulated, and validated.
In addition to data collection, engineering capabilities are also reflected in the construction of big data systems and the deployment of computing power, which is not an easy task.
21st Century Business Herald: Are so many "martial arts masters" accumulated by Xiaopeng in the era of cold weapons no longer needed? What advantages can be leveraged from his past accumulation?
Li Liyun: To collect efficient data, the most important thing is that the autonomous driving team needs to do a lot of work on the vehicle side. Otherwise, if a large amount of data is collected and stored, it becomes a cost.
If it weren't for unlimited resources, the collection of vehicle data would require strong algorithmic and even AI capabilities. This is consistent with our previous accumulation. For example, using rules to supervise data collection, such as the path generated by AI, may be very geometrically unreasonable and obviously not like what humans would open, which can be quickly identified through rules.
Compared with traditional technical solutions, end-to-end is often considered to have a high upper limit and a low lower limit. But this may be a unique aspect of what we do. In the previous era, we established a large number of simulation datasets, all of which were calibrated according to rules. When AI reached the upper limit of a new model, we would run these datasets to quickly detect the unreasonable lower limit of the model and perform rapid model calibration. The rules we have accumulated in the past have provided a safety net for AI.
In addition, currently only Xiaopeng and Tesla can achieve it, without relying on high-definition maps or LiDAR, using a set of software to adapt to all advanced intelligent driving models.
21st Century Business Herald: Why other car companies can't do it, and where do they fall short?
Li Liyun: Firstly, Xiaopeng's data collection efficiency is higher; Secondly, Xiaopeng has strong platform engineering capabilities. In the era of end-to-end AI, whether or not there is a LiDAR, regardless of the type of vehicle, it is a set of intelligent driving solutions for us.
21st Century Business Herald: After Tesla V12, many car companies are embracing end-to-end, hoping to overtake on the bend. Is it easier or harder to overtake on the bend?
Li Liyun: Originally, the engineering ability was to recruit and stack various cold weapon experts in different directions. As long as they were gathered together, they could fight.
In the era of hot weapons, greater computing power and data are needed. Behind this, the mechanism that can make so much computing power and data flow needs to be deployed on the car, and during the process of getting on the car, Tesla and we do not deny that there are occasional times when some rules are needed to provide a safety net. I think companies that keep up with the trend of transformation may also have success, but overall, they will systematically widen the gap between the first and second tiers.
Yuan Tingting: In terms of engineering, we have invested a lot of effort in AI Infra (i.e. AI infrastructure, the AI intermediate layer infrastructure that connects computing power and applications). For example, to stir fry a dish, you can use a good stove, firewood, and fruit wood, or use an alcohol lamp and a small aluminum pot on top. It may seem like you can quickly serve a dish, but in the long run, it's completely different.
Doing end-to-end is like having a baby in October. Having a baby in October really requires ten months of adequate nutrition and care for it to have the moment of birth. It's not something I planned to do, I invested enough money, so I can "produce" it in a month with ten people. It requires a solid foundation and sufficient effort to achieve the ultimate harvest.
Three attempts of 'smart driving veteran'
21st Century Business Herald: When was Xiaopeng first tested end-to-end? What was the form and performance of end-to-end at that time?
Li Liyun: In September 2022, Xiaopeng City's assisted driving system landed in Guangzhou, becoming the first car company to mass produce city navigation assisted driving. However, our entire research and development was completed in the first half of 2022, and time was spent on reviewing drawings. At that time, we thought high-precision maps were a crutch. To do a good job in urban navigation assisted driving, we need to use more generalized and better technical solutions to adapt to various road conditions. We began to switch to the non graph scheme.
At first, the graph free solution required more complex algorithms to detect various types of vehicles such as tricycles and electric vehicles, which was far less convenient than defining a model to generalize. Therefore, we tried to stack small models and assembled dozens of excellent algorithm engineers to solve the problem through the coupling of some rules.
However, the interface of manually defined rules means that these models have not yet freed themselves from algorithm rules, and it is also difficult to pile up more excellent algorithm engineers.
21st Century Business Herald: What is the most difficult problem for small models to solve? What special cases did you encounter at that time?
Li Liyun: The problem cannot be solved by coupling the rules of various small models, because the models themselves need to transmit more information.
During the small model period, scenes such as roundabouts, narrow roads, small paths, turns, and large intersections were very difficult, and we may have to spend 3-5 months.
For example, in some cities, the intersections are very complex. When a driver needs to turn left at an intersection, but finds that the road ahead is an overpass, the road ahead is a side road, and there is another road next to it, the system may directly slow down to 0.
And the end-to-end big model is very smart, it solves two major problems: first, it can never be turned on in special scenarios; The second is to enhance anthropomorphism. For example, when the driver is at the above-mentioned intersection, the system will not stop or switch to another lane, but will hesitate like a human, slow down slightly, and confidently choose a road to walk over. The slight feeling is like a chef cooking, adding a little salt, the taste is just right. This kind of change is very anthropomorphic and has a strong "taste".
With the support of data and big models, we can complete the above scenario in about one or two months.
21st Century Business Herald: What is the opportunity for teams to start thinking about the need to transition to big models?
To become a top global AI enterprise, it is essential to closely monitor the latest developments in AI technology. In March 2023, OpenAI released GPT 4. Afterwards, with the release of new models such as Sora and o1 by OpenAI, and the emergence of AI, these important events sparked our thinking.
We have accumulated data and architecture before, and at the beginning of last year, we began to think about how to apply big models to the field of autonomous driving. At the beginning of this year, we began exploring the transition from big models to cloud based big models.
I think cloud based big models are more attractive. In the future, at a crossroads, the system can even more confidently choose a better road based on memory. It can reduce dimensionality and attack big models, empowering intelligent driving.
21st Century Business Herald: In May of this year, Xiaopeng announced the mass production of an end-to-end intelligent driving model, becoming the only car company in the world and the first in China to mass produce an end-to-end intelligent driving model after Tesla. What are the differences between the design ideas of the intelligent driving big model at that time and today?
Li Liyun: The first version of the end-to-end intelligent driving model for getting on the car is a process of gradually getting on the car according to the scene. In Xiaopeng's upcoming AI Dimensity XOS 5.4.0 system, we have used end-to-end large models regardless of the scenario, and the overall anthropomorphism will take a big step forward.
One piece is effective end-to-end, but it has significant side effects
21st Century Business Herald: There are currently two mainstream views on the selection of end-to-end solutions: One model end-to-end and segmented end-to-end. Xiaopeng is classified as a segmented end-to-end solution. Do you agree with this view?
Li Liyun: Segmented end-to-end is a concept artificially created, Xiaopeng is not segmented end-to-end.
In Xiaopeng auto drive system, XNet, XBrain and XPlanner, which play the roles of human eyes, brain and cerebellum respectively, overlap and couple with each other. In deep learning, the three major networks will pre train each part and then jointly train them.
21st Century Business Herald: Why is it designed like this?
Li Liyun: There are two reasons. The first important reason is that I feel we are standing on a cognitive high ground because we have been investing in end-to-end research and development since a very early age, and have designed XNet, XBrain, and XPlanner based on the principle of complete anthropomorphism. What's even more important behind this is that we have a cloud based model, also known as a foundation model, which was pre trained into three networks for interpretability and reasonable allocation and deployment of computing power.
In fact, Huawei's end-to-end architecture also includes a perception network, a regulatory network, and an innate security network. We have similarities with Huawei in terms of model cognition, that is, in an end-to-end nature, we are more concerned with lossless transmission of information and maximizing information retention, rather than deliberately pursuing training and deployment of one piece.
On the other hand, allowing AI to drive is itself very radical. In the design of end-to-end large models, if a gradual approach is adopted, the three networks will have both emphasis and collaboration, which can increase more interpretability and controllability, and the allocation and deployment of computing power will also be more reasonable. At least during the debugging process, we are more likely to know where the problem lies.
21st Century Business Herald: Does One Piece have its own advantages end-to-end, and what are the challenges?
Li Liyun: A large One piece model on the car end may take effect quickly, so the outside world may think it has the potential to overtake on bends. But it has significant side effects - in the future, as the amount of data increases, the limited computing power in the car may not be able to handle so much data, which could bring many challenges.
21st Century Business Herald: Three networks cannot train together as quickly as One piece. How can Xiaopeng solve this problem?
Li Liyun: In terms of methodology, slow is fast. I now agree more with cloud based models like Open AI, which are completely one piece intelligent agents. So we will lay out a large model in the cloud and consider the safety guarantee of interpretability on the vehicle side.
Although the effectiveness is a gradual process, we do not need to repeat the construction, as the upper limit will be higher. The parameters of cloud models will be 80 to 100 times higher than those of the current car end. By the end of 2025, our cloud computing power will reach over 10 EFlops, an increase of 2.6 times compared to the 2024 plan.
21st Century Business Herald: In May of this year, Xiaopeng announced the completion of 100% mapping. There is a view that after Xiaopeng achieved the ultimate goal of 'no picture', the intelligent driving force went on to develop end-to-end, and the route was relatively conservative.
Li Liyun: At the beginning of our research and development, we had some end-to-end pre embedded components. To achieve true graph free, graph free means generalization, which means car companies need to have a certain level of understanding ability. Therefore, we started from the beginning of graph free (end-to-end), graph free