Companies allowed to test autonomous vehicles in California with a safety driver reported driving nearly 2.9 million miles during the latest reporting period, an increase of more than 800,000 miles from the previous reporting cycle, the California Department of Motor Vehicles reported this week.

The state’s DMV released 60 annual reports submitted by companies testing vehicles in autonomous mode on California public roads during the reporting period – Dec. 1, 2018 through Nov. 30, 2019. Reports include the total number of disengagements, the circumstances or testing conditions, location, and total miles traveled in autonomous mode on public roads for each permit holder.

A disengagement can occur when a failure of the technology is detected, or when a safety driver decides to take control of the vehicle. The DMV said that reports provide insights on a company’s testing activities in California, “but are not intended to compare one company with another or reach broad conclusions on technological capabilities.”

Under California’s current regulations, companies are not required to report testing on private roads or test tracks, testing that occurs out of state, below SAE Level 3, or testing done in simulation. The DMV said 24 permit holders reported they did not test autonomous vehicles on California public roads. Two companies – Roadstar.AI and Xmotors.AI, failed to submit a disengagement report, and their permits were revoked on Feb. 26, 2020.

Currently 64 companies have valid permits to test autonomous vehicles with a safety driver on California public roads, and one company (Waymo) has a permit for driverless testing. With its driverless testing, Waymo reported no miles driven on public roads for this permit.

Similar to the 2018 reports, Waymo logged the most miles driven during the period – it recorded 1.45 million miles driven autonomously, with 110 disengagements. Next up was Cruise LLC, which drove 831,039 miles with 68 disengagements. Combined, those two companies represented 79% of all of the autonomous miles driven during the period.

Miles driven

The number of miles driven between the 2019 reports and 2018 reports indicate that many companies have ramped up their testing on California public roads. For example, Baidu USA ramped up its miles driven to 108,300.2 in 2019, compared with 18,093 miles in 2018. Waymo and Cruise also saw big increases in the miles driven between the two years, with Cruise logging 383,418 more miles in 2019 than in 2018, and Waymo increasing its miles driven by 182,550.

Companies reducing its miles driven were also shown in the report. This includes Apple, which reduced its miles driven in 2019 – it only logged 7,544 miles driven during the 2019 reporting period, compared with 79,745 miles in 2018. Aurora Innovation, Nissan, Phantom.AI, PlusAI, and WeRide also saw significantly fewer miles driven in 2019 compared with 2018.

This doesn’t necessarily mean that these companies are going away from the autonomous vehicle space, but rather they just show less testing on public roads. Many could be testing on private roads, testing in other states, or testing through simulation.

Company	Miles driven (2019)	Miles driven (2018)	Increase/decrease
AImotive	6,056	3,428	2,628
Apple	7,544	79,745	-72,201
Aurora Innovation	13,429.49	32,858	-19,428.51
AutoX Technologies	32,054	22,710	9,344
Baidu USA	108,300.2	18,093	90,207.2
BMW of North America	21.36	41	-19.64
Cruise LLC	831,039.88	447,621	383,418.88
Drive.AI	3,974.25	4,616.69	-642.44
Lyft	42,930.96	0	42,930.96
Mercedes Benz R&D North America	14,238.06	1,749.39	12,488.67
Nissan North America	2,277.4	5,473	-3,195.6
Nullmax	2,430	3,036	-606
Nuro	68,761.94	24,680	44,081.94
NVIDIA	7,218	4,142	3,076
Phantom AI	1,125	4,149	-3,024
PlusAI	1,880	10,816	-8,936
Pony.AI	174,845.29	16,356	158,489.29
Qualcomm	2,164.19	240.02	1,924.17
SAIC Innovation	2,229.66	634.03	1,595.63
SF Motors	3,492.8	2,561	931.8
Telenav	22	30	-8
Tesla	12.2	0	12.2
Toyota Research Institute	1,817	381	1,436
Udelv	706.55	219	487.55
Valeo North America	99.56	0	99.56
Waymo	1,454,137.32	1,271,587	182,550.32
WeRide	5,917	15,440.8	-9,523.8
Zoox	67,015	30,764	36,251
Total	2,855,739.11	2,001,370,93	854,368.18

Miles per disengagement

This is a tricky area to analyze – while it’s easy to take the number of miles driven and divide by disengagement to give a sense of how many miles a car has driven before a safety driver disengaged, this does not tell the whole story. Safety drivers and companies have different policies and procedures for when to disengage, as outlined by the company’s individual disengagement reports. Sometimes this can be weather related, sometimes it’s software, sometimes it’s another reckless or aggressive driver on the road. Subjective decisions by safety drivers make these numbers less solid in terms of deciding whether one company is better than another.

However, we are presenting these numbers to show whether an individual company has progressed in increasing the number of miles driven per disengagement between the 2018 and 2019 reporting periods. It is true that company policies could have changed to become more or less cautious during those periods – we would have no way of knowing this, however.

The “leader” in the clubhouse would be Baidu, which reported 18,050 miles driven per disengagement, followed by Waymo (13,219.43 miles/disengagement), Cruise (12,221.17 miles/disengagement), and AutoX (10,684.67). These companies are the only ones that were able to log more than 10,000 miles per disengagement.

Baidu’s leap into the top spot has raised some eyebrows. For example, in this blog post, the author questions Baidu’s 8,697% increase in its disengagement rate. Other sites, including The Verge, write that the reports are the only real glimpse that the public (and the media) have in terms of progress on autonomous vehicles.

Responding to the reports, Waymo put out a series of seven tweets commenting on the California disengagement reports and the process:

“We appreciate what the California DMV was trying to do when creating this requirement, but the disengagement metric does not provide relevant insights intot he capabilities of the Waymo Driver or distinguish its performance from others in the self-driving space.”

“While much of the development, learning, and validation of the Waymo Driver comes through billions of miles driven within our simulation environments, our real-world driving experience is primarily outside of California, in markets like Detroit and Phoenix.”

“Most of our large-scale real-world driving, which is critical for full-system validation (including validating the realism of our simulator) comes from Phoenix. Indeed, our production releases are already serving thousands of Waymo riders in Phoenix.”

“This real-world production performance is largely unrelated to our California testing outputs. At this stage our real-world driving in California is predominately engineering development, and not production releases.”

“This is even more the case in 2019 and 2020, as we are now developing our 5^th-generation Waymo Driver in Silicon Valley, SF and LA. We don’t think CA disengagement data should be used to compare performance, or judge readiness or competency.”

“We also expect our values to fluctuate depending on release timing, geography and other non-engineering reasons, which invalidates time-series comparisons.”

“Later this year, we’ll share more on the safety framework we’ve developed to ensure the safe performance of our self-driving technology as we continue to scale our fully driverless operations.”

Company	Disengagements (2019)	Miles/disengagement (2019)	Miles/disengagement (2018)
AImotive	26	232.92	201.60
Apple	64	117.88	1.10
Aurora Innovation	141	95.24	99.90
AutoX Technologies	3	10684.67	190.8
Baidu USA	6	18050.03	205.60
BMW of North America	8	2.67	4.60
Cruise LLC	68	12221.17	5,204.90
Drive.AI	75	52.99	83.90
Lyft	1667	25.75	0
Mercedes Benz R&D North America	2054	6.93	1.5
Nissan North America	47	48.46	210.5
Nullmax	70	34.71	44.6
Nuro	34	2022.41	1028.3
NVIDIA	655	11.02	20.10
Phantom AI	43	26.16	20.70
PlusAI	2	940	54.40
Pony.AI	27	6475.75	1022.30
Qualcomm	37	58.49	2.4
SAIC Innovation	40	55.74	1.2
SF Motors	140	24.95	11
Telenav	3	7.33	6
Toyota Research Institute	2947	0.62	2.5
Udelv	444	1.59	3.8
Valeo North America	92	1.08	0
Waymo	110	13219.43	11017.00
WeRide	39	151.72	173.5
Zoox	42	1595.6	1922.8

Then there’s Tesla. In the 2018 reports, the company announced they were testing on private roads, but in the 2019 reports they have driven a grand total of 12.2 miles with zero disengagements. This gives a divide-by-zero figure for their miles per disengagement, but it also indicates that it’s likely that their testing is continuing on private roads.

First-timers

First-time filers (those who received permits between December 2017 and November 2018) logged a total of 24,872.86 miles, the reports indicated.

Company	Disengagements	Miles driven	Miles/disengagement
Ambarella	52	3,161	60.79
Apex.AI	63	448.25	7.12
BoxBot	109	38.5	0.35
DiDi Research	8	12,279.42	1,534.93
Gatik AI	24	2,553	106.38
Intel Corp	165	1,295	7.85
RideCell	6	8.69	1.45
ThorDrive	27	5,089	188.48

Companies that have permits, but reported no autonomous testing on public roads, included:

• AAA NCNU (did not have active permits at the submission deadline)
• Aptiv Solutions
• Bosch
• Changan Automobile
• Continental
• CYNGN Inc
• EasyMile
• Faraday & Future Inc
• Ford
• Helm.AI Inc
• Honda
• Imagry Inc.
• Mando America Corporation
• Navya Inc.
• Nio USA, Inc.
• Renovo.auto
• Samsung
• Subaru
• TORC Robotics Inc
• TuSimple
• Udacity, Inc.
• Volkswagen Group of America
• Voyage

What does this mean?

These reports give us some insights into where different companies are progressing in their testing of autonomous vehicles, at least on public roads in California. Some have argued that these disengagement reports are misleading, while others have argued that the reports mean that the world isn’t ready for self-driving cars.

For us, it’s a nice set of numbers that indicate that progress is being made by some companies, as well as to see the types of issues that are causing safety drivers to disengage (weather, other drivers, software issues). Apart from that, the rest is subjective.