This week Nvidia released and open-sourced an autonomous driving VLA model, Alpamayo, and demonstrated its driving performance in real road environments at CES. Judging from the publicly released videos, the model runs fairly smoothly on complex urban roads, though there are still some safety interventions. As soon as the project was announced, it sparked a huge amount of discussion online — including whether it would challenge the moat Tesla has spent years building around autonomous driving. Both Elon and Tesla’s Ashok publicly responded to NV’s system. Since NV did not demonstrate the system’s capabilities at scale, and there is no single-take video, I’m inclined to believe the system’s capability is mediocre.

From the published technical report, Alpamayo is a fairly standard VLA framework that borrows heavily from techniques in the embodied AI field. By aligning the CoT reasoning with the trajectories output by the system, the model can explicitly supervise the causal chain of logic during training and deployment. The technical report also details how data should be annotated to prevent the system from causal confusion and superficial learning. It looks like this is the canonical autonomous driving exemplar under a standard VLA framework — academia will surely benefit a lot and can use it as a starting point for plenty of exploration (especially since NV also open-sourced a closed-loop simulation system; leaderboard-chasers are in for a treat).

As for the open-source model’s actual performance, it also fell short of expectations. On the one hand, although the technical report repeatedly emphasizes causal consistency, in practice many driving scenarios show no strict matching between CoT and trajectory output. On the other hand, it’s hard to see where the advantage of VLA lies in the showcased scenarios — a decent end-to-end model should handle them just fine. There is likely a huge gap between the open-source model and NV’s actual road testing, let alone the gap between NV and Tesla.

I do believe that VLA and CoT are important building blocks toward open-world L4 autonomous driving. The main reason is that such systems can leverage internet data, improving data efficiency through few-shot learning and gradually eliminating the long-tail problem of autonomous driving. For example, a VLA can establish the concept that various obstacles must not be collided with through its pre-training, thereby improving safety in driving scenarios. By annotating problematic data with CoT, we can retrain the VLA model so it understands the why, and therefore the how.

In the short term, Tesla’s system remains absolutely ahead: it has a publicly testable version, has accumulated a massive set of long-tail cases, and continues to accumulate at a very high speed. In the long run, as VLM and embodied AI technology develop, more companies will have a decent autonomous driving stack. Perhaps autonomous driving in the future really will be like the difference between iOS and Android — the difference between various systems may come down to handling long-tail cases. That small but important distinction determines consumer perception and the profit split in commercialization.

Other

• The papers One Sample to Rule Them All: Extreme Data Efficiency in RL Scaling and LIMR: Less is More for RL Scaling mainly discuss data curation in LLM RL training, emphasizing that data quality matters far more than quantity in RL. By constructing small but refined datasets, we can substantially boost model capability (especially for small models).
• The project How Much 3D Do Video Foundation Models Encode?, https://vidfm-3d-probe.github.io/, demonstrates that video diffusion models do indeed learn a great deal of scene geometry. On one hand, it directly justifies fine-tuning depth estimation out of diffusion networks; on the other, it broadens everyone’s thinking — diffusion really is an excellent pre-training task for 3D information.
• Large Video Planner Enables Generalizable Robot Control is another downstream application of video diffusion. It still looks fairly rough — more of a proof of concept — but it undeniably points out that video diffusion models are a rich mine of information worth digging into.