This week I read Fei-Fei Li’s autobiography The World I See, which devotes a large portion to the story of ImageNet (what else — it is the work that made her name).

I used to be ignorant and didn’t give Fei-Fei Li enough respect, always thinking her contribution was just a dataset with little technical depth, and that she wasn’t the inventor of any of the many algorithms. After reading this book and learning the full story behind the dataset — the challenges and doubts encountered along the way — I deeply admire her.

First, about the dataset itself. Fei-Fei Li is not a typical computer scientist: her undergraduate and PhD research intersected with neuroscience and brain science, focused mostly on cognition. During her PhD she realized how critical dataset scale is for algorithms — an algorithm’s capability needs a sufficiently large dataset to support it. In 2004 she built a dataset, Caltech 101, with 9,000 images, which was already the world’s largest classification dataset at the time. After finishing her PhD, she happened to learn about a project called WordNet, which catalogued concepts in the world and their relations, and had a spin-off called ImageNet intended to add image information to concepts. That ImageNet project was in fact not successful, but it sparked her interest: she wanted to borrow the idea and build a dataset with tens of thousands of categories.

When she started building the dataset she had just become an AP at Princeton and was still under plenty of tenure pressure. Most peers she talked to couldn’t quite understand what she was doing — everyone agreed data was important, but not that academia needed such a large dataset. Driven by her own conviction, she and her first student, Jia Deng, began the annotation work. It was through various engineering optimizations and the rise of crowdsourcing platforms that the dataset gradually came together. Even after it was built, it didn’t immediately attract attention. Fei-Fei Li kept promoting it at conferences, handed out ImageNet-branded souvenirs, gave talks at different schools, and organized an annual classification competition (the first two years had rather mediocre results). It wasn’t until 2012, when AlexNet demonstrated the enormous advantage of deep learning on ImageNet, that the dataset became widely used alongside the rise of deep learning.

Without Fei-Fei Li’s earlier research experience, without Google Search and Amazon’s crowdsourcing platform, without Hinton’s persistent pursuit of deep learning, it’s hard to say what deep learning would look like today. One could say ImageNet set the direction for deep learning’s early years, with classification as the most universal task for exploring network architectures and learning paradigms.

Other

• VLM4VLA: Revisiting Vision-Language Models in Vision-Language-Action Models — VLM pre-training is necessary but not sufficient for VLAs; their general capabilities do not fully predict VLA performance, and VLAs need representations that go beyond existing VLM capabilities. Fine-tuning with embodied auxiliary tasks does not improve VLA performance — existing embodied VQA tasks don’t match downstream manipulation needs. The VLM’s visual encoder is the main bottleneck for VLA performance; injecting control-related supervision into the visual module yields consistent gains. The domain gap between VLM pre-training objectives and embodied action planning needs to be bridged.