CRAY Colloquium - Where the Data Runs Out: ToyBlocks and the Science of Robot Foundation Models
Abstract
Recent advances in frontier AI models and in self-driving vehicles have shown that learning at scale unlocks capabilities that smaller, narrower training regimes do not. Robotics stands to benefit from the same recipe, but there is no naturally occurring source of large-scale, high-quality robot interaction data. Efforts at generating such data at scale have produced manipulation capabilities well beyond what was possible a few years ago, but the resulting models remain brittle in ways we cannot fully explain. We still lack an understanding of how data, model architectures, and training regimes should combine to produce truly powerful robot foundation models.
I will illustrate the capabilities and limitations of such models with our own work on Flex-pi, a world action model trained by behavior cloning on teleoperated demonstrations. I will then turn to our ongoing work on ToyBlocks, a simulated robot environment built to enable a scientific investigation of robot foundation models. ToyBlocks combines physically faithful simulation with automatic generation of tasks, demonstrations, and tests, so that questions regarding data scaling, model architecture, and training regime can be investigated under controlled conditions, and the answers transfer to real robots.
Biography
Dieter Fox is a Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. He grew up in Bonn, Germany, and received his Ph.D. in 1998 from the Computer Science Department at the University of Bonn. He joined the UW faculty in the fall of 2000.
His research interests are in robotics, artificial intelligence, and state estimation. The goal of his research is to enable systems to interact with people and their environment in an intelligent way. A lot of his work focuses on perception and its connection to control, where he develops techniques to extract relevant information from raw sensor data, with applications in human activity recognition, 3D mapping and tracking, and robot manipulation and control.
Fox shares his time between UW, where he is the head of the UW Robotics and State Estimation Lab (RSE-Lab), and the Allen Institute for AI (Ai2), where he focuses on building foundation models for robotics. Prior to Ai2, Fox spent a portion of his time at Nvidia, where he led the company’s Robotics Research Lab in Seattle for nearly eight years.
Fox is a Fellow of the AAAI, ACM, and IEEE, recipient of the IEEE RAS Pioneer Award and the IJCAI John McCarthy Award, and served as an editor of the IEEE Transactions on Robotics.
