Past events

Abstract

Recent advances in frontier AI models and in self-driving vehicles have shown that learning at scale unlocks capabilities that smaller, narrower training regimes do not. Robotics stands to benefit from the same recipe, but there is no naturally occurring source of large-scale, high-quality robot interaction data. Efforts at generating such data at scale have produced manipulation capabilities well beyond what was possible a few years ago, but the resulting models remain brittle in ways we cannot fully explain. We still lack an understanding of how data, model architectures, and training regimes should combine to produce truly powerful robot foundation models.

I will illustrate the capabilities and limitations of such models with our own work on Flex-pi, a world action model trained by behavior cloning on teleoperated demonstrations.  I will then turn to our ongoing work on ToyBlocks, a simulated robot environment built to enable a scientific investigation of robot foundation models. ToyBlocks combines physically faithful simulation with automatic generation of tasks, demonstrations, and tests, so that questions regarding data scaling, model architecture, and training regime can be investigated under controlled conditions, and the answers transfer to real robots.

Biography

Dieter Fox is a Professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. He grew up in Bonn, Germany, and received his Ph.D. in 1998 from the Computer Science Department at the University of Bonn. He joined the UW faculty in the fall of 2000.

His research interests are in robotics, artificial intelligence, and state estimation. The goal of his research is to enable systems to interact with people and their environment in an intelligent way. A lot of his work focuses on perception and its connection to control, where he develops techniques to extract relevant information from raw sensor data, with applications in human activity recognition, 3D mapping and tracking, and robot manipulation and control.

Fox shares his time between UW, where he is the head of the UW Robotics and State Estimation Lab (RSE-Lab), and the Allen Institute for AI (Ai2), where he focuses on building foundation models for robotics. Prior to Ai2, Fox spent a portion of his time at Nvidia, where he led the company’s Robotics Research Lab in Seattle for nearly eight years.

Fox is a Fellow of the AAAI, ACM, and IEEE, recipient of the IEEE RAS Pioneer Award and the IJCAI John McCarthy Award, and served as an editor of the IEEE Transactions on Robotics.

Abstract

How do we ensure that sharing network measurements does not disclose information that should not be disclosed? Today's approaches to this problem rely on best effort and, ultimately, trusting that network data is not misused. This talk explores a different technical approach that involves sound, machine-checked guarantees around non-disclosure. The talk introduces an eBPF-based approach to encode disclosure policies and presents the design, implementation, and evaluation of a general—yet disclosure-controlled—approach to carry out flexible measurements over packet streams. The talk also reports on an early deployment experience of this approach that builds on the Patchwork project.

Biography

This event is presented by the College of Science and Engineering Data Science Initiative.

This week's speaker, Ken Perlin from New York University, will be giving a talk titled, "Future Reality: A research agenda for XR glasses."

Abstract

But what if we could remove those limitations and work with the kinds of XR glasses that will only become available in another 5 to 10 years? At the NYU Future Reality Lab we are using the technology of today to emulate the XR glasses of tomorrow. We are developing a research platform called VELDT (Visually Emulated Lightweight Display Testbed) which allows research participants to experience the future of XR now, and to collaborate with one another as though that future has already arrived.

We are particularly interested in how an individual or a small group of collaborators can use existing notebook computers together with powerful future XR glasses in a seamless way, since the combination of those two technologies makes particularly good use of the complementary features of both. Our lab is exploring the challenges and opportunities of that future reality, and the many new and exciting opportunities for HCI research that it enables. For example, one of our research projects examines how collaborators, whether co-located or remote, can use a mix of keyboard, trackpad, eye gaze and spatial micro-gestures to interact with a shared “holographic” workspace which contains virtual objects that are present both on the computer screen and in the surrounding 3D space. We also study ways to leverage artificial intelligence (AI) to manage such shared spaces, as well as to ensure data privacy, optimize layout readability across different environments, intelligently filter notifications to reduce cognitive load, and generate high fidelity expressive avatars of remote collaborators.

Biography

Social Hack is a FREE 24-hour hackathon hosted by Social Coding on October 24–25 at Coffman Memorial Union! We’re bringing together UMN students of all experience levels to build projects, network with industry professionals, and compete for prize money! We’ll also have free food throughout the event!

Register today!

The Knowledge Computing Lab is hosting Dalton Lunga and Jacob Arndt from the GeoAI group from Oak Ridge National Laboratory for a special seminar titled, "Advancing Geospatial AI: From Research to Mission Impacts." Their work spans geospatial AI, foundation models, computer vision, and remote sensing. 

Abstract

The Geospatial Science and Engineering Division at ORNL advances GeoAI by combining large-scale remote sensing data, high-performance computing, and fundamental AI research to address complex scientific and national security mission challenges. This talk will highlight our capabilities in developing large-scale mission datasets, multimodal geospatial foundation models, efficient learning methods, and benchmarks that advance AI research and enable mission-aligned evaluations. Through applications in man-made and natural hazards, satellite image authenticity, vision-language reasoning, and geospatial decision support, we will demonstrate how mission-driven challenges are inspiring AI innovation and transforming ORNL’s national security science programs.

Biographies

Dalton Lunga, Ph.D., is a Staff Scientist at Oak Ridge National Laboratory, where he leads the Geospatial Artificial Intelligence (GeoAI) Group in the Geospatial Science and Engineering Division. His research focuses on computer vision, machine learning, high-performance computing, and remote sensing, with emphasis on imagery analysis. He chaired the IEEE Geoscience and Remote Sensing Society (GRSS) Machine Learning and Imagery Analysis Working Group from 2021 to 2024 and currently serves as Chair-Elect of the GRSS Technical Committee on Image Analysis and Data Fusion. He has been an Associate Editor of IEEE Geoscience and Remote Sensing Letters since 2020 and has guest-edited several GRSS special issues. He is also guest editor to the special issue of ACM Transactions on Spatial Algorithms and Systems on GeoAI Foundation Models and Their Applications. He is lead editor of the Elsevier 2026 book on GeoAI for Earth Observation Imagery: Fundamentals and Practical Applications.

 

Jacob Arndt is a Research Scientist in the Geospatial Science and Engineering Division at Oak Ridge National Laboratory (ORNL). Prior to joining ORNL, he earned both his bachelor of science in geography (2016) and master's in geographic information science (2018) from the University of Minnesota Twin Cities. His research lies at the intersection of computer vision, geographic information science, and machine learning. Most recently, his work focuses on deepfake detection and remote sensing image authenticity, the development of datasets and benchmarks for geospatial applications, and machine learning-assisted analysis and QA/QC for maritime navigation charts.

 

This event is presented by the College of Science and Engineering Data Science Initiative.

This week's speaker, Subbarao Kambhampati from Arizona State University, will be giving a talk titled, "On the Mythos of LLM Thinking Tokens."

Abstract

Ever since DeepSeek R1, the intermediate tokens output by the Large Reasoning Models (LRMs) have been ascribed several magical properties-- that they provide an interpretable window into the "thinking" of the model, that their length is a reliable indicator of the computational complexity of the task, and that the main contribution of RL-based post-training is to improve model's thinking--as evidenced by the average length of the thinking tokens increasing during training. In this talk, I will discuss several recent efforts from our group that systematically dissect these claims and offer counterintuitive conclusions. I will also discuss implications of these efforts to agentic systems, as well as some alternate ways to get LLMs to provide verifiable traces of their problem solving--that can be used in safety critical applications.

Biography

This event is presented by the College of Science and Engineering Data Science Initiative.