CRAY Colloquium: Human-AI for Physical World Accessibility

The computer science colloquium takes place on Mondays from 11:15 a.m. - 12:15 p.m. This week's speaker, Jon Froehlich (University of Washington), will be giving a talk titled "Human-AI for Physical World Accessibility"

Abstract

“Does the café entrance look accessible? Are there curb ramps along my route?” Digital maps have revolutionized how people move about the physical world but fail to support individual needs—routing wheelchair users along non-traversable paths, showing maps without screenreader support, or recommending inaccessible activities. In my lab, we create Human-AI systems that leverage human insight with computer vision and emerging GeoAI methods to transform physical world accessibility. I will describe our ambitious vision to map and assess every sidewalk in the world via remote crowdsourcing and computer vision (Project Sidewalk, RampNet), to make street-level imagery accessible via multimodal LLMs (StreetReaderAI), and to support complex physical tasks like cooking and sports via real-time visual augmentations in AR headsets (CookAR, ARSports). We envision a new class of interactive accessibility agents that sense, understand, and reason about physical spaces in real-time, providing personalized guidance adapted to each user's unique abilities and preferences.

Biography

Jon is a Professor in the Allen School of Computer Science at the University of Washington (UW), a part-time Visiting Faculty Researcher at Google Research, and co-founder of projectsidewalk.org, a Crowd+AI platform for urban accessibility analytics. His research in Human-Centered AI and Accessibility has produced 140+ scientific publications; 23 have received awards, including seven Best Papers at CHI and ASSETS—the top venues in their respective areas—and a 10-Year Impact Award at UbiComp. His work has been honored with the Sloan Fellowship, Google Faculty Award (3x), NSF CAREER Award, UW College of Engineering Outstanding Faculty Award, and the PacTrans Outstanding Researcher Award. At UW, he directs the Makeability Lab under the broad mission “to build and study interactive technology for a social purpose.”