Leveraging AI in Real World Spaces with Augmented Reality

Department of Computer Science & Engineering (CS&E) Assistant Professor Zhu-Tian Chen is exploring how people can naturally collaborate with artificial intelligence (AI) in real-world spaces by designing augmented reality (AR) systems that make AI’s insights visible, tangible, and actionable in the physical world. His lab aims to take AI off screens and bring its capabilities into physical spaces via AR glasses, robotic arms, and headsets that allow you to interact with proxies of real locations.

“We all know AI is powerful and can improve our productivity, but in most scenarios, AI is constrained to the desktop or phone,” Chen said. “There are many other scenarios where we need to interact with the physical world, and I am working on problems where AI could help us in day-to-day physical tasks. This could be useful in many fields, such as manufacturing, medicine, and agriculture.”

One of Chen’s biggest projects is a collaborative effort between himself, Victoria Interrante, Evan Suma Rosenberg, and their various graduate students. Their paper, titled, “Can AR Embedded Visualizations Foster Appropriate Reliance on AI in Spatial Decision-Making? A Comparative Study of AR X-Ray vs. 2D Minimap” was recently accepted by the 2026 Association of Computing Machinery (ACM) Computer Human Interaction (CHI) Conference in Barcelona this April. This project tests different ways of presenting AI generated information on AR glasses in a way that best facilitates a decision making process without causing an overreliance on the technology.

“Imagine in the future that everyone is using AR glasses in their daily lives,” Chen said. “Those glasses would be AI assisted and could talk to you as you interact with the world. This could also be used in professional high-stakes scenarios, like for firefighters, security, or pilots. The problem is that AI is imperfect and can make mistakes, so how can the user evaluate the results and make decisions in real time? Our research looks at how the presentation of AI suggestions can make a huge impact on how users rely on the technology.” 

Chen’s team designed an experiment where users had to pick a coffee machine within a building to get their coffee in the shortest amount of time. The glasses would display information about the distance to each machine and approximately how many people were in line at each location. Users had one of two visual presentations of this information: 1) X-ray see-through tech where the information is embedded in the space; or 2) a 2D map displayed on the side. Users who interacted with the x-ray tech had better spatial reasoning, but tended to be over reliant on the AI suggestions, while users who had the 2D map had a better performance overall. 

“This work will provide some empirical knowledge about the impact of AI visualizations on human’s collaboration with AI. Future designers can better decide how user interfaces in AR should look.”

Another project is working on remote operation of a robotic arm to help with day-to-day tasks. In collaboration with robotics researcher Karthik Desingh, Chen uses an AR interface to use the robotic arm as a proxy. Both Chen and Karthik work in the nearly renovated space in Shepherd Labs, allowing for closer collaboration between the visual computing group and robotics group within CS&E.

“We envision in the future that you could purchase a robot arm and use it to give you a hand where you need it, like in the kitchen. The question we are working on is how can humans better interact with the robot arm so it does what you want it to do. We developed an AR user interface that allows you to better control the robot arm. In the future, we hope this can create a fluid, intuitive user experience. We want to build towards the ability to control the arm without having to demonstrate what the arm should do.”

Chen’s work with real-world proxy objects in AR has gained significant traction on Reddit and tech blogs. The Reality Proxy project allows AR users to interact with a “digital twin” or proxies of real-world objects that are typically difficult to interact with in a virtual space because of their size, distance from the user, or intricate components. Chen’s team used the example of a book shelf to demonstrate this technology, allowing users to more easily select books on the shelf via the proxy. 

“This is a fundamental mixed reality interaction technique and can be used in any scenario where users want to interact with real-world objects via proxies. It helps the user more intuitively select physical objects, which was previously difficult if the object was far away or near other objects.”

This type of technology could be applicable in architecture design, remote drone control, and any mixed reality application. The end goal is to make it easier for users to interact with a physical object while using an AR device. 

“We worked with the Department of Architecture to develop a design technology using this proxy technique. We used a drone to collect layout data of the University of Minnesota campus, then we reconstructed the campus in AR to give users the ability to look at real buildings and make edits and design decisions in AR. The proxy technique makes it easier to select specific objects and implement design options.”

Learn more about Chen’s work on his personal website.