VR + Robotics: Enabling Collaboration Between Humans and Robots
Department of Computer Science & Engineering PhD student Walter Sands is integrating virtual reality (VR) observational tools with robotics to facilitate collaborative work between humans and robots. Using VR headsets that allow the user to see their real-world environment alongside graphics, Sands is developing the ability for the user to simultaneously see what a robot sees in order to foster collaboration.
“We have had a long line of research doing 3D sketching in VR,” Sands said. “It brings the concepts of sketching a real-life subject into a virtual space so you can do 3D sketches of objects and locations. From there, I was interested in adapting that technology to work for other versions of perceptions that fit in with this pass-through VR capability. We are combining that technology with Spot, the robot dog from Boston Dynamics.”
This interdisciplinary project is a joint venture between Sands and his advisor, Professor Daniel Keefe’s Interactive Visualization Lab, and PhD candidate Xun Tu from Assistant Professor Karthik Desingh’s Robotics: Perception and Manipulation (RPM) Lab. Sands is working to integrate the VR system with Spot’s code so the two can communicate with each other in a shared coordinate space.
“Something that we have toyed with is trying to utilize this work in a search and rescue situation where the robot could go places that are not conducive to humans, like a wreckage site,” Sands said. “That would allow people to get new information that they couldn’t previously attain while also continuing their own important work during those high stress situations. This technology allows you to accomplish two goals at the same time.”
In August 2025, members of CS&E’s visual computing and human-centered computing research divisions moved into the newly renovated space in Shepherd Labs, which is also home to CS&E’s robotics researchers. This project is just one example of the new collaborations that have been fostered in this dynamic new space.
“My office is now across the way from Kathik’s lab, so I started to interact with them more. It makes it so much easier to test, debug, and work together on this project. I think the new space as a whole gives all of us more opportunities to do collaborative, interesting things instead of being siloed.”
In the future, Sands will be demonstrating this integrated technology with Spot at the Bell Museum. The headset users will work to sketch a woolly mammoth in a room that has chairs and other obstacles in the way. Spot will help the user identify where obstacles are and move them out of the way as the user moves around the mammoth.
“Spot sends video footage to the headset user so they can see the robot’s view alongside the user’s view. The second robot view is currently a 3D representation using a point cloud overlay within the VR environment. The user can transition to a full screen view that would black out what you are seeing and transition to a view of what Spot sees, allowing you to go back and forth between your view and the robot’s view.”
Learn more about Sand’s research at the IV Lab Website.