Blending Worlds: How 3D Drawing Promotes Observation and Discovery
The highest bandwidth input into the human brain is the visual system, so how can we leverage this phenomenon to help humans communicate with computers? And better yet, how can we build computer programs that help us better observe and interpret the world around us? For Professor Daniel F. Keefe, the answer to blending these two worlds lies at the intersection of art and virtual reality (VR).
“In my life, there are certain things I have learned over the years that change the way I look at the world,” Keefe said. “Learning to draw and learning about physics have both had that impact. It makes you revisit the world and understand it in a different way. It is one of the most fundamental forms of human creativity and is a special process.”
A world-renowned researcher in visual computing, Keefe has been exploring how 3D drawing in a virtual space can aid observation and discovery since his days as an undergraduate at Tufts University in the late 1990s. A self-proclaimed undercover artist in the engineering program, Keefe made a point to take as many art classes as he could while pursuing his degree in computer science, which eventually led him to the world of virtual reality and computer graphics during a summer program with NASA. After discovering his fascination with visual computing, Keefe went on to pursue his PhD at Brown University, which was conveniently situated near the Rhode Island School of Design.
“One of the very first projects that I did during my PhD program was a small group project on 3D painting in virtual reality. That project ended up being my first ever research publication on a system called CavePainting, which is the precursor to my current work. That paper came out in 2001, so this work has been going on for almost 25 years. I did my dissertation on this topic and have used 3D painting as a tool in many different applications.”
Since planting those roots, much of Keefe’s research has focused on bringing art into virtual spaces in a meaningful way.
“The biggest thing that VR can add to this practice is the third dimension. What we can create in VR has the immediacy of working with a paper and pencil, but you can work off the page and situate it into a space. You can scale and inhabit your creations in a true 3D space. It gives you a way to move the power of a sketch pad into your environment. Once you include observation in these tools, you can situate your creations in the world.”
As a fundamental form of human creativity and inquiry, the practice of drawing from real-world observation has opened the doors to scientific discoveries for tens of thousands of years. In order to draw, an artist must first learn to observe, making drawing a fundamental practice of studying and interpreting a subject. Blending this practice into a VR environment provides important tools for more impactful observation as the user can fully immerse themselves in their creations.
What if a surgeon could more deeply explore their patient’s anatomy in a virtual space to plan their next move during a procedure? Or if a field scientist could place their field notes and sketches into their physical space to get a more accurate representation of their environment? 3D drawing presents endless opportunities and applications that could improve how humans interpret and learn from the world around them. However, most VR applications fall short when blending the real world and virtual spaces, completely removing users from their physical surroundings.
“Currently, we do not use VR tools in a way that meets their potential. If you go online and you search for examples of what people have done with Google Tilt Brush and tools of that nature, you find a lot of very quick sketches that look like doodles and cartoons. When people move into VR headsets to do a creative task like drawing, they are shutting out the real world, so all that is left to draw are things that come from your imagination. I want to expand the opportunities in this space and use drawing as a mode of study and observation.”
In order to bridge this gap, Keefe is advancing the underlying software that enables 3D drawing, while simultaneously exploring the best ways to teach students how to make real-world observations in a virtual space effectively. Now in the second year of a $600K grant from the National Science Foundation, Keefe’s latest 3D drawing project is a collaboration between his team at the University of Minnesota and Macalester College’s Associate Professor Bret Jackson, who earned his PhD from Minnesota in 2014 with Keefe as an advisor.
On the technological side, the team has developed a technique called 3D sighting, which translates a common art technique into a blended virtual environment. When learning to draw, students are taught to take visual measurements of their subject by using a pencil as a guide. If you want to draw a human figure with correct proportions, then it helps to know how tall the torso or the legs are relative to the head. You can learn to measure this visually, from a distance, by looking carefully at a human. First, close one eye and hold a pencil vertically at arm's length then align the tip of the pencil with the top of the head and move your thumb along the pencil until it lines up with the chin. The distance between the tip of the pencil and your thumb is now equivalent to "1 head". Keeping your thumb in place and your arm at the same distance from your eye, you can now move the pencil to align with other parts of the body. If your subject is an adult, you will probably find that they are about 7 to 7.5 "heads" tall. These techniques help judge relative proportions while drawing. Keefe’s team has been working to translate this idea into a mixed reality software using 3D trackers to line up points of interest from multiple angles. This technique is one of many Keefe is putting into practice with a semester-long interdisciplinary course on 3D drawing.
“This spring, we taught an interdisciplinary class on 3D drawing that was a joint effort between the Department of Computer Science & Engineering (CS&E) and the Department of Art at the University of Minnesota. It was the first full-semester course on 3D drawing in the world. This course is explicitly integrated with our research, so students are getting the most recent tech developments in real time as we create them. We also put some of that curriculum out online so it is publicly available for others to replicate, and we will continue to add to that course and make it better.”
The philosophy of putting theory into practice is central to visual computing and the larger human-centered computing group at the University of Minnesota. By testing software in real-time with students, Keefe has the opportunity to optimize 3D drawing for human use, ensuring this technology will have uses and applications that touch multiple domains in the real world.
“Ultimately, what we are interested in is the user being able to insert their own interpretation to something in the real world. As a human-centered project, we are not so interested in creating the perfect model; we want the human to be involved with what they are seeing. That is what will help us bring drawing to the next level.”
Interdisciplinary collaboration is at the heart of this project and central to visual computing as a whole. The 3D seminar was hosted in the Art Technology and Science (ArTeS) studio, an initiative co-founded by Keefe and Professor Diane Willow in the Department of Art. Willow played a key role in the seminar and helped bring an artist’s perspective to the curriculum and VR techniques. The course itself had student representatives from four different colleges across the University of Minnesota, which was ranked the No. 1 public University in the Times Higher Education’s Interdisciplinary Science Ranking.
“When I think of why I came to UMN, it was because of people like Victoria Interrante (CS&E Professor and Director of Graduate Studies for Cognitive Science) who proved that you can collaborate across disciplines and colleges in the University. There are a number of art and design schools across the country, but it is more rare to have those types of programs at an R1 university. We have a great set up and we can work with colleagues in those art and creative spaces. You have to really be committed to interdisciplinary collaboration and recognize that it is necessary for this type of work to succeed.”
This interdisciplinary work is getting a boost this fall as the Shepherd Labs building on the west bank of campus reopens with a new look and new tenants. In addition to the existing spaces for the Minnesota Robotics Institute, two floors have been redesigned into a more collaborative space that houses the entire human-centered computing branch of the department, including visual computing faculty as well as members of the GroupLens Research Lab. Previously spread across three buildings on campus, this new space will open the doors for collaborations across disciplines within computer science and beyond.
“This move to Shepherd Labs is so incredible, because we are getting closer ties to the robotics group, as well as the other human-centered computing folks in GroupLens. Having that kind of interdisciplinary and interconnected expertise within shouting distance is really special and one-of-a-kind. It is really a human-centered, interactive group that crosses a number of computer science disciplines. We collectively have the ability to mix the physical and the virtual in this new research space, which will position us well for future research endeavors.”
One such endeavor is a blossoming collaboration between Keefe and CS&E Assistant Professor Karthik Desingh who specializes in robotics. While Keefe focuses on aiding real-world observation within a virtual environment, Desingh is working to improve how robots observe their surroundings.
“Karthik and I are working on similar problems. He works with Spot the robot dog to observe physical surroundings in order to walk around to engage with a space. Directing the robot around the room is not so different from guiding an artist around a subject. It is a parallel construction that we are trying to build upon with some of our students. It really ties back to the bigger picture vision of enhancing observation of real and virtual spaces.”
Whether it is with a VR headset or a robot, CS&E researchers are committed to exploring how human-computer interactions can facilitate discovery and improve how we move in the world.
“Our project is about drawing, but it really looks at a bigger question: how do humans work creatively with computers in the future? Drawing is an awesome use-case to study how to move between the physical and digital world and intermix human and computer insights. We want to bring all of these aspects together and do something with it, create something tangible.”