Researchers use 3D motion capture to study Minnesota Zoo monkeys

Study holds new insights for neuroscience and behavioral science

The words “3D motion capture” might conjure up images of Hollywood stars decked out in green suits with ping pong ball-like markers attached to them. Techniques like this have brought to life memorable movie characters, such as Gollum of “The Lord of the Rings” and James Cameron’s “Avatar” cast.

Now, University of Minnesota researchers are doing something similar. Except they’re using motion capture to track the movement of snow monkeys at the Minnesota Zoo—and, they’re doing it without the ping pong balls.

Computer science & engineering assistant professor Hyun Soo Park is collaborating with Medical School neuroscience faculty Ben Hayden and Jan Zimmerman on the project. The researchers are working to enable “markerless” motion capture—tracking without attaching anything to the subjects—in order to better study how monkeys’ movement and social behavior relates to brain activity.

“We want to do this without the markers for the monkey because monkeys always tear off the [physical] markers,” explained Park. “The reason why we want to do this is that monkeys have a lot of similar behavior to humans—their social behavior, their individual social interactions with others, their interacting with objects—so there are a lot of things that we can study.”

The research could ultimately open new opportunities to study early behavioral markers, such as those for people with autism, schizophrenia, or obsessive-compulsive disorder.

The optics of motion capture

Dubbed “Open Monkey Studio,” the project started in 2017 with the researchers tracking one or two monkeys in a controlled environment in a lab. Within the Gemini-Huntley Robotics Research Lab, Park and his computer science team set up 64 cameras, which capture images, recognize limb and joint locations, then make a 3D reconstruction of the monkeys’ movement.

A monkey moving around in a controlled lab environment (left), and the researchers' motion capture system creates the corresponding images to track the monkey's movement (right). Credit: Hyun Soo Park

Meanwhile, Hayden’s neuroscience lab was responsible for measuring the monkeys’ brain signals so that the researchers could relate the signals to specific gestures. Hayden said the process was completely safe for the monkeys, and they were able to move around unconstrained in a large cage.

“We had two sets of data. One was brain signals, and the other one was motion signals,” Park said. “The project was about building the infrastructure that can correlate these two signals. What behaviors constitute which brain activities? That’s what we really wanted to understand from that."

Open Monkey Studio was supported by a Minnesota Futures Grant, and the researchers' paper was recently accepted to the Nature Communications journal. In 2020, the team received a National Science Foundation (NSF) grant to scale up their research by tracking monkeys in a larger, more natural environment.

Enter the Minnesota Zoo, which has a population of nearly 30 snow monkeys.

“It's a huge area where the monkeys are doing a lot of interesting natural activities,” Park said. “But the real challenge here is that the space is too large. If we just scale up the number of cameras for this space, maybe we'll end up with 500, 600 cameras.”

To solve this problem, the computer scientists must create a 3D reconstruction of the environment first, which is what Park and his graduate student, Yuan Yao, are currently working on.

Cameras do the work

While the research will likely reveal many insights for the neuroscientists, the motion capture infrastructure Park’s team has built is also an important step forward for research processes in the behavioral science field.

Typically, in order to study behavior, scientists observe animals and write down their subjective observations. According to Park, this method can pave the way for a more quantitative approach to gathering behavioral data. Essentially, the cameras do the work for the scientists.

“We’re providing a tool for data that has been measured before by manual visual inspections, but we are actually transforming this to the computational in order to give a more objective analysis of the behaviors,” he added.

Hayden, an associate professor in the U of M Medical School's neuroscience department, said that the ability to track animals more precisely can help in a large number of fields.

“For example, we anticipate [the research] will have benefits to veterinary medicine,” he said. “We can track the animals and tell how healthy they are, and develop novel treatments that can improve animal welfare. In addition, we can also help zoos to develop exhibits that are more pleasant for the animals that live there.”

This research was also supported by seed grants from the University of Minnesota’s Digital Technology Center and the Minnesota Robotics Institute.

Story by Olivia Hultgren