MnRI Newsletter — April 2022
From the Director
Papanikolopolous,
Computer Science & Engineering
As we come to the end of the Spring semester, we want to share some exciting news about the M.S. Program. We currently have more than 150 applicants from all over the world (17% more than at the same time last year). Nineteen students have committed to attend the M.S. program as part of the third cohort in academic year 2022/2023. We also continue to learn about the placement of our graduates— from academia, including top Ph.D. programs, to industry, with a variety of leaders in the robotics landscape. For Spring 2022, the M.S. program in Robotics was in the top five M.S. programs in the college (with respect to student enrollment). We are very proud of the accomplishments of these students and eager to welcome our third cohort.
Youbing Wang and Saad Bedros are working on a project with the Timberwolves, creating a robotic system that follows the players and passes the ball to them. Recently, as I watched a demonstration, I was reminded of the importance of robotics in sports and entertainment. Multiple cameras and machine learning are used to monitor athletes as they train, helping coaches and the athletes themselves assess their performance and possible improvement. Many professional teams are also using mobile robots during football training. MVP, for example, has created the Robotic Self-Righting Tackling Dummy, which can move with speeds up to 16 mph. Toyota has created the Basketball Robot Cue3, which can shoot three-pointers. Also in the game are RoboCup, Boston Dynamics’ Atlas (gymnastics), Omron’s FORPHEUS (ping-pong), and others.
In the entertainment industry, the impact of robotics is profound but often not widely known. Jurassic Park’s T-Rex, for instance, is a large robotic system based on hydraulics and enclosed in foam rubber. It was so difficult to operate that it almost killed crewmember Alan Scott when the studio power went out. Disney roboticists (Disney calls them “imagineers”) have been developing intricate robotic actors for several decades, and Disney is one of the largest employers of roboticists. The use of immersive environments and robotics in combination with computer vision and graphics has revolutionized the film industry. For example, Sandra Bullock and George Clooney were filmed using big industrial robots in the film Gravity. The result was stunning!
MnRI is eager to support you all in research activities, not only in these areas but in medicine, agriculture, transportation, or anything else that may have a positive impact on human well-being. We look forward to your ideas.
Have a great rest of the academic year,
Nikos Papanikolopoulos
Minnesota Robotics Institute Director
LoCO: A Low-Cost, Open-Source Autonomous Underwater Robot
Junaed Sattar
Computer Science & Engineering
Autonomous underwater vehicles (AUVs) are a key tool in scientific and industrial work in marine and aquatic environments. They are used to explore shipwrecks, chart biological habitats, destroy subsea mines, inspect and repair undersea architecture such as pipelines or cables, and do a plethora of other tasks. However, AUVs are often expensive, large, and difficult to deploy, limiting their use to well-funded research groups, such as oceanography institutes, underwater robotics research labs, and university research teams. Additionally, AUVs are often not sold commercially and those which are, tend to be expensive to acquire and maintain. There are a great number of areas where the need for AUVs is outweighed by the challenges outlined above, such as use in marine science at a state and local government level, education at lower-funded universities and secondary schools, and in hobbyist development. If an AUV were available at a lower cost, with less overhead in deployment, and with less constraint on additions and modifications to the platform, those in these under-represented groups would be able to leverage the capabilities of an underwater autonomous agent to achieve their goals in research and education.
To this end, the Interaction Robotics and Vision Laboratory (IRVLab) at the University of Minnesota, under the direction of Assistant Professor Junaed Sattar, designed and built the LoCO AUV, a Low-Cost, Open, Autonomous Underwater Vehicle. Developed with support from a MnRI Seed grant for aquatic species conservation, LoCO is a human-portable, easy to deploy AUV built from approximately $4,000 worth of parts, largely off-the-shelf electronics, and is additively manufactured. While LoCO is not as specialized as more expensive AUVs, it boasts an impressive array of capabilities, from autonomous swimming in defined patterns to diver following and control by hand gestures. Its hardware is designed to run deep learning for perception, enabling a great number of applications in the future.
Moreover, the LoCO platform’s design lends itself well to modularity and modification, making it relatively simple to add new sensor payloads or other hardware. With the openness of the design, this AUV is well-suited as a tool for those without access to the resources required to acquire and deploy more expensive AUVs. The IRVLab has chosen to release the entire diagram of the LoCO AUV 3 software-hardware design of the LoCO platform under open-source licenses (GPLV3 for the code, Creative Commons Share-A-Like for the designs), and a number of institutions globally (e.g., in India, Singapore, and Finland) are building their own version of the robot at this time.
LoCO was designed with easing the development of underwater autonomous behaviors with a focus on human-robot collaboration. The design team is vastly multidisciplinary, with members coming from diverse backgrounds such as Computer Science, Electrical, Mechanical, Aeronautical Engineering, and Math. The robot is capable of on-board deep learning inference, leading to advanced capabilities. A number of open research problems have been addressed and validated on board LoCO. For example, former IRVLab Ph.D. student and now an Assistant Professor at the University of Florida Dr. Md Jahidul Islam’s doctoral dissertation work created methods for diver following, underwater gesture recognition, and underwater image enhancement. Dr. Islam’s work focused on the design of efficient convolutional and generative deeplearned models for such purposes, in the process addressing the data scarcity and specificity problems often faced in underwater robotics research. Ph.D. candidate Michael Fulton’s research in Robot Communication Via Motion has enabled LoCO to use motion-based gestures, or Kinemes, to communicate visually with its human companions. To further address the challenges of data scarcity in machine learning tasks, Professor Sattar and Ph.D. candidate Jungseok Hong have recently been working on zero-shot learning models to make it possible for the robot to detect objects of interest simply from textual descriptions. Ph.D. candidate Jiawei Mo’s research in localization and mapping (SLAM) in visually-degraded and GPSdenied environments has demonstrated promising results, enabling robot localization under one of the most extreme circumstances imaginable.
These capabilities are quite mature, and a number of other capabilities are under active development. Students at the IRVLab are working on ocular, light-based communication methods, user-defined gestural control, an “autopilot” system for stable control underwater, a gesture-based unified language for multi-human-robot (m/HRI) teamwork, and detection and mapping of underwater trash and invasive species colonies. One ongoing work of note is with Professor Changhyun Choi that is looking into using underwater manipulation for grasping and removing marine debris for the long-term preservation of aquatic ecosystems.
LoCO has been extensively field-tested, including in the lakes of Minnesota and the Caribbean Sea off the coast of Barbados. Also, it has generated strong interest outside of the University, with numerous high-school groups around Minnesota actively considering underwater robotics as part of their extracurricular activities. LoCO has just not been a platform for the IRVLab to forge ahead with for their own research agenda but has shown that affordable, open-source underwater autonomous robotics is indeed a possibility for a broad population of robotics enthusiasts.
For more information on the LoCO project, including design and software, please see our GitHub repository. Visit the IRVLab website and you can follow on Twitter at @irvlab.
Wearable Technology Laboratory (WTL)
Lucy Dunne
Apparel Design
The University of Minnesota Wearable Technology Laboratory (WTL), located in the UMN College of Design, was founded in 2009 and is Co-Directed by Drs. Lucy Dunne and Brad Holschuh. The lab is well known as a leader in the design, manufacture, and evaluation of garment-integrated technologies, and brings together experts in Apparel Design, Robotics, Mechanical / Electrical Engineering, and Human Factors and Ergonomics to investigate challenges and opportunities associated with wearable technology.
Work in the WTL focuses on both body-worn technologies and the form factor and wearability of these technologies. This dual focus is made possible by the interdisciplinary expertise represented in Dr. Dunne and Dr. Holschuh’s respective backgrounds and the lab’s diverse cohort of students. The WTL research portfolio includes textile-based sensing and actuation technologies, e-textile manufacturing and smart garment assembly methods, and human factors of wearable systems.
Brad Holschuh
Apparel Design
Garment-based form factors for wearable technology offer exciting opportunities that cannot be achieved in smartwatch or wrist-band form factors alone. Garments cover much of the body surface and are already part of our everyday life – by seamlessly integrating actuators and sensors into everyday clothing we can create augmented experiences without sacrificing the important qualities that define our clothing (comfort, fit, aesthetics, etc.). Examples of wearable technology systems that have been developed by the WTL include:
- Stitch-based strain sensors that enable perceptually-invisible kinematic sensing in garments (e.g., leggings that can dynamically measure knee angle during exercise) [1]
- E-textile circuits are manufactured using industry-standard apparel equipment to create entirely soft wearable electronics [2]
- Shape memory alloy (SMA) actuated garments to create spatially / temporally dynamic compression for health and wellness [3], [4]
- Entirely-soft exoskeletons combining soft sensors and actuators for children with upper-body mobility disorders [5]
References
[1] G. Gioberto and L. E. Dunne, “Garment-integrated bend sensor,” Electron. , 2014, doi: 10.3390/electronics3040564.
[2] M. T. I. Molla et al., “Surface-mount manufacturing for E- textile circuits,” in Proceedings - International Symposium on Wearable Computers, ISWC, 2017, doi: 10.1145/3123021.3123058.
[3] J. C. Duvall, N. Schleif, L. E. Dunne, and B. Holschuh, “Dynamic Compression Garments for Sensory Processing Disorder Treatment Using Integrated Active Materials,” J. Med. Device., vol. 13, no. 2, p. 021001, Mar. 2019, doi: 10.1115/1.4042599.
[4] R. Granberry, K. Eschen, B. Holschuh, and J. Abel, “Functionally Graded Knitted Actuators with NiTi-Based Shape Memory Alloys for Topographically Self-Fitting Wearables,” Adv. Mater. Technol., vol. 4, no. 11, 2019, doi: 10.1002/admt.201900548.
[5] A. Golgouneh et al., “Design of a Hybrid SMA-Pneumatic based Wearable Upper Limb Exoskeleton,” Proc. - Int. Symp. Wearable Comput. ISWC, pp. 179–183, Sep. 2020, doi: 10.1145/3460421.3478838
Awards and Major Grants
Changhyun Choi (Electrical and Computer Engineering) received the prestigious NSF CAREER Award for his research titled “Visual Manipulation Learning for Challenging Object Grasping.” The research seeks to significantly enhance the capabilities of robotic systems to grasp objects. The $538,863 project will develop novel computational algorithms that enable robots to visually understand scenes, learn to perform a proper manipulation action sequence, and adapt to different environmental settings. The project is supported by the cross-directorate Foundational Research in Robotics program, jointly managed and funded by the Directorates for Engineering (ENG) and Computer and Information Science and Engineering (CISE).
Maria Gini (Computer Science and Engineering) won the 2022 ACM/SIGAI Autonomous Agents Research Award. This prestigious prize recognizes years of research and leadership in the field of robotics and multi-agent systems. Prof. Gini has been an active member and leader of the agents community since its inception. She has been a consistent mentor and role model, deeply committed to bringing diversity to the fields of AI, robotics, and computing. She is also the former President of International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). Prof. Gini will be giving an invited talk at AAMAS 2022.
Andrew Lamperski (Electrical and Computer Engineering) received a $335,337 grant from the National Science Foundation, titled “Mechanics-Based Algorithms for Sampling, Control, and Learning in Non-Convex Domains.” This grant is from the NSF CMMI – Dynamics, Control and Sensor Diagnostics Program and will fund research that enables smart devices to adapt automatically to novel situations, with reliable guarantees of safe and efficient behavior. The research aims to make fundamental contributions to the development of a model-based reinforcement learning methodology that guarantees stability and near-optimal performance for a wide class of unknown nonlinear stochastic systems.
Demoz Gebre-Egziabher (Aerospace Engineering and Mechanics) and Lindsay Glesener (Physics and Astronomy) in collaboration with Mats Heimdahl (Computer Science and Engineering) are supervising a large group of students from various disciplines in CSE to build a CubeSat (10 x 10 x 30 cm satellite) known as EXACT. EXACT, sponsored by the Air Force Research Labs (AFRL), will fly a novel X-Ray spectrometer designed and built by a collaboration between the University of Minnesota, Montana State University, University of California, Santa Cruz, and the SouthWest Research Institute. The spectrometer will be used to explore novel positioning, navigation and timing techniques that rely on X-ray signals emitted by astrophysical objects. EXACT is being built in the University of Minnesota's Small Satellite Research Lab (SSRL) and is slated for delivery to the Air Force for launch in 2023.
Derya Aksaray (Aerospace Engineering) and Mechanics has been appointed an Associate Editor of the IEEE Robotics and Automation Letters (RAL). The scope of the IEEE RAL journal is to publish peer-reviewed articles that provide a timely and concise account of innovative research ideas and application results, reporting significant theoretical findings and application case studies in areas of robotics and automation.
Brad Holschuh (Design, Housing & Apparel, College of Design) received the 2022 University of Minnesota Award for Outstanding Contributions to Graduate and Professional Education. Prof. Brad Holschuh is the Director of the Human Factors and Ergonomics graduate program and is an Associate Professor of Apparel Design. An aerospace engineer by training, he teaches several interdisciplinary graduate courses and is appointed as faculty in four graduate programs across the Colleges of Design and Science/Engineering. He co-directs the Wearable Technology Laboratory, where engineers, designers, and human factors students work together to create a future that blends clothing and technology.
Nikos Papanikolopoulos (Computer Science & Engineering) received the 2022 University of Minnesota Award for Outstanding Contributions to Graduate and Professional Education. He was recognized for a long and outstanding tradition of quality graduate education in the areas of robotics and computer vision. His efforts have instigated the creation of the Minnesota Robotics Institute and its associated MS program in Robotics. Through extensive and dedicated guidance, his graduate students have become leaders in their respective fields and the program’s emphasis on research has propelled it to become globally recognized in the field.