Robotics Colloquium: Guest Speaker Kostas Bekris

Title: “Towards Closing the Perception-Planning and Sim2Real Gaps in Robotics”

Abstract:  There is now wider deployment of robotics solutions across applications, such as logistics, service and field robotics. There are still critical gaps, however, that limit the adaptability, robustness and safety of robots at: (a) the interface of domains, such as perception, planning/control and learning, that must be viewed in a holistic manner, and (b) the sim2real gap, i.e., the deviation between internal models and the real world.
This talk will first describe efforts in tighter integration of perception and planning for vision-driven robot manipulation. We have developed high-fidelity open-vocabulary 3D segmentation as well as high-frequency tracking of rigid bodies’ 6D poses - without needing CAD models or human annotations - by utilizing progress both in deep learning, LVMs and pose graph optimization. These solutions together with appropriate shared representations, tighter closed-loop operation and – critically - compliant end-effectors are unblocking the deployment of full-stack robot manipulation systems. This talk will provide examples of using such tools for robotic packing, assembly under tight tolerances as well as constrained placement given a single demonstration that generalizes across an object category.
The second part of the talk is motivated by tensegrity robots, which combine rigid and soft elements, to achieve safety and adaptability. They also complicate, however, modeling and control given their high-dimensionality and complex dynamics. This sim2real gap of analytical models made us look into reinforcement learning (RL) for controlling robot tensegrities. RL is challenging in this domain, however, due to data requirements. Training RL in simulation is promising but is blocked again by the sim2real gap. For this reason, we are developing a differentiable physics engines for tensegrity robots based on first principles. It is trained with few example ground truth trajectories from the real robot. It then provides accurate-enough simulations to train a controller that is directly transferrable back to the real system. We report our success in such a real2sim2real transfer for a 3-bar tensegrity robot.

Bio: Kostas Bekris is a Professor of Computer Science at Rutgers University in New Jersey and an Amazon Scholar with Amazon Robotics since 2019. He is working in algorithmic robotics, where his group is developing algorithms for robot planning, learning and perception especially in the context of robot manipulation problems, robots with significant dynamics and a focus on taking advantage of novel soft, adaptive mechanisms. Applications include logistics and manufacturing as well as field robotics. He is serving as Editor of the Intern. Journal of Robotics Research (IJRR) and has served as Program Chair for the Robotics: Science and Systems (RSS) and the Workshop on the Algorithmic Foundations of Robotics (WAFR) conferences. His research has been supported by NSF, DHS, DOD and NASA, including a NASA Early Career Faculty award.