Learning from Reality Without Giving Up the Model

A hybrid approach helps autonomous systems adapt quickly by blending simple models with real-world experience.

One of the central challenges in autonomous systems is deceptively simple: the mathematical model used to design a controller is never quite the same as the physical system it is intended to control. A model of a vehicle, drone, or robot may capture the essential physics while omitting tire nonlinearities, aerodynamic effects, friction, parameter variations, or other complexities. Yet making the model detailed enough to reproduce all these effects can make control design computationally difficult—or require identifying aspects of the physical system that are themselves hard to determine.  Modern machine learning offers another possibility: dispense with an accurate model and learn directly from experience. Reinforcement learning has demonstrated impressive capabilities along these lines. But learning primarily through physical experimentation can require enormous amounts of data, making it expensive, slow, and potentially unsafe for real robotic systems.

Our work explores a middle ground: use an imperfect model for what it knows, and real-system experience for what the model gets wrong.

We start with a method we call Gradient-based Reprogrammable Iterative Learning Control, or GRILC [1]. The key observation behind GRILC is that a model need not predict a physical system's behavior accurately to provide useful information about how that behavior changes when control inputs are modified. In mathematical terms, even when the model itself is inaccurate, its gradients—the local directions in which the system responds to changes—may still be informative.  This suggests a different way to combine models and data. Rather than continually trying to make the mathematical model reproduce the real system, GRILC takes the trajectory actually produced by the physical system as its starting point. A simplified model then provides gradient information indicating how the controls should change to improve performance. The modified controls are tried on the real system, the resulting trajectory is measured, and the process repeats. In this way, reality supplies the baseline behavior, while the model supplies directions for improving it [1].

An autonomous race car provides an intuitive example. A simplified vehicle model might incorrectly estimate the car's mass, inertia, or tire behavior. A conventional controller that relies heavily on that model can consequently make poor decisions. GRILC instead observes what the real car actually did and uses the approximate model primarily to determine which changes in steering and speed should improve the next attempt. In simulations  [1], GRILC remained effective even when the model contained very large errors in mass and inertia, and compared extremely favorably with model-predictive control. Experiments with a physical 1/10-scale autonomous car similarly showed the vehicle progressively learning faster trajectories and cutting corners more aggressively [1].

However, this kind of iterative learning has an important limitation. Traditionally, the system must perform essentially the same task repeatedly, beginning from the same initial state. A robot operating in the real world rarely enjoys that luxury. Each new task may begin from a different position, velocity, orientation, or other operating condition.

Our more recent work addresses this limitation through a method called Experience-Based Hybrid Gradient Optimization (EHGO) [2]. Its key addition is an adaptive experience database. Before deploying the physical system, we can generate a large collection of trajectories inexpensively using the approximate model in simulation. These provide the robot with a useful, though imperfect, initial repertoire of behaviors.  Once the robot begins operating, this database evolves. When faced with a new situation, the system searches its accumulated experience for trajectories relevant to its current initial condition. It then uses GRILC to improve the selected trajectory using information from the real system, and stores the resulting improved trajectory back in the database [2]. Thus, each physical experiment not only addresses the immediate control problem but can also improve the system's ability to handle future situations.

Choosing which past experiences to use requires an interesting tradeoff. A simulated trajectory may begin almost exactly at the current state but has never been tested on the real system. Another trajectory may begin farther away but have already undergone several rounds of real-world refinement. EHGO therefore learns a relevance measure that balances situation similarity against the quality of accumulated real-world experience [2]. As the database grows, simulated knowledge is progressively supplemented by—and in useful regions effectively displaced by—better experience obtained from the actual system.

This combination of prior modeling and accumulated experience can substantially reduce the amount of real-world learning required. We evaluated EHGO on three nonlinear control problems involving a pendulum, a car, and a quadcopter, deliberately introducing significant discrepancies between the simulated and “real” dynamics [2]. In these experiments, EHGO adapted rapidly and performed very well compared with several versions of Proximal Policy Optimization, a widely used deep reinforcement-learning algorithm [2].

The broader lesson is not that models should replace learning, or that learning should replace models. A useful model does not necessarily have to be accurate. A simplified representation of a physical system may contain valuable structural information even when its quantitative predictions are wrong. At the same time, direct experience provides information that no imperfect model can fully anticipate.

By letting these two sources of knowledge play different roles—real-world trajectories describing what actually happens, and approximate models indicating how to improve what happens—hybrid approaches such as GRILC and EHGO offer a path toward autonomous systems that learn rapidly without starting from scratch. The ultimate objective is a machine that arrives with useful knowledge from simulation, improves that knowledge through experience, and continually builds a repertoire that makes each subsequent encounter with the physical world more informed than the last.

References

[1] K.-Y. Tseng, J. S. Shamma, and G. E. Dullerud, “Low-Fidelity Gradient Updates for High-Fidelity Reprogrammable Iterative Learning Control,” Proceedings of the American Control Conference (ACC), 2022.

[2] K.-Y. Tseng, M. Zhang, K. Hauser, and G. E. Dullerud, “Adaptive Trajectory Database Learning for Nonlinear Control with Hybrid Gradient Optimization,” Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.