Algorithm-Hardware Codesign for Real-Time Robotics
Associate Professor,
Electrical and Computer Engineering
As robots become increasingly capable, they also become increasingly computationally demanding. A modern robot may need to perceive its surroundings through cameras and LiDAR, understand complex scenes using AI, plan its motion, and continuously adapt its actions—all while operating in real time on compact, energy-constrained computing platforms. These demands create a fundamental challenge: How do we equip robots with increasingly sophisticated intelligence while enabling real-time operation on resource- and energy-constrained edge devices?
The HiAccel Laboratory, directed by Prof. Zhenman Fang, addresses this challenge through algorithm–hardware codesign. His lab codesigns algorithms and hardware architectures to unlock greater parallelism and efficient dataflow, maximize data reuse and memory efficiency, and leverage low-precision computation and data access. By co-optimizing these dimensions across the algorithm–hardware stack, the HiAccel lab develops customized computing solutions on edge FPGAs (field-programmable gate arrays) that often deliver orders-of-magnitude improvements in performance and energy efficiency.
FLi-SLAM: Real-Time, Energy-Efficient LiDAR SLAM at the Edge. Consider a robot equipped with a 3D LiDAR sensor, which provides accurate geometric information and remains effective in challenging lighting conditions. To localize itself and construct a map of its surroundings, the robot must perform computation-intensive scan matching between successive LiDAR frames represented as point clouds, where the core computation is multiple iterations of k-nearest neighbors (KNN) point correspondence search between the point cloud of the current LiDAR frame and that of the currently built map. This computation involves tens of billions of distance calculations and comparisons for every LiDAR frame—arriving every 50 ms for a 3D LiDAR sensor operating at 20 Hz—creating a major challenge for real-time processing on resource-constrained edge devices.
To address this challenge, we have developed FLi-SLAM, the first FPGA-based algorithm-hardware codesign that accelerates the entire point-to-feature iterative closest point registration pipeline for 3D LiDAR SLAM. First, we build an approximate KDTree on chip—to partition k-dimensional map points and accelerate query-point search—with fully pipelined tree traversal and theoretical maximum data reuse for each bucket of off-chip points. To accelerate the subsequent exact KNN search, we propose a simple yet effective accuracy-preserving point cloud pruning algorithm and develop a high-throughput (fully pipelined) KNN hardware design. After accelerating the KDTree-based KNN, we also optimize the new bottlenecks in point-to-feature calculations by codesigning specialized algorithms and hardware for their complex QR and Eigen decomposition modules.
As shown in Fig 1, evaluated on widely used open-source LiDAR datasets, our FLi-SLAM running on the AMD/Xilinx ZCU102 embedded FPGA board achieves an average speedup of 56.4x and 22.9x over the state-of-the-art LiDAR SLAM framework LIO-SAM on a quad-core ARM A53 CPU and LIO-SAM-GPU on an Nvidia Jetson Orin Nano GPU, which is future-proof for emerging LiDAR sensors (up to 100 Hz). Moreover, FLi-SLAM also achieves energy efficiency gains of 19.5x and 10.8x.
Algorithm-Hardware Codesign for AI Inference at the Edge. AI is at the core of modern robots, enabling them to understand complex scenes and commands, plan their motion, and continuously adapt their actions. However, achieving real-time, energy-efficient AI inference remains a major challenge given the limited computing resources and power budgets of robotic edge platforms.
We have contributed to multiple AI acceleration projects by developing hardware-efficient AI model compression techniques, including quantization and pruning, and by codesigning customized hardware architectures on edge FPGAs to execute these models in real time. Our work spans a broad range of AI workloads, including convolutional neural networks (CNNs), vision transformers (ViTs), Stable Diffusion, and spiking neural networks (SNNs), demonstrating the potential of algorithm–hardware codesign to bring increasingly sophisticated AI capabilities to resource-constrained robotic platforms.
For example, in our Caffeine project, we were among the first to accelerate widely used CNNs on FPGAs—a core AI model of the deep learning revolution. We have introduced a unified convolutional representation for both convolutional and fully connected layers, jointly optimized computation and memory access, and developed an end-to-end Python-to-FPGA tool flow that enables AI developers to deploy efficient CNN accelerators without hardware expertise. This work won the IEEE TCAD Best Paper Award (2019) and the IEEE/ACM ICCAD 10-Year Retrospective Most Influential Paper Award (2025), and has become a foundational reference in the field, with more than 800 citations.
Building on this foundation, we have developed further algorithm–hardware co-optimizations for CNN and ViT acceleration, including hardware-efficient mixed-precision quantization from 8-bit to 1-bit, static and dynamic model pruning, and hardware architecture optimizations for layer-sharing systolic arrays and fully on-chip streaming architectures, as summarized in Fig. 2.
Future Work and Collaboration. We are now exploring how to extend the same codesign philosophy to emerging world action models (WAMs), with the goal of enabling efficient robotic intelligence that can perceive, reason about, and ultimately act in the physical world. We welcome collaborations with researchers working on challenging robotic algorithms that require real-time, energy-efficient processing. If you have an algorithm or application that could benefit from customized hardware acceleration, please reach out at [email protected].
