College
College of Engineering
Mentor Information
Xiaomin Lin
Description
Robotic manipulation is transitioning from specialized, pre-programmed industrial subroutines to generalized, data-driven foundation models. This transition allows robots to utilize semantic reasoning to operate in unstructured environments like households or warehouses. Recent advancements in Vision-Language-Action (VLA) architectures, like RT-2, Octo, and OpenVLA, have improved robotic learning generalizability but still encounter some limitations. VLA research has only been deployed on expensive manipulator hardware, which restricts accessibility and large-scale data collection. In addition, VLA models underperform when encountering out-of-distribution (OOD) semantic instructions, novel object geometries, or complex, long-horizon tasks. Traditionally, expert demonstrations can address the OOD instructions and novel geometries. However, continuous fine-tuning on new demonstrations can degrade proficiencies from previously mastered tasks. To overcome these bottlenecks, this study employs the XLeRobot—an open-source, budget mobile manipulator—and a novel Retrieval-Augmented Generation (RAG) framework for robotics. On novel task encounters, the RAG system queries a database of trajectories, identifies the nearest semantic and kinematic neighbor, and uses it as a reference. Rather than continuously fine-tuning the model, the system freezes the VLA as a reasoning engine and utilizes memory-augmented prompting to retrieve relevant semantic and kinematic experiences from a hierarchical two-layer vector database. This study will implement the proposed Robotic RAG framework on the XLeRobot platform to measure generalization across novel tasks. We’ll compare our method with state-of-the-art models like pi0.5 and SmolVLA over metrics like success rates, inference latency, and recovery using unfamiliar objects and compositional sequences. In conclusion, this research decreases expert demonstrations and enables long-horizon tasks, contributing to generalistic manipulation systems.
Synergizing Low-Cost Mobile Manipulation with Retrieval-Augmented Vision-Language-Action Models
Robotic manipulation is transitioning from specialized, pre-programmed industrial subroutines to generalized, data-driven foundation models. This transition allows robots to utilize semantic reasoning to operate in unstructured environments like households or warehouses. Recent advancements in Vision-Language-Action (VLA) architectures, like RT-2, Octo, and OpenVLA, have improved robotic learning generalizability but still encounter some limitations. VLA research has only been deployed on expensive manipulator hardware, which restricts accessibility and large-scale data collection. In addition, VLA models underperform when encountering out-of-distribution (OOD) semantic instructions, novel object geometries, or complex, long-horizon tasks. Traditionally, expert demonstrations can address the OOD instructions and novel geometries. However, continuous fine-tuning on new demonstrations can degrade proficiencies from previously mastered tasks. To overcome these bottlenecks, this study employs the XLeRobot—an open-source, budget mobile manipulator—and a novel Retrieval-Augmented Generation (RAG) framework for robotics. On novel task encounters, the RAG system queries a database of trajectories, identifies the nearest semantic and kinematic neighbor, and uses it as a reference. Rather than continuously fine-tuning the model, the system freezes the VLA as a reasoning engine and utilizes memory-augmented prompting to retrieve relevant semantic and kinematic experiences from a hierarchical two-layer vector database. This study will implement the proposed Robotic RAG framework on the XLeRobot platform to measure generalization across novel tasks. We’ll compare our method with state-of-the-art models like pi0.5 and SmolVLA over metrics like success rates, inference latency, and recovery using unfamiliar objects and compositional sequences. In conclusion, this research decreases expert demonstrations and enables long-horizon tasks, contributing to generalistic manipulation systems.
