Bilkent University
Department of Computer Engineering
CS 590/690 SEMINAR
Contact-Rich Adaptive LLM-based Control for Robotic Manipulation
Mert Kaan Er
Master Student
(Supervisor:Asst.Prof.Özgür S.Öğüz)
(Co-advisor:Assoc.Prof.Hamdi Dibeklioğlu)
Computer Engineering Department
Bilkent University
Abstract: While Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate capabilities in reasoning and semantic understanding, applying them directly to contact-rich manipulation remains a challenge due to their lack of physical grounding and adaptive control. We propose CoRAL (Contact-Rich Adaptive LLM-based control), a modular framework enabling zero-shot planning by decoupling high-level reasoning from low-level control. Unlike black-box policies, CoRAL utilizes LLMs not as direct controllers, but as cost designers that synthesize context-aware objective functions for a sampling-based motion planner (MPPI). To address visual ambiguity in physical parameters, a VLM first provides semantic priors for environmental dynamics (e.g., mass and friction). The LLM then refines these estimates using interaction history and iteratively updates the cost function structure to correct strategic errors. A retrieval-based memory unit allows the system to reuse successful strategies across recurrent tasks. This architecture ensures real-time control stability, bridging the gap between slow LLM inference and dynamic contact requirements. We validate CoRAL in simulation and on real-world hardware across challenging tasks. Experiments demonstrate that CoRAL outperforms state-of-the-art VLA and foundation-model-based planner baselines by boosting success rates over 50% on average in unseen contact-rich scenarios, effectively handling sim-to-real gaps through its adaptive physical understanding.
DATE: March 23, Monday @ 15:50 Place: EA 502