Start Date
5-12-2025 12:00 PM
End Date
5-12-2025 1:00 PM
Description
Meal-planning technologies such as recipe apps and diet trackers are widely used but typically rely on text or static visuals and lack social engagement, emotional responsiveness, and conversational adaptability. Social robots, with expressive speech, gesture, and display capabilities, offer the potential to make everyday tasks like cooking more accessible and enjoyable. Recent work highlights the growing role of large language models (LLMs) in robotics and embodied AI systems. This project presents REPS (Recipe Embodied Partner System), a multimodal recipe recommendation system for the Misty social robot that integrates structured recipe retrieval, LLM-based reasoning, and multimodal robot output to create a natural and engaging cooking-assistant experience.
The system pipeline includes a backend hosting a curated recipe database, which is queried based on user-provided ingredients, preferences, and dietary constraints. A large language model interprets user input, extracts relevant information, and refines recipe instructions for clarity and accessibility, similar to recent multimodal robot-LLM integration approaches. Misty delivers recipes through synchronized speech, expressive gestures, LED cues, and a connected tablet displaying ingredients, step-by-step instructions, and images. Safety and personalization filters ensure adherence to dietary constraints and appropriate content.
Evaluation of REPS focuses on technical metrics such as ingredient extraction accuracy, latency, refinement quality, and multimodal synchronization. Results demonstrate high ingredient extraction accuracy and sub-second response latency while maintaining synchronized multimodal delivery. REPS illustrates how LLM-powered social robots can support everyday domestic tasks, contributing toward accessible, embodied AI assistants for everyday domestic environments.
REPS: A Multimodal Recipe Recommendation System for the Misty Social Robot
Meal-planning technologies such as recipe apps and diet trackers are widely used but typically rely on text or static visuals and lack social engagement, emotional responsiveness, and conversational adaptability. Social robots, with expressive speech, gesture, and display capabilities, offer the potential to make everyday tasks like cooking more accessible and enjoyable. Recent work highlights the growing role of large language models (LLMs) in robotics and embodied AI systems. This project presents REPS (Recipe Embodied Partner System), a multimodal recipe recommendation system for the Misty social robot that integrates structured recipe retrieval, LLM-based reasoning, and multimodal robot output to create a natural and engaging cooking-assistant experience.
The system pipeline includes a backend hosting a curated recipe database, which is queried based on user-provided ingredients, preferences, and dietary constraints. A large language model interprets user input, extracts relevant information, and refines recipe instructions for clarity and accessibility, similar to recent multimodal robot-LLM integration approaches. Misty delivers recipes through synchronized speech, expressive gestures, LED cues, and a connected tablet displaying ingredients, step-by-step instructions, and images. Safety and personalization filters ensure adherence to dietary constraints and appropriate content.
Evaluation of REPS focuses on technical metrics such as ingredient extraction accuracy, latency, refinement quality, and multimodal synchronization. Results demonstrate high ingredient extraction accuracy and sub-second response latency while maintaining synchronized multimodal delivery. REPS illustrates how LLM-powered social robots can support everyday domestic tasks, contributing toward accessible, embodied AI assistants for everyday domestic environments.