Start Date
1-5-2026 12:00 PM
End Date
1-5-2026 1:00 PM
Description
Introduction & Problem Statement
Meal planning requires balancing nutritional needs, available ingredients, and user preferences, which can be time-consuming and cognitively demanding. While mobile recipe applications provide personalized recommendations, they lack interactive and socially engaging communication. In contrast, social robots such as Misty enable expressive interaction through speech, gestures, and emotional cues, but typically rely on rigid, pre-programmed behaviors and lack domain-specific knowledge for tasks such as recipe recommendation.
This gap highlights the need for a system that combines the strengths of both technologies. Recipe recommendation is inherently multimodal, requiring both social interaction for engagement and visual presentation for detailed information. This project addresses the problem of how to design a synchronized robot–tablet system that delivers interactive, personalized, and information-rich recipe recommendations in real time.
Methods
We developed a multimodal system that integrates a Misty II social robot with a companion tablet interface, coordinated through a Raspberry Pi-based backend. The system uses a structured large language model (LLM) pipeline to process user input, extracting ingredients, preferences, and dietary constraints such as allergies.
Recipes are retrieved from a curated database of over 500 entries or generated using controlled LLM prompting when no suitable match is found. Rule-based validation mechanisms ensure that dietary constraints are respected and reduce the likelihood of incorrect or unsafe outputs. The tablet interface displays recipe details visually, while Misty delivers synchronized responses through speech, gestures, and LED expressions. This coordinated design supports clear, multimodal interaction and improves overall usability.
Results & Findings
Experimental evaluation demonstrates that the system achieves 92% accuracy in ingredient extraction and generates recipe recommendations within a few seconds under typical conditions. The integration of rule-based validation with LLM reasoning improves consistency and reduces irrelevant or unsafe recommendations.
The multimodal interaction approach enhances user engagement by combining verbal communication with visual guidance. However, several challenges remain, including handling complex or ambiguous user inputs, optimizing response time for longer prompts, and improving synchronization between Misty’s speech and physical behaviors. These findings highlight both the effectiveness and current limitations of integrating LLMs into real-time robotic systems.
Misty: An LLM-Driven Recipe Recommender Robot
Introduction & Problem Statement
Meal planning requires balancing nutritional needs, available ingredients, and user preferences, which can be time-consuming and cognitively demanding. While mobile recipe applications provide personalized recommendations, they lack interactive and socially engaging communication. In contrast, social robots such as Misty enable expressive interaction through speech, gestures, and emotional cues, but typically rely on rigid, pre-programmed behaviors and lack domain-specific knowledge for tasks such as recipe recommendation.
This gap highlights the need for a system that combines the strengths of both technologies. Recipe recommendation is inherently multimodal, requiring both social interaction for engagement and visual presentation for detailed information. This project addresses the problem of how to design a synchronized robot–tablet system that delivers interactive, personalized, and information-rich recipe recommendations in real time.
Methods
We developed a multimodal system that integrates a Misty II social robot with a companion tablet interface, coordinated through a Raspberry Pi-based backend. The system uses a structured large language model (LLM) pipeline to process user input, extracting ingredients, preferences, and dietary constraints such as allergies.
Recipes are retrieved from a curated database of over 500 entries or generated using controlled LLM prompting when no suitable match is found. Rule-based validation mechanisms ensure that dietary constraints are respected and reduce the likelihood of incorrect or unsafe outputs. The tablet interface displays recipe details visually, while Misty delivers synchronized responses through speech, gestures, and LED expressions. This coordinated design supports clear, multimodal interaction and improves overall usability.
Results & Findings
Experimental evaluation demonstrates that the system achieves 92% accuracy in ingredient extraction and generates recipe recommendations within a few seconds under typical conditions. The integration of rule-based validation with LLM reasoning improves consistency and reduces irrelevant or unsafe recommendations.
The multimodal interaction approach enhances user engagement by combining verbal communication with visual guidance. However, several challenges remain, including handling complex or ambiguous user inputs, optimizing response time for longer prompts, and improving synchronization between Misty’s speech and physical behaviors. These findings highlight both the effectiveness and current limitations of integrating LLMs into real-time robotic systems.