Robot assistant (TIAGo) – Thinking effectively using Large Vision Language models

...to act effectively – the future home and restaurant waiter to act fast and properly – the future home and restaurant waiter.

One of the key challenges for robot assistant are their lack of understanding of what they are capable of doing when they have a new task to perform. Humans do most of their acting intuitively without thinking about how they do a task while we are also able reason and think when needed. We would like robots to become better in themselves deciding how to do new tasks rather having to ask their user about advice or training.

Large Language Models (LLMs) has recently progressed a lot but their relevance to robotics first came with the introduction of Vision Language Models (VLMs) where text and image input can be analysed together (Brohan et al, 2023).

Bildet kan inneholde: mat, servise, ingrediens, oppskrift, mat.

Approach

In this project we would like to demonstrate a robots’ ability to find, put together and bring food based on different food ingrediencies and user food preferences. The project can be undertaken in simulation only or a mix of simulation and real testing with TIAGo or UR5 arm.

 

Project Plan

The project could contain the following parts:

  • Review literature on VLMs applied in robotics and with a special focus on food preparation applications
  • Implement a prototype system for receiving text input from a user about requested food
  • Consider the food ingrediencies available and select what seems to be a best possible match to the request
  • Consider undertaking user testing of the prototype
  • Consider adding a health consideration part taking healthy choices into consideration

Existing models:

Consider using the gpt-4o model via API provided by OpenAI, see https://platform.openai.com/overview                                                                                                 

Segment Anything Model (SAM) (Kirillow et al, 2023) can be applied to identify and segment any object in an image. A Contrastive Language-Image Pre-training (CLIP) (Radfor et al, 2021) model can be applied to understand and generate representations for a wide range of visual concepts described in text by a user. There are work a tidy up tasks that can be relevant (Wu et al, 2023).

 

 

References

  • ICRA2024 Workshop on cooking robotics: https://sites.google.com/view/icra2024cookingrobotics/
  • J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assistance with large language models,” arXiv preprint arXiv:2305.05658, 2023.
  • Brohan, A., et al. (2023). RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control. [https://arxiv.org/abs/2307.15818]
  • A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo et al., “Segment anything,” arXiv preprint arXiv:2304.02643, 2023.
  • A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763.
Published Oct. 21, 2024 6:27 AM - Last modified Oct. 5, 2026 4:00 PM

Supervisor(s)

Scope (credits)

60