One of the challenges regarding this is that the robot can only be in one place at once. Therefore, strategies regarding how to search for these become important. In addition, it is beneficial that the robot can find objects in the shortest amount of time, so the method of searching needs to be efficient. Some methods have already been developed, but there is more to do, which can include machine learning, VLM/LLMs, physical robots, and probabilistic methods. The following are some ideas of what could be worked on. Different points can be combined, and other ideas can also be included.
Improve on temporal maps
«Temporal maps» use prior data to create a heat map over the most likely places of the object. The method will then search the most likely place based on the map. These maps are only based on data from the time of day. There is a high probability that including weekday or other data could increase the model's accuracy.
Last seen and merge temporal heat maps
In addition to the method using temporal maps, a simple method called «last seen» could also be interesting. It uses the last position the object was seen and searches in that place. One possible direction is to combine both. The challenge becomes how to combine them. For example, should the time since the object was last seen be considered?
Include LLMs
One problem for the robot could be that the object is hidden under or inside something. Could it be possible to include VLMs (Vision Language Models) in the system to find hidden things? The VLMs could, for example, help understand that it is more likely that a key or phone is under a piece of paper at a table than a cup. In this way, the VLM could be included in systems, such as last-seen or temporal maps, to further increase the capacity of the search algorithm.
Real-world robots
A lot of the methods have only been tested in simulation. Using the TIAGo robot or another robot could be very interesting. Testing in a more dynamic and less predictable environment than simulation would probably lead to unforeseen problems that need to be solved.
Project tasks:
- Get an overview of earlier related work
- Implement a set of alternative approaches according to the introduction above.
- Perform experiments, preferably both in simulation and in the real world.
- Compare the results to previous work available.
- Write the master thesis
References:
Mantelli, M., Noori, F.M., Pittol, D. et al. Semantic Temporal Object Search System Based on Heat Maps. J Intell Robot Syst 106, 69 (2022). https://doi.org/10.1007/s10846-022-01760-8