Trial lecture
"Multimodal LLMs for image captioning, VQA, visual dialogue and situated dialogue systems”
Time and place: October 11, 2024 11:15 AM. Auditorium SmallTalk, Ole Johan Dahls hus.
Main research findings
Artificial agents such as robots that communicate with humans in natural language must make use of a diverse array of information in order to complete their tasks. In some cases, such an agent may have difficulty naturally and accurately talking to a user because of uncertainty about this information. Our work addresses this difficulty by representing such information with graphs, which are collections of objects linked to each other with descriptions of how they are related. Using graphs that contain words, sentences, physical objects, and even abstract ideas, we developed novel methods to improve a conversational artificial agent’s ability to discuss the world accurately and assist users. The results of this thesis contribute new methods that can assist in anchoring a conversational AI system’s capability to “ground” itself in the world and the nuanced and complex relations between people, places, things, and the language used to describe these relations.
Adjudication committee
- Professor Simon Dobnik, Department of Philosophy, Linguistics and Theory of Science, University of Gothenburg, Sweden
- Associate professor Ondřej Dušek, Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic
- Researcher Benedikte Wallace, Department of Informatics, University of Oslo, Norway
Supervisors
-
Associate Professor Pierre Lison, Department of Informatics, University of Oslo, Norway
-
Research Leader Stefan Ultes, University of Bamberg, Germany
Chair of defence
Professor Emeritus Dag Langmyhr, Departement of Informatics, University of Oslo
Contact information at Department: Mozhdeh Sheibani Harat