Reinforcement Learning aided by Large Language Models
Deep Learning has allowed Reinforcement Learning to scale up to handling complex problems with high-dimensional inputs. However, RL agents still take a lot more time to learn how to solve a task than humans - partly because they solve it without really understanding the task or their environment. The knowledge contained in Large Language Models (LLMs) may be a way towards giving them this understanding, allowing much more efficient and robust Deep Reinforcement Learning.
Thought cloning: A recent paper showed that RL agents can learn more efficiently by not only learning how to act, but also how to "think about" their actions in a way similar to humans. Humans have been proposed to have two main forms of reasoning: One fast and instinctive, and another slower and with more reasoning involved. If balanced properly, these two forms of reasoning can allow fast response when needed, and more carefully planned responses when necessary (or when there is plenty of time).
Thought cloning. Image from https://arxiv.org/pdf/2306.00323
The processing in the Thought Cloning framework resembles the slow "System 2" thinking in humans. But it is reasonable to expect that an intelligent agent, similarly to humans, could benefit from sometimes acting quickly without too much thinking, like our instinctive "System 1" responses. This could be explored by training a thought cloning agent like the one above, and a simpler RL agent mapping states directly to actions, and a separate meta-agent to select between them. Can the meta-agent learn when to perform deliberate, language-guided action selection, and when to act more rapidly and instinctively?
Guiding RL exploration with LLMs: Reinforcement Learning agents struggle to learn useful skills when rewards are very sparse, that is, when very many correct actions are needed in a sequence to obtain a reward. A famous example is the game Montezuma's revenge (see image below), where an agent may have to climb some ladders, jump over some skulls, pick up a key, and unlock a door before reaching it's goal. This reward sparseness leads to traditional Deep Reinforcement Learning methods failing to solve Montezuma's Revenge.
A way forward could be to give the RL agent an understanding of it's surroundings, which could help it identify what to learn next. For instance, if the agent were to ask a language model what it's subgoals should be, perhaps it could break the very sparse learning problem here down into a sequence of skills that are easier to learn, such as jumping over skulls, climbing ladders, picking up the key, etc. Similar ideas are explored in the algorithms Intelligent Go-Explore and OMNI, which also use the knowledge contained in LLMs to guide an agent towards its next learning goal in an RL context. However, while those two algorithms are focused on open-ended search (identifying all possible useful skills/states), the current project proposes to aim for solving one specific problem (e.g. a level in Montezuma's revenge), and seeing if guidance by an LLM can split the task into smaller pieces.
Guiding RL exploration for robotics with LLMs: The project above could also shift its focus towards robotics, where there are many sparse-reward tasks. For instance, given a high-level robot task ("Fetch me a coffee"), can an LLM help identify and learn the relevant sub-components ("Learn these skills: 1) driving to the coffee machine, 2) opening the drawer, 3) picking up a cup, etc."). Previous work on automatic reward function designwith LLMs is also relevant here.
Making new RL environments with LLMs: The projects above suggested identifying useful challenges in a given environment to guide the skill acquisition of an RL agent. However, one could also consider to let an LLM generate a whole new environment, which would guide an RL agent to learn something we care about. For instance, imagine we want to teach a robot to pick up objects and place them elsewhere. As a step on the way, an LLM could design a "picking game", where a useful selection of objects are placed around, and the robot is rewarded for learning picking skills, and next a "placement game" with it's own set of rules. OMNI-EPIC follows a similar methodology for open-ended exploration of agent skills, and Eurekaverse extends this idea to robot locomotion learning. This project proposes to draw inspiration from OMNI-EPIC, Eurekaverse and other methods for automatic environment generation with LLMs and see if they can be beneficial for robotic tasks.
OMNI-EPIC proposes a way to automatically generate environments where RL agents can learn useful skills. Can we apply this to make robots learn useful skills?
Your own idea? If you have your own ideas around ways to use the knowledge contained in LLMs to improve learning in RL agents (robots or simulated agents), let's chat!
