NB: This project is now merged with "Mitigating Sycophancy in LLMs with Guardrails".
Large language models (LLMs), also known as "generative AI", are nowadays routinely used for almost any text-related task. Generating a text on a given topic from scratch is one popular use-case, but another one gaining traction is assessing a text written by someone else, be it in the form of academic peer review or a critical summary of a high school student essay.
It is known that in this task, modern LLMs are biased towards being overly positive. They tend to focus on strong or impressive aspects of the text under analysis, while ignoring downsides. Take Google’s NotebookLM web service. Even given a scientific paper with extremely bad human reviews and low scores, it still happily generates a super positive podcast about it. Of course, this makes reviewing with LLMs a very unreliable enterprise.
One possible reason for such a bias can be LLM optimization to align with human preferences. Reinforcement learning with human feedback (RLHF) was one of the recipes to the success of ChatGPT, and now almost all widely used language models are employing this or that sort of preference alignment. It is supposed to make the models: 1) helpful, 2) harmless, 3) honest, since it is assumed this is what a typical chatbot user wants. But this is not exactly what a critical review is.
Thus, research question within this Master project will be to find out whether preference alignment indeed pushes LLMs towards overly positive generation in the cases when it is actually detrimental to the task at hand. If true, we will move to to devising ways to mitigate this bias without harming the performance of the model in other tasks.
Experiments can be conducted on English, Norwegian or other languages.
A group project is possible (up to 3 students).
References
- Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks (Zhou et al., LREC-COLING 2024)
Kuznetsov, Ilia, et al. "What Can Natural Language Processing Do for Peer Review?." arXiv preprint arXiv:2405.06563 (2024).
Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021.
Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).