Mitigating Sycophancy in LLMs with Guardrails

NB: this project is now merged together with "Analyzing and mitigating positive bias of large language models".

Large Language Models (LLMs) often exhibit sycophantic behaviour - that is, a tendency to flatter the user (e.g., beginning responses with phrases like “What an insightful question”), agree with their opinions, or otherwise respond in ways that aim to please, sometimes at the expense of factual accuracy. These behaviors are not accidental; they emerge in part because human users (often unconsciously) prefer agreeable or affirming responses, a bias that is reflected in the preference datasets used during LLM alignment (see here and here). As a result, sycophancy becomes a learned behavior, reinforced during fine-tuning and/or preference alignment.

Sycophancy poses substantial risks to the truthfulness and reliability of model outputs and raises multiple ethical concerns, particularly in sensitive domains where user opinions may be misinformed or harmful. This thesis will explore how guardrails (that is, mechanisms designed to constrain or guide model behavior) can be used to mitigate sycophantic tendencies in LLMs. The investigation will cover a range of approaches, from simple prompt-based techniques to more advanced systems such as NVIDIA’s NeMo or LlamaGuard, which use trained classifiers or policy models to enforce behavioral constraints. These methods will be empirically evaluated using the ELEPHANT benchmark, a recently introduced dataset specifically designed to assess social sycophancy in LLMs. 

Experiments can be conducted on English, Norwegian or other languages.

A group project is possible (up to 3 students).

References

1. Is LLM a Reliable Reviewer? A Comprehensive Evaluation of LLM on Automatic Paper Reviewing Tasks (Zhou et al., LREC-COLING 2024)
2. Kuznetsov, Ilia, et al. "What Can Natural Language Processing Do for Peer Review?." arXiv preprint arXiv:2405.06563 (2024).
3. Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems 33 (2020): 3008-3021.
4. Touvron, Hugo, et al. "Llama 2: Open foundation and fine-tuned chat models." arXiv preprint arXiv:2307.09288 (2023).