This thesis will investigate how reliably large language models can predict human preferences between pairs of generated responses. Using LLMs as judges offers a way to evaluate generated text at scale, but their usefulness depends on how closely their assessments correspond to human judgments. The project will examine this correspondence across languages and evaluation criteria, investigating whether agreement on overall quality conceals differences in the models’ ability to assess specific aspects of response quality.
The project will use a multilingual extension of the dataset introduced in the ELOQUENT shared task on preference prediction at CLEF 2025. Each example consists of a prompt and two model-generated responses, with human preference annotations covering relevance, naturalness, truthfulness, safety, and overall quality. Following the original task formulation, judges will predict whether response A or response B is preferable, or whether both responses are equally good or equally bad. The multilingual setting will enable comparisons of performance across languages and investigation of whether the relative strengths of different judges vary with language and evaluation criterion.
Experiments will compare a selection of judge models and prompting strategies, including zero-shot and few-shot approaches. Evaluation will measure agreement with human annotations and examine sensitivity to response ordering and prompt formulation. Quantitative results will be complemented by qualitative analysis of disagreements, including possible influences of response length and linguistic fluency. An optional extension will examine the explanations generated by judge models, comparing them with human explanations to identify differences in how evaluation criteria are interpreted and applied. The thesis will contribute an empirical assessment of the reliability and limitations of LLM-based judges for multilingual preference prediction.