Through their abilities to analyze language and generate human-like text, Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP) and continue to find new application areas across research, industry, and society at large. By being able to interact with the system through natural dialogue, end-users can prompt or instruct the model to solve tasks or answer queries, without requiring knowledge of how to (fine-)tune a model on their own task-specific data.
An essential part of LLM development is so-called benchmarking, i.e. systematic testing and comparison of models to uncover what knowledge, skills, capabilities, limitations, biases, etc. that they have acquired through training. This requires gold-standard datasets for a wide variety of different types of tasks or domains, and with corresponding metrics to quantify performance. Concretely, such data could take the form of multiple-choice or free-form questions, reasoning tasks, writing aid tasks, etc., where a given instruction or prompt is paired with the correct response (possibly among several available alternatives).
While a substantial volume of standardized benchmarking datasets has been developed for widely used languages like English, resources and evaluation benchmarks for less-resourced languages like Norwegian remain limited. Moreover, relying on machine translation, e.g. from English to Norwegian, is not a viable solution. First, it often results in unnatural-sounding phrasing and fails to capture idiomatic expressions unique to the target language. Secondly, it fails to appropriately adapt or "localize" the tests to the relevant cultural, political, and geographical context. For these reasons, it is desirable to create benchmarking data from scratch for a given target language. At the same time though, manually creating such datasets is a labor-intensive and time-consuming process, representing a major bottleneck.
As an important contribution to alleviating these challenges, this project aims to develop techniques for semi-automatically creating benchmarking data tailored for Norwegian LLMs. A potential direction would include (i) utilizing textual data from publicly available resources under permissive licenses, (ii) applying LLMs to create semi-synthetic examples (e.g. by generating questions/instructions and/or answers/responses) and (iii) filtering out irrelevant or noisy examples.