The availability of benchmark datasets is a crucial prerequisite for evaluating and comparing large language models. There has been good progress on increasing the linguistic coverage of benchmark datasets in recent years, including both benchmarks translated from English and benchmarks created from scratch for specific languages. At the same time, as LLM development moves on, we see increasingly short "shelf lives" of benchmark datasets, in the sense that they quickly become too easy for the LLMs and cease to be relevant benchmarks. One possible way around this dilemma is to focus on non-standard varieties for which little training data exist. Such benchmarks rather test the robustness of LLMs to non-standard input.
For Norwegian, several dialectal datasets have been collected and annotated, e.g.:
- morphosyntactically annotated subsets of the NDC and LIA dialect corpora
- NorDial, a dataset of dialectal tweets, annotated with specific dialect features
- NoMusic, a set of dialogue system instructions translated from English into 10 Norwegian dialects.
The main goal of this thesis will be to investigate how these existing resources can be turned into benchmark datasets for instruction-tuned LLMs. Additionally, it is also possible to create entirely new dialectal benchmarks, for example by translating or adapting English benchmarks (similarly to what has been done for other languages, e.g. DIALECT-COPA).