The data used for pretraining large language models (LLMs) are constantly evolving. The first models were trained on large dumps of mostly unfiltered web crawls -- then, over time, the importance of data quality became more and more apparent, and we started to use increasingly complex ways of selecting the best data for training. The most recent step in this direction applies LLMs themselves to further increase the data quality, which typically means rephrasing the existing sources in some clever way -- remove off-topic text segments, re-structure, rephrase as a detailed textbook, reformulate as a sequence of questions and answers? Which exact approach works best is still an open, and heavily studied, research question.
When it comes to Norwegian LLMs, we still have some catching up to do. We have limited pretraining data compared to English; can we apply the same techniques for Norwegian as for the large languages? Can rephrasing help us augment the amount of pretraining data? And do we have capable language models that can generate Norwegian fluently? Are there some trade-offs when training on such data?
The primary goal of this project is to explore and develop automatic methods for rephrasing pretraining data for Norwegian LLMs. You will then use the synthesized data to train small language models on the Olivia HPC cluster, and carefully evaluate the effect on performance.
If you're interested in this topic, you can read more about it in this nice blog post: The Synthetic Data Playbook: Generating Trillions of the Finest Tokens