The Language Technology Group (LTG) coordinates the training data management and preparation in two large European research projects, High-Performance Language Technologies (HPLT) and Foundation Models for Trasparent AI in Europe (OpenEuroLLM). This involves collaboration with web crawl initiatives (like Common Crawl, the Internet Archive, and OpenWebSearch), document extraction from petabytes of raw web archives, the selection of “running text” from document formats like HTML and PDF, data annotation with e.g. language identification, different text registers, personally identifiable information, and maybe most importantly estimation of “text quality” and data selection for LLM pre-training. Most of the steps in these pipelines apply various machine learning techniques, and they all offer large degrees of design choices and parameters. How to assemble the “best” collection of LLM pre-training data (for a given language and model architecture), thus, is an empirical question, that needs to be informed by systematic experimentation of different design choices and their downstream effects on LLM behavior. Using different types of data, for example, in different training phases can lead to better model performance, or it can reduce the necessary computational effort (i.e. environmental footprint), or both.
MSc projects in this space will typically address one sub-problem in the data preparation pipeline, for example language identification, or register classification, or different notions of “text quality” estimation, or data mix design in multi-stage pre- and mid-training. Such a project will seek to experimentally determine the interdependencies between different sub-approaches and end-to-end model behavior and, based on these findings, work to refine and improve the specific sub-problem. These projects will have access to the resources of the HPLT and OpenEuroLLM initiatives, including the ability to work on the most advanced supercomputers in Europe. If desirable, one could define a group project, or two closely related but individual projects. For 2027, OpenEuroLLM will offer some student summer jobs, and ongoing MSc candidates in this space would of course be welcome to apply for these.