Our latest publications - LTG

• Wan, Jiajing; Touileb, Samia; Steskal, Lubos & Øvrelid, Lilja (2026). Personalizing News Headlines with Retrieval-Augmented Generation. Association for Computational Linguistics (ACL). Annual Meeting Conference Proceedings. ISSN 0736-587X. p. 55–67. doi: 10.18653/v1/2026.customnlp4u-1.6. Full text in Research Archive Show summary

  We focus on personalized news headline generation, where we aim to improve headline generation by extending the generation context to incorporate the news reading history of users. In particular, we study a RAG-LLM-based system that customizes news headlines with user histories to improve news headline personalization. Our experiments show that our approach not only produces better headlines for specific users, but also makes the generated headlines closer to the original headlines. We experiment with different retrievers and analyze the generated outputs through systematic comparisons with both original and rewritten headlines. These analyses provide insights into the role of retrieval and personalization in headline generation, highlighting how the user history contributes to meaningful improvement while remaining aligned with original headlines

• Torunoğlu-Selamet, Dilara; Arslan, Doğukan; Wilkens, Rodrigo; He, Wei; Eryiğit, Doruk & Pickard, Thomas [Show all 78 contributors for this article] (2026). A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding - ACL Anthology. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 9434–9448. doi: 10.63317/5cvnbcoktfo2. Full text in Research Archive
• Norman, Rebecka Maria; Storset, Lilja Charlotte; Mæhlum, Petter; Iversen, Hilde Karin Hestad; Jelin, Elma & Velldal, Erik [Show all 8 contributors for this article] (2026). Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis. Family Practice. ISSN 0263-2136. 43(5). doi: 10.1093/fampra/cmag071. Full text in Research Archive
• Marinelli, Ryan Anthony (2026). Bayesian Benchmarking: Applications of Statistics for Efficient Model Ranking. In Bin, Riccardo De; Salucci, Clara Bertinelli & Hubin, Aliaksandr (Ed.), Proceedings of the 40th International Workshop on Statistical Modelling 28.06 – 03.07.2026, Oslo, Norway. International Workshop on Statistical Modelling. ISSN 9788255325000. doi: https:/www.mn.uio.no/math/english/research/groups/statistics-data-science/events/conferences/iwsm2026/proceedings/proceedings_iwsm2026_proofonline.pdf. Full text in Research Archive Show summary

  This work explores more efficient benchmarking techniques to com- pare the performance between models. By leveraging Bayesian techniques, it is possible to confidently rank models by their performance without excessive compute.

• Samuel, David; Øvrelid, Lilja; Velldal, Erik & Kutuzov, Andrey (2026). Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages. In Vondrick, C.; Hariharan, B.; Raffel, C.; Pinto, L.; Yang, D. & Faust, A. (Ed.), International Conference on Learning Representations (ICLR) 2026. International Conference on Learning Representations (ICLR). ISSN 9798331339678. p. 94000–94027. doi: https:/proceedings.iclr.cc/paper_files/paper/2026/hash/97a057ab1235218027dc07c50eedee0d-Abstract-Conference.html. Full text in Research Archive Show summary

  We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

• Aulamo, Mikko; Virpioja, Sami; Scherrer, Yves & Tiedemann, Jörg (2026). The Challenge of Finding Robust and Efficient Strategies for Training Machine Translation Models with Noisy Data. In Shterionov, Dimitar; Vanmassenhove, Eva; Sisto, Mirella De; Blain, Fred; Pourmostafa, Javed; Sharami, Roshan; Lepp, Lisa; Manna, Chiara; Rescigno, Argentina Anna; Karakanta, Aliana; Terryn, Ayla Rigouts; Lardelli, Manuel; Resende, Natalia; Murgolo, Elena; Hackenbuchner, Janica; Zaretskaya, Anna; Esplà-Gomis, Miquel; Etchegoyhen, Thierry; Gromann, Dagmar; Bawden, Rachel; Haddow, Barry; Szoc, Sara; Forcada, Mikel & Moniz, Helena (Ed.), Proceedings of the 26th Annual Conference of the European Association for Machine Translation. European Association for Machine Translation (EAMT). ISSN 9789403901404. p. 173–187. doi: 10.26116/9789403901404. Full text in Research Archive Show summary

  Most machine translation datasets come with a certain level of noise, and strategies for handling such data need to be robust and efficient. Data selection and filtering are challenging and may depend on expensive language-specific tools that are not necessarily available, especially for low-resource languages. This paper looks at training strategies that combine cheap heuristic filters with curriculum learning to implement iterative procedures that robustly operate on raw noisy data without expensive prior preprocessing and data selection. The intuition is that we can cluster data into buckets with varying noise levels and use different sets of buckets at different stages of MT model training. We test various strategies and compare them to pre-filtering approaches for a diverse set of low-resource languages and conclude that curriculum learning can improve robustness but does not necessarily lead to improved translation performance. Overall, the experiments demonstrate the importance of proper experimental workflows, which cannot easily generalize from one language pair and scenario to another.

• Qian, Shenbin & Scherrer, Yves (2026). Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation. In Shterionov, Dimitar; Vanmassenhove, Eva; Sisto, Mirella De; Blain, Fred; Pourmostafa, Javed; Sharami, Roshan; Lepp, Lisa; Manna, Chiara; Rescigno, Argentina Anna; Karakanta, Aliana; Terryn, Ayla Rigouts; Lardelli, Manuel; Resende, Natalia; Murgolo, Elena; Hackenbuchner, Janica; Zaretskaya, Anna; Esplà-Gomis, Miquel; Etchegoyhen, Thierry; Gromann, Dagmar; Bawden, Rachel; Haddow, Barry; Szoc, Sara; Forcada, Mikel & Moniz, Helena (Ed.), Proceedings of the 26th Annual Conference of the European Association for Machine Translation. European Association for Machine Translation (EAMT). ISSN 9789403901404. p. 31–56. doi: 10.26116/9789403901404. Full text in Research Archive Show summary

  Large Language Models (LLMs) have recently demonstrated strong performance in machine translation (MT). However, most prior work focuses on improving or benchmarking translation quality, offering limited insight into when and why LLM-based translation fails. In this work, we systematically analyze failure modes of LLMs in MT by evaluating 15 models, including four reasoning LLMs, across 22 language pairs (LPs) with varying resource levels. We find that non-English-centric LPs consistently yield lower COMET scores than English-centric pairs. To investigate the underlying causes, we introduce Token Activation Rate (TAR), a metric that captures how effectively a model utilizes language-specific tokens in its vocabulary during generation. We validate TAR as a proxy for language representation using models with known language distributions in the training data, and show that lower TAR is strongly associated with poorer translation performance. Furthermore, reasoning LLMs tend to generate more tokens when translating into low-TAR languages, suggesting a compensatory mechanism, although its impact on translation quality varies across models. Overall, our findings emphasize the importance of token-level dynamics in understanding MT performance of LLMs.

• Milano, Emilia; Plum, Alistair; Scherrer, Yves & Purschke, Christoph (2026). Language Ideologies in a Multilingual Society: An LLM-based Analysis of Luxembourgish News Comments. In Stranisci, Marco Antonio; Falk, Neele; Labat, Sofie; Lo, Soda Marem; Velutharambath, Aswathy; Weber, Sabine; Damiano, Rossana; Frenda, Simona; Hoste, Véronique; Kleinberg, Bennett; Klinger, Roman; Patti, Viviana; Plaza-del-Arco, Flor Miriam; Sap, Maarten & Yimam, Seid Muhie (Ed.), Proceedings of the 1st Workshop on Social Context (SoCon) and the 2nd Workshop on Integrating NLP and Psychology to Study Social Interactions (NLPSI) @ LREC 2026. European Language Resources Association. ISSN 9782493814654. p. 114–131. doi: 10.63317/48kwvvr63gms. Full text in Research Archive Show summary

  Detecting language ideologies is a valuable yet complex task for understanding how identities are constructed through discourse. In Luxembourg’s multicultural and multilingual society, language ideologies reflect more than simple preferences: they carry deep cultural and social meanings, shaping identities and social belonging. Following recent developments in applying Natural Language Processing tools to linguistics and social science, this paper explores the potential of large language models to assist in the detection of language ideologies. We manually annotate a corpus of user comments in Luxembourgish with predefined ideological categories and then evaluate the performance of large language models under varying prompt conditions to assess their ability to replicate these human annotations. Since Luxembourgish is a small language and poorly represented in the LLMs’ training data, we also investigate whether machine-translating the data to high-resource languages increases performance on the ideology detection task. Our findings suggest that, while LLMs are not yet fully optimized for a multi-class ideological annotation task, they are practical tools to identify language ideological content.

• Othamar, Elisabeth & Scherrer, Yves (2026). Norwegian Sign Language: Overview of Resources and Experiments with Automatic SignWriting Transcription. In Efthimiou, Eleni; Fotinea, Stavroula-Evita; Hanke, Thomas; Hochgesang, Julie A.; Mesch, Johanna & Schulder, Marc (Ed.), Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion. European Language Resources Association. ISSN 9782493814821. p. 408–418. doi: 10.63317/3unezo3kea9w. Full text in Research Archive Show summary

  Norwegian Sign Language (NTS) remains an under-resourced sign language despite its official recognition in Norway since 2022. The limited availability of structured, reusable, and publicly accessible datasets continues to hinder both linguistic research and the development of sign language technologies such as recognition and translation systems. This paper presents an overview of existing datasets and potential data sources for NTS, categorizing them by accessibility, format, and suitability for computational research. We further discuss legal, ethical, and practical considerations related to data reuse, including copyright and privacy constraints. In addition, we report on a series of pilot experiments exploring alternative data acquisition strategies, including dictionary videos, SignWriting resources, and broadcast news material. These preliminary experiments explore whether automatic SignWriting transcription can serve as an intermediate representation for NTS, and examine its potential role in sign identification within continuous signing. The aim of this work is both to document ongoing efforts and to support future initiatives toward the sustainable development of NTS resources.

• Rønningstad, Egil; Klinger, Roman; Øvrelid, Lilja & Velldal, Erik (2026). Entity-Level Sentiment Analysis with Sentence Relevance Detection. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 8040–8055. doi: 10.63317/35ideqx4jk89. Full text in Research Archive Show summary

  The task of entity-level sentiment analysis (Elsa) is to extract sentiment scores for a given entity (such as person names or organization names) from a text. Elsa is a challenging task and involves processing of longer documents, where several entities may be mentioned with varying importance for the final score aggregation. Fine-tuning encoder-based Transformers (such as BERT) constitutes the state of the art for sentiment predictions, however, these models are still limited by their restricted input lengths. Decoder-only models so far still underperform on the task. We approach the context limitation by learning to extract segments that are relevant for the sentiment prediction for a given entity, without preprocessing by chunking and aggregation. For decoder models, we explore fine-tuning these through supervised fine-tuning and pairwise comparison, a method borrowed from reward modeling for preference optimization. Both methods perform well and set a new standard for the Elsa task. We further show that pairwise classification is faster, simpler, and shows less variance than the more common direct supervision for this task.

• Olsen, Helene Bøsei; Velldal, Erik & Øvrelid, Lilja (2026). MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 7766–7780. doi: 10.63317/2kaxd8nu44bx. Full text in Research Archive Show summary

  Automatically predicting structured representations of events has long been a central goal in information extraction, yet most contemporary work remains limited to identifying contiguous text spans as event arguments. This span-centric formulation fails to capture higher-level aspects of real-world events, such as actor identities, temporal scope, and aggregated outcomes, that many event-centred applications depend on. While commonly treated as a standard extractive benchmark, MUC-4 originally combined span-based arguments with normalised, inferred, and categorical fields, reflecting a richer, application-driven design. In this paper, we revisit MUC-4 in its full original formulation, casting it as an abstractive event analysis task that connects traditional event extraction goals with modern generative and document-level paradigms. We provide the first systematic evaluation of fine-tuned generative models in this extended formulation on MUC-4, examining how post-training stages and model size affect performance across both span-based and higher-level, semantically grounded event information. An extensive error analysis highlights practical challenges and directions for future work.

• Clausen, Hannah; Kutuzov, Andrey; Smajdor, Anna & Velldal, Erik (2026). Assessing Moral Judgment by Large Language Models – A Survey of Available Datasets. Northern European Journal of Language Technology (NEJLT). ISSN 2000-1533. 12(1), p. 88–116. doi: 10.3384/nejlt.2000-1533.2026.6366. Full text in Research Archive Show summary

  Recent advances in language modeling have contributed to a growing emphasis on machine ethics, including researching and assessing moral judgments made by large language models (LLMs). This paper provides a critical survey of the existing datasets for exactly this assessment, with a special focus on the respective data sources. We address the current lack of theoretical grounding by providing an introduction to ethics and different frameworks from moral philosophy and moral psychology. Moreover, we identify four main data sources: webcrawled corpora, scholars, laypeople, and synthetic data generation. By discussing the strengths and weaknesses of these sources, we analyze their implications for the assessment of moral judgment. Importantly, systemizing the available datasets reveals an over-reliance on previous work, reinforcing existing shortcomings. Addressing the current limitations, we recommend adopting a consistent terminology and creating independently curated datasets based on interdisciplinary work. To ensure a clear delineation of normative approaches, we propose focusing on the assessment of moral consistency and certainty of LLMs as effective and well-defined indicators of their performance on moral judgment.

• Oepen, Stephan; Arefyev, Nikolay; Aulamo, Mikko; Bañón, Marta; Buljan, Maja & Burchell, Laurie V. [Show all 29 contributors for this article] (2026). HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 1409–1434. doi: 10.63317/25xbdofco9od. Full text in Research Archive Show summary

  We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for some 20 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder–decoder models, as well as about 30 “smallish” monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

• Fedorova, Mariia; Arefyev, Nikolay; Buljan, Maja; Helcl, Jindřich; Oepen, Stephan & Rønningstad, Egil [Show all 7 contributors for this article] (2026). OpenLID-v3: Improving the Precision of Closely Related Language Identification – An Experience Report. In Scherrer, Yves; Aepli, Noëmi; Blaschke, Verena; Jauhiainen, Tommi; Ljubešić, Nikola; Nakov, Preslav; Tiedemann, Jörg & Zampieri, Marcos (Ed.), Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISSN 9798891763722. p. 275–292. doi: 10.18653/v1/2026.vardial-1.23. Full text in Research Archive Show summary

  Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During the development we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages.

• Storset, Lilja Charlotte; Jelin, Elma; Norman, Rebecka Maria; Bjertnæs, Øyvind Andresen; Øvrelid, Lilja & Velldal, Erik (2026). From Pain to Praise: Aspect-Based Sentiment Analysis for Norwegian Patient Feedback. In Danilova, Vera; Kurfalı, Murathan; Söderfeldt, Ylva; Reed, Julia & Burchell, Andrew (Ed.), Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026). Association for Computational Linguistics (ACL). ISSN 9798891763678. p. 191–202. doi: 10.18653/v1/2026.healing-1.16. Full text in Research Archive Show summary

  This paper describes a new dataset for aspect-based sentiment analysis (ABSA) for analyzing patient feedback about healthcare services. In an interdisciplinary collaboration spanning the fields of natural language processing and healthcare research, we manually annotate a dataset of 2382 free-text comments collected from national patient experience surveys in Norway, covering two sub-fields of services – special mental healthcare and general practitioners. Annotations are provided on both the sentence- and comment-level, covering a fine-grained set of 25 unique healthcare-related aspects and their polarities. We also report results for fine-tuning both encoder- and decoder models on the resulting dataset, comparing different modeling strategies, like joint and sequential prediction of aspects and polarity. The resources developed in this work can assist healthcare researchers in the analysis of patient feedback, bringing a much more efficient approach compared to today’s manual analysis, potentially leading to improved patient satisfaction and clinical outcomes.

• Papadopoulou, Anthi; Lison, Pierre; Anderson, Mark David; Øvrelid, Lilja & Pilán, Ildikó (2026). Neural text sanitization with privacy risk indicators: an empirical analysis. Language Resources and Evaluation. ISSN 1574-020X. 60. doi: 10.1007/s10579-026-09911-1. Full text in Research Archive Show summary

  Text sanitization is the task of redacting a document to mask all occurrences of (direct or indirect) personal identifiers, with the goal of concealing the identity of the individual(s) referred in it. In this paper, we consider a two-step approach to text sanitization and provide a detailed analysis of its empirical performance on two recently published datasets: the Text Anonymization Benchmark (Pilán et al., 2022) and a collection of Wikipedia biographies (Papadopoulou et al., 2022a). The text sanitization process starts with a privacy-oriented entity recognizer that seeks to determine the text spans expressing identifiable personal information. This privacy-oriented entity recognizer is trained by combining a standard named entity recognition model with a gazetteer populated by person-related terms extracted from Wikidata. The second step of the text sanitization process consists in assessing the privacy risk associated with each detected text span, either isolated or in combination with other text spans. We present five distinct indicators of the re-identification risk, respectively based on language model probabilities, text span classification, sequence labelling, perturbations, and web search. We provide a contrastive analysis of each privacy indicator and highlight their benefits and limitations, notably in relation to the available labeled data.

• Fedorova, Mariia; Kutuzov, Andrey & Umarova, Khonzoda (2026). DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling. In Tahmasebi, Nina; Cassotti, Pierluigi; Montariol, Syrielle; Kutuzov, Andrey; Huebscher, Netta; Spaziani, Elena & Baes, Naomi (Ed.), The Proceedings for the 6th International Workshop on Computational Approaches to Language Change (LChange’26). Association for Computational Linguistics (ACL). ISSN 9798891763623. p. 87–96. doi: 10.18653/v1/2026.lchange-1.7. Full text in Research Archive Show summary

  In this resource paper, we present DHPLT, an open collection of diachronic corpora in 41 diverse languages. DHPLT is based on the web-crawled HPLT datasets; we use web crawl timestamps as the approximate signal of document creation time. The collection covers three time periods: 2011-2015, 2020-2021 and 2024-present (1 million documents per time period for each language). We additionally provide pre-computed word type and token embeddings and lexical substitutions for our chosen target words, while at the same time leaving it open for the other researchers to come up with their own target words using the same datasets. DHPLT aims at filling in the current lack of multilingual diachronic corpora for semantic change modelling (beyond a dozen of high-resource languages). It opens the way for a variety of new experimental setups in this field.

• Wold, Sondre; Simon, Étienne; Velldal, Erik & Øvrelid, Lilja (2026). Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality. In Demberg, Vera; Inui, Kentaro & Marquez, Lluís (Ed.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891763807. p. 2239–2252. doi: https:/aclanthology.org/2026.eacl-long.99/. Full text in Research Archive Show summary

  The principle of compositionality, which concerns the construction of meaning from constituent parts, is a longstanding topic in various disciplines, most commonly associated with formal semantics. In NLP, recent studies have focused on the compositional properties of text embedding models, particularly regarding their sensitivity to idiomatic expression, as idioms have traditionally been seen as non-compositional. In this paper, we argue that it is unclear how previous work relates to formal definitions of the principle. To address this limitation, we take a theoretically motivated approach based on definitions in formal semantics. We present 𝜀-compositionality, a continuous relaxation of compositionality derived from these definitions. We measure 𝜀-compositionality on a dataset containing both idiomatic and non-idiomatic sentences, providing a theoretically motivated assessment of sensitivity to idiomaticity. Our findings indicate that most text embedding models differentiate between idiomatic and non-idiomatic phrases, although to varying degrees.

• Intelligence, Network Regional Language and Artificial; Fischer, Hanna; Lameli, Alfred & Scherrer, Yves (2026). Regionale Sprache und Künstliche Intelligenz im Zeitalter der digitalen Transformation. Zeitschrift für Dialektologie und Linguistik. ISSN 0044-1449. doi: 10.25162/zdl-2026-0001. Full text in Research Archive Show summary

  For research on regional languages, the rapid development of AI technology in recent years offers significant innovative potential. At the same time, the discipline carries a particular responsibility when it comes to representing regional language in the digital sphere. The digital transformation confronts the field with three overarching development goals: (1) harnessing the potential of AI, (2) depicting cultural diversity, and (3) shaping digital transformation responsibly. This position paper defines these developmental goals and, through an assessment of the current situation, outlines the prerequisites and conditions currently in place for the use of AI-based methods in the field. The resulting challenges and desiderata lead to central guidelines that support these goals. Finally, concrete recommendations present measures for advancing research on regional languages.

• Miletić, Aleksandra; Mickus, Timothee & Scherrer, Yves (2026). Explainable linguistic variety classification: The case of Bosnian-Croatian-Montenegrin-Serbian. In Samardžić, Tanja (Eds.), PROCEEDINGS OF THE 1ST ReLDI CONFERENCE ON LANGUAGE SCIENCE AND TECHNOLOGY. ReLDI Centre, Belgrade. ISSN 9788690792207. p. 17–25. doi: 10.5281/zenodo.18488312. Full text in Research Archive Show summary

  This paper presents a preliminary evaluation and analysis of a method for explainable linguistic variety classification applied to Bosnian, Croatian, Montenegrin, and Serbian. Neural models have become the de-facto standard for most NLP tasks and applications, but the factors influencing their predictions remain to a large extent opaque. Using Twitter data, we aim to detect the linguistic features that contribute most to the classification decisions and conduct an extensive manual analysis of these features. The results show that the global model behaviour of neural classifiers aligns with existing knowledge about variation in these varieties, although the models struggle to identify features specific to a single variety, especially in the case of Bosnian and Montenegrin. Overall, the paper showcases the applicability of the examined explainability method in sociolinguistic contexts characterized by complex variation patterns, thereby demonstrating the interest of explainable machine learning methods for corpus-based variation research.

• Samuel, David; Mikhailov, Vladislav; Velldal, Erik; Øvrelid, Lilja; Charpentier, Lucas Georges Gabriel & Kutuzov, Andrey [Show all 7 contributors for this article] (2025). Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 573–608. doi: https:/aclanthology.org/2025.nodalida-1.61/. Full text in Research Archive Show summary

  Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern S{\'a}mi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokm{\r{a}}l, Nynorsk, and Northern S{\'a}mi with 11.4 billion parameters: NorMistral-11B.

• Touileb, Samia; Mikhailov, Vladislav; Kroka, Marie Ingeborg; Øvrelid, Lilja & Velldal, Erik (2025). Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 729–738. doi: https:/aclanthology.org/2025.nodalida-1.73/. Full text in Research Archive Show summary

  We introduce a dataset of high-quality human-authored summaries of news articles in Norwegian. The dataset is intended for benchmarking of the abstractive summarisation capabilities of generative language models. Each document in the dataset is provided with three different candidate gold-standard summaries written by native Norwegian speakers and all summaries are provided in both of the written variants of Norwegian – Bokmål and Nynorsk. The paper describes details on the data creation effort as well as an evaluation of existing open LLMs for Norwegian on the dataset. We also provide insights from a manual human evaluation, comparing human-authored to model generated summaries. Our results indicate that the dataset provides a challenging LLM benchmark for Norwegian summarisation capabilities.

• Bian, Jie; Arefyev, Nikolay; Mühlhäuser, Max & Welzl, Michael (2025). Automated Insights Into GitHub Collaboration Dynamics. IEEE Access. 13, p. 85526–85542. doi: 10.1109/access.2025.3566309. Full text in Research Archive Show summary

  Today, GitHub (Trademark) is the most widely used platform for open-source software development. Large projects may comprise hundreds of distributed collaborators and thousands of GitHub “issues” (structured discussions). It includes basic support for dealing with issues, via “Pull Requests” (PRs)—document changes that can manually be defined to “close” them (i.e., they address and thereby conclude the issue discussion). Unresolved issues can pile up. For example, at the time of writing, the Kubernetes repository has almost 2000 open issues; finding which ones a PR might close is a hard task by itself. We address this by automatically identifying issue-PR relationships using language models (LMs) and leveraging Information Retrieval (IR) techniques. To foster further research, we contribute a carefully curated novel dataset called CodeConvo, reflecting the most influential open-source repositories for code development as well as some technical document repositories. We use this dataset to benchmark several state-of-the-art (SOTA) non-proprietary models that show exceptional performance on the MTEB benchmark, as well as to train and evaluate the performance of Smart Insights into GitHub Issue-PR Relations (SIGIR), our tailored model. The best SIGIR model/data combination yields an average Mean Reciprocal Rank (MRR) above 0.7, around 20% higher than the best baseline performance. Notably, ablation studies revealed that knowledge transfer occurs not only between different programming languages but also between code and technical documents, albeit to a lesser extent. We believe these results are encouraging and can stimulate the practical application of LMs for taming the complexity of very large projects, in GitHub and beyond.

• Simon, Étienne; Olsen, Helene Bøsei; Villar, Ramón Carreño; Mishra, Rahul; Arefyev, Nikolay & Yilmaz, Mert Can [Show all 8 contributors for this article] (2025). Abstractive Event Analysis of Armed Conflicts: Introducing the UCDP-AEC Dataset. In Wartena, Christian & Heid, Ulrich (Ed.), Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops. HsH Applied Academics. ISSN 9783690180177. p. 104–119. doi: https:/aclanthology.org/2025.konvens-2.8.pdf. Full text in Research Archive Show summary

  This paper introduces a new dataset of document-level event annotations in the domain of armed conflict. By augmenting the event database from the Uppsala Conflict Data Program (UCDP) with source documents identified in public web archives, we create the UCDP Abstractive Event analysis Corpus (UCDP-AEC). While a large part of research on information extraction is focused on extracting text spans, realworld use cases often require inferring more high-level information that is not necessarily explicitly mentioned in texts. UCDP-AEC differs from traditional event extraction datasets in that the document-level annotations do not correspond to mere text spans of the input, but capture expert-interpreted and often implicit information. With more than 10 000 documents, UCDP-AEC is of comparable size to the largest human-annotated traditional event extraction datasets. We also report preliminary experimental results for various generative approaches, by fine-tuning both decoder models and existing event argument extraction models that require minimal adaptation to our abstractive formulation of the task

• Rowe, Jacqueline; Gibert, Ona De; Klimaszewski, Mateusz; Haley, Coleman; Birch, Alexandra & Scherrer, Yves (2025). EdinHelsOW WMT 2025 CreoleMT System Description: Improving Lusophone Creole Translation through Data Augmentation, Model Merging and LLM Post-editing. In Haddow, Barry; Kocmi, Tom; Koehn, Philipp & Monz, Christof (Ed.), Proceedings of the Tenth Conference on Machine Translation. Association for Computational Linguistics (ACL). ISSN 9798891763418. p. 1166–1182. doi: 10.18653/v1/2025.wmt-1.91. Full text in Research Archive Show summary

  In this work, we present our submissions to the unconstrained track of the System subtask of the WMT 2025 Creole Language Translation Shared Task. Of the 52 Creole languages included in the task, we focus on translation between English and seven Lusophone Creoles. Our approach leverages known strategies for low-resource machine translation, including back-translation and distillation of data, fine-tuning pre-trained multilingual models, and post-editing with large language models and lexicons. We also demonstrate that adding high-quality parallel Portuguese data in training, initialising Creole embeddings with Portuguese embedding weights, and strategically merging best checkpoints of different fine-tuned models all produce considerable gains in performance in certain translation directions. Our best models outperform the baselines on the Task test set for eight out of fourteen translation directions. When evaluated on a more diverse test set, they surpass the baselines in all but one direction.

• Mæhlum, Petter; Evensen, Anders Næss & Scherrer, Yves (2025). Improved Norwegian Bokmål Translations for FLORES. In Haddow, Barry; Kocmi, Tom; Koehn, Philipp & Monz, Christof (Ed.), Proceedings of the Tenth Conference on Machine Translation. Association for Computational Linguistics (ACL). ISSN 9798891763418. p. 1124–1132. doi: 10.18653/v1/2025.wmt-1.86. Full text in Research Archive Show summary

  FLORES+ is a collection of parallel datasets obtained by translation from originally English source texts. FLORES+ contains Norwegian translations for the two official written variants of Norwegian: Norwegian Bokmål and Norwegian Nynorsk. However, the earliest Bokmål version contained non-native-like mistakes, and even after a later revision, the dataset contained grammatical and lexical errors. This paper aims at correcting unambiguous mistakes, and thus creating a new version of the Bokmål dataset. At the same time, we provide a translation into Radical Bokmål, a sub-variety of Norwegian which is closer to Nynorsk in some aspects, while still being within the official norms for Bokmål. We discuss existing errors and differences in the various translations and the corrections that we provide.

• Fedorova, Mariia; Kutuzov, Andrey; Periti, Francesco & Scherrer, Yves (2025). Explaining novel senses using definition generation with open language models. In Christodoulopoulos, Christos; Chakraborty, Tanmoy; Rose, Carolyn & Peng, Violet (Ed.), Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics (ACL). ISSN 9798891763357. p. 22294–22302. doi: 10.18653/v1/2025.findings-emnlp.1214. Full text in Research Archive Show summary

  We apply definition generators based on open-weights large language models to the task of creating explanations of novel senses, taking target word usages as an input. To this end, we employ the datasets from the AXOLOTL’24 shared task on explainable semantic change modeling, which features Finnish, Russian and German languages. We fine-tune and provide publicly the open-source models performing higher than the best submissions of the aforementioned shared task, which employed closed proprietary LLMs. In addition, we find that encoder-decoder definition generators perform on par with their decoder-only counterparts.

• Jelin, Elma; Storset, Lilja Charlotte; Norman, Rebecka Maria; Iversen, Hilde Karin Hestad; Ellingsen-Dalskau, Lina Harvold & Mæhlum, Petter [Show all 9 contributors for this article] (2025). From words to action? A scoping review on automatic sentiment analysis of patient experience comments from online sources and surveys. BMJ Health & Care Informatics. 32(1). doi: 10.1136/bmjhci-2025-101631. Full text in Research Archive Show summary

  Background Automatic analysis of free-text patient comments enables the efficient processing of large feedback volumes, reducing reliance on manual review. A 2021 review examined natural language processing (NLP) and sentiment analysis (SA) in patient experience research; however, recent advances in deep learning and generative artificial intelligence (AI) call for an updated synthesis. Objectives This scoping review aims to map and summarise recent studies applying SA to unstructured patient experience data related to healthcare services. Methods Following Joanna Briggs Institute methodology and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we conducted a comprehensive search across Medline, CINAHL, Web of Science, Cochrane, Embase and APA PsycINFO. We included studies published from January 2020 to March 2024 in English or a Scandinavian language. Eligible studies analysed patient feedback using NLP techniques and described the development or validation of SA models. Two reviewers independently screened the studies and extracted data, which were presented in tables both tabular and narrative forms. Results 30 studies were included, primarily from the USA, Europe and Asia. Patient comments were mostly sourced from online platforms such as social media. Feedback largely concerned hospital care. 18 studies employed rule-based SA approaches, while 12 applied supervised machine learning (ML) and only 4 studies used deep learning models. Few addressed the visualisation or practical applications in healthcare. Conclusion Despite significant progress, modern methods like deep learning and generative AI remain underused in SA of patient-experience data. Limited focus on implementation restricts SA’s role in quality improvement. Future research should assess advanced methods and their cost-effectiveness versus traditional ML.

• Mikhailov, Vladislav; Artemova, Ekaterina; Butenko, Zoia; Øvrelid, Lilja & Velldal, Erik (2025). Overview of the Preference Prediction Task at the ELOQUENT 2025 lab for evaluating generative language model quality. CEUR Workshop Proceedings. 4038, p. 1320–1329. doi: https:/ceur-ws.org/Vol-4038/paper_105.pdf. Full text in Research Archive Show summary

  The Preference Prediction shared task, part of the ELOQUENT Lab at CLEF-2025, challenges participants to build systems that simulate human judgment in evaluating pairs of LLM-generated responses. This novel task includes two sub-tasks. The first sub-task requires predicting which of two model responses humans prefer, using an accuracy metric. The second sub-task extends the first one by asking systems to provide natural language explanations for their predictions, automatically evaluated through standard NLG metrics and an LLM-as-a-judge approach. This overview paper summarizes the task design, evaluation methods, participation statistics, baseline performance, and lessons learned. We discuss the system performance in both sub-tasks. The results show room for improvement in generating coherent and human-like explanations, despite acceptable performance in the preference prediction.

• Karlgren, Jussi; Artemova, Ekaterina; Bojar, Ondřej; Engels, Marie Isabel; Mikhailov, Vladislav & Šindelář, Pavel [Show all 8 contributors for this article] (2025). Overview of ELOQUENT 2025: Shared Tasks for Evaluating Generative Language Model Quality. In Carrillo-de-Albornoz, Jorge (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2025).. Springer Publishing Company. ISSN 9783032043542. doi: 10.1007/978-3-032-04354-2_14. Full text in Research Archive
• Berg, Margareta; Pilán, Ildikó; Falkum, Ingrid Lossius & Lison, Pierre (2025). Pragmatic Reasoning for Irony Detection With Large Language Models in English and Norwegian - SEMDIAL. SemDial Proceedings. p. 204–209. doi: https:/www.semdial.org/anthology/Z25-Berg_semdial_4421.pdf. Full text in Research Archive Show summary

  This study investigates the ‘pragmatic abilities’ of large language models (LLMs) – both standard and reasoning-optimized – across two languages (English and Norwegian). Based on an existing experimental study on children’s irony comprehension, we found that LLMs largely identified irony, but performance was poorer in Norwegian due to translation challenges.

• You, Huiling; Touileb, Samia; Velldal, Erik & Øvrelid, Lilja (2025). Event-based evaluation of abstractive news summarization. In Arviv, Ofir; Clinciu, Miruna; Dhole, Kaustubh; Dror, Rotem; Gehrmann, Sebastian; Habba, Eliya; Itzhak, Itay; Mille, Simon; Perlitz, Yotam; Santus, Enrico; Sedoc, João; Scheuer, Michal Shmueli; Stanovsky, Gabriel & Tafjord, Oyvind (Ed.), Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). Association for Computational Linguistics (ACL). ISSN 9798891762619. p. 504–510. doi: https:/aclanthology.org/2025.gem-1.43/. Full text in Research Archive
• Scherrer, Yves; Goot, Rob van der & Mæhlum, Petter (2025). Findings of the VarDial Evaluation Campaign 2025: The NorSID Shared Task on Norwegian Slot, Intent and Dialect Identification. In Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Nakov, Preslav; Tiedemann, Jörg & Zampieri, Marcos (Ed.), Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISSN 9798891762084. p. 1–8. Full text in Research Archive
• Fedorova, Mariia; Frydenberg, Jonas Sebulon; Handford, Victoria; Langø, Victoria Ovedie Chruickshank; Willoch, Solveig Helene & Midtgaard, Marthe Løken [Show all 9 contributors for this article] (2025). Multi-label Scandinavian Language Identification (SLIDE). In Holdt, Špela Arhar; Ilinykh, Nikolai; Scalvini, Barbara; Bruton, Micaella; Debess, Iben Nyholm & Tudor, Crina Madalina (Ed.), Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025). Tartu Ülikool Kirjastus. ISSN 9789908531212. p. 179–189. Full text in Research Archive Show summary

  Identifying closely related languages at sentence level is difficult, in particular because it is often impossible to assign a sentence to a single language. In this paper, we focus on multi-label sentence-level Scandinavian language identification (LID) for Danish, Norwegian Bokmål, Norwegian Nynorsk, and Swedish. We present the Scandinavian Language Identification and Evaluation, SLIDE, a manually curated multi-label evaluation dataset and a suite of LID models with varying speed–accuracy tradeoffs. We demonstrate that the ability to identify multiple languages simultaneously is necessary for any accurate LID method, and present a novel approach to training such multi-label LID models.

• Midtgaard, Marthe Løken; Mæhlum, Petter & Scherrer, Yves (2025). LTG at VarDial 2025 NorSID: More and Better Training Data for Slot and Intent Detection. In Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Nakov, Preslav; Tiedemann, Jörg & Zampieri, Marcos (Ed.), Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISSN 9798891762084. p. 200–208. Full text in Research Archive Show summary

  This paper describes the LTG submission to the VarDial 2025 shared task, where we participate in the Norwegian slot and intent detection subtasks. The shared task focuses on Norwegian dialects, which present challenges due to their low-resource nature and variation. We test a variety of neural models and training data configurations, with the focus on improving and extending the available Norwegian training data. This includes automatically re-aligning slot spans in Norwegian Bokmål, as well as re-translating the original English training data into both Bokmål and Nynorsk. % to address dialectal diversity. We also re-annotate an external Norwegian dataset to augment the training data. Our best models achieve first place in both subtasks, achieving an span F1 score of 0.893 for slot filling and an accuracy of 0.980 for intent detection. Our results indicate that while translation quality is less critical, improving the slot labels has a notable impact on slot performance. Moreover, adding more standard Norwegian data improves performance, but incorporating even small amounts of dialectal data leads to greater gains.

• Rabus, Achim & Scherrer, Yves (2025). Dialects, Topic Models, and Border Effects: The Rusyn Case, Proceedings of the 10th Workshop on Slavic Natural Language Processing (Slavic NLP 2025). Association for Computational Linguistics (ACL). ISSN 9781959429579. p. 38–43. doi: 10.18653/v1/2025.bsnlp-1.5. Full text in Research Archive Show summary

  In this contribution, we present, discuss, and apply a data-driven approach for analyzing varieties of the Slavic minority language Carpathian Rusyn spoken in different countries in the Carpathian region. Using topic modeling, a method originally developed for text mining, we show that the Rusyn varieties are subject to border effects, i.e., vertical convergence and horizontal divergence, due to language contacts with their respective umbrella languages Polish, Slovak and Standard Ukrainian. Additionally, we show that the method is suitable for uncovering fieldworker isoglosses, i.e., different transcription principles in an otherwise homogeneous dataset.

• Storset, Lilja Charlotte; Rønningstad, Egil; Mæhlum, Petter; Velldal, Erik & Øvrelid, Lilja (2025). Mixed Feelings: Cross-Domain Sentiment Classification of Patient Feedback. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 537–543. doi: https:/aclanthology.org/2025.nodalida-1.58/. Full text in Research Archive Show summary

  Sentiment analysis of patient feedback from the public health domain can aid decision makers in evaluating the provided services. The current paper focuses on free-text comments in patient surveys about general practitioners and psychiatric healthcare, annotated with four sentence-level polarity classes - positive, negative, mixed and neutral - while also attempting to alleviate data scarcity by leveraging general-domain sources in the form of reviews. For several different architectures, we compare in-domain and out-of-domain effects, as well as the effects of training joint multi-domain models.

• Wold, Sondre; Charpentier, Lucas Georges Gabriel & Simon, Étienne (2025). Systematic Generalization in Language Models Scales with Information Entropy. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). ISSN 9798891762565. p. 1807–1819. doi: 10.18653/v1/2025.findings-acl.90. Full text in Research Archive Show summary

  Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although benchmarks exist for assessing compositional behavior, it is unclear how to measure the difficulty of a systematic generalization problem. In this work, we show how one aspect of systematic generalization can be described by the entropy of the distribution of component parts in the training data. We formalize a framework for measuring entropy in a sequence-to-sequence task and find that the performance of popular model architectures scales with the entropy. Our work connects systematic generalization to information efficiency, and our results indicate that success at high entropy can be achieved even without built-in priors, and that success at low entropy can serve as a target for assessing progress towards robust systematic generalization.

• Mondorf, Philipp; Wold, Sondre & Plank, Barbara (2025). Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 14934–14955. doi: 10.18653/v1/2025.acl-long.727. Full text in Research Archive Show summary

  A fundamental question in interpretability research is to what extent neural networks, particularly language models, implement reusable functions through subnetworks that can be composed to perform more complex tasks. Recent advances in mechanistic interpretability have made progress in identifying circuits, the minimal computational subgraphs responsible for a model’s behavior on specific tasks. However, most studies focus on identifying circuits for individual tasks without investigating how functionally similar circuits relate to each other. To address this gap, we study the modularity of neural networks by analyzing circuits for highly compositional subtasks within a transformer-based language model. Specifically, given a probabilistic context-free grammar, we identify and compare circuits responsible for ten modular string-edit operations. Our results indicate that functionally similar circuits exhibit both notable node overlap and cross-task faithfulness. Moreover, we demonstrate that the circuits identified can be reused and combined through set operations to represent more complex functional model capabilities.

• Charpentier, Lucas Georges Gabriel & Lison, Pierre (2025). Re-identification of De-identified Documents with Autoregressive Infilling. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 1192–1209. doi: 10.18653/v1/2025.acl-long.60. Full text in Research Archive Show summary

  Documents revealing sensitive information about individuals must typically be de-identified. This de-identification is often done by masking all mentions of personally identifiable information (PII), thereby making it more difficult to uncover the identity of the person(s) in question. To investigate the robustness of de-identification methods, we present a novel, RAG-inspired approach that attempts the reverse process of re-identification based on a database of documents representing background knowledge. Given a text in which personal identifiers have been masked, the re-identification proceeds in two steps. A retriever first selects from the background knowledge passages deemed relevant for the re-identification. Those passages are then provided to an infilling model which seeks to infer the original content of each text span. This process is repeated until all masked spans are replaced. We evaluate the re-identification on three datasets (Wikipedia biographies, court rulings and clinical notes). Results show that (1) as many as 80% of de-identified text spans can be successfully recovered and (2) the re-identification accuracy increases along with the level of background knowledge.

• Mikhailov, Vladislav; Enstad, Tita Ranveig; Samuel, David; Farsethås, Hans Christian; Velldal, Erik & Øvrelid, Lilja [Show all 7 contributors for this article] (2025). NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). ISSN 9798891762565. p. 3495–3541. doi: 10.18653/v1/2025.findings-acl.181. Full text in Research Archive Show summary

  This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets – of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-created prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pretrained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

• Mikhailov, Vladislav; Mæhlum, Petter; Langø, Victoria; Velldal, Erik & Øvrelid, Lilja (2025). A Collection of Question Answering Datasets for Norwegian. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 397–407. doi: https:/aclanthology.org/2025.nodalida-1.43/. Full text in Research Archive Show summary

  This paper introduces a new suite of question answering datasets for Norwegian; NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA. The data covers a wide range of skills and knowledge domains, including world knowledge, commonsense reasoning, truthfulness, and knowledge about Norway. Covering both of the written standards of Norwegian – Bokmål and Nynorsk – our datasets comprise over 10k question-answer pairs, created by native speakers. We detail our dataset creation approach and present the results of evaluating 11 language models (LMs) in zero- and few-shot regimes. Most LMs perform better in Bokmål than Nynorsk, struggle most with commonsense reasoning, and are often untruthful in generating answers to questions. All our datasets and annotation materials are publicly available.

• You, Huiling; Touileb, Samia; Velldal, Erik & Øvrelid, Lilja (2025). NorEventGen: generative event extraction from Norwegian news. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 801–811. doi: https:/aclanthology.org/2025.nodalida-1.79/. Full text in Research Archive Show summary

  In this work, we approach event extraction from Norwegian news text using a generation-based approach which formulates the task as text-to-structure generation. We present experiments assessing the effect of different modeling configurations and provide an analysis of the model predictions and typical system errors. Finally, we apply our system to a large corpus of raw news texts and analyze the resulting distribution of event structures in a fairly representative snap-shot of the Norwegian news landscape.

• Burchell, Laurie; Gibert, Ona de; Arefev, Nikolay; Aulamo, Mikko; Bañón, Marta & Chen, Pinzhen [Show all 35 contributors for this article] (2025). An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT). In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 17452–17485. doi: 10.18653/v1/2025.acl-long.854. Full text in Research Archive Show summary

  Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

• Kennington, Casey; Lison, Pierre & Schlangen, David (2025). Incremental Dialogue Management: Survey, Discussion, and Implications for HRI. arXiv. Full text in Research Archive Show summary

  Efforts towards endowing robots with the ability to speak have benefited from recent advancements in NLP, in particular large language models. However, as powerful as current models have become, they still operate on sentence or multi-sentence level input, not on the word-by-word input that humans operate on, affecting the degree of responsiveness that they offer, which is critical in situations where humans interact with robots using speech. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and the implications of incremental dialogue for embodied, robotic platforms.

• Scherrer, Yves & Kuparinen, Olli (2025). Interactive maps for corpus-based dialectology. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 634–638. doi: https:/aclanthology.org/2025.nodalida-1.64/. Full text in Research Archive Show summary

  Traditional data collection methods in dialectology rely on structured surveys, whose results can be easily presented on printed or digital maps. But in recent years, corpora of transcribed dialect speech have become a precious alternative data source for data-driven linguistic analysis. For example, topic models can be advantageously used to discover both general dialectal variation patterns and specific linguistic features that are most characteristic for certain dialects. Multilingual (or rather, multilectal) language modeling tasks can also be used to learn speaker-specific embeddings. In connection with this paper, we introduce a website that presents the results of two recent studies in the form of interactive maps, allowing visitors to explore the effects of various parameter settings. The website covers two tasks (topic models and speaker embeddings) and three language areas (Finland, Norway, and German-speaking Switzerland). It is available at https://www.corcodial.net/ .

• Hopton, Zachary; Scherrer, Yves & Samardžić, Tanja (2025). Functional Lexicon in Subword Tokenization, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics (ACL). ISSN 9798891761896. p. 7839–7854. doi: 10.18653/v1/2025.naacl-long.398. Full text in Research Archive Show summary

  The distinction between function and content units of the lexicon has been somewhat neglected in recent NLP work, but it could still be useful when working with low-resource languages, and, in particular, to improve cross-lingual transfer. In this paper, we investigate to what extent BPE subword tokenization can be used to identify units of the functional lexicon in a language without any annotated data. We analyze subword tokens in terms of their productivity and attempt to find thresholds that best distinguish function from content tokens. On a sample of seven diverse languages, we find that the best results are obtained with 50 BPE merges. We also show that this subword tokenization setting can be beneficial for the interlinear glossing task.

• Gibert, Ona de; Nieminen, Tommi; Scherrer, Yves & Tiedemann, Jörg (2025). OpusDistillery: A Configurable End-to-End Pipeline for Systematic Multilingual Distillation of Open NMT Models. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 201–208. doi: https:/aclanthology.org/2025.nodalida-1.20/. Full text in Research Archive Show summary

  In this work, we introduce OpusDistillery, a novel framework to streamline the Knowledge Distillation (KD) process of multilingual NMT models. OpusDistillery’s main features are the integration of openly available teacher models from OPUS-MT and Hugging Face, comprehensive multilingual support and robust GPU utilization tracking. We describe the tool in detail and discuss the individual contributions of its pipeline components, demonstrating its flexibility for different use cases. OpusDistillery is open-source and released under a permissive license, aiming to facilitate further research and development in the field of multilingual KD for any sequence-to-sequence task. Our code is available at https://github.com/Helsinki-NLP/OpusDistillery.

• Rosa, Javier De La; Mikhailov, Vladislav; Zhang, Lemei; Wetjen, Freddy; Samuel, David & Liu, Peng [Show all 19 contributors for this article] (2025). The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 544–560. Full text in Research Archive
• Wold, Sondre; Simon, Étienne; Charpentier, Lucas Georges Gabriel; Kostylev, Egor V.; Velldal, Erik & Øvrelid, Lilja (2024). Compositional Generalization with Grounded Language Models, Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760998. p. 3447–3460. doi: 10.18653/v1/2024.findings-acl.205. Full text in Research Archive Show summary

  Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous work on compositional generalization in semantic parsing, we allow for a controlled evaluation of the degree to which these models learn and generalize from patterns in knowledge graphs. We develop a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and further avoids grounding the language models in information already encoded implicitly in their weights. We evaluate existing methods for combining language models with knowledge graphs and find them to struggle with generalization to sequences of unseen lengths and to novel combinations of seen base components. While our experimental results provide some insight into the expressive power of these models, we hope our work and released datasets motivate future research on how to better combine language models with structured knowledge representations.

• Hussiny, Mohammad Ali; Payenda, Mohammad Arif & Øvrelid, Lilja (2024). PersianEmo: Enhancing Farsi-Dari Emotion Analysis with a Hybrid Transformer and Recurrent Neural Network Model. Proceedings of LREC. p. 257–263. Full text in Research Archive Show summary

  Emotion analysis is a critical research domain within the field of natural language processing (NLP). While substantial progress has been made in this area for the Persian language, there is still a need for more precise models and larger datasets specifically focusing on the Farsi and Dari dialects. In this research, we introduce “LearnArmanEmo” as a new dataset and a superior ensemble approach for Persian text emotion classification. Our proposed model, which combines XLM-RoBERTa-large and BiGRU, undergoes evaluation on LetHerLearn for the Dari dialect, ARMANEMO for the Farsi dialect, and LearnArmanEmo for both Dari and Farsi dialects. The empirical results substantiate the efficacy of our approach with the combined model demonstrating superior performance. Specifically, our model achieves an F1 score of 72.9% on LetHerLearn, an F1 score of 77.1% on ARMANEMO, and an F1 score of 78.8% on the LearnArmanEmo dataset, establishing it as a better ensemble model for these datasets. These findings underscore the potential of this hybrid model as a useful tool for enhancing the performance of emotion analysis in Persian language processing.

• Chen, Pinzhen; Ji, Shaoxiong; Bogoychev, Nikolay; Kutuzov, Andrei; Haddow, Barry & Heafield, Kenneth (2024). Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca, Findings of the Association for Computational Linguistics: EACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760936. p. 1347–1356. Full text in Research Archive
• Olsen, Helene Bøsei; Simon, Étienne; Velldal, Erik & Øvrelid, Lilja (2024). Socio-political Events of Conflict and Unrest: A Survey of Available Datasets. In Hürriyetoğlu, Ali; Tanev, Hristo; Surendrabikram, Thapa & Uludoğan, Gökçe (Ed.), Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024). Association for Computational Linguistics (ACL). ISSN 9798891760707. p. 40–53. Full text in Research Archive
• Charpentier, Lucas Georges Gabriel & Samuel, David (2024). GPT or BERT: why not both? In Hu, Michael Y.; Mueller, Aaron; Ross, Candace; Williams, Adina; Linzen, Tal; Zhuang, Chengxu; Choshen, Leshem; Cotterell, Ryan; Warstadt, Alex & Wilcox, Ethan Gotlieb (Ed.), The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning. Association for Computational Linguistics (ACL). ISSN 9798891762220. p. 262–283. Full text in Research Archive
• Fedorova, Mariia; Mickus, Timothee; Partanen, Niko; Siewert, Janine; Spaziani, Elena & Kutuzov, Andrei (2024). AXOLOTL’24 Shared Task on Multilingual Explainable Semantic Change Modeling, Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9798891761384. p. 72–91. Full text in Research Archive Show summary

  This paper describes the organization and findings of AXOLOTL’24, the first multilingual explainable semantic change modeling shared task. We present new sense-annotated diachronic semantic change datasets for Finnish and Russian which were employed in the shared task, along with a surprise test-only German dataset borrowed from an existing source. The setup of AXOLOTL’24 is new to the semantic change modeling field, and involves subtasks of identifying unknown (novel) senses and providing dictionary-like definitions to these senses. The methods of the winning teams are described and compared, thus paving a path towards explainability in computational approaches to historical change of meaning.

• Fedorova, Mariia; Kutuzov, Andrei & Scherrer, Yves (2024). Definition generation for lexical semantic change detection, Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760998. p. 5712–5724. Full text in Research Archive Show summary

  We use contextualized word definitions generated by large language models as semantic representations in the task of diachronic lexical semantic change detection (LSCD). In short, generated definitions are used as ‘senses’, and the change score of a target word is retrieved by comparing their distributions in two time periods under comparison. On the material of five datasets and three languages, we show that generated definitions are indeed specific and general enough to convey a signal sufficient to rank sets of words by the degree of their semantic change over time. Our approach is on par with or outperforms prior non-supervised sense-based LSCD methods. At the same time, it preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. This is another step in the direction of explainable semantic change modeling.

• Samuel, David (2024). BERTs are Generative In-Context Learners. In Globerson, Amir; Mackey, Lester; Belgrave, Danielle; Fan, Angela; Paquet, Ulrich; Tomczak, Jakub & Zhang, Cheng (Ed.), Advances in Neural Information Processing Systems 37 (NeurIPS 2024). Curran Associates, Inc.. ISSN 9781713899921. p. 2558–2589. doi: 10.52202/079017-0084. Full text in Research Archive Show summary

  While in-context learning is commonly associated with causal language models, such as GPT, we demonstrate that this capability also 'emerges' in masked language models. Through an embarrassingly simple inference technique, we enable an existing masked model, DeBERTa, to perform generative tasks without additional training or architectural changes. Our evaluation reveals that the masked and causal language models behave very differently, as they clearly outperform each other on different categories of tasks. These complementary strengths suggest that the field's focus on causal models for in-context learning may be limiting – both architectures can develop these capabilities, but with distinct advantages; pointing toward promising hybrid approaches that combine the strengths of both objectives.

• Mæhlum, Petter; Samuel, David; Norman, Rebecka Maria; Jelin, Elma; Bjertnæs, Øyvind Andresen & Øvrelid, Lilja [Show all 7 contributors for this article] (2024). It’s Difficult to be Neutral – Human and LLM-based Sentiment Annotation of Patient Comments, Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health). ELRA Language Resources Association. ISSN 9782493814258. p. 8–19. Full text in Research Archive
• Fedorova, Mariia; Kutuzov, Andrei; Arefev, Nikolay & Schlechtweg, Dominik (2024). Enriching Word Usage Graphs with Cluster Definitions. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 6189–6198. Full text in Research Archive Show summary

  We present a dataset of word usage graphs (WUGs), where the existing WUGs for multiple languages are enriched with cluster labels functioning as sense definitions. They are generated from scratch by fine-tuned encoder-decoder language models. The conducted human evaluation has shown that these definitions match the existing clusters in WUGs better than the definitions chosen from WordNet by two baseline systems. At the same time, the method is straightforward to use and easy to extend to new languages. The resulting enriched datasets can be extremely helpful for moving on to explainable semantic change modeling.

• Attieh, Joseph; Hopton, Zachary; Scherrer, Yves & Samardžić, Tanja (2024). System Description of the NordicsAlps Submission to the AmericasNLP 2024 Machine Translation Shared Task. In Mager, Manuel; Ebrahimi, Abteen; Rijhwani, Shruti; Oncevay, Arturo; Chiruzzo, Luis; Pugh, Robert & Wense, Katharina von der (Ed.), Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024). Association for Computational Linguistics (ACL). ISSN 9798891761087. p. 150–158. doi: 10.18653/v1/2024.americasnlp-1.18. Full text in Research Archive
• Kuparinen, Olli & Scherrer, Yves (2024). Corpus-based dialectometry with topic models. Journal of Linguistic Geography. 12(1), p. 1–12. doi: 10.1017/jlg.2024.6. Full text in Research Archive
• Mæhlum, Petter & Scherrer, Yves (2024). NoMusic - The Norwegian Multi-Dialectal Slot and Intent Detection Corpus. In Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Zampieri, Marcos; Nakov, Preslav & Tiedemann, Jörg (Ed.), Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024). Association for Computational Linguistics (ACL). ISSN 9798891761049. doi: 10.18653/v1/2024.vardial-1.9. Full text in Research Archive
• Gibert, Ona de; Aulamo, Mikko; Scherrer, Yves & Tiedemann, Jörg (2024). Hybrid Distillation from RBMT and NMT: Helsinki-NLP’s Submission to the Shared Task on Translation into Low-Resource Languages of Spain. In Haddow, Barry; Kocmi, Tom; Koehn, Philipp & Monz, Christof (Ed.), Proceedings of the Ninth Conference on Machine Translation. Association for Computational Linguistics (ACL). ISSN 9798891761797. p. 908–917. doi: 10.18653/v1/2024.wmt-1.88. Full text in Research Archive
• Chifu, Adrian-Gabriel; Glavaš, Goran; Ionescu, Radu Tudor; Ljubešic, Nikola; Miletić, Aleksandra & Miletić, Filip [Show all 8 contributors for this article] (2024). VarDial Evaluation Campaign 2024: Commonsense Reasoning in Dialects and Multi-Label Similar Language Identification. In Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Zampieri, Marcos; Nakov, Preslav & Tiedemann, Jörg (Ed.), Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024). Association for Computational Linguistics (ACL). ISSN 9798891761049. doi: 10.18653/v1/2024.vardial-1.1. Full text in Research Archive
• Gibert, Ona de; Nail, Graeme; Arefev, Nikolay; Bañón, Marta; Linde, Jelmer van der & Ji, Shaoxiong [Show all 13 contributors for this article] (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 1116–1128. Full text in Research Archive Show summary

  We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

• Wold, Sondre; Mæhlum, Petter & Hove, Oddbjørn (2024). Estimating Lexical Complexity from Document-Level Distributions. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 6309–6318. Full text in Research Archive
• Samuel, David; Charpentier, Lucas Georges Gabriel & Wold, Sondre (2024). More room for language: Investigating the effect of retrieval on language models. In Duh, Kevin; Gomez, Helena & Bethard, Steven (Ed.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). Association for Computational Linguistics (ACL). ISSN 9798891761155. p. 282–305. Full text in Research Archive
• Bian, Jie; Welzl, Michael; Kutuzov, Andrei & Arefyev, Nikolay (2024). Tell Me Why: Language Models Help Explain the Rationale Behind Internet Protocol Design. In LI, Geoffrey Ye; Liang, Le; Gündüz, Deniz & Antón-Haro, Carles (Ed.), 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798350343199. p. 447–453. doi: 10.1109/ICMLCN59089.2024.10624781. Full text in Research Archive Show summary

  Request for Comments (RFCs) serve as guidebooks for the implementation of Internet protocols or network mechanisms. They reveal how these protocols and mechanisms work, but the underlying reasons for their operation are not always available in RFCs. We present an attempt to automate the discovery of these reasons in mailing list archives via natural language processing methods. Our approach leverages the known relationship between text changes and the discussions leading to these changes in the Working Group (WG) GitHub repositories of the Internet Engineering Task Force (IETF) to obtain labeled training data. We find that our model is able to generalize, and it can indeed discover emails that have led to formulations in RFCs. This is a first step towards facilitating a deeper understanding of these often complex documents, which can be helpful for developers, protocol designers, and educators alike.

• Touileb, Samia; Murstad, Jeanett; Mæhlum, Petter; Steskal, Lubos; Storset, Lilja Charlotte & You, Huiling [Show all 7 contributors for this article] (2024). EDEN: A Dataset for Event Detection in Norwegian News. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 5495–5506. Full text in Research Archive Show summary

  We present EDEN, the first Norwegian dataset annotated with event information at the sentence level, adapting the widely used ACE event schema to Norwegian. The paper describes the manual annotation of Norwegian text as well as transcribed speech in the news domain, together with inter-annotator agreement and discussions of relevant dataset statistics. We also present preliminary modeling results using a graph-based event parser. The resulting dataset will be made freely available for download and use.

• Rønningstad, Egil; Velldal, Erik & Øvrelid, Lilja (2024). A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation. In Henning, Sophie & Stede, Manfred (Ed.), Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII). Association for Computational Linguistics (ACL). ISSN 9798891760738. p. 133–139. Full text in Research Archive Show summary

  We investigate annotator variation for the novel task of Entity-Level Sentiment Analysis (ELSA) which annotates the aggregated sentiment directed towards volitional entities in a text. More specifically, we analyze the annotations of a newly constructed Norwegian ELSA dataset and release additional data with each annotator’s labels for the 247 entities in the dataset’s test split. We also perform a number of experiments prompting ChatGPT for these sentiment labels regarding each entity in the text and compare the generated annotations with the human labels. Cohen’s Kappa for agreement between the best LLM-generated labels and curated gold was 0.425, which indicates that these labels would not have high quality. Our analyses further investigate the errors that ChatGPT outputs, and compare them with the variations that we find among the 5 trained annotators that all annotated the same test data.

• Hassan, Syed Zohaib; Lison, Pierre & Halvorsen, Pål (2024). Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion. arXiv. doi: 10.48550/arXiv.2412.12710. Full text in Research Archive Show summary

  Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

• Simon, Étienne; Olsen, Helene Bøsei; You, Huiling; Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2024). Generative Approaches to Event Extraction: Survey and Outlook. In Tetreault, Joel; Nguyen, Thien Huu; Lamba, Hemank & Hughes, Amanda (Ed.), Proceedings of the Workshop on the Future of Event Detection (FuturED). Association for Computational Linguistics (ACL). ISSN 9798891761735. p. 73–86. doi: 10.18653/v1/2024.futured-1.7. Full text in Research Archive
• Kokosinskii, Denis; Kuklin, Mikhail & Arefyev, Nikolay (2024). Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modeling, Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9798891761384. p. 168–179. doi: 10.18653/v1/2024.lchange-1.16. Full text in Research Archive Show summary

  This paper describes our solution of the first subtask from the AXOLOTL-24 shared task on Semantic Change Modeling. The goal of this subtask is to distribute a given set of usages of a polysemous word from a newer time period between senses of this word from an older time period and clusters representing gained senses of this word. We propose and experiment with three new methods solving this task. Our methods achieve SOTA results according to both official metrics of the first substask. Additionally, we develop a model that can tell if a given word usage is not described by any of the provided sense definitions. This model serves as a component in one of our methods, but can potentially be useful on its own.

• Kokosinskii, Denis & Arefyev, Nikolay (2024). Multilingual Substitution-based Word Sense Induction. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 11859–11872. Full text in Research Archive
• Rønningstad, Egil; Klinger, Roman; Øvrelid, Lilja & Velldal, Erik (2024). Entity-Level Sentiment: More than the Sum of Its Parts, The 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis Proceedings of the Workshop. Association for Computational Linguistics (ACL). ISSN 9798891761568. p. 84–96. Full text in Research Archive Show summary

  In sentiment analysis of longer texts, there may be a variety of topics discussed, of entities mentioned, and of sentiments expressed regarding each entity. We find a lack of studies exploring how such texts express their sentiment towards each entity of interest, and how these sentiments can be modelled. In order to better understand how sentiment regarding persons and organizations (each entity in our scope) is expressed in longer texts, we have collected a dataset of expert annotations where the overall sentiment regarding each entity is identified, together with the sentence-level sentiment for these entities separately. We show that the reader's perceived sentiment regarding an entity often differs from an arithmetic aggregation of sentiments at the sentence level. Only 70% of the positive and 55% of the negative entities receive a correct overall sentiment label when we aggregate the (human-annotated) sentiment labels for the sentences where the entity is mentioned. Our dataset reveals the complexity of entity-specific sentiment in longer texts, and allows for more precise modelling and evaluation of such sentiment expressions.

• Schlechtweg, Dominik; Zamora-Reina, Frank D.; Bravo-Marquez, Felipe & Arefyev, Nikolay (2024). Sense through time: diachronic word sense annotations for word sense induction and Lexical Semantic Change Detection. Language Resources and Evaluation. ISSN 1574-020X. doi: 10.1007/s10579-024-09771-7. Full text in Research Archive Show summary

  There has been extensive work on human word sense annotation, i.e., manually labeling word uses in natural texts according to their senses. Such labels were primarily created for the tasks of Word Sense Disambiguation (WSD) and Word Sense Induction (WSI). However, almost all datasets annotated with word senses are synchronic datasets, i.e., contain texts created in a relatively short period of time and often do not provide the creation date of the texts. This ignores possible applications in diachronic-historic settings, where the aim is to induce or disambiguate historical word senses or changes in senses across time. To facilitate investigations into historical WSD and WSI and to establish connections with the task of Lexical Semantic Change Detection (LSCD), there is a crucial need for historical word sense-annotated data. Hence, we created a new reliable diachronic WSD/WSI dataset ‘DWUG DE Sense’. We describe the preparation and annotation and analyze central statistics. We then describe a thorough evaluation of different prediction systems for jointly solving both WSI and LSCD tasks. All our systems are based on a state-of-the-art architecture that combines Word-in-Context models and graph clustering techniques with different hyperparameter settings. Our findings reveal that using the WSI task as optimization criterion yields better results for both tasks even when the LSCD task is the focal point of optimization. This underscores that although both tasks are related, WSI seems to be more general and able to incorporate the LSCD task.

• Payenda, Mohammad Arif; Vahidi, Abdul Razaq; Hussiny, Mohammad Ali; Prinz, Andreas & Øvrelid, Lilja (2024). PashtoEmo: Enhancing Text-Based Emotion Analysis in the Pashto Language Through Dataset Creation. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 14763, p. 225–236. doi: 10.1007/978-3-031-70242-6_22. Full text in Research Archive Show summary

  This paper presents the comprehensive PashtoEmo dataset for emotion analysis in the Pashto language. PashtoEmo contains 8016 text instances from X social media platform covering several topics, including politics, women’s rights, social justice, culture, sports, and education. The texts are annotated with six emotional categories and an additional ‘Other’ class. We tested PashtoEmo for emotion analysis with several machine learning, deep learning, and transformer models. Even the best model, XLM-RoBERTa-large, achieved only a 76.85% F1 score and a 78.11% accuracy. This indicates that the data set must be extended with adequate resources in Pashto language text-based applications.

View all works in NVA

• Scherrer, Yves; Aepli, Noëmi; Blaschke, Verena; Jauhiainen, Tommi; Ljubešić, Nikola & Nakov, Preslav [Show all 8 contributors for this article] (2026). Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISBN 9798891763722. 404 p. doi: 10.18653/v1/2026.vardial-1. Full text in Research Archive
• Tahmasebi, Nina; Cassotti, Pierluigi; Montariol, Syrielle; Kutuzov, Andrey; Huebscher, Netta & Spaziani, Elena [Show all 7 contributors for this article] (2026). The Proceedings for the 6th International Workshop on Computational Approaches to Language Change (LChange’26). Association for Computational Linguistics (ACL). ISBN 9798891763623. doi: https:/aclanthology.org/volumes/2026.lchange-1/. Full text in Research Archive
• Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Nakov, Preslav; Tiedemann, Jörg & Zampieri, Marcos (2025). Proceedings of the 12th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISBN 9798891762084. 223 p. Full text in Research Archive
• Tahmasebi, Nina; Montariol, Syrielle; Kutuzov, Andrei; Alfter, David; Periti, Francesco & Cassotti, Pierluigi [Show all 7 contributors for this article] (2024). Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISBN 9798891761384. 193 p. Full text in Research Archive
• Scherrer, Yves; Jauhiainen, Tommi; Ljubešić, Nikola; Zampieri, Marcos; Nakov, Preslav & Tiedemann, Jörg (2024). Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024). Association for Computational Linguistics (ACL). ISBN 9798891761049. 252 p. Full text in Research Archive

View all works in NVA

• Melteig, Elina; Glad, Ingrid Kristine; Ramezani-Kebrya, Ali; Pérez, Johan Manuel de Aguas; Holmsen, Sigurd & Palomares, Alfonso Diz-Lois [Show all 7 contributors for this article] (2026). Blir KI vår dommedag? Ekspertene strides. [Journal]. Titan. Full text in Research Archive
• Poiesz, Emil & Holmsen, Sigurd (2026). Hva skjer i kunstig intelligens? Siste nytt fra forskningsfronten. doi: https:/www.arendalsuka.no/program/hva-skjer-i-kunstig-intelligens-siste-nytt-fra-forskningsfronten. Full text in Research Archive Show summary

  Presentasjon under Arendalsuka om energibruk til KI modeller

• Tahmasebi, Nina; Kutuzov, Andrey; Dubossarsky, Haim & Giulianelli, Mario (2026). Computational Models of Semantic Change. In Ledgeway, Adam; Everaert, Martin; Riemsdijk, Henk van; Aldridge, Edith; Breitbarth, Anne; Kiss, Katalin; Salmons, Joseph & Simonenko, Alexandra (Ed.), The Wiley Blackwell Companion to Diachronic and Historical Linguistics. John Wiley & Sons. ISSN 9781119898023. doi: 10.1002/9781119898023.wbcdl140. Full text in Research Archive Show summary

  This chapter provides an overview of key considerations when using computational methods to scan digital text for evidence of semantic change. It lays out the foundations of the field, from the distributional hypothesis to the methods employed and the resources available. It also shows how the computational study of semantic change relates to and supplements the corresponding field of historical linguistics. Computational approaches to semantic change are a relatively new field of study where much remains to be done. Therefore, we end by discussing future research directions and possible extensions of the field to address changing cultural and societal phenomena encoded in text.

• Clausen, Hannah (2025). Ethical Issues in the Use of Surveys on LLMs. Full text in Research Archive
• Clausen, Hannah; Kutuzov, Andrey; Smajdor, Anna & Velldal, Erik (2025). Assessing Morality in Large Language Models - A Survey. Full text in Research Archive Show summary

  Poster Presentation of the current state of research presenting datasets, approaches and theories used to evaluate moral judgment by large language models

• Clausen, Hannah (2025). Benchmarking Ethical Reasoning in Large Language Models - Visit of the Scientific Advisory Committee. Full text in Research Archive Show summary

  Project presentation for the Scientific Advisory Committee (SAC) discussing the state of research, relevant shortcomings therein and how to address these

• Grødeland, Gunnveig; Knutsen, Carl Henrik; MOENE, KALLE & Øvrelid, Lilja (2025). Panelsamtale: 80 Years of Democracy – Is There Hope for Another 80? Full text in Research Archive
• Clausen, Hannah (2024). Benchmarking Ethical Reasoning in Large Language Models - Integreat Annual Retreat 2024. Full text in Research Archive
• Jelin, Elma; Bjertnæs, Øyvind Andresen; Norman, Rebecka Maria; Iversen, Hilde Karin Hestad & Mæhlum, Petter (2024). Mapping the Landscape: A Scoping Review on Advancing Patient Experience Feedback Through Natural Language Processing (NLP) and Machine Learning (ML), including sentiment Analysis (SA). Full text in Research Archive
• Norman, Rebecka Maria; Jelin, Elma; Mæhlum, Petter; Velldal, Erik; Øvrelid, Lilja & Bjertnæs, Øyvind Andresen (2024). From words to action: development of a machine learning model using natural language processing to automatically turn free-text patient comments into insights for quality improvement. Full text in Research Archive
• Wold, Sondre & Mæhlum, Petter (2024). Nesten 60 år etter LIX -- Lesbarhet på ordnivå. Full text in Research Archive
• Mæhlum, Petter & Scherrer, Yves (2024). NoMusic – Et parallellkorpus for skriftlig dialekt. Full text in Research Archive
• Scherrer, Yves (2024). NoMusic: Norwegian Multi-Parallel Slot and Intent Detection Corpus. Full text in Research Archive
• Øvrelid, Lilja (2024). Norwegian Large Language Models: current status and the road ahead. Full text in Research Archive
• Lison, Pierre (2024). Nå kan KI-generert tekst vannmerkes. Full text in Research Archive
• Chandler, David; Chandnani, Krisha; Goswami, Aditi; Marinelli, Ryan Anthony; Ness, Preben Monteiro & O’Donovan, Brian [Show all 20 contributors for this article] (2026). Data Study Group Final Report: CEFAS - Automated identification of sea pens using OpenCV and machine learning. The Alan Turing Institute. doi: https:/zenodo.org/records/20447259. Full text in Research Archive
• Adib, Shady; Bezalwar, Rashmi; Bolanos, Francisco; Brau, Wendy; Coghi, Francesco & Forshaw, Matthew [Show all 23 contributors for this article] (2026). Data Study Group Final Report: The Alan Turing Institute and Innovate UK BridgeAI - Understanding skill supply–demand gaps and recruitment trends for AI roles in the UK labour market. The Alan Turing Institute. doi: 10.5281/ZENODO.20607530. Full text in Research Archive Show summary

  Data Study Groups are week-long events at The Alan Turing Institute bringing together some of the country’s top talent from data science, artificial intelligence, and wider fields, to analyse real-world data science challenges. This report was produced as part of The Alan Turing Institute Data Study Group in collaboration with the Innovate UK BridgeAI programme [6]. The Innovate UK BridgeAI programme is an Innovate UK initiative that aims to accelerate the adoption of artificial intelligence (AI) across key UK sectors by supporting skills development, workforce readiness, and responsible AI deployment. It addresses a central challenge in AI skills policy: how to connect structured competency frameworks and occupational standards with fast-changing labour market evidence from job advertisements and macro-level labour market indicators. This analysis focuses on three related questions: how AI-related roles and skills can be identified and aligned across Innovate UK BridgeAI, Skills England, and labour market data; how AI-related hiring demand relates to broader labour market conditions and whether vacancy data improve macroeconomic monitoring and forecasting; and how job-posting evidence can support occupational progression analysis and prototype AI job classification.

• Samuel, David; Kutuzov, Andrey; Øvrelid, Lilja & Velldal, Erik (2026). Data-Constrained Language Modeling. Universitetet i Oslo. Full text in Research Archive Show summary

  Language model performance scales with training data, yet the reservoir of human-written text is finite– a constraint that already limits most languages and will eventually affect even English as models continue to scale. This thesis addresses data constraints across the entire life cycle of language models: we develop architectural improvements and training methods that maximize learning from limited and repeated data, and we propose post-training methods that enable fluency-preserving preference optimization without native instruction-tuning datasets.

View all works in NVA