Andrey Kutuzov

• Samuel, David; Øvrelid, Lilja; Velldal, Erik & Kutuzov, Andrey (2026). Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages. In Vondrick, C.; Hariharan, B.; Raffel, C.; Pinto, L.; Yang, D. & Faust, A. (Ed.), International Conference on Learning Representations (ICLR) 2026. International Conference on Learning Representations (ICLR). ISSN 9798331339678. p. 94000–94027. doi: https:/proceedings.iclr.cc/paper_files/paper/2026/hash/97a057ab1235218027dc07c50eedee0d-Abstract-Conference.html. Full text in Research Archive Show summary

  We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

• Clausen, Hannah; Kutuzov, Andrey; Smajdor, Anna & Velldal, Erik (2026). Assessing Moral Judgment by Large Language Models – A Survey of Available Datasets. Northern European Journal of Language Technology (NEJLT). ISSN 2000-1533. 12(1), p. 88–116. doi: 10.3384/nejlt.2000-1533.2026.6366. Full text in Research Archive Show summary

  Recent advances in language modeling have contributed to a growing emphasis on machine ethics, including researching and assessing moral judgments made by large language models (LLMs). This paper provides a critical survey of the existing datasets for exactly this assessment, with a special focus on the respective data sources. We address the current lack of theoretical grounding by providing an introduction to ethics and different frameworks from moral philosophy and moral psychology. Moreover, we identify four main data sources: webcrawled corpora, scholars, laypeople, and synthetic data generation. By discussing the strengths and weaknesses of these sources, we analyze their implications for the assessment of moral judgment. Importantly, systemizing the available datasets reveals an over-reliance on previous work, reinforcing existing shortcomings. Addressing the current limitations, we recommend adopting a consistent terminology and creating independently curated datasets based on interdisciplinary work. To ensure a clear delineation of normative approaches, we propose focusing on the assessment of moral consistency and certainty of LLMs as effective and well-defined indicators of their performance on moral judgment.

• Oepen, Stephan; Arefyev, Nikolay; Aulamo, Mikko; Bañón, Marta; Buljan, Maja & Burchell, Laurie V. [Show all 29 contributors for this article] (2026). HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 1409–1434. doi: 10.63317/25xbdofco9od. Full text in Research Archive Show summary

  We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for some 20 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder–decoder models, as well as about 30 “smallish” monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

• Fedorova, Mariia; Kutuzov, Andrey & Umarova, Khonzoda (2026). DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling. In Tahmasebi, Nina; Cassotti, Pierluigi; Montariol, Syrielle; Kutuzov, Andrey; Huebscher, Netta; Spaziani, Elena & Baes, Naomi (Ed.), The Proceedings for the 6th International Workshop on Computational Approaches to Language Change (LChange’26). Association for Computational Linguistics (ACL). ISSN 9798891763623. p. 87–96. doi: 10.18653/v1/2026.lchange-1.7. Full text in Research Archive Show summary

  In this resource paper, we present DHPLT, an open collection of diachronic corpora in 41 diverse languages. DHPLT is based on the web-crawled HPLT datasets; we use web crawl timestamps as the approximate signal of document creation time. The collection covers three time periods: 2011-2015, 2020-2021 and 2024-present (1 million documents per time period for each language). We additionally provide pre-computed word type and token embeddings and lexical substitutions for our chosen target words, while at the same time leaving it open for the other researchers to come up with their own target words using the same datasets. DHPLT aims at filling in the current lack of multilingual diachronic corpora for semantic change modelling (beyond a dozen of high-resource languages). It opens the way for a variety of new experimental setups in this field.

• Samuel, David; Mikhailov, Vladislav; Velldal, Erik; Øvrelid, Lilja; Charpentier, Lucas Georges Gabriel & Kutuzov, Andrey [Show all 7 contributors for this article] (2025). Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 573–608. doi: https:/aclanthology.org/2025.nodalida-1.61/. Full text in Research Archive Show summary

  Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern S{\'a}mi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokm{\r{a}}l, Nynorsk, and Northern S{\'a}mi with 11.4 billion parameters: NorMistral-11B.

• Fedorova, Mariia; Kutuzov, Andrey; Periti, Francesco & Scherrer, Yves (2025). Explaining novel senses using definition generation with open language models. In Christodoulopoulos, Christos; Chakraborty, Tanmoy; Rose, Carolyn & Peng, Violet (Ed.), Findings of the Association for Computational Linguistics: EMNLP 2025. Association for Computational Linguistics (ACL). ISSN 9798891763357. p. 22294–22302. doi: 10.18653/v1/2025.findings-emnlp.1214. Full text in Research Archive Show summary

  We apply definition generators based on open-weights large language models to the task of creating explanations of novel senses, taking target word usages as an input. To this end, we employ the datasets from the AXOLOTL’24 shared task on explainable semantic change modeling, which features Finnish, Russian and German languages. We fine-tune and provide publicly the open-source models performing higher than the best submissions of the aforementioned shared task, which employed closed proprietary LLMs. In addition, we find that encoder-decoder definition generators perform on par with their decoder-only counterparts.

• Mikhailov, Vladislav; Enstad, Tita Ranveig; Samuel, David; Farsethås, Hans Christian; Velldal, Erik & Øvrelid, Lilja [Show all 7 contributors for this article] (2025). NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). ISSN 9798891762565. p. 3495–3541. doi: 10.18653/v1/2025.findings-acl.181. Full text in Research Archive Show summary

  This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets – of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-created prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pretrained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

• Burchell, Laurie; Gibert, Ona de; Arefev, Nikolay; Aulamo, Mikko; Bañón, Marta & Chen, Pinzhen [Show all 35 contributors for this article] (2025). An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT). In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 17452–17485. doi: 10.18653/v1/2025.acl-long.854. Full text in Research Archive Show summary

  Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

• Rosa, Javier De La; Mikhailov, Vladislav; Zhang, Lemei; Wetjen, Freddy; Samuel, David & Liu, Peng [Show all 19 contributors for this article] (2025). The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 544–560. Full text in Research Archive
• Chen, Pinzhen; Ji, Shaoxiong; Bogoychev, Nikolay; Kutuzov, Andrei; Haddow, Barry & Heafield, Kenneth (2024). Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca, Findings of the Association for Computational Linguistics: EACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760936. p. 1347–1356. Full text in Research Archive
• Fedorova, Mariia; Mickus, Timothee; Partanen, Niko; Siewert, Janine; Spaziani, Elena & Kutuzov, Andrei (2024). AXOLOTL’24 Shared Task on Multilingual Explainable Semantic Change Modeling, Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9798891761384. p. 72–91. Full text in Research Archive Show summary

  This paper describes the organization and findings of AXOLOTL’24, the first multilingual explainable semantic change modeling shared task. We present new sense-annotated diachronic semantic change datasets for Finnish and Russian which were employed in the shared task, along with a surprise test-only German dataset borrowed from an existing source. The setup of AXOLOTL’24 is new to the semantic change modeling field, and involves subtasks of identifying unknown (novel) senses and providing dictionary-like definitions to these senses. The methods of the winning teams are described and compared, thus paving a path towards explainability in computational approaches to historical change of meaning.

• Fedorova, Mariia; Kutuzov, Andrei & Scherrer, Yves (2024). Definition generation for lexical semantic change detection, Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760998. p. 5712–5724. Full text in Research Archive Show summary

  We use contextualized word definitions generated by large language models as semantic representations in the task of diachronic lexical semantic change detection (LSCD). In short, generated definitions are used as ‘senses’, and the change score of a target word is retrieved by comparing their distributions in two time periods under comparison. On the material of five datasets and three languages, we show that generated definitions are indeed specific and general enough to convey a signal sufficient to rank sets of words by the degree of their semantic change over time. Our approach is on par with or outperforms prior non-supervised sense-based LSCD methods. At the same time, it preserves interpretability and allows to inspect the reasons behind a specific shift in terms of discrete definitions-as-senses. This is another step in the direction of explainable semantic change modeling.

• Fedorova, Mariia; Kutuzov, Andrei; Arefev, Nikolay & Schlechtweg, Dominik (2024). Enriching Word Usage Graphs with Cluster Definitions. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 6189–6198. Full text in Research Archive Show summary

  We present a dataset of word usage graphs (WUGs), where the existing WUGs for multiple languages are enriched with cluster labels functioning as sense definitions. They are generated from scratch by fine-tuned encoder-decoder language models. The conducted human evaluation has shown that these definitions match the existing clusters in WUGs better than the definitions chosen from WordNet by two baseline systems. At the same time, the method is straightforward to use and easy to extend to new languages. The resulting enriched datasets can be extremely helpful for moving on to explainable semantic change modeling.

• Gibert, Ona de; Nail, Graeme; Arefev, Nikolay; Bañón, Marta; Linde, Jelmer van der & Ji, Shaoxiong [Show all 13 contributors for this article] (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 1116–1128. Full text in Research Archive Show summary

  We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

• Bian, Jie; Welzl, Michael; Kutuzov, Andrei & Arefyev, Nikolay (2024). Tell Me Why: Language Models Help Explain the Rationale Behind Internet Protocol Design. In LI, Geoffrey Ye; Liang, Le; Gündüz, Deniz & Antón-Haro, Carles (Ed.), 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798350343199. p. 447–453. doi: 10.1109/ICMLCN59089.2024.10624781. Full text in Research Archive Show summary

  Request for Comments (RFCs) serve as guidebooks for the implementation of Internet protocols or network mechanisms. They reveal how these protocols and mechanisms work, but the underlying reasons for their operation are not always available in RFCs. We present an attempt to automate the discovery of these reasons in mailing list archives via natural language processing methods. Our approach leverages the known relationship between text changes and the discussions leading to these changes in the Working Group (WG) GitHub repositories of the Internet Engineering Task Force (IETF) to obtain labeled training data. We find that our model is able to generalize, and it can indeed discover emails that have led to formulations in RFCs. This is a first step towards facilitating a deeper understanding of these often complex documents, which can be helpful for developers, protocol designers, and educators alike.

• Giulianelli, Mario; Luden, Iris; Fernandez, Raquel & Kutuzov, Andrei (2023). Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9781959429722. p. 3130–3148. Full text in Research Archive Show summary

  We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations.Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label. We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users — historical linguists, lexicographers, or social scientists — to explore and intuitively explain diachronic trajectories of word meaning. Semantic change analysis is only one of many possible applications of the ‘definitions as representations’ paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.

• Samuel, David; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2023). Trained on 100 million words and still in shape: BERT meets British National Corpus. In Vlachos, Andreas & Augenstein, Isabelle (Ed.), Findings of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics (ACL). ISSN 9781959429470. p. 1954–1974. doi: 10.18653/v1/2023.findings-eacl.146. Full text in Research Archive Show summary

  While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source – the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.

• Samuel, David; Kutuzov, Andrei; Touileb, Samia; Velldal, Erik; Øvrelid, Lilja & Rønningstad, Egil [Show all 8 contributors for this article] (2023). NorBench – A Benchmark for Norwegian Language Models. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. p. 618–633. Full text in Research Archive Show summary

  We present NorBench: a streamlined suite of NLP tasks and probes for evaluating Norwegian language models (LMs) on standardized data splits and evaluation metrics. We also introduce a range of new Norwegian language models (both encoder and encoder-decoder based). Finally, we compare and analyze their performance, along with other existing LMs, across the different benchmark tests of NorBench.

• Giulianelli, Mario; Kutuzov, Andrei & Pivovarova, Lidia (2022). Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change, Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781955917421. p. 54–67. Full text in Research Archive Show summary

  Morphological and syntactic changes in word usage — as captured, e.g., by grammatical profiles — have been shown to be good predictors of a word’s meaning change. In this work, we explore whether large pre-trained contextualised language models, a common tool for lexical semantic change detection, are sensitive to such morphosyntactic changes. To this end, we first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages, and then combine the two approaches in ensembles to assess their complementarity. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages. This indicates that language models do not fully cover the fine-grained morphological and syntactic signals that are explicitly represented in grammatical profiles. An interesting exception are the test sets where the time spans under analysis are much longer than the time gap between them (for example, century-long spans with a one-year gap between them). Morphosyntactic change is slow so grammatical profiles do not detect in such cases. In contrast, language models, thanks to their access to lexical information, are able to detect fast topical changes.

• Kutuzov, Andrei; Velldal, Erik & Øvrelid, Lilja (2022). Contextualized embeddings for semantic change detection: Lessons learned. Northern European Journal of Language Technology (NEJLT). ISSN 2000-1533. 8(1). doi: 10.3384/nejlt.2000-1533.2022.3478. Full text in Research Archive Show summary

  We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.

• Barnes, Jeremy; Oberlaender, Laura; Troiano, Enrica; Kutuzov, Andrei; Buchmann, Jan & Agerri, Rodrigo [Show all 8 contributors for this article] (2022). SemEval 2022 Task 10: Structured Sentiment Analysis. In Emerson, Guy; Schluter, Natalie; Stanovsky, Gabriel; Kumar, Ritesh; Palmer, Alexis; Schneider, Nathan; Singh, Siddarth & Ratan, Shyam (Ed.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics (ACL). ISSN 9781955917803. p. 1280–1295. Full text in Research Archive
• Aksenova, Anna; Gavrishina, Ekaterina; Rykov, Elisei & Kutuzov, Andrei (2022). RuDSI: Graph-based Word Sense Induction Dataset for Russian, Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781955917223. p. 77–88. Full text in Research Archive Show summary

  We present RuDSI, a new benchmark for word sense induction (WSI) in Russian. The dataset was created using manual annotation and semi-automatic clustering of Word Usage Graphs (WUGs). RuDSI is completely data-driven (based on texts from Russian National Corpus), with no external word senses imposed on annotators. We present and analyze RuDSI, describe our annotation workflow, show how graph clustering parameters affect the dataset, report the performance that several baseline WSI methods obtain on RuDSI and discuss possibilities for improving these scores.

• Kutuzov, Andrei; Touileb, Samia; Mæhlum, Petter; Enstad, Tita & Witteman, Alexandra (2022). NorDiaChange: Diachronic Semantic Change Dataset for Norwegian, Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546726. p. 2563–2572. Full text in Research Archive Show summary

  We describe NorDiaChange: the first diachronic semantic change dataset for Norwegian. NorDiaChange comprises two novel subsets, covering about 80 Norwegian nouns manually annotated with graded semantic change over time. Both datasets follow the same annotation procedure and can be used interchangeably as train and test splits for each other. NorDiaChange covers the time periods related to pre- and post-war events, oil and gas discovery in Norway, and technological developments. The annotation was done using the DURel framework and two large historical Norwegian corpora. NorDiaChange is published in full under a permissive licence, complete with raw annotation data and inferred diachronic word usage graphs (DWUGs).

• Rodina, Julia; Trofimova, Yuliya; Kutuzov, Andrei & Artemova, Ekaterina (2021). ELMo and BERT in Semantic Change Detection for Russian, Proceedings of AIST 2020: Analysis of Images, Social Networks and Texts. Springer. ISSN 9783030726102. p. 175–186. Full text in Research Archive Show summary

  We study the effectiveness of contextualized embeddings for the task of diachronic semantic change detection for Russian language data. Evaluation test sets consist of Russian nouns and adjectives annotated based on their occurrences in texts created in pre-Soviet, Soviet and post-Soviet time periods. ELMo and BERT architectures are compared on the task of ranking Russian words according to the degree of their semantic change over time. We use several methods for aggregation of contextualized embeddings from these architectures and evaluate their performance. Finally, we compare unsupervised and supervised techniques in this task.

• Kutuzov, Andrei; Barnes, Jeremy; Velldal, Erik; Øvrelid, Lilja & Oepen, Stephan (2021). Large-Scale Contextualised Language Modelling for Norwegian. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 30–40. Full text in Research Archive Show summary

  We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first large-scale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian. For additional background and access to the data, models, and software, please see: http://norlm.nlpl.eu

• Kutuzov, Andrei; Giulianelli, Mario & Pivovarova, Lidia (2021). Grammatical Profiling for Semantic Change Detection, Proceedings of the 25th Conference on Computational Natural Language Learning. Association for Computational Linguistics (ACL). ISSN 9781955917056. p. 423–434. Full text in Research Archive Show summary

  Semantics, morphology and syntax are strongly interdependent. However, the majority of computational methods for semantic change detection use distributional word representations which encode mostly semantics. We investigate an alternative method, grammatical profiling, based entirely on changes in the morphosyntactic behaviour of words. We demonstrate that it can be used for semantic change detection and even outperforms some distributional semantic methods. We present an in-depth qualitative and quantitative analysis of the predictions made by our grammatical profiling system, showing that they are plausible and interpretable.

• Iazykova, Tatyana; Kapelyushnik, Denis; Bystrova, Olga & Kutuzov, Andrei (2021). Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 20, p. 302–318. doi: 10.28995/2075-7182-2021-20-302-317. Full text in Research Archive Show summary

  Leaderboards like SuperGLUE are seen as important incentives for active development of NLP, since they provide standard benchmarks for fair comparison of modern language models. They have driven the world's best engineering teams as well as their resources to collaborate and solve a set of tasks for general language understanding. Their performance scores are often claimed to be close to or even higher than the human performance. These results encouraged more thorough analysis of whether the benchmark datasets featured any statistical cues that machine learning based language models can exploit. For English datasets, it was shown that they often contain annotation artifacts. This allows solving certain tasks with very simple rules and achieving competitive rankings. In this paper, a similar analysis was done for the Russian SuperGLUE (RSG), a recently published benchmark set and leaderboard for Russian natural language understanding. We show that its test datasets are vulnerable to shallow heuristics. Often approaches based on simple rules outperform or come close to the results of the notorious pre-trained language models like GPT-3 or BERT. It is likely (as the simplest explanation) that a significant part of the SOTA models performance in the RSG leaderboard is due to exploit these shallow heuristics and that has nothing in common with real language understanding. We provide a set of recommendations on how to improve these datasets, making the RSG leaderboard even more representative of the real progress in Russian NLU.

• Kutuzov, Andrei & Pivovarova, Lidia (2021). Three-part diachronic semantic change dataset for Russian, Proceedings of the 2nd International Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781954085602. p. 7–13. Full text in Research Archive Show summary

  We present a manually annotated lexical semantic change dataset for Russian: RuShiftEval. Its novelty is ensured by a single set of target words annotated for their diachronic semantic shifts across three time periods, while the previous work either used only two time periods, or different sets of target words. The paper describes the composition and annotation procedure for the dataset. In addition, it is shown how the ternary nature of RuShiftEval allows to trace specific diachronic trajectories: ‘changed at a particular time period and stable afterwards’ or ‘was changing throughout all time periods’. Based on the analysis of the submissions to the recent shared task on semantic change detection for Russian, we argue that correctly identifying such trajectories can be an interesting sub-task itself.

• Ravishankar, Vinit; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2021). Multilingual ELMo and the Effects of Corpus Sampling. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 378–383. Full text in Research Archive
• Kutuzov, Andrei & Kuzmenko, Elizaveta (2021). Representing ELMo embeddings as two-dimensional text online, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics (ACL). ISSN 9781954085053. p. 143–148. Full text in Research Archive Show summary

  We describe a new addition to the WebVectors toolkit which is used to serve word embedding models over the Web. The new ELMoViz module adds support for contextualized embedding architectures, in particular for ELMo models. The provided visualizations follow the metaphor of ‘two-dimensional text’ by showing lexical substitutes: words which are most semantically similar in context to the words of the input sentence. The system allows the user to change the ELMo layers from which token embeddings are inferred. It also conveys corpus information about the query words and their lexical substitutes (namely their frequency tiers and parts of speech). The module is well integrated into the rest of the WebVectors toolkit, providing lexical hyperlinks to word representations in static embedding models. Two web services have already implemented the new functionality with pre-trained ELMo models for Russian, Norwegian and English.

• Kutuzov, Andrei; Fomin, Vadim; Rodina, Julia & Mikhailov, Vladislav (2020). ShiftRy: web service for diachronic analysis of Russian news. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 19, p. 485–501. Full text in Research Archive Show summary

  We present the ShiftRy web service. It helps to analyze temporal changes in the usage of words in news texts from Russian mass media. For that, we employ diachronic word embedding models trained on large Russian news corpora from 2010 up to 2019. The users can explore the usage history of any given query word, or browse the lists of words ranked by the degree of their semantic drift in any couple of years. Visualizations of the words’ trajectories through time are provided. Importantly, users can obtain corpus examples with the query word before and after the semantic shift (if any). The aim of ShiftRy is to ease the task of studying word history on short-term time spans, and the influence of social and political events on word usage. The service will be updated with new data yearly.

• Kunilovskaya, Maria; Kutuzov, Andrei & Plum, Alistair (2020). Taxonomy enrichment for Russian: synset classification outperforms linear hyponym-hypernym projections. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 19, p. 459–469. doi: 10.28995/2075-7182-2020-19-474-484. Full text in Research Archive
• Rodina, Julia & Kutuzov, Andrei (2020). RuSemShift: a dataset of historical lexical semantic change in Russian, Proceedings of the 28th International Conference on Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781952148279. p. 1037–1047. Full text in Research Archive Show summary

  We present RuSemShift, a large-scale manually annotated test set for the task of semantic change modeling in Russian for two long-term time period pairs: from the pre-Soviet through the Soviet times and from the Soviet through the post-Soviet times. Target words were annotated by multiple crowd-source workers. The annotation process was organized following the DURel framework and was based on sentence contexts extracted from the Russian National Corpus. Additionally, we report the performance of several distributional approaches on RuSemShift, achieving promising results, which at the same time leave room for other researchers to improve.

• Kutuzov, Andrei & Giulianelli, Mario (2020). UiO-UvA at SemEval-2020 Task 1: Contextualised Embeddings for Lexical Semantic Change Detection, Proceedings of the Fourteenth Workshop on Semantic Evaluation. Association for Computational Linguistics (ACL). ISSN 9781952148316. p. 126–134. Full text in Research Archive Show summary

  We apply contextualised word embeddings to lexical semantic change detection in the SemEval-2020 Shared Task 1. This paper focuses on Subtask 2, ranking words by the degree of their semantic drift over time. We analyse the performance of two contextualising architectures (BERT and ELMo) and three change detection algorithms. We find that the most effective algorithms rely on the cosine similarity between averaged token embeddings and the pairwise distances between token embeddings. They outperform strong baselines by a large margin (in the post-evaluation phase, we have the best Subtask 2 submission for SemEval-2020 Task 1), but interestingly, the choice of a particular algorithm depends on the distribution of gold scores in the test set.

• Katricheva, Nadezda; Yaskevich, Alyaxey; Lisitsina, Anastasiya; Zhordaniya, Tamara; Kutuzov, Andrei & Kuzmenko, Elizaveta (2020). Vec2graph: A Python Library for Visualizing Word Embeddings as Graphs. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 1086, p. 190–198. doi: 10.1007/978-3-030-39575-9_20. Full text in Research Archive
• Kutuzov, Andrei; Fomin, V.; Mikhailov, V. Nikola & Rodina, Julia (2020). Shiftry: Web service for diachronic analysis of Russian news. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 2020-(19), p. 500–516. doi: 10.28995/2075-7182-2020-19-500-516. Full text in Research Archive
• Logacheva, Varvara; Teslenko, Denis; Shelmanov, Artem; Remus, S.; Ustalov, Dmitry & Kutuzov, Andrei [Show all 10 contributors for this article] (2020). Word Sense Disambiguation for 158 Languages using Word Embeddings Only. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of The 12th Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546344. p. 5945–5954. Full text in Research Archive Show summary

  Disambiguation of word senses in context is easy for humans, but is a major challenge for automatic approaches. Sophisticated supervised and knowledge-based models were developed to solve this task. However, (i) the inherent Zipfian distribution of supervised training instances for a given word and/or (ii) the quality of linguistic knowledge representations motivate the development of completely unsupervised and knowledge-free approaches to word sense disambiguation (WSD). They are particularly useful for under-resourced languages which do not have any resources for building either supervised and/or knowledge-based models. In this paper, we present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory, which can be used for disambiguation in context. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings by Grave et al. (2018), enabling WSD in these languages. Models and system are available online.

• Kutuzov, Andrei; Dorgham, Mohammad; Oliynyk, Oleksiy; Biemann, Chris & Panchenko, Alexander (2019). Making Fast Graph-based Algorithms with Graph Metric Embeddings. In Korhonen, Anna; Traum, David & Màrquez, Lluís (Ed.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781950737482. p. 3349–3355. doi: 10.18653/v1/P19-1325. Full text in Research Archive Show summary

  Graph measures, such as node distances, are inefficient to compute. We explore dense vector representations as an effective way to approximate the same information. We introduce a simple yet efficient and effective approach for learning graph embeddings. Instead of directly operating on the graph structure, our method takes structural measures of pairwise node similarities into account and learns dense node representations reflecting user-defined graph distance measures, such as e.g. the shortest path distance or distance measures that take information beyond the graph structure into account. We demonstrate a speed-up of several orders of magnitude when predicting word similarity by vector operations on our embeddings as opposed to directly computing the respective path-based measures, while outperforming various other graph embeddings on semantic similarity and word sense disambiguation tasks.

• Kutuzov, Andrei; Velldal, Erik & Øvrelid, Lilja (2019). One-to-X Analogical Reasoning on Word Embeddings: a Case for Diachronic Armed Conflict Prediction from News Texts. In Tahmasebi, Nina; Borin, Lars; Jatowt, Adam & Xu, Yang (Ed.), Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781950737314. p. 196–201. doi: 10.18653/v1/W19-4724. Full text in Research Archive Show summary

  We extend the well-known word analogy task to a one-to-X formulation, including one-to-none cases, when no correct answer exists. The task is cast as a relation discovery problem and applied to historical armed conflicts datasets, attempting to predict new relations of type ‘location:armed-group’ based on data about past events. As the source of semantic information, we use diachronic word embedding models trained on English news texts. A simple technique to improve diachronic performance in such task is demonstrated, using a threshold based on a function of cosine distance to decrease the number of false positives; this approach is shown to be beneficial on two different corpora. Finally, we publish a ready-to-use test set for one-to-X analogy evaluation on historical armed conflicts data.

• Rodina, Julia; Bakshandaeva, Daria; Fomin, Vadim; Kutuzov, Andrei; Touileb, Samia & Velldal, Erik (2019). Measuring Diachronic Evolution of Evaluative Adjectives with Word Embeddings: the Case for English, Norwegian, and Russian. In Tahmasebi, Nina; Borin, Lars; Jatowt, Adam & Xu, Yang (Ed.), Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781950737314. p. 202–209. doi: 10.18653/v1/W19-4725. Full text in Research Archive Show summary

  We measure the intensity of diachronic semantic shifts in adjectives in English, Norwegian and Russian across 5 decades. This is done in order to test the hypothesis that evaluative adjectives are more prone to temporal semantic change. To this end, 6 different methods of quantifying semantic change are used. Frequency-controlled experimental results show that, depending on the particular method, evaluative adjectives either do not differ from other types of adjectives in terms of semantic change or appear to actually be less prone to shifting (particularly, to ‘jitter’-type shifting). Thus, in spite of many well-known examples of semantically changing evaluative adjectives (like ‘terrific’ or ‘incredible’), it seems that such cases are not specific to this particular type of words.

• Kutuzov, Andrei & Nikishina, Irina (2019). Double-Blind Peer-Reviewing and Inclusiveness in Russian NLP Conferences, Analysis of Images, Social Networks and Texts (Revised Selected Papers). Springer Publishing Company. ISSN 9783030373344. p. 3–8. doi: 10.1007/978-3-030-37334-4_1. Full text in Research Archive

View all works in NVA

• Tahmasebi, Nina; Cassotti, Pierluigi; Montariol, Syrielle; Kutuzov, Andrey; Huebscher, Netta & Spaziani, Elena [Show all 7 contributors for this article] (2026). The Proceedings for the 6th International Workshop on Computational Approaches to Language Change (LChange’26). Association for Computational Linguistics (ACL). ISBN 9798891763623. doi: https:/aclanthology.org/volumes/2026.lchange-1/. Full text in Research Archive
• Tahmasebi, Nina; Montariol, Syrielle; Kutuzov, Andrei; Alfter, David; Periti, Francesco & Cassotti, Pierluigi [Show all 7 contributors for this article] (2024). Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISBN 9798891761384. 193 p. Full text in Research Archive
• Tahmasebi, Nina; Montariol, Syrielle; Dubossarsky, Haim; Kutuzov, Andrei; Hengchen, Simon & Alfter, David [Show all 8 contributors for this article] (2023). Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISBN 9798891760431. 174 p. Full text in Research Archive
• Tahmasebi, Nina; Montariol, Syrielle; Kutuzov, Andrei; Hengchen, Simon; Dubossarsky, Haim & Borin, Lars (2022). Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISBN 9781955917421. 219 p. Full text in Research Archive
• Kutuzov, Andrei; Aalst, Wil M. P. van der; Batagelj, Vladimir & Ignatov, Dmitry (2021). Proceedings of AIST 2020: Analysis of Images, Social Networks and Texts. Springer. ISBN 9783030726102. 480 p. Full text in Research Archive

View all works in NVA

• Tahmasebi, Nina; Kutuzov, Andrey; Dubossarsky, Haim & Giulianelli, Mario (2026). Computational Models of Semantic Change. In Ledgeway, Adam; Everaert, Martin; Riemsdijk, Henk van; Aldridge, Edith; Breitbarth, Anne; Kiss, Katalin; Salmons, Joseph & Simonenko, Alexandra (Ed.), The Wiley Blackwell Companion to Diachronic and Historical Linguistics. John Wiley & Sons. ISSN 9781119898023. doi: 10.1002/9781119898023.wbcdl140. Full text in Research Archive Show summary

  This chapter provides an overview of key considerations when using computational methods to scan digital text for evidence of semantic change. It lays out the foundations of the field, from the distributional hypothesis to the methods employed and the resources available. It also shows how the computational study of semantic change relates to and supplements the corresponding field of historical linguistics. Computational approaches to semantic change are a relatively new field of study where much remains to be done. Therefore, we end by discussing future research directions and possible extensions of the field to address changing cultural and societal phenomena encoded in text.

• Clausen, Hannah; Kutuzov, Andrey; Smajdor, Anna & Velldal, Erik (2025). Assessing Morality in Large Language Models - A Survey. Full text in Research Archive Show summary

  Poster Presentation of the current state of research presenting datasets, approaches and theories used to evaluate moral judgment by large language models

• Aalst, Wil M. P. van der; Batagelj, Vladimir; Buzmakov, Alexey; Ignatov, Dmitry I.; Kalenkova, Anna & Khachay, Michael [Show all 18 contributors for this article] (2021). Preface. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 1357, p. vii–x. Full text in Research Archive
• Samuel, David; Kutuzov, Andrey; Øvrelid, Lilja & Velldal, Erik (2026). Data-Constrained Language Modeling. Universitetet i Oslo. Full text in Research Archive Show summary

  Language model performance scales with training data, yet the reservoir of human-written text is finite– a constraint that already limits most languages and will eventually affect even English as models continue to scale. This thesis addresses data constraints across the entire life cycle of language models: we develop architectural improvements and training methods that maximize learning from limited and repeated data, and we propose post-training methods that enable fluency-preserving preference optimization without native instruction-tuning datasets.

View all works in NVA