Nikolay Arefev

• Oepen, Stephan; Arefyev, Nikolay; Aulamo, Mikko; Bañón, Marta; Buljan, Maja & Burchell, Laurie V. [Show all 29 contributors for this article] (2026). HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 1409–1434. doi: 10.63317/25xbdofco9od. Full text in Research Archive Show summary

  We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for some 20 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder–decoder models, as well as about 30 “smallish” monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

• Fedorova, Mariia; Arefyev, Nikolay; Buljan, Maja; Helcl, Jindřich; Oepen, Stephan & Rønningstad, Egil [Show all 7 contributors for this article] (2026). OpenLID-v3: Improving the Precision of Closely Related Language Identification – An Experience Report. In Scherrer, Yves; Aepli, Noëmi; Blaschke, Verena; Jauhiainen, Tommi; Ljubešić, Nikola; Nakov, Preslav; Tiedemann, Jörg & Zampieri, Marcos (Ed.), Proceedings of the 13th Workshop on NLP for Similar Languages, Varieties and Dialects. Association for Computational Linguistics (ACL). ISSN 9798891763722. p. 275–292. doi: 10.18653/v1/2026.vardial-1.23. Full text in Research Archive Show summary

  Language identification (LID) is an essential step in building high-quality multilingual datasets from web data. Existing LID tools (such as OpenLID or GlotLID) often struggle to identify closely related languages and to distinguish valid natural language from noise, which contaminates language-specific subsets, especially for low-resource languages. In this work we extend the OpenLID classifier by adding more training data, merging problematic language variant clusters, and introducing a special label for marking noise. We call this extended system OpenLID-v3 and evaluate it against GlotLID on multiple benchmarks. During the development we focus on three groups of closely related languages (Bosnian, Croatian, and Serbian; Romance varieties of Northern Italy and Southern France; and Scandinavian languages) and contribute new evaluation datasets where existing ones are inadequate. We find that ensemble approaches improve precision but also substantially reduce coverage for low-resource languages.

• Bian, Jie; Arefyev, Nikolay; Mühlhäuser, Max & Welzl, Michael (2025). Automated Insights Into GitHub Collaboration Dynamics. IEEE Access. 13, p. 85526–85542. doi: 10.1109/access.2025.3566309. Full text in Research Archive Show summary

  Today, GitHub (Trademark) is the most widely used platform for open-source software development. Large projects may comprise hundreds of distributed collaborators and thousands of GitHub “issues” (structured discussions). It includes basic support for dealing with issues, via “Pull Requests” (PRs)—document changes that can manually be defined to “close” them (i.e., they address and thereby conclude the issue discussion). Unresolved issues can pile up. For example, at the time of writing, the Kubernetes repository has almost 2000 open issues; finding which ones a PR might close is a hard task by itself. We address this by automatically identifying issue-PR relationships using language models (LMs) and leveraging Information Retrieval (IR) techniques. To foster further research, we contribute a carefully curated novel dataset called CodeConvo, reflecting the most influential open-source repositories for code development as well as some technical document repositories. We use this dataset to benchmark several state-of-the-art (SOTA) non-proprietary models that show exceptional performance on the MTEB benchmark, as well as to train and evaluate the performance of Smart Insights into GitHub Issue-PR Relations (SIGIR), our tailored model. The best SIGIR model/data combination yields an average Mean Reciprocal Rank (MRR) above 0.7, around 20% higher than the best baseline performance. Notably, ablation studies revealed that knowledge transfer occurs not only between different programming languages but also between code and technical documents, albeit to a lesser extent. We believe these results are encouraging and can stimulate the practical application of LMs for taming the complexity of very large projects, in GitHub and beyond.

• Simon, Étienne; Olsen, Helene Bøsei; Villar, Ramón Carreño; Mishra, Rahul; Arefyev, Nikolay & Yilmaz, Mert Can [Show all 8 contributors for this article] (2025). Abstractive Event Analysis of Armed Conflicts: Introducing the UCDP-AEC Dataset. In Wartena, Christian & Heid, Ulrich (Ed.), Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops. HsH Applied Academics. ISSN 9783690180177. p. 104–119. doi: https:/aclanthology.org/2025.konvens-2.8.pdf. Full text in Research Archive Show summary

  This paper introduces a new dataset of document-level event annotations in the domain of armed conflict. By augmenting the event database from the Uppsala Conflict Data Program (UCDP) with source documents identified in public web archives, we create the UCDP Abstractive Event analysis Corpus (UCDP-AEC). While a large part of research on information extraction is focused on extracting text spans, realworld use cases often require inferring more high-level information that is not necessarily explicitly mentioned in texts. UCDP-AEC differs from traditional event extraction datasets in that the document-level annotations do not correspond to mere text spans of the input, but capture expert-interpreted and often implicit information. With more than 10 000 documents, UCDP-AEC is of comparable size to the largest human-annotated traditional event extraction datasets. We also report preliminary experimental results for various generative approaches, by fine-tuning both decoder models and existing event argument extraction models that require minimal adaptation to our abstractive formulation of the task

• Burchell, Laurie; Gibert, Ona de; Arefev, Nikolay; Aulamo, Mikko; Bañón, Marta & Chen, Pinzhen [Show all 35 contributors for this article] (2025). An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT). In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 17452–17485. doi: 10.18653/v1/2025.acl-long.854. Full text in Research Archive Show summary

  Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value.

• Fedorova, Mariia; Kutuzov, Andrei; Arefev, Nikolay & Schlechtweg, Dominik (2024). Enriching Word Usage Graphs with Cluster Definitions. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 6189–6198. Full text in Research Archive Show summary

  We present a dataset of word usage graphs (WUGs), where the existing WUGs for multiple languages are enriched with cluster labels functioning as sense definitions. They are generated from scratch by fine-tuned encoder-decoder language models. The conducted human evaluation has shown that these definitions match the existing clusters in WUGs better than the definitions chosen from WordNet by two baseline systems. At the same time, the method is straightforward to use and easy to extend to new languages. The resulting enriched datasets can be extremely helpful for moving on to explainable semantic change modeling.

• Gibert, Ona de; Nail, Graeme; Arefev, Nikolay; Bañón, Marta; Linde, Jelmer van der & Ji, Shaoxiong [Show all 13 contributors for this article] (2024). A New Massive Multilingual Dataset for High-Performance Language Technologies. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 1116–1128. Full text in Research Archive Show summary

  We present the HPLT (High Performance Language Technologies) language resources, a new massive multilingual dataset including both monolingual and bilingual corpora extracted from CommonCrawl and previously unused web crawls from the Internet Archive. We describe our methods for data acquisition, management and processing of large corpora, which rely on open-source software tools and high-performance computing. Our monolingual collection focuses on low- to medium-resourced languages and covers 75 languages and a total of ≈ 5.6 trillion word tokens de-duplicated on the document level. Our English-centric parallel corpus is derived from its monolingual counterpart and covers 18 language pairs and more than 96 million aligned sentence pairs with roughly 1.4 billion English tokens. The HPLT language resources are one of the largest open text corpora ever released, providing a great resource for language modeling and machine translation training. We publicly release the corpora, the software, and the tools used in this work.

• Bian, Jie; Welzl, Michael; Kutuzov, Andrei & Arefyev, Nikolay (2024). Tell Me Why: Language Models Help Explain the Rationale Behind Internet Protocol Design. In LI, Geoffrey Ye; Liang, Le; Gündüz, Deniz & Antón-Haro, Carles (Ed.), 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798350343199. p. 447–453. doi: 10.1109/ICMLCN59089.2024.10624781. Full text in Research Archive Show summary

  Request for Comments (RFCs) serve as guidebooks for the implementation of Internet protocols or network mechanisms. They reveal how these protocols and mechanisms work, but the underlying reasons for their operation are not always available in RFCs. We present an attempt to automate the discovery of these reasons in mailing list archives via natural language processing methods. Our approach leverages the known relationship between text changes and the discussions leading to these changes in the Working Group (WG) GitHub repositories of the Internet Engineering Task Force (IETF) to obtain labeled training data. We find that our model is able to generalize, and it can indeed discover emails that have led to formulations in RFCs. This is a first step towards facilitating a deeper understanding of these often complex documents, which can be helpful for developers, protocol designers, and educators alike.

• Kokosinskii, Denis; Kuklin, Mikhail & Arefyev, Nikolay (2024). Deep-change at AXOLOTL-24: Orchestrating WSD and WSI Models for Semantic Change Modeling, Proceedings of the 5th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9798891761384. p. 168–179. doi: 10.18653/v1/2024.lchange-1.16. Full text in Research Archive Show summary

  This paper describes our solution of the first subtask from the AXOLOTL-24 shared task on Semantic Change Modeling. The goal of this subtask is to distribute a given set of usages of a polysemous word from a newer time period between senses of this word from an older time period and clusters representing gained senses of this word. We propose and experiment with three new methods solving this task. Our methods achieve SOTA results according to both official metrics of the first substask. Additionally, we develop a model that can tell if a given word usage is not described by any of the provided sense definitions. This model serves as a component in one of our methods, but can potentially be useful on its own.

• Kokosinskii, Denis & Arefyev, Nikolay (2024). Multilingual Substitution-based Word Sense Induction. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 11859–11872. Full text in Research Archive
• Schlechtweg, Dominik; Zamora-Reina, Frank D.; Bravo-Marquez, Felipe & Arefyev, Nikolay (2024). Sense through time: diachronic word sense annotations for word sense induction and Lexical Semantic Change Detection. Language Resources and Evaluation. ISSN 1574-020X. doi: 10.1007/s10579-024-09771-7. Full text in Research Archive Show summary

  There has been extensive work on human word sense annotation, i.e., manually labeling word uses in natural texts according to their senses. Such labels were primarily created for the tasks of Word Sense Disambiguation (WSD) and Word Sense Induction (WSI). However, almost all datasets annotated with word senses are synchronic datasets, i.e., contain texts created in a relatively short period of time and often do not provide the creation date of the texts. This ignores possible applications in diachronic-historic settings, where the aim is to induce or disambiguate historical word senses or changes in senses across time. To facilitate investigations into historical WSD and WSI and to establish connections with the task of Lexical Semantic Change Detection (LSCD), there is a crucial need for historical word sense-annotated data. Hence, we created a new reliable diachronic WSD/WSI dataset ‘DWUG DE Sense’. We describe the preparation and annotation and analyze central statistics. We then describe a thorough evaluation of different prediction systems for jointly solving both WSI and LSCD tasks. All our systems are based on a state-of-the-art architecture that combines Word-in-Context models and graph clustering techniques with different hyperparameter settings. Our findings reveal that using the WSI task as optimization criterion yields better results for both tasks even when the LSCD task is the focal point of optimization. This underscores that although both tasks are related, WSI seems to be more general and able to incorporate the LSCD task.

• Anwar, Saba; Shelmanov, Artem; Arefyev, Nikolay; Panchenko, Alexander & Biemann, Chris (2023). Text augmentation for semantic frame induction and parsing. Language Resources and Evaluation. ISSN 1574-020X. doi: 10.1007/s10579-023-09679-8. Full text in Research Archive Show summary

  Abstract Semantic frames are formal structures describing situations, actions or events, e.g., Commerce buy , Kidnapping , or Exchange . Each frame provides a set of frame elements or semantic roles corresponding to participants of the situation and lexical units (LUs)—words and phrases that can evoke this particular frame in texts. For example, for the frame Kidnapping , two key roles are Perpetrator and the Victim , and this frame can be evoked with lexical units abduct, kidnap , or snatcher . While formally sound, the scarce availability of semantic frame resources and their limited lexical coverage hinders the wider adoption of frame semantics across languages and domains. To tackle this problem, firstly, we propose a method that takes as input a few frame-annotated sentences and generates alternative lexical realizations of lexical units and semantic roles matching the original frame definition. Secondly, we show that the obtained synthetically generated semantic frame annotated examples help to improve the quality of frame-semantic parsing. To evaluate our proposed approach, we decompose our work into two parts. In the first part of text augmentation for LUs and roles, we experiment with various types of models such as distributional thesauri, non-contextualized word embeddings (word2vec, fastText, GloVe), and Transformer-based contextualized models, such as BERT or XLNet. We perform the intrinsic evaluation of these induced lexical substitutes using FrameNet gold annotations. Models based on Transformers show overall superior performance, however, they do not always outperform simpler models (based on static embeddings) unless information about the target word is suitably injected. However, we observe that non-contextualized models also show comparable performance on the task of LU expansion. We also show that combining substitutes of individual models can significantly improve the quality of final substitutes. Because intrinsic evaluation scores are highly dependent on the gold dataset and the frame preservation, and cannot be ensured by an automatic evaluation mechanism because of the incompleteness of gold datasets, we also carried out experiments with manual evaluation on sample datasets to further analyze the usefulness of our approach. The results show that the manual evaluation framework significantly outperforms automatic evaluation for lexical substitution. For extrinsic evaluation, the second part of this work assesses the utility of these lexical substitutes for the improvement of frame-semantic parsing. We took a small set of frame-annotated sentences and augmented them by replacing corresponding target words with their closest substitutes, obtained from best-performing models. Our extensive experiments on the original and augmented set of annotations with two semantic parsers show that our method is effective for improving the downstream parsing task by training set augmentation, as well as for quickly building FrameNet-like resources for new languages or subject domains.

• Rachinskiy, Maxim & Arefyev, Nikolay (2022). GlossReader at LSCDiscovery: Train to Select a Proper Gloss in English – Discover Lexical Semantic Change in Spanish, Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781955917421. p. 198–203. doi: 10.18653/v1/2022.lchange-1.22. Full text in Research Archive
• Homskiy, Daniil & Arefyev, Nikolay (2022). DeepMistake at LSCDiscovery: Can a Multilingual Word-in-Context Model Replace Human Annotators? Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781955917421. p. 173–179. doi: 10.18653/v1/2022.lchange-1.18. Full text in Research Archive
• Kudisov, Artem & Arefyev, Nikolay (2022). BOS at LSCDiscovery: Lexical Substitution for Interpretable Lexical Semantic Change Detection, Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781955917421. p. 165–172. doi: 10.18653/v1/2022.lchange-1.17. Full text in Research Archive
• Bingyu, Zhang & Arefyev, Nikolay (2022). The Document Vectors Using Cosine Similarity Revisited. In Tafreshi, Shabnam; Sedoc, João; Rogers, Anna; Drozd, Aleksandr; Rumshisky, Anna & Akula, Arjun (Ed.), Proceedings of the Third Workshop on Insights from Negative Results in NLP. Association for Computational Linguistics (ACL). ISSN 9781955917407. p. 129–133. doi: 10.18653/v1/2022.insights-1.17. Full text in Research Archive
• Rachinskiy, Maxim & Arefyev, Nikolay (2021). GlossReader at SemEval-2021 Task 2: Reading Definitions Improves Contextualized Word Embeddings. In Palmer, Alexis; Schneider, Nathan; Schluter, Natalie; Emerson, Guy; Herbelot, Aurelie & Zhu, Xiaodan (Ed.), Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). Association for Computational Linguistics (ACL). ISSN 9781954085701. p. 756–762. doi: 10.18653/v1/2021.semeval-1.100. Full text in Research Archive
• Razzhigaev, Anton; Arefyev, Nikolay & Panchenko, Alexander (2021). SkoltechNLP at SemEval-2021 Task 2: Generating Cross-Lingual Training Data for the Word-in-Context Task. In Palmer, Alexis; Schneider, Nathan; Schluter, Natalie; Emerson, Guy; Herbelot, Aurelie & Zhu, Xiaodan (Ed.), Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). Association for Computational Linguistics (ACL). ISSN 9781954085701. p. 157–162. doi: 10.18653/v1/2021.semeval-1.16. Full text in Research Archive
• Arefyev, Nikolay; Kharchev, Dmitrii & Shelmanov, Artem (2021). NB-MLM: Efficient Domain Adaptation of Masked Language Models for Sentiment Analysis, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781955917094. p. 9114–9124. doi: 10.18653/v1/2021.emnlp-main.717. Full text in Research Archive
• Davletov, Adis; Gordeev, Denis; Arefyev, Nikolay & Davletov, Emil (2021). LIORI at SemEval-2021 Task 8: Ask Transformer for measurements. In Palmer, Alexis; Schneider, Nathan; Schluter, Natalie; Emerson, Guy; Herbelot, Aurelie & Zhu, Xiaodan (Ed.), Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). Association for Computational Linguistics (ACL). ISSN 9781954085701. p. 1249–1254. doi: 10.18653/v1/2021.semeval-1.178. Full text in Research Archive
• Davletov, Adis; Arefyev, Nikolay; Gordeev, Denis & Rey, Alexey (2021). LIORI at SemEval-2021 Task 2: Span Prediction and Binary Classification approaches to Word-in-Context Disambiguation. In Palmer, Alexis; Schneider, Nathan; Schluter, Natalie; Emerson, Guy; Herbelot, Aurelie & Zhu, Xiaodan (Ed.), Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021). Association for Computational Linguistics (ACL). ISSN 9781954085701. p. 780–786. doi: 10.18653/v1/2021.semeval-1.103. Full text in Research Archive
• Arefyev, Nikolay; Sheludko, Boris; Podolskiy, Alexander & Panchenko, Alexander (2020). Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution, Proceedings of the 28th International Conference on Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781952148279. p. 1242–1255. doi: 10.18653/v1/2020.coling-main.107. Full text in Research Archive

View all works in NVA

• Vyacheslav, Stroev; Dmitry, Grechka; Maria, Eliseeva & Arefyev, Nikolay (2022). Kashtanka.pet: employing AI to help lost pets return to their homes. Full text in Research Archive Show summary

  Аннотация доклада: The Kashtanka.pet project addresses the problem of searching for lost pets efficiently. There are numerous platforms and groups in social networks collecting ads about missing and found cats and dogs. However, it is often almost impossible for a human to find a specific lost pet among millions of ads about found pets distributed across all those websites. We develop a system that allows pet owners and volunteers to find lost pets efficiently with the help of AI. It crawls websites for ads about lost and found pets, and retrieves the pairs of ads announcing the same pet was lost and then found. The retrieved pairs are then inspected and further processed by humans. On the poster we present the architecture of the Kashtanka.pet system. Then we address the problem of evaluating and improving the quality of underlying AI models of lost pets retrieval. The standard manual annotation of a dataset for our task requires finding the matching pairs of lost and found ads, which makes the annotation process prohibitively difficult. Thus, we generate the matching pairs automatically by splitting sets of photos from the ads containing several photos into two parts. However, the simple random splitting results in both parts sharing some photos made in the same place. This may promote models searching for the same background rather than the same pet, and makes the setup unrealistically simple because in reality a pet is lost and found in different places. To mitigate this we propose a method that finds those ads that contain photos of a pet taken in several places and splits them accordingly. In order to estimate the quality of this method and select its hyperparameters, we additionally annotated pairs of photos from random ads asking annotators if those photos were taken in the same or different places. Several methods solving the task are proposed and compared. The pipeline currently deployed at kashtanka.pet is based on YOLOv4 for pet detection and cropping, EfficientNet for obtaining their embeddings, and GRU for aggregation of those embeddings across several images from the same ad. The model is trained with the triplet loss on the target dataset. Two other methods employ the pre-trained multimodal BLIP and SLIP models, which are the recently introduced improvements over the popular CLIP model. We found that even without fine-tuning on the data from our target domain the image embeddings from the multimodal models significantly outperform the currently deployed pipeline.

View all works in NVA