Our latest publications - LTG

• Chen, Pinzhen; Ji, Shaoxiong; Bogoychev, Nikolay; Kutuzov, Andrei; Haddow, Barry & Heafield, Kenneth (2024). Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca, Findings of the Association for Computational Linguistics: EACL 2024. Association for Computational Linguistics. ISSN 979-8-89176-093-6. p. 1347–1356.
• Rønningstad, Egil; Velldal, Erik & Øvrelid, Lilja (2024). A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation. In Henning, Sophie & Stede, Manfred (Ed.), Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII). Association for Computational Linguistics. ISSN 979-8-89176-073-8. p. 133–139. Full text in Research Archive
• Buljan, Maja (2023). What quantifying word order freedom can tell us about dependency corpora. In Schneider, Nathan (Eds.), The Seventh International Conference on Dependency Linguistics (Depling, GURT/SyntaxFest 2023) -- Proceedings of the Conference. Association for Computational Linguistics. ISSN 978-1-959429-32-6. p. 54–67. Full text in Research Archive
• Olsen, Helene Bøsei; Touileb, Samia & Velldal, Erik (2023). Arabic dialect identification: An in-depth error analysis on the MADAR parallel corpus. In Sawaf, Hassan; El-Beltagy, Samhaa; Zaghouani, Wajdi; Magdy, Walid; Abdelali, Ahmed; Tomeh, Nadi; Abu Farha, Ibrahim; Habash, Nizar; Khalifa, Salam; Keleg, Amr; Haddad, Hatem; Zitouni, Imed; Mrini, Khalil & Almatham, Rawan (Ed.), Proceedings of ArabicNLP 2023. Association for Computational Linguistics. ISSN 978-1-959429-27-2. p. 370–384. doi: 10.18653/v1/2023.arabicnlp-1.30. Full text in Research Archive
• Tiedemann, Jörg; Aulamo, Mikko; Bakshandaeva, Daria; Boggia, Michele; Grönroos, Stig-Arne & Nieminen, Tommi [Show all 10 contributors for this article] (2023). Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation. ISSN 1574-020X. p. 67–79. doi: 10.1007/s10579-023-09704-w. Full text in Research Archive
• Mæhlum, Petter & Ivanova, Sardana (2023). Phonotactics as an Aid in Low Resource Loan Word Detection and Morphological Analysis in Sakha. In Ilinykh, Nikolai; Morger, Felix; Dannélls, Dana; Dobnik, Simon; Megyesi, Beáta & Nivre, Joakim (Ed.), Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). Association for Computational Linguistics. ISSN 978-1-959429-73-9. p. 111–120. Show summary

  Obtaining information about loan words and irregular morphological patterns can be difficult for low-resource languages. Using Sakha as an example, we show that it is possible to exploit known phonemic regularities such as vowel harmony and consonant distributions to identify loan words and irregular patterns, which can be helpful in rule-based downstream tasks such as parsing and POS-tagging. We evaluate phonemically inspired methods for loanword detection, combined with bi-gram vowel transition probabilities to inspect irregularities in the morphology of loanwords. We show that both these techniques can be useful for the detection of such patterns. Finally, we inspect the plural suffix -ЛАр [-LAr] to observe some of the variation in morphology between native and foreign words.

• Charpentier, Lucas Georges Gabriel & Samuel, David (2023). Not all layers are equally as important: Every Layer Counts BERT. In Warstadt, Alex; Mueller, Aaron; Choshen, Leshem; Wilcox, Ethan; Zhuang, Chengxu; Ciro, Juan; Mosquera, Rafael; Paranjabe, Bhargavi; Williams, Adina; Linzen, Tal & Cotterell, Ryan (Ed.), Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. Association for Computational Linguistics. ISSN 978-1-952148-02-6. p. 238–252. doi: 10.18653/v1/2023.conll-babylm.20.
• Samuel, David (2023). Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings. In Warstadt, Alex; Mueller, Aaron; Choshen, Leshem; Wilcox, Ethan; Zhuang, Chengxu; Ciro, Juan; Mosquera, Rafael; Paranjabe, Bhargavi; Williams, Adina; Linzen, Tal & Cotterell, Ryan (Ed.), Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning. Association for Computational Linguistics. ISSN 978-1-952148-02-6. p. 221–237. doi: 10.18653/v1/2023.conll-babylm.19.
• Kuparinen, Olli; Miletic, Aleksandra & Scherrer, Yves (2023). Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation. In Bouamor, Houda; Pino, Juan Miguel Rey & Bali, Kalika (Ed.), Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics. ISSN 979-8-89176-061-5. p. 13814–13828. Full text in Research Archive Show summary

  Text normalization methods have been commonly applied to historical language or user-generated content, but less often to dialectal transcriptions. In this paper, we introduce dialect-to-standard normalization – i.e., mapping phonetic transcriptions from different dialects to the orthographic norm of the standard variety – as a distinct sentence-level character transduction task and provide a large-scale analysis of dialect-to-standard normalization methods. To this end, we compile a multilingual dataset covering four languages: Finnish, Norwegian, Swiss German and Slovene. For the two biggest corpora, we provide three different data splits corresponding to different use cases for automatic normalization. We evaluate the most successful sequence-to-sequence model architectures proposed for text normalization tasks using different tokenization approaches and context sizes. We find that a character-level Transformer trained on sliding windows of three words works best for Finnish, Swiss German and Slovene, whereas the pre-trained byT5 model using full sentences obtains the best results for Norwegian. Finally, we perform an error analysis to evaluate the effect of different data splits on model performance.

• Siewert, Janine; Wieling, Martijn & Scherrer, Yves (2023). Changing usage of Low Saxon auxiliary and modal verbs. In Tahmasebi, Nina; Montariol, Syrielle; Dubossarsky, Haim; Kutuzov, Andrei; Hengchen, Simon; Alfter, David; Periti, Francesco & Cassotti, Pierluigi (Ed.), Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics. ISSN 979-8-89176-043-1. p. 112–118. Full text in Research Archive Show summary

  We investigate the usage of auxiliary and modal verbs in Low Saxon dialects from both Germany and the Netherlands based on word vectors, and compare developments in the modern language to Middle Low Saxon. Although most of these function words have not been affected by lexical replacement, changes in usage that likely at least partly result from contact with the state languages can still be observed.

• Scherrer, Yves; Miletić, Aleksandra & Kuparinen, Olli (2023). The Helsinki-NLP Submissions at NADI 2023 Shared Task: Walking the Baseline. In Sawaf, Hassan; El-Beltagy, Samhaa; Zaghouani, Wajdi; Magdy, Walid; Abdelali, Ahmed; Tomeh, Nadi; Abu Farha, Ibrahim; Habash, Nizar; Khalifa, Salam; Keleg, Amr; Haddad, Hatem; Zitouni, Imed; Mrini, Khalil & Almatham, Rawan (Ed.), Proceedings of ArabicNLP 2023. Association for Computational Linguistics. ISSN 978-1-959429-27-2. p. 670–677. Full text in Research Archive Show summary

  The Helsinki-NLP team participated in the NADI 2023 shared tasks on Arabic dialect translation with seven submissions. We used statistical (SMT) and neural machine translation (NMT) methods and explored character- and subword-based data preprocessing. Our submissions placed second in both tracks. In the open track, our winning submission is a character-level SMT system with additional Modern Standard Arabic language models. In the closed track, our best BLEU scores were obtained with the leave-as-is baseline, a simple copy of the input, and narrowly followed by SMT systems. In both tracks, fine-tuning existing multilingual models such as AraT5 or ByT5 did not yield superior performance compared to SMT.

• Rønningstad, Egil (2023). UIO at SemEval-2023 Task 12: Multilingual fine-tuning for sentiment classification in low-resource languages. Association for Computational Linguistics (ACL). Annual Meeting Conference Proceedings. ISSN 0736-587X. Full text in Research Archive
• Anwar, Saba; Shelmanov, Artem; Arefyev, Nikolay; Panchenko, Alexander & Biemann, Chris (2023). Text augmentation for semantic frame induction and parsing. Language Resources and Evaluation. ISSN 1574-020X. doi: 10.1007/s10579-023-09679-8. Full text in Research Archive
• Hussiny, Mohammad Ali; Payenda, Mohammad Arif & Vahidi, Abdul Razaq (2023). SentiDariPers: Sentiment Analysis of Dari-Persian Tweets Based on People’s Views and Opinion. In Valencia-Garcia, Rafael; Bucaram-Leverone, Martha; Del Cioppo-Morstadt, Javier; Vera-Lucio, Nestor & Centanaro-Quiroz, Pablo Humberto (Ed.), International Conference on Technologies and Innovation (CITI 2023). Springer. ISSN 978-3-031-45681-7. p. 138–156. doi: 10.1007/978-3-031-45682-4_11. Show summary

  In the research area of sentiment analysis, there is a noticeable gap when it comes to the Dari-Persian dialect. To bridge this gap, our research aimed to curate a comprehensive dataset encompassing people’s opinions in this specific language variant. This paper presents the development of a benchmark sentiment annotated dataset for the Dari dialect of Persian, which serves as an official language of Afghanistan. The dataset, named “SentiDariPers”, comprises 43,089 tweets posted between August 2021 and April 2023. It has been manually annotated with four sentiment classes: Negative, Positive, Neutral, and Mixed. We applied a range of models, such as Support Vector Machine (SVM), Long Short-Term Memory (LSTM), Bi-directional Long Short-Term Memory (Bi-LSTM), Gated Recurrent Unit (GRU), and Convolutional Neural Network (CNN). Additionally, we develop an ensemble model that combines different sets of sentiment classes for each system. We present a detailed comparative analysis of the results obtained from these models. Experimental findings demonstrate that the ensemble model achieves the highest accuracy 91%. We provide insights into the data collection and annotation process, offer relevant dataset statistics, discuss the experimental results, and provide further analysis of the data.

• Samuel, David & Øvrelid, Lilja (2023). Tokenization with Factorized Subword Encoding, Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics. ISSN 978-1-959429-62-3.
• You, Huiling; Touileb, Samia & Øvrelid, Lilja (2023). JSEEGraph: Joint Structured Event Extraction as Graph Parsing. In Palmer, Alexis & Camacho-Collados, Jose (Ed.), Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023). Association for Computational Linguistics. ISSN 978-1-959429-76-0. Full text in Research Archive
• Hussiny, Mohammad Ali & Øvrelid, Lilja (2023). Emotion Analysis of Tweets Banning Education in Afghanistan. In Barnes, Jeremy; De Clercq, Orphee & Klinger, Roman (Ed.), The 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. Association for Computational Linguistics. ISSN 978-1-959429-87-6. Full text in Research Archive
• Wold, Sondre; Øvrelid, Lilja & Velldal, Erik (2023). Text-To-KG Alignment: Comparing Current Methods on Classification Tasks, Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023). Association for Computational Linguistics. ISSN 978-1-959429-82-1. p. 1–14. doi: https:/aclanthology.org/2023.matching-1.1/.
• Giulianelli, Mario; Luden, Iris; Fernandez, Raquel & Kutuzov, Andrei (2023). Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics. ISSN 978-1-959429-72-2. p. 3130–3148. Full text in Research Archive Show summary

  We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations.Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialised Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label. We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users — historical linguists, lexicographers, or social scientists — to explore and intuitively explain diachronic trajectories of word meaning. Semantic change analysis is only one of many possible applications of the ‘definitions as representations’ paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.

• Barnes, Jeremy Claude; Touileb, Samia; Mæhlum, Petter & Lison, Pierre (2023). Identifying Token-Level Dialectal Features in Social Media. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 978-99-1621-999-7. Full text in Research Archive Show summary

  Dialectal variation is present in many human languages and is attracting a growing interest in NLP. Most previous work concentrated on either (1) classifying dialectal varieties at the document or sentence level or (2) performing standard NLP tasks on dialectal data. In this paper, we propose the novel task of token-level dialectal feature prediction. We present a set of fine-grained annotation guidelines for Norwegian dialects, expand a corpus of dialectal tweets, and manually annotate them using the introduced guidelines. Furthermore, to evaluate the learnability of our task, we conduct labeling experiments using a collection of baselines, weakly supervised and supervised sequence labeling models. The obtained results show that, despite the difficulty of the task and the scarcity of training data, many dialectal features can be predicted with reasonably high accuracy.

• Mæhlum, Petter; Velldal, Erik & Øvrelid, Lilja (2023). A Diagnostic Dataset for Sentiment and Negation Modeling for Norwegian. In Ilinykh, Nikolai; Morger, Felix; Dannélls, Dana; Dobnik, Simon; Megyesi, Beáta & Nivre, Joakim (Ed.), Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). Association for Computational Linguistics. ISSN 978-1-959429-73-9. Full text in Research Archive
• Kolesnichenko, Larisa; Velldal, Erik & Øvrelid, Lilja (2023). Word Substitution with Masked Language Models as Data Augmentation for Sentiment Analysis. In Ilinykh, Nikolai; Morger, Felix; Dannélls, Dana; Dobnik, Simon; Megyesi, Beáta & Nivre, Joakim (Ed.), Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). Association for Computational Linguistics. ISSN 978-1-959429-73-9. Full text in Research Archive
• Samuel, David; Kutuzov, Andrei; Touileb, Samia; Velldal, Erik; Øvrelid, Lilja & Rønningstad, Egil [Show all 8 contributors for this article] (2023). NorBench – A Benchmark for Norwegian Language Models. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 978-99-1621-999-7. p. 618–633. Full text in Research Archive
• Olstad, Annika Willoch; Papadopoulou, Anthi & Lison, Pierre (2023). Generation of Replacement Options in Text Sanitization. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 978-99-1621-999-7. p. 292–300. Show summary

  The purpose of text sanitization is to edit text documents to mask text spans that may directly or indirectly reveal personal information. An important problem in text sanitization is to find less specific, yet still informative replacements for each text span to mask. We present an approach to generate possible replacements using a combination of heuristic rules and an ontology derived from Wikidata. Those replacement options are hierarchically structured and cover various types of personal identifiers. Using this approach, we extend a recently released text sanitization dataset with manually selected replacements. The outcome of this data collection shows that the approach is able to suggest appropriate replacement options for most text spans.

• Ivanova, Sardana; Andreassen, Fredrik Aas; Jentoft, Matias; Wold, Sondre & Øvrelid, Lilja (2023). NorQuAD: Norwegian Question Answering Dataset. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 978-99-1621-999-7.
• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2023). Measuring normative and descriptive biases in language models using census data. In Vlachos, Andreas & Augenstein, Isabelle (Ed.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics. ISSN 978-1-959429-44-9. Full text in Research Archive
• Charpentier, Lucas Georges Gabriel; Wold, Sondre; Samuel, David & Rønningstad, Egil (2023). BRENT: Bidirectional Retrieval Enhanced Norwegian Transformer. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 978-99-1621-999-7.
• Samuel, David; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2023). Trained on 100 million words and still in shape: BERT meets British National Corpus. In Vlachos, Andreas & Augenstein, Isabelle (Ed.), Findings of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics. ISSN 978-1-959429-47-0. p. 1954–1974. Show summary

  While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source – the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.

• Mæhlum, Petter; Haug, Dag Trygve Truslew; Jørgensen, Tollef Emil; Kåsen, Andre; Nøklestad, Anders & Rønningstad, Egil [Show all 9 contributors for this article] (2022). NARC – Norwegian Anaphora Resolution Corpus. International Conference on Computational Linguistics (ICCL) (COLING). ISSN 1525-2477. 29(7), p. 48–60. Full text in Research Archive Show summary

  Published in: Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC): https://aclanthology.org/venues/coling/. We present the Norwegian Anaphora Resolution Corpus (NARC), the first publicly available corpus annotated with anaphoric relations between noun phrases for Norwegian. The paper describes the annotated data for 326 documents in Norwegian Bokmål, together with inter-annotator agreement and discussions of relevant statistics. We also present preliminary modelling results which are comparable to existing corpora for other languages, and discuss relevant problems in relation to both modelling and the annotations themselves.

• You, Huiling; Samuel, David; Touileb, Samia & Øvrelid, Lilja (2022). EventGraph at CASE 2021 Task 1: A General Graph-based Approach to Protest Event Extraction, The 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Association for Computational Linguistics. ISSN 978-1-959429-05-0. p. 155–160.
• You, Huiling; Samuel, David; Touileb, Samia & Øvrelid, Lilja (2022). EventGraph: Event Extraction as Semantic Graph Parsing, The 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Association for Computational Linguistics. ISSN 978-1-959429-05-0. p. 7–15.
• Pilán, Ildikó; Lison, Pierre; Øvrelid, Lilja; Papadopoulou, Anthi; Sánchez, David & Batet, Montserrat (2022). The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics. ISSN 0891-2017. 48(4), p. 1053–1101. doi: 10.1162/coli_a_00458. Show summary

  We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared with previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored toward measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts, and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymization-benchmark.

• Papadopoulou, Anthi; Yu, Yunhao; Lison, Pierre & Øvrelid, Lilja (2022). Neural Text Sanitization with Explicit Measures of Privacy Risk, The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-955917-65-0. p. 217–229. Show summary

  We present a novel approach for text sanitization, which is the task of editing a document to mask all (direct and indirect) personal identifiers and thereby conceal the identity of the individuals(s) mentioned in the text. In contrast to previous work, the approach relies on explicit measures of privacy risk, making it possible to explicitly control the trade-off between privacy protection and data utility. The approach proceeds in three steps. A neural, privacy-enhanced entity recognizer is first employed to detect and classify potential personal identifiers. We then determine which entities, or combination of entities, are likely to pose a re-identification risk through a range of privacy risk assessment measures. We present three such measures of privacy risk, respectively based on (1) span probabilities derived from a BERT language model, (2) web search queries and (3) a classifier trained on labelled data. Finally, a linear optimization solver decides which entities to mask to minimize the semantic loss while simultaneously ensuring that the estimated privacy risk remains under a given threshold. We evaluate the approach both in the absence and presence of manually annotated data. Our results highlight the potential of the approach, as well as issues specific types of personal data can introduce to the process.

• Rønningstad, Egil; Øvrelid, Lilja & Velldal, Erik (2022). Entity-Level Sentiment Analysis (ELSA): An Exploratory Task Survey. International Conference on Computational Linguistics (ICCL) (COLING). ISSN 1525-2477. p. 6773–6783. Full text in Research Archive Show summary

  This paper explores the task of identifying the overall sentiment expressed towards volitional entities (persons and organizations) in a document - what we refer to as Entity-Level Sentiment Analysis (ELSA). While identifying sentiment conveyed towards an entity is well researched for shorter texts like tweets, we find little to no research on this specific task for longer texts with multiple mentions and opinions towards the same entity. This lack of research would be understandable if ELSA can be derived from existing tasks and models. To assess this, we annotate a set of professional reviews for their overall sentiment towards each volitional entity in the text. We sample from data already annotated for document-level, sentence-level, and target-level sentiment in a multi-domain review corpus, and our results indicate that there is no single proxy task that provides this overall sentiment we seek for the entities at a satisfactory level of performance. We present a suite of experiments aiming to assess the contribution towards ELSA provided by document-, sentence-, and target-level sentiment analysis, and provide a discussion of their shortcomings. We show that sentiment in our dataset is expressed not only with an entity mention as target, but also towards targets with a sentiment-relevant relation to a volitional entity. In our data, these relations extend beyond anaphoric coreference resolution, and our findings call for further research of the topic. Finally, we also present a survey of previous relevant work.

• Wold, Sondre (2022). The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection, Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-955917-22-3. p. 54–59.
• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2022). Occupational Biases in Norwegian and Multilingual Language Models. In Christine, Basta; Marta R., Costa-jussà; Hila, Gonen; Christian, Hardmeier & Gabriel, Stanovsky (Ed.), Proceedings of The 4th Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-955917-68-1. p. 200–211. doi: 10.18653/v1/2022.gebnlp-1.21. Show summary

  In this paper we explore how a demographic distribution of occupations, along gender dimensions, is reflected in pre-trained language models. We give a descriptive assessment of the distribution of occupations, and investigate to what extent these are reflected in four Norwegian and two multilingual models. To this end, we introduce a set of simple bias probes, and perform five different tasks combining gendered pronouns, first names, and a set of occupations from the Norwegian statistics bureau. We show that language specific models obtain more accurate results, and are much closer to the real-world distribution of clearly gendered occupations. However, we see that none of the models have correct representations of the occupations that are demo-graphically balanced between genders. We also discuss the importance of the training data on which the models were trained on, and argue that template-based bias probes can some-times be fragile, and a simple alteration in a template can change a model’s behavior.

• Aksenova, Anna; Gavrishina, Ekaterina; Rykov, Elisei & Kutuzov, Andrei (2022). RuDSI: Graph-based Word Sense Induction Dataset for Russian, Proceedings of TextGraphs-16: Graph-based Methods for Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-955917-22-3. p. 77–88. Show summary

  We present RuDSI, a new benchmark for word sense induction (WSI) in Russian. The dataset was created using manual annotation and semi-automatic clustering of Word Usage Graphs (WUGs). RuDSI is completely data-driven (based on texts from Russian National Corpus), with no external word senses imposed on annotators. We present and analyze RuDSI, describe our annotation workflow, show how graph clustering parameters affect the dataset, report the performance that several baseline WSI methods obtain on RuDSI and discuss possibilities for improving these scores.

• Walker, Nicholas Thomas; Dahl, Torbjørn & Lison, Pierre (2022). Dialogue Management as Graph Transformations. In Stoyanchev, Svetlana; Ultes, Stephan & Li, Haizhou (Ed.), Conversational AI for Natural Human-Centric Interaction . Springer. ISSN 978-981-19-5537-2. p. 219–227. doi: 10.1007/978-981-19-5538-9_15. Show summary

  We present ongoing work on a new dialogue management framework using graphs as core representation for the current dialogue state. Dialogue management tasks such as state tracking and action selection are framed as sequences of graph transformations that repeatedly update this graph based on incoming observations. Those graph transformations are expressed using a graph query language, making it possible to specify all dialogue management operations through a unified, declarative syntax. We argue that graphs are particularly well suited to model the dialogue state of complex, open-ended domains. In contrast to traditional dialogue state representations that are limited to fixed, predefined slots, graphs can naturally express dialogue domains with rich relational structures and variable numbers of entities to track. We describe how dialogue state tracking and action selection can be modelled in such graph-centric view of dialogue management, using either handcrafted rules or data-driven models. We also briefly discuss how to account for some aspects of dialogue management such as uncertainties, incremental inputs and contextual knowledge. Finally, we describe a proof-of-concept study of this dialogue management framework in a human–robot interaction scenario.

• Kutuzov, Andrei; Velldal, Erik & Øvrelid, Lilja (2022). Contextualized embeddings for semantic change detection: Lessons learned . Northern European Journal of Language Technology (NEJLT). ISSN 2000-1533. 8(1). doi: 10.3384/nejlt.2000-1533.2022.3478. Show summary

  We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.

• Buljan, Maja; Nivre, Joakim; Oepen, Stephan & Øvrelid, Lilja (2022). A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing. Language Resources and Evaluation. ISSN 1574-020X. p. 1–28. doi: 10.1007/s10579-022-09591-7. Full text in Research Archive
• Barnes, Jeremy; Oberlaender, Laura; Troiano, Enrica; Kutuzov, Andrei; Buchmann, Jan & Agerri, Rodrigo [Show all 8 contributors for this article] (2022). SemEval 2022 Task 10: Structured Sentiment Analysis. In Emerson, Guy; Schluter, Natalie; Stanovsky, Gabriel; Kumar, Ritesh; Palmer, Alexis; Schneider, Nathan; Singh, Siddarth & Ratan, Shyam (Ed.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics. ISSN 978-1-955917-80-3. p. 1280–1295.
• Papadopoulou, Anthi; Lison, Pierre; Øvrelid, Lilja & Pilán, Ildikó (2022). Bootstrapping Text Anonymization Models with Distant Supervision. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association. ISSN 979-10-95546-72-6. p. 4477–4487. Show summary

  We propose a novel method to bootstrap text anonymization models based on distant supervision. Instead of requiring manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee k-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a RoBERTa-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise if the knowledge graph is noisy or incomplete. The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.

• Kutuzov, Andrei; Touileb, Samia; Mæhlum, Petter; Enstad, Tita & Witteman, Alexandra (2022). NorDiaChange: Diachronic Semantic Change Dataset for Norwegian, Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association. ISSN 979-10-95546-72-6. p. 2563–2572. Full text in Research Archive Show summary

  We describe NorDiaChange: the first diachronic semantic change dataset for Norwegian. NorDiaChange comprises two novel subsets, covering about 80 Norwegian nouns manually annotated with graded semantic change over time. Both datasets follow the same annotation procedure and can be used interchangeably as train and test splits for each other. NorDiaChange covers the time periods related to pre- and post-war events, oil and gas discovery in Norway, and technological developments. The annotation was done using the DURel framework and two large historical Norwegian corpora. NorDiaChange is published in full under a permissive licence, complete with raw annotation data and inferred diachronic word usage graphs (DWUGs).

• Samuel, David; Barnes, Jeremy; Kurtz, Robin; Oepen, Stephan; Øvrelid, Lilja & Velldal, Erik (2022). Direct parsing to sentiment graphs. Association for Computational Linguistics (ACL). Annual Meeting Conference Proceedings. ISSN 0736-587X. 2(60), p. 470–478. doi: 10.18653/v1/2022.acl-short.51.
• Giulianelli, Mario; Kutuzov, Andrei & Pivovarova, Lidia (2022). Do Not Fire the Linguist: Grammatical Profiles Help Language Models Detect Semantic Change, Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics. ISSN 978-1-955917-42-1. p. 54–67. Show summary

  Morphological and syntactic changes in word usage — as captured, e.g., by grammatical profiles — have been shown to be good predictors of a word’s meaning change. In this work, we explore whether large pre-trained contextualised language models, a common tool for lexical semantic change detection, are sensitive to such morphosyntactic changes. To this end, we first compare the performance of grammatical profiles against that of a multilingual neural language model (XLM-R) on 10 datasets, covering 7 languages, and then combine the two approaches in ensembles to assess their complementarity. Our results show that ensembling grammatical profiles with XLM-R improves semantic change detection performance for most datasets and languages. This indicates that language models do not fully cover the fine-grained morphological and syntactic signals that are explicitly represented in grammatical profiles. An interesting exception are the test sets where the time spans under analysis are much longer than the time gap between them (for example, century-long spans with a one-year gap between them). Morphosyntactic change is slow so grammatical profiles do not detect in such cases. In contrast, language models, thanks to their access to lexical information, are able to detect fast topical changes.

• Ravishankar, Vinit & Søgaard, Anders (2021). The Impact of Positional Encodings on Multilingual Compression, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-955917-09-4. p. 763–777. doi: 10.18653/v1/2021.emnlp-main.59.
• Ravishankar, Vinit; Kulmizev, Artur; Abdou, Mostafa; Søgaard, Anders & Nivre, Joakim (2021). Attention Can Reflect Syntactic Structure (If You Let It). In Merlo, Paola; Tiedemann, Jörg & Tsarfaty, Reut (Ed.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics. ISSN 978-1-954085-02-2. p. 3031–3045. doi: 10.18653/v1/2021.eacl-main.264.
• Trattner, Christoph; Jannach, Dietmar; Motta, Enrico; Meijer, Irene Costera; Diakopoulos, Nicholas & Elahi, Mehdi [Show all 13 contributors for this article] (2021). Responsible media technology and AI: challenges and research directions. AI and Ethics. ISSN 2730-5953. doi: 10.1007/s43681-021-00126-4. Full text in Research Archive Show summary

  The last two decades have witnessed major disruptions to the traditional media industry as a result of technological breakthroughs. New opportunities and challenges continue to arise, most recently as a result of the rapid advance and adoption of artificial intelligence technologies. On the one hand, the broad adoption of these technologies may introduce new opportunities for diversifying media offerings, fighting disinformation, and advancing data-driven journalism. On the other hand, techniques such as algorithmic content selection and user personalization can introduce risks and societal threats. The challenge of balancing these opportunities and benefits against their potential for negative impacts underscores the need for more research in responsible media technology. In this paper, we first describe the major challenges—both for societies and the media industry—that come with modern media technology. We then outline various places in the media production and dissemination chain, where research gaps exist, where better technical approaches are needed, and where technology must be designed in a way that can effectively support responsible editorial processes and principles. We argue that a comprehensive approach to research in responsible media technology, leveraging an interdisciplinary approach and a close cooperation between the media industry and academic institutions, is urgently needed.

• Laake, Signe & Øvrelid, Lilja (2021). Forskjeller mellom talemål og skriftspråk: Hva kan trebanker fortelle oss? In Hagen, Kristin; Kristoffersen, Gjert; Vangsnes, Øystein Alexander & Åfarli, Tor Anders (Ed.), Språk i arkiva. Ny forsking om eldre talemål frå LIA-prosjektet . Novus Forlag. ISSN 9788283900811. p. 65–86.
• Yeghaneh Abkenar, Mohammad; Stede, Manfred & Oepen, Stephan (2021). Neural Argumentation Mining on Essays and Microtexts with Contextualized Word Embeddings. CEUR Workshop Proceedings. ISSN 1613-0073. 2957.
• Kutuzov, Andrei; Giulianelli, Mario & Pivovarova, Lidia (2021). Grammatical Profiling for Semantic Change Detection, Proceedings of the 25th Conference on Computational Natural Language Learning. Association for Computational Linguistics. ISSN 978-1-955917-05-6. p. 423–434. Show summary

  Semantics, morphology and syntax are strongly interdependent. However, the majority of computational methods for semantic change detection use distributional word representations which encode mostly semantics. We investigate an alternative method, grammatical profiling, based entirely on changes in the morphosyntactic behaviour of words. We demonstrate that it can be used for semantic change detection and even outperforms some distributional semantic methods. We present an in-depth qualitative and quantitative analysis of the predictions made by our grammatical profiling system, showing that they are plausible and interpretable.

• Moore, Andrew & Barnes, Jeremy Claude (2021). Multi-task Learning of Negation and Speculation for Targeted Sentiment Classification, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. ISSN 978-1-954085-46-6. p. 2838–2869.
• Lison, Pierre; Barnes, Jeremy & Hubin, Aliaksandr (2021). skweak: Weak Supervision Made Easy for NLP. In Ji, Heng & Park, Jong (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. Association for Computational Linguistics. ISSN 978-1-954085-56-5. p. 337–346. doi: 10.18653/v1/2021.acl-demo.40. Full text in Research Archive Show summary

  We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak

• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2021). Using Gender- and Polarity-Informed Models to Investigate Bias. In Costa-jussà, Marta R.; Gonen, Hila; Hardmeier, Christian & Webster, Kellie (Ed.), Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics. ISSN 978-1-954085-61-9. p. 66–74. doi: 10.18653/v1/2021.gebnlp-1.8.
• Touileb, Samia & Barnes, Jeremy (2021). The interplay between language similarity and script on a novel multi-layer Algerian dialect corpus. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics. ISSN 978-1-954085-54-1. p. 3700–3712. doi: 10.18653/v1/2021.findings-acl.324. Show summary

  Recent years have seen a rise in interest for cross-lingual transfer between languages with similar typology, and between languages of various scripts.However, the interplay be-tween language similarity and difference in script on cross-lingual transfer is a less studied problem. We explore this interplay on cross-lingual transfer for two supervised tasks,namely part-of-speech tagging and sentiment analysis.We introduce a newly annotated corpus of Algerian user-generated comments comprising parallel annotations of Algerian written in Latin, Arabic, and code-switched scripts, as well as annotations for sentiment and topic categories.We perform baseline experiments by fine-tuning multi-lingual language models. We further explore the effect of script vs. language similarity in cross-lingual transfer by fine-tuning multi-lingual models on languages which are a) typologically distinct, but use the same script, b) typologically similar, but use a distinct script, or c) are typo-logically similar and use the same script. We find there is a delicate relationship between script and typology for part-of-speech, while sentiment analysis is less sensitive.

• Iazykova, Tatyana; Kapelyushnik, Denis; Bystrova, Olga & Kutuzov, Andrei (2021). Unreasonable Effectiveness of Rule-Based Heuristics in Solving Russian SuperGLUE Tasks. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 20, p. 302–318. doi: 10.28995/2075-7182-2021-20-302-317. Show summary

  Leaderboards like SuperGLUE are seen as important incentives for active development of NLP, since they provide standard benchmarks for fair comparison of modern language models. They have driven the world's best engineering teams as well as their resources to collaborate and solve a set of tasks for general language understanding. Their performance scores are often claimed to be close to or even higher than the human performance. These results encouraged more thorough analysis of whether the benchmark datasets featured any statistical cues that machine learning based language models can exploit. For English datasets, it was shown that they often contain annotation artifacts. This allows solving certain tasks with very simple rules and achieving competitive rankings. In this paper, a similar analysis was done for the Russian SuperGLUE (RSG), a recently published benchmark set and leaderboard for Russian natural language understanding. We show that its test datasets are vulnerable to shallow heuristics. Often approaches based on simple rules outperform or come close to the results of the notorious pre-trained language models like GPT-3 or BERT. It is likely (as the simplest explanation) that a significant part of the SOTA models performance in the RSG leaderboard is due to exploit these shallow heuristics and that has nothing in common with real language understanding. We provide a set of recommendations on how to improve these datasets, making the RSG leaderboard even more representative of the real progress in Russian NLU.

• Kutuzov, Andrei & Pivovarova, Lidia (2021). Three-part diachronic semantic change dataset for Russian, Proceedings of the 2nd International Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics. ISSN 978-1-954085-60-2. p. 7–13. Show summary

  We present a manually annotated lexical semantic change dataset for Russian: RuShiftEval. Its novelty is ensured by a single set of target words annotated for their diachronic semantic shifts across three time periods, while the previous work either used only two time periods, or different sets of target words. The paper describes the composition and annotation procedure for the dataset. In addition, it is shown how the ternary nature of RuShiftEval allows to trace specific diachronic trajectories: ‘changed at a particular time period and stable afterwards’ or ‘was changing throughout all time periods’. Based on the analysis of the submissions to the recent shared task on semantic change detection for Russian, we argue that correctly identifying such trajectories can be an interesting sub-task itself.

• Lison, Pierre; Pilán, Ildikó; Sánchez, David; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics. ISSN 978-1-954085-52-7. p. 4188–4203. doi: 10.18653/v1/2021.acl-long.323. Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Barnes, Jeremy; Kurtz, Robin; Oepen, Stephan; Øvrelid, Lilja & Velldal, Erik (2021). Structured Sentiment Analysis as Dependency Graph Parsing. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics. ISSN 978-1-954085-52-7. p. 3387–3402. doi: http:/dx.doi.org/10.18653/v1/2021.acl-long.263.
• Garbulowski, Mateusz; Diamanti, Klev; Smolińska, Karolina; Baltzer, Nicholas; Stoll, Patricia & Bornelöv, Susanne [Show all 9 contributors for this article] (2021). R.ROSETTA: an interpretable machine learning framework. BMC Bioinformatics. ISSN 1471-2105. 22(1), p. 1–18. doi: 10.1186/s12859-021-04049-z.
• Barnes, Jeremy; Mæhlum, Petter & Touileb, Samia (2021). NorDial: A Preliminary Corpus of Written Norwegian Dialect Use. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 978-91-7929-614-8. p. 445–451. Show summary

  Norway has a large amount of dialectal variation, as well as a general tolerance to its use in the public sphere. There are, however, few available resources to study this variation and its change over time and in more informal areas, on social media. In this paper, we propose a first step to creating a corpus of dialectal variation of written Norwegian. We collect a small corpus of tweets and manually annotate them as Bokmål, Nynorsk, any dialect, or a mix. We further perform preliminary experiments with state-of-the-art models, as well as an analysis of the data to expand this corpus in the future. Finally, we make the annotations available for future work.

• Barnes, Jeremy; Øvrelid, Lilja & Velldal, Erik (2021). If you've got it, flaunt it: Making the most of fine-grained sentiment annotations. In Merlo, Paola; Tiedemann, Jörg & Tsarfaty, Reut (Ed.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics. ISSN 978-1-954085-02-2. p. 49–62.
• Ravishankar, Vinit; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2021). Multilingual ELMo and the Effects of Corpus Sampling. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 978-91-7929-614-8. p. 378–383.
• Mæhlum, Petter; Barnes, Jeremy; Kurtz, Robin; Øvrelid, Lilja & Velldal, Erik (2021). Negation in Norwegian: an annotated dataset . In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 978-91-7929-614-8. p. 299–308.
• Kutuzov, Andrei; Barnes, Jeremy; Velldal, Erik; Øvrelid, Lilja & Oepen, Stephan (2021). Large-Scale Contextualised Language Modelling for Norwegian. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 978-91-7929-614-8. p. 30–40. Show summary

  We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first large-scale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian. For additional background and access to the data, models, and software, please see: http://norlm.nlpl.eu

• Rodina, Julia; Trofimova, Yuliya; Kutuzov, Andrei & Artemova, Ekaterina (2021). ELMo and BERT in Semantic Change Detection for Russian, Proceedings of AIST 2020: Analysis of Images, Social Networks and Texts. Springer. ISSN 978-3-030-72610-2. p. 175–186. Show summary

  We study the effectiveness of contextualized embeddings for the task of diachronic semantic change detection for Russian language data. Evaluation test sets consist of Russian nouns and adjectives annotated based on their occurrences in texts created in pre-Soviet, Soviet and post-Soviet time periods. ELMo and BERT architectures are compared on the task of ranking Russian words according to the degree of their semantic change over time. We use several methods for aggregation of contextualized embeddings from these architectures and evaluate their performance. Finally, we compare unsupervised and supervised techniques in this task.

• Kutuzov, Andrei & Kuzmenko, Elizaveta (2021). Representing ELMo embeddings as two-dimensional text online, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics. ISSN 978-1-954085-05-3. p. 143–148. Show summary

  We describe a new addition to the WebVectors toolkit which is used to serve word embedding models over the Web. The new ELMoViz module adds support for contextualized embedding architectures, in particular for ELMo models. The provided visualizations follow the metaphor of ‘two-dimensional text’ by showing lexical substitutes: words which are most semantically similar in context to the words of the input sentence. The system allows the user to change the ELMo layers from which token embeddings are inferred. It also conveys corpus information about the query words and their lexical substitutes (namely their frequency tiers and parts of speech). The module is well integrated into the rest of the WebVectors toolkit, providing lexical hyperlinks to word representations in static embedding models. Two web services have already implemented the new functionality with pre-trained ELMo models for Russian, Norwegian and English.

• Dahl, Fredrik Andreas; Kasicheyanula, Taraka Rama; Hurlen, Petter; Brekke, Pål Haugar; Husby, Haldor & Gundersen, Tore [Show all 8 contributors for this article] (2021). Neural classification of Norwegian radiology reports: using NLP to detect findings in CT-scans of children. BMC Medical Informatics and Decision Making. ISSN 1472-6947. 21(1), p. 1–8. doi: 10.1186/s12911-021-01451-8. Full text in Research Archive
• Kutuzov, Andrei; Fomin, V.; Mikhailov, V. Nikola & Rodina, Julia (2020). Shiftry: Web service for diachronic analysis of Russian news. Komp'yuternaya Lingvistika i Intellektual'nye Tekhnologii. ISSN 2221-7932. 2020-(19), p. 500–516. doi: 10.28995/2075-7182-2020-19-500-516.
• Katricheva, Nadezda; Yaskevich, Alyaxey; Lisitsina, Anastasiya; Zhordaniya, Tamara; Kutuzov, Andrei & Kuzmenko, Elizaveta (2020). Vec2graph: A Python Library for Visualizing Word Embeddings as Graphs. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 1086, p. 190–198. doi: 10.1007/978-3-030-39575-9_20.
• Kurtz, Robin; Oepen, Stephan & Kuhlmann, Marco (2020). End-to-End Negation Resolution as Graph Parsing. In Bouma, Gosse; Matsumoto, Yuji; Oepen, Stephan; Sagae, Kenji; Seddah, Djame; Sun, Weiwei; Søgaard, Anders & Tsarfaty, Reut (Ed.), Proceedings of the 16th International Conference on Parsing Technologies and the IWPT 2020 Shared Task on Parsing into Enhanced Universal Dependencies. Association for Computational Linguistics. ISSN 978-1-952148-11-8. p. 14–24.

View all works in Cristin

• Tahmasebi, Nina; Montariol, Syrielle; Dubossarsky, Haim; Kutuzov, Andrei; Hengchen, Simon & Alfter, David [Show all 8 contributors for this article] (2023). Proceedings of the 4th Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics. ISBN 979-8-89176-043-1. 174 p.
• Tahmasebi, Nina; Montariol, Syrielle; Kutuzov, Andrei; Hengchen, Simon; Dubossarsky, Haim & Borin, Lars (2022). Proceedings of the 3rd Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics. ISBN 978-1-955917-42-1. 219 p.
• Barnes, Jeremy (2021). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. ISBN 978-1-954085-46-6. 5953 p.
• Dobnik, Simon & Øvrelid, Lilja (2021). Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISBN 978-91-7929-614-8. 467 p.
• Kutuzov, Andrei; van der Aalst, Wil M. P.; Batagelj, Vladimir & Ignatov, Dmitry (2021). Proceedings of AIST 2020: Analysis of Images, Social Networks and Texts. Springer. ISBN 978-3-030-72610-2. 480 p.

View all works in Cristin

• Riegler, Michael; Lison, Pierre & Falkum, Ingrid Lossius (2023). Kunstig intelligens-modeller er ikke miniatyr­versjoner av den menneskelige hjernen. Forskersonen.no.
• Lison, Pierre & Touileb, Samia (2023). Chat GPT egner seg dårlig til eksamenssensuren. Morgenbladet. ISSN 0805-3847.
• Walker, Nicholas Thomas & Lison, Pierre (2023). GraphWOZ: Dialogue Management with Conversational Knowledge Graphs.
• Falkum, Ingrid Lossius & Lison, Pierre (2023). Er prateroboten ChatGPT en klok samtale­partner eller papegøye? Forskersonen.no. Show summary

  https://forskersonen.no/chatgpt-kronikk-kunstig-intelligens/er-prateroboten-chatgpt-en-klok-samtalepartner-eller-papegoye/2157206

• Korsvoll, Nils Hallvard; Lison, Pierre; Reinertsen, Hilde; Elken, Mari; De Korne, Haley & Hansen, Kai Arne [Show all 7 contributors for this article] (2023). Fire tiltak for en bedre språkpolitikk i akademia. Khrono.no. ISSN 1894-8995.
• Mæhlum, Petter & Eivind, Torgersen (2022). Det er mange måter å si nei på – her er noen av de fineste . [Internet]. titan.no.
• Eivind, Torgersen & Mæhlum, Petter (2022). Ordet «ikke» skaper trøbbel for kunstig intelligens. [Internet]. forskning.no.
• Vyacheslav, Stroev; Dmitry, Grechka; Maria, Eliseeva & Arefyev, Nikolay (2022). Kashtanka.pet: employing AI to help lost pets return to their homes. Show summary

  Аннотация доклада: The Kashtanka.pet project addresses the problem of searching for lost pets efficiently. There are numerous platforms and groups in social networks collecting ads about missing and found cats and dogs. However, it is often almost impossible for a human to find a specific lost pet among millions of ads about found pets distributed across all those websites. We develop a system that allows pet owners and volunteers to find lost pets efficiently with the help of AI. It crawls websites for ads about lost and found pets, and retrieves the pairs of ads announcing the same pet was lost and then found. The retrieved pairs are then inspected and further processed by humans. On the poster we present the architecture of the Kashtanka.pet system. Then we address the problem of evaluating and improving the quality of underlying AI models of lost pets retrieval. The standard manual annotation of a dataset for our task requires finding the matching pairs of lost and found ads, which makes the annotation process prohibitively difficult. Thus, we generate the matching pairs automatically by splitting sets of photos from the ads containing several photos into two parts. However, the simple random splitting results in both parts sharing some photos made in the same place. This may promote models searching for the same background rather than the same pet, and makes the setup unrealistically simple because in reality a pet is lost and found in different places. To mitigate this we propose a method that finds those ads that contain photos of a pet taken in several places and splits them accordingly. In order to estimate the quality of this method and select its hyperparameters, we additionally annotated pairs of photos from random ads asking annotators if those photos were taken in the same or different places. Several methods solving the task are proposed and compared. The pipeline currently deployed at kashtanka.pet is based on YOLOv4 for pet detection and cropping, EfficientNet for obtaining their embeddings, and GRU for aggregation of those embeddings across several images from the same ad. The model is trained with the triplet loss on the target dataset. Two other methods employ the pre-trained multimodal BLIP and SLIP models, which are the recently introduced improvements over the popular CLIP model. We found that even without fine-tuning on the data from our target domain the image embeddings from the multimodal models significantly outperform the currently deployed pipeline.

• Løland, Anders; Fuglerud, Kristin Skeide & Lison, Pierre (2022). Hva er universell utforming? . [Internet]. Sannsynligvis VIKTIG .
• Lison, Pierre; Rots, Aike Peter & De Korne, Haley (2022). Hvilket fremmedspråk bør man lære seg i Google-oversettelsenes tidsalder? Morgenbladet. ISSN 0805-3847.
• Lison, Pierre; Korsvoll, Nils Hallvard & Lundberg, Aase-Kristine Aasen (2022). Kjernekraft -er det farlig, eller er det fremtiden? Morgenbladet. ISSN 0805-3847.
• van der Aalst, Wil M. P.; Batagelj, Vladimir; Buzmakov, Alexey; Ignatov, Dmitry I.; Kalenkova, Anna & Khachay, Michael [Show all 18 contributors for this article] (2021). Preface. Communications in Computer and Information Science (CCIS). ISSN 1865-0929. 1357, p. vii–x.
• Welzl, Michael; Oepen, Stephan; Cezary Jaskula, Cezary; Griwodz, Carsten & Islam, Safiqul (2021). Collaboration in the IETF: an initial analysis of two decades in email discussions. Computer communication review. ISSN 0146-4833. 51(3), p. 29–32. doi: 10.1145/3477482.3477488. Show summary

  RFC 9000, published in May 2021, marks an important milestone for the Internet's standardization body, the Internet Engineering Task Force (IETF): finally, the specification of the QUIC protocol is available. QUIC is the result of a five-year effort - and it is also the second of two major protocols (the first being SPDY, which later became HTTP/2) that Google LLC first deployed, and then brought to the IETF for standardization. This begs the question: when big players follow such a "shoot first, discuss later" approach, is IETF collaboration still "real", or is the IETF now being (mis-)used to approve protocols for standardization when they are already practically established, without really actively involving anyone but the main proponents?

• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2021). Using Gender- and Polarity-informed Models to Investigate Bias.
• Lison, Pierre; Bølstad, Jørgen & Kvellestad, Anders (2021). Forvirrende pandemistatistikk: Hva skal vi med logaritmer i grafer? Morgenbladet. ISSN 0805-3847.
• Velldal, Erik; Walsø, Claes Lyth & Øvrelid, Lilja (2021). Nå kan maskinene tolke hva du føler og mener. [Internet]. forskning,no.
• Velldal, Erik; Walsø, Claes Lyth & Øvrelid, Lilja (2021). Nå er maskinene klare til å tolke hva du føler og mener. [Internet]. titan.uio.no.
• Lison, Pierre (2021). Skweak: Weak Supervision Made Easy for NLP.
• Lison, Pierre (2021). Fremdrift i forskningsprosjekter.
• Lison, Pierre; Pilán, Ildikó; Sánchez Ruenes, David; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions.
• Lison, Pierre; Pilán, Ildikó; Øvrelid, Lilja; Sánchez Ruenes, David & Batet, Montserrat (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Lison, Pierre; Barnes, Jeremy Claude & Hubin, Aliaksandr (2021). skweak: weak supervision made easy for NLP. Show summary

  We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak

• Lison, Pierre & Falkum, Ingrid Lossius (2020). Hva skjedde med «Don’t be evil»? Morgenbladet. ISSN 0805-3847.
• Rønningstad, Egil & Øvrelid, Lilja (2020). Targeted Sentiment Analysis for Norwegian text. Universitetet i Oslo.

View all works in Cristin