Lilja Øvrelid

• Wan, Jiajing; Touileb, Samia; Steskal, Lubos & Øvrelid, Lilja (2026). Personalizing News Headlines with Retrieval-Augmented Generation. Association for Computational Linguistics (ACL). Annual Meeting Conference Proceedings. ISSN 0736-587X. p. 55–67. doi: 10.18653/v1/2026.customnlp4u-1.6. Full text in Research Archive Show summary

  We focus on personalized news headline generation, where we aim to improve headline generation by extending the generation context to incorporate the news reading history of users. In particular, we study a RAG-LLM-based system that customizes news headlines with user histories to improve news headline personalization. Our experiments show that our approach not only produces better headlines for specific users, but also makes the generated headlines closer to the original headlines. We experiment with different retrievers and analyze the generated outputs through systematic comparisons with both original and rewritten headlines. These analyses provide insights into the role of retrieval and personalization in headline generation, highlighting how the user history contributes to meaningful improvement while remaining aligned with original headlines

• Torunoğlu-Selamet, Dilara; Arslan, Doğukan; Wilkens, Rodrigo; He, Wei; Eryiğit, Doruk & Pickard, Thomas [Show all 78 contributors for this article] (2026). A Parallel Cross-Lingual Benchmark for Multimodal Idiomaticity Understanding - ACL Anthology. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 9434–9448. doi: 10.63317/5cvnbcoktfo2. Full text in Research Archive
• Norman, Rebecka Maria; Storset, Lilja Charlotte; Mæhlum, Petter; Iversen, Hilde Karin Hestad; Jelin, Elma & Velldal, Erik [Show all 8 contributors for this article] (2026). Exploiting survey and online patient experience comments in general practice: validating a fine-tuned language model for automatic sentiment analysis. Family Practice. ISSN 0263-2136. 43(5). doi: 10.1093/fampra/cmag071. Full text in Research Archive
• Samuel, David; Øvrelid, Lilja; Velldal, Erik & Kutuzov, Andrey (2026). Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages. In Vondrick, C.; Hariharan, B.; Raffel, C.; Pinto, L.; Yang, D. & Faust, A. (Ed.), International Conference on Learning Representations (ICLR) 2026. International Conference on Learning Representations (ICLR). ISSN 9798331339678. p. 94000–94027. doi: https:/proceedings.iclr.cc/paper_files/paper/2026/hash/97a057ab1235218027dc07c50eedee0d-Abstract-Conference.html. Full text in Research Archive Show summary

  We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

• Rønningstad, Egil; Klinger, Roman; Øvrelid, Lilja & Velldal, Erik (2026). Entity-Level Sentiment Analysis with Sentence Relevance Detection. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 8040–8055. doi: 10.63317/35ideqx4jk89. Full text in Research Archive Show summary

  The task of entity-level sentiment analysis (Elsa) is to extract sentiment scores for a given entity (such as person names or organization names) from a text. Elsa is a challenging task and involves processing of longer documents, where several entities may be mentioned with varying importance for the final score aggregation. Fine-tuning encoder-based Transformers (such as BERT) constitutes the state of the art for sentiment predictions, however, these models are still limited by their restricted input lengths. Decoder-only models so far still underperform on the task. We approach the context limitation by learning to extract segments that are relevant for the sentiment prediction for a given entity, without preprocessing by chunking and aggregation. For decoder models, we explore fine-tuning these through supervised fine-tuning and pairwise comparison, a method borrowed from reward modeling for preference optimization. Both methods perform well and set a new standard for the Elsa task. We further show that pairwise classification is faster, simpler, and shows less variance than the more common direct supervision for this task.

• Olsen, Helene Bøsei; Velldal, Erik & Øvrelid, Lilja (2026). MUC-4 Revisited: Document-level Event Analysis beyond Span-based Arguments. In Piperidis,, Stelios; Bel, Núria; Heuvel, Henk van den; Ide, Nancy; Krek, Simon & Toral, Antonio (Ed.), Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026). European Language Resources Association. ISSN 9782493814494. p. 7766–7780. doi: 10.63317/2kaxd8nu44bx. Full text in Research Archive Show summary

  Automatically predicting structured representations of events has long been a central goal in information extraction, yet most contemporary work remains limited to identifying contiguous text spans as event arguments. This span-centric formulation fails to capture higher-level aspects of real-world events, such as actor identities, temporal scope, and aggregated outcomes, that many event-centred applications depend on. While commonly treated as a standard extractive benchmark, MUC-4 originally combined span-based arguments with normalised, inferred, and categorical fields, reflecting a richer, application-driven design. In this paper, we revisit MUC-4 in its full original formulation, casting it as an abstractive event analysis task that connects traditional event extraction goals with modern generative and document-level paradigms. We provide the first systematic evaluation of fine-tuned generative models in this extended formulation on MUC-4, examining how post-training stages and model size affect performance across both span-based and higher-level, semantically grounded event information. An extensive error analysis highlights practical challenges and directions for future work.

• Storset, Lilja Charlotte; Jelin, Elma; Norman, Rebecka Maria; Bjertnæs, Øyvind Andresen; Øvrelid, Lilja & Velldal, Erik (2026). From Pain to Praise: Aspect-Based Sentiment Analysis for Norwegian Patient Feedback. In Danilova, Vera; Kurfalı, Murathan; Söderfeldt, Ylva; Reed, Julia & Burchell, Andrew (Ed.), Proceedings of the 1st Workshop on Linguistic Analysis for Health (HeaLing 2026). Association for Computational Linguistics (ACL). ISSN 9798891763678. p. 191–202. doi: 10.18653/v1/2026.healing-1.16. Full text in Research Archive Show summary

  This paper describes a new dataset for aspect-based sentiment analysis (ABSA) for analyzing patient feedback about healthcare services. In an interdisciplinary collaboration spanning the fields of natural language processing and healthcare research, we manually annotate a dataset of 2382 free-text comments collected from national patient experience surveys in Norway, covering two sub-fields of services – special mental healthcare and general practitioners. Annotations are provided on both the sentence- and comment-level, covering a fine-grained set of 25 unique healthcare-related aspects and their polarities. We also report results for fine-tuning both encoder- and decoder models on the resulting dataset, comparing different modeling strategies, like joint and sequential prediction of aspects and polarity. The resources developed in this work can assist healthcare researchers in the analysis of patient feedback, bringing a much more efficient approach compared to today’s manual analysis, potentially leading to improved patient satisfaction and clinical outcomes.

• Papadopoulou, Anthi; Lison, Pierre; Anderson, Mark David; Øvrelid, Lilja & Pilán, Ildikó (2026). Neural text sanitization with privacy risk indicators: an empirical analysis. Language Resources and Evaluation. ISSN 1574-020X. 60. doi: 10.1007/s10579-026-09911-1. Full text in Research Archive Show summary

  Text sanitization is the task of redacting a document to mask all occurrences of (direct or indirect) personal identifiers, with the goal of concealing the identity of the individual(s) referred in it. In this paper, we consider a two-step approach to text sanitization and provide a detailed analysis of its empirical performance on two recently published datasets: the Text Anonymization Benchmark (Pilán et al., 2022) and a collection of Wikipedia biographies (Papadopoulou et al., 2022a). The text sanitization process starts with a privacy-oriented entity recognizer that seeks to determine the text spans expressing identifiable personal information. This privacy-oriented entity recognizer is trained by combining a standard named entity recognition model with a gazetteer populated by person-related terms extracted from Wikidata. The second step of the text sanitization process consists in assessing the privacy risk associated with each detected text span, either isolated or in combination with other text spans. We present five distinct indicators of the re-identification risk, respectively based on language model probabilities, text span classification, sequence labelling, perturbations, and web search. We provide a contrastive analysis of each privacy indicator and highlight their benefits and limitations, notably in relation to the available labeled data.

• Wold, Sondre; Simon, Étienne; Velldal, Erik & Øvrelid, Lilja (2026). Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality. In Demberg, Vera; Inui, Kentaro & Marquez, Lluís (Ed.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891763807. p. 2239–2252. doi: https:/aclanthology.org/2026.eacl-long.99/. Full text in Research Archive Show summary

  The principle of compositionality, which concerns the construction of meaning from constituent parts, is a longstanding topic in various disciplines, most commonly associated with formal semantics. In NLP, recent studies have focused on the compositional properties of text embedding models, particularly regarding their sensitivity to idiomatic expression, as idioms have traditionally been seen as non-compositional. In this paper, we argue that it is unclear how previous work relates to formal definitions of the principle. To address this limitation, we take a theoretically motivated approach based on definitions in formal semantics. We present 𝜀-compositionality, a continuous relaxation of compositionality derived from these definitions. We measure 𝜀-compositionality on a dataset containing both idiomatic and non-idiomatic sentences, providing a theoretically motivated assessment of sensitivity to idiomaticity. Our findings indicate that most text embedding models differentiate between idiomatic and non-idiomatic phrases, although to varying degrees.

• Samuel, David; Mikhailov, Vladislav; Velldal, Erik; Øvrelid, Lilja; Charpentier, Lucas Georges Gabriel & Kutuzov, Andrey [Show all 7 contributors for this article] (2025). Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 573–608. doi: https:/aclanthology.org/2025.nodalida-1.61/. Full text in Research Archive Show summary

  Training large language models requires vast amounts of data, posing a challenge for less widely spoken languages like Norwegian and even more so for truly low-resource languages like Northern S{\'a}mi. To address this issue, we present a novel three-stage continual training approach that substantially improves the downstream performance together with the inference efficiency for the target languages. Based on our findings, we train, evaluate, and openly release a new generative language model for Norwegian Bokm{\r{a}}l, Nynorsk, and Northern S{\'a}mi with 11.4 billion parameters: NorMistral-11B.

• Touileb, Samia; Mikhailov, Vladislav; Kroka, Marie Ingeborg; Øvrelid, Lilja & Velldal, Erik (2025). Benchmarking Abstractive Summarisation: A Dataset of Human-authored Summaries of Norwegian News Articles. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 729–738. doi: https:/aclanthology.org/2025.nodalida-1.73/. Full text in Research Archive Show summary

  We introduce a dataset of high-quality human-authored summaries of news articles in Norwegian. The dataset is intended for benchmarking of the abstractive summarisation capabilities of generative language models. Each document in the dataset is provided with three different candidate gold-standard summaries written by native Norwegian speakers and all summaries are provided in both of the written variants of Norwegian – Bokmål and Nynorsk. The paper describes details on the data creation effort as well as an evaluation of existing open LLMs for Norwegian on the dataset. We also provide insights from a manual human evaluation, comparing human-authored to model generated summaries. Our results indicate that the dataset provides a challenging LLM benchmark for Norwegian summarisation capabilities.

• Simon, Étienne; Olsen, Helene Bøsei; Villar, Ramón Carreño; Mishra, Rahul; Arefyev, Nikolay & Yilmaz, Mert Can [Show all 8 contributors for this article] (2025). Abstractive Event Analysis of Armed Conflicts: Introducing the UCDP-AEC Dataset. In Wartena, Christian & Heid, Ulrich (Ed.), Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops. HsH Applied Academics. ISSN 9783690180177. p. 104–119. doi: https:/aclanthology.org/2025.konvens-2.8.pdf. Full text in Research Archive Show summary

  This paper introduces a new dataset of document-level event annotations in the domain of armed conflict. By augmenting the event database from the Uppsala Conflict Data Program (UCDP) with source documents identified in public web archives, we create the UCDP Abstractive Event analysis Corpus (UCDP-AEC). While a large part of research on information extraction is focused on extracting text spans, realworld use cases often require inferring more high-level information that is not necessarily explicitly mentioned in texts. UCDP-AEC differs from traditional event extraction datasets in that the document-level annotations do not correspond to mere text spans of the input, but capture expert-interpreted and often implicit information. With more than 10 000 documents, UCDP-AEC is of comparable size to the largest human-annotated traditional event extraction datasets. We also report preliminary experimental results for various generative approaches, by fine-tuning both decoder models and existing event argument extraction models that require minimal adaptation to our abstractive formulation of the task

• Jelin, Elma; Storset, Lilja Charlotte; Norman, Rebecka Maria; Iversen, Hilde Karin Hestad; Ellingsen-Dalskau, Lina Harvold & Mæhlum, Petter [Show all 9 contributors for this article] (2025). From words to action? A scoping review on automatic sentiment analysis of patient experience comments from online sources and surveys. BMJ Health & Care Informatics. 32(1). doi: 10.1136/bmjhci-2025-101631. Full text in Research Archive Show summary

  Background Automatic analysis of free-text patient comments enables the efficient processing of large feedback volumes, reducing reliance on manual review. A 2021 review examined natural language processing (NLP) and sentiment analysis (SA) in patient experience research; however, recent advances in deep learning and generative artificial intelligence (AI) call for an updated synthesis. Objectives This scoping review aims to map and summarise recent studies applying SA to unstructured patient experience data related to healthcare services. Methods Following Joanna Briggs Institute methodology and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we conducted a comprehensive search across Medline, CINAHL, Web of Science, Cochrane, Embase and APA PsycINFO. We included studies published from January 2020 to March 2024 in English or a Scandinavian language. Eligible studies analysed patient feedback using NLP techniques and described the development or validation of SA models. Two reviewers independently screened the studies and extracted data, which were presented in tables both tabular and narrative forms. Results 30 studies were included, primarily from the USA, Europe and Asia. Patient comments were mostly sourced from online platforms such as social media. Feedback largely concerned hospital care. 18 studies employed rule-based SA approaches, while 12 applied supervised machine learning (ML) and only 4 studies used deep learning models. Few addressed the visualisation or practical applications in healthcare. Conclusion Despite significant progress, modern methods like deep learning and generative AI remain underused in SA of patient-experience data. Limited focus on implementation restricts SA’s role in quality improvement. Future research should assess advanced methods and their cost-effectiveness versus traditional ML.

• Mikhailov, Vladislav; Artemova, Ekaterina; Butenko, Zoia; Øvrelid, Lilja & Velldal, Erik (2025). Overview of the Preference Prediction Task at the ELOQUENT 2025 lab for evaluating generative language model quality. CEUR Workshop Proceedings. 4038, p. 1320–1329. doi: https:/ceur-ws.org/Vol-4038/paper_105.pdf. Full text in Research Archive Show summary

  The Preference Prediction shared task, part of the ELOQUENT Lab at CLEF-2025, challenges participants to build systems that simulate human judgment in evaluating pairs of LLM-generated responses. This novel task includes two sub-tasks. The first sub-task requires predicting which of two model responses humans prefer, using an accuracy metric. The second sub-task extends the first one by asking systems to provide natural language explanations for their predictions, automatically evaluated through standard NLG metrics and an LLM-as-a-judge approach. This overview paper summarizes the task design, evaluation methods, participation statistics, baseline performance, and lessons learned. We discuss the system performance in both sub-tasks. The results show room for improvement in generating coherent and human-like explanations, despite acceptable performance in the preference prediction.

• Karlgren, Jussi; Artemova, Ekaterina; Bojar, Ondřej; Engels, Marie Isabel; Mikhailov, Vladislav & Šindelář, Pavel [Show all 8 contributors for this article] (2025). Overview of ELOQUENT 2025: Shared Tasks for Evaluating Generative Language Model Quality. In Carrillo-de-Albornoz, Jorge (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction (CLEF 2025).. Springer Publishing Company. ISSN 9783032043542. doi: 10.1007/978-3-032-04354-2_14. Full text in Research Archive
• You, Huiling; Touileb, Samia; Velldal, Erik & Øvrelid, Lilja (2025). Event-based evaluation of abstractive news summarization. In Arviv, Ofir; Clinciu, Miruna; Dhole, Kaustubh; Dror, Rotem; Gehrmann, Sebastian; Habba, Eliya; Itzhak, Itay; Mille, Simon; Perlitz, Yotam; Santus, Enrico; Sedoc, João; Scheuer, Michal Shmueli; Stanovsky, Gabriel & Tafjord, Oyvind (Ed.), Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM²). Association for Computational Linguistics (ACL). ISSN 9798891762619. p. 504–510. doi: https:/aclanthology.org/2025.gem-1.43/. Full text in Research Archive
• Storset, Lilja Charlotte; Rønningstad, Egil; Mæhlum, Petter; Velldal, Erik & Øvrelid, Lilja (2025). Mixed Feelings: Cross-Domain Sentiment Classification of Patient Feedback. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 537–543. doi: https:/aclanthology.org/2025.nodalida-1.58/. Full text in Research Archive Show summary

  Sentiment analysis of patient feedback from the public health domain can aid decision makers in evaluating the provided services. The current paper focuses on free-text comments in patient surveys about general practitioners and psychiatric healthcare, annotated with four sentence-level polarity classes - positive, negative, mixed and neutral - while also attempting to alleviate data scarcity by leveraging general-domain sources in the form of reviews. For several different architectures, we compare in-domain and out-of-domain effects, as well as the effects of training joint multi-domain models.

• Mikhailov, Vladislav; Enstad, Tita Ranveig; Samuel, David; Farsethås, Hans Christian; Velldal, Erik & Øvrelid, Lilja [Show all 7 contributors for this article] (2025). NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). ISSN 9798891762565. p. 3495–3541. doi: 10.18653/v1/2025.findings-acl.181. Full text in Research Archive Show summary

  This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets – of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-created prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pretrained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available.

• Mikhailov, Vladislav; Mæhlum, Petter; Langø, Victoria; Velldal, Erik & Øvrelid, Lilja (2025). A Collection of Question Answering Datasets for Norwegian. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 397–407. doi: https:/aclanthology.org/2025.nodalida-1.43/. Full text in Research Archive Show summary

  This paper introduces a new suite of question answering datasets for Norwegian; NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA. The data covers a wide range of skills and knowledge domains, including world knowledge, commonsense reasoning, truthfulness, and knowledge about Norway. Covering both of the written standards of Norwegian – Bokmål and Nynorsk – our datasets comprise over 10k question-answer pairs, created by native speakers. We detail our dataset creation approach and present the results of evaluating 11 language models (LMs) in zero- and few-shot regimes. Most LMs perform better in Bokmål than Nynorsk, struggle most with commonsense reasoning, and are often untruthful in generating answers to questions. All our datasets and annotation materials are publicly available.

• You, Huiling; Touileb, Samia; Velldal, Erik & Øvrelid, Lilja (2025). NorEventGen: generative event extraction from Norwegian news. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 801–811. doi: https:/aclanthology.org/2025.nodalida-1.79/. Full text in Research Archive Show summary

  In this work, we approach event extraction from Norwegian news text using a generation-based approach which formulates the task as text-to-structure generation. We present experiments assessing the effect of different modeling configurations and provide an analysis of the model predictions and typical system errors. Finally, we apply our system to a large corpus of raw news texts and analyze the resulting distribution of event structures in a fairly representative snap-shot of the Norwegian news landscape.

• Rosa, Javier De La; Mikhailov, Vladislav; Zhang, Lemei; Wetjen, Freddy; Samuel, David & Liu, Peng [Show all 19 contributors for this article] (2025). The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Johansson, Richard & Stymne, Sara (Ed.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025). University of Tartu Library. ISSN 9789908531090. p. 544–560. Full text in Research Archive
• Wold, Sondre; Simon, Étienne; Charpentier, Lucas Georges Gabriel; Kostylev, Egor V.; Velldal, Erik & Øvrelid, Lilja (2024). Compositional Generalization with Grounded Language Models, Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760998. p. 3447–3460. doi: 10.18653/v1/2024.findings-acl.205. Full text in Research Archive Show summary

  Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous work on compositional generalization in semantic parsing, we allow for a controlled evaluation of the degree to which these models learn and generalize from patterns in knowledge graphs. We develop a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and further avoids grounding the language models in information already encoded implicitly in their weights. We evaluate existing methods for combining language models with knowledge graphs and find them to struggle with generalization to sequences of unseen lengths and to novel combinations of seen base components. While our experimental results provide some insight into the expressive power of these models, we hope our work and released datasets motivate future research on how to better combine language models with structured knowledge representations.

• Hussiny, Mohammad Ali; Payenda, Mohammad Arif & Øvrelid, Lilja (2024). PersianEmo: Enhancing Farsi-Dari Emotion Analysis with a Hybrid Transformer and Recurrent Neural Network Model. Proceedings of LREC. p. 257–263. Full text in Research Archive Show summary

  Emotion analysis is a critical research domain within the field of natural language processing (NLP). While substantial progress has been made in this area for the Persian language, there is still a need for more precise models and larger datasets specifically focusing on the Farsi and Dari dialects. In this research, we introduce “LearnArmanEmo” as a new dataset and a superior ensemble approach for Persian text emotion classification. Our proposed model, which combines XLM-RoBERTa-large and BiGRU, undergoes evaluation on LetHerLearn for the Dari dialect, ARMANEMO for the Farsi dialect, and LearnArmanEmo for both Dari and Farsi dialects. The empirical results substantiate the efficacy of our approach with the combined model demonstrating superior performance. Specifically, our model achieves an F1 score of 72.9% on LetHerLearn, an F1 score of 77.1% on ARMANEMO, and an F1 score of 78.8% on the LearnArmanEmo dataset, establishing it as a better ensemble model for these datasets. These findings underscore the potential of this hybrid model as a useful tool for enhancing the performance of emotion analysis in Persian language processing.

• Olsen, Helene Bøsei; Simon, Étienne; Velldal, Erik & Øvrelid, Lilja (2024). Socio-political Events of Conflict and Unrest: A Survey of Available Datasets. In Hürriyetoğlu, Ali; Tanev, Hristo; Surendrabikram, Thapa & Uludoğan, Gökçe (Ed.), Proceedings of the 7th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE 2024). Association for Computational Linguistics (ACL). ISSN 9798891760707. p. 40–53. Full text in Research Archive
• Mæhlum, Petter; Samuel, David; Norman, Rebecka Maria; Jelin, Elma; Bjertnæs, Øyvind Andresen & Øvrelid, Lilja [Show all 7 contributors for this article] (2024). It’s Difficult to be Neutral – Human and LLM-based Sentiment Annotation of Patient Comments, Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health). ELRA Language Resources Association. ISSN 9782493814258. p. 8–19. Full text in Research Archive
• Touileb, Samia; Murstad, Jeanett; Mæhlum, Petter; Steskal, Lubos; Storset, Lilja Charlotte & You, Huiling [Show all 7 contributors for this article] (2024). EDEN: A Dataset for Event Detection in Norwegian News. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 5495–5506. Full text in Research Archive Show summary

  We present EDEN, the first Norwegian dataset annotated with event information at the sentence level, adapting the widely used ACE event schema to Norwegian. The paper describes the manual annotation of Norwegian text as well as transcribed speech in the news domain, together with inter-annotator agreement and discussions of relevant dataset statistics. We also present preliminary modeling results using a graph-based event parser. The resulting dataset will be made freely available for download and use.

• Rønningstad, Egil; Velldal, Erik & Øvrelid, Lilja (2024). A GPT among Annotators: LLM-based Entity-Level Sentiment Annotation. In Henning, Sophie & Stede, Manfred (Ed.), Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII). Association for Computational Linguistics (ACL). ISSN 9798891760738. p. 133–139. Full text in Research Archive Show summary

  We investigate annotator variation for the novel task of Entity-Level Sentiment Analysis (ELSA) which annotates the aggregated sentiment directed towards volitional entities in a text. More specifically, we analyze the annotations of a newly constructed Norwegian ELSA dataset and release additional data with each annotator’s labels for the 247 entities in the dataset’s test split. We also perform a number of experiments prompting ChatGPT for these sentiment labels regarding each entity in the text and compare the generated annotations with the human labels. Cohen’s Kappa for agreement between the best LLM-generated labels and curated gold was 0.425, which indicates that these labels would not have high quality. Our analyses further investigate the errors that ChatGPT outputs, and compare them with the variations that we find among the 5 trained annotators that all annotated the same test data.

• Simon, Étienne; Olsen, Helene Bøsei; You, Huiling; Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2024). Generative Approaches to Event Extraction: Survey and Outlook. In Tetreault, Joel; Nguyen, Thien Huu; Lamba, Hemank & Hughes, Amanda (Ed.), Proceedings of the Workshop on the Future of Event Detection (FuturED). Association for Computational Linguistics (ACL). ISSN 9798891761735. p. 73–86. doi: 10.18653/v1/2024.futured-1.7. Full text in Research Archive
• Rønningstad, Egil; Klinger, Roman; Øvrelid, Lilja & Velldal, Erik (2024). Entity-Level Sentiment: More than the Sum of Its Parts, The 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis Proceedings of the Workshop. Association for Computational Linguistics (ACL). ISSN 9798891761568. p. 84–96. Full text in Research Archive Show summary

  In sentiment analysis of longer texts, there may be a variety of topics discussed, of entities mentioned, and of sentiments expressed regarding each entity. We find a lack of studies exploring how such texts express their sentiment towards each entity of interest, and how these sentiments can be modelled. In order to better understand how sentiment regarding persons and organizations (each entity in our scope) is expressed in longer texts, we have collected a dataset of expert annotations where the overall sentiment regarding each entity is identified, together with the sentence-level sentiment for these entities separately. We show that the reader's perceived sentiment regarding an entity often differs from an arithmetic aggregation of sentiments at the sentence level. Only 70% of the positive and 55% of the negative entities receive a correct overall sentiment label when we aggregate the (human-annotated) sentiment labels for the sentences where the entity is mentioned. Our dataset reveals the complexity of entity-specific sentiment in longer texts, and allows for more precise modelling and evaluation of such sentiment expressions.

• Payenda, Mohammad Arif; Vahidi, Abdul Razaq; Hussiny, Mohammad Ali; Prinz, Andreas & Øvrelid, Lilja (2024). PashtoEmo: Enhancing Text-Based Emotion Analysis in the Pashto Language Through Dataset Creation. Lecture Notes in Computer Science (LNCS). ISSN 0302-9743. 14763, p. 225–236. doi: 10.1007/978-3-031-70242-6_22. Full text in Research Archive Show summary

  This paper presents the comprehensive PashtoEmo dataset for emotion analysis in the Pashto language. PashtoEmo contains 8016 text instances from X social media platform covering several topics, including politics, women’s rights, social justice, culture, sports, and education. The texts are annotated with six emotional categories and an additional ‘Other’ class. We tested PashtoEmo for emotion analysis with several machine learning, deep learning, and transformer models. Even the best model, XLM-RoBERTa-large, achieved only a 76.85% F1 score and a 78.11% accuracy. This indicates that the data set must be extended with adequate resources in Pashto language text-based applications.

• Samuel, David & Øvrelid, Lilja (2023). Tokenization with Factorized Subword Encoding, Findings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics (ACL). ISSN 9781959429623. Full text in Research Archive
• Hussiny, Mohammad Ali & Øvrelid, Lilja (2023). Emotion Analysis of Tweets Banning Education in Afghanistan. In Barnes, Jeremy; Clercq, Orphee De & Klinger, Roman (Ed.), The 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis. Association for Computational Linguistics (ACL). ISSN 9781959429876. Full text in Research Archive Show summary

  This paper introduces the first emotion-annotated dataset for the Dari variant of Persian spoken in Afghanistan. The LetHerLearn dataset contains 7,600 tweets posted in reaction to the Taliban’s ban of women’s rights to education in 2022 and has been manually annotated according to Ekman’s emotion categories. We here detail the data collection and annotation process, present relevant dataset statistics as well as initial experiments on the resulting dataset, benchmarking a number of different neural architectures for the task of Dari emotion classification.

• You, Huiling; Touileb, Samia & Øvrelid, Lilja (2023). JSEEGraph: Joint Structured Event Extraction as Graph Parsing. In Palmer, Alexis & Camacho-Collados, Jose (Ed.), Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023). Association for Computational Linguistics (ACL). ISSN 9781959429760. doi: 10.18653/v1/2023.starsem-1.11. Full text in Research Archive Show summary

  We propose a graph-based event extraction framework JSEEGraph that approaches the task of event extraction as general graph parsing in the tradition of Meaning Representation Parsing. It explicitly encodes entities and events in a single semantic graph, and further has the flexibility to encode a wider range of additional IE relations and jointly infer individual tasks. JSEEGraph performs in an end-to-end manner via general graph parsing: (1) instead of flat sequence labelling, nested structures between entities/triggers are efficiently encoded as separate nodes in the graph, allowing for nested and overlapping entities and triggers; (2) both entities, relations, and events can be encoded in the same graph, where entities and event triggers are represented as nodes and entity relations and event arguments are constructed via edges; (3) joint inference avoids error propagation and enhances the interpolation of different IE tasks. We experiment on two benchmark datasets of varying structural complexities; ACE05 and Rich ERE, covering three languages: English, Chinese, and Spanish. Experimental results show that JSEEGraph can handle nested event structures, that it is beneficial to solve different IE tasks jointly, and that event argument extraction in particular benefits from entity extraction. Our code and models are released as open-source.

• Samuel, David; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2023). Trained on 100 million words and still in shape: BERT meets British National Corpus. In Vlachos, Andreas & Augenstein, Isabelle (Ed.), Findings of the Association for Computational Linguistics: EACL 2023. Association for Computational Linguistics (ACL). ISSN 9781959429470. p. 1954–1974. doi: 10.18653/v1/2023.findings-eacl.146. Full text in Research Archive Show summary

  While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source – the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.

• Wold, Sondre; Øvrelid, Lilja & Velldal, Erik (2023). Text-To-KG Alignment: Comparing Current Methods on Classification Tasks, Proceedings of the First Workshop on Matching From Unstructured and Structured Data (MATCHING 2023). Association for Computational Linguistics (ACL). ISSN 9781959429821. p. 1–14. Full text in Research Archive
• Ivanova, Sardana; Andreassen, Fredrik Aas; Jentoft, Matias; Wold, Sondre & Øvrelid, Lilja (2023). NorQuAD: Norwegian Question Answering Dataset. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. Full text in Research Archive
• Kolesnichenko, Larisa; Velldal, Erik & Øvrelid, Lilja (2023). Word Substitution with Masked Language Models as Data Augmentation for Sentiment Analysis. In Ilinykh, Nikolai; Morger, Felix; Dannélls, Dana; Dobnik, Simon; Megyesi, Beáta & Nivre, Joakim (Ed.), Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). Association for Computational Linguistics (ACL). ISSN 9781959429739. Full text in Research Archive Show summary

  This paper explores the use of masked language modeling (MLM) for data augmentation (DA), targeting structured sentiment analysis (SSA) for Norwegian based on a dataset of annotated reviews. Considering the limited resources for Norwegian language and the complexity of the annotation task, the aim is to investigate whether this approach to data augmentation can help boost the performance. We report on experiments with substituting words both inside and outside of sentiment annotations, and we also present an error analysis, discussing some of the potential pitfalls of using MLM-based DA for SSA, and suggest directions for future work.

• Mæhlum, Petter; Velldal, Erik & Øvrelid, Lilja (2023). A Diagnostic Dataset for Sentiment and Negation Modeling for Norwegian. In Ilinykh, Nikolai; Morger, Felix; Dannélls, Dana; Dobnik, Simon; Megyesi, Beáta & Nivre, Joakim (Ed.), Proceedings of the Second Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2023). Association for Computational Linguistics (ACL). ISSN 9781959429739. Full text in Research Archive Show summary

  Negation constitutes a challenging phenomenon for many natural language processing tasks, such as sentiment analysis (SA). In this paper we investigate the relationship between negation and sentiment in the context of Norwegian professional reviews. The first part of this paper includes a corpus study which investigates how negation is tied to sentiment in this domain, based on existing annotations. In the second part, we introduce NoReC-NegSynt, a synthetically augmented test set for negation and sentiment, to allow for a more detailed analysis of the role of negation in current neural SA models. This diagnostic test set, containing both clausal and non-clausal negation, allows for analyzing and comparing models’ abilities to treat several different types of negation. We also present a case-study, applying several neural SA models to the diagnostic data.

• Samuel, David; Kutuzov, Andrei; Touileb, Samia; Velldal, Erik; Øvrelid, Lilja & Rønningstad, Egil [Show all 8 contributors for this article] (2023). NorBench – A Benchmark for Norwegian Language Models. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. p. 618–633. Full text in Research Archive Show summary

  We present NorBench: a streamlined suite of NLP tasks and probes for evaluating Norwegian language models (LMs) on standardized data splits and evaluation metrics. We also introduce a range of new Norwegian language models (both encoder and encoder-decoder based). Finally, we compare and analyze their performance, along with other existing LMs, across the different benchmark tests of NorBench.

• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2023). Measuring normative and descriptive biases in language models using census data. In Vlachos, Andreas & Augenstein, Isabelle (Ed.), Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781959429449. Full text in Research Archive Show summary

  We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender–occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.

• Buljan, Maja; Nivre, Joakim; Oepen, Stephan & Øvrelid, Lilja (2022). A tale of four parsers: methodological reflections on diagnostic evaluation and in-depth error analysis for meaning representation parsing. Language Resources and Evaluation. ISSN 1574-020X. p. 1–28. doi: 10.1007/s10579-022-09591-7. Full text in Research Archive Show summary

  Abstract We discuss methodological choices in diagnostic evaluation and error analysis in meaning representation parsing (MRP), i.e. mapping from natural language utterances to graph-based encodings of semantic structure. We expand on a pilot quantitative study in contrastive diagnostic evaluation, inspired by earlier work in syntactic dependency parsing, and propose a novel methodology for qualitative error analysis. This two-pronged study is performed using a selection of submissions, data, and evaluation tools featured in the 2019 shared task on MRP. Our aim is to devise methods for identifying strengths and weaknesses in different broad families of parsing techniques, as well as investigating the relations between specific parsing approaches, different meaning representation frameworks, and individual linguistic phenomena—by identifying and comparing common error patterns. Our preliminary empirical results suggest that the proposed methodologies can be meaningfully applied to parsing into graph-structured target representations, as a side-effect uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.

• Rønningstad, Egil; Øvrelid, Lilja & Velldal, Erik (2022). Entity-Level Sentiment Analysis (ELSA): An Exploratory Task Survey. International Conference on Computational Linguistics (ICCL) (COLING). ISSN 1525-2477. p. 6773–6783. Full text in Research Archive Show summary

  This paper explores the task of identifying the overall sentiment expressed towards volitional entities (persons and organizations) in a document - what we refer to as Entity-Level Sentiment Analysis (ELSA). While identifying sentiment conveyed towards an entity is well researched for shorter texts like tweets, we find little to no research on this specific task for longer texts with multiple mentions and opinions towards the same entity. This lack of research would be understandable if ELSA can be derived from existing tasks and models. To assess this, we annotate a set of professional reviews for their overall sentiment towards each volitional entity in the text. We sample from data already annotated for document-level, sentence-level, and target-level sentiment in a multi-domain review corpus, and our results indicate that there is no single proxy task that provides this overall sentiment we seek for the entities at a satisfactory level of performance. We present a suite of experiments aiming to assess the contribution towards ELSA provided by document-, sentence-, and target-level sentiment analysis, and provide a discussion of their shortcomings. We show that sentiment in our dataset is expressed not only with an entity mention as target, but also towards targets with a sentiment-relevant relation to a volitional entity. In our data, these relations extend beyond anaphoric coreference resolution, and our findings call for further research of the topic. Finally, we also present a survey of previous relevant work.

• You, Huiling; Samuel, David; Touileb, Samia & Øvrelid, Lilja (2022). EventGraph: Event Extraction as Semantic Graph Parsing, The 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Association for Computational Linguistics (ACL). ISSN 9781959429050. p. 7–15. Full text in Research Archive
• You, Huiling; Samuel, David; Touileb, Samia & Øvrelid, Lilja (2022). EventGraph at CASE 2021 Task 1: A General Graph-based Approach to Protest Event Extraction, The 5th Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text. Association for Computational Linguistics (ACL). ISSN 9781959429050. p. 155–160. Full text in Research Archive
• Samuel, David; Barnes, Jeremy; Kurtz, Robin; Oepen, Stephan; Øvrelid, Lilja & Velldal, Erik (2022). Direct parsing to sentiment graphs. Association for Computational Linguistics (ACL). Annual Meeting Conference Proceedings. ISSN 0736-587X. 2(60), p. 470–478. doi: 10.18653/v1/2022.acl-short.51. Full text in Research Archive
• Kutuzov, Andrei; Velldal, Erik & Øvrelid, Lilja (2022). Contextualized embeddings for semantic change detection: Lessons learned. Northern European Journal of Language Technology (NEJLT). ISSN 2000-1533. 8(1). doi: 10.3384/nejlt.2000-1533.2022.3478. Full text in Research Archive Show summary

  We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.

• Barnes, Jeremy; Oberlaender, Laura; Troiano, Enrica; Kutuzov, Andrei; Buchmann, Jan & Agerri, Rodrigo [Show all 8 contributors for this article] (2022). SemEval 2022 Task 10: Structured Sentiment Analysis. In Emerson, Guy; Schluter, Natalie; Stanovsky, Gabriel; Kumar, Ritesh; Palmer, Alexis; Schneider, Nathan; Singh, Siddarth & Ratan, Shyam (Ed.), Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). Association for Computational Linguistics (ACL). ISSN 9781955917803. p. 1280–1295. Full text in Research Archive
• Mæhlum, Petter; Haug, Dag Trygve Truslew; Jørgensen, Tollef Emil; Kåsen, Andre; Nøklestad, Anders & Rønningstad, Egil [Show all 9 contributors for this article] (2022). NARC – Norwegian Anaphora Resolution Corpus. International Conference on Computational Linguistics (ICCL) (COLING). ISSN 1525-2477. 29(7), p. 48–60. Full text in Research Archive Show summary

  Published in: Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC): https://aclanthology.org/venues/coling/. We present the Norwegian Anaphora Resolution Corpus (NARC), the first publicly available corpus annotated with anaphoric relations between noun phrases for Norwegian. The paper describes the annotated data for 326 documents in Norwegian Bokmål, together with inter-annotator agreement and discussions of relevant statistics. We also present preliminary modelling results which are comparable to existing corpora for other languages, and discuss relevant problems in relation to both modelling and the annotations themselves.

• Pilán, Ildikó; Lison, Pierre; Øvrelid, Lilja; Papadopoulou, Anthi; Sánchez, David & Batet, Montserrat (2022). The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics. ISSN 0891-2017. 48(4), p. 1053–1101. doi: 10.1162/coli_a_00458. Full text in Research Archive Show summary

  We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared with previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored toward measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts, and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymization-benchmark.

• Papadopoulou, Anthi; Yu, Yunhao; Lison, Pierre & Øvrelid, Lilja (2022). Neural Text Sanitization with Explicit Measures of Privacy Risk, The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781955917650. p. 217–229. Full text in Research Archive Show summary

  We present a novel approach for text sanitization, which is the task of editing a document to mask all (direct and indirect) personal identifiers and thereby conceal the identity of the individuals(s) mentioned in the text. In contrast to previous work, the approach relies on explicit measures of privacy risk, making it possible to explicitly control the trade-off between privacy protection and data utility. The approach proceeds in three steps. A neural, privacy-enhanced entity recognizer is first employed to detect and classify potential personal identifiers. We then determine which entities, or combination of entities, are likely to pose a re-identification risk through a range of privacy risk assessment measures. We present three such measures of privacy risk, respectively based on (1) span probabilities derived from a BERT language model, (2) web search queries and (3) a classifier trained on labelled data. Finally, a linear optimization solver decides which entities to mask to minimize the semantic loss while simultaneously ensuring that the estimated privacy risk remains under a given threshold. We evaluate the approach both in the absence and presence of manually annotated data. Our results highlight the potential of the approach, as well as issues specific types of personal data can introduce to the process.

• Papadopoulou, Anthi; Lison, Pierre; Øvrelid, Lilja & Pilán, Ildikó (2022). Bootstrapping Text Anonymization Models with Distant Supervision. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546726. p. 4477–4487. Full text in Research Archive Show summary

  We propose a novel method to bootstrap text anonymization models based on distant supervision. Instead of requiring manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee k-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a RoBERTa-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise if the knowledge graph is noisy or incomplete. The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.

• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2022). Occupational Biases in Norwegian and Multilingual Language Models. In Christine, Basta; R., Costa-jussà Marta; Hila, Gonen; Christian, Hardmeier & Gabriel, Stanovsky (Ed.), Proceedings of The 4th Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781955917681. p. 200–211. doi: 10.18653/v1/2022.gebnlp-1.21. Full text in Research Archive Show summary

  In this paper we explore how a demographic distribution of occupations, along gender dimensions, is reflected in pre-trained language models. We give a descriptive assessment of the distribution of occupations, and investigate to what extent these are reflected in four Norwegian and two multilingual models. To this end, we introduce a set of simple bias probes, and perform five different tasks combining gendered pronouns, first names, and a set of occupations from the Norwegian statistics bureau. We show that language specific models obtain more accurate results, and are much closer to the real-world distribution of clearly gendered occupations. However, we see that none of the models have correct representations of the occupations that are demo-graphically balanced between genders. We also discuss the importance of the training data on which the models were trained on, and argue that template-based bias probes can some-times be fragile, and a simple alteration in a template can change a model’s behavior.

• Kutuzov, Andrei; Barnes, Jeremy; Velldal, Erik; Øvrelid, Lilja & Oepen, Stephan (2021). Large-Scale Contextualised Language Modelling for Norwegian. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 30–40. Full text in Research Archive Show summary

  We present the ongoing NorLM initiative to support the creation and use of very large contextualised language models for Norwegian (and in principle other Nordic languages), including a ready-to-use software environment, as well as an experience report for data preparation and training. This paper introduces the first large-scale monolingual language models for Norwegian, based on both the ELMo and BERT frameworks. In addition to detailing the training process, we present contrastive benchmark results on a suite of NLP tasks for Norwegian. For additional background and access to the data, models, and software, please see: http://norlm.nlpl.eu

• Ravishankar, Vinit; Kutuzov, Andrei; Øvrelid, Lilja & Velldal, Erik (2021). Multilingual ELMo and the Effects of Corpus Sampling. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 378–383. Full text in Research Archive
• Mæhlum, Petter; Barnes, Jeremy; Kurtz, Robin; Øvrelid, Lilja & Velldal, Erik (2021). Negation in Norwegian: an annotated dataset. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 299–308. Full text in Research Archive
• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2021). Using Gender- and Polarity-Informed Models to Investigate Bias. In Costa-jussà, Marta R.; Gonen, Hila; Hardmeier, Christian & Webster, Kellie (Ed.), Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781954085619. p. 66–74. doi: 10.18653/v1/2021.gebnlp-1.8. Full text in Research Archive
• Lison, Pierre; Pilán, Ildikó; Sánchez, David; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9781954085527. p. 4188–4203. doi: 10.18653/v1/2021.acl-long.323. Full text in Research Archive Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Laake, Signe & Øvrelid, Lilja (2021). Forskjeller mellom talemål og skriftspråk: Hva kan trebanker fortelle oss? In Hagen, Kristin; Kristoffersen, Gjert; Vangsnes, Øystein Alexander & Åfarli, Tor Anders (Ed.), Språk i arkiva. Ny forsking om eldre talemål frå LIA-prosjektet. Novus Forlag. ISSN 9788283900811. p. 65–86. Full text in Research Archive
• Barnes, Jeremy; Kurtz, Robin; Oepen, Stephan; Øvrelid, Lilja & Velldal, Erik (2021). Structured Sentiment Analysis as Dependency Graph Parsing. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9781954085527. p. 3387–3402. doi: 10.18653/v1/2021.acl-long.263. Full text in Research Archive
• Barnes, Jeremy; Øvrelid, Lilja & Velldal, Erik (2021). If you've got it, flaunt it: Making the most of fine-grained sentiment annotations. In Merlo, Paola; Tiedemann, Jörg & Tsarfaty, Reut (Ed.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781954085022. p. 49–62. Full text in Research Archive
• Dahl, Fredrik Andreas; Kasicheyanula, Taraka Rama; Hurlen, Petter; Brekke, Pål Haugar; Husby, Haldor & Gundersen, Tore [Show all 8 contributors for this article] (2021). Neural classification of Norwegian radiology reports: using NLP to detect findings in CT-scans of children. BMC Medical Informatics and Decision Making. 21(1), p. 1–8. doi: 10.1186/s12911-021-01451-8. Full text in Research Archive Show summary

  Background With a motivation of quality assurance, machine learning techniques were trained to classify Norwegian radiology reports of paediatric CT examinations according to their description of abnormal findings. Methods 13.506 reports from CT-scans of children, 1000 reports from CT scan of adults and 1000 reports from X-ray examination of adults were classified as positive or negative by a radiologist, according to the presence of abnormal findings. Inter-rater reliability was evaluated by comparison with a clinician’s classifications of 500 reports. Test–retest reliability of the radiologist was performed on the same 500 reports. A convolutional neural network model (CNN), a bidirectional recurrent neural network model (bi-LSTM) and a support vector machine model (SVM) were trained on a random selection of the children’s data set. Models were evaluated on the remaining CT-children reports and the adult data sets. Results Test–retest reliability: Cohen’s Kappa = 0.86 and F1 = 0.919. Inter-rater reliability: Kappa = 0.80 and F1 = 0.885. Model performances on the Children-CT data were as follows. CNN: (AUC = 0.981, F1 = 0.930), bi-LSTM: (AUC = 0.978, F1 = 0.927), SVM: (AUC = 0.975, F1 = 0.912). On the adult data sets, the models had AUC around 0.95 and F1 around 0.91. Conclusions The models performed close to perfectly on its defined domain, and also performed convincingly on reports pertaining to a different patient group and a different modality. The models were deemed suitable for classifying radiology reports for future quality assurance purposes, where the fraction of the examinations with abnormal findings for different sub-groups of patients is a parameter of interest.

• Brekke, Pål Haugar; Rama, Taraka; Pilán, Ildikó; Nytrø, Øystein & Øvrelid, Lilja (2021). Synthetic data for annotation and extraction of family history information from clinical text. Journal of Biomedical Semantics. 12(11). doi: 10.1186/s13326-021-00244-2. Full text in Research Archive Show summary

  Background The limited availability of clinical texts for Natural Language Processing purposes is hindering the progress of the field. This article investigates the use of synthetic data for the annotation and automated extraction of family history information from Norwegian clinical text. We make use of incrementally developed synthetic clinical text describing patients’ family history relating to cases of cardiac disease and present a general methodology which integrates the synthetically produced clinical statements and annotation guideline development. The resulting synthetic corpus contains 477 sentences and 6030 tokens. In this work we experimentally assess the validity and applicability of the annotated synthetic corpus using machine learning techniques and furthermore evaluate the system trained on synthetic text on a corpus of real clinical text, consisting of de-identified records for patients with genetic heart disease. Results For entity recognition, an SVM trained on synthetic data had class weighted precision, recall and F1-scores of 0.83, 0.81 and 0.82, respectively. For relation extraction precision, recall and F1-scores were 0.74, 0.75 and 0.74. Conclusions A system for extraction of family history information developed on synthetic data generalizes well to real, clinical notes with a small loss of accuracy. The methodology outlined in this paper may be useful in other situations where limited availability of clinical text hinders NLP tasks. Both the annotation guidelines and the annotated synthetic corpus are made freely available and as such constitutes the first publicly available resource of Norwegian clinical text.

• Trattner, Christoph; Jannach, Dietmar; Motta, Enrico; Meijer, Irene Costera; Diakopoulos, Nicholas & Elahi, Mehdi [Show all 13 contributors for this article] (2021). Responsible media technology and AI: challenges and research directions. AI and Ethics. ISSN 2730-5953. doi: 10.1007/s43681-021-00126-4. Full text in Research Archive Show summary

  The last two decades have witnessed major disruptions to the traditional media industry as a result of technological breakthroughs. New opportunities and challenges continue to arise, most recently as a result of the rapid advance and adoption of artificial intelligence technologies. On the one hand, the broad adoption of these technologies may introduce new opportunities for diversifying media offerings, fighting disinformation, and advancing data-driven journalism. On the other hand, techniques such as algorithmic content selection and user personalization can introduce risks and societal threats. The challenge of balancing these opportunities and benefits against their potential for negative impacts underscores the need for more research in responsible media technology. In this paper, we first describe the major challenges—both for societies and the media industry—that come with modern media technology. We then outline various places in the media production and dissemination chain, where research gaps exist, where better technical approaches are needed, and where technology must be designed in a way that can effectively support responsible editorial processes and principles. We argue that a comprehensive approach to research in responsible media technology, leveraging an interdisciplinary approach and a close cooperation between the media industry and academic institutions, is urgently needed.

• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2020). Gender and sentiment, critics and authors: a dataset of Norwegian book reviews. In Costa-jussà, Marta R.; Hardmeier, Christian; Radford, Will & Webster, Kellie (Ed.), Proceedings of the Second Workshop on Gender Bias in Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781952148439. p. 125–138. Full text in Research Archive Show summary

  Gender bias in models and datasets is widely studied in NLP. The focus has usually been on analysing how females and males express themselves, or how females and males are described. However, a less studied aspect is the combination of these two perspectives, how female and male describe the same or opposite gender. In this paper, we present a new gender annotated sentiment dataset of critics reviewing the works of female and male authors. We investigate if this newly annotated dataset contains differences in how the works of male and female authors are critiqued, in particular in terms of positive and negative sentiment. We also explore the differences in how this is done by male and female critics. We show that there are differences in how critics assess the works of authors of the same or opposite gender. For example, male critics rate crime novels written by females, and romantic and sentimental works written by males, more negatively.

• Pilán, Ildikó; Brekke, Pål Haugar & Øvrelid, Lilja (2020). Building a Norwegian Lexical Resource for Medical Entity Recognition. In Melero, Maite (Eds.), Proceedings of the LREC 2020 Workshop on Multilingual Biomedical Text Processing (MultiligualBIO 2020). European Language Resources Association. ISSN 9791095546658. p. 9–14. Full text in Research Archive Show summary

  We present a large Norwegian lexical resource of categorized medical terms. The resource, which merges information from large medical databases, contains over 56,000 entries, including automatically mapped terms from a Norwegian medical dictionary. We describe the methodology behind this automatic dictionary entry mapping based on keywords and suffixes and further present the results of a manual evaluation performed on a subset by a domain expert. The evaluation indicated that ca. 80% of the mappings were correct.

• Buljan, Maja; Nivre, Joakim; Oepen, Stephan & Øvrelid, Lilja (2020). A tale of three parsers: Towards diagnostic evaluation for meaning representation parsing. In Calzolari, Nicoletta (Eds.), Proceedings of the 12th Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546344. p. 1902–1909. Full text in Research Archive Show summary

  We discuss methodological choices in contrastive and diagnostic evaluation in meaning representation parsing, i.e. mapping from natural language utterances to graph-based encodings of its semantic structure. Drawing inspiration from earlier work in syntactic dependency parsing, we transfer and refine several quantitative diagnosis techniques for use in the context of the 2019 shared task on Meaning Representation Parsing (MRP). As in parsing proper, moving evaluation from simple rooted trees to general graphs brings along its own range of challenges. Specifically, we seek to begin to shed light on relative strenghts and weaknesses in different broad families of parsing techniques. In addition to these theoretical reflections, we conduct a pilot experiment on a selection of top-performing MRP systems and one of the five meaning representation frameworks in the shared task. Empirical results suggest that the proposed methodology can be meaningfully applied to parsing into graph-structured target representations, uncovering hitherto unknown properties of the different systems that can inform future development and cross-fertilization across approaches.

• Øvrelid, Lilja; Mæhlum, Petter; Barnes, Jeremy & Velldal, Erik (2020). A Fine-Grained Sentiment Dataset for Norwegian. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of The 12th Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546344. Full text in Research Archive
• Barnes, Jeremy Claude; Velldal, Erik & Øvrelid, Lilja (2020). Improving Sentiment Analysis with Multi-task Learning of Negation. Natural Language Engineering. ISSN 1351-3249. 1(21). doi: 10.1017/S1351324920000510. Full text in Research Archive Show summary

  Sentiment analysis is directly affected by compositional phenomena in language that act on the prior polarity of the words and phrases found in the text. Negation is the most prevalent of these phenomena and in order to correctly predict sentiment, a classifier must be able to identify negation and disentangle the effect that its scope has on the final polarity of a text. This paper proposes a multi-task approach to explicitly incorporate information about negation in sentiment analysis, which we show outperforms learning negation implicitly in a data-driven manner. We describe our approach, a cascading neural architecture with selective sharing of LSTM layers, and show that explicitly training the model with negation as an auxiliary task helps improve the main task of sentiment analysis. The effect is demonstrated across several different standard English-language data sets for both tasks and we analyze several aspects of our system related to its performance, varying types and amounts of input data and different multi-task setups.

• Jørgensen, Fredrik; Aasmoe, Tobias; Husevåg, Anne-Stine Ruud; Øvrelid, Lilja & Velldal, Erik (2020). NorNE: Annotating Named Entities for Norwegian. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of The 12th Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546344. p. 4547–4556. Full text in Research Archive Show summary

  This paper presents NorNE, a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names. We here present details on the annotation effort, guidelines, inter-annotator agreement and an experimental analysis of the corpus using a neural sequence labeling architecture.

• Pilán, Ildikó; Brekke, Pål Haugar; Dahl, Fredrik Andreas; Gundersen, Tore; Husby, Haldor & Nytrø, Øystein [Show all 7 contributors for this article] (2020). Classification of Syncope Cases in Norwegian Medical Records. In Rumshisky, Anna; Roberts, Kirk; Bethard, Steven & Naumann, Tristan (Ed.), Proceedings of the 3rd Clinical Natural Language Processing Workshop. Association for Computational Linguistics (ACL). ISSN 9781952148743. p. 79–84. doi: 10.18653/v1/2020.clinicalnlp-1.9. Full text in Research Archive Show summary

  Loss of consciousness, so-called syncope, is a commonly occurring symptom associated with worse prognosis for a number of heart-related diseases. We present a comparison of methods for a diagnosis classification task in Norwegian clinical notes, targeting syncope, i.e. fainting cases. We find that an often neglected baseline with keyword matching constitutes a rather strong basis, but more advanced methods do offer some improvement in classification performance, especially a convolutional neural network model. The developed pipeline is planned to be used for quantifying unregistered syncope cases in Norway.

• Mæhlum, Petter; Barnes, Jeremy Claude; Øvrelid, Lilja & Velldal, Erik (2019). Annotating evaluative sentences for sentiment analysis: a dataset for Norwegian. In Hartmann, Mareike & Plank, Barbara (Ed.), Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179299958. p. 121–130. Full text in Research Archive Show summary

  This paper documents the creation of a large-scale dataset of evaluative sentences – i.e. both subjective and objective sentences that are found to be sentiment-bearing – based on mixed-domain professional reviews from various news-sources. We present both the annotation scheme and first results for classification experiments. The effort represents a step toward creating a Norwegian dataset for fine-grained sentiment analysis.

• Barnes, Jeremy Claude; Øvrelid, Lilja & Velldal, Erik (2019). Sentiment analysis is not solved! Assessing and probing sentiment classification. In Linzen, Tal; Chrupała, Grzegorz; Belinkov, Yonatan & Hupkes, Dieuwke (Ed.), The BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP at ACL 2019: Proceedings of the Second Workshop. Association for Computational Linguistics (ACL). ISSN 9781950737307. p. 12–23. doi: 10.18653/v1/w19-4802. Full text in Research Archive Show summary

  Neural methods for sentiment analysis have led to quantitative improvements over previous approaches, but these advances are not always accompanied with a thorough analysis of the qualitative differences. Therefore, it is not clear what outstanding conceptual challenges for sentiment analysis remain. In this work, we attempt to discover what challenges still prove a problem for sentiment classifiers for English and to provide a challenging dataset. We collect the subset of sentences that an (oracle) ensemble of state-of-the-art sentiment classifiers misclassify and then annotate them for 18 linguistic and paralinguistic phenomena, such as negation, sarcasm, modality, etc. Finally, we provide a case study that demonstrates the usefulness of the dataset to probe the performance of a given sentiment classifier with respect to linguistic phenomena.

• Kutuzov, Andrei; Velldal, Erik & Øvrelid, Lilja (2019). One-to-X Analogical Reasoning on Word Embeddings: a Case for Diachronic Armed Conflict Prediction from News Texts. In Tahmasebi, Nina; Borin, Lars; Jatowt, Adam & Xu, Yang (Ed.), Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change. Association for Computational Linguistics (ACL). ISSN 9781950737314. p. 196–201. doi: 10.18653/v1/W19-4724. Full text in Research Archive Show summary

  We extend the well-known word analogy task to a one-to-X formulation, including one-to-none cases, when no correct answer exists. The task is cast as a relation discovery problem and applied to historical armed conflicts datasets, attempting to predict new relations of type ‘location:armed-group’ based on data about past events. As the source of semantic information, we use diachronic word embedding models trained on English news texts. A simple technique to improve diachronic performance in such task is demonstrated, using a threshold based on a function of cosine distance to decrease the number of false positives; this approach is shown to be beneficial on two different corpora. Finally, we publish a ready-to-use test set for one-to-X analogy evaluation on historical armed conflicts data.

• Barnes, Jeremy Claude; Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2019). Lexicon information in neural sentiment analysis: a multi-task learning approach. In Hartmann, Mareike & Plank, Barbara (Ed.), Proceedings of the 22nd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179299958. p. 175–186. Full text in Research Archive Show summary

  This paper explores the use of multi-task learning (MTL) for incorporating external knowledge in neural models. Specifically, we show how MTL can enable a BiLSTM sentiment classifier to incorporate information from sentiment lexicons. Our MTL set-up is shown to improve model performance (compared to a single-task set-up) on both English and Norwegian sentence-level sentiment datasets. The paper also introduces a new sentiment lexicon for Norwegian.

• Berggren, Stig Johan; Rama, Taraka & Øvrelid, Lilja (2019). Regression or classification? Automated Essay Scoring for Norwegian. In Yannakoudakis, Helen; Kochmar, Ekaterina; Leacock, Claudia; Madnani, Nitin; Pilán, Ildikó & Zesch, Torsten (Ed.), Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications. Association for Computational Linguistics (ACL). ISSN 9781950737345. p. 92–102. Full text in Research Archive
• Nooralahzadeh, Farhad; Lønning, Jan Tore & Øvrelid, Lilja (2019). Reinforcement-based denoising of distantly supervised NER with partial annotation, Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019). Association for Computational Linguistics (ACL). ISSN 9781950737789. p. 225–234. doi: 10.18653/v1/D19-6125. Full text in Research Archive Show summary

  Existing named entity recognition (NER) systems rely on large amounts of human-labeled data for supervision. However, obtaining large-scale annotated data is challenging particularly in specific domains like health-care, e-commerce and so on. Given the availability of domain specific knowledge resources, (e.g., ontologies, dictionaries), distant supervision is a solution to generate automatically labeled training data to reduce human effort. The outcome of distant supervision for NER, however, is often noisy. False positive and false negative instances are the main issues that reduce performance on this kind of auto-generated data. In this paper, we explore distant supervision in a supervised setup. We adopt a technique of partial annotation to address false negative cases and implement a reinforcement learning strategy with a neural network policy to identify false positive instances. Our results establish a new state-of-the-art on four benchmark datasets taken from different domains and different languages. We then go on to show that our model reduces the amount of manually annotated data required to perform NER in a new domain.

• Ravishankar, Vinit; Gökirmak, Memduh; Øvrelid, Lilja & Velldal, Erik (2019). Multilingual Probing of Deep Pre-Trained Contextual Encoders. In Nivre, Joakim; Derczynski, Leon; Ginter, Filip; Lindi, Bjørn; Oepen, Stephan; Søgaard, Anders & Tidemann, Jorg (Ed.), Proceedings of the First NLPL Workshop on Deep Learning for Natural Language Processing. Linköping University Electronic Press. ISSN 9789179299996. p. 37–47. Full text in Research Archive
• Ravishankar, Vinit; Øvrelid, Lilja & Velldal, Erik (2019). Probing Multilingual Sentence Representations With X-Probe. In Augenstein, Isabelle; Gella, Spandana; Ruder, Sebastian; Can, Burcu; Welbl, Johannes; Ren, Xiang & Rei, Marek (Ed.), The 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) : Proceedings of the Workshop. Association for Computational Linguistics (ACL). ISSN 9781950737352. p. 156–168. doi: 10.18653/v1/W19-4318. Full text in Research Archive Show summary

  This paper extends the task of probing sentence representations for linguistic insight in a multilingual domain. In doing so, we make two contributions: first, we provide datasets for multilingual probing, derived from Wikipedia, in five languages, viz. English, French, German, Spanish and Russian. Second, we evaluate six sentence encoders for each language, each trained by mapping sentence representations to English sentence representations, using sentences in a parallel corpus. We discover that cross-lingually mapped representations are often better at retaining certain linguistic information than representations derived from English encoders trained on natural language inference (NLI) as a downstream task.

• Hammer, Hugo Lewi; Riegler, Michael Alexander; Øvrelid, Lilja & Velldal, Erik (2019). THREAT: A Large Annotated Corpus for Detection of Violent Threats, Proceedings of Content Based Multimedia Information (CBMI 2019). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781728146737. p. 1–5. doi: 10.1109/CBMI.2019.8877435. Full text in Research Archive Show summary

  Understanding, detecting, moderating and in extreme cases deleting hateful comments in online discussions and social media are well-known challenges. In this paper we present a dataset consisting of a total of around 30000 sentences from around 10000 YouTube comments. Each sentence is manually annotated as either being a violent threat or not. Violent threats is the most extreme form of hateful communication and is of particular importance from an online radicalization and national security perspective. This is the first publicly available dataset with such an annotation. The dataset can further be useful to develop automatic moderation tools or may even be useful from a social science perspective for analyzing the characteristics of online threats and how hateful discussions evolve.

• Kasicheyanula, Taraka Rama; Brekke, Pål Haugar; Nytrø, Øystein & Øvrelid, Lilja (2018). Iterative development of family history annotation guidelines using a synthetic corpus of clinical text, Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis (LOUHI 2018). Association for Computational Linguistics (ACL). ISSN 9781948087742. p. 111–121. doi: 10.18653/v1/w18-5613. Full text in Research Archive
• Bouma, Gosse; Hajic, Jan; Haug, Dag Trygve Truslew; Nivre, Joakim; Solberg, Per Erik & Øvrelid, Lilja (2018). Expletives in Universal Dependency Treebanks, Proceedings of the Second Workshop on Universal Dependencies (UDW 2018). Association for Computational Linguistics (ACL). ISSN 9781948087780. p. 18–26. doi: 10.18653/v1/w18-6003. Full text in Research Archive
• Nooralahzadeh, Farhad & Øvrelid, Lilja (2018). SIRIUS-LTG: An Entity Linking Approach to Fact Extraction and Verification, Proceedings of the First Workshop on Fact Extraction and VERification (FEVER). Association for Computational Linguistics (ACL). ISSN 9781948087735. p. 119–123. doi: 10.18653/v1/w18-5519. Full text in Research Archive
• Øvrelid, Lilja; Kåsen, Andre; Hagen, Kristin; Solberg, Per Erik; Johannessen, Janne Bondi & Nøklestad, Anders (2018). The LIA Treebank of Spoken Norwegian Dialects. In Calzolari, Nicoletta; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Hasida, Koiti; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan; Piperidis, Stelios & Tokunaga, Takenobu (Ed.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association. ISSN 9791095546009. p. 4482–4488. Full text in Research Archive Show summary

  This article presents the LIA treebank of transcribed spoken Norwegian dialects. It consists of dialect recordings made in the period between 1950--1990, which have been digitised, transcribed, and subsequently annotated with morphological and dependency-style syntactic analysis as part of the LIA (Language Infrastructure made Accessible) project at the University of Oslo. In this article, we describe the LIA material of dialect recordings and its transcription, transliteration and further morphosyntactic annotation. We focus in particular on the extension of the native NDT annotation scheme to spoken language phenomena, such as pauses and various types of disfluencies, and present the subsequent conversion of the treebank to the Universal Dependencies scheme. The treebank currently consists of 13,608 tokens, distributed over 1396 segments taken from three different dialects of spoken Norwegian. The LIA treebank annotation is an on-going effort and future releases will extend on the current data set.

• Nooralahzadeh, Farhad; Øvrelid, Lilja & Lønning, Jan Tore (2018). Evaluation of Domain-specific Word Embeddings using Knowledge Resources. In Calzolari, Nicoletta; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Hasida, Koiti; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan; Piperidis, Stelios & Tokunaga, Takenobu (Ed.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association. ISSN 9791095546009. p. 1438–1445. Full text in Research Archive Show summary

  In this work we evaluate domain-specific embedding models induced from textual resources in the Oil and Gas domain. We conduct intrinsic and extrinsic evaluations of both general and domain-specific embeddings and we observe that constructing domain-specific word embeddings is worthwhile even with a considerably smaller corpus size. Although the intrinsic evaluation shows low performance in synonymy detection, an in-depth error analysis reveals the ability of these models to discover additional semantic relations such as hyponymy, co-hyponymy and relatedness in the target domain. Extrinsic evaluation of the embedding models is provided by a domain-specific sentence classification task, which we solve using a convolutional neural network. We further adapt embedding enhancement methods to provide vector representations for infrequent and unseen terms. Experiments show that the adapted technique can provide improvements both in intrinsic and extrinsic evaluation.

• Fares, Murhaf; Oepen, Stephan; Øvrelid, Lilja; Björne, Jari & Johansson, Richard (2018). The 2018 shared task on extrinsic parser evaluation: On the downstream utility of English universal dependency parsers. In Zeman, Daniel & Hajic, Jan (Ed.), Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Association for Computational Linguistics (ACL). ISSN 9781948087827. p. 22–33. doi: 10.18653/v1/K18-2002. Full text in Research Archive Show summary

  We summarize empirical results and tentative conclusions from the Second Extrinsic Parser Evaluation Initiative (EPE 2018). We review the basic task setup, downstream applications involved, and end-to-end results for seventeen participating parsers. Based on both quantitative and qualitative analysis, we correlate intrinsic evaluation results at different layers of morphsyntactic analysis with observed downstream behavior.

• Kutuzov, Andrei; Øvrelid, Lilja; Szymanski, Terrence & Velldal, Erik (2018). Diachronic word embeddings and semantic shifts: a survey, Proceedings of the 27th International Conference on Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781948087506. p. 1384–1397. Full text in Research Archive Show summary

  Recent years have witnessed a surge of publications aimed at tracing temporal changes in lexical semantics using distributional methods, particularly prediction-based word embedding models. However, this vein of research lacks the cohesion, common terminology and shared practices of more established areas of natural language processing. In this paper, we survey the current state of academic research related to diachronic word embeddings and semantic shifts detection. We start with discussing the notion of semantic shifts, and then continue with an overview of the existing methods for tracing such time-related shifts with word embedding models. We propose several axes along which these methods can be compared, and outline the main challenges before this emerging subfield of NLP, as well as prospects and possible applications.

View all works in NVA

• Dobnik, Simon & Øvrelid, Lilja (2021). Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISBN 9789179296148. 467 p. Full text in Research Archive

View all works in NVA

• Grødeland, Gunnveig; Knutsen, Carl Henrik; MOENE, KALLE & Øvrelid, Lilja (2025). Panelsamtale: 80 Years of Democracy – Is There Hope for Another 80? Full text in Research Archive
• Norman, Rebecka Maria; Jelin, Elma; Mæhlum, Petter; Velldal, Erik; Øvrelid, Lilja & Bjertnæs, Øyvind Andresen (2024). From words to action: development of a machine learning model using natural language processing to automatically turn free-text patient comments into insights for quality improvement. Full text in Research Archive
• Øvrelid, Lilja (2024). Norwegian Large Language Models: current status and the road ahead. Full text in Research Archive
• Velldal, Erik; Walsø, Claes Lyth & Øvrelid, Lilja (2021). Nå kan maskinene tolke hva du føler og mener. [Internet]. forskning,no. Full text in Research Archive
• Velldal, Erik; Walsø, Claes Lyth & Øvrelid, Lilja (2021). Nå er maskinene klare til å tolke hva du føler og mener. [Internet]. titan.uio.no. Full text in Research Archive
• Lison, Pierre; Pilán, Ildikó; Øvrelid, Lilja; Ruenes, David Sánchez & Batet, Montserrat (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Full text in Research Archive Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Lison, Pierre; Pilán, Ildikó; Ruenes, David Sánchez; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Full text in Research Archive
• Touileb, Samia; Øvrelid, Lilja & Velldal, Erik (2021). Using Gender- and Polarity-informed Models to Investigate Bias. Full text in Research Archive
• Pilán, Ildikó; Brekke, Pål H. & Øvrelid, Lilja (2020). Building a Norwegian Lexical Resource for Medical Entity Recognition. Full text in Research Archive Show summary

  We present a large Norwegian lexical resource of categorized medical terms. The resource, which merges information from large medical databases, contains over 56,000 entries, including automatically mapped terms from a Norwegian medical dictionary. We describe the methodology behind this automatic dictionary entry mapping based on keywords and suffixes and further present the results of a manual evaluation performed on a subset by a domain expert. The evaluation indicated that ca. 80% of the mappings were correct.

• Dahl, Fredrik Andreas; Kasicheyanula, Taraka Rama; Hurlen, Petter; Brekke, Pål Haugar; Husby, Haldor & Gundersen, Tore [Show all 8 contributors for this article] (2019). Classifying Norwegian radiology reports with deep learning. Full text in Research Archive
• Øvrelid, Lilja (2019). Forskjeller mellom talemål og skrift. Hva kan trebanker fortelle oss? Full text in Research Archive
• Samuel, David; Kutuzov, Andrey; Øvrelid, Lilja & Velldal, Erik (2026). Data-Constrained Language Modeling. Universitetet i Oslo. Full text in Research Archive Show summary

  Language model performance scales with training data, yet the reservoir of human-written text is finite– a constraint that already limits most languages and will eventually affect even English as models continue to scale. This thesis addresses data constraints across the entire life cycle of language models: we develop architectural improvements and training methods that maximize learning from limited and repeated data, and we propose post-training methods that enable fluency-preserving preference optimization without native instruction-tuning datasets.

• Rønningstad, Egil & Øvrelid, Lilja (2020). Targeted Sentiment Analysis for Norwegian text. Universitetet i Oslo. Full text in Research Archive Show summary

  Targeted Sentiment Analysis attempts to extract sentiment targets and the sentiment polarity towards these targets, as explicitly expressed in text. Targeted Sentiment Analysis is a difficult task where there may be multiple sentiment targets in one sentence, and there may be conflicting sentiments towards one target. Sentiment may be expressed through nuances and combinations of words at different positions in the sentence. State-of-the-art models for Targeted Sentiment Analysis therefore require large amount of data. In our thesis we explore approaches to Targeted Sentiment Analysis in scenarios where a) we have a large annotated dataset, b) we have a very limited amount of annotated data, and c) we have no annotated data for the target language and domain. Given a large monolingual dataset, we provide a state-of-the art model through the multilingual BERT (M-BERT) pretrained language model. Given more limited data we show how bilingual training data allows for noteworthy improvements over monolingual training. Given a scenario with no labeled data for the target domain and language, we demonstrate the cross-lingual performance of M-BERT for the Norwegian and English language pair. We isolate and compare the effect of domain and language differences, and demonstrate the option of machine-translating text for Targeted Sentiment Analysis.

View all works in NVA