.jpeg)
The language technology group research seminar is a biweekly event. The regular time slot for the seminar is Tuesdays, 11:15 - 12:00 CEST. Talks usually consist of a 30-minute presentation and 15 minutes of discussion with the audience.
The seminar is conducted in a hybrid form: both offline at Ole-Johan Dahls hus, UiO, and online in Zoom (link available by request). With questions or suggestions related to the LTG seminar, please contact Hannah Clausen or Andrey Kutuzov.
Forthcoming talks
November 3rd, 2026
Topic: TBA
Mikita Pilinka (UiO)
Room: Styrerom 4118
Past talks
LTG research seminar has a long history, but this page starts from Fall 2021 only.
October 20th, 2026
Topic: Risks in the Wild: Mapping AI Agent Posts to a Risk Taxonomy
Annika Willoch Olstad (Simula, LTG)
Room: Styrerom 4118
AI agents and agent social networks are evolving rapidly, raising questions about risks that we may not yet fully understand. What happens when agents interact at scale, and what new dangers might emerge? In this project, we aim to explore these questions by mapping agent-generated text against established AI risk taxonomies, using the Moltbook Observatory Archive and the MIT AI Risk Repository as starting points. Our approach is to identify known risks while looking for patterns that existing categories may miss. Ultimately, we hope to explore how current risk frameworks can help us understand emerging agentic risks, and where they may fall short. In this talk, we present the motivation behind the project, our initial starting points, and the approach we are beginning to explore.
October 1st, 2026 (Thursday, 12:15 - 13:00)
Topic: Interpretability of Language Models across Abstraction Levels
Oliver Eberle (TU Berlin)
Room: Styrerom 4118
How do modern AI systems, such as language models, solve complex reasoning tasks, and how do they organize the underlying computational processes? This talk provides an overview of my research on explainable AI and mechanistic interpretability methods for (i) uncovering internal model mechanisms, (ii) evaluating and aligning the strategies they implement, and ultimately (iii) steering their capabilities and behaviors. By studying model mechanisms across multiple levels of abstraction, this research connects input features, internal representations and feature descriptions, feature interactions, and algorithmic primitives that emerge in language models. Linking these levels of analysis offers insights into how systems process information, generalize across tasks, and produce outputs like multi-step reasoning in solving the Traveling Salesperson Problem. Beyond advancing our understanding of AI models, these methods offer ways for analyzing and improving systems that are increasingly used for scientific insights, as demonstrated through case studies in cognitive science and the humanities.
September 22nd, 2026
Topic: Can Language Models Understand Visual Information?
Petter Mæhlum (LTG)
Room: Styrerom 4118
We introduce a novel benchmark for evaluating whether language models can recognize visual concepts through Scalable Vector Graphics (SVG) representations. SVG is a wide-spread format that encodes images as sequences of elementary geometric shapes with text-based markup, making visual information accessible to language models without requiring additional training or multi-modal architectures. This enables us to investigate a fundamental question: Do models trained on text develop an understanding of the visual properties -- shapes, proportions, and colors -- of real-world objects? To address this question, we present a benchmark consisting of minimal manually-created SVG depictions of common objects, and compare our results to a human-annotated baseline.
September 3rd, 2026 (Thursday, 12:15 - 13:00)
Topic: Apertus: Democratizing Open and Compliant LLMs for Global Language Environments
Syrielle Montariol (Sorbonne Université)
Room: Black Box 3217
Apertus is a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Apertus models are pretrained exclusively on openly available data, retroactively respecting `robots.txt` exclusions and filtering for non-permissive, toxic, and personally identifiable content. The Apertus models also expand multilingual coverage, pre-training on 15T tokens from over 1800 languages and post-training on over 149 languages. In this talk, I will put Apertus in the context of the EU AI Act and detail the steps taken to make it compliant, with a focus on training data creation.
August 18th, 2026
Topic: Can It Happen, Will It Happen, and Where Should We Look? Towards Practical LLM Evaluation and Interpretability
Gianluca Barmina (University of Southern Denmark)
Room: Styrerom 4118
Large language models are increasingly evaluated not only for what they can do, but also for how they are likely to behave in realistic settings and for what their internal representations can tell us about that behavior. However, there is often a gap between what is convenient to measure and what would actually be useful to know about a deployed model.
In this talk, I will discuss two projects that approach this problem from complementary perspectives. First, I will present PropMe, a propensity-aware framework for memorization evaluation that distinguishes between a model's capability to reproduce training data when deliberately elicited and its propensity to do so under more ordinary prompting conditions. Second, I will turn to interpretability and ongoing work on Natural Language Autoencoders, which translate internal model activations into natural-language descriptions. Since generating such explanations for every token is prohibitively expensive, we ask a practical question: where should we look? We investigate whether inexpensive signals available from a standard forward pass can identify positions whose activations yield task-relevant explanations, and examine where useful internal information emerges across different models and auditing scenarios.
June 16th, 2026
Topic: Conservative Models and Radical Writers
Petter Mæhlum (LTG)
Room: Styrerom 4118
The Norwegian language is in a special situation of written diglossia, where the same spoken language has two separately defined written standards, Norwegian Bokmål and Norwegian Nynorsk. However, within each of the norms there is still considerable freedom, and many lemmas have official variants. In many -- but not all -- cases, these variants can be seen as being more conservative or moderate. We explore radicality in two settings: language feedback and machine translation, and look at how radical and moderate language is treated differently.
May 26th, 2026
Topic: Estimating Logical Faults in Chain-of-Thought Inference and Effectiveness of Adversarial Prompting
Ryan Anthony Marinelli (LTG)
Room: Styrerom 4118
In this work, Number of Mistakes and Average Angle Error are proposed to measure faults in logic in reasoning models. Segmenting chain-of-thought and determining distance measures informs how models process adversarial prompts.
May 12th, 2026
Topic: From Pain to Praise: Aspect-Based Sentiment Analysis for Norwegian Patient Feedback
Lilja Charlotte Storset (LTG)
Room: Styrerom 4118
The Norwegian Institute of Public Health (NIPH) conducts national patient experience surveys every year in order to define experience and outcome measures. These surveys include open-ended questions, generating tens of thousands of valuable comments each year. However, these are rarely analyzed because this requires a significant amount of time and resources. In this work, we describe a new dataset for aspect-based sentiment analysis (ABSA) for analyzing patient feedback about healthcare services. We report results for fine-tuning both encoder- and decoder-models on several text- and label-granularities, and compare different modeling strategies. We show how resources developed in this work can assist healthcare researchers in the analysis of patient feedback, bringing a much more efficient approach compared to today’s manual analysis.
April 28th, 2026
Topic: Unlearning Sensitive Information from LLMs
Jan Bronec (Charles University)
Room: Styrerom 4118
The sheer volume of data – often multilingual – required to pre-train LLMs rules out manual sanitization by human annotators. Heuristic approaches for filtering out sensitive, dangerous, or hateful content may miss out on various paraphrases of the same content, leaving different kinds of sensitive personal information, dangerous knowledge, or copyright-protected content unfiltered. LLMs trained on these datasets are susceptible to sensitive data leakage. As these inconsistencies are generally unveiled after pre-training, we find it necessary to sanitize pre-trained LLMs for downstream tasks, as well as in-between expensive re-training runs. Contemporary unlearning methods exhibit major model degradation and low robustness to paraphrased prompts. In our previous work, we sought to alleviate this degradation by imposing a constraint on parameter updates, enabling efficient computation of additional regularization terms in a commonly used unlearning method.
April 14th, 2026
Topic: Compositionality And Generalization in Language Modeling
Sondre Wold (LTG, NVE)
Room: Styrerom 4118
Our ability to understand and use language is not limited to expressions we are already familiar with. Instead, we continually create novel expressions by systematically combining words and phrases from a finite vocabulary. This ability is often described by the principle of semantic compositionality, which defines the meaning of a compound expression as determined by the meaning of the expression’s parts and their syntactic composition.
Today, a large portion of all text is generated by language models built with artificial neural networks. Unlike the principle of compositionality, these networks do not rely on an explicit definition of meaning composition, but on general-purpose machine learning techniques.
In this thesis, we evaluate several types of neural networks for both graph classification and sequence modeling, targeting productive and systematic properties of generalization. We achieve this through enforcing different constraints on the data distribution, introducing several evaluation frameworks. Our results show that most networks lack productive and systematic generalization. We further demonstrate the connection between systematic generalization and the information entropy of the training distribution, and we present novel ways of grounding the evaluation of neural networks in formal definitions of compositionality. Our findings advocate for a more principled approach to evaluation in NLP.
March 17th, 2026
DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling
Andrey Kutuzov (LTG)
Room: Styrerom 4118
We present DHPLT, an open collection of diachronic corpora in 41 diverse languages. DHPLT is based on the web-crawled HPLT datasets; we use web crawl timestamps as the approximate signal of document creation time. The collection covers three time periods: 2011-2015, 2020-2021 and 2024-present (1 million documents per time period for each language). We additionally provide pre-computed word type and token embeddings and lexical substitutions for our chosen target words, while at the same time leaving it open for the other researchers to come up with their own target words using the same datasets. DHPLT aims at filling in the current lack of multilingual diachronic corpora for semantic change modelling (beyond a dozen of high-resource languages). It opens the way for a variety of new experimental setups in this field.
March 3rd, 2026
Topic: Collecting a cultural-specific open-ended question answering dataset in Czech, Slovak and Ukrainian
Jindřich Helcl (LTG)
Room: Styrerom 4118
We introduce CUS-QA, a benchmark for evaluation of open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art large language models (LLMs). Our dataset consists of manually curated questions and answers grounded in Wikipedia, created by native speakers from Czechia, Slovakia, and Ukraine, with accompanying English translations. It includes both purely textual questions and those requiring visual understanding. We evaluate state-of-the-art LLMs through prompting and add human judgments of answer correctness. Using these human evaluations, we analyze the reliability of existing automatic evaluation metrics. Our baseline results show that even the best open-weight LLMs achieve only over 40% accuracy on textual questions and below 30% on visual questions. LLM-based evaluation metrics show strong correlation with human judgment, while traditional string-overlap metrics perform surprisingly well due to the prevalence of named entities in answers.
February 17th, 2026
Topic: Fortifying AI in Deployment Contexts
Ryan Anthony Marinelli (LTG)
Room: Styrerom 4118
Ryan Marinelli presents work from his PhD thesis, “Fortifying AI in Deployment Contexts”. The research introduces a new framework for AI security aimed at understanding emerging AI ecosystems and addressing the challenges of securing this evolving threat landscape. The proposed mitigations include applications of homomorphic encryption, mechanistic interpretability, agentic penetration testing, model calibration, and uncertainty quantification, among other techniques.
January 20th, 2026
Topic: Learning the (un)attested: Modeling acquisition of island constraints using symbolic and neural language models
Anastasia Philipps (LTG)
Room: Styrerom 4118
In this talk, I will introduce myself and present my PhD research on how computational modeling can be used to study learnability of linguistic constraints. My thesis focused on filler–gap dependencies – that is, dependencies between fillers such as "what" and the positions where they are interpreted, as in "What did you do __ yesterday" – and on island constraints that restrict such dependencies. Islands have traditionally been taken as evidence for rich innate linguistic knowledge, but recent modeling work suggests that they may instead be learnable from linguistic input using more general learning mechanisms.
I compared two fundamentally different modeling approaches – symbolic cognitive models and neural language models – and evaluated their ability to learn filler–gap dependencies and islands across languages. Using Norwegian, which shows systematic exceptions to widely attested island patterns, I tested whether these models could capture both general constraints on filler-gap dependencies and cross-linguistic variation. The results showed that both model types could learn some important generalizations from data, but that each also exhibited significant limitations. Overall, the talk discusses how model biases and properties of the input shape learning outcomes, and how different linguistic traditions view the usefulness of neural language models for questions related to language acquisition.
December 16th, 2025
Topic: NovelCR: A Large-Scale Bilingual Dataset Tailored for Long-Span Coreference Resolution
Meihan Tong (LTG, Integreat)
Room: Styrerom 4118
Coreference resolution (CR) links pronouns and noun phrases to their referent entities, serving as a key step in deep text understanding. Presently available CR datasets are either small in scale or restrict coreference resolution to a limited text span. We present NovelCR, a large-scale bilingual benchmark designed for long-span coreference resolution. NovelCR features extensive annotations, including 148k mentions in NovelCR-en and 311k mentions in NovelCR-zh. Moreover, the dataset is notably rich in long-span coreference pairs, with 85% of pairs in NovelCR-en and 83% in NovelCR-zh spanning across three or more sentences. Experiments on NovelCR reveal a large gap between state-of-the-art baselines and human performance, highlighting that NovelCR remains an open issue.
December 2nd, 2025
Topic: Analyzing LLMs Using Surveys
Hannah Clausen (LTG, Integreat)
Room: Styrerom 4118
The performance increase as well as wide availability of LLMs to the public resulted in an increasing research interest to examine the models themselves as well as their usefulness in other disciplines. One major interest lies in the applicability of measurement instruments created for humans to LLMs. Here, I will specifically focus on surveys consisting of a questionnaire as well as the respective human responses. Surveys offer high-quality data that is theoretically grounded and developed following methodological standards serving as a valuable source for the evaluation of LLMs. However, little research is done regarding the validity and reliability of LLM responses to questionnaires. In this talk, I will present an experimental framework for the analysis of LLMs using surveys addressing this gap.
November 18th, 2025
Topic: Lower Resourced Means Higher Risk in LLM Security
Heather Lent (Aalborg University)
Room: Styrerom 4118
This talk will introduce the emerging field of LLM security and highlight the ethically concerning research practices ongoing in this space. Among these concerns is the alarming lack of attention on lower-resourced languages and varieties, which I will argue puts the public at greater risk of security incursions, regardless of the language they speak. In addition to discussing ethical challenges of this new research direction, I will present a first empirical attempt to better understand language model security for medium and lower-resourced languages and varieties. This talk will draw on work published at TACL this year and as well as a pre-print (under review).
November 4th, 2025
Topic: Evaluating Machine Translation Quality Using Large Language Models
Shenbin Qian (LTG)
Room: Styrerom 4118
This study investigates the use of large language models (LLMs) as automatic evaluators for machine translation (MT) quality, an approach known as LLM-as-a-Judge. We explore their ability to perform reference-free quality estimation (QE) across both Chinese–English user-generated content and less-resourced language pairs. We first present HADQAET, a human-annotated dataset for assessing Chinese–English translation of microblog texts, and evaluate the performance of several LLMs (e.g., Llama-2, DeepSeek, Mixtral) against strong encoder-based baselines such as COMET-KIWI and TransQuest. Then, extending to less-resourced language pairs from the WMT 2023 (The Eighth Conference for Machine Translation) QE Shared Task, we investigate different training strategies and propose ALOPE (Adaptive Layer Optimisation for QE), which adapts intermediate LLM layers for regression-based MT evaluation. Our findings show that LLMs can approach, though not yet surpass, state-of-the-art QE systems, revealing both their multilingual potential and limitations in less-resourced contexts.
October 21st, 2025
Topic: Extracting knowledge embedded in an LLM
Emil Poiesz (LTG)
Room: Styrerom 4118
Using LLMs as teachers to extract knowledge embedded in the LLM. We implement adapted learning algorithms from Learning Theory with modern decoder models. Presenting initial results, challenges with the current approach, and looking ahead to future work.
June 10th, 2025
Topic: Text re-identification with language models
Benet Manzanares Salor (University Rovira i Virgili)
Room: Styrerom 4118
Time: 13:15-14:00
Text anonymization methods do not offer formal privacy guarantees, requiring an ex-post privacy assessment. To this end, we proposed Text Re-Identification Attack (TRIA), a method analogous to record linkage but specifically designed for textual data. Based on this attack, we define Text Re-Identification Risk (TRIR), a privacy metric that empirically quantifies the re-identification risk of a set of documents.
TRIA is implemented by leveraging a BERT-based language model classifier, trained on the background knowledge expected to be available for the attacker. Experimental evaluations reveal high re-identification risks across most anonymization methods, particularly those relying on Named Entity Recognition (NER). Notably, even manually anonymized texts—often treated as the ground truth—showed non-negligible risks.
Looking ahead, the plan is to enhance TRIA using state-of-the-art tools, including infilling of masked terms, Large Language Models (LLMs) and Retrieval Augmented Generation (RAG).
May 27th, 2025
Topic: How LLMs gatecrashed class: framing language technology in a learning and privacy perspective
Alexander Joyce and Tom Jarle Christiansen (Vestfold school municipality)
LLMs «gatecrashed» upper secondary education in the winter of 2023. The technology not only challenged learning, with a focus on assessment validity and academic integrity, but also student’s and employee’s privacy, who were keen to try out the technology by using services primarily set up to gather and process user data. Vestfold and Telemark county council initiated a development project that resulted in a chatbot-service spring 2023, which was gradually implemented with teachers and other staff, and then students. In the summer of 2024 we launched an experimental functionality with access to Norwegian language technology; implementing normistral-7b-warm-instruct through a HuggingFace inference endpoint, as well as a RAG pipeline built on an authoritative text corpus containing a mixture of Norwegian public sector data (NLOD) as well as open math textbook material. The service has been continually developed to this day and is the primary tool for exploring LLMs in both the schools and administration, in Telemark county council and Vestfold county council.
May 13th, 2025
Topic: HPLT & OpenEuroLLM: Towards Open European Foundation Models
HPLT and OpenEuroLLM are two R&D consortia funded by the European Union, who both invoked notions of “democratizing” LLM development and contributing to European “sovereignty” in their proposals. The projects work to create large volumes of multilingual pre-training data, to enable meaningful multilingual evaluation, to create the infrastructure for efficient LLM training in the EuroHPC ecosystem, and of course to create fully open LLMs between 7–400B parameters, with special emphasis on the languages of Europe.
May 6th, 2025
Topic: Stance detection: a sociolinguistic approach
Emilia Milano (University of Luxembourg)
The analysis of language ideologies and stance-taking in Luxembourgish online comments is the core of my PhD as part of the project TRAVOLTA (Tracing Attitudes And Variation In Online Luxembourgish Text Archives). In this context, we understand stance as a complex socio-pragmatic phenomenon which gives speakers the possibility to index ideologies, values, and social and personal identities in interaction. We study this phenomenon in Luxembourg, a highly multilingual country with official trilingualism, a high number of foreign residents as well as cross-border commuters.
In the public discourse on language in the country, stances about the language situation appear to be a proxy for questions of identity and social belonging. We focus on stance-taking strategies in interaction, keeping in mind that the genre "online comment" adds a layer of opacity to analysis due to the lack of context.
In this talk, after introducing the theoretical framework for my PhD, I will present the methods used to analyse our project data and possible future developments. Starting from a qualitative foundation, this project aims to use computational tools to better understand language ideologies and stereotypes in Luxembourgish written texts, as well as to inform NLP research on stance-taking on a sociolinguistic understanding of the phenomenon.
April 1st, 2025
Topic: Introduction to Reinforcement Learning
Egil Rønningstad (LTG)
This presentation is part of the mandatory work in my special syllabus course “Deep Reinforcement Learning”. The emphasis is on the basic concepts of reinforcement learning (RL). From there we move to neural networks for parametrized value function approximation, and present how Proximal Policy Optimization (PPO) estimates a policy function directly. We finally look at how aligning Large Language Models with human dialogue preferences can be posed as an RL problem, with Reinforcement Learning from Human Feedback (RLHF).
February 25th, 2025
Topic: Multi-label Scandinavian Language Identification (SLIDE)
Mariia Fedorova (LTG)
Identifying closely related languages at sentence level is difficult, in particular because it is often impossible to assign a sentence to a single language. In this paper, we focus on multi-label sentence-level Scandinavian language identification (LID) for Danish, Norwegian Bokmål, Norwegian Nynorsk, and Swedish. We present the Scandinavian Language Identification and Evaluation, SLIDE, a manually curated multi-label evaluation dataset and a suite of LID models with varying speed-accuracy tradeoffs. We demonstrate that the ability to identify multiple languages simultaneously is necessary for any accurate LID method, and present a novel approach to training such multi-label LID models. (Fedorova et al., 2025)
January 28th, 2025
Topic: HalluCounter: Reference-free Hallucination Detection in LLMs
Rahul Mishra (LTG)
Slides
In this talk, I'll present our upcoming ACL 2025 submission on detecting hallucinations in Large Language Models (LLMs) without reference text. I'll begin by highlighting the significance and challenges of reference-free hallucination detection, as well as existing approaches. Next, I'll dive into our proposed HalluCounter model and introduce novel benchmark datasets for the Reference-Free Hallucination Detection (RFHD) task. Finally, I'll discuss our evaluation strategies, share key results, and provide insightful takeaways.
January 14th, 2025
Topic: Ontological Methods for Opinion Interpretation
Gaurav Negi (University of Galway)
Opinion mining has been a highly active research area over the past few decades, driven by its importance in understanding and leveraging human perspectives. In this talk, I will present our recent work aimed at unifying the formulations of sentiment analysis and opinion mining within and across the fields of Natural Language Processing (NLP) and the Semantic Web. Central to our approach is the development of the Unified Opinion Concepts (UOC) ontology—a framework for semantically structured representation of opinions. I will briefly discuss the foundational concepts that underpin this ontology and highlight its application.
December 17th, 2025
Topic: Translation as Rational Communication
Maria Kunilovskaya (University of Saarland)
The talk gives an overview of the applications of information theory in translationese studies based on the publications by B7 project, a part of a large SFB1102 grant called Information Density and Linguistic Encoding (IDeaL, https://sfb1102.uni-saarland.de/projects/). Translationese studies is a central direction in the empirical translation studies that aims to capture and explain the specificity of mediated language (translation and interpreting). Information theory provides a formal basis for a rational account of variation in language use, which hypothesises that linguistic choices are guided by the pressure for an optimal balance between the amount of transmitted information and the cognitive effort needed to encode the message (Levy, 2008; Jaeger, 2010). I will summarise the studies of translated language built on surprisal theory (Bizzoni and Teich, 2019; Przybyl et al., 2022; Kunilovskaya et al., 2023) and arguing that the specificity of linguistic choice in translation might be a rational reaction to the increased cognitive load (compared to monolingual production). Our results shed light on how the translation process affects the information density of the message. We highlight the differences between outcomes of the written and spoken mediation modes, which are attributed to dissimilar production conditions, especially with regard to the cognitive load. I also hope to discuss our current agenda and the on-going research focused on two research tasks: (i) implementing the memory-surprisal tradeoff (Hahn et al., 2021) to model the online interpreting process, and (ii) representing the cross-lingual cognitive difficulty of a translation task and tracing its impact on translational behaviour.
November 5th, 2024
Topic: GPT or BERT: why not both?
David Samuel (LTG)
I will swiftly go through three related papers: First, BERTs are Generative In-Context Learners (https://arxiv.org/abs/2406.04823) demonstrates that masked language models are as capable in (generative) in-context learning as causal language models, then GPT or BERT: why not both? (https://arxiv.org/abs/2410.24159) shows how these two training objectives can be merged together and result in a stronger model, and lastly the NorMistral paper (not published yet) suggests that this hybrid masked-causal language modeling can also work at a very large scale.
October 22nd, 2024
Topic: LLMs: Code generation abilities, security vulnerabilities, and problem-solving abilities
Rebeka Tóth and Tamas György Bisztray (SEC)
This presentation synthesizes key findings from three distinct studies on large language models (LLMs), focusing on their code generation capabilities, security vulnerabilities, and problem-solving abilities. The first study explores the security risks of C programs generated using neutral prompts, introducing the FormAI-v2 dataset of 265,000 programs, and finding that 63.47% contain vulnerabilities. The second study examines the security of GPT-4-generated PHP code for web development from the security and compelxity aspect, revealing that 26% of deployed websites had exploitable vulnerabilities, particularly in file upload scenarios. The third study introduces the Dynamic Intelligence Assessment (DIA) framework, benchmarking LLMs' adaptive intelligence and model confidence across complex problem domains. Together, these studies underscore the importance of rigorous testing and validation of LLM-generated content, highlighting the risks and limitations of using such models in production environments.
September 24th, 2024
Topic: Orchestrating WSD and WSI Models for Semantic Change Modeling
Nikolay Arefyev (LTG)
In this talk I will describe our experiments and solutions of the first subtask of the AXOLOTL-24 shared task on Semantic Change Modeling. The goal of this subtask is to distribute a given set of usages of a polysemous word from a newer time period between senses of this word from an older time period and clusters representing gained senses of this word. We propose and experiment with three types of methods solving this task, pure WSD and WSI methods and combined methods developed specifically for this Semantic Change Modeling task. Our methods achieve SOTA results according to both official metrics of the first substask. Additionally, we develop a model that can tell if a given word usage is not described by any of the provided sense definitions. This model serves as a component in one of our methods, but can potentially be useful on its own.
September 10th, 2024
Topic: Machine Translation for Norwegian
Petter Mæhlum (LTG)
Slides: TBA
The topic of my presentation will be two-fold. I will first present a brief overview of resources related to machine translation for Norwegian, focusing on translation from Bokmål into Nynorsk. This part will also touch on the background of why translation into Nynorsk is a thing. I will then talk about my PhD project, and how I plan to look into various sources of variation in relation to machine translation for Norwegian.
This paper shows how modern NLP technology can be used to expose the rationale behind RFC design decisions. A commentary is a collection of all the comments posted by those participants within a working group (WG) who are actively involved in drafting during each of the I-D iterations. They may suggest further edits, ask for clarification, explain the rationale, defend certain design choices, reject other options, etc. It is not common, however, for there to be clear links between specific design decisions written in an I-D and these arguments. The communication is carried out in such a way that ``everybody" already knows what it is that they are discussing. We create the Internet Engineering Task Force (IETF) commentary on Internet-Drafts (I-Ds) dataset NetD2R (Network Design Decisions and Rationales), in order, firstly, to establish these missing links, secondly, to find relevant discussions given specific text snippets from an I-D, and finally, to understand the rationale behind the design decisions described in the Internet Protocol specifications (Request for Comments (RFCs)). Specifically, we apply the modern information retrieval (IR) technique to this dataset to automate the mining of the relevant discussions that lead to a specific version of an I-D. Our work lies at the intersection of Internet protocol design and standardization and debate machine reading.
May 7th, 2024
Topic: A journey from tokenization to instruction-finetuning (NorMistral)
David Samuel and Lucas Charpentier (LTG)
We will present the whole process of creating the Norwegian Langauge Models, specifically NorMistral, from tokenisation to pretraining. Then passing by evaluating the models, followed by quantization. To finish with instruction fine-tuning, limitations and future plans.
April 9th, 2024
Topic: Socio-political Events of Conflict and Unrest
Helene Bøsei Olsen (LTG)
There is a large and growing body of literature on datasets created to facilitate the study of socio-political events of conflict and unrest. However, the datasets, and the approaches taken to create them, vary a lot depending on the type of research they are intended to support. For example, while scholars from natural language processing (NLP) tend to focus on annotating specific spans of text indicating various components of an event, scholars from the disciplines of political science and conflict studies tend to focus on creating databases that code an abstract but structured representation of the event, less tied to a specific source text. The survey aims to map out the current landscape of available event datasets within the domain of social and political conflict and unrest – both from the NLP and political science communities – offering a unified view of the work done across different disciplines.
March 26th, 2024
Topic: Emotion Analysis of Tweets Banning Education in Afghanistan
Mohammad Ali Hussiny (LTG)
We introduce the first emotion-annotated dataset for the Dari variant of Persian spoken in Afghanistan. The LetHerLearn dataset contains 7,600 tweets posted in reaction to the Taliban’s ban of women’s rights to education in 2022 and has been manually annotated according to Ekman’s emotion categories. We here detail the data collection and annotation process, present relevant dataset statistics as well as initial experiments on the resulting dataset, benchmarking a number of different neural architectures for the task of Dari emotion classification.
March 12th, 2024
Topic: Third-semester report
Sondre Wold (LTG)
February 27th, 2024
Topic: Consistent anomalies in the grammatical number of nouns: an overview
Elena Spaziani (La Sapienza)
Grammatical number is one of the most studied categories in linguistics. Given the apparently straightforward relationship between grammatical values and the numerical properties they denote, grammatical number can be considered one of the simplest categories. However, there are many difficulties, epitomised by what have long been called “anomalies”, that arise on closer inspection. An overview will be given of two anomalous cases, namely lexical plurals, i.e. plural forms that have a non-grammatical meaning, and singularia and pluralia tanta nouns, i.e. nouns that are used in only one grammatical form.
February 13th, 2023
Topic: More Room For Language: Investigating the Effect of Retrieval on Language Models
Retrieval-augmented language models pose a promising alternative to standard language modeling. During pretraining, these models search in a corpus of documents for contextually relevant information that could aid the language modeling objective. We introduce an `ideal retrieval' methodology to study these models in a fully controllable setting. We conduct an extensive evaluation to examine how retrieval augmentation affects the behavior of the underlying language model. Among other things, we observe that these models: {i) save substantially less world knowledge in their weights, (ii) are better in understanding local context and inter-word dependencies, but {iii) are worse in comprehending global context.
December 14th, 2023
Topic: BabyLM Challenge: What is it? Best Models and Papers
The shared task for the CoNLL workshop was the BabyLM challenge. One of the objectives of this shared task was to democratize pre-training by creating two small datasets (10M and 100M words) of high quality. The shared task showed some interesting results that will be quickly presented. In addition, we will present our model that won both the strict and strict-small tracks as well as the winner of the loose track and the two outstanding papers.
November 30th, 2023
Topic: Dictionary-like definitions for Semantic Change datasets
Mariia Fedorova (LTG)
Slides
Semantic change is a change of the meanings of a language unit (a morpheme, a word, a phrase etc.). The concept of meaning (sense) in computational linguistics is data-driven, namely, the senses may be induced from a large number of word usages without any additional knowledge. The senses may be defined as clusters of target word usages, but one has to explore the word usages manually in order to understand which particular senses have changed. Since a word may have tens of senses, an automated approach to labeling the sense clusters is required. One of the possible approaches is generating dictionary-like definitions of the word usage clusters. We will discuss some definition generation methods and their evaluation.
November 16th, 2023
Topic: Employing AI to Help Lost Pets Return to Their Homes
Nikolay Arefev (LTG)
The Kashtanka.pet (https://kashtanka.pet) project addresses the problem of searching for lost pets efficiently. There are numerous platforms and groups in social networks collecting ads about missing and found cats and dogs. However, it is often almost impossible for a human to find a specific lost pet among millions of ads about found pets distributed across all those websites. The project aims at helping pet owners and volunteers to find lost pets efficiently with the help of AI. It crawls websites for ads about lost and found pets, and retrieves the pairs of ads announcing the same pet was lost and then found. The retrieved pairs are then inspected and further processed by humans.
November 2nd, 2023
Topic: Benchmarking transformer language models on natural language understanding tasks
Vladislav Mikhailov (LTG)
Benchmarking has found broad acceptance in NLP as a conventional approach to comparing language models LMs with respect to specific evaluation criteria, such as performance, efficiency, and fairness. However, benchmarking suffers from low linguistic diversity and the inappropriateness of the result aggregation procedures that do not account for benchmark complexity. To this end, we propose the first large-scale benchmarks for the Russian language that cover a broad scope of NLU tasks. We develop novel aggregation procedures that rely on rankings in each evaluation criterion and allow aggregating heterogeneous information. We present the results of benchmarking over one hundred LMs, comparing their performance to human-level performance across various experimental setups.
October 19th, 2023
Topic: Graph-based Anomaly detection
Shiva Shadrooh (NTNU)
Graph-based anomaly detection utilizes graph theory and network analysis to identify abnormal patterns or outliers within complex data structures, representing data elements as nodes and their interactions as edges. This approach is crucial due to the intricate relationships graph structures can capture, which traditional machine learning models may struggle to handle effectively. Moreover, the real-time aspect of graph-based algorithms is vital, providing swift anomaly detection for immediate action in critical domains such as network security and fraud detection. These algorithms swiftly uncover atypical behaviors or events, aiding timely interventions and responses in interconnected data scenarios.
September 28, 2023
Topic: Deep Learning For Unsupervised Relation Extraction
Étienne Simon (LTG)
Capturing concepts' interrelations is a fundamental of natural language understanding. It constitutes a bridge between two historically separate approaches of artificial intelligence: the use of symbolic and distributed representations. However, tackling this problem without human supervision poses several issues, and unsupervised models have difficulties echoing the expressive breakthroughs of supervised ones. We address two supervision gaps we identified: the problem of regularization of sentence-level discriminative models and the problem of leveraging relational information from dataset-level structures. The first gap arises following the increased use of discriminative approaches, such as deep neural network classifiers, in the supervised setting. These models tend to collapse without supervision. To overcome this limitation, we introduce two relation distribution losses to constrain the relation classifier into a trainable state. The second gap arises from the development of dataset-level (aggregate) approaches. We show that unsupervised models can leverage a large amount of additional information from the structure of the dataset, even more so than supervised models. We close this gap by adapting existing unsupervised methods to capture topological information using graph convolutional networks. Furthermore, we show that we can exploit the mutual information between topological (dataset-level) and linguistic (sentence-level) information to design a new training paradigm for unsupervised relation extraction.
September 14, 2023
Topic: Corpus-based computational dialectology: Data, methods and results
Yves Scherrer (LTG)
OBS: This will be held in Kristen Nygaards sal, Room 5370
The CorCoDial (corpus-based computational dialectology) project aims to infer dialect classifications from variation-rich corpora, focusing in particular on the dialect-to-standard normalization task to introduce comparability between different texts. I will start by presenting a multilingual collection of phonetically transcribed and orthographically normalized corpora. This collection forms the data basis of several case studies. In the first study, we investigate to what extent topic models can find dialectological rather than semantic topics. In the second experiment, we evaluate character alignment methods from different research traditions on a range of desirable and undesirable characteristics. In the last study, we focus on neural dialect-to-standard normalization and investigate what the embeddings of speaker labels can tell us about the origin of the speakers.
June 15, 2023
Topic: Interpretable Word Sense Representations via Definition Generation: The Case of Semantic Change Analysis
Andrey Kutuzov (LTG)
We propose using automatically generated natural language definitions of contextualised word usages as interpretable word and word sense representations. Given a collection of usage examples for a target word, and the corresponding data-driven usage clusters (i.e., word senses), a definition is generated for each usage with a specialized Flan-T5 language model, and the most prototypical definition in a usage cluster is chosen as the sense label.
We demonstrate how the resulting sense labels can make existing approaches to semantic change analysis more interpretable, and how they can allow users (historical linguists, lexicographers, or social scientists) to explore and intuitively explain diachronic trajectories of word meaning. Semantic change analysis is only one of many possible applications of the "definitions as representations" paradigm. Beyond being human-readable, contextualised definitions also outperform token or usage sentence embeddings in word-in-context semantic similarity judgements, making them a new promising type of lexical representation for NLP.
April 27, 2023
Topic: Training a new generation of LTG language models
David Samuel (LTG)
Can a competitive masked language model be trained on a small text corpus of 100 million words? The short answer is yes! And as a reward, we can also get an optimized training recipe for language models. Can we then apply this recipe on a huge Norwegian text corpus? Can we apply it on generative language models? And what Norwegian corpus should we use? And how should we evaluate the models? This talk will try to answer all of these questions and much more
April 20, 2023
Topic: The RWKV language model: An RNN with the advantages of a transformer
Johan Sokrates Wind (UiO, Dep. of Mathematics)
Large pretrained transformers such as ChatGPT have demonstrated impressive performance across a diverse range of tasks. Recently, the RWKV architecture was developed as an RNN that can be trained like a transformer, resulting in a model with 14 billion parameters, making it the largest RNN ever. Moreover, unlike traditional RNNs, RWKV performs comparably to transformers on benchmark tests. In this presentation, I will explain how the RWKV architecture enables effective training and inference.
March 30, 2023
Topic: Phonotactics as an Aid in Low Resource Loan Word Detection and Morphological Analysis in Sakha
Petter Mæhlum (LTG)
Obtaining information about loan words and irregular morphological patterns can be difficult for low-resource languages. Using Sakha as an example, we show that it is possible to exploit known phonemic regularities such as vowel harmony and consonant distributions to identify loan words and irregular patterns, which can be helpful in rule-based downstream tasks such as parsing and POS-tagging. We evaluate phonemically inspired methods for loanword detection, combined with bi-gram vowel transition probabilities to inspect irregularities in the morphology of loanwords. We show that both these techniques can be useful for the detection of such patterns.
March 16, 2023
Topic: Fine-tuning of cross-lingual language models for lexical semantic change detection
Nikolay Arefev (LTG)
Contextualized word embeddings from out-of-the-box neural language and masked language models are known to be bad representations of word meaning due to large orthographic / grammatical bias. Fine-tuning such models on some labeled datasets for the final task is the standard way to achieve good performance. However, it is not obvious how to fine-tune them directly for the lexical semantic change detection (LSCD) task in which examples are single words and targets are some scores corresponding to the change in their meaning between two time periods.
In this talk I will present our experiments on fine-tuning language models for the LSCD task. Our models achieved SOTA results in the RuShiftEval-2021 and the LSCDiscovery-2022 shared tasks on lexical semantic change detection for the Russian and the Spanish language correspondingly. Surprisingly, even models fine-tuned on English data only achieve near SOTA performance due to zero-shot cross-lingual abilities of the underlying cross-lingual masked language model. Another appealing property of the fine-tuned models is their ability to annotate word occurrences with glosses which suggests a way to make the LSCD predictions interpretable.
March 2, 2023
Topic: Language Models and the Discomfort of Not Knowing
Mark Anderson (Norwegian Computing Center)
In language technology, we need to evaluate the capabilities of models. Often the apparent concrete nature of quantitative metrics can gives a false sense of certainty. In this talk, I will vaguely, and at times indirectly, discuss my general feeling of discomfort about evaluation in NLP and about claims made about the nature of models.
February 9, 2023
Topic: Meaning Making with Artificial Interlocutors and Risks of Language Technology
Emily M. Bender (University of Washington)
OBS: This will be held in Kristen Nygaards sal, Room 5370
Humans make sense of language in context, bringing to bear their own understanding of the world including their model of their interlocutor's understanding of the world. In this talk, I will explore various potential risks that arise when we as humans bring this sense-making capacity to interactions with artificial interlocutors. That is, I will ask what happens in conversations where one party has no (or extremely limited) access to meaning and all of the interpretative work rests with the other, and briefly explore what this entails for the design of language technology.
February 2, 2023
Topic: Learning Linguistic Tree Structures with Text and Graph Methods
Irina Nikishina (University of Hamburg)
Knowledge graphs such as DBpedia, Freebase or Wikidata always contain a taxo- nomic backbone that allows the arrangement and structuring of various concepts in accordance with the hypo-hypernym (“class-subclass”) relationship. With the rapid growth of lexical resources for specific domains, the problem of automatic extension of the existing knowledge bases with new words is becoming more and more widespread. In this talk, we address the problem of Taxonomy Enrichment which aims at adding new words to the existing taxonomy.
We formulate two task settings on the automatic taxonomy extension. The first one aims at predicting hypernyms (“parents, words with broader meanings”) from taxonomy given a predefined list of new words with no definition. The second task setting considers taxonomy enrichment with no predefined candidates. We assume that the the compressed information from pre-trained language models like BERT or T5 can be leveraged to predict new words missing in taxonomic resources.
January 19, 2023
Topic: Developing a Conversational Agent for Engineering Design Collaboration
Joseph Makokha (UiO, DIG group)
Intelligent systems incorporating artificial intelligence (AI) have been around for decades and have steadily gained capabilities approaching humans in several domains ranging from medicine where they aid in detection of fractures and tumours; data science where they automate tasks; in art generation; robotics and self-driving cars, and many others. As digital technologies like artificial intelligence become increasingly pervasive around us, we find humans frequently collaborating with machines in Human-AI (HAI) teams in diverse contexts. There is therefore a need for researchers, practitioners, decision makers, and others to understand ways that AI will influence everyday processes and outcomes. Many questions arise from prospective scenarios, such as what will happen when an AI outperforms humans on common tasks at work - and researchers are attempting to answer these questions regarding technical, practical, philosophical and other aspects of AI. However, there exist few replicable examples - where an AI outperforms humans - beyond strategy games like AlphaGo, chess and similar contests. Thus we seek to narrow this gap.
We propose a conceptual model of an AI collaborative system comprising a human-AI team similar to how such teams might operate; then demonstrate a way of developing a class of AI systems, the Disruptive Interjector (DI), which observe what a human is doing, then interject with suggestions that aid in idea generation or problem solving in human-AI (HAI) teams. We note that this kind of system goes beyond current creativity support systems by replacing one teammate in a human-human (HH) team with an AI, to create a HAI team. The proposed DI diverges from a solution by encouraging consideration of other possibilities, and is therefore distinct from tutors, chatbots, recommenders and other similar systems that seek to converge towards a “correct” solution. To this end, we develop a conceptual design of the DI system, then apply deep Convolution Neural Networks (CNNs) - in form of an LSTM network and two datasets - to generate new conversations that can be used in collaborating with a human. This talk will highlight barriers as well as possible solutions that will enable the successful development of such collaborative systems.
December 5, 2022
Topic: Entity-level sentiment analysis (3rd semester evaluation)
Egil Rønningstad (LTG)
Slides (TBA)
November 21, 2022
Topic: Graph neural networks and how they are used in NLP
Sondre Wold (LTG)
A lot of data fit nicely into a graph formalism. Social networks, biological systems, and maps can all be thought of as relationships between objects. The Graph Neural Network (GNN) architecture has become the most expressive way to do representation learning over such structures, providing a unified framework for generating embeddings for nodes, relations, and even entire graphs. These embeddings can be used for tasks like predicting fraudulent transactions in a financial network, drug discovery, and shortest path computation for services like Google Maps. The first half of this talk will introduce the most common GNN models, focusing on how they differ from "standard" NNs. The second half will present and discuss how GNNs are currently being used to solve tasks within NLP, particularly for integrating external knowledge for tasks like QA and language modeling.
November 7, 2022
EventGraph: Event Extraction as Semantic Graph Parsing
Huiling You (LTG)
Event extraction involves the detection and extraction of both the event triggers and corresponding event arguments. Existing systems often decompose event extraction into multiple subtasks, without considering their possible interactions. We propose EventGraph, a joint framework for event extraction, which encodes events as graphs. We represent event triggers and arguments as nodes in a semantic graph. Event extraction therefore becomes a graph parsing problem, which provides the following advantages: 1) performing event detection and argument extraction jointly; 2) detecting and extracting multiple events from a piece of text; and 3) capturing the complicated interaction between event arguments and triggers. Experimental results on ACE2005 show that our model is competitive to state-of-the-art systems and has substantially improved the results on argument extraction.
October 24, 2022
Towards more robust text anonymization through adversarial models
This PhD project will aim at developing adversarial models to text anonymization (i.e. models that re-identify masked identifiers). To do this we plan to use retrieval-based transformers/models and various databases that act as levels of background knowledge available to an attacker.
September 26, 2022
Introducing two datasets: NARC and a small PoS-tagged dataset for dialectal tweets
Petter Mæhlum (LTG)
The completion of NARC (Norwegian Anaphora Resolution Corpus) is getting close. We present a corpus for coreference resolution and bridging for Norwegian. Our forthcoming publication focuses on the Bokmål part of the corpus, with basic corpus statistics and preliminary modelling results. The PoS-tagged tweets were made in order to evaluate commonly used PoS-taggers on informal Norwegian data in three categories: Bokmål, Nynorsk and dialectal tweets. The dataset is small, but allows us to highlight some problems seen when trying to PoS-tag informal text and written dialect.
June 13, 2022
Entity-Level Sentiment Analysis ELSA: What we have found and where to go from here
Egil Rønningstad (LTG)
Work in progress
We have recently created an exploratory dataset to find similarities and differences between Entity-Level Sentiment Analysis (ELSA) and other sentiment analysis tasks, in particular Targeted Sentiment Analysis (TSA). We see that ELSA can (partially) be derived from NER, coreferences and TSA, but error propagation is an issue. Our next steps are 1) To create annotation guidelines for a proper ELSA dataset, 2) Explore alternative, probably summarization-related approaches to ELSA. After the presentation we will have room for discussion and brainstorming.
May 16, 2022
Language technology tools to support low-resource languages: case study of Sakha
Sardana Ivanova (University of Helsinki)
Paper 1 (no longer available), Paper 2
Slides
This presentation gives an overview on language technology tools for supporting low-resource languages, in particular, the Sakha language. Tools include: a morphological analyser, a computer-assisted language learning (CALL) platform, and two natural language generation (NLG) systems.
We extended an earlier, preliminary version of the morphological analyser, built on the Apertium rule-based machine translation platform. The transducer, developed using Helsinki Finite-State Toolkit (HFST), has coverage of solidly above 90%, and high precision. Based on the morphological analyser, we implemented a language learning environment for Sakha in the Revita CALL platform. Revita is a freely available online platform for learners beyond the beginner level.
Currently we have implemented two NLG systems for Finnish and a few other languages: a transformer-based poetry generation system and a template-based news generation system. We plan to extend those systems to support Sakha.
May 9, 2022
Graph-Based Entity Models for Dialogue Management
Nicholas Walker (University of Oslo)
Modern task-oriented spoken dialogue systems often rely on dialogue management modules, which keep track of information for an autonomous dialogue agent to complete tasks. Dialogue systems are also frequently deployed with physical robotic agents for Human-Robot Interaction (HRI).
This seminar will detail completed and planned work in dialogue management and HRI which comprise the titular doctoral project as part of the third semester evaluation of the work. The present state-of-the-art and proposed graph-based approaches for dialogue management will be discussed in addition to methodological challenges. We will present an overview of the current theory, investigations into methodologies and prototypes, data collection, and an outline of future work for the project. Discussion will also include description of the work in the context of HRI and experiments with the project’s Pepper robot.
April 25, 2022
NorDiaChange: diachronic semantic change dataset for Norwegian
Andrey Kutuzov (LTG)
We describe NorDiaChange: the first diachronic semantic change dataset for Norwegian. NorDiaChange comprises two novel subsets, covering about 80 Norwegian nouns manually annotated with graded semantic change over time. Both datasets follow the same annotation procedure and can be used interchangeably as train and test splits for each other. NorDiaChange covers the time periods related to pre- and post-war events, oil and gas discovery in Norway, and technological developments. The annotation was done using the DURel framework and two large historical Norwegian corpora. NorDiaChange is published in full under a permissive license, complete with raw annotation data and inferred diachronic word usage graphs (DWUGs).
March 31, 2022 (12:30)
Parsing into semantic dependency graphs
Maja Buljan (University of Oslo)
Natural language processing encompasses a spectrum of tasks whose goal, on a superficial level, is to structure the information contained in "raw" human language input. In this talk, we focus on meaning representation parsing -- i.e. mapping from natural language utterances to graph-based encodings of semantic structure. As the halfway-point progress report (third semester evaluation) of the titular doctoral project, we will give an overview of the methodological challenges of the task, review the current state-of-the-art, and summarise completed and ongoing work that comprises the project. This includes an in-depth dive into different meaning representation frameworks, parsing architectures, diagnostic evaluation of systems, and framework-specific error analysis, as well as a look forward to (currently) unsolved challenges in model development, e.g. multitask learning for cross-framework and cross-lingual parsing.
March 14, 2022
Efficient Strategies of Language Production: An Information-Theoretic Analysis
Mario Giulianelli (University of Amsterdam)
Speakers are thought to use efficient information transmission strategies for effective communication. For example, they transmit information at a constant rate in written text and they use repetitions extensively in spoken dialogue. We analyze these strategies in monologue and dialogue datasets, combining information-theoretic measures with probability estimates obtained from Transformer-based language models.
We find (i) that information density decreases overall in spoken open domain and written task-oriented dialogues, while it remains uniform in written texts; (ii) that speakers’ choices are oriented towards global, rather than local, uniformity of information; (iii) that uniform information density strategies are at play in dialogue when we zoom in on topically and referentially coherent contextual units; (iv) and that repetitions of non-topical and non-referential expressions, too, can be interpreted as an efficient production strategy.
Besides providing new empirical evidence on written and spoken language production, we believe that our studies can directly inform the development of more human-like natural language generation models.
February 28, 2022
Internet Protocol Standardization in the IETF: An Introduction to the Textual Archive and Potential NLP Applications
Michael Welzl (University of Oslo)
Internet protocols are standardized in the Internet Engineering Task Force (IETF). The standardization process in this organization involves a large amount of freely accessible textual artifacts: discussions in mailing lists, on GitHub, as well as recorded meeting minutes of online and in-person meetings. The results are “RFCs” - prose documents which lend themselves to NLP analysis just as well as the textual body that precedes their finalization. This talk will introduce the IETF process, the available means to access the text archives, and share some ideas on NLP analyses that might be useful to the Internet community or computer scientists in general.
February 14, 2022
Case Studies in BERTology: shallow heuristics or verbal reasoning?
Anna Rogers (University of Copenhagen)
What does BERT learn from the current Natural Language Understanding datasets - verbal reasoning skills, or shallow heuristics? This talk discusses available evidence and presents a case study of generalization in NLI (from MNLI to the adversarially constructed HANS dataset) in a range of BERT-based architectures (adapters, Siamese Transformers, HEX debiasing), as well as with subsampling the data and increasing the model size. Most strategies are unsuccessful, but they are all providing insights into how Transformer-based models learn to generalize.
January 31, 2022
The DECRYPT project - tools, methods and strategies in historical NLP
Crina Tudor (Uppsala University)
Historical ciphers and keys represent a rich source of information that can provide great insights into our past. The main drawback, however, is that such sources are spread out in archives all over the world, which makes it rather difficult to analyze and compare various manuscripts.
The DECRYPT project started out as an effort to address this issue by building a reliable and comprehensive system that aims to make such sources easily available to the general public. This interdisciplinary endeavor brings together historians, linguists, cryptographers and programmers who work together to develop tools that can facilitate the automatic analysis of encoded manuscripts.
January 17, 2022
Targeted Sentiment Analysis (TSA), and how to make the best use of your data
Egil Rønningstad (LTG)
Slides (based on Master thesis)
Targeted Sentiment Analysis aims at for each sentence to detect the words representing that what is spoken positively or negatively about. In the sentence "I admire my dog", "my dog" is spoken positively about. In TSA, we do not include finding the holder/source "I", or the words that express this positivity or negativity, like "admire".
I have done TSA on the Norwegian NoReC-fine dataset. I have compared different word embeddings to use with LSTM, and compared with some pretrained BERT-related models. NoReC-fine consists of newspaper reviews of topics from various domains. We look at the cross-domain effect on the results, and compare them with cross-lingual experiments with same-domain data.
December 13, 2021
Multilingual Language Models for Fine-tuning and Feature Extraction in Word-in-Context Disambiguation
Huling You (LTG)
SemEval-2021 Task 2: Multilingual and Crosslingual Word-in-Context Disambiguation (MCL-WiC) is proposed as a benchmark to evaluate context-sensitive word representations. Our main interest is to investigate the usefulness of pre-trained multilingual language models (LMs) in this MCL-WiC task, without resorting to sense inventories, dictionaries, or other resources. As our main method, we fine-tune the language models with a span classification head. We also experiment with using the multilingual language models as feature extractors, extracting contextual embeddings for the target word. We compare three different LMs: XLM-RoBERTa (XLMR), multilingual BERT (mBERT) and multilingual distilled BERT (mDistilBERT).
We find that fine-tuning is better than feature extraction. XLMR performs better than mBERT in the cross-lingual setting both with fine-tuning and feature extraction, whereas these two models give a similar performance in the multilingual setting. mDistilBERT performs poorly with fine-tuning but gives similar results to the other models when used as a feature extractor.
November 29, 2021
Analyzing public sentiment towards wind energy in Norway
Alexandra Wittemann (LTG)
The work was done within this project
Wind power has with technological development and cost decrease as of lately become profitable in Norway even without any subsidies, which have led to an increase in new developments across the country. These new developments have not always been welcomed and as development continued to increase, so did the opposition. In addition to that the distribution of benefits and burdens of energy infrastructure and energy policies should be fair and equitable, transitioning to a low-carbon society or energy system requires support for political decisions and policies.
A traditional way to collect information on public opinions is via questionnaires, surveys or interviews. These methods may however be prone to selection bias and response bias, as well as missing data and incomplete information. There is as such a value in exploring alternative methods to acquire information on public opinion. In this study, we follow the work of Kim et al.[2020] and assess the public sentiment in Norway towards on- and offshore wind energy via a machine learning approach for natural language processing based on data scrapping from social media sites such as Twitter. We collected about 70 000 Norwegian tweets which we manually annotated a subset of and fed into NorBERT for fine-tuning. We then used the fine-tuned model to classify the rest of the tweets.
November 8, 2021
Text anonymization with explicit measures of disclosure risk
Anthi Papadopoulou (LTG)
We present a new approach to text anonymization, one that moves past a NER task and incorporates disclosure risk in the process, combining NLP and PPDP. Making use of Wikipedia biographies and background knowledge from Wikidata we propose an automatic annotation method based on k-anonymity that can produce large amounts of labeled data for sensitive information. We train two BERT models on these data following two different approaches of picking sensitive terms to mask. We also manually annotate and release a sample of a 1000 article summaries, and use it to check the performance of our models.
October 25, 2021
What Quantifying Word Order Freedom Reveals about Dependency Corpora
Maja Buljan (LTG)
Slides
This is an overview of ongoing work on word order freedom and syntactic annotation, with the goal of differentiating between findings that reveal inherent properties of languages vs. features dependent on annotation styles. Following previous work on defining a quantifiable and linguistically interpretable measure of word order freedom in language, we take a closer look at the robustness of the basic measure (word order entropy) to variations in dependency corpora used in the analysis. We compare measures at three levels of generality, applied to treebanks annotated according to the Universal Dependencies v1 and v2 guidelines, spanning 31 language. Preliminary results show that certain measures, such as subject-object order freedom, are sensitive to changes in annotation guidelines, highlighting aspects of these metrics that should be taken into consideration when using dependency corpora for linguistic analysis.
October 11, 2021
"Improving Multilingual Lexical Normalization by Fine-tuning ByT5"
David Samuel (LTG)
We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021, which evaluates lexical normalization systems on 12 social media datasets in 11 languages.
Our system is based on a pre-trained byte-level language model, ByT5, which we further pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. We release both the source code and the fine-tuned models.
September 27, 2021
"Grammatical Profiling for Semantic Change Detection"
Andrey Kutuzov (LTG)
https://arxiv.org/abs/2109.10397
Semantics, morphology and syntax are strongly interdependent. However, the majority of computational methods for semantic change detection use distributional word representations which encode mostly semantics. We investigate an alternative method, grammatical profiling, based entirely on changes in the morphosyntactic behaviour of words.
We demonstrate that it can be used for semantic change detection and even outperforms some distributional semantic methods. We present an in-depth qualitative and quantitative analysis of the predictions made by our grammatical profiling system, showing that they are plausible and interpretable.