Pierre Lison

• Manzanares-Salor, Benet; Sánchez, David; Lison, Pierre & Pilán, Ildikó (2026). TAE: Text anonymization evaluator. SoftwareX. 35. doi: 10.1016/j.softx.2026.102851. Full text in Research Archive Show summary

  Text anonymization is essential for privacy-compliant data sharing, yet its evaluation remains challenging. Existing evaluation metrics are fragmented, and many rely on manual annotations, which hampers applicability and introduces subjectivity. We present the Text Anonymization Evaluator (TAE), a Python-based package that consolidates both traditional and state-of-the-art metrics for assessing privacy and utility in anonymized text. By integrating these metrics into a unified and extensible toolkit, TAE enables systematic, scalable, and objective evaluation and comparison of text anonymization methods.

• Manzanares-Salor, Benet; Sánchez, David & Lison, Pierre (2026). Unsupervised utility evaluation of text anonymization methods via neural language models. Neural Networks. ISSN 0893-6080. 202, p. 109079–109079. doi: 10.1016/j.neunet.2026.109079. Full text in Research Archive
• Papadopoulou, Anthi; Lison, Pierre; Anderson, Mark David; Øvrelid, Lilja & Pilán, Ildikó (2026). Neural text sanitization with privacy risk indicators: an empirical analysis. Language Resources and Evaluation. ISSN 1574-020X. 60. doi: 10.1007/s10579-026-09911-1. Full text in Research Archive Show summary

  Text sanitization is the task of redacting a document to mask all occurrences of (direct or indirect) personal identifiers, with the goal of concealing the identity of the individual(s) referred in it. In this paper, we consider a two-step approach to text sanitization and provide a detailed analysis of its empirical performance on two recently published datasets: the Text Anonymization Benchmark (Pilán et al., 2022) and a collection of Wikipedia biographies (Papadopoulou et al., 2022a). The text sanitization process starts with a privacy-oriented entity recognizer that seeks to determine the text spans expressing identifiable personal information. This privacy-oriented entity recognizer is trained by combining a standard named entity recognition model with a gazetteer populated by person-related terms extracted from Wikidata. The second step of the text sanitization process consists in assessing the privacy risk associated with each detected text span, either isolated or in combination with other text spans. We present five distinct indicators of the re-identification risk, respectively based on language model probabilities, text span classification, sequence labelling, perturbations, and web search. We provide a contrastive analysis of each privacy indicator and highlight their benefits and limitations, notably in relation to the available labeled data.

• Høst, Anders Mølmen; Lison, Pierre & Moonen, Leon (2026). A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs. In Yuan, Linlin; Xu, Minghui & Siew, Marie (Ed.), 2025 IEEE 24th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE (Institute of Electrical and Electronics Engineers). ISSN 9798331565336. p. 1598–1607. doi: 10.1109/Trustcom66490.2025.00186. Full text in Research Archive Show summary

  Vulnerability databases, such as the National Vulnerability Database (NVD), offer detailed descriptions of Common Vulnerabilities and Exposures (CVEs), but often lack information on their real-world impact, such as the tactics, techniques, and procedures (TTPs) that adversaries may use to exploit the vulnerability. However, manually linking CVEs to their corresponding TTPs is a challenging and time-consuming task, and the high volume of new vulnerabilities published annually makes automated support desirable.This paper introduces Triage, a two-pronged automated approach that uses Large Language Models (LLMs) to map CVEs to relevant techniques from the Att&ck knowledge base. We first prompt an LLM with instructions based on MITRE’s CVE Mapping Methodology to predict an initial list of techniques. This list is then combined with the results from a second LLM-based module that uses in-context learning to map a CVE to relevant techniques. This hybrid approach strategically combines rule-based reasoning with data-driven inference. Our evaluation reveals that in-context learning outperforms the individual mapping methods, and the hybrid approach improves recall of exploitation techniques. We also find that GPT-4o-mini performs better than Llama3.3-70B on this task. Overall, our results show that LLMs can be used to automatically predict the impact of cybersecurity vulnerabilities and Triage makes the process of mapping CVEs to Att&ck more efficient.

• Kennington, Casey; Lison, Pierre & Schlangen, David (2025). Prior Lessons of Incremental Dialogue and Robot Action Management for the Age of Language Models. Dialogue and Discourse. 16(3), p. 96–130. doi: 10.5210/dad.2025.305. Full text in Research Archive Show summary

  Efforts towards endowing robots with the ability to speak have benefited from recent advancements in natural language processing, in particular large language models. However, current language models are not fully incremental, as their processing is inherently monotonic and thus lack the ability to revise their interpretations or output in light of newer observations. This monotonicity has important implications for the development of dialogue systems for human–robot interaction. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and implications of incremental dialogue for embodied, robotic platforms in the age of large language models.

• Walker, Nicholas Thomas; Ultes, Stefan & Lison, Pierre (2025). Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human–Robot Interaction. Dialogue and Discourse. 16(3), p. 60–95. doi: 10.5210/dad.2025.304. Full text in Research Archive Show summary

  Knowledge graphs are often used to represent structured information in a flexible and efficient manner, but their use in situated dialogue remains under-explored. This paper presents a novel conversational model for human--robot interaction that rests upon a graph-based representation of the dialogue state. The knowledge graph representing the dialogue state is continuously updated with new observations from the robot sensors, including linguistic, situated and multimodal inputs, and is further enriched by other modules, in particular for spatial understanding. The neural conversational model employed to respond to user utterances relies on a simple but effective graph-to-text mechanism that traverses the dialogue state graph and converts the traversals into a natural language form. This conversion of the state graph into text is performed using a set of parameterized functions, and the values for those parameters are optimized based on a small set of Wizard-of-Oz interactions. After this conversion, the text representation of the dialogue state graph is included as part of the prompt of a large language model used to decode the agent response. The proposed approach is empirically evaluated through a user study with a humanoid robot that acts as conversation partner to evaluate the impact of the graph-to-text mechanism on the response generation. After moving a robot along a tour of an indoor environment, participants interacted with the robot using spoken dialogue and evaluated how well the robot was able to answer questions about what the robot observed during the tour. User scores suggest an improvement in the perceived factuality of the robot responses when the graph-to-text approach is employed compared to a baseline using inputs structured as semantic triples.

• Helstad, Gina; Lison, Pierre; Bjørnstad-Tuveng, Elin & Nytrøen, Kari (2025). Digitising health history: The creation, function and implementation of the Norwegian Health Archives Registry. Health Information Management Journal. ISSN 1833-3583. 55(1), p. 166–172. doi: 10.1177/18333583251389095. Full text in Research Archive Show summary

  Context: The Norwegian Health Archives Registry (NHAR) is a national initiative dedicated to digitising, centralising, and providing access to historical full-text patient health records (PHRs) for research purposes. Established in 2019, NHAR includes PHRs from the deceased population in Norway’s specialist healthcare services, offering a unique long-term data source for future research. NHAR has now digitised 1.7 million paper-based PHRs, covering medical history dating back to 1875. The registry is now expanding to include digital-born PHRs. Aim: This article describes NHAR’s innovation potential as a health registry, its data management processes, and the integration of artificial intelligence (AI) tools to facilitate data management and research in compliance with strict health data regulations. Practice innovation: NHAR’s data value chain includes structured metadata acquisition, large-scale digitisation and secure data delivery for research. The workflow includes a custom optical character recognition (OCR) tool tailored to Norwegian medical terminology, concept-based search tools for unstructured clinical full text and robust strategies for long-term data management. A novel AI-based de-identification system automatically detects and masks personal identifiers in digitised PHRs. Lessons learned: Despite these innovations, challenges persist in processing handwritten and historical PHRs due to OCR limitations and language-specific complexities. Key challenges include improving data quality, enhancing OCR accuracy and refining AI tools for information retrieval, data extraction and de-identification. Conclusion: NHAR offers significant potential for interdisciplinary research across various medical fields. Implications for health information management practice: NHAR establishes a foundation for secure access to historical health data and introduces advanced data management strategies to facilitate future research.

• Pilán, Ildikó; Manzanares-Salor, Benet; Sánchez, David & Lison, Pierre (2025). Truthful text sanitization guided by inference attacks. Applied Soft Computing. ISSN 1568-4946. 185. doi: 10.1016/j.asoc.2025.114013. Full text in Research Archive Show summary

  Text sanitization aims to rewrite parts of a document to prevent disclosure of personal information. The central challenge of text sanitization is to strike a balance between privacy protection (avoiding the leakage of personal information) and utility preservation (retaining as much as possible of the document’s original content). To this end, we introduce a novel text sanitization method based on generalizations, that is, broader but still informative terms that subsume the semantic content of the original text spans. The approach relies on the use of instruction-tuned large language models (LLMs) and is divided into two stages. Given a document including text spans expressing personally identifiable information (PII), the LLM is first applied to obtain truth-preserving replacement candidates for each text span and rank them according to their abstraction level. Those candidates are then evaluated for their ability to protect privacy by conducting inference attacks with the LLM. Finally, the system selects the most informative replacement candidate shown to be resistant to those attacks. This two-stage process produces replacements that effectively balance privacy and utility. We also present novel metrics to evaluate these two aspects without needing to manually annotate documents. Results on the Text Anonymization Benchmark show that the proposed approach, implemented with Mistral 7B Instruct, leads to enhanced utility, with only a marginal (<1 p.p.) increase in re-identification risk compared to fully suppressing the original spans. Furthermore, our approach is shown to be more truth-preserving than existing methods such as Microsoft Presidio’s synthetic replacements.

• Kåsene, Vebjørn Haug & Lison, Pierre (2025). Following Route Instructions using Large Vision-Language Models: A Comparison between Low-level and Panoramic Action Spaces. In Abbas, Mourad; Yousef, Tariq & Galke, Lukas (Ed.), Proceedings of the 8th International Conference on Natural Language and Speech Processing (ICNLSP-2025). Association for Computational Linguistics (ACL). ISSN 9798891762978. p. 449–463. doi: https:/aclanthology.org/2025.icnlsp-1.43/. Full text in Research Archive Show summary

  Vision-and-Language Navigation (VLN) refers to the task of enabling autonomous robots to navigate unfamiliar environments by following natural language instructions. While recent Large Vision-Language Models (LVLMs) have shown promise in this task, most current VLM systems rely on models specifically designed and optimized for navigation, leaving the potential of off-the-shelf LVLMs underexplored. Furthermore, while older VLN approaches used low-level action spaces with egocentric views and atomic actions (such as "turn left" or "move forward"), newer models tend to favor panoramic action spaces with discrete navigable viewpoints. This paper investigates (1) whether off-the-shelf LVLMs (fine-tuned without architectural modifications or simulator-based training) can effectively support VLN tasks and (2) whether such models can support both low-level and panoramic action paradigms. To this end, we fine-tune the open-source model Qwen2.5-VL-3B-Instruct on the Room-to-Room (R2R) dataset and evaluate its empirical performance across both low-level and panoramic action spaces. The best resulting model achieves a 41% success rate on the R2R test set, demonstrating that while off-the-shelf LVLMs can learn to perform Vision-and-Language Navigation, they still lag behind models specifically designed for this task.

• Walker, Nicholas Thomas; Lison, Pierre; Hilgendorf, Laetitia; Wagner, Nicolas & Ultes, Stefan (2025). Retrieving Relevant Knowledge Subgraphs for Task-Oriented Dialogue. In Béchet, Frédéric; Lefèvre, Fabrice; Asher, Nicholas; Kim, Seokhwan & Merlin, Teva (Ed.), Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics (ACL). ISSN 9798891763296. p. 513–526. doi: https:/aclanthology.org/2025.sigdial-1.42.pdf. Full text in Research Archive Show summary

  In this paper, we present an approach for extracting knowledge graph information for retrieval augmented generation in dialogue systems. Knowledge graphs are a rich source of background information, but the inclusion of more potentially useful information in a system prompt risks decreased model performance from excess context. We investigate a method of retrieving relevant subgraphs of maximum relevance and minimum size by framing this trade-off as a Prize-collecting Steiner Tree problem. The results of our user study and analysis indicate promising efficacy of a simple subgraph retrieval approach compared with a top-K retrieval model.

• Berg, Margareta; Pilán, Ildikó; Falkum, Ingrid Lossius & Lison, Pierre (2025). Pragmatic Reasoning for Irony Detection With Large Language Models in English and Norwegian - SEMDIAL. SemDial Proceedings. p. 204–209. doi: https:/www.semdial.org/anthology/Z25-Berg_semdial_4421.pdf. Full text in Research Archive Show summary

  This study investigates the ‘pragmatic abilities’ of large language models (LLMs) – both standard and reasoning-optimized – across two languages (English and Norwegian). Based on an existing experimental study on children’s irony comprehension, we found that LLMs largely identified irony, but performance was poorer in Norwegian due to translation challenges.

• Charpentier, Lucas Georges Gabriel & Lison, Pierre (2025). Re-identification of De-identified Documents with Autoregressive Infilling. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 1192–1209. doi: 10.18653/v1/2025.acl-long.60. Full text in Research Archive Show summary

  Documents revealing sensitive information about individuals must typically be de-identified. This de-identification is often done by masking all mentions of personally identifiable information (PII), thereby making it more difficult to uncover the identity of the person(s) in question. To investigate the robustness of de-identification methods, we present a novel, RAG-inspired approach that attempts the reverse process of re-identification based on a database of documents representing background knowledge. Given a text in which personal identifiers have been masked, the re-identification proceeds in two steps. A retriever first selects from the background knowledge passages deemed relevant for the re-identification. Those passages are then provided to an infilling model which seeks to infer the original content of each text span. This process is repeated until all masked spans are replaced. We evaluate the re-identification on three datasets (Wikipedia biographies, court rulings and clinical notes). Results show that (1) as many as 80% of de-identified text spans can be successfully recovered and (2) the re-identification accuracy increases along with the level of background knowledge.

• Kennington, Casey; Lison, Pierre & Schlangen, David (2025). Incremental Dialogue Management: Survey, Discussion, and Implications for HRI. arXiv. Full text in Research Archive Show summary

  Efforts towards endowing robots with the ability to speak have benefited from recent advancements in NLP, in particular large language models. However, as powerful as current models have become, they still operate on sentence or multi-sentence level input, not on the word-by-word input that humans operate on, affecting the degree of responsiveness that they offer, which is critical in situations where humans interact with robots using speech. In this paper, we review the literature on interactive systems that operate incrementally (i.e., at the word level or below it). We motivate the need for incremental systems, survey incremental modeling of important aspects of dialogue like speech recognition and language generation. Primary focus is on the part of the system that makes decisions, known as the dialogue manager. We find that there is very little research on incremental dialogue management, offer some requirements for practical incremental dialogue management, and the implications of incremental dialogue for embodied, robotic platforms.

• Manzanares-Salor, Benet; Sánchez, David & Lison, Pierre (2024). Evaluating the disclosure risk of anonymized documents via a machine learning-based re-identification attack. Data mining and knowledge discovery. ISSN 1384-5810. 38, p. 4040–4075. doi: 10.1007/s10618-024-01066-3. Full text in Research Archive Show summary

  The availability of textual data depicting human-centered features and behaviors is crucial for many data mining and machine learning tasks. However, data containing personal information should be anonymized prior making them available for secondary use. A variety of text anonymization methods have been proposed in the last years, which are standardly evaluated by comparing their outputs with human-based anonymizations. The residual disclosure risk is estimated with the recall metric, which quantifies the proportion of manually annotated re-identifying terms successfully detected by the anonymization algorithm. Nevertheless, recall is not a risk metric, which leads to several drawbacks. First, it requires a unique ground truth, and this does not hold for text anonymization, where several masking choices could be equally valid to prevent re-identification. Second, it relies on human judgements, which are inherently subjective and prone to errors. Finally, the recall metric weights terms uniformly, thereby ignoring the fact that the influence on the disclosure risk of some missed terms may be much larger than of others. To overcome these drawbacks, in this paper we propose a novel method to evaluate the disclosure risk of anonymized texts by means of an automated re-identification attack. We formalize the attack as a multi-class classification task and leverage state-of-the-art neural language models to aggregate the data sources that attackers may use to build the classifier. We illustrate the effectiveness of our method by assessing the disclosure risk of several methods for text anonymization under different attack configurations. Empirical results show substantial privacy risks for most existing anonymization methods.

• Hassan, Syed Zohaib; Lison, Pierre & Halvorsen, Pål (2024). Enhancing Naturalness in LLM-Generated Utterances through Disfluency Insertion. arXiv. doi: 10.48550/arXiv.2412.12710. Full text in Research Archive Show summary

  Disfluencies are a natural feature of spontaneous human speech but are typically absent from the outputs of Large Language Models (LLMs). This absence can diminish the perceived naturalness of synthesized speech, which is an important criteria when building conversational agents that aim to mimick human behaviours. We show how the insertion of disfluencies can alleviate this shortcoming. The proposed approach involves (1) fine-tuning an LLM with Low-Rank Adaptation (LoRA) to incorporate various types of disfluencies into LLM-generated utterances and (2) synthesizing those utterances using a text-to-speech model that supports the generation of speech phenomena such as disfluencies. We evaluated the quality of the generated speech across two metrics: intelligibility and perceived spontaneity. We demonstrate through a user study that the insertion of disfluencies significantly increase the perceived spontaneity of the generated speech. This increase came, however, along with a slight reduction in intelligibility.

• Pilán, Ildikó; Prévot, Laurent; Buschmeier, Hendrik & Lison, Pierre (2024). Conversational Feedback in Scripted versus Spontaneous Dialogues: A Comparative Analysis. In Kawahara, Tatsuya; Demberg, Vera; Ultes, Stefan; Inoue, Koji; Mehri, Shikib; Howcroft, David & Komatani, Kazunori (Ed.), Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics (ACL). ISSN 9798891761612. p. 440–457. doi: 10.18653/v1/2024.sigdial-1.38. Full text in Research Archive Show summary

  Scripted dialogues such as movie and TV subtitles constitute a widespread source of training data for conversational NLP models. However, there are notable linguistic differences between these dialogues and spontaneous interactions, especially regarding the occurrence of communicative feedback such as backchannels, acknowledgments, or clarification requests. This paper presents a quantitative analysis of such feedback phenomena in both subtitles and spontaneous conversations. Based on conversational data spanning eight languages and multiple genres, we extract lexical statistics, classifications from a dialogue act tagger, expert annotations and labels derived from a fine-tuned Large Language Model (LLM). Our main empirical findings are that (1) communicative feedback is markedly less frequent in subtitles than in spontaneous dialogues and (2) subtitles contain a higher proportion of negative feedback. We also show that dialogues generated by standard LLMs lie much closer to scripted dialogues than spontaneous interactions in terms of communicative feedback.

• Olstad, Annika Willoch; Papadopoulou, Anthi & Lison, Pierre (2023). Generation of Replacement Options in Text Sanitization. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. p. 292–300. Full text in Research Archive Show summary

  The purpose of text sanitization is to edit text documents to mask text spans that may directly or indirectly reveal personal information. An important problem in text sanitization is to find less specific, yet still informative replacements for each text span to mask. We present an approach to generate possible replacements using a combination of heuristic rules and an ontology derived from Wikidata. Those replacement options are hierarchically structured and cover various types of personal identifiers. Using this approach, we extend a recently released text sanitization dataset with manually selected replacements. The outcome of this data collection shows that the approach is able to suggest appropriate replacement options for most text spans.

• Høst, Anders Mølmen; Lison, Pierre & Moonen, Leon (2023). Constructing a Knowledge Graph from Textual Descriptions of Software Vulnerabilities in the National Vulnerability Database. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. p. 386–391. Full text in Research Archive Show summary

  Knowledge graphs have shown promise for several cybersecurity tasks, such as vulnerability assessment and threat analysis. In this work, we present a new method for constructing a vulnerability knowledge graph from information in the National Vulnerability Database (NVD). Our approach combines named entity recognition (NER), relation extraction (RE), and entity prediction using a combination of neural models, heuristic rules, and knowledge graph embeddings. We demonstrate how our method helps to fix missing entities in knowledge graphs used for cybersecurity and evaluate the performance.

• Barnes, Jeremy Claude; Touileb, Samia; Mæhlum, Petter & Lison, Pierre (2023). Identifying Token-Level Dialectal Features in Social Media. In Alumäe, Tanel & Fishel, Mark (Ed.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). University of Tartu. ISSN 9789916219997. Full text in Research Archive Show summary

  Dialectal variation is present in many human languages and is attracting a growing interest in NLP. Most previous work concentrated on either (1) classifying dialectal varieties at the document or sentence level or (2) performing standard NLP tasks on dialectal data. In this paper, we propose the novel task of token-level dialectal feature prediction. We present a set of fine-grained annotation guidelines for Norwegian dialects, expand a corpus of dialectal tweets, and manually annotate them using the introduced guidelines. Furthermore, to evaluate the learnability of our task, we conduct labeling experiments using a collection of baselines, weakly supervised and supervised sequence labeling models. The obtained results show that, despite the difficulty of the task and the scarcity of training data, many dialectal features can be predicted with reasonably high accuracy.

• Walker, Nicholas Thomas; Ultes, Stefan & Lison, Pierre (2023). Retrieval-Augmented Neural Response Generation Using Logical Reasoning and Relevance Scoring. SemDial Proceedings. Full text in Research Archive Show summary

  Constructing responses in task-oriented dialogue systems typically relies on information sources such the current dialogue state or external databases. This paper presents a novel approach to knowledge-grounded response generation that combines retrieval-augmented language models with logical reasoning. The approach revolves around a knowledge graph representing the current dialogue state and background information, and proceeds in three steps. The knowledge graph is first enriched with logically derived facts inferred using probabilistic logical programming. A neural model is then employed at each turn to score the conversational relevance of each node and edge of this extended graph. Finally, the elements with highest relevance scores are converted to a natural language form, and are integrated into the prompt for the neural conversational model employed to generate the system response. We investigate the benefits of the proposed approach on two datasets (KVRET and Graph-WOZ) along with a human evaluation. Experimental results show that the combination of (probabilistic) logical reasoning with conversational relevance scoring does increase both the factuality and fluency of the responses.

• Walker, Nicholas Thomas; Ultes, Stefan & Lison, Pierre (2023). A Graph-to-Text Approach to Knowledge-Grounded Response Generation in Human-Robot Interaction. arXiv. Full text in Research Archive Show summary

  Knowledge graphs are often used to represent structured information in a flexible and efficient manner, but their use in situated dialogue remains under-explored. This paper presents a novel conversational model for human--robot interaction that rests upon a graph-based representation of the dialogue state. The knowledge graph representing the dialogue state is continuously updated with new observations from the robot sensors, including linguistic, situated and multimodal inputs, and is further enriched by other modules, in particular for spatial understanding. The neural conversational model employed to respond to user utterances relies on a simple but effective graph-to-text mechanism that traverses the dialogue state graph and converts the traversals into a natural language form. This conversion of the state graph into text is performed using a set of parameterized functions, and the values for those parameters are optimized based on a small set of Wizard-of-Oz interactions. After this conversion, the text representation of the dialogue state graph is included as part of the prompt of a large language model used to decode the agent response. The proposed approach is empirically evaluated through a user study with a humanoid robot that acts as conversation partner to evaluate the impact of the graph-to-text mechanism on the response generation. After moving a robot along a tour of an indoor environment, participants interacted with the robot using spoken dialogue and evaluated how well the robot was able to answer questions about what the robot observed during the tour. User scores show a statistically significant improvement in the …

• Walker, Nicholas Thomas; Dahl, Torbjørn & Lison, Pierre (2022). Dialogue Management as Graph Transformations. In Stoyanchev, Svetlana; Ultes, Stephan & Li, Haizhou (Ed.), Conversational AI for Natural Human-Centric Interaction. Springer. ISSN 9789811955372. p. 219–227. doi: 10.1007/978-981-19-5538-9_15. Full text in Research Archive Show summary

  We present ongoing work on a new dialogue management framework using graphs as core representation for the current dialogue state. Dialogue management tasks such as state tracking and action selection are framed as sequences of graph transformations that repeatedly update this graph based on incoming observations. Those graph transformations are expressed using a graph query language, making it possible to specify all dialogue management operations through a unified, declarative syntax. We argue that graphs are particularly well suited to model the dialogue state of complex, open-ended domains. In contrast to traditional dialogue state representations that are limited to fixed, predefined slots, graphs can naturally express dialogue domains with rich relational structures and variable numbers of entities to track. We describe how dialogue state tracking and action selection can be modelled in such graph-centric view of dialogue management, using either handcrafted rules or data-driven models. We also briefly discuss how to account for some aspects of dialogue management such as uncertainties, incremental inputs and contextual knowledge. Finally, we describe a proof-of-concept study of this dialogue management framework in a human–robot interaction scenario.

• Hassan, Syed Zohaib; Salehi, Pegah; Røed, Ragnhild Klingenberg; Halvorsen, Pål; Baugerud, Gunn Astrid & Johnson, Miriam S. [Show all 11 contributors for this article] (2022). Towards an AI-driven talking avatar in virtual reality for investigative interviews of children. In Claypool, Mark & Eg, Ragnhild (Ed.), GameSys '22: Proceedings of the 2nd Workshop on Games Systems. Association for Computing Machinery (ACM). ISSN 9781450393812. p. 9–15. doi: 10.1145/3534085.3534340. Full text in Research Archive Show summary

  Artificial intelligence (AI) and gaming systems have advanced to the stage where the current models and technologies can be used to address real-world problems. The development of such systems comes with different challenges, e.g., most of them related to system performance, complexity and user testing. Using a virtual reality (VR) environment, we have designed and developed a game-like system aiming to mimic an abused child that can help to assist police and child protection service (CPS) personnel in interview training of maltreated children. Current research in this area points to the poor quality of conducted interviews, and emphasises the need for better training methods. Information obtained in these interviews is the core piece of evidence in the prosecution process. We utilised advanced dialogue models, talking visual avatars, and VR to build a virtual child avatar that can interact with users. We discuss our proposed architecture and the performance of the developed child avatar prototype, and we present the results from the user study conducted with CPS personnel. The user study investigates the users' perceived quality of experience (QoE) and their learning effects. Our study confirms that such a gaming system can increase the knowledge and skills of the users. We also benchmark and discuss the system performance aspects of the child avatar. Our results show that the proposed prototype works well in practice and is well received by the interview experts.

• Pilán, Ildikó; Lison, Pierre; Øvrelid, Lilja; Papadopoulou, Anthi; Sánchez, David & Batet, Montserrat (2022). The text anonymization benchmark (TAB): A dedicated corpus and evaluation framework for text anonymization. Computational Linguistics. ISSN 0891-2017. 48(4), p. 1053–1101. doi: 10.1162/coli_a_00458. Full text in Research Archive Show summary

  We present a novel benchmark and associated evaluation metrics for assessing the performance of text anonymization methods. Text anonymization, defined as the task of editing a text document to prevent the disclosure of personal information, currently suffers from a shortage of privacy-oriented annotated text resources, making it difficult to properly evaluate the level of privacy protection offered by various anonymization methods. This paper presents TAB (Text Anonymization Benchmark), a new, open-source annotated corpus developed to address this shortage. The corpus comprises 1,268 English-language court cases from the European Court of Human Rights (ECHR) enriched with comprehensive annotations about the personal information appearing in each document, including their semantic category, identifier type, confidential attributes, and co-reference relations. Compared with previous work, the TAB corpus is designed to go beyond traditional de-identification (which is limited to the detection of predefined semantic categories), and explicitly marks which text spans ought to be masked in order to conceal the identity of the person to be protected. Along with presenting the corpus and its annotation layers, we also propose a set of evaluation metrics that are specifically tailored toward measuring the performance of text anonymization, both in terms of privacy protection and utility preservation. We illustrate the use of the benchmark and the proposed metrics by assessing the empirical performance of several baseline text anonymization models. The full corpus along with its privacy-oriented annotation guidelines, evaluation scripts, and baseline models are available on: https://github.com/NorskRegnesentral/text-anonymization-benchmark.

• Papadopoulou, Anthi; Yu, Yunhao; Lison, Pierre & Øvrelid, Lilja (2022). Neural Text Sanitization with Explicit Measures of Privacy Risk, The 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. Association for Computational Linguistics (ACL). ISSN 9781955917650. p. 217–229. Full text in Research Archive Show summary

  We present a novel approach for text sanitization, which is the task of editing a document to mask all (direct and indirect) personal identifiers and thereby conceal the identity of the individuals(s) mentioned in the text. In contrast to previous work, the approach relies on explicit measures of privacy risk, making it possible to explicitly control the trade-off between privacy protection and data utility. The approach proceeds in three steps. A neural, privacy-enhanced entity recognizer is first employed to detect and classify potential personal identifiers. We then determine which entities, or combination of entities, are likely to pose a re-identification risk through a range of privacy risk assessment measures. We present three such measures of privacy risk, respectively based on (1) span probabilities derived from a BERT language model, (2) web search queries and (3) a classifier trained on labelled data. Finally, a linear optimization solver decides which entities to mask to minimize the semantic loss while simultaneously ensuring that the estimated privacy risk remains under a given threshold. We evaluate the approach both in the absence and presence of manually annotated data. Our results highlight the potential of the approach, as well as issues specific types of personal data can introduce to the process.

• Papadopoulou, Anthi; Lison, Pierre; Øvrelid, Lilja & Pilán, Ildikó (2022). Bootstrapping Text Anonymization Models with Distant Supervision. In Calzolari, Nicoletta; Béchet, Frédéric; Blache, Philippe; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Odijk, Jan & Piperidis, Stelios (Ed.), Proceedings of the Thirteenth Language Resources and Evaluation Conference. European Language Resources Association. ISSN 9791095546726. p. 4477–4487. Full text in Research Archive Show summary

  We propose a novel method to bootstrap text anonymization models based on distant supervision. Instead of requiring manually labeled training data, the approach relies on a knowledge graph expressing the background information assumed to be publicly available about various individuals. This knowledge graph is employed to automatically annotate text documents including personal data about a subset of those individuals. More precisely, the method determines which text spans ought to be masked in order to guarantee k-anonymity, assuming an adversary with access to both the text documents and the background information expressed in the knowledge graph. The resulting collection of labeled documents is then used as training data to fine-tune a pre-trained language model for text anonymization. We illustrate this approach using a knowledge graph extracted from Wikidata and short biographical texts from Wikipedia. Evaluation results with a RoBERTa-based model and a manually annotated collection of 553 summaries showcase the potential of the approach, but also unveil a number of issues that may arise if the knowledge graph is noisy or incomplete. The results also illustrate that, contrary to most sequence labeling problems, the text anonymization task may admit several alternative solutions.

• Manzanares-Salor, Benet; Sánchez, David & Lison, Pierre (2022). Automatic Evaluation of Disclosure Risks of Text Anonymization Methods. In Domingo-Ferrer, Josep & Laurent, Maryline (Ed.), Privacy in Statistical Databases. Springer. ISSN 9783031139444. p. 157–171. doi: 10.1007/978-3-031-13945-1_12. Full text in Research Archive Show summary

  The standard approach to evaluate text anonymization methods consists of comparing their outcomes with the anonymization performed by human experts. The degree of privacy protection attained is then measured with the IR-based recall metric, which expresses the proportion of re-identifying terms that were correctly detected by the anonymization method. However, the use of recall to estimate the degree of privacy protection suffers from several limitations. The first is that it assigns a uniform weight to each re-identifying term, thereby ignoring the fact that some missed re-identifying terms may have a larger influence on the disclosure risk than others. Furthermore, IR-based metrics assume the existence of a single gold standard annotation. This assumption does not hold for text anonymization, where several maskings (each one encompassing a different combination of terms) could be equally valid to prevent disclosure. Finally, those metrics rely on manually anonymized datasets, which are inherently subjective and may be prone to various errors, omissions and inconsistencies. To tackle these issues, we propose an automatic re-identification attack for (anonymized) texts that provides a realistic assessment of disclosure risks. Our method follows a similar premise as the well-known record linkage methods employed to evaluate anonymized structured data, and leverages state-of-the-art deep learning language models to exploit the background knowledge available to potential attackers. We also report empirical evaluations of several well-known methods and tools for text anonymization. Results show significant re-identification risks for all methods, including also manual anonymization efforts.

• Weitzenboeck, Emily Mary; Lison, Pierre; Cyndecka, Malgorzata Agnieszka & Langford, Malcolm (2022). The GDPR and Unstructured Data: Is Anonymisation Possible? International Data Privacy Law (IDPL). ISSN 2044-3994. 12(3), p. 184–206. doi: 10.1093/idpl/ipac008. Full text in Research Archive Show summary

  Much of the legal and technical literature on data anonymization has focused on structured data such as tables. However, unstructured data such as text documents or images are far more common, and the legal requirements that must be fulfilled to properly anonymize such data formats remain unclear and underaddressed by the literature. In the absence of a definition of the term ‘anonymous data’ in the General Data Protection Regulation (GDPR), we examine its antithesis—personal data—and the identifiability test in Recital 26 GDPR to understand what conditions must be in place for the anonymization of unstructured data. This article examines the two contrasting approaches for determining identifiability that are prevalent today: (i) the risk-based approach and (ii) the strict approach in the Article 29 Working Party’s Opinion on Anonymization Techniques (WP 216). Through two case studies, we illustrate the challenges encountered when trying to anonymize unstructured datasets. We show that, while the risk-based approach offers a more nuanced test consistent with the purposes of the GDPR, the strict approach of WP 216 makes anonymization of unstructured data virtually impossible as long as the original data continues to exist. The concluding section considers the policy implications of the strict approach and technological developments that assist identification, and proposes a way forward.

• Lison, Pierre; Barnes, Jeremy & Hubin, Aliaksandr (2021). skweak: Weak Supervision Made Easy for NLP. In Ji, Heng & Park, Jong (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. Association for Computational Linguistics (ACL). ISSN 9781954085565. p. 337–346. doi: 10.18653/v1/2021.acl-demo.40. Full text in Research Archive Show summary

  We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak

• Olsen, Joakim; Næss, Arild Brandrud & Lison, Pierre (2021). Assessing the Quality of Human-Generated Summaries with Weakly Supervised Learning. In Dobnik, Simon & Øvrelid, Lilja (Ed.), Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789179296148. p. 112–123. Full text in Research Archive Show summary

  This paper explores how to automatically measure the quality of human-generated summaries, based on a Norwegian corpus of real estate condition reports and their corresponding summaries. The proposed approach proceeds in two steps. First, the real estate reports and their associated summaries are automatically labelled using a set of heuristic rules gathered from human experts and aggregated using weak supervision. The aggregated labels are then employed to learn a neural model that takes a document and its summary as inputs and outputs a score reflecting the predicted quality of the summary. The neural model maps the document and its summary to a shared “summary content space” and computes the cosine similarity between the two document embeddings to predict the final summary quality score. The best performance is achieved by a CNN-based model with an accuracy (measured against the aggregated labels obtained via weak supervision) of 89.5%, compared to 72.6% for the best unsupervised model. Manual inspection of examples indicate that the weak supervision labels do capture important indicators of summary quality, but the correlation of those labels with human judgements remains to be validated. Our models of summary quality predict that approximately 30% of the real estate reports in the corpus have a summary of poor quality.

• Lison, Pierre; Pilán, Ildikó; Sánchez, David; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. In Zong, Chengqing; Xia, Fei; Li, Wenjie & Navigli, Roberto (Ed.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9781954085527. p. 4188–4203. doi: 10.18653/v1/2021.acl-long.323. Full text in Research Archive Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Lison, Pierre; Barnes, Jeremy; Hubin, Aliaksandr & Touileb, Samia (2020). Named Entity Recognition without Labelled Data: A Weak Supervision Approach. In Jurafsky, Dan; Chai, Joyce; Schluter, Natalie & Tetreault, Joel (Ed.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics (ACL). ISSN 9781952148255. p. 1518–1533. Full text in Research Archive Show summary

  Named Entity Recognition (NER) performance often degrades rapidly when applied to target domains that differ from the texts observed during training. When in-domain labelled data is available, transfer learning techniques can be used to adapt existing NER models to the target domain. But what should one do when there is no hand-labelled data for the target domain? This paper presents a simple but powerful approach to learn NER models in the absence of labelled data through weak supervision. The approach relies on a broad spectrum of labelling functions to automatically annotate texts from the target domain. These annotations are then merged together using a hidden Markov model which captures the varying accuracies and confusions of the labelling functions. A sequence labelling model can finally be trained on the basis of this unified annotation. We evaluate the approach on two English datasets (CoNLL 2003 and news articles from Reuters and Bloomberg) and demonstrate an improvement of about 7 percentage points in entity-level F1 scores compared to an out-of-domain neural NER model.

• Jang, Youngsoo; Lee, Jongmin; Park, Jaeyoung; Lee, Kyeng-Hun; Lison, Pierre & Kee-Eung, Kim (2019). PyOpenDial: A Python-based Domain-Independent Toolkit for Developing Spoken Dialogue Systems with Probabilistic Rules. In Huang, Ruihong & Padó, Sebastian (Ed.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019), System Demonstrations. Association for Computational Linguistics (ACL). ISSN 9781950737925. p. 187–192. doi: 10.18653/v1/D19-3032. Full text in Research Archive Show summary

  Welcome to the proceedings of the system demonstrations session. This volume contains the papers of the system demonstrations presented at the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), which was held in Hong Kong, China on November 3-7, 2019. The system demonstrations session includes papers describing systems ranging from early research prototypes to mature production-ready software. We received 110 submissions, 22 of which were either invalid or withdrawn by the authors. Of the 88 valid submissions, 44 were selected for inclusion in the proceedings after review of three members of the program committee, achieving an overall acceptance rate of 40%. We thank all authors for their submissions, and the 130 members of the program committee for their timely and thoughtful reviews. In addition, EMNLP-IJCNLP 2019 has a Best Demo Award for the first time, and the best paper award was selected among outstanding papers following a second round of reviewing by five senior members of the program committee who are due a special recognition for their efforts. Ruihong Huang and Sebastian Padó EMNLP-IJCNLP 2019 System Demonstration Co-Chairs

• Lison, Pierre & Dogruöz, A. Seza (2018). Detecting Machine-translated Documents in Large Parallel Corpora. In Rapp, Reinhard; Zweigenbaum, Pierre & Sharoff, Serge (Ed.), 11th Workshop on Building and Using Comparable Corpora (BUCC 2018). European Language Resources Association. ISSN 9791095546078. p. 25–32. Full text in Research Archive Show summary

  Parallel corpora extracted from online repositories of movie and TV subtitles are employed in a wide range of NLP applications, from language modelling to machine translation and dialogue systems. However, the subtitles uploaded in such repositories exhibit varying levels of quality. A particularly difficult problem stems from the fact that a substantial number of these subtitles are not written by human subtitlers but are simply generated through the use of online translation engines. This paper investigates whether these machine-generated subtitles can be detected automatically using a combination of linguistic and extra-linguistic features. We show that a feedforward neural network trained on a small dataset of subtitles can detect machine-generated subtitles with a F1-score of 0.64. Furthermore, applying this detection model on an unlabelled sample of subtitles allows us to provide a statistical estimate for the proportion of subtitles that are machine-translated (or are at least of very low quality) in the full corpus.

• Lison, Pierre; Tiedemann, Jörg & Kouylekov, Milen (2018). OpenSubtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Calzolari, Nicoletta; Choukri, Khalid; Cieri, Christopher; Declerck, Thierry; Goggi, Sara; Hasida, Koiti; Isahara, Hitoshi; Maegaard, Bente; Mariani, Joseph; Mazo, Hélène; Moreno, Asuncion; Odijk, Jan; Piperidis, Stelios & Tokunaga, Takenobu (Ed.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association. ISSN 9791095546009. p. 1742–1748. Full text in Research Archive Show summary

  Movie and TV subtitles are a highly valuable resource for the compilation of parallel corpora thanks to their availability in large numbers and across many languages. However, the quality of the resulting sentence alignments is often lower than for other parallel corpora. This paper presents a new major release of the OpenSubtitles collection of parallel corpora, which is extracted from a total of 3.7 million subtitles spread over 60 languages. In addition to a substantial increase in the corpus size (about 30 % compared to the previous version), this new release associates explicit quality scores to each sentence alignment. These scores are determined by a statistical regression model based on simple language-independent features and estimated on a small sample of aligned sentence pairs. Evaluation results show that the model is able predict lexical translation probabilities with a root mean square error of 0.07 (coefficient of determination R2 = 0.47). Based on the scores produced by this regression model, the parallel corpora can be filtered to prune out alignments with a score below a given threshold.

• Lison, Pierre & Mavroeidis, Vasileios (2017). Automatic Detection of Malware-Generated Domains with Recurrent Neural Models. Norsk Informasjonssikkerhetskonferanse (NISK). ISSN 1893-6563. Full text in Research Archive Show summary

  Modern malware families often rely on domain-generation algorithms (DGAs) to determine rendezvous points to their command-and-control server. Traditional defence strategies (such as blacklisting domains or IP addresses) are inadequate against such techniques due to the large and continuously changing list of domains produced by these algorithms. This paper demonstrates that a machine learning approach based on recurrent neural networks is able to detect domain names generated by DGAs with high precision. The neural models are estimated on a large training set of domains generated by various malwares. Experimental results show that this data-driven approach can detect malware-generated domain names with a F1 score of 0.971. To put it differently, the model can automatically detect 93 % of malware-generated domain names for a false positive rate of 1:100.

• Lison, Pierre & Kutuzov, Andrei (2017). Redefining Context Windows for Word Embedding Models: An Experimental Study. In Tiedemann, Jörg (Eds.), Proceedings of the 21st Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press. ISSN 9789176856017. p. 284–288. Full text in Research Archive Show summary

  Distributional semantic models learn vector representations of words through the contexts they occur in. Although the choice of context (which often takes the form of a sliding window) has a direct influence on the resulting embeddings, the exact role of this model component is still not fully understood. This paper presents a systematic analysis of context windows based on a set of four distinct hyperparameters. We train continuous Skip- Gram models on two English-language corpora for various combinations of these hyper-parameters, and evaluate them on both lexical similarity and analogy tasks. Notable experimental results are the positive impact of cross-sentential contexts and the surprisingly good performance of right-context windows.

• Lison, Pierre & Mavroeidis, Vasileios (2017). Neural Reputation Models learned from Passive DNS data, IEEE Big Data 1st International Workshop on Big Data Analytic for Cyber Crime Investigation and Prevention 2017. IEEE (Institute of Electrical and Electronics Engineers). ISSN 9781538627150. p. 3662–3671. doi: 10.1109/BigData.2017.8258361. Full text in Research Archive Show summary

  Blacklists and whitelists are often employed to filter outgoing and incoming traffic on computer networks. One central function of these lists is to mitigate the security risks posed by malware threats by associating a reputation (for instance benign or malicious) to end-point hosts. The creation and maintenance of these lists is a complex and time-consuming process for security experts. As a consequence, blacklists and whitelists are prone to various errors, inconsistencies and omissions, as only a tiny fraction of end-point hosts are effectively covered by the reputation lists. In this paper, we present a machine learning model that is able to automatically detect whether domain names and IP addresses are benign, malicious or sinkholes. The model relies on a deep neural architecture and is trained on a large passive DNS database. Evaluation results demonstrate the effectiveness of the approach, as the model is able to detect malicious DNS records with a F1 score of 0.96. In other words, the model is able to detect 95 % of the malicious hosts with a false positive rate of 1:1000.

View all works in NVA

• Lison, Pierre (2022). Dis, c'est quoi l'intelligence artificielle? Renaissance Du Livre. ISBN 9782507057299. 96 p. Full text in Research Archive Show summary

  L’intelligence artificielle est aujourd’hui sur toutes les lèvres. Certains y voient une révolution technologique qui viendra nous libérer de nombreuses tâches pénibles, répétitives ou dangereuses et transformera notre rapport au monde et au travail. Pour d’autres, elle représente au contraire un danger pour l’équilibre déjà fragile de nos sociétés, voire une menace existentielle pour l’humanité. Mais de quoi s’agit-il au juste ? Cet ouvrage aborde de manière pédagogique les idées-clés qui sous-tendent l’intelligence artificielle. Comment une machine peut-elle apprendre, raisonner et résoudre par elle-même des problèmes complexes ? Quelles sont les applications pratiques de l’intelligence artificielle ? Et en quoi l’intelligence d’une machine se différencie-t-elle du fonctionnement de notre cerveau humain ? Au fil d’une conversation, ce livre permet de mieux comprendre cette discipline au confluent de l’informatique, des mathématiques et des sciences cognitives.

View all works in NVA

• Lison, Pierre; Ruenes, David Sánchez & Stalla-Bourdillon, Sophie (2026). Search Data, Privacy, and the Limits of Heuristics: A Critical Reading of the EC's Preliminary Findings against Alphabet. Full text in Research Archive
• Lison, Pierre & Schwemer, Sebastian Felix (2026). Ideen om en innholdsavgift for KI brer seg. Full text in Research Archive Show summary

  Å kompensere mennesker som bidrar med originalt innhold, handler ikke bare om rettferdighet. Det er også en investering i et bærekraftig, digitalt økosystem.

• Høst, Anders Mølmen; Lison, Pierre & Moonen, Leon (2025). Replication package for "A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs". doi: 10.5281/zenodo.17341503. Full text in Research Archive Show summary

  This repository contains the replication package for the paper "A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs", by Anders Mølmen Høst, Pierre Lison, and Leon Moonen, accepted for publication in the 24th IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom).

• Høst, Anders Mølmen; Lison, Pierre & Moonen, Leon (2025). A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs. doi: 10.48550/arXiv.2508.18439. Full text in Research Archive Show summary

  Vulnerability databases, such as the National Vulnerability Database (NVD), offer detailed descriptions of Common Vulnerabilities and Exposures (CVEs), but often lack information on their real-world impact, such as the tactics, techniques, and procedures (TTPs) that adversaries may use to exploit the vulnerability. However, manually linking CVEs to their corresponding TTPs is a challenging and time-consuming task, and the high volume of new vulnerabilities published annually makes automated support desirable. This paper introduces TRIAGE, a two-pronged automated approach that uses Large Language Models (LLMs) to map CVEs to relevant techniques from the ATT&CK knowledge base. We first prompt an LLM with instructions based on MITRE’s CVE Mapping Methodology to predict an initial list of techniques. This list is then combined with the results from a second LLM-based module that uses in-context learning to map a CVE to relevant techniques. This hybrid approach strategically combines rule-based reasoning with data-driven inference. Our evaluation reveals that in-context learning outperforms the individual mapping methods, and the hybrid approach improves recall of exploitation techniques. We also find that GPT-4o-mini performs better than Llama3.3-70B on this task. Overall, our results show that LLMs can be used to automatically predict the impact of cybersecurity vulnerabilities and TRIAGE makes the process of mapping CVEs to ATT&CK more efficient.

• Høst, Anders Mølmen; Lison, Pierre & Moonen, Leon (2025). A Systematic Approach to Predict the Impact of Cybersecurity Vulnerabilities Using LLMs. Full text in Research Archive Show summary

  Vulnerability databases, such as the National Vulnerability Database (NVD), offer detailed descriptions of Common Vulnerabilities and Exposures (CVEs), but often lack information on their real-world impact, such as the tactics, techniques, and procedures (TTPs) that adversaries may use to exploit the vulnerability. However, manually linking CVEs to their corresponding TTPs is a challenging and time-consuming task, and the high volume of new vulnerabilities published annually makes automated support desirable. This paper introduces TRIAGE, a two-pronged automated approach that uses Large Language Models (LLMs) to map CVEs to relevant techniques from the ATT&CK knowledge base. We first prompt an LLM with instructions based on MITRE's CVE Mapping Methodology to predict an initial list of techniques. This list is then combined with the results from a second LLM-based module that uses in-context learning to map a CVE to relevant techniques. This hybrid approach strategically combines rule-based reasoning with data-driven inference. Our evaluation reveals that in-context learning outperforms the individual mapping methods, and the hybrid approach improves recall of exploitation techniques. We also find that GPT-4o-mini performs better than Llama3.3-70B on this task. Overall, our results show that LLMs can be used to automatically predict the impact of cybersecurity vulnerabilities and TRIAGE makes the process of mapping CVEs to ATT&CK more efficient. A replication package is available for download from https://doi.org/10.5281/zenodo.17341503. Keywords: vulnerability impact, CVE, ATT&CK techniques, large language models, automated mapping.

• Lison, Pierre (2024). Nå kan KI-generert tekst vannmerkes. Full text in Research Archive
• Falkum, Ingrid Lossius & Lison, Pierre (2023). Er prateroboten ChatGPT en klok samtale­partner eller papegøye? Full text in Research Archive Show summary

  https://forskersonen.no/chatgpt-kronikk-kunstig-intelligens/er-prateroboten-chatgpt-en-klok-samtalepartner-eller-papegoye/2157206

• Lison, Pierre & Touileb, Samia (2023). Chat GPT egner seg dårlig til eksamenssensuren. Full text in Research Archive
• Korsvoll, Nils Hallvard; Lison, Pierre; Reinertsen, Hilde; Elken, Mari; Korne, Haley De & Hansen, Kai Arne [Show all 7 contributors for this article] (2023). Fire tiltak for en bedre språkpolitikk i akademia. Full text in Research Archive
• Lison, Pierre & Kennington, Casey (2023). Who's in Charge? Roles and Responsibilities of Decision-Making Components in Conversational Robots. Full text in Research Archive
• Walker, Nicholas Thomas & Lison, Pierre (2023). GraphWOZ: Dialogue Management with Conversational Knowledge Graphs. Full text in Research Archive
• Riegler, Michael; Lison, Pierre & Falkum, Ingrid Lossius (2023). Kunstig intelligens-modeller er ikke miniatyr­versjoner av den menneskelige hjernen. Full text in Research Archive
• Lison, Pierre (2023). Venn med kunstig intelligens. [TV]. NRK Dagsrevyen. Full text in Research Archive
• Lison, Pierre (2023). Kunstig Intelligens, en fare for menneskeheten? [Radio]. NRK P2. Full text in Research Archive
• Engebretsen, Solveig; Løland, Anders & Lison, Pierre (2023). Alt du kan lære om statistisk modellering og maskinlæring på en dag. Full text in Research Archive
• Engebretsen, Solveig; Løland, Anders & Lison, Pierre (2023). Alt du kan lære om statistisk modellering og maskinlæring på en dag. Full text in Research Archive
• Løland, Anders; Fuglerud, Kristin Skeide & Lison, Pierre (2022). Hva er universell utforming? [Internet]. Sannsynligvis VIKTIG . Full text in Research Archive
• Lund, Bjarte Aarmo; Sandsør, Astrid Marie Jorde & Lison, Pierre (2022). Problemer på kontoret: Alltid jeg som må trakte kaffe. Full text in Research Archive
• Lison, Pierre; Rots, Aike Peter & Korne, Haley De (2022). Hvilket fremmedspråk bør man lære seg i Google-oversettelsenes tidsalder? Full text in Research Archive
• Lison, Pierre (2022). Anonymization of sensitive information. Full text in Research Archive
• Lison, Pierre; Korsvoll, Nils Hallvard & Lundberg, Aase-Kristine Aasen (2022). Kjernekraft -er det farlig, eller er det fremtiden? Full text in Research Archive
• Korsvoll, Nils Hallvard; Elken, Mari; Sandtorv, Alexander Harald & Lison, Pierre (2021). Welcome to Norway! Full text in Research Archive
• Lison, Pierre (2021). Språkteknologi: siste trender og vanlige fallgruver. Full text in Research Archive
• Lison, Pierre; Bølstad, Jørgen & Kvellestad, Anders (2021). Forvirrende pandemistatistikk: Hva skal vi med logaritmer i grafer? Full text in Research Archive
• Lison, Pierre (2021). Skweak: Weak Supervision Made Easy for NLP. Full text in Research Archive
• Lison, Pierre (2021). Fremdrift i forskningsprosjekter. Full text in Research Archive
• Lison, Pierre; Pilán, Ildikó; Øvrelid, Lilja; Ruenes, David Sánchez & Batet, Montserrat (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Full text in Research Archive Show summary

  This position paper investigates the problem of automated text anonymisation, which is a prerequisite for secure sharing of documents containing sensitive information about individuals. We summarise the key concepts behind text anonymisation and provide a review of current approaches. Anonymisation methods have so far been developed in two fields with little mutual interaction, namely natural language processing and privacy-preserving data publishing. Based on a case study, we outline the benefits and limitations of these approaches and discuss a number of open challenges, such as (1) how to account for multiple types of semantic inferences, (2) how to strike a balance between disclosure risk and data utility and (3) how to evaluate the quality of the resulting anonymisation. We lay out a case for moving beyond sequence labelling models and incorporate explicit measures of disclosure risk into the text anonymisation process.

• Lison, Pierre; Pilán, Ildikó; Ruenes, David Sánchez; Batet, Montserrat & Øvrelid, Lilja (2021). Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. Full text in Research Archive
• Lison, Pierre; Barnes, Jeremy Claude & Hubin, Aliaksandr (2021). skweak: weak supervision made easy for NLP. Full text in Research Archive Show summary

  We present skweak, a versatile, Python-based software toolkit enabling NLP developers to apply weak supervision to a wide range of NLP tasks. Weak supervision is an emerging machine learning paradigm based on a simple idea: instead of labelling data points by hand, we use labelling functions derived from domain knowledge to automatically obtain annotations for a given dataset. The resulting labels are then aggregated with a generative model that estimates the accuracy (and possible confusions) of each labelling function. The skweak toolkit makes it easy to implement a large spectrum of labelling functions (such as heuristics, gazetteers, neural models or linguistic constraints) on text data, apply them on a corpus, and aggregate their results in a fully unsupervised fashion. skweak is especially designed to facilitate the use of weak supervision for NLP tasks such as text classification and sequence labelling. We illustrate the use of skweak for NER and sentiment analysis. skweak is released under an open-source license and is available at https://github.com/NorskRegnesentral/skweak

• Lison, Pierre (2021). Vi må snakke om Bitcoin. Full text in Research Archive
• Lison, Pierre & Falkum, Ingrid Lossius (2020). Hva skjedde med «Don’t be evil»? Full text in Research Archive
• Løland, Anders; Lison, Pierre & Falkum, Ingrid Lossius (2020). Episode 6: Kan språkteknologi virkelig forstå språk? Med Ingrid Lossius Falkum og Pierre Lison. [Internet]. Sannsynligvis VIKTIG (podkast). Full text in Research Archive
• Løland, Anders & Lison, Pierre (2020). Episode 5: Hva er språkteknologi (eller NLP)? Med Pierre Lison. [Internet]. Sannsynligvis VIKTIG (podkast). Full text in Research Archive
• Lison, Pierre (2020). Ethical and social impacts of AI. Full text in Research Archive
• Lison, Pierre (2020). Developing NLP models without labelled data using weak supervision. Full text in Research Archive
• Riegler, Michael; Lison, Pierre; Strümke, Inga & Løland, Anders (2020). For enkelt om kunstig intelligens: – Diskriminerende og fordomsfull AI er ikke alltid lett å løse. Forskning.no. ISSN 1891-635X. Full text in Research Archive
• Lison, Pierre & Falkum, Ingrid Lossius (2020). Kan kunstig intelligens "forstå" språk? Aftenposten (morgenutg. : trykt utg.). ISSN 0804-3116. Full text in Research Archive
• Lison, Pierre (2020). Named Entity Recognition without Labelled Data: A Weak Supervision Approach. Full text in Research Archive
• Lison, Pierre (2019). Data-driven models of reputation for cybersecurity. Full text in Research Archive
• Lison, Pierre (2019). Dialogue Modelling: Small data, Big data. Full text in Research Archive
• Lison, Pierre (2019). Open challenges in anonymisation. Full text in Research Archive
• Lison, Pierre (2019). Modélisation du dialogue: contrôle du dialogue et corpus multilingues. Full text in Research Archive
• Lison, Pierre (2019). Modellering av omdømme i cybersikkerhet med nevralske nettverk. Full text in Research Archive
• Prévot, Laurent; Magistry, Pierre & Lison, Pierre (2019). Should we use movie subtitles to study linguistic patterns of conversational speech? A study based on French, English and Taiwan Mandarin. Full text in Research Archive
• Lison, Pierre (2018). Data-driven models of reputation in cyber-security. Full text in Research Archive Show summary

  In this talk, I will present our work on developing data-driven, predictive models of reputation (such as benign or malicious) for end-point hosts. I'll focus on two particular questions: 1) Malware often relies on so-called domain-generation algorithms (DGAs) to produce "fake" domain names that are used to connect compromised hosts with a command-and-control server. Many types of DGAs are been developed, from simple hashing techniques to more sophisticated approaches based on wordlists. I will show that these malware-generated domain names can be detected through recurrent neural networks such as LSTMs or GRUs. 2) The second part of the talk will focus on neural models of traffic reputation learned from passive DNS data. Passive DNS data are collections of inter-server DNS queries captured by sensors distributed on the network. This data is a goldmine for predicting whether a given domain name or IP address is likely to be benign or malicious. I will describe a deep neural architecture that predicts the reputation of end-point hosts with high accuracy. The neural model is trained on a large passive DNS dataset (745 million entries) and relies on a broad range of features extracted from the DNs graph.

• Lison, Pierre; Tiedemann, Jörg & Kouylekov, Milen (2018). OpenSubtitles 2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora. Full text in Research Archive Show summary

  Movie and TV subtitles are a highly valuable resource for the compilation of parallel corpora thanks to their availability in large numbers and across many languages. However, the quality of the resulting sentence alignments is often lower than for other parallel corpora. This paper presents a new major release of the OpenSubtitles collection of parallel corpora, which is extracted from a total of 3.7 million subtitles spread over 60 languages. In addition to a substantial increase in the corpus size (about 30 % compared to the previous version), this new release associates explicit quality scores to each sentence alignment. These scores are determined by a statistical regression model based on simple language-independent features and estimated on a small sample of aligned sentence pairs. Evaluation results show that the model is able predict lexical translation probabilities with a root mean square error of 0.07 (coefficient of determination R2 = 0.47). Based on the scores produced by this regression model, the parallel corpora can be filtered to prune out alignments with a score below a given threshold

• Lison, Pierre (2018). SAFERS: Talegjenkjenning og maskinlæring for nødmeldetjenester. Full text in Research Archive
• Lison, Pierre (2018). Tekstmining: En kort innføring. Full text in Research Archive
• Lison, Pierre (2018). Modélisation du dialogue : systèmes de dialogue parlé et corpus multilingues. Full text in Research Archive
• Lison, Pierre & Dogruöz, Sega (2018). Detecting Machine-translated Subtitles in Large Parallel Corpora. Full text in Research Archive Show summary

  Parallel corpora extracted from online repositories of movie and TV subtitles are employed in a wide range of NLP applications, from language modelling to machine translation and dialogue systems. However, the subtitles uploaded in such repositories exhibit varying levels of quality. A particularly difficult problem stems from the fact that a substantial number of these subtitles are not written by human subtitlers but are simply generated through the use of online translation engines. This paper investigates whether these machine-generated subtitles can be detected automatically using a combination of linguistic and extra-linguistic features. We show that a feedforward neural network trained on a small dataset of subtitles can detect machine-generated subtitles with a F1-score of 0.64. Furthermore, applying this detection model on an unlabelled sample of subtitles allows us to provide a statistical estimate for the proportion of subtitles that are machine-translated (or are at least of very low quality) in the full corpus.

• Lison, Pierre (2018). Neural models for predicting the reputation of end-point hosts. Full text in Research Archive
• Lison, Pierre (2017). Automatic Detection of Malware-Generated Domains with Recurrent Neural Models. Full text in Research Archive Show summary

  Modern malware families often rely on domain-generation algorithms (DGAs) to determine rendezvous points to their command-and-control server. Traditional defence strategies (such as blacklisting domains or IP addresses) are inadequate against such techniques due to the large and continuously changing list of domains produced by these algorithms. This paper demonstrates that a machine learning approach based on recurrent neural networks is able to detect domain names generated by DGAs with high precision. The neural models are estimated on a large training set of domains generated by various malwares. Experimental results show that this data-driven approach can detect malware-generated domain names with a F1 score of 0.971. To put it differently, the model can automatically detect 93 % of malware-generated domain names for a false positive rate of 1:100.

• Lison, Pierre (2017). SAFERS - Speech Analytics for Emergency Response Services. Kan taleteknologi og maskinlæring brukes for å effektivisere nødmeldetjenester? Full text in Research Archive
• Lison, Pierre (2017). Neural Reputation Models learned from Passive DNS Data. Full text in Research Archive
• Lison, Pierre (2017). Opptreden i God Morgen Norge (TV2) for å vise Lenny roboten som ble brukt ved Forskningstorget. [TV]. God Morgen Norge (TV2). Full text in Research Archive
• Baste, Øystein Flø; Cyndecka, Malgorzata Agnieszka; Esayas, Samson Yoseph; Langford, Malcolm; Lison, Pierre & Weitzenboeck, Emily Mary (2025). Open Justice Data in Europe: A Patchwork. Social Science Research Network. doi: 10.2139/ssrn.5207840. Full text in Research Archive Show summary

  The publication of court judgments is essential to upholding rule of law and democratic norms as well as facilitating legal research, and new legal technologies. However, many European states struggled to transition to online publication at scale. In this article we address three questions: what are the obligations of states to publish judgments; which states are making progress; and what are the challenges and solutions in ensuring greater publicity? We examine the overarching duties in the ECHR and EU law and the relevant legal requirements and practice in 12 national jurisdictions and two regional courts. Our findings show tremendous variation in duties and practice, and identify barriers to progress (legal, organisational, and budgetary) but also promising innovative solutions in certain jurisdictions. Ultimately, while this publication diversity provides a form of experimental governance, it would be timely to move towards common standards and approaches.

• Lison, Pierre (2024). Automated de-identication of scanned patient records: Evaluation report. Norsk Regnesentral. Full text in Research Archive
• Dahl, Fredrik Andreas; Eikvil, Line; Tvete, Ingunn Fride; Lison, Pierre; Pilán, Ildikó & Fuglerud, Kristin Skeide [Show all 7 contributors for this article] (2023). Helse-effektivisering - et mulig satsningsområde for NR. Norsk Regnesentral . Full text in Research Archive
• Jokinen, Kristiina; Heckmann, Martin; Lala, Dinesh & Lison, Pierre (2021). Proceedings of the 1st RobotDial Workshop on Dialogue Models for Human-Robot Interaction. RobotDial workshop. Full text in Research Archive
• Redelmeier, Annabelle Alice; Lison, Pierre; Løland, Anders & Pilán, Ildikó (2021). Predicting insurance fraud with the help of a sentiment analysis model. Norsk Regnesentral. Full text in Research Archive
• Lison, Pierre (2018). Anonymisering av rettsavgjørelser. Norsk Regnesentral. Full text in Research Archive

View all works in NVA