sondrewo

• Wold, Sondre; Simon, Étienne; Velldal, Erik & Øvrelid, Lilja (2026). Measuring Idiomaticity in Text Embedding Models with epsilon-compositionality. In Demberg, Vera; Inui, Kentaro & Marquez, Lluís (Ed.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891763807. p. 2239–2252. doi: https:/aclanthology.org/2026.eacl-long.99/. Full text in Research Archive Show summary

  The principle of compositionality, which concerns the construction of meaning from constituent parts, is a longstanding topic in various disciplines, most commonly associated with formal semantics. In NLP, recent studies have focused on the compositional properties of text embedding models, particularly regarding their sensitivity to idiomatic expression, as idioms have traditionally been seen as non-compositional. In this paper, we argue that it is unclear how previous work relates to formal definitions of the principle. To address this limitation, we take a theoretically motivated approach based on definitions in formal semantics. We present 𝜀-compositionality, a continuous relaxation of compositionality derived from these definitions. We measure 𝜀-compositionality on a dataset containing both idiomatic and non-idiomatic sentences, providing a theoretically motivated assessment of sensitivity to idiomaticity. Our findings indicate that most text embedding models differentiate between idiomatic and non-idiomatic phrases, although to varying degrees.

• Wold, Sondre; Charpentier, Lucas Georges Gabriel & Simon, Étienne (2025). Systematic Generalization in Language Models Scales with Information Entropy. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics (ACL). ISSN 9798891762565. p. 1807–1819. doi: 10.18653/v1/2025.findings-acl.90. Full text in Research Archive Show summary

  Systematic generalization remains challenging for current language models, which are known to be both sensitive to semantically similar permutations of the input and to struggle with known concepts presented in novel contexts. Although benchmarks exist for assessing compositional behavior, it is unclear how to measure the difficulty of a systematic generalization problem. In this work, we show how one aspect of systematic generalization can be described by the entropy of the distribution of component parts in the training data. We formalize a framework for measuring entropy in a sequence-to-sequence task and find that the performance of popular model architectures scales with the entropy. Our work connects systematic generalization to information efficiency, and our results indicate that success at high entropy can be achieved even without built-in priors, and that success at low entropy can serve as a target for assessing progress towards robust systematic generalization.

• Mondorf, Philipp; Wold, Sondre & Plank, Barbara (2025). Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Models. In Che, Wanxiang; Nabende, Joyce; Shutova, Ekaterina & Pilehvar, Mohammad (Ed.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics (ACL). ISSN 9798891762510. p. 14934–14955. doi: 10.18653/v1/2025.acl-long.727. Full text in Research Archive Show summary

  A fundamental question in interpretability research is to what extent neural networks, particularly language models, implement reusable functions through subnetworks that can be composed to perform more complex tasks. Recent advances in mechanistic interpretability have made progress in identifying circuits, the minimal computational subgraphs responsible for a model’s behavior on specific tasks. However, most studies focus on identifying circuits for individual tasks without investigating how functionally similar circuits relate to each other. To address this gap, we study the modularity of neural networks by analyzing circuits for highly compositional subtasks within a transformer-based language model. Specifically, given a probabilistic context-free grammar, we identify and compare circuits responsible for ten modular string-edit operations. Our results indicate that functionally similar circuits exhibit both notable node overlap and cross-task faithfulness. Moreover, we demonstrate that the circuits identified can be reused and combined through set operations to represent more complex functional model capabilities.

• Wold, Sondre; Simon, Étienne; Charpentier, Lucas Georges Gabriel; Kostylev, Egor V.; Velldal, Erik & Øvrelid, Lilja (2024). Compositional Generalization with Grounded Language Models, Findings of the Association for Computational Linguistics ACL 2024. Association for Computational Linguistics (ACL). ISSN 9798891760998. p. 3447–3460. doi: 10.18653/v1/2024.findings-acl.205. Full text in Research Archive Show summary

  Grounded language models use external sources of information, such as knowledge graphs, to meet some of the general challenges associated with pre-training. By extending previous work on compositional generalization in semantic parsing, we allow for a controlled evaluation of the degree to which these models learn and generalize from patterns in knowledge graphs. We develop a procedure for generating natural language questions paired with knowledge graphs that targets different aspects of compositionality and further avoids grounding the language models in information already encoded implicitly in their weights. We evaluate existing methods for combining language models with knowledge graphs and find them to struggle with generalization to sequences of unseen lengths and to novel combinations of seen base components. While our experimental results provide some insight into the expressive power of these models, we hope our work and released datasets motivate future research on how to better combine language models with structured knowledge representations.

• Wold, Sondre; Mæhlum, Petter & Hove, Oddbjørn (2024). Estimating Lexical Complexity from Document-Level Distributions. In Calzolari, Nicoletta; Kan, Min-Yen; Hoste, Veronique; Lenci, Alessandro; Sakti, Sakriani & Xue, Nianwen (Ed.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). European Language Resources Association. ISSN 9782493814104. p. 6309–6318. Full text in Research Archive
• Samuel, David; Charpentier, Lucas Georges Gabriel & Wold, Sondre (2024). More room for language: Investigating the effect of retrieval on language models. In Duh, Kevin; Gomez, Helena & Bethard, Steven (Ed.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). Association for Computational Linguistics (ACL). ISSN 9798891761155. p. 282–305. Full text in Research Archive

View all works in NVA

• Wold, Sondre (2026). A PRELIMINARY STATISTICAL ASSESSMENT OF INSTABILITY PREDICTABILITY IN OBSERVED SNOW PROFILES. International Snow Science Workshops (ISSW) Proceedings. doi: 10.15788/1790098791. Full text in Research Archive
• Wold, Sondre & Mæhlum, Petter (2024). Nesten 60 år etter LIX -- Lesbarhet på ordnivå. Full text in Research Archive
• Wold, Sondre (2023). Maskinoversettelse. In Bolstad, Erik (Eds.), Store norske leksikon (snl.no 2023). Store norske leksikon. Full text in Research Archive
• Wold, Sondre (2026). Compositionality And Generalization in Language Modeling. Universitetet i Oslo. Full text in Research Archive Show summary

  Our ability to understand and use language is not limited to expressions we are already familiar with. Instead, we continually create novel expressions by systematically combining words and phrases from a finite vocabulary. This ability is often described by the principle of semantic compositionality, which defines the meaning of a compound expression as determined by the meaning of the expression’s parts and their syntactic composition. Today, a large portion of all text is generated by language models built with artificial neural networks. Unlike the principle of compositionality, these networks do not rely on an explicit definition of meaning composition, but on general-purpose machine learning techniques. In this thesis, we evaluate several types of neural networks for both graph classification and sequence modeling, targeting productive and systematic properties of generalization. We achieve this through enforcing different constraints on the data distribution, introducing several evaluation frameworks. Our results show that most networks lack productive and systematic generalization. We further demonstrate the connection between systematic generalization and the information entropy of the training distribution, and we present novel ways of grounding the evaluation of neural networks in formal definitions of compositionality. Our findings advocate for a more principled approach to evaluation in NLP.

View all works in NVA