Diagnostic Semantic Dataset for Norwegian

Modern Natural Language Processing (NLP) relies on various sorts of language resources. This includes large corpora of texts, corpora annotated for various purposes, and hand-crafted resources, e.g., lexica or word-nets. Such resources are abundant for English. To build similar resources for Norwegian is vital for developing NLP for Norwegian to the same level as NLP for English.

Test suites, aslo known as challenge sets, are hand-crafted examples of phenomena used for evaluating NLP systems; “can this system handle that phenomenon?”. Test suites used to be more common some years ago, and have for a period been considered less useful than annotated corpora which may be used both for training and testing. Lately, test suites have been reintroduced as an evaluation tool as part of GLUE, a larger system containing several different ways of testing end-to-end systems for English. The GLUE Diagnostic Dataset contains 550 examples annotated with explanations.

Example from Wang et al 2018

Source: Wang et. al 2018

The first goal of the current project is to build a similar test suite for Norwegian.

The second part is more open. Here the goal is to use the test suite for evaluating various systems.

Emneord: NLP, test suites, semantics, GLUE
Publisert 25. okt. 2020 18:06 - Sist endret 26. okt. 2020 09:50


Omfang (studiepoeng)