Language identification (LID) tools perform quite well on high-resource languages, but their performance on low-resource language varieties is hard to assess, for several reasons:
- There is (by definition) not a lot of training data. The imbalance of training data across languages leads to trash bin effects (general noise is wrongly labeled as a specific low-resource language).
- The test set often just consists of the Universal Declaration of Human Rights, which is very short and not very representative of text typically found on the web.
- The commonly used language labels are based on the ISO 639-3 codes, which sometimes imperfectly represent the linguistic reality.
- Many language varieties subsumed under one ISO code are in fact continua of dialects and/or do not have a standardized spelling, and it is unclear how such situations should best be annotated.
In this thesis, you will:
- take a closer look at LID systems that claim high language coverage, such as GlotLID or ConLID (~1600 languages)
- select a set of low-resource language varieties with different properties and create challenging test sets for them
- try to improve existing LID systems, propose different labelling schemes or annotation types to better reflect the (socio-)linguistic realities of the varieties in question.