Shenbin Qian

• Qian, Shenbin & Scherrer, Yves (2026). Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation. In Shterionov, Dimitar; Vanmassenhove, Eva; Sisto, Mirella De; Blain, Fred; Pourmostafa, Javed; Sharami, Roshan; Lepp, Lisa; Manna, Chiara; Rescigno, Argentina Anna; Karakanta, Aliana; Terryn, Ayla Rigouts; Lardelli, Manuel; Resende, Natalia; Murgolo, Elena; Hackenbuchner, Janica; Zaretskaya, Anna; Esplà-Gomis, Miquel; Etchegoyhen, Thierry; Gromann, Dagmar; Bawden, Rachel; Haddow, Barry; Szoc, Sara; Forcada, Mikel & Moniz, Helena (Ed.), Proceedings of the 26th Annual Conference of the European Association for Machine Translation. European Association for Machine Translation (EAMT). ISSN 9789403901404. p. 31–56. doi: 10.26116/9789403901404. Full text in Research Archive Show summary

  Large Language Models (LLMs) have recently demonstrated strong performance in machine translation (MT). However, most prior work focuses on improving or benchmarking translation quality, offering limited insight into when and why LLM-based translation fails. In this work, we systematically analyze failure modes of LLMs in MT by evaluating 15 models, including four reasoning LLMs, across 22 language pairs (LPs) with varying resource levels. We find that non-English-centric LPs consistently yield lower COMET scores than English-centric pairs. To investigate the underlying causes, we introduce Token Activation Rate (TAR), a metric that captures how effectively a model utilizes language-specific tokens in its vocabulary during generation. We validate TAR as a proxy for language representation using models with known language distributions in the training data, and show that lower TAR is strongly associated with poorer translation performance. Furthermore, reasoning LLMs tend to generate more tokens when translating into low-TAR languages, suggesting a compensatory mechanism, although its impact on translation quality varies across models. Overall, our findings emphasize the importance of token-level dynamics in understanding MT performance of LLMs.

View all works in NVA