Energy-efficient Large-scale Training and Inference

Large language models (LLMs) and Vision Transformers (ViTs) have become the core of the digital infrastructure that underpins several areas of our lives.Training and inference of those models, however, is computationally expensive and emits tons of CO²e every year. In 2025, global data centres used 485TWhof electricity - about 3% of the world electricity [1]. The training and inference of machine learning (ML) models takes a big part in this. During training of the Olmo 3 series, 421t CO²e have been emitted which is approximately 210 times the annual footprint of an average Norwegian. In recent years, however, inference has overtaken the training of those models in terms of energy consumption and emissions. A single median Gemini prompt consumes 0.24Wh electricity which on average (worldwide) emits 0.03g CO²e [2]. This highlights the importance for both energy-efficient training but also inference of LLMs and ViTs. 

One way of making models more energy-efficient is through compression.Model compression describes a set of techniques that make trained models smaller, faster and/or cheaper to run while keeping the accuracy as close to the original model as possible. While there exist many different types of model compression methods, we are mostly interested in quantisation, low-rank approximation and probabilistic methods.

Building on previous works within the group, potential master thesis will be offered for the following topics:

1. Combining sketching with post-training quantisation (PTQ): In a previous project we showed that by learning the distribution of sketching matrices, we can increase performance on model compression tasks. A popular choice of post training compression method is quantisation. We aim to combine sketching with PTQ to achieve even higher compression levels.

2. Obtaining generalisation error bounds for compressed model: We learn the distribution of sketching matrices for model compression rather than sampling form a generic, data-oblivious distribution. This MS project aims to understand what distribution the learned sketching matrices induce on the model and studies generalisation bounds for those compressed models e.g., using PAC-Bayes. 

3.  Layer-wise gradient compression for distributed training: To minimise inter-GPU communication overhead and shorten distributed training, gradient compression is typically employed to reduce the necessary number of bits to communicate. Existing layer-wise approaches rely on L2 error on gradients at fixed intervals during training to adapt the choice of per-layer compression levels to training dynamics; our recent work generalizes this framework. This project aims to build upon and extend the range of applications of our framework —possible directions for interested students include federated learning (FL), upscaling to fully sharded data parallel (FSDP) training with LLMs, and finding optimal framework configurations for these settings.

4. Layer-wise compression level selection post-training: Inspired by layer-wise gradient compression during training, we want to explore layer-wise compression level selection for post-training model compression. We are especially interested in applications to compression schemes with a wide range of possible per-layer configurations, such as low-rank methods or unstructured pruning.

Sample of our MS students/interns

Next one maybe you?!

Josep Peiro Ramos (First author of ICLR submission)
Ola Karlstad (Co-author of ICLR submission)
Frida Marie Engøy Westby (First author of ICLRW 2026 paper)
Preben Nicholai Castberg (Co-author of ICLR submission)
Brage Eilertsen (First author of AAAI 26 paper)
Røskva Bjørgfinsdóttir (Co-author of AAAI 26 paper)
Iman Tabrizian (Co-first author of NeurIPS paper)