Deep neural networks are usually built on a simple intuition: more layers allow a model to perform more steps of computation. A twelve-layer neural network should, in some meaningful sense, be doing more than a six-layer one.
But increasingly, that picture is beginning to crack.
In our new paper, A Smaller Transformer in Your Transformer, accepted at BMVC 2026, we study what happens inside Vision Transformers as information passes from one layer to the next. We find that several consecutive layers can enter periods where they perform strikingly similar transformations. Instead of each layer representing a clearly distinct computational step, the network appears to move slowly through broader phases of computation.
That raises an obvious question: if several layers are all working through essentially the same phase, do we really need to execute every intermediate step? Our answer is: often times, no.
We introduce Transformer-Within-Transformer (TWT), a method for finding these redundant phases in an already trained model and replacing each of them with a single learned Transformer layer. Rather than merely deleting layers, TWT teaches the replacement layer to jump directly from the beginning of a computational phase to its end.
The result can be a model with roughly half the original depth and around half the inference computation, while retaining much of the original performance.
Taking the shortcut
One way to picture a deep neural network is as a route through a complicated landscape of possible representations.
A conventional Transformer takes many small steps through this landscape. Previous work has shown that some groups of steps look remarkably similar, and one interpretation is that the model is effectively repeating the same operation several times.
TWT starts from a different possibility: perhaps those repeated-looking steps are not themselves essential. They may simply be the route that training happened to discover. Instead of reproducing every footstep, we ask whether a new layer can learn the shortcut.
The distinction is important. Some previous approaches reduce the number of unique parameters while still repeating computations at inference time. Others replace Transformer layers with simpler operations, but sacrifice the ability of attention to dynamically mix information between image regions. TWT instead replaces an entire redundant phase with a normal Transformer layer, preserving the basic machinery of attention while removing the intermediate computation.
We first identify groups of layers whose representations remain highly similar. We then audition existing layers from the original network as starting points for the replacement, before training the compressed network to reproduce the important intermediate and final representations of its larger teacher.
The illustration above shows pairwise cosine similarity between block outputs from ImageNet1k, and reveals consistent block-diagonal structure across architectures and training paradigms, both for self-supervised (DINOv2, DINOv3) and supervised (DeiT-III) models. Contiguous groups of blocks produce redundant representations, which can be leveraged to produce more effective models. Possible blocks are outlined in white.
Half the journey, nearly the same destination
Across standard natural-image models, we found that twelve-layer networks could often be reduced to six layers while remaining close to their original classification accuracy. For example, a six-layer TWT model (DINOv2) requires 11.71 billion floating-point operations per image instead of 23.42 billion, while ImageNet classification accuracy drops from 84.6% to 84.2%.
The effect becomes particularly interesting in histopathology. Whole-slide images can contain billions of pixels, and are commonly divided into thousands of smaller image regions before being processed by a foundation model. Even modest reductions in the cost of analysing each region therefore accumulate rapidly.
On two histopathology foundation models, H0-mini and Hibou-B, TWT could reduce twelve-layer networks to as few as three or four layers while preserving much of their downstream performance. In several experiments on prostate cancer grading and breast cancer metastasis detection, the compressed networks even performed better than the original models on external datasets.
That result deserves some caution. Making a neural network smaller does not magically make it better, and we emphasize that compression does not generally improve accuracy. But it raises an interesting possibility: some of the computation performed by an oversized model may not merely be unnecessary. Under a shift in data distribution, it may amplify variation that is irrelevant to the task.
What disappears when we remove the layers?
This led us to look more closely at the geometry of the representations inside the networks.
We measured their intrinsic dimension, loosely speaking, the number of independent directions required to describe the structure of the learned representation. The compressed models largely followed the same broad trajectory through representation space as their full-sized teachers. In the histopathology models, however, the compressed network also showed substantially less variation under changes in image colour and geometry.
At several stages of the histopathology network, this variation was reduced by roughly 38 to 50 percent.
One interpretation is that the removed layers were spending computation on small, noisy adjustments rather than constructing genuinely new representations. TWT keeps the destination while discarding some of the wandering required to get there.
How deep does a deep network need to be?
The practical implication of TWT is straightforward: large pretrained vision models can sometimes be made substantially cheaper after training, without redesigning or retraining them from the beginning.
The more intriguing implication concerns what depth means in neural networks.
We normally describe a twelve-layer Transformer as a sequence of twelve computational stages. But if several adjacent layers belong to a common computational phase, architectural depth and effective computational depth may be very different things.
A network may be deep partly because the optimisation process found a long route to a solution, not because the underlying computation intrinsically requires that many steps. TWT does not answer why these redundant phases emerge. In fact, that may be the more important question. If we can identify which parts of a trained model are doing genuinely distinct work, the next generation of models may not need to learn the redundant computation in the first place.
Conference and Venue
We are happy to announce that our paper was accepted as part of the British Machine Vision Conference (BMVC) which will be held in Lancaster, UK on the 23rd to 26th of November, 2026. We’re looking forward to presenting our work, discussing the research with the community, and connecting with fellow researchers in computer vision and related fields.
More details on the paper and presentation schedule will follow soon. See you in Lancaster!