Constantin Venkov Bechkov

• Beshkov, Kosio & Malthe-Sørenssen, Anders (2025). [2509.24895] Towards Understanding the Shape of Representations in Protein Language Models. arXiv. Fulltekst i vitenarkiv Vis sammendrag

  While protein language models (PLMs) are one of the most promising avenues of research for future de novo protein design, the way in which they transform sequences to hidden representations, as well as the information encoded in such representations is yet to be fully understood. Several works have attempted to propose interpretability tools for PLMs, but they have focused on understanding how individual sequences are transformed by such models. Therefore, the way in which PLMs transform the whole space of sequences along with their relations is still unknown. In this work we attempt to understand this transformed space of sequences by identifying protein structure and representation with square-root velocity (SRV) representations and graph filtrations. Both approaches naturally lead to a metric space in which pairs of proteins or protein representations can be compared with each other. We analyze different types of proteins from the SCOP dataset and show that the Karcher mean and effective dimension of the SRV shape space follow a non-linear pattern as a function of the layers in ESM2 models of different sizes. Furthermore, we use graph filtrations as a tool to study the context lengths at which models encode the structural features of proteins. We find that PLMs preferentially encode immediate as well as local relations between residues, but start to degrade for larger context lengths. The most structurally faithful encoding tends to occur close to, but before the last layer of the models, indicating that training a folding model ontop of these layers might lead to improved folding performance.

• Beshkov, Kosio (2025). [2502.01360] A Quotient Homology Theory of Representation in Neural Networks. arXiv. Fulltekst i vitenarkiv Vis sammendrag

  Previous research has proven that the set of maps implemented by neural networks with a ReLU activation function is identical to the set of piecewise linear continuous maps. Furthermore, such networks induce a hyperplane arrangement splitting the input domain of the network into convex polyhedra $G_J$ over which a network $Φ$ operates in an affine manner. In this work, we leverage these properties to define an equivalence class $\sim_Φ$ on top of an input dataset, which can be split into two sets related to the local rank of $Φ_J$ and the intersections $\cap \text{Im}Φ_{J_i}$. We refer to the latter as the \textit{overlap decomposition} $\mathcal{O}_Φ$ and prove that if the intersections between each polyhedron and an input manifold are convex, the homology groups of neural representations are isomorphic to quotient homology groups $H_k(Φ(\mathcal{M})) \simeq H_k(\mathcal{M}/\mathcal{O}_Φ)$. This lets us intrinsically calculate the Betti numbers of neural representations without the choice of an external metric. We develop methods to numerically compute the overlap decomposition through linear programming and a union-find algorithm. Using this framework, we perform several experiments on toy datasets showing that, compared to standard persistent homology, our overlap homology-based computation of Betti numbers tracks purely topological rather than geometric features. Finally, we study the evolution of the overlap decomposition during training on several classification problems while varying network width and depth and discuss some shortcomings of our method.

• Beshkov, Kosio; Verhellen, Jonas & Lepperød, Mikkel Elle (2025). Isometric representations in neural networks improve robustness. Scientific Reports. 15. doi: 10.1038/s41598-025-20619-0. Fulltekst i vitenarkiv Vis sammendrag

  Artificial and biological agents are unable to learn given completely random and unstructured data. The structure of data is encoded in the distance or similarity relationships between data points. In the context of neural networks, the neuronal activity within a layer forms a representation reflecting the transformation that the layer implements on its inputs. In order to utilize the structure in the data in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, findings in neuroscience propose that generalization and robustness are tied to neural representations being continuously differentiable. Furthermore, representations of objects have the capacity of being hierarchical. Combined together, these two conditions imply that neural networks need to both preserve the distances between inputs as well as have the capacity to apply cuts at different resolutions, corresponding to different levels of a hierarchy. During cross-entropy classification, the metric and structural properties of network representations are usually broken both between and within classes. To achieve and study this behavior, we train neural networks to perform classification while simultaneously maintaining the metric structure within each class at potentially different levels of a hierarchy, leading to continuous and isometric within-class representations. We show that such network representations turn out to be a beneficial component for making accurate and robust inferences about the world. We come up with a network architecture that facilitates hierarchical manipulation of internal neural representations. We verify that our isometric regularization term improves the robustness to adversarial attacks on MNIST and CIFAR10. Finally, we use toy datasets and show that the learned map is isometric everywhere, except around decision boundaries.

• Schøyen, Vemund Sigmundson; Beshkov, Kosio; Pettersen, Markus Borud; Hermansen, Erik; Holzhausen, Konstantin & Malthe-Sørenssen, Anders [Vis alle 8 forfattere av denne artikkelen] (2025). Hexagons all the way down: grid cells as a conformal isometric map of space. PLoS Computational Biology. ISSN 1553-734X. 21(2). doi: 10.1371/journal.pcbi.1012804. Fulltekst i vitenarkiv Vis sammendrag

  Grid cells in the entorhinal cortex are known for their hexagonal spatial activity patterns and are thought to provide a neural metric for space, and support path integration. In this study, we further investigate grid cells as a metric of space by optimising them for a conformal isometric (CI) map of space using a model based on a superposition of plane waves. By optimising the phases within a single grid cell module, we find that the module can form a CI of two-dimensional flat space with phases arranging into a regular hexagonal pattern, supporting an accurate spatial metric. Additionally, we find that experimentally recorded grid cells exhibit CI properties, with one example module showing a phase arrangement similar to the hexagonal pattern observed in our model. These findings provide computational and preliminary experimental support for grid cells as a CI-based spatial representation. We also examine other properties that emerge in CI-optimised modules, including consistent energy expenditure across space and the minimal cell count required to support unique representation of space and maximally topologically persistent toroidal population activity. Altogether, our results suggest that grid cells are well-suited to form a CI map, with several beneficial properties arising from this organisation.

• Beshkov, Kosio & Einevoll, Gaute (2024). [2404.19710] A rank decomposition for the topological classification of neural representations. arXiv. Fulltekst i vitenarkiv Vis sammendrag

  Neural networks can be thought of as applying a transformation to an input dataset. The way in which they change the topology of such a dataset often holds practical significance for many tasks, particularly those demanding non-homeomorphic mappings for optimal solutions, such as classification problems. In this work, we leverage the fact that neural networks are equivalent to continuous piecewise-affine maps, whose rank can be used to pinpoint regions in the input space that undergo non-homeomorphic transformations, leading to alterations in the topological structure of the input dataset. Our approach enables us to make use of the relative homology sequence, with which one can study the homology groups of the quotient of a manifold $\mathcal{M}$ and a subset $A$, assuming some minimal properties on these spaces. As a proof of principle, we empirically investigate the presence of low-rank (topology-changing) affine maps as a function of network width and mean weight. We show that in randomly initialized narrow networks, there will be regions in which the (co)homology groups of a data manifold can change. As the width increases, the homology groups of the input manifold become more likely to be preserved. We end this part of our work by constructing highly non-random wide networks that do not have this property and relating this non-random regime to Dale's principle, which is a defining characteristic of biological neural networks. Finally, we study simple feedforward networks trained on MNIST, as well as on toy classification and regression tasks, and show that networks manipulate the topology of data differently depending on the continuity of the task they are trained on.

• Verhellen, Jonas; Beshkov, Kosio; Amundsen, Sebastian; Ness, Torbjørn V & Einevoll, Gaute (2024). Multitask learning of a biophysically-detailed neuron model. PLoS Computational Biology. ISSN 1553-734X. 20(7). doi: 10.1371/journal.pcbi.1011728. Fulltekst i vitenarkiv Vis sammendrag

  The human brain operates at multiple levels, from molecules to circuits, and understanding these complex processes requires integrated research efforts. Simulating biophysically-detailed neuron models is a computationally expensive but effective method for studying local neural circuits. Recent innovations have shown that artificial neural networks (ANNs) can accurately predict the behavior of these detailed models in terms of spikes, electrical potentials, and optical readouts. While these methods have the potential to accelerate large network simulations by several orders of magnitude compared to conventional differential equation based modelling, they currently only predict voltage outputs for the soma or a select few neuron compartments. Our novel approach, based on enhanced state-of-the-art architectures for multitask learning (MTL), allows for the simultaneous prediction of membrane potentials in each compartment of a neuron model, at a speed of up to two orders of magnitude faster than classical simulation methods. By predicting all membrane potentials together, our approach not only allows for comparison of model output with a wider range of experimental recordings (patch-electrode, voltage-sensitive dye imaging), it also provides the first stepping stone towards predicting local field potentials (LFPs), electroencephalogram (EEG) signals, and magnetoencephalography (MEG) signals from ANN-based simulations. While LFP and EEG are an important downstream application, the main focus of this paper lies in predicting dendritic voltages within each compartment to capture the entire electrophysiology of a biophysically-detailed neuron model. It further presents a challenging benchmark for MTL architectures due to the large amount of data involved, the presence of correlations between neighbouring compartments, and the non-Gaussian distribution of membrane potentials.

• Beshkov, Kosio; Fyhn, Marianne; Hafting, Torkel & Einevoll, Gaute (2024). Topological structure of population activity in mouse visual cortex encodes densely sampled stimulus rotations. iScience. 27(4). doi: 10.1016/j.isci.2024.109370. Fulltekst i vitenarkiv Vis sammendrag

  The primary visual cortex is one of the most well understood regions supporting the processing involved in sensory computation. Following the popularization of high-density neural recordings, it has been observed that the activity of large neural populations is often constrained to low dimensional manifolds. In this work, we quantify the structure of such neural manifolds in the visual cortex. We do this by analyzing publicly available two-photon optical recordings of mouse primary visual cortex in response to visual stimuli with a densely sampled rotation angle. Using a geodesic metric along with persistent homology, we discover that population activity in response to such stimuli generates a circular manifold, encoding the angle of rotation. Furthermore, we observe that this circular manifold is expressed differently in subpopulations of neurons with differing orientation and direction selectivity. Finally, we discuss some of the obstacles to reliably retrieving the truthful topology generated by a neural population.

• Beshkov, Kosio; Fyhn, Marianne; Hafting, Torkel & Einevoll, Gaute Tomas (2023). Topological Structure of Population Activity in Mouse Visual Cortex Encodes Densely Sampled Visual Scene Rotations. Social Science Research Network (SSRN). doi: 10.2139/ssrn.4495668. Fulltekst i vitenarkiv
• Verhellen, Jonas; Beshkov, Kosio; Amundsen, Sebastian; Ness, Torbjørn Vefferstad & Einevoll, Gaute Tomas (2023). Multitask Learning of Biophysically-Detailed Neuron Models. BioRxiv. ISSN 0362-4331. doi: 10.1101/2023.12.05.570220. Fulltekst i vitenarkiv
• Beshkov, Kosio; Verhellen, Jonas & Lepperød, Mikkel Elle (2022). Isometric Representations in Neural Networks Improve Robustness. arXiv.org. ISSN 2331-8422. doi: 10.48550/arXiv.2211.01236. Fulltekst i vitenarkiv
• Beshkov, Kosio & Tiesinga, Paul (2021). Geodesic-based distance reveals nonlinear topological features in neural activity from mouse visual cortex. Biological cybernetics. ISSN 0340-1200. 116, s. 53–68. doi: 10.1007/s00422-021-00906-5. Fulltekst i vitenarkiv

Se alle arbeider i NVA

• Beshkov, Kosio & Einevoll, Gaute Tomas (2024). Towards a Topological Classification of Representations in Neural Networks. Fulltekst i vitenarkiv
• Beshkov, Kosio; Fyhn, Marianne; Hafting, Torkel & Einevoll, Gaute Tomas (2023). Topological Structure of Population Activity in Mouse Visual Cortex Encodes Densely Sampled Visual Scene Rotations. Fulltekst i vitenarkiv
• Beshkov, Kosio; Fyhn, Marianne; Hafting, Torkel & Einevoll, Gaute Tomas (2023). Topological Structure of Population Activity in Mouse Visual Cortex Encodes Densely Sampled Visual Scene Rotations. Fulltekst i vitenarkiv
• Beshkov, Kosio; Verhellen, Jonas & Lepperød, Mikkel Elle (2022). Manifold Propagation through Recurrent Neural Networks. Fulltekst i vitenarkiv
• Vang, Georg; Einevoll, Gaute Tomas; Stasik, Alexander Johannes & Beshkov, Kosio (2024). Exploration of reservoir computing and artificial neural network architectures as saliency detectors. Norges miljø- og biovitenskapelige universitet (NMBU). Fulltekst i vitenarkiv Vis sammendrag

  This study explores the performance of reservoir-type and conventional artificial neural networks as saliency detectors, inspired by Li Zhaoping's hypothesis regarding the V1's role as a saliency detector. A comparative analysis evaluates various models on a simple input image time series task, focusing on their effectiveness in training and out-of-sample validation data. Results indicate that while reservoir models exhibit higher training data loss compared to non-reservoir networks, they outperform other models on new data. Modifications to the reservoir structure show promise in improving both loss and Intersection over Union (IoU) score performance on the training and out-of-sample validation data. However, the study does not find that imposing a more visual cortex-like structure in the reservoir enhances its performance. Further analysis suggests that while reservoir models offer advantages such as cost-effectiveness and robustness for changes in input data, they may lack the precision of more trainable models like CNNs. Limitations to the thesis include the scope of hyperparameter exploration and the lack of overfitting mitigation techniques like the use of dropout. Future research should focus on refining all models to measure the potential performance on both training and out-of-sample data. This includes exploring ensemble methods and improving weight initialization techniques to enhance model precision and adaptability, leading to advancements in various real-world applications.

• Beshkov, Kosio (2024). Neural Representations and the Topology of Cognition. Universitetet i Oslo. Fulltekst i vitenarkiv

Se alle arbeider i NVA