Publications
-
Beshkov, Kosio; Verhellen, Jonas & Lepperød, Mikkel Elle
(2025).
Isometric representations in neural networks improve robustness.
Scientific Reports.
15.
doi:
10.1038/s41598-025-20619-0.
Full text in Research Archive
Show summary
Artificial and biological agents are unable to learn given completely random and unstructured data. The structure of data is encoded in the distance or similarity relationships between data points. In the context of neural networks, the neuronal activity within a layer forms a representation reflecting the transformation that the layer implements on its inputs. In order to utilize the structure in the data in a truthful manner, such representations should reflect the input distances and thus be continuous and isometric. Supporting this statement, findings in neuroscience propose that generalization and robustness are tied to neural representations being continuously differentiable. Furthermore, representations of objects have the capacity of being hierarchical. Combined together, these two conditions imply that neural networks need to both preserve the distances between inputs as well as have the capacity to apply cuts at different resolutions, corresponding to different levels of a hierarchy. During cross-entropy classification, the metric and structural properties of network representations are usually broken both between and within classes. To achieve and study this behavior, we train neural networks to perform classification while simultaneously maintaining the metric structure within each class at potentially different levels of a hierarchy, leading to continuous and isometric within-class representations. We show that such network representations turn out to be a beneficial component for making accurate and robust inferences about the world. We come up with a network architecture that facilitates hierarchical manipulation of internal neural representations. We verify that our isometric regularization term improves the robustness to adversarial attacks on MNIST and CIFAR10. Finally, we use toy datasets and show that the learned map is isometric everywhere, except around decision boundaries.
-
Verhellen, Jonas; Beshkov, Kosio; Amundsen, Sebastian; Ness, Torbjørn V & Einevoll, Gaute
(2024).
Multitask learning of a biophysically-detailed neuron model.
PLoS Computational Biology.
ISSN 1553-734X.
20(7).
doi:
10.1371/journal.pcbi.1011728.
Full text in Research Archive
Show summary
The human brain operates at multiple levels, from molecules to circuits, and understanding these complex processes requires integrated research efforts. Simulating biophysically-detailed neuron models is a computationally expensive but effective method for studying local neural circuits. Recent innovations have shown that artificial neural networks (ANNs) can accurately predict the behavior of these detailed models in terms of spikes, electrical potentials, and optical readouts. While these methods have the potential to accelerate large network simulations by several orders of magnitude compared to conventional differential equation based modelling, they currently only predict voltage outputs for the soma or a select few neuron compartments. Our novel approach, based on enhanced state-of-the-art architectures for multitask learning (MTL), allows for the simultaneous prediction of membrane potentials in each compartment of a neuron model, at a speed of up to two orders of magnitude faster than classical simulation methods. By predicting all membrane potentials together, our approach not only allows for comparison of model output with a wider range of experimental recordings (patch-electrode, voltage-sensitive dye imaging), it also provides the first stepping stone towards predicting local field potentials (LFPs), electroencephalogram (EEG) signals, and magnetoencephalography (MEG) signals from ANN-based simulations. While LFP and EEG are an important downstream application, the main focus of this paper lies in predicting dendritic voltages within each compartment to capture the entire electrophysiology of a biophysically-detailed neuron model. It further presents a challenging benchmark for MTL architectures due to the large amount of data involved, the presence of correlations between neighbouring compartments, and the non-Gaussian distribution of membrane potentials.
-
-
-
Verhellen, Jonas
(2022).
Graph-based molecular Pareto optimisation.
Chemical Science.
ISSN 2041-6520.
13(25),
p. 7526–7535.
doi:
10.1039/d2sc00821a.
Full text in Research Archive
Show summary
Computer-assisted design of small molecules has experienced a resurgence in academic and industrial interest due to the widespread use of data-driven techniques such as deep generative models. While the ability to generate molecules that fulfil required chemical properties is encouraging, the use of deep learning models requires significant, if not prohibitive, amounts of data and computational power. At the same time, open-sourcing of more traditional techniques such as graph-based genetic algorithms for molecular optimisation [Jensen, Chem. Sci., 2019, 12, 3567–3572] has shown that simple and training-free algorithms can be efficient and robust alternatives. Further research alleviated the common genetic algorithm issue of evolutionary stagnation by enforcing molecular diversity during optimisation [Van den Abeele, Chem. Sci., 2020, 42, 11485–11491]. The crucial lesson distilled from the simultaneous development of deep generative models and advanced genetic algorithms has been the importance of chemical space exploration [Aspuru-Guzik, Chem. Sci., 2021, 12, 7079–7090]. For single-objective optimisation problems, chemical space exploration had to be discovered as a useable resource but in multi-objective optimisation problems, an exploration of trade-offs between conflicting objectives is inherently present. In this paper we provide state-of-the-art and open-source implementations of two generations of graph-based non-dominated sorting genetic algorithms (NSGA-II, NSGA-III) for molecular multi-objective optimisation. We provide the results of a series of benchmarks for the inverse design of small molecule drugs for both the NSGA-II and NSGA-III algorithms. In addition, we introduce the dominated hypervolume and extended fingerprint based internal similarity as novel metrics for these benchmarks. By design, NSGA-II, and NSGA-III outperform a single optimisation method baseline in terms of dominated hypervolume, but remarkably our results show they do so without relying on a greater internal chemical diversity.
View all works in NVA
Published
Oct. 2, 2020 11:31 AM
- Last modified
Dec. 1, 2020 10:43 AM
