Wallace, Benedikte; Glette, Kyrre & Szorkovszky, Alexander
(2025).
How can we make robot dance expressive and responsive? A survey of methods and future directions.
Frontiers in Computer Science.
7.
doi: 10.3389/fcomp.2025.1575667.
Full text in Research ArchiveShow summary
The development of robots that can dance like humans presents a complex challenge due to the disparate abilities involved and various aesthetic qualities that need to be achieved. This article reviews recent advances in robotics, artificial intelligence, and human-robot interaction toward enabling various aspects of realistic dance, and examines potential paths toward a fully embodied dancing agent. We begin by outlining the essential abilities required for a robot to perform human-like dance movements and the resulting aesthetic qualities, summarized under the terms expressiveness and responsiveness . Subsequently, we present a review of the current state-of-the-art in dance-related robot technology, highlighting notable achievements, limitations and trade-offs in existing systems. Our analysis covers various approaches, including traditional control systems, machine learning algorithms, and hybrid systems that aim to imbue robots with the capacity for responsive, expressive movement. Finally, we identify and discuss the critical gaps in current research and technology that need to be addressed for the full realization of realistic dancing robots. These include challenges in real-time motion planning, adaptive learning from human dancers, and morphology independence. By mapping out current methods and challenges, we aim to provide insights that may guide future innovations in creating more engaging, responsive, and expressive robotic systems.
Wallace, Benedikte; Nymoen, Kristian; Tørresen, Jim & Martin, Charles Patrick
(2024).
Breaking from realism: exploring the potential of glitch in AI-generated dance.
Digital Creativity.
ISSN 1462-6268.
35,
p. 125–142.
doi: 10.1080/14626268.2024.2327006.
Full text in Research ArchiveShow summary
What role does deviation from realism play in the potential for generative artificial intelligence (AI) as a creative tool? A deep case-study was performed to explore interactions with AI-generated dance sequences as an inspiration source in dance composition and improvization. We present a simple interface created in collaboration with an experienced dancer for browsing AI-generated dance. By including glitches, the physics-breaking mistakes often encountered in AI-generated artefacts, we examine their affordances and possible use cases through sessions with the dancer. Through a process of reflexive thematic analysis, we identified that generative AI can engage a dancer through surprise, inspiring a transformation from abstract to physical movements. Our work challenges existing notions of the importance of realism in dance generation models, exemplifies the importance of close collaboration with practitioners in evaluating AI-generated artefacts and proposes glitch as a potential use case for dance ideation as it encourages dancers to embody unfamiliar movement qualities and break from ingrained patterns.
Erdem, Cagri; Wallace, Benedikte; Glette, Kyrre & Jensenius, Alexander Refsum
(2023).
Tool or Actor? Expert Improvisers’ Evaluation of a Musical AI “Toddler”.
Computer Music Journal.
ISSN 0148-9267.
46(4),
p. 26–42.
doi: 10.1162/comj_a_00657.
Full text in Research ArchiveShow summary
Abstract In this article, we introduce the coadaptive audiovisual instrument, CAVI. This instrument uses deep learning to generate control signals based on muscle and motion data of a performer's actions. The generated signals control time-based live sound-processing modules. How does a performer perceive such an instrument? Does it feel like a machine learning–based musical tool? Or is it an actor with the potential to become a musical partner? We report on an evaluation of CAVI after it had been used in two public performances. The evaluation is based on interviews with the performers, audience questionnaires, and the creator's self-analysis. Our findings suggest that the perception of CAVI as a tool or actor correlates with the performer's sense of agency. The perceived agency changes throughout a performance based on several factors, including perceived musical coordination, the balance between surprise and familiarity, a “common sense,” and the physical characteristics of the performance setting.
Wallace, Benedikte; Hilton, C.; Nymoen, Kristian; Tørresen, Jim; Martin, Charles & Fiebrink, Rebecca
(2023).
Embodying an Interactive AI for Dance Through Movement Ideation,
C&C '23: Creativity and Cognition Virtual Event USA June 19 - 21, 2023.
Association for Computing Machinery (ACM).
ISSN 9798400701801.p. 454–464.
doi: 10.1145/3591196.3593336.
Full text in Research Archive
Bentsen, Lars Ødegaard; Simionato, Riccardo; Wallace, Benedikte & Krzyzaniak, Michael Joseph
(2022).
Transformer and LSTM Models for Automatic Counterpoint Generation using Raw Audio.
Proceedings of the SMC Conferences.
doi: 10.5281/zenodo.6572847.
Full text in Research ArchiveShow summary
A study investigating Transformer and LSTM models applied to raw audio for automatic generation of counterpoint was conducted. In particular, the models learned to generate missing voices from an input melody, using a collection of raw audio waveforms of various pieces of Bach’s work, played on different instruments. The research demonstrated the efficacy and behaviour of the two deep learning (DL) architectures when applied to raw audio data, which are typically characterised by much longer sequences than symbolic music representations, such as MIDI. Currently, the LSTM model has been the quintessential DL model for sequence-based tasks, such as generative audio models, but the research conducted in this study shows that the Transformer model can achieve competitive results on a fairly complex raw audio task. The research therefore aims to spark further research and investigation into how Trans- former models can be used for applications typically dominated by recurrent neural networks (RNN). In general, both models yielded excellent results and generated sequences with temporal patterns similar to the input targets for songs that were not present in the training data, as well as for a sample taken from a completely different dataset.
Ruud, Markus Toverud; Sandberg, Tale Hisdal; Tranvaag, Ulrik Johan Vedde; Wallace, Benedikte; Karbasi, Seyed Mojtaba & Tørresen, Jim
(2022).
Reinforcement Learning Based Dance Movement Generation.
In Carlson, Kristin (Eds.),
MOCO '22: Proceedings of the 8th International Conference on Movement and Computing.
Association for Computing Machinery (ACM).
ISSN 9781450387163.
doi: 10.1145/3537972.3538007.
Full text in Research ArchiveShow summary
Generating genuinely creative and novel artifacts with machine learning is still a challenge in the world of computational science. A creative machine learning agent can be beneficial for applications where novel solutions are desired and may also optimize search. Reinforcement Learnings’ (RL) interactive properties can make it an effective tool to investigate these possibilities in creative contexts. This paper shows how a Reinforcement learning-based technique, in combination with Principal Component Analysis (PCA), can be utilized for generating varying movements based on a goal picking policy. The proposed model is trained on a data set of motion capture recordings of dance improvisation. Our study shows that the trained RL agent can learn to pick sequences of dance poses that are coherent, have compound movement, and can resemble dance.
Wallace, Benedikte; Martin, Charles Patrick; Tørresen, Jim & Nymoen, Kristian
(2021).
Exploring the Effect of Sampling Strategy on Movement Generation with Generative Neural Networks,
EvoMUSART 2021: Artificial Intelligence in Music, Sound, Art and Design.
Springer Nature.
ISSN 9783030729134.p. 344–359.
doi: 10.1007/978-3-030-72914-1_23.
Full text in Research Archive
Wallace, Benedikte & Martin, Charles Patrick
(2019).
Comparing models for harmony prediction in an interactive audio looper.
Lecture Notes in Computer Science (LNCS).
ISSN 0302-9743.
11453 LNCS,
p. 173–187.
doi: 10.1007/978-3-030-16667-0_12.
Full text in Research Archive
Noori, Farzan Majeed; Wallace, Benedikte; Uddin, Md Zia & Tørresen, Jim
(2019).
A Robust Human Activity Recognition Approach Using OpenPose, Motion Features, and Deep Recurrent Neural Network.
Lecture Notes in Computer Science (LNCS).
ISSN 0302-9743.
11482 LNCS,
p. 299–310.
doi: 10.1007/978-3-030-20205-7_25.
Full text in Research Archive
Wallace, Benedikte; Martin, Charles Patrick & Nymoen, Kristian
(2019).
Tracing from Sound to Movement with Mixture Density Recurrent Neural Networks.
In Coleman, Grisha (Eds.),
Proceedings of the 6th International Conference on Movement and Computing.
ACM Publications.
ISSN 9781450376549.
doi: 10.1145/3347122.3371376.
Full text in Research ArchiveShow summary
In this work, we present a method for generating sound-tracings using a mixture density recurrent neural network (MDRNN). A sound-tracing is a rendering of perceptual qualities of short sound objects through body motion. The model is trained on a dataset of single point sound-tracings with multimodal input data and learns to generate novel tracings. We use a second neural network classifier to show that the input sound can be identified from generated tracings. This is part of an ongoing research effort to examine the complex correlations between sound and movement and the possibility of modelling these relationships using deep learning.
Bøe, Sunniva Monclair; Frøjd, Elise Koppang; Jensenius, Alexander Refsum; Marin-Bucio, Diego; Wallace, Benedikte & Lartillot, Olivier
(2026).
Debatten raser om KI i kulturfeltet: Trenger vi egentlig musikere, forfattere og filmprodusenter nå?
[Journal].
Forskning.no.
Full text in Research ArchiveShow summary
Norske forskere vil finne ut hvordan kreative utøvere kan være med på å utvikle kunstig intelligens – og motsatt.
Wallace, Benedikte
(2024).
Imitation or Innovation? Translating Features of Expressive Motion from Humans to Robots. Full text in Research Archive
Pileberg, Silje & Wallace, Benedikte
(2023).
I kunsten kan kunstig intelligens få ein heilt ny funksjon.
[Internet].
uio.no.
Full text in Research Archive
Pileberg, Silje & Wallace, Benedikte
(2023).
In art, AI can acquire an entirely new purpose.
[Internet].
sciencenorway.no.
Full text in Research Archive
Wallace, Benedikte & Melteig, Elina
(2022).
Benedikte ba maskinen lage egne danser, basert på data fra ekte dansere.
[Internet].
titan.uio.no.
Full text in Research Archive
Ruud, Markus Toverud; Sandberg, Tale Hisdal; Tranvaag, Ulrik Johan Vedde; Wallace, Benedikte; Karbasi, Seyed Mojtaba & Tørresen, Jim
(2022).
Reinforcement Learning Based Dance Movement Generation.
Full text in Research ArchiveShow summary
Generating genuinely creative and novel artifacts with machine learning is still a challenge in the world of computational science. A creative machine learning agent can be beneficial for applications where novel solutions are desired and may also optimize search. Reinforcement Learnings’ (RL) interactive properties can make it an effective tool to investigate these possibilities in creative contexts. This paper shows how a Reinforcement learning-based technique, in combination with Principal Component Analysis (PCA), can be utilized for generating varying movements based on a goal picking policy. The proposed model is trained on a data set of motion capture recordings of dance improvisation. Our study shows that the trained RL agent can learn to pick sequences of dance poses that are coherent, have compound movement, and can resemble dance.
Wallace, Benedikte
(2021).
Learning Embodied Sound-Motion Mappings: Evaluating AI-Generated Dance Improvisation.
Full text in Research Archive
Wallace, Benedikte
(2021).
Exploring the Effect of Sampling Strategy on Movement Generation with Generative Neural Networks.
Full text in Research Archive
Wallace, Benedikte
(2020).
Deep Learning with Multi-Dimensional Time-Series Data: Examining the Effect of Music on Movement.
Full text in Research Archive
Wallace, Benedikte; Nymoen, Kristian; Martin, Charles Patrick & Tørresen, Jim
(2020).
Towards Movement Generation with Audio Features.
Full text in Research Archive
Wallace, Benedikte & Krüger, Frida Jørstad
(2020).
Dansestilen din avslører deg.
[Internet].
nrk.no.
Full text in Research Archive
Å lage kunst eller bevege seg til musikk er noe vi mennesker gjør helt instinktivt, vil vi kunne modellere disse kreative prosessene ved hjelp av kunstig intelligens? Og hva menes med å trene nevrale nettverk i musikk? I denne episoden av #LØRN snakker Silvija med PhD kandidat, RITMO senter for tverrfaglig forskning på rytme, tid og bevegelse ved UiO, Benedikte Wallace, om hvordan de kobler kunstig intelligens, robotikk og sensorikk med ting som har med dans og musikk å gjøre fordi det er denne type kombinasjoner som gir nye innsikter.
Wallace, Benedikte; Nymoen, Kristian & Martin, Charles Patrick
(2019).
Tracing from Sound to Movement with Mixture Density Recurrent Neural Networks.
Full text in Research Archive
Slåttum, Sigmund Kleivene; Høvin, Mats Erling; Wallace, Benedikte & Lindblom, Diana Saplacan
(2026).
Supervisor for Master thesis: "Moving with the Music: A Novel Robotic Floor for Motion-Enhanced Music Experience" (60 ECTS).
University of Oslo (UiO).
Full text in Research ArchiveShow summary
This thesis investigates how synchronized whole-body movement shapes the experience of listening
to music. The two research questions ask whether floor movement changes the listening
experience, and whether the specific character of the movement matters for that experience.
To address these questions, a one-degree-of-freedom robotic floor was designed and built from
scratch. The system, named Harmonic Audio-Reactive Platform (HARP), lifts a person standing
on it in a vertical rolling motion derived from features extracted from the music. Two motion patterns
were implemented for use in the study: a simple pattern oscillating at the beat frequency,
and a complex pattern that combines two sinusoids whose amplitudes vary with the spectral content
of the music. The hardware, software, and trajectory generation are described in detail in the
thesis, and form a methodological contribution alongside the empirical findings. Six participants
took part in a within-group user study. Each experienced three songs from contrasting genres
(classical, EDM, and metal) under three motion conditions (no motion, simple, and complex),
giving nine conditions per participant. A qualitatively-driven mixed methods design was used.
Semi-structured interviews were analyzed with reflexive thematic analysis, alongside Likert ratings,
amplitude adjustment logs, synchronization measurements, and video observations. Three
themes were constructed from the interviews: Enjoyment, Emotion, and Congruence. Across
all data sources, the most consistent finding is that motion pattern preference is personal. No
single pattern worked for everyone, and the same pattern could feel mismatched or stressful to
one participant and meaningful to another. The movement was mostly received as adding to the
listening experience, with Likert ratings, qualitative themes, and participants’ own descriptions
of real-world use pointing the same way. Synchronization between music and motion shaped the
experience but its specific role could not be settled within this study.
The inherently subjective notion of quality in creative artefacts gives rise to both technical and philosophical questions regarding AI-generated art. In this thesis, the development and evaluation of AI-generated dance forms the context within which these questions are explored. Dance is a multifaceted art form and a rich, complex data source. The main objectives of this thesis are to understand more about how deep learning can be used to capture salient features of dance movement using full-body motion capture data. We further explore how such technology might be aimed to benefit the creative practice of dancers. Our work includes the collection of a data set of improvisation performed by individual dancers and the implementation of generative models trained on this data. We evaluate the models’ ability to learn the intricate relationship between music and movement and investigate the trained model's generative versatility. We also examine the generated dance through subjective assessment in a survey, and through dialogue and embodiment exercises with experienced dancers and choreographers. This work broadens our understanding of how leveraging the ability of AI to go beyond a mere replica of human movement can itself lead to engagement and inspiration.
Musicians often use tools such as loop-pedals and multitrack recorders to assist in improvisation and songwriting. While these devices are useful in creating new compositions from scratch, they do not contribute to the composition directly. In recent years, new musical instruments, interfaces and controllers using machine learning algorithms to create new sounds, generate accompaniment or construct novel compositions, have become available for both professional musicians and novices to enjoy. This thesis describes the design, implementation and evaluation of a system for predictive songwriting and improvisation using concatenative accompaniment which has been given the nickname PSCA. In its most simple form, the PSCA functions as an audio looper for vocal improvisation, but the system also utilises machine learning approaches to predict suitable harmonies to accompany the playback loop. Two machine learning algorithms were compared and implemented into the PSCA to facilitate harmony prediction: the hidden Markov model (HMM) and the Bidirectional Long Short-Term Memory (BLSTM). The HMM and BLSTM algorithms are trained on a dataset of lead sheets in order to learn the relationship between the notes in a melody and the chord which accompanies it as well as learning dependencies between chords to model chord progressions. In quantitative testing, the BLSTM model was found to be able to learn the harmony prediction task more effectively than the HMM model, this was also supported by a qualitative analysis of musicians using the PSCA system. The system proposed in this thesis provides a novel approach in which these two machine learning models are compared with regards to prediction accuracy on the dataset as well as the perceived musicality of each model when used for harmony prediction in the PSCA. This approach results in a system which can contribute to the improvisation and songwriting process by adding harmonies to the audio loop on-the-fly.
