Data Collection
A dedicated dataset will be collected at the RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, University of Oslo, which is equipped with motion capture systems, EMG sensors, gaze trackers, high-quality microphones, and drums. The dataset will include synchronized recordings of all modalities along with audio and MIDI data.
Participants and Recording Procedure
Participants will perform drumming patterns under various conditions. The recordings will include:
-
Motion capture markers on shoulder, elbow, wrist, drumstick tip, and optionally head
-
EMG sensors on key drumming muscles
-
Gaze tracking data to capture visual attention and anticipation
-
Video recordings for visual reference, pose tracking, or direct image-based modeling
-
Audio and MIDI recordings of each strike
Variations will include different striking intensities, speeds, and positions on each drum zone.
Dataset Preparation
-
Clean and normalize motion, EMG, gaze, and video data
-
Parse MIDI files to extract ground truth parameters
-
Align all modalities frame-by-frame
-
Organize into training, validation, and test datasets
Deep Learning Development and Multimodal Fusion
-
Implement models that take multimodal sequences as input and output predicted MIDI parameters
-
Explore architectures such as CNNs (for spatial features), RNNs/LSTMs or Transformers (for temporal modeling), or hybrid approaches
-
Investigate different fusion strategies for multimodal data:
-
Data-level (Early) Fusion: Raw signals from all modalities are combined before feature extraction, allowing the model to learn joint representations from the start.
-
Feature-level Fusion: Features are extracted separately from each modality and then concatenated or combined, providing more structured integration while retaining modality-specific patterns.
-
Decision-level (Late) Fusion: Each modality is processed independently, producing separate predictions that are combined at the decision stage, which can reduce noise from less informative modalities.
-
-
Evaluate trained models quantitatively (e.g., prediction error on MIDI parameters) and qualitatively (e.g., perceptual evaluation of generated sounds)
Requirements
-
Basic knowledge of Python (and preferably PyTorch or TensorFlow)
-
Familiarity with machine learning and deep learning concepts
-
Interest in music technology, motion capture, and multimodal data analysis
References
-
Jensenius, A. R., & Lyons, M. J. (2017). A nod to gesture in music performance. Gesture-Based Human-Computer Interaction and Simulation, 223–235.
-
Maestre, E., et al. (2010). Modeling instrumental gestures: An analysis/synthesis framework for violin bowing. Computer Music Journal, 34(3), 37–55.
-
Makris, D., et al. (2021). Mapping human motion to sound with deep learning: A review. Journal of New Music Research, 50(4), 354–373.
-
Hawthorne, C., et al. (2019). Enabling factorized piano music modeling and generation with the MAESTRO dataset. International Conference on Learning Representations (ICLR).