This project focuses on the development and application of perceptually inspired metrics in music and sound generation using machine learning. Drawing from psychoacoustics, music theory, and signal processing, the objective is to create metrics that better reflect how listeners evaluate the quality and coherence of generated music. The work will explore a variety of perceptual models—ranging from timbral similarity to harmonic progression—to inform loss functions and validation strategies in music generation tasks. By developing such metrics, we can train models to generate sound and music that not only meets technical criteria but also resonates with human listeners.
Objective
- To investigate current machine learning approaches in sound and music generation and identify gaps in evaluation methods.
- To develop perceptually inspired validation metrics grounded in psychoacoustic and music-theoretic principles.
- To implement these metrics into state-of-the-art generative models and evaluate their effectiveness in improving the perceptual quality of generated music.
- To conduct listener studies to validate these perceptual metrics against human judgments.
Project Description
The project will explore various perceptual aspects of sound and music generation, including timbre, rhythm, harmony, and melodic structure. Students will develop new perceptually grounded metrics that are tailored to human auditory perception and music cognition. These metrics will then be integrated into existing machine learning models for music generation (e.g., WaveNet, Music Transformer) and tested on a variety of datasets. The project will culminate in an evaluation of the perceptual quality of generated music using both automated metrics and human listener studies.
Key Tasks
- Literature Review: Explore existing perceptually grounded metrics in music generation and identify gaps in current evaluation methods.
- Metric Development: Design and implement perceptual metrics based on psychoacoustic and music-theoretic principles (e.g., timbre, harmony, rhythm) in Python using frameworks like TensorFlow/PyTorch.
- Model Integration: Integrate these metrics into generative models (e.g., WaveNet, Music Transformer), using them as loss functions or evaluation criteria.
- Dataset Preparation: Select or prepare diverse music datasets for model training and testing.
- Evaluation: Test models using both traditional and perceptual metrics, and conduct listener studies to validate perceptual quality.
- Documentation: Analyze and document the results from automated evaluations and listener studies.
The students will join the In Sustainable Immersive Networking Lab (SIN-LAB) and work in a team setting. SIN-LAB provides ample opportunities for experimental studies as it is equipped with a wide range of multimedia (cameras, LIDARs, headsets, glasses VR) and haptic devices (haptic gloves, robot arms and hands, in-air touch feedback, and haptic body suit) along with networking equipment to support immersive networking applications.
Learning outcome
- A strong understanding of machine learning techniques for sound and music generation.
- Experience in psychoacoustics, music theory, and their applications in computational models.
- Skills in developing and testing perceptually grounded metrics for evaluating complex generative models.
- Hands-on experience with generative models such as WaveNet and Music Transformer, including model training and fine-tuning.
- Practical skills in conducting perceptual evaluations through listener studies.
- Possibility to publish the result and attend the top international conference.
Conditions
- Participate actively in the weekly SINLab meetings
- Include the course IN5060 in the study plan, unless you have already completed a course on classical (non-ML) data analysis
- Include the course IN4260MCT in the study plan
- Include the course MCT4052 in the study plan
- A knowledge about the state of the art machine learning models for audio processing
- Interest on computational music theory, music perception, and machine learning.
- Interest on psychoacoustics to provide insights into human auditory perception, essential for designing perceptually grounded metrics.
- Have or acquire the following skills: Python programming, deep learning (TensorFlow, PyTorch), signal processing, familiarity with music theory (helpful but not mandatory), and experience in audio data handling.