Objective

• To investigate and evaluate existing DDSP techniques for modifying input sounds' time and frequency characteristics.
• To design and develop DDSP modules that simulate frequency shifting and amplitude variations common in guitar playing techniques (e.g., vibrato, blending, fingerstyle, and pick techniques).
• To evaluate the perceptual quality of synthesized audio through quantitative performance analysis.

Project Description

The proposed project will explore DDSP approaches for real-time sound modification. The goal is to develop models capable of sample-by-sample sound processing that integrates with existing machine learning frameworks, specifically for accurately modeling time-frequency features associated with expressive guitar performance. After development, the synthesized audio will be evaluated using automated perceptual metrics and human listener studies to ensure the models produce realistic and expressive guitar sounds.

Key Tasks

• Literature Review: Conduct a comprehensive review of DDSP methodologies and their application in sound synthesis. Explore previous work on modeling acoustic instrument techniques, focusing on time-frequency transformations.
• Model Design: Design DDSP modules that capture frequency shifts and amplitude variations in guitar playing, ensuring modularity for flexibility in sound manipulation. 
• Model Training and Evaluation: Train the developed models using guitar datasets and evaluate them using standard objective metrics (e.g., spectral similarity, signal-to-noise ratio) and perceptually grounded metrics. Conduct listener studies to assess the perceptual quality of the synthesized audio.
• Documentation: Prepare thorough documentation, including technical specifications of the models, experimental results, and a final thesis write-up.

The students will join the In Sustainable Immersive Networking Lab (SIN-LAB) and work in a team setting. SIN-LAB provides ample opportunities for experimental studies as it is equipped with a wide range of multimedia (cameras, LIDARs, headsets, glasses VR) and haptic devices (haptic gloves, robot arms, and hands, in-air touch feedback, and haptic body suit) along with networking equipment to support immersive networking applications.

Learning outcome

• A functional DDSP-based model capable of real-time, high-quality synthesis of guitar sounds, mainly focusing on vibrato, blending, and excitation technique variations.
• A comparison between machine-generated and human-judged perceptual qualities of synthesized sounds contributes new insights into using DDSP in musical instrument modeling.
• Contributions to the broader field of audio synthesis through the development of perceptually grounded validation techniques for DDSP models.

Suggested courses & skills

• IN5060
• IN4260MCT
• MCT4052
• Familiarity with the state-of-the-art machine learning models for audio processing
• Skills Required: Python programming, deep learning (TensorFlow, PyTorch), signal processing, familiarity with music theory (helpful but not mandatory), and experience in audio data handling.