Objective

This project allows students to focus on one or more key research areas:

• Long-Term Dependencies and Memory: To handle long-term patterns in music, students can design models that retain session-level information by combining learned and rule-based approaches. Temporal processing units like attention mechanisms can dynamically highlight recurring motifs, rhythms, or harmonic progressions across time. Additionally, explicit memory systems enable agents to recall key interactions, such as thematic elements or user inputs, from earlier in a session. 
• Variable Input and Output Durations: Instead of fixing sequence lengths, the model can dynamically adapt by analyzing input patterns to infer the structure (e.g., phrase or measure boundaries) and adjust outputs accordingly.
• Multimodal Data Processing: When integrating data types like audio, symbolic sequences, or visual cues, students could explore pairing feature extraction pipelines (e.g., spectrogram analysis for audio or keypoint tracking for visuals) with models that align these data streams. For example, a system might employ perceptual embeddings derived from face and pose landmarks, or spectral analysis of audio to reinforce the microtiming.
• Multi-Agent Structure: Agents can collaborate through role-specific or music-theoretical rules; for instance, one agent focusing on rhythm generation while another improvises over it using predefined harmonic constraints. For interaction, agents can dynamically adapt their behavior by monitoring user inputs and responding within established constraints (e.g., tempo synchronization). 
• Creative Variation and Diversity: To ensure musical outputs are varied yet coherent, generative rules for embellishments (e.g., drum fills, arpeggios) can be coupled with probabilistic systems that decide when and how often these appear. A generative agent might add ornaments and fills based on interval rules or rhythmic densities while occasionally breaking patterns to maintain interest. 
• Data Processing and Feature Extraction: Processing inputs like audio and symbolic data involves music information retrieval techniques at large. For audio, this might include extracting rhythmic onsets, pitch contours, or timbral features (e.g., spectrograms or MFCCs). For symbolic data, these could be analyzing rhythmic patterns like beat histograms and harmony features.

Project Description

This project focuses on designing and developing interactive multi-agent systems for music generation with multimodal capabilities. Students will start by reviewing relevant literature on multimodal interaction and generative music systems to identify gaps and opportunities. They will then explore agent architectures that combine rule-based and learning-based approaches to handle multimodal inputs (e.g., symbolic, audio, video) and enable real-time collaboration with human users.

The project includes building an interactive interface and conducting user studies to evaluate the system’s performance and user experiences.

The students will join the In Sustainable Immersive Networking Lab (SIN-LAB) and work in a team setting. SIN-LAB provides ample opportunities for experimental studies as it is equipped with a wide range of multimedia (cameras, LIDARs, headsets, glasses VR) and haptic devices (haptic gloves, robot arms, and hands, in-air touch feedback, and haptic body suit) along with networking equipment to support immersive networking applications.

Learning outcome

By the end of the project, students are expected to:

• Gain an understanding of multi-agent systems and human-computer interaction concepts in the context of music generation.
• Develop the ability to design flexible frameworks that integrate diverse data types and support dynamic, real-time user interactions.
• A critical perspective on rule-based vs. learning-based approaches in generative and interactive tasks in terms of creative, technical, and perceptual challenges.
• Build and apply evaluation methodologies that consider technical performance, aesthetic coherence, and user satisfaction in generative music systems.

Suggested courses & skills

• IN5060
• IN4260MCT
• MCT4052 (or equivalent)
• Basic understanding of music theory and rhythm structures,
• Knowledge of data processing and feature extraction techniques
• Experience with Python and machine learning libraries such as TensorFlow or PyTorch.