Facial action analysis (FAA) has shifted from controlled laboratory environments to complex real-world conditions, with deep neural networks now playing a central role in learning FAA representations. This project explores the affect and engagement of musicians and listeners in these natural settings, leveraging everyday equipment such as cameras to track subtle facial cues without the need for complex setups. The focus is on addressing challenges like varying lighting, occlusions, non-frontal head poses, identity biases, and subtle emotional expressions, while providing opportunities for students to explore areas such as analytical studies using public datasets (e.g., YouTube), real-time interactive systems, ethics, data privacy, and signal processing.
FAA is gaining widespread interest across fields like healthcare, entertainment, and education. It can monitor patient well-being, enhance therapeutic interventions, optimize content based on real-time audience reactions, and improve learning by tracking student engagement. Musical tasks are ideal for this research because they involve complex emotional expressions, offering rich data for analysis. Decoding the emotional dynamics of music-making can enhance our understanding of human cognition and lead to innovative tools and interfaces that support creativity and well-being.
Objective
The goal of this project is to develop non-invasive systems to track and interpret facial expressions during music-related activities. By leveraging machine learning in real-world settings, we aim to overcome obstacles like lighting variations and subtle expression recognition. A diverse dataset, with labels for age, gender, and ethnicity, is essential for robust FAA across demographic groups. Techniques such as multitask learning and transfer learning will be employed to meet these challenges.
Project Description
This project will explore facial action analysis in music-related tasks using computer vision and deep learning. Students will focus on analyzing physical indicators of changes in affective states and moods such as boredom or stress, testing the feasibility of using common equipment in real-world conditions, and may focus on the following areas:
- Real-time interactive systems
- Ethics and data privacy considerations
- Signal processing techniques
Key Tasks
- Facial Action Analysis: Investigate the application of deep learning techniques for the analysis and recognition during music-related tasks.
- Deep Learning Deployment: Using advanced algorithms to interpret physical cues linked to the users’ affective states and moods
- Equipment Utilization: Evaluate the feasibility and efficiency of non-specialized equipment for emotion tracking in real-world settings.
The students will join the In Sustainable Immersive Networking Lab (SIN-LAB) and work in a team setting. SIN-LAB provides ample opportunities for experimental studies as it is equipped with a wide range of multimedia (cameras, LIDARs, headsets, glasses VR) and haptic devices (haptic gloves, robot arms and hands, in-air touch feedback, and haptic body suit) along with networking equipment to support immersive networking applications.
Learning outcome
- Develop Cutting-Edge Skills: Gain expertise in state-of-the-art deep learning, FAA, and human-computer interaction technologies.
- Hands-On System Design: Experience designing and deploying real-time, interactive systems that operate in diverse real-world settings.
- Cross-Disciplinary Research Impact: Contribute to groundbreaking studies in areas ranging from music technology to healthcare, mental well-being, and education.
- Data Privacy & Ethics Mastery: Engage with the ethical challenges of facial data analysis and learn how to address privacy concerns in sensitive environments.
- Teamwork & Collaboration: Work in a collaborative lab environment, fostering teamwork with peers and mentors from various fields, and contributing to real-world problem-solving.
Conditions
- Participate actively in the weekly SINLab meetings
- Include the course IN5060 in the study plan, unless you have already completed a course on classical (non-ML) data analysis
- Include the course IN4260MCT in the study plan
- Experience with Python and relevant ML libraries (TensorFlow, PyTorch). Familiarity with web development techniques is a big bonus
- A good foundation in machine learning and computer vision
- Interest in Music Technology is not required but a big plus, such as a familiarity with music-related data analysis, music information retrieval, audio signal processing, or human-computer interaction.
- Awareness of ethical implications and regulations related to facial data collection, especially in diverse demographic settings.