Imagine you play the guitar and wish to enjoy with your band the most rad rock n roll riffs you learned, but you don't have a drummer. Drum sets are expensive, loud, and hard to carry around. What if there was a "drummer buddy" that lives in the browser and is always ready to rock with you as soon as you put on your VR headset? That can be possible with state-of-the-art artificial intelligence (AI) systems. See, for instance, Google's DrumBot.
However, AI doesn't have a body. But drummers do, and seeing them hitting the drums hard is undoubtedly essential to the fun of playing music together. Moreover, musicians often use their bodies to communicate, keep the groove, signal the accented beats, or express their enjoyment. Thus, the task of creating an engaging virtual representation of such a "drummer buddy" is not trivial.
First, it is a design problem. What kind of "drummer buddy" would you like to play with if you had to choose an avatar? Would you prefer it to be a human, an octopus with tentacles all around the drum set, or a scary fictional character, such as Skeletor? If that looks like an animal, what kind of facial expressions and gestures should it have so that you can enjoy playing music together?
Second, it is a mapping problem. How do you relate your avatar's movements to the sonic and musical features and your bodily expressions that can be captured on the webcam so that the user perceives them as meaningful and engaging?
Your starting point in this master's thesis is a jam session scenario in the style of rock music. You will design a number of avatars and create mappings between your avatar's actions and pre-recorded drum loops. Then, you will evaluate your design by conducting user studies. We want you to investigate how your design considerations influence the users' QoE and in what ways they can be improved in future work.
The students will join the In Sustainable Immersive Networking Lab (SIN-LAB) and work in a team setting. SIN-LAB provides ample opportunities for experimental studies as it is equipped with a wide range of multimedia (cameras, LIDARs, headsets, glasses VR) and haptic devices (haptic gloves, robot arms and hands, in-air touch feedback, and haptic body suit) along with networking equipment to support immersive networking applications.
Learning outcome
- Combine your technical and creative skills;
- Hands-on experience in interdisciplinary system design and user study;
- Gain experience in applying machine learning (data-based design) and/or mathematical optimization (model-based design) in interaction design;
- Contribute to the rapidly improving CS sub-field that focuses on creative human-AI interactions in XR contexts;
- Possibility to publish the result and attend the top international conference.
Suggested courses & skills
- IN5060
- Programming skills: basic Python and C++ (Unreal) or C# (Unity)
- Interest in VR and music, obviously.