1. Project description
In this project, you will train a deep neural network to translate movement data from one emotion to another and explore how these transformations might work when applied to creatures with different morphologies using a simulation environment such as Gazebo, Unity, Blender, or others. The research questions addressed can be, but are not limited to the following:
RQ1. How can we translate emotional content from one human/or other creatures’ movement to another in virtual creatures?
RQ2. How can we develop mappings that work regardless of morphology?
Inspiration for expressive body movement can be taken, for instance, from silent body human movement in theater/acting lessons or from animation. If the student wishes to do some user studies, this is also welcome. The student may collect data from human users at the beginning of the process, before translating the movement data, through surveys or interviews. There is also the possibility of collecting respondent data after the mapping has been made, to validate the mapping/translation and explore how human respondents interpret the translation.
2. Project tasks
A suggested set of tasks could be the following:
- Get an overview of earlier work on expressive movements on robots, using deep learning models and other methods.
- Find a suitable dataset of movements labeled with emotion content (Or create your own in our MotionCapture Lab)
- Prepare a simulation environment for visualizing the movement sequences on various robots.
- Implement a sequence prediction or translation model such as the Transformer
- Train the model to take a motion sequence tagged with an emotional state as input and produce the same motion sequence performed in a different emotional state.
- Verify your approach using a questionnaire to see if your generated movements are perceived correctly. (user study)
- Compare the results to previous work available.
- Write the master thesis.
3. Starting point for references
[1] Paiva, Pedro VV, et al. "Emotion Transformer: Attention Model for Pose-Based Emotion Recognition." VISIGRAPP (5: VISAPP). 2023.
[2] F. Ahmed, A. S. M. H. Bari and M. L. Gavrilova, "Emotion Recognition From Body Movement," in IEEE Access, vol. 8, pp. 11761-11781, 2020, doi: 10.1109/ACCESS.2019.2963113.
[3] R. Santhoshkumar, M. Kalaiselvi Geetha, “Deep Learning Approach for Emotion Recognition from Human Body Movements with Feedforward Deep Convolution Neural Networks”, Procedia Computer Science, Volume 152, 2019, https://doi.org/10.1016/j.procs.2019.05.038.
[4] Aberman, Kfir, et al. "Unpaired motion style transfer from video to animation." ACM Transactions on Graphics (TOG) 39.4 (2020): 64-1.
[5] Christensen, J.F., Fernández, A., Smith, R.A. et al. EMOKINE: A software package and computational framework for scaling up the creation of highly controlled emotional full-body movement datasets. Behav Res (2024). https://doi.org/10.3758/s13428-024-02433-0
[6] T. Schulz, J. Torresen, and J. Herstad, “Animation Techniques in Human-Robot Interaction User Studies: a Systematic Literature Review,” ACM Trans. Hum.-Robot Interact., vol. 8, no. 2, Jun. 2019, doi: 10.1145/3317325.
[7] Osorio, P., Sagawa, R., Abe, N., and Venture, G. (2024). A generative model to
embed human expressivity into robot motions. Sensors, 24(2):569.
4. Course background
IN4140 Introduction to robot technology or good familiarity with ROS will be necessary to work with the robots.
IN4050 Introduction to artificial intelligence and machine learning can be useful for implementing or understanding algorithms
IN5590 Rapid Prototyping of Robotic Systems may be useful if you want to add to a robot
IN5490 Advanced Topics in Artificial Intelligence for Intelligent Systems can be helpful for understanding how to read articles and write your master's.