Background
Electromyography (EMG) is a technique which is used for gesture recognition, enabling control of prosthetics, human-computer interfaces, and robotics. These systems work by measuring the electrical signals generated by muscle contractions. While effective in controlled lab settings, their performance in the real-world are severely degraded by real-world environmental factors. Specifically, temperature substantially impacts physiological processes, altering skin impedance, muscle fiber conduction velocity, and autonomic nervous system responses. These changes can distort EMG signals and affect other biosignals like Photoplethysmography (PPG) and actigraphy, which are crucial for monitoring physiological state. By simultaneously recording EMG, Inertial Measurement Unit (IMU), PPG, EDA, ECG and direct skin temperature data, a multimodal system can be developed to understand and mitigate the effects of temperature, leading to more robust and reliable gesture recognition.
Problem Statement
Current EMG-based gesture recognition systems are typically trained and tested under stable, thermoneutral conditions (~22 °C). Consequently, their accuracy degrades substantially when the user is exposed to different temperature (e.g. going outside in the winter in Norway). This performance decline is due to temperature-induced changes in the user's physiological state that are not accounted for by the models. This lack of environmental robustness is a major barrier to the deployment of EMG-based interfaces in practical, everyday applications where temperature fluctuations are common. There is thus a need to develop machine learning models that can intelligently fuse multimodal biosensor data to maintain high gesture recognition accuracy regardless of the ambient thermal conditions.
Objective and Methodology
This project's primary objective is to develop and validate a machine learning framework that leverages multimodal biosignals to create an environmentally robust hand gesture recognition system. The goal is to design a model that maintains high classification accuracy across neutral, hot, and cold conditions by using PPG, EDA, and temperature data to contextualize and compensate for changes in EMG and IMU signals. The core challenge is to effectively fuse these disparate data streams to create a unified model resilient to temperature-induced physiological variations.
Methodology
This project will be done over three semesters, using a pre-existing dataset.
- Semester 1: Data Exploration and Literature Review. The project will begin with a thorough exploration of the provided dataset, which contains synchronized EMG, IMU, PPG, EDA, and temperature recordings from participants performing gestures in neutral, hot, and cold environments. The student will perform exploratory data analysis to understand the data structure and visualize the impact of temperature on each signal type. In parallel, a comprehensive literature review will be conducted to investigate the physiological effects of temperature on neuromuscular and autonomic signals, and to identify existing machine learning strategies for sensor fusion and domain adaptation in biometrics.
- Semester 2: Signal Processing and Baseline Modeling. A data processing pipeline will be developed for each modality. This includes filtering and segmenting the signals and extracting relevant features. For EMG and IMU, time-domain, frequency-domain, and time-frequency features will be extracted. For PPG, EDA, and temperature, features quantifying autonomic response and thermal state will be derived. To establish a performance benchmark, a baseline gesture recognition model (e.g., a Convolutional Neural Network or Random Forest) will be trained using only EMG data from the neutral environment. This model's performance will be evaluated on data from all three thermal conditions to quantify the degradation caused by temperature changes.
- Semester 3: Multimodal Fusion and System Validation. The final phase will focus on designing, training, and validating advanced multimodal fusion models. Several architectures will be investigated, from simple early-fusion (feature concatenation) to more complex hybrid models that use separate processing streams for gesture (EMG/IMU) and context (PPG/EDA/Temp) data. Advanced deep learning architectures, such as those incorporating attention mechanisms, will be explored to allow the model to dynamically weigh the importance of different features and compensate for environmental effects. The performance of the final fused models will be rigorously compared against the EMG-only baseline across all temperature conditions. The project will culminate in the writing of a master's thesis and the preparation of a manuscript for submission to a peer-reviewed conference or journal.
Expected Outcomes
The anticipated outcome of this project is a novel multimodal machine learning model that outperforms traditional EMG-only systems in gesture recognition accuracy when operating across a wide range of ambient temperatures. The project is also expected to deliver a detailed quantitative analysis of how temperature variations impact different biosignals. Further outcomes will include a comprehensive comparison of various sensor fusion strategies, a publicly available open-source codebase for reproducing the results, and a master's thesis that contributes new knowledge to the field of robust human-computer interfacing.