Artificial intelligence (AI) and computer vision are becoming increasingly important in applications that require machines to understand and interact with the real world. For example, smart environments, healthcare systems, robotics, and interactive applications often need to analyze human activities in real time. However, building such systems is challenging because real-world environments are complex: different sensors provide different types of information, AI models require significant computing resources, and applications need to respond quickly.
This project explores how multimodal sensing systems can be designed to combine information from different sensors, such as RGB cameras and infrared cameras, to achieve reliable human analysis. Students will investigate how it can be developed and deployed efficiently across different computing platforms, including edge devices and cloud servers.
The goal of this project is to build and evaluate a real-time system that can collect sensor data, process information using different technologies, and provide fast and reliable results. Depending on the project direction, students may work on areas such as sensor data processing, computer vision models, AI optimization, or edge-cloud system design.
A typical project may include the following steps:
- Review related work in multimodal sensing.
- Collect and process data from different data sources such as RGB and infrared sensors.
- Define approaches for human analysis tasks.
- Run system on edge devices, cloud platforms, or a combination of both.
- Test the system and analyze accuracy, speed, and resource usage.
The final system could, for example, demonstrate real-time human detection using multiple sensors, an activity recognition system running on an edge device, or an intelligent sensing platform connected to cloud-based services.
Learning Outcome
By the end of the project, students are expected to:
- Gain an understanding of multimodal systems, computer vision, and/or real-time technologies.
- Develop practical skills in designing system pipelines that combine different types of sensor data.
- Understand the challenges of deploying sensing system and machine learning models on edge devices or cloud platforms.
- Learn how to optimize AI models for real-time applications with limited computational resources.
- Gain experience in developing and evaluating systems in collaboration with researchers from both academia and industry.
Requirements for the Student
We are looking for highly motivated master students who are interested in tackling challenging problems in real-time systems, multimodal sensing, computer vision, and edge-cloud computing. Building such a system requires students who are comfortable working across different technical areas and are motivated to develop solutions for complex engineering and research challenges.
The master thesis will be conducted in a collaboration of IFE and ND. ND has adopted a model where master students take 60 credits worth of courses in the first year of the master studies, and work full-time on the master thesis in the second year. We ask you to plan accordingly. The following are some relevant courses for this thesis topic:
- IN5050, IN5060, IN5020, and IN4260MCT provide relevant background.
Students are expected to have a strong technical background and experience in some of the following areas:
- Programming skills in Python, alternatively MATLAB. Students should be comfortable developing, debugging, and integrating software components for AI and sensing systems.
- Programming skills in C/C++. This is required to integrate sensor platforms and real-time systems.
- Machine Learning and Computer Vision: Basic knowledge of machine learning, deep learning, and computer vision concepts is expected.
- System Architecture and Software Development: Interest and ability to design and implement complex systems that integrate sensors, AI models, data processing pipelines, and computing platforms.
- Data Processing and Sensor Systems: Experience with processing, analyzing, and handling data from different sources, including image, video, or other sensor modalities.
The project is designed for students who enjoy solving difficult technical problems and are motivated to work at the intersection of AI, sensing, and system engineering. Students should be prepared to work independently, explore new technologies, and contribute actively to the development of a complete real-time multimodal AI system.
At the same time, this project is a collaboration between IFI and IFE. Thus, students will be supported throughout the project by an experienced research and engineering team. The collaboration provides access to dedicated technical guidance, research expertise, facilities, experimental equipment, and computing resources. Students will have the opportunity to work closely with researchers and engineers, gaining hands-on experience in a professional research environment while developing their own solutions. In addition, students are expected to spend one day per week onsite at IFE facility in Halden during the implementation.