External: Where to drill next? (NGI)

Goal

In this project, your goal is to:

1. Train a reinforcement learning (RL) agent (policy) to decide the location of the next ground investigation drilling.
2. Use the RL to optimise the drillplan for a given surface area.

Problem (a more thorough description is found here)

Drilling takes time and is expensive. Sometimes we don’t discover all risk zones (zones of quick clay, deep trenches in the soil/rock contact etc.). Often, we drill more holes than necessary in specific locations. When carrying out ground investigations, we want:

• to drill as few holes as possible in the shortest possible time
• to detect all risk zones.
• Reduce the uncertainty of the ground model below a certain threshold

Impact

• Drilling fewer holes might considerably reduce the cost and increase the speed of constructing a ground model. Improved ground models reduce risk in construction projects.
• Detecting more risk zones with the correct drilling might avoid a construction failure
• Detecting natural hazards more effectively might save lives

Suggest approach

Research questions

RQ1: How can a realistic simulation environment for ground investigation be developed as a basis to train an RL-agent (in 2D and 3D) to achieve its goals?

RQ2: How can a real-time reinforcement learning model optimize (cost, speed, uncertainty-level) the drillplan in ground investigation?

Subquestion 1: How can the trained policy be used to decide where to drill next?

Subquestion 2: How can this model be used to minimize the number of necessary holes in order to achieve a given level of uncertainty in the ground model which is based on the information from the drilling.

Code setup

• Make a directory structure for the code and commit it to a private Github account. In the following development, you should regularly commit code to Github. Supervisors will then be able to give better guidance.
• Control the environment using Python Poetry and the Python version using pyenv. Choose the latest possible version.
• Install the necessary packages, such as Stable Baselines 3 for RL development, Gymnasium, to build a custom RL environment, and Pytorch for the neural networks.
• Code each of your solutions in a new branch.

Implementation – high-level description

The list below involves several levels of increasing complexity and solutions closer to reality. You should at least try to complete steps 1-6.

It is recommended to use generative AI for guidance/drafting in designing the system and architecture, required that the candidate fully understand system.

Read up and summarize existing literature and methods in ways to use machine learning to decide on an optimal drillplan in ground investigation. This will be your literature review part in your thesis.

1. Implement the solution described in the first referenced paper in the reference list, using Python, the RL-library StableBaselines3 and Gymnasium from Farama Foundation. This involves a couple of subtasks:
   1. Make a simulation environment for a 2D soil profile with layers of different soils and set up the custom environment. Either use the technique with random fields described in the paper, or generate profiles using GemPy (https://www.gempy.org/), or some other technique.
   2. Implement the reward function
   3. Define the actions.
   4. Drill a hole and update the state (the 2D profile with interpolated properties from drill holes). Train the agent
   5. Test the trained agent on a simulated environment that is not used for training
   6. Provide a plot of the development of reward and plots of “discovered” vs. true soil-profile
2. Test on real data from NGI. You should build 2D profiles from single drillhole data from NGI and test your model on these. You must first assign the right soil type index after Robertson (2015).
3. Instead of the MLP solution outlined in the paper, you should use a CNN to capture the state of the 2D ground profile. The actions should be to drill holes anywhere along the profile, and the reward must be changed accordingly. Still, a central part of the reward is to minimize the difference between the interpolated 2D-profile and the real profile.
4. Extend to 3D. Create a 3D simulation environment. Use the open-source cloud compare program to visualize your environment. Use a 3D CNN (or other ML methods to capture the 3D information) to capture the state. For each drilling, the 3D state is updated by interpolation.
5. Extend to an irregular surface shape (so far, this has been a cube). This will lead to challenges in capturing the state with a CNN.
6. Extend to an irregular bedrock surface. You will now also capture and make a model of the bedrock surface.
7. Include the surface image so the model can see where it drilled.
8. Include logistical operations, such as the distance to the next drilling location and other factors that should be optimized for the next drilling.

Deliveries

• Progress plan for the project, including splitting the task into manageable parts
• The master thesis
• A GitHub repo of structured code, ready to be cloned and run, with a Readme file describing how to run the functionality.
• Optional (but recommended 😊). A scientific paper describing your findings. This paper could be delivered to a conference for review, such as the yearly NORA conference.

Earlier work in the topic

A first investigation into the task is carried out in a student project in IN5490 the autumn semester 2024. The master thesis project should inspect the code (Github repo) and the final paper from this project.

A paper presented at the ISMLG conference in Cork 2023 by Coello and Smyrniou, titled “Optimisation of in-situ soil investigation with Deep Reinforcement Learning” (https://www.researchgate.net/publication/373659484_Optimisation_of_in-situ_soil_investigation_with_Deep_Reinforcement_Learning).

The candidate should search for related literature and methods for solutions.

References

• A recent “lightweight” RL model for drilling optimisation: https://www.researchgate.net/publication/373659484_Optimisation_of_in-situ_soil_investigation_with_Deep_Reinforcement_Learning
• https://www.frontiersin.org/journals/applied-mathematics-and-statistics/articles/10.3389/fams.2021.689934/full
• https://www.mdpi.com/2079-9292/11/3/368
• https://typeset.io/questions/how-reinforcement-learning-can-help-mineral-exploration-3v2yukxvz2
• Deep reinforcement learning for optimal well control in subsurface systems with uncertain geology - ScienceDirect
• Information about data and the CPT-testing procedure: https://www.data-in-brief.com/article/S2352-3409(20)31498-0/fulltext

This is only a start. The candidate should search for related literature and methods for solutions. Try using litmaps with the references as a starting point: Litmaps | Your Literature Review Assistant. Example: https://app.litmaps.com/preview/81962282

Requirements/Qualifications

• Experience with Python programming
• Experience with machine learning, CNN, etc.
• Knowledge about reinforcement learning
• Beneficial to have knowledge of the Python libraries, Stable Baselines 3, Gymnasium, Pytorch

Recommended courses

• IN5400 Machine learning for image analysis
• TEK5040 Deep learning for autonomous systems 
• IN4050 Introduction to artificial intelligence and machine learning
• IN5490 Advanced Topics in Artificial Intelligence for Intelligent Systems