Merge branch '3d' of https://github.com/equinor/team-dogleg into webp…

…age_plot
jakvah · Jul 29, 2020 · 2e32d3d · 2e32d3d
2 parents 85d3374 + ba4c2db
commit 2e32d3d
Show file tree

Hide file tree

Showing 10 changed files with 611 additions and 303 deletions.
diff --git a/README.md b/README.md
@@ -1,10 +1,51 @@
-# team-dogleg
-Team repo for Team Dogleg - Virtual summer internship 2020
+# Automate well trajectory planning using artificial intelligence - Team Dogleg
+Team repo for Team Dogleg, working on the case *Test reinforcement learning to automate well trajectories* - Equinor Virtual Summer Internship 2020. 
 
-## Run locally
-It is recommended to [run using docker](#run-with-docker-using-windows), however if you don't have Docker available it is possible to run it *normally* on you computer.
+## Table of Contents
 
-### Installations
+- [What it is](#what-it-is)
+    - [The problem](#the-problem)
+    - [The soltuion](#the-solution)
+- [Quickstart](#quickstart)
+    - [Run locally](#run-locally)
+        - [Installations](#installations-(local))
+        - [Run the program](#run-the-program)
+        - [Examples of running the program](#examples-of-running-the-program)
+    - [Run with Docker using Windows](#run-with-docker-using-windows)
+        - [Installations](#installations-(docker))
+        - [Building and running the Docker container](#building-and-running-the-docker-container)
+    - [Run with Docker using MacOS/Linux](#run-with-docker-using-macos/linux)
+- [Available algorithms](#available-algorithms)
+- [Adjust training parameters](#adjust-training-parameters) 
+- [Using the custom environment](#using-the-custom-environment)
+    - [Installation](#installation-(gym-environment))
+    - [Quick start](#quick-start)
+    - [Advanced use](#advanced-use)
+        - [Overwriting the starting conditions](#overwriting-the-starting-conditions)
+        - [Toggle hazards, Monte Carlo simulated training and episode log](#toggle-hazards,-monte-carlo-simulated-training-and-episode-log)
+        - [Adjust environment parameters](#adjust-environment-parameters)
+        - [Key functions to utilize when training](#key-functions-and-attributes-to-utilize-when-training)
+
+
+
+## What it is
+
+### The problem
+Planning the trajectory of a subsurface well is to a large extent a manual process. This is resoruce demanding and the resulting well path is not usually not the optimal path to the desired target location. It is therefore desirable to automate this process with the goal of simplify the well planning process, saving resources and creating more optimal well paths.
+
+### The solution
+
+This repository proposes software attempting to automate the planning of a well path trajectory using reinforcement learning. An agent is trained in an environment modelling a subsurface environment with hazards (no-go areas) and a set of target balls. It uses a [custom made](gym-drill/README.md) environment compatible with [OpenAI Gym](https://gym.openai.com/). The reinforcement learning is implemented using a set of implementations of reinforcement learning algorithms from [stable-baselines](https://stable-baselines.readthedocs.io/en/master/).
+
+
+## Quickstart
+
+This repository offers two ways to quickly train, retrain or load a trained model using a variety of the [available stable-baslines reinforcement learning algortihms](#available-algorithms). The solution has been dockerized for easy "shipping around-ability", however it is also possible to simply execute the python code outside of Docker on your host machine.  
+
+### Run locally
+It is recommended to [run using docker](#run-with-docker-using-windows), however if you don't have Docker available it is possible to run it *normally* on your computer.
+
+### Installations (local)
 
 Install the requried dependencies:
 
@@ -48,7 +89,7 @@ The arguments are summarized in the table below.
 | Further train existing agent                       	| retrain  	| \<save_name>  \<algorithm_used> 	| \<timesteps>  \<new_save_name> 	|
 | Load and display existing agent in the environment 	| load     	| \<save_name>  \<algorithm_used> 	| -                              	|
 
-### Examples
+### Examples of running the program
 To further clarify how to properly run the program three examples demonstrating the three possible actions are shown.
 
 1. To **train** a new agent for a total of **600** timesteps using the **DQN** algorithm and saving the agent as **my_trained_agent** do:
@@ -67,17 +108,18 @@ To further clarify how to properly run the program three examples demonstrating
 ```
 
 ## Run with Docker using Windows
-### Installations
+Running with Docker ensures that the code can execute on any computer with Docker installed. For user convenience all interactions with the Docker container is done from a PowerShell script. Should detalis regarding the specific Docker commands be of interest the user is reffered to [the script itself](run.ps1). It is recommended that you run the script from a PowerShell terminal as administrator. To get started, make sure you are allowed to execute scripts by setting the execution policy to unrestriced. In PowerShell, run
+
+```
+~$ Set-ExecutionPolicy Unrestricted
+``` 
+### Installations (Docker)
 
 Running with Docker is recommended to ensure a satisfying runtime environment. If you haven't already, install [Docker for Windows](https://docs.docker.com/docker-for-windows/install/).
 
 ### Building and running the Docker container
 
-For user convenience all interactions with the Docker container is done from a PowerShell script. Should detalis regarding the specific Docker commands be of interest the user is reffered to [the script itself](run.ps1). It is recommended that you run the script from a PowerShell terminal as administrator. To get started, make sure you are allowed to execute scripts by setting the execution policy to unrestriced. In PowerShell, run
-
-```
-~$ Set-ExecutionPolicy Unrestricted
-```
+Building and running the docker container is done by passing different flags as arguments to the ``./run.ps1`` script.
 
 #### Building the Docker container
 Build the Docker container using the ``-build`` flag:
@@ -116,6 +158,27 @@ All previously mentioned flags that can be passed to ```run.ps1``` have a corres
 
 Currently we don't have a fancy script for Linux users. To run with linux you would have to manually pass the commands from the ```run.ps1``` script into the terminal.
 
+Build the container using
+```cmd
+~$ docker build -t auto_well_path .
+```
+
+Then, run the conatiner using
+```cmd
+~$ docker run -dit --mount type=bind,source="$(pwd)",target=/usr/src/app -p 0.0.0.0:6006:6006 -p  0.0.0.0:8988:8988 --name auto_well_path_running auto_well_path
+```
+If wanted, start the tensorboard server displaying the tensorboard graphs in the browser using
+
+```cmd
+~$ docker exec -dit auto_well_path_running tensorboard --logdir /usr/src/app/tensorboard_logs --host 0.0.0.0 --port 6006
+```
+
+Finally, start the python program using
+```
+~$ docker exec -it auto_well_path_running python main.py <action> <save_name> <algorithm> <timesteps>* <new_save_name>*
+```
+
+where the arguments are the same as the ones in the [running the program locally section](#run-the-program).
 ## Available algorithms
 
 The following algorithms to train agents are available 
@@ -125,9 +188,114 @@ The following algorithms to train agents are available
 
 More algorithms comming soon!
 
-## Life advice when using docker
+## Adjust training parameters
+
+To tweak parameters related to the training in stable-baselines, see the [agent_training](gym-drill/agent_training.py) file together with the stable baselines documentation for the [specific algorithm you want to alter](#available-algorithms).
+
+## Using the custom environment
+
+If it is desirable to perform your own non-stable-baselines training or just customize the training in general, using the custom made OpenAI Gym compatible environment you can use the custom Gym-Drill environment.
+<!--## Custom Well Trajectory Environment-->
+
+Gym-Drill is a custom, OpenAI Gym compatible environment modelling a subsurface reservoar. Its main purpose is to be used in a reinforcement learning context to train an agent to find the best path from a given starting point, to a set of target balls. The environment inflicts a curvature constraint (dogleg severity limitation) on the agents well path. An option to include Hazards (hard constraints that the path cannot intersect) into the subsurface environment also exist.
+
+<!--### Table of Contents
+
+- [Installation](#installation-(gym-environment))
+- [Quick start](#quick-start)
+- [Advanced use](#advanced-use)
+    - [Overwriting the starting conditions](#overwriting-the-starting-conditions)
+    - [Toggle hazards, Monte Carlo simulated training and episode log](#toggle-hazards,-monte-carlo-simulated-training-and-episode-log)
+    - [Adjust environment parameters](#adjust-environment-parameters)
+    - [Key functions to utilize when training](#key-functions-and-attributes-to-utilize-when-training)
+-->
+### Installation (Gym environment)
+
+Using the custom environment requires [OpenAI Gym](https://gym.openai.com/) (duh), [matplotlib](https://matplotlib.org/) for displaying well paths and [numpy](https://numpy.org/) for number crunching. If not already installed, run from the terminal:
+
+```
+~$ pip install gym numpy matplotlib
+```
+
+To install the custom gym environment, navigate to the ``gym-drill`` folder
+
+```cmd
+~$ cd gym-drill
+```
+
+and run:
+
+```cmd
+~$ pip install -e .
+```
+
+Thats it. Happy training!
+### Quick start
+
+To use the environment both ``gym`` and ``gym_drill`` must be imported to the python file where an instance of the environment will be created. Notice the difference between the ``gym-drill`` and the ``gym_drill`` folder (TODO: what is the difference, haha). To create an instance of the environment, use OpenAI Gyms ``make()`` function and pass the environment name ``drill-v0`` as argument
+
+```python
+import gym
+import gym_drill
+
+env = gym.make("drill-v0")
+```
+This will create an instance of the environment with the agent (bit) starting in position **(1000,1000,0)** with the values for the  [Inclination and Azimuth angles](https://www.researchgate.net/figure/a-Azimuth-and-inclination-when-drilling-a-directional-well-b-The-azimuth-of-a_fig2_320511730) beeing drawn uniformly with intervals (0,pi/4) and (0,2pi) respectively. See [register function](gym_drill/__init__.py) for details.
+
+### Advanced use
+
+There are several parameters that can be adjusted to change the environment, both the "physical" aspects and the rewards for different actions.
+
+#### Overwriting the starting conditions
+To initialize an instance of the environment with your own specificed parameters do
+
+```python
+env = gym.make(env_name,startLocation = STARTLOCATION, bitInitialization = BIT_INITIALIZATION)
+```
+where `STARTLOCATION` is of type [**Coordinate**](gym_drill/envs/Coordinate.py) and `BIT_INITIALIZATION` is a list/tuple on the form ```[initial_azimuth_angle, initial_azimuth_angular_velocity, initial_azimuth_angular_acceleration,initial_inclination_angle, initial_inclination_angular_velocity, initial_inclination_angular_acceleration```. An example of creating an environment with custom parameters would be:
+
+```python
+import gym
+import gym_drill
+import numpy as np 
+
+from gym_drill.envs.Coordinate import Coordinate
+
+STARTLOCATION = Coordinate(0.0,0.0,0.0)
+BIT_INITIALIZATION = [0.0,0.0,0.0,0.0,0.0,0.0]
+
+env_name = 'drill-v0'
+env = gym.make(env_name,startLocation = STARTLOCATION, bitInitialization = BIT_INITIALIZATION)
+```
+#### Toggle hazards, Monte Carlo simulated training and episode log
+
+In addition to [overwriting the starting conditions] of the agent (bit), there exist options to toggle hazards in the environment, to train with Monte Carlo simulated environments in order ensure the existence of a feasible path and a episode based log that gives you realtime updates regarding the training.
+
+- Toggle hazards by passing ``activate_hazards = True`` as a keyword argument to the ``make()`` function. This will enrich the environment with hazards of amount and size as specified in the [environment config file](gym_drill/envs/environment_config.py). See the [Adjust environment parameters](#adjust-environment-parameters) section for details. By default this is set to ``True``
+
+- Toggle Monte Carlo simulated training by passing ``monte_carlo = True`` as a keyword argument to the ``make()`` function. This will ensure that an agent training in the environment always will be exposed to an environment where a feasible path to all targets exist. This is done by first generating a set of random paths and then populate those paths with targets. The details of the Monte Carlo simulation is specified in the [environment config file](gym_drill/envs/environment_config.py). See the [Adjust environment parameters](#adjust-environment-parameters) section for details. By default this is set to ``False``
+
+- Toggle the episode log by passing ``activate_log == True`` as a keyword argument to the ``make()`` function. This will write the amount of steps and total reward from each episode to a file named "drill_log.txt". This log will contain the total amount of steps **NOTE: Using the log will greatly reduce performance during training.** It is recommended that the log is used when tweaking the reward system or during very thorough training. By default this is set to ``False``.  
+
+As an example, if you want to turn of hazards and Monte Carlo simulated training, but see behind the scenes magic written in the log, you would do
+
+```python
+env = gym.make("drill-v0",activate_hazards = False,monte_carlo = False,activate_log = True)
+```
+
+#### Adjust environment parameters
+
+The environment and an agent exeperience in the environment is described by a set of variables that control physical attributes, movement limitations, rewards, what is contained in the observation space and more. These are all stored in the [environment_config](gym_drill/envs/environment_config.py) file. If you feel like changing aspects of the environment for yourself by tweaking these variables all you have to do is update the values inside this file.
+#### Key functions and attributes to utilize when training
+As the environment is OpenAI gym compatible it has all the attributes and functions you would expect to be in an OpenAI gym environement pr [the documentation](https://gym.openai.com/docs/). These include, but are not limited to:
 
-- List all your docker images by running `docker image`
-- Delete a docker image using `docker image rm [IMAGE ID]`. You can find the ID of the image by listing all images using `docker image`
+- A ``reset()`` function which resets the environment to its initial state. Note that even if you are [overwriting the starting conditions](#overwriting-the-starting-conditions) the Azimuth and Inclination angle will be drawn randomly, to ensure that the training is not beeing overfitted for one particular starting angle.
+- A ``step()`` function that accepts an ``action`` and executes that action in the environment. In accordance with the documentation the ``step()`` function returns:
+    - An ``observation`` object, containing the new state of the environment after the action has been executed
+    - A ``reward`` (float), which is the reward the agent recieved for exectuing the particular action 
+    - A ``done`` signal (boolean) indicating wheter or not the episode is over
+    - An ``info`` (dictionary) message containing diagnostic information useful for debugging. 
 
+The only deviation from the functions describes in the documentation is that the ``render()`` function that most OpenAI gym environment use to display the environment has been replaced with two seperate functions. ``display_planes()`` for displaying the horizontal (xy) and vertical (zy) planes and ``display_3d_environment()`` which displays the path and environment in a 3D plot.
 
+*Last updated 28.07.2020*