Train a Reinforcement Learning AI to Play Minesweeper

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the rl_minesweeper.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Within your Google Drive, click on ‘MyDrive,’ select the ‘+ New’ button and ‘File upload.’ Select the rl_minesweeper.ipynb file you just downloaded.
4. Double-click on the rl_minesweeper.ipynb file. This should automatically open in Google Colab.
   1. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find in that section.
   2. Run the block under Importing Libraries to ensure you have access to all the functions we will use for this project.
      1. After running all the blocks in this section, return to your Google Drive, and you should see a new folder called rl_minesweeper. This folder contains all of the files necessary to run our Minesweeper environment. You can read more about the code on the GitHub page here.

Editing the Hyperparameters

In this section, you will improve your Minesweeper AI the same way engineers improve a design: pick a goal (like “win more games”), set limits (how long training takes), then change one hyperparameter at a time, train, and test to see what works best. You may have to make a tradeoff–training longer can improve win rate but it can also take a lot more time. This cycle of “try->test->improve” matches the Engineering Design Process steps.

1. In this code block, you will see that the #TODO comment is where you can edit the model's hyperparameters. You will see that there is a long list, but here are the ones to focus on:
   1. Within board, there are height, width, and num_mines.
      1. Here, you can adjust the height and width of the Minesweeper board, as well as the number of mines on the board. It is recommended that you start with a small board (4x4, 5x5, 6x6) and a low number of mines (1-3).
   2. epsilon_decay controls how fast epsilon decreases. 
      1. ɛ (epsilon) is the chance of taking a random action; it determines the balance between exploration (trying new moves) and exploitation (using the best known move). If ɛ decays too fast, the agent may stop exploring too early; if it’s too slow, learning can be inefficient. Watch this video to learn more about the exploration/exploitation trade-off.
      2. Our agent starts with an ɛ of 1, meaning it is 100% exploring at the beginning and choosing random actions instead of following what it currently thinks is best. As ɛ decreases over time ( epsilon_decay), the agent explores less and exploits more:
         1. With ɛ = 0.5, it acts randomly about 50% of the time.
         2. With ɛ = 0.1, it acts randomly about 10% of the time.
         3. With ε = 0.05, the agent chooses a random action about 5% of the time. Since 0.05 is the minimum ε, once the agent reaches this value, it continues exploring at a steady 5% rate for the rest of training. If you want, you can change this minimum–the lowest you can set it to is 0, which would mean the agent eventually stops taking random actions completely and always uses what it thinks is best.
      3. learning_rate_a is how big of a “step” the model takes when it tries to fix a mistake. Watch this video to learn more about the learning rate.
         1. A big learning rate = big steps, which means it can learn faster but might “overshoot” or overcorrect and get worse or act unstable.
         2. A small learning rate means taking small steps, which is steadier and more cautious, but it learns more slowly.
      4. For all other hyperparameters, we recommend only changing them if you are more familiar with machine learning.
      5. Run this code block after you finish making your changes.

Training the Agent

1. This code block will train the agent for a number of episodes. By default, it is set to 100, but to see meaningful results, it is recommended to change it to between 300,000-1,000,000.
   1. If you have a smaller board (6x6 and smaller), you should try starting with 300,000 and see how your agent performs in the later sections.
   2. If you have a larger board (6x6 and larger), try starting with 500,000 and increasing up to 1,000,000.
   3. Run this code block once you have changed the number of episodes.
   4. IMPORTANT: Even if your runtime disconnects, this code block will continue training your model. For example, if you finished training for 300,000 episodes, running this code block again will train the agent for another 300,000 episodes, for a total of 600,000. If you want to start a new model from scratch, it is best to rename your folder from rl_minesweeper to something like rl_minesweeper_v1, then download a new notebook and start from the section "Setting Up the Google Colab Environment" again. 

Visualizing the Data

1. Run this code block to display training data. You will see four graphs displayed below.
   1. The “Episode Length per Episode” graph shows the number of steps the agent can take on the board before the game ends, either when the agent finishes the game or when it hits a mine. Ideally, the agent would be able to take more steps per episode over time. 
   2. The “Epsilon Decay Over Time” graph displays the epsilon value over several episodes. Here, you can see how quickly epsilon decays and decide whether you want your epsilon to decay more slowly or faster.
   3. The “Average Loss per Episode” graph shows how much the agent’s predictions differ from the training target while it learns. In general, you want the loss to decrease over time, which suggests the agent is learning. Some ups and downs are normal, but if the loss remains very high or is highly unstable, you may need to adjust hyperparameters, such as the learning rate.
   4. The “Total Reward per Episode” graph shows how many reward points the agent earned in each episode. This is one of the best overall signs of improvement: if the agent is learning, the total reward should usually increase over time (though it may be noisy). If the reward stays flat or gets worse, it may mean the agent isn’t learning effectively–or that the reward system needs to be adjusted.

Evaluating the Data

1. Run this code block to have your agent run 100 different Minesweeper games and calculate its win rate. How well did your agent perform?

Changing the Agent

1. Iteration is a required part of the Engineering Design Process, so you should keep testing and improving your agent. To do this, make a data table to track each version of your model and decide whether your changes are worth it based on your criteria and constraints (for example: higher win rate vs. longer training time). After each run, record your settings and results, then choose your next change and try again. To improve your agent, you have two main options:
   1. Keeping training the same agent: Go back to Training the Agent and rerun that section (and the sections after it) to train for more episodes. Remember that continuing training means that the agent continues learning on top of what it has already learned.
   2. Start a new version of the agent (recommended for comparing changes to the hyperparameters): Rename your current folder from rl_minesweeper to rl_minesweeper_v1, then download a fresh copy of the notebook and repeat the Setting Up the Google Colab Environment steps. This lets you create a new agent with different settings while keeping your old results.
      
      
      Swipe left to see more

      Table 1. Example data table. You can add more rows for additional changes. 

      Version #	Episodes Trained For	Board Size	# of Mines	Other Hyperparameters Changed	Win Rate (%)
      V1	300,000	6x6	3	None	-
      V1	600,000	6x6	3	None	-
      V2	500,000	5x5	3	Changed epsilon_decay to 0.99999	-