Detecting Emotion with Computer Vision

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the emotion_detection.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Download the emotions.zip file. This contains the images our model will train on.
4. Within your Google Drive, click on ‘MyDrive,’ then create a new folder and rename it emotion_detection. Inside the folder, upload the emotion_detection.ipynb file and the emotions.zip file. You do NOT need to unzip this file before uploading it.
5. Double-click on the emotion_detection.ipynb file. This should automatically open in Google Colab.
   1. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find in that section.
   2. Run the block under Importing Libraries to ensure you have access to all the functions we will use for this project.

Viewing the Dataset

1. (Code Block 1A) This first code block will unzip the emotions.zip file you uploaded earlier. Run this code block.
   1. Once this code block is done running, you should see a new folder called images inside your emotion_detection folder. Within the images folder, you should see seven more folders named after the seven emotions: Anger, Disgust, Fear, Happy, Neutral, Sad, and Surprised. 
   2. If you don’t see the folders right away, wait a few minutes and refresh.
2. (Code Block 1B) This code block will display a count of images in each folder.
   1. Does the data look evenly distributed? If not, how can we address the class imbalance?
3. (Code Block 1C) This code block defines a helper function that lets us view some of the images in the notebook. Run this code block.
4. (Code Block 1D) Under the #TODO comment in this code block, you can change the name of the folder to view images from that folder. It is set to “Anger” by default. You can run this code block multiple times to view different images.
   1. Note: Make sure to type the emotion folder exactly as it appears under Code Block 1B. For example, “Neutral” is correct but not “neutral”. 

Preprocessing

1. (Code Block 2A) This code block converts the images to grayscale. Images are often converted to grayscale because it makes them simpler for the computer to study. Instead of looking at many different colors, the model only needs to focus on light and dark areas, which helps it notice shapes, edges, and facial expressions more easily. Run this code block.
   1. Converting to grayscale means the computer has less information to process, so training can be faster and use less computing power. For emotion detection, the most important clues usually come from facial features and expressions, not from the exact colors in the image.
   2. At the bottom of this code block, you should see the grayscale image's size printed. The image should be 64x64 pixels.
      1. Note: 64x64 is typically large enough for basic emotion classification if the image is a tightly cropped face (as in this dataset). In fact, some datasets use smaller images, although more subtle emotions may suffer at low resolution. 

Creating the DataFrame

1. (Code Block 2B) This code block creates a DataFrame, or data table, to organize our information. This makes the data easier for the machine learning model to understand and learn from. Run this code block.

Splitting the Data into Train, Validation, and Test Sets

1. (Code Block 2C) This code block splits our dataset into training, validation, and test sets to help prevent overfitting, which occurs when a model memorizes the training data rather than learning generalizable patterns. It also makes sure the model is tested on data it has never seen before. Click on this link to learn more about splitting our data into train, validation, and test sets. Run this code block.

Creating Image Data Generators

1. (Code Block 2D) This code block creates an ImageDataGenerator that prepares images for training by loading them in small batches and generating slightly modified versions to help the model learn more effectively. For example, it may create images that are slightly rotated, flipped, or more or less saturated. Run this code block.

Building the CNN Model

1. (Code Block 3A) Under the #TODO comment, you can edit the filters. Filters are pattern detectors. Early filters might detect edges or simple shapes, while deeper filters can detect more complex facial features. You can check out the documentation for the convolutional layer here. filter_1 is the number of filters for the first layer, filter_2 is the number of filters for the second layer, etc. There are four hidden layers. A hidden layer is any layer in a neural network that comes between the input layer and the output layer.
   1. Note: The number of filters has a big effect on how much the model can learn. More filters give the model greater power to find patterns, but too many can slow training and lead to overfitting.
   2. Try these 3 different models and compare the results:
      1. Small model: 8, 16, 32, 32 
      2. Medium model: 32, 64, 128, 128 
      3. Large model: 64, 128, 256, 256 

Compiling and Training the Model

1. (Code Block 4A) This code block will compile and train the model you defined in Code Block 3A. Next to the #TODO comment, you can adjust the number of epochs the model trains for. An epoch is one full pass through the entire training dataset. By default, the model is set to train for 10 epochs.
   1. Note: This code may take a few minutes to run, so this is a good time to stretch, take a short break, or grab a snack.

Evaluating the Model

1. (Code Block 5A) This code block creates two graphs to show how your model performed during training.
   1. Training and Validation Accuracy Graph: This graph shows how often the model correctly predicts the emotion at each epoch. 
      1. Training accuracy shows how well the model does on the data it was trained on. 
      2. Validation accuracy indicates how well the model performs on new data it was not trained on.
      3. If both lines go up, the model is learning.
      4. If validation accuracy is also high, the model is performing well on new images rather than just memorizing the training set.
      5. If training accuracy gets much higher than validation accuracy, the model may be overfitting. That means it learned the trained images too specifically and does not generalize as well to new ones.
   2. Training and Validation Loss: This graph shows how far off the model is at each epoch. 
      1. Training loss measures error on the training data.
      2. Validation loss measures error on the validation data.
      3. If both lines go down, that is usually a good sign.
      4. Lower loss means the model’s predictions are getting closer to the correct answers.
      5. If training loss keeps going down but validation loss starts going up, that is another sign of overfitting.
   3. How to read both graphs together:
      1. A strong model usually has:
         1. Accuracy increasing
         2. Loss decreasing
         3. Training and validation lines that stay fairly close together
      2. A model that is overfitting often has:
         1. Training accuracy much higher than validation accuracy
         2. Training loss is going down while validation loss goes up
2. (Code Block 5B) This code block will print the final validation accuracy. Run this code block.
   1. What was your model’s final validation accuracy?
   2. If your validation accuracy was low, what changes could you make to improve your model?
3. (Code Block 5C) This code block prints the model’s test accuracy, which indicates how well the trained model performs on the test dataset. The test dataset contains images the model has not used during training or validation, so test accuracy gives a final check of how well the model might perform on brand-new images. Run this code block.
   1. What was your model’s test accuracy?
   2. Was your test accuracy higher, lower, or about the same as your validation accuracy? What might that mean?
4. (Code Block 5D) This code block displays a confusion matrix, a table that shows how well the model classified each emotion.
   1. In a confusion matrix, the actual emotion labels are compared with the model’s predicted emotion labels. Correct predictions usually appear along the diagonal of the matrix. Values outside the diagonal show where the model made mistakes, such as predicting “happy” when the actual label was “surprised.”
   2. Use the confusion matrix to look for patterns in the model’s errors. For example, are there certain emotions the model predicts very well? Are there emotions that are often confused with it?
5. (Code Block 5E) This code block will randomly display one image that the model classified incorrectly. It will show the image, print the actual emotion label, and print the emotion the model predicted instead. You can re-run this block multiple times to show a different image each time.
   1. Look closely at the image. Why do you think the model confused these two emotions? Are there facial features, lighting, image quality issues, or ambiguous expressions that might have made the prediction difficult?
   2. Do you think the model could improve with more training data or a better architecture, or are some emotions naturally difficult to predict from facial expressions alone?

Testing Your Emotion Detection Model with Your Own Images

In this section, you will test your emotion detection model using either a webcam photo or an image uploaded from Google Drive.

You only need to choose one option:

1. Use Code Blocks 6A-6B to take a picture directly in Google Colab using your webcam.
   1. (Code Block 6A) This block defines a helper function that lets us capture images directly within the Google Colab notebook. Run this code block.
   2. (Code Block 6B) This code block runs the helper function from Code Block 6A, allowing you to take a picture using your webcam. When you run this block, Google Colab will ask for permission to access your camera. Click Allow. If you do not have a webcam or do not want to grant camera access, you can use your own images instead, using Code Blocks 6C-6D.
      1. After running the block, your camera preview will appear with a Capture button. Click Capture to take a picture. The model will then print its predicted emotion.
      2. Was the predicted emotion the same as the emotion you were showing? Try making different facial expressions and see how well the model recognizes them.
2.  Use Code Blocks 6C-6D if you do not have a webcam, or do not want to give camera access, or prefer to test an image from Google Drive.
   1. (Code Block 6C) This code block defines a helper function that lets us test images stored in Google Drive. It also creates a folder called test if one does not already exist.
      1. After running this block, check your Google Drive for the new test folder. If you do not see it right away, refresh the Google Drive page.
   2. (Code Block 6D) This code block runs the helper function from Code Block 6C, allowing you to test your model with an image from Google Drive.
      1. First, upload the image you want to test to the test folder in your Google Drive. Then, under the #TODO comment, enter the name of the image file you uploaded (e.g., "happy.jpg")
      2. When you run this block, the model will print its predicted emotion below the code.
3. Fill out a table similar to the one below. To fill out the confusion matrix table, record the emotion you were trying to show and the emotion the model predicted.
   1. The rows show the actual emotion. This is the emotion you were trying to show in your webcam photo or uploaded image.
   2. The columns show the predicted emotion. This is the emotion the model printed after testing your image.
   3. Test at least 10 images or facial expressions for each emotion: surprised, sad, neutral, happy, fear, disgust, and anger. This means you should test at least 70 total images or webcam photos. Testing multiple examples of each emotion gives you more reliable results and makes it easier to see patterns in the model’s predictions.
   4. Each time you test an image, find the row for the actual emotion and the column for the predicted emotion. Then add a tally mark or increase the number in that box by 1.
      1. For example, if you made a happy face but the model predicted neutral, you would add 1 to the box where the Happy row and Neutral column meet.
      2. If the model correctly predicts the emotion, the count will appear on the diagonal of the table, where the actual and predicted emotions match. For example, if you made a sad face and the model predicted sad, add 1 to the Sad-Sad box.
      3. After you finish testing, look at the table to see which emotions the model predicted well and which it confused with others.

4. After you complete the confusion matrix, review your results and look for patterns. Which emotions did the model predict correctly most often? Which emotions were confused with each other?
   1. Then go back to Code Block 3A and test different numbers of filters in the CNN model. Train the model again, test it with your images, and compare the new confusion matrix to your first one.
   2. You can also check the project variations section for additional parts of the model you may be able to change. After each change, record your results so you can compare how different model settings affect the model’s performance.