Happy or Sad? Use Artificial Intelligence to Classify Faces

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Just like little kids learn to distinguish happy from sad faces by seeing examples, Teachable Machine will learn from examples you provide. We will refer to this set of examples as the training data because your model will learn from this set of drawings. 

Getting Familiar with Teachable Machine

1. Before you start this project, it might be fun and helpful to play with Teachable Machine. Go to the Teachable Machine new project web page and use the webcam to train the model to recognize whether you or a stuffed animal is looking at the camera.

   The Teachable Machine Tutorial 1 video explains how to gather training data. Go ahead and gather training data. If your computer does not have a webcam, take digital pictures and upload those to the model instead. Then, watch the Teachable Machine Tutorial 2 video to learn how to train the model.

   The Teachable Machine Tutorial 3 video shows how to understand the predictions.

   Try these steps to get familiar with Teachable Machine and be sure to watch the tutorials if you get stuck.

Creating Training Data

1. Draw examples of happy faces in twenty circles (two sheets of the template) and sad faces in another set of twenty circles (two templates). Always use the same pencil or pen to draw your faces.
2. Cut along the lines and make a stack of happy and a stack of sad faces.
3. If you do not have a digital camera or scanner you can use the webcam to take pictures in Teachable Machine. This will be explained below.

   If you have a digital camera or scanner and can easily transfer the files to the computer:

   1. For each drawing, place the face on a plain background like a sheet of colored construction paper, as shown in Figure 5. Take a picture or scan. Try your best to take every picture the same way, laying the sad/happy face in the same spot, at the same angle, same distance, etc.
   2. Transfer the files to the computer. Organize them (by giving them clear filenames or putting them in a folder) so you can clearly identify them as the training data later.

Image Credit: Sabine De Brabandere, Science Buddies / Science Buddies
Figure 5. A drawing that is ready to be photographed (lower right corner), and six drawings that have already been photographed (upper left corner).

Preparing the Model

1. Go to the Teachable Machine new project web page for recognizing images. 
2. Label the classes.

   The model will classify drawings of faces into two groups or classes: happy faces and sad faces. Rename class 1 as "Happy" and class 2 as "Sad."

Uploading Training Data

1. If you already saved the learning data files to your computer, use the "Upload" button to upload the training data files for happy faces to the "Happy" class, and the training data files for the sad face to the "Sad" class.
2. If you will use the webcam in Teachable Machine to take pictures:
   1. Start the webcam for the "Happy" class by clicking the "Webcam" button.
   2. Place a happy face drawing on a plain background like a sheet of colored construction paper.
   3. Hold the drawing close to the lens, so the drawing fills most of the space.
   4. Take all the pictures with the camera at the same angle so the lighting stays consistent.
   5. Try to keep your fingers out of the picture. Teachable Machine has a function that crops the pictures as you are taking them. This can help you crop your fingers out of the pictures.
   6. Repeat step b–e for all happy drawings in your training data. Try your best to take every picture in the same way.
   7. Once all happy faces of the learning data are uploaded, move on to the "Sad" class, repeat steps b–f for uploading the sad faces learning data. Try your best to take these pictures the same way as you took the happy face pictures.

You now have twenty pictures in the training data for each class. If this is not the case, remove doubles or add pictures where needed.

It is fine if the training data shows some variation in lighting, distance, etc. It is important though that these variations occur in both classes. For example, it is not OK to have all the pictures in the "Happy" class to be close-ups and all the pictures of the "Sad" class to be taken from further away. But having about the same portion of close-ups and pictures taken from further away in both classes is fine.

Training the Model

1. Start the model's learning process by clicking on the "Train Model" button. The model should take less than a minute to complete this step. In that time, the model for searches patterns in the training data it can use to distinguish between happy and sad drawings.

Saving the Model

1. All the uploaded data on the machine can be saved to Google Drive and later uploaded from the Drive to continue. Look for the commands "Save project to Drive" and "Open project from Drive" under the "Teachable Machine" menu. It will ask you to log in to a Google account to access your Google Drive.

   The project can also be saved as a file on your computer and uploaded later. Look for the commands "Download project as file" and "open project from file" under the "Teachable Machine" menu.

Testing the Model

1. To see how the model performs, you can use the webcam or files. Figure 6 shows how to toggle between the two.
   Image Credit: Sabine De Brabandere, Science Buddies / Science Buddies
   Figure 6. Teachable Machine allows you to use files you upload or the webcam to test the tool.

   One by one, choose a few happy and a few sad drawings from the training data, and test how the model classifies these drawings.

   The bars under the test picture in the "output" box inform you how the model classifies the drawing. Your model will probably classify its training data with high confidence, as shown in the left image of Figure 7 where we see the model is certain the image belongs to the "Happy" class.

   Sometimes, the model is not clear about how to classify a drawing. The right image in Figure 7 is an example. The orange bar next to "Happy" shows the model classifies this drawing with a confidence of 59% in the "Happy" class. The red bar next to "Sad" shows it is 41% confident the drawing belongs to the "Sad" class.

   Image Credit: Sabine De Brabandere, Science Buddies / Science Buddies
   Figure 7. Two examples of an AI tool that learned to classify happy and sad face drawings.

   In this experiment, you classify a drawing in the class for which the tool shows a confidence of over 50%. With this rule, the picture on the right in Figure 8 is classified under "Happy" because 59 is higher than 50. Because this drawing is in fact a sad face, the tool receives a mark in "Sad drawing misclassified as "Happy."

   If you use the webcam, try your best to show the drawings the same way as you took pictures previously.

2. In your notebook, draw a confusion matrix like the one shown in Figure 8. Do not forget to fill in the amount of training data you used in the title.
   Image Credit: Sabine De Brabandere, Science Buddies / Science Buddies
   Figure 8. A confusion matrix helps organize the test results.

Testing New Data

The real test is seeing how well the tool can classify drawings it has never seen (drawings that were not part of the training data). 

1. Create five more happy face drawings and five more sad face drawings. 
2. One at a time, test each new face.
3. Record a new confusion matrix. How did your model do with the new faces that were not part of the training data?

Adding More Training Data

You have trained and tested a model that used twenty drawings in each class to learn the difference between a happy and sad face. You will now find out what happens to your model's accuracy if you add more training data.

1. Save the model under a new name. This will keep this model separate from the one that was trained with twenty drawings per class. You can update the name of your model by clicking the pencil icon located under the "Teachable Machine" menu.
2. Draw 15 more happy faces and 15 more sad faces. Combined with the 5 each that you drew in the previous section, you should now have 20 new faces for each class.
3. Using the webcam or file upload feature, add 20 new faces to each one of the classes (the files that you uploaded previously should still be there). 
4. Re-train the model.
5. Re-test the model as you did previously, first with faces that were included in the training data, then with new faces. Record a new confusion matrix each time.

Analyzing the Data

1. Compare and contrast the confusion matrices you have. The background section can help you understand the confusion matrix. Remember, blue represents accuracy, yellow represents misclassification. Higher numbers in the blue squares and lower numbers in the yellow squares indicate a better-performing model.
2. Did adding more training data improve the accuracy of your model? If not, you can continue to iteratively add more training data, re-test, and re-train the model. Do you reach a point of diminishing returns, where adding more training data does not significantly improve the model's accuracy? 

Communicating the Results

1. Add pictures of the learning data and classifications of the AI tool you created in your report. Figure 2 and Figure 3 are good examples.