Create a Facial Recognition System with Machine Learning

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. You have two options for downloading the zip file:
   1. Option 1: The facial_recognition.zip file does not include anchor and positive data. For this dataset, you will need to take your pictures to serve as anchor and positive data.
   2. Option 2: The facial_recognition_with_dataset.zip file. This dataset includes anchor and positive images of former Prime Minister Tony Blair.
3. Once you have downloaded one of the zip files, unzip it by right-clicking it in your Downloads and clicking 'Extract all.’ This will create a new file folder alongside the Compressed zipped file. Upload the folder called ‘facial_recognition’ or ‘facial_recognition_with_dataset’ (depending on which zip file you chose) inside the newly extracted folder.
4. Upload the file folder to Google Drive. You will need to sign in to your Google account at this point or make an account. Upload the file by clicking on the 'New’ button on the upper left corner of the main Google Drive screen, then selecting the 'Folder upload' option. Be sure to select the file folder, not the compressed zipped file.
5. There are two ways you can access the Google Colab notebook:
   1. Within your Google Drive, double-click the folder you just uploaded. Then, double-click on the file called 'facial_recognition.ipynb'. This should automatically open the notebook in Google Colaboratory.
   2. Another option is to go directly to Google Colaboratory. On the pop-up menu, select 'Google Drive’ then select the file called ‘facial_recognition.ipynb’.
6. Change the Runtime type by clicking on ‘Runtime,’ then ‘Change runtime type.’ Next, click on the ‘T4 GPU’ option and save.
7. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find there.

Overview

Before you start, briefly review the purpose of each section, which is listed below. The explanations provided here are brief, but you can read more detailed explanations by expanding the block using the triangle icon to the left of each section title. You can also collapse it again by clicking on the triangle icon again. You can also expand the blocks to check their status; a green checkmark indicates the block has been run.

1. Setup. This section installs and imports libraries that include special machine-learning functions we will use in our code.
2. Collect Positives and Anchors. In this section, if you are doing Option 1, you will collect positive and anchor images necessary for training our model. If you are doing Option 2, this section has already been completed for you.
3. Load and Preprocess Images. This section prepares the positive and anchor images you provided and the negative images we provided for training.
4. Model Engineering. This section creates the Siamese neural network model.
5. Training. This section trains the model using the data that you collected and preprocessed.
6. Evaluate Model. In this section, you will analyze the model's results to see how accurate it is. Was it able to tell the difference between you and other people?
7. Save the Model. This section saves the model so that you can reuse it for another project without having to wait for a new model to train.
8. Real-Time Test. In this section, you can further test your model by testing it in real-time, taking new pictures, and seeing if it will still be able to tell the difference between you and other people.

Now, let us start the project!

1. Setup

This section should have only hidden cells. You can run all of them at once by clicking on the play icon to prepare your coding environment. You can also expand the cells by clicking the arrow to the left of the section title.

2. Collect Positives and Anchors

If you are using the facial_recognition_with_dataset.zip file (Option 2), all you need to do for this section is run the entire section 2. For Option 1, follow the instructions as follows:

1. Helper Functions for Collecting Images: These code blocks contain code that will allow you to take photos and save them to your Google Drive. Run this code block.
2. Collect Positive and Anchor Classes: This code block will use your webcam to let you take anchor and positive images of yourself. You will need to give Google Colab permission to use the webcam. If you’d prefer, you can also upload images directly to the folder by opening Google Drive, locating the folder, and uploading your images in the ‘positive’ and ‘anchor’ folders. Once you’ve taken or uploaded the pictures, run this code block.
   1. The function asks for user input. Press ‘a’ to save an anchor image, ‘p’ to save a positive image, and ‘q’ to quit the function. Press ‘Enter’ on your keyboard after you type in one of the options. Click ‘Capture’ under the code block to take the picture.
      1. These two types of images should be of the same person. Take multiple images, alternating between saving anchor and positive images. Aim to take between 100 to 300 images for each type (anchor and positive). It is recommended to alternate between taking anchor and positive images so that you have an equal amount of data for each type, but it does not matter as long as you end up with roughly the same number of each type.
         1. You can turn your head and take multiple images of your face at different angles. This helps the model to recognize you from all angles.
         2. You can also make different facial expressions. This helps the model recognize you no matter what facial expression you may be making.
         3. You can also try different hairstyles, makeup styles, and accessories (e.g., glasses) to help the model learn to recognize your face even if you change your look.
      2. If you quit the program before taking enough pictures, you can simply run the cell again to take more.
3. Function to Count Images in Anchor and Positive Directories
   1. The ‘count_files_in_directory’ function can help you keep track of how many images you have saved so far. When you run it, it will print how many pictures you have in the ANC_PATH and POS_PATH directories. Feel free to run this block multiple times. You may re-run the block above it to continue taking pictures.

3. Load and Preprocess Images

This code prepares images for facial recognition, organizes them for training and testing, and ensures efficient model use. Expand the cells to read more details in the comments. Run this code.

4. Model Engineering

This code creates two models: one to generate embeddings (numerical representations of images capturing their important features) and the Siamese neural network to compare facial similarities using these embeddings. Expand the cells to read more details in the comments. Run this code.

5. Training

1. Sections 5.1-5.4 prepare the Siamese neural network for training. Expand the cells to read more details in the comments. Run each cell.
2. Section 5.5 trains the Siamese neural network. This training may take a while because of the data size, model complexity, and other factors. Run this code.

6. Evaluate Model

1. Make Predictions: Run these code blocks.
   1. The first code block gets a batch of test data from the test dataset for evaluating the trained Siamese network. It includes input images (‘test_input’), validation images (‘test_val’), and true labels (‘y_true’).
   2. The second code block predicts similarities between input and validation images using the trained Siamese network. It generates predictions stored in ‘y_hat’, which helps assess the network’s performance and accuracy against true labels. This step aids in evaluating the Siamese neural network’s performance on unseen data.
   3. The third code block simplifies predictions from the Siamese network by categorizing them as 1 for similarity if they are greater than 0.5 and 0 for dissimilarity if they are less than 0.5. It helps analyze the network’s ability to differentiate between similar and dissimilar image pairs.
   4. The fourth code block contains the true class labels. You can run the fourth code block to compare the predicted values with the true class labels.
2. Calculate Metrics: These blocks will output a value between 0 and 1, with 0 meaning 0% and 1 meaning 100%. Run these code blocks.
   1. The first code block calculates the accuracy metric for evaluating the Siamese neural network’s performance.
      1. Accuracy is a measure used to evaluate the performance of a machine-learning model. It represents the proportion of correctly predicted outcomes or labels compared to the total number of instances in the dataset. In simpler terms, accuracy tells you how often the model’s predictions are correct. For example, if the model has an accuracy of 0.25, it means it gets the answer right 25% of the time, or once every four tries. 
         
         Mathematically, accuracy is calculated as: 
         
         
         $$accuracy = \frac{number\ \:of \:correct \:predictions}{total\: number\: of\: predictions}$$
   2. The second code block calculates the recall metric for evaluating the Siamese neural network’s performance.
      1. Recall is another measurement used to evaluate the performance of a machine-learning model. Recall measures the ability of the model to identify all positive instances correctly. It answers the question: Of all the actual positive instances, how many did the model predict correctly? For instance, if the model has a recall of 0.90, it means that out of all of the true data, the model correctly identified 90% of them and 10% were missed or false negatives. 
         
         Mathematically, recall is calculated as: 
         
         
         $$recall = \frac{true\: positives}{true\:positives\: +\: false\: negatives}$$
   3. The third code block calculates the precision metric for evaluating the Siamese neural network’s performance.
      1. Precision is yet another measurement used to evaluate the performance of a machine learning model, and it focuses on the accuracy of positive predictions by the model. It answers the question: Of the instances the model predicted as positive, how many are actually positive? For instance, if the model has a precision of 0.80, it means that out of all the true data, 80% were actually true and 20% were false. 
         
         Mathematically, precision is calculated as: 
         
         
         $$precision = \frac{true\: positives} {true\: positives\: +\: false\: positives}$$
   4. In these formulas:
      1. True Positives (TP) are the cases where the model correctly predicted the positive class
      2. True Negatives (TN) are the cases where the model correctly predicted the negative class
      3. False Positives (FP) are the cases where the model incorrectly predicted the positive class when it was actually negative
      4. False Negatives (FN) are the cases where the model incorrectly predicted the negative class when it was actually positive

      		Actual
      		Positive	Negative
      Predicted	Positive	True Positive	False Positive
      	Negative	False Negative	True Negative

      Table 1. Comparing the predicted result with the actual result, we can determine whether the prediction was a true positive, false positive, true negative, or false negative.
3. Visualize Results: Run this code.
   1. This code block allows us to visually compare image pairs from the test dataset. You can change the value of the index (‘input_index’) to view different comparisons.
   2. Check if the model was accurate.
      1. If the y_hat and y_true values are both 0, that means that the model correctly identified the faces as not a match.
      2. If the y_hat and y_true values are both 1, that means that the model correctly identified the faces as a match.

7. Save Model

1. Run the code blocks.
   1. The first code block saves the trained Siamese model's weights to a file named “siamesemodel.h5” in a designated directory. This saves the model's learned parameters and configurations for later use, making sure it works the same each time and can be used with other applications or systems.
   2. The second block reloads your saved Siamese model. You can resume training, make predictions, or continue analyses without starting from scratch.
   3. The third block makes additional predictions with your reloaded model.
   4. The fourth block prints a summary of the Siamese neural network model's architecture.

8. Real Time Test

1. Sections 8.1-8.2 include code that allows real-time facial image verification. Expand the cells to read more details in the comments. Run these code blocks.
2. Section 8.3 calls the ‘real_time_verification’ function. When this code runs, you can press 'v’ then ‘Enter' on your keyboard to open the webcam again. Make sure to click on 'Capture’ to capture a new image. The code will take a moment to verify whether the image matches the existing anchor images. If it matches, it will output 'True’, and if the image does not match, it will output ‘False.’
   1. Experiment with changing the ‘detection_threshold’ and ‘verification_threshold’ values.
   2. Start by changing only the detection_threshold value. Start at 0 and increase by 0.1 (0.1, 0.2, 0.3, etc.) until you get to 1.  At which level does it start to recognize faces?
   3. Next, keep the detection_threshold at 0.5, and change the verification_threshold. Start at 0 and increase by 0.1 (0.1, 0.2, 0.3, etc.) until you reach 1. At which level does it start to recognize your face?
   4. Finally, experiment with changing both values to see which values of detection_threshold and verification_threshold are best for recognizing a face as yours or not.
3. Try seeing if the model can tell the difference between you and a family member. If you are using the facial_recognition_with_dataset.zip file, try seeing if the model can tell the difference between you and Tony Blair (search for an image of Tony Blair on the internet and show it to the camera). If it cannot, try increasing the threshold by 0.1 until it can. Experiment with changing the threshold values for both the detection_threshold and verification_threshold to find which combination is the best for recognizing your face or Tony Blair’s face.