Using AI to Detect Proper Exercise Form

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the exercise_form.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Within your Google Drive, click on ‘MyDrive,’ then create a new folder and rename it exercise_form. Inside the folder, upload the exercise_form.ipynb folder.
4. Double-click on the exercise_form.ipynb. This should automatically open in Google Colab. You will need to do the following in the notebook:
   1. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find in that section.
   2. Run the block under Importing Libraries to ensure you have access to all the functions we will use for this project.

Collecting the Data

You have the flexibility to choose any exercise for your model to classify as having good or bad form–such as pull-ups, push-ups, squats, or lunges. Just make sure that you (or whoever is demonstrating) can consistently perform the chosen exercise with proper form.

1. Record at least 100 short videos of the chosen exercise performed with good form using a phone or any camera. Each video should feature one person completing a single repetition and should be consistent in length (e.g., each video should be between 2 and 4 seconds). Ensure the footage focuses solely on the exercise–avoid including actions like walking to or from the camera. If you are filming alone, trim out these parts. Having a second person record the exercise can make this process easier and result in cleaner videos. Here are some video-taking tips to make your model robust: 
   1. Change the camera angle between shots by rotating around the person performing the exercise.
   2. Have the person rotate their body before starting the exercise in different recordings.
   3. Vary the zoom level between recordings, taking closer shots and further shots (make sure the whole person can still be seen).
   4. These are just a few ideas–feel free to explore other variations to make your dataset more robust!
2. Inside the exercise_form folder is another folder called exercise_videos, inside that is a folder called good_form. Upload the videos of the exercise with good form to the good_form folder.
3. Repeat Step 1, but this time perform the exercise using bad form. While there are many ways to perform an exercise incorrectly, it is important to stay consistent and collect multiple videos of the same type of poor form. For example, if you are recording squats, one common mistake is letting the knees extend past the feet. In this case, you should capture at least 50 videos from different angles showing this specific error.
   1. If you want your model to detect additional types of poor form, you will need to collect separate sets of videos for each. Aim to gather at least 50 videos for every variation of poor form you plan to include.
4. Once you have recorded the poor form videos, upload them to the exercise_form/exercise_videos/bad_form folder.
5. For your test data, record at least two additional videos of the same person performing the exercise – one with good form and one with bad form. To keep things organized, it is a good idea to rename the files to something like good_form_test(1).MP4 and bad_form_test(2).MP4 so you can easily tell them apart. Upload these test videos to the exercise_form/test_videos folder.

Saving Keypoints

Now, head back to the Google Colab environment to move on to the next section of the project.

1. (Code Block 1A) This code block processes videos of exercise repetitions using a YOLOv8 pose estimation model. It loops through folders of good and bad form videos, runs pose detection on each frame, and creates annotated versions of the videos with drawn keypoints (specific body landmarks like shoulders, elbows, knees, etc.) and skeletons (lines connecting those keypoints to represent the body’s posture). At the same time, it extracts the keypoint coordinates and confidence scores for each frame, saving them into CSV files for further analysis. This prepares clean, labeled data for training or evaluating a model to classify exercise form. Run this code block. This may take a few minutes. 
   1. You can view the labeled videos in the exercise_form/exercise_labeled_videos folder. Inside, you will find two subfolders–one for good form and one for bad form–each containing their respective labeled videos.
   2. You can view the CSV files the model will use for training in the exercise_form/exercise_keypoints folder. You will again find two subfolders–one for good form and one for bad form–each containing their respective CSV files.

Training the Model

1. (Code Block 2A) This code block extracts features from keypoints CSV files, which contain detected body keypoints for each frame of a video. From these points, the algorithm creates feature vectors for machine learning. It calculates the mean and standard deviation of keypoint positions to summarize posture and movement, and then stores these feature vectors for each video. The code loops through both good and bad form video folders to extract the features, and labels them accordingly (0 for good form, 1 for bad form). It then prepares the data as arrays (X for features, Y for labels) to be used for training a machine learning model. Run this code block. This may take a few minutes. 
2. (Code Block 2B) This code block splits the exercise form dataset into training and test sets. It trains an XGBoost classifier on training data, predicts the form labels for the test set, and evaluates the model’s performance using a classification report. Run this code block.
   1. We split our dataset into train and test to avoid overfitting and ensure the model is tested on data it has never seen before. You can check out this video to learn more about why it’s important to use a training dataset for this AI algorithm.
   2. We have used an XGBoost classifier for this project, but feel free to experiment with other machine learning models and algorithms to compare their performance. To learn more about the XGBoost algorithm, check out this resource.
   3. The classification report shows performance metrics like precision, recall, F1-score, and support for both good and bad form exercises, as well as overall accuracy. To read more about these metrics and how to read a classification report, here is an article we recommend reading.
3. (Code Block 2C) This code block saves the trained machine learning model (the XGBoost classifier) to a file so you can reuse it later without needing to retrain it. Run this code block.

Testing the Model

1. (Code Block 3A) Under the #TODO comment in this code block, replace the placeholder with the name of your test video file. Remember, your test videos are located in the exercise_form/test_videos folder. The code will then run pose detection on the video, saving an annotated version and a CSV file with keypoint positions to the same folder. 
2. (Code Block 3B) This code loads the trained model, gets features from the test video’s CSV, and predicts if the form is good or bad. It then prints a message based on the result. Was your model able to tell if the exercise was good or bad form? Here are some suggestions for improving your model:
   1. Train the model with data from different people to help it generalize better across various body types and exercise forms, which can vary based on body proportions. This will enhance its ability to differentiate between good and bad form for more people.
   2. Include a wide variety of bad form examples in the training data to teach the model to recognize multiple forms of improper movement.