Detect Litter with Machine Learning

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the litter_detection.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Within your Google Drive, click on ‘MyDrive,’ then create a new folder and rename it “litter_detection.” Inside the folder, upload the litter_detection.ipynb folder.
4. Double-click on the litter_detection.ipynb file. This should automatically open in Google Colab. You will do the following in the Google Colab environment:
   1. Change the Runtime type by clicking on ‘Runtime,’ then ‘Change runtime type.’ Next, click on the ‘T4 GPU’ option if it is not already selected and save.
   2. Read the Troubleshooting Tips and How to Use This Notebook sections in the notebook. Follow the instructions you find in that section.
   3. Run the block under Importing Libraries to ensure you have access to all the functions we will use for this project.

Collecting the Data

You have the flexibility to choose the type of data you collect. To begin, we will focus on three types of trash: cans, cardboard, and plastic. However, feel free to adjust the categories based on what you have in your trash or recycling bin, or what you find outdoors.

1. Using a phone or any other camera, capture at least 100 images of each object. While multiple objects can appear in the same image, ensure that no two images are too similar. Specifically, you need to collect images of three distinct objects: one can, one piece of cardboard, and one plastic item.
   
   For each object, you must take 100 images. This can be achieved in two ways: either by capturing 100 images that include all three objects together or by taking 100 images of each object individually. 
   
   To create a diverse dataset, vary the images in different ways, such as: 
   1. Changing camera angles by rotating around the object.
   2. Rotating the object itself.
   3. Using different backgrounds (e.g., gravel, grass, dirt).
   4. Stacking or nesting objects (e.g., placing a bottle inside a cardboard box).
   5. Experimenting with different lighting conditions (e.g., adjusting for sunlight direction, taking photos under a tree).
   6. Varying the zoom level to capture both close-up details and wider shots.
   7. These are just a few ideas–feel free to explore other variations to make your dataset more robust! 
2. Inside the litter_detection folder is another folder called data, inside that is images, and finally, within images, there is a folder named train. Upload your images to the train folder.
3. For the test data, record a video of the objects. You can either pan across all of the objects or move them around while filming. You can try tossing or flipping the objects around, but remember that the model might not recognize the object while it is in your hand, so remember to set it down every time you move the object. The video should be in MP4 format. Once recorded, upload your videos to the litter_detection/test folder.

Labeling the Data

Note: The instructions for this part of the procedure are written for an external website. The interface for the website or the exact steps you need to take may change if the website is updated.

1. Next, we will label our data using the Computer Vision Annotation Tool, CVAT.ai, a free online tool for annotating images and videos. You may use other tools such as LabelMe or VIA if you would like, but you will need to adapt your procedure since the steps below are written for CVAT.ai. Labeling means drawing boxes around objects in your images and assigning them a category, such as ‘can’ or ‘plastic.’ This step is crucial for training an AI model to recognize different types of trash. To get started, create an account by clicking ‘Sign In’ at the top right corner of the website. 
2. To start a new project, click on ‘Projects,’ then select the plus (+) icon, and choose ‘Create a new project.’
3. Give your project a name, such as ‘litter_detection.’
4. Click the ‘Add label’ button, enter a label name (e.g., can, cardboard, or plastic), and select ‘Continue.’ Repeat this process for each label you need.
   1. If you accidentally create an unwanted label, you can delete it by clicking ‘Cancel,’ then selecting the trash icon next to the label you want to remove.
5. Once you have entered all your labels, click ‘Submit & Open.’
6. Then, click the plus (+) icon and choose ‘Create a new task.’
7. Give the task a name, like ‘litter_detection_annotation.’
8. Upload your images here as well, then click ‘Submit & Open.’ This upload may take a while depending on the speed of your internet connection and the number and size of your image files. Wait for it to finish.
9. You will see a job labeled ‘Job #_______.’ The blanks will be filled with numbers. Click on that job to proceed.
10. To start labeling the images, press ‘n’ on your keyboard. You will see horizontal and vertical lines appear, which are just guidelines. Use these guidelines to draw a box around every piece of litter in the image. On the ‘Object’ tab on the right side of the screen, ensure that each rectangle you have drawn is assigned the correct label (e.g., can, cardboard, plastic, etc.).
11. You can save your progress by clicking Ctrl + S. It is good practice to save frequently. To move to the next image, press 'f.' You can press 'd' to return to a previous image. You can also use the arrows at the top to navigate.
    
    
    Image Credit: Science BuddiesFigure 2. Example of what the object tab in the labeling tool might look like, with three different rectangles labeling a can, cardboard, and plastic.
12. Repeat steps 10-11 until all of the images have been labeled.
13. Next, go to the ‘Tasks’ tab in your CVAT account. Find the task you just completed, click the three dots next to it, and select ‘Export Task Dataset.’ Choose ‘YOLO 1.1’ as the export format, then click ‘OK.’
    1.  A pop-up will appear indicating that the export process has started, followed by another pop-up with a download link once it is finished. 
    2. Note: If you miss the pop-up, you can find the download link under the ‘Requests’ tab.
14. After downloading the annotations and unzipping the folder, navigate to the inside of the folder and you should see that there are .txt files for each image. Upload these files to the litter_detection/data/labels/train folder on your Google Drive.

Preparing the YAML file

Now, we will prepare the YAML (a recursive acronym for ‘YAML Ain’t Markup Language’) file in the Google Colab notebook. The YAML file is used to define the dataset configuration. It tells the model where to find the images and labels, which classes are included, and how many categories exist. This file is essential for training because it helps the model understand how the dataset is structured and what it needs to learn.

1. (Code Block 3A) Below the #TODO comment, you will find the section where you can update the class names. Replace them with the labels you used in CVAT, assigning each class name a unique integer.

Training the Model

1. (Code Block 4A) We have provided the code to create and train a YOLO (You Only Look Once) model. Simply run the code block–training may take a few hours depending on the number of images, so please be patient.

Testing the Model

1. (Code Block 5A) Under the #TODO comment in this code block, enter the name of your test video, then run the block. The results will be saved as an MP4 file with the same name, prefixed by “output_”.
   1. Once the code finishes running, you can find the processed video in the litter_detection/test folder. If you have multiple videos, update the name and run the block again for each one. Do not worry if it takes some time to complete–it may take a few minutes.
   2. How well did your model perform? Watch the entire video and keep a tally of how many pieces of trash are correctly and incorrectly identified in a data table (see example table below). (Optional: For a more in-depth analysis, calculate the accuracy, precision, and recall for each object, and investigate why one object might be misclassified as another). If the results are not as accurate as you would like, here are some ways to improve it:
      1. Gather more data – Increasing the dataset size can help the model learn better. Focus on collecting more images, especially for any classes that were frequently misclassified. Try capturing images in different lighting conditions, angles, and backgrounds to improve model robustness.
      2. Adjust training parameters – Inside Code Block 4A, you can find a hyperparameter called ‘epochs.’ Increase this number to 150, 200, etc. to see if that improves your model.
         
         Swipe left to see more

         Table 1. Example data table.

         		Identified as
         		Can	Cardboard	Plastic
         Item	Can
         	Cardboard
         	Plastic