Abstract
Driverless cars use machine learning, a type of artificial intelligence, to identify road signs and other things they encounter on the road, like pedestrians and other vehicles. In this project you will take your own pictures of different types of road signs and use machine learning to identify the signs using a free online tool (no programming is required). You will then attempt to improve your machine learning model so it can identify signs in a variety of environments.Summary
Objective
Train a machine learning model to classify pictures of different types of road signs.
Introduction
Humans are very good at identifying objects visually. For example, you can look at Figure 1 and immediately tell that one sign is a "curve ahead" sign and the other is a stop sign. This is a very important skill when driving, since drivers constantly need to identify and obey traffic signs.
Image Credit: Science Buddies
Figure 1. Left: A curve sign. Right: A stop sign.
While it is very easy for humans to perform this task, it is very difficult to write a computer program that can do it. Traditional programs follow strict predefined rules that are written by a human. If you were to write a program like this, you might write a description of each sign and then write rules telling the computer to look for certain characteristics. For example, the curve sign is yellow and diamond-shaped and has a curved black arrow in the middle. The stop sign is red and octagonal and has the word "stop" in white in the middle. You could write a computer vision program—a computer program that analyzes images or videos to extract information—to search for these characteristics in an image.
This approach only works well, however, if you have near-perfect pictures of the signs. What if a sign is splattered with mud, partially covered with snow, or marked with graffiti? What if it is nighttime, so the lighting in the picture is different and the sign appears to be a slightly different color?
Or what if the picture of the sign is taken at an odd or extreme angle, like the sign in Figure 2? As a human, you can still readily identify the curve sign, but a computer program written to look for a wider diamond shape, like the one in Figure 1, might have trouble.
All of these factors make identifying road signs and other objects a challenging task for driverless cars (also called autonomous or self-driving vehicles). To operate safely, they must be able to identify signs and objects in a wide variety of conditions and from many different angles.
Image Credit: Science Buddies
Figure 2. A picture of a curve sign taken at an extreme angle.
Ultimately, it is not feasible to write a set of rules to describe every possible view of a sign that a driver might encounter. Given that limitation, a driverless car that is programmed to follow a strict set of predefined rules might fail to properly identify a sign in a new or unexpected situation, and that could result in a crash. This is where machine learning comes in. Machine learning is a type of artificial intelligence.
With machine learning, instead of writing a program with pre-defined rules, programmers use a type of program called a neural network that functions more like the human brain. The program is given a massive amount of training data (see Figure 3), typically consisting of thousands or even millions of images—more images than humans could ever sit down to write rules for or descriptions of. Using advanced math, the neural network looks for patterns and develops its own rules. In other words, it "learns" to identify the images.
Image Credit: Science Buddies
Figure 3. Example training images of stop signs. Training images could also include pictures taken at different times of day or in different weather conditions.
There are many different types of machine learning. Supervised learning, the type you will use in this project, uses training data that has been pre-classified by a human (the "supervisor"). This type of learning works well when you want to teach a machine learning model (program) to classify different sets of objects, like road signs. This video provides an overview of supervised learning:
One challenge with machine learning models is that they can be subject to bias from their training data. For example, notice how all of the pictures in Figure 3 have a clear blue sky in the background, some snow on the ground, and trees without leaves. All of these pictures were taken near each other on the same day. A machine learning model trained with only this data might learn to associate these features with stop signs and have trouble identifying pictures of stop signs with other backgrounds (for example, a cloudy day, an urban environment with buildings, or a summer day with green plants). This video provides a brief overview of bias in machine learning models:
You can help prevent a model from developing bias by providing a diverse set of training data. In this case, that would mean pictures of road signs in a wide variety of settings: in different locations, at different times of day, and in different weather conditions.
In this project you will use a free online tool called Teachable Machine by Google® to train a machine learning model. The tool allows you to upload sets of training images to different classes (categories), train the model, and then evaluate the model by giving it new images to see if it classifies them correctly. You will attempt to improve your model by providing it with more training data, to see how accurate you can make it. You can do all of this without doing any programming yourself. Those who do have programming knowledge can export the model in order to examine it in more detail (see the Teachable Machine website for more information).
Terms and Concepts
- Computer vision
- Driverless car
- Autonomous
- Self-driving
- Machine learning
- Artificial intelligence
- Neural network
- Training data
- Supervised learning
- Model
- Bias
- Accuracy
- Confusion matrix
Questions
- Why is it difficult for a computer program to identify objects in images?
- What is the difference between machine learning and a traditional computer program?
- What are some advantages of using a machine learning program over a traditional program? What about disadvantages?
- What is bias in a machine learning model, and what causes it?
- Why is it important to provide a wide variety of training images to a supervised machine learning model?
Bibliography
- Google. (n.d.). Teachable Machine FAQ. Retrieved March 8, 2022.
- PBS LearningMedia. (2019). Supervised Learning #2 Crash Course: Artificial Intelligence [Video]. Retrieved March 8, 2022.
- Google. (2017, August 25). Machine Learning and Human Bias [Video]. YouTube. Retrieved March 8, 2022.
- Shroff, R. (2019, September 25). Artificial Intelligence Explained in Simple Terms. Medium. Retrieved March 8, 2022.
Materials and Equipment
- Smartphone or digital camera
- Computer with Internet access
- Lab notebook
Experimental Procedure
- Choose two different types of road signs that you will use for your machine learning model. You should choose signs where you will have safe, easy access to the same type of sign in different locations in order to take pictures. You may need to explore your neighborhood with an adult to look for signs or use an online tool like Street View in Google Maps®.
- Take at least 10 pictures of each type of sign. For example, if there are two stop signs within walking distance, you could take five pictures of each sign. Note that Teachable Machine will crop each image to a square aspect ratio, so make sure the sign is not too close to the edges of the image or it will get cropped out.
- Load the pictures onto your computer so they are ready to upload to Teachable Machine.
Note: As of March 2022, you must use Teachable Machine on a computer; it will not run directly on your phone.
Note: Websites can change frequently. If the following instructions do not match exactly what you see, just follow the on-screen instructions to use Teachable Machine.
- Go to Teachable Machine and click Get Started.
- Click Image Project.
- Click Standard image model.
- Edit the names of the two Classes in the model. For example, you could name them "Stop sign" and "Curve sign."
- Under Add Image Samples for your first class, click Upload. Select and upload the pictures of that type of sign from your computer. It may take some time to upload if you have a lot of images.
- Repeat Step 9 to add images for your other class.
- Click Train Model. Wait for the model to finish training. The model is "learning" to identify your two classes of images while you wait.
- Click the drop-down arrow next to Advanced and then click Under the hood.
- Read the information that pops up on your screen.
To summarize, the model only uses 85% of your images for training. It saves the other 15% to use as "test" images. That way, the model can test itself with images it has never seen before. It uses the results from the test images to determine the model accuracy, the percentage of images that were classified correctly for each class (expressed as a number between 0 and 1). It also produces a confusion matrix, which identifies how many images were classified correctly and incorrectly for each class. Hover your mouse over the "?" icon next to each item to learn more about it.
Note: Each time you re-train your model, it will choose different images as "tests," so the resulting accuracy and confusion matrix might be different.
- Now try testing your model with additional images. Try to find images that are different from the ones you used for training. You can go explore your neighborhood to take pictures of new signs, search for pictures of signs online, take screenshots from a tool like Google Maps, or even use an image editing program to edit pictures you already have (e.g., cutting and pasting signs onto different backgrounds).
- Under Preview, change the selection in the drop-down menu from Webcam to File.
- Under Preview, upload one of your new images. Under Output, the model will assign a probability that the image shows one of your two classes of signs (Figures 4 and 5).
Image Credit: Google Teachable Machine / Science Buddies
Figure 4. A machine learning model trained to classify stop signs and curve signs. The output detects a stop sign with 99% probability in the test image.
Image Credit: Google Teachable Machine / Science Buddiesscreenshot shows uploaded training images of curve signs and bump signs on the left. On the right is an output preview showing the results for a photo of a bump sign. The program has classified the image as a bump sign with 84% certainty.
Figure 5. A machine learning model trained to classify curve signs and bump signs. Even though it is obvious to a human viewer that the test image contains a bump sign, the model only assigns 84% probability that the image contains a bump sign. - Test your model's output with a variety of pictures of signs, in as many different environments as you can (different locations, different times of day, different weather, different objects in the foreground or background of the picture, etc.). How accurately does the model classify each image? Are there any images the model gets wrong or has trouble with (e.g., assigns roughly 50/50 probabilities), even if they are obvious to you? Can you identify features in the images that you think are causing problems?
For example, the test image in Figure 5 has a rectangular "no stopping or standing" sign below the bump sign in the foreground, but none of the training images in that model had additional signs in them. It also has a darker tree in the background, with very little blue sky visible compared to the training images.
- Go back and review your training images. Based on problems you saw with your test images, can you identify any potential sources of bias in your training images? For example, maybe you took all your pictures of one type of sign with a blue sky in the background, and all your pictures of another type of sign with a building in the background. Can you figure out how you could add more training images to help eliminate that bias?
- Save your model by clicking the Teachable Machine menu in the top left, then click Save to Google Drive.
- Based on your analysis in Steps 15 and 16, gather more training images. You may need to explore farther to find more signs, or ask an adult to drive you around or take public transportation to new locations so you can find more signs. You can also take pictures at a different time of day for different lighting or wait for the weather to change and take pictures on a different day. You can also use image editing software to cut and paste signs onto different backgrounds.
- Create a new model using the New Project option in the upper left menu. Train it with all of your training images, not just the new ones.
Note: while you can upload more images and re-train an existing model, this will overwrite the model. Creating a new project for each iteration of your model will allow you to go back and compare them, as opposed to continuously overwriting the same model each time you train it.
- Test your new model with the same test images you used on your first model. (Make sure you do not include the test images in your training data). Does your new model do a better job identifying images that the first model had trouble with? What is the result for each individual test image? How do the accuracy and confusion matrix compare to those of your previous model?
- You can now repeat the process of testing your model, taking more training images, training a new model, and testing it again. Try to push the limits of your model and test it with more difficult images.
Think about all the scenarios a self-driving car might encounter where it would be easy for a person to identify the sign, but possibly difficult for the machine learning model—for example, a stop sign partially obscured by a tree branch. If your training data did not include any images of partially obscured signs, can it still identify them in test images? What about damaged or bent signs?
- Keep trying to improve your model. This is an open-ended project with no defined "end point." You can always give your model more training data and keep trying to improve it. However, realistically, at some point you have to decide when your model is "good enough." Can you get your model to a point where you think it would be safe to use in a self-driving car?
If you want more challenges and more ways to expand your model, see the Variations section.
Ask an Expert
Global Connections
The United Nations Sustainable Development Goals (UNSDGs) are a blueprint to achieve a better and more sustainable future for all.
Variations
- You can add more classes to your Teachable Machine model by clicking Add a class. Try training a model that can recognize multiple types of signs.
- Take pictures with no signs in them and include them as a class in your model. Does your model ever mistakenly identify a sign in an image even when none is present?
- Try training a model that can recognize other objects an autonomous car would see when driving, like pedestrians, cyclists, and other cars, or red, yellow, or green lights. Remember to be careful and follow all traffic and pedestrian safety laws (e.g., do not stand in the middle of the road to take pictures). You may need to check whether it is legal to take pictures of other people in public in your location.
- Click the Advanced drop-down under the Train model button to access several different parameters that affect how your model learns. Hover your mouse over the "?" icon next to each parameter to learn more about it. Try adjusting these parameters to see how they influence your model.
Careers
If you like this project, you might enjoy exploring these related careers:
Related Links
- Science Fair Project Guide
- Other Ideas Like This
- Computer Science Project Ideas
- Artificial Intelligence Project Ideas
- Self-Driving Cars Project Ideas
- My Favorites
