Scientific Method or EDP? Machine Learning

[AmyCowen]

Hi UNDEFINEDNULL -

I am glad to hear you have moved forward with your project. You are right that many projects involve three trials in order to ensure the results are repeatable. I talked with one of our staff scientists about your questions about three trials specifically in the context of a computer science or machine learning project. Here is some feedback for you.

"The advice to repeat the experiment at least three times means that you should typically do at least three trials for each value of the independent variable in your experiment. This is usually easier to think about in terms of physical experiments. For example, if you are measuring how far a ball will go when launched at different angles, you should measure the distance for each angle at least three times, because it is impossible to launch a ball at exactly the same speed and angle each time. As a result, there will always be some error or variation in the distance.

This can be a little different in some computer science projects. For example, typically if you run the exact same computer program three times without changing any of the inputs, it will produce the exact same output. It will not have any variation or randomness like you would get with something physical like throwing a ball, so running the program three times is not really doing three 'trials.'

For a machine learning project, you could conduct multiple trials by re-training the model using different training data each time. For example, in a previous post you suggested that 'group 1' would be a small sample (15 photos of each sign type). Your first trial for group 1 would be with one set of 15 photos for each sign. Then you would re-train the model with another set of 15 new photos for each sign, etc., and repeat the experiment. This will require a lot of images, but if you are using a phone/digital camera, you can take a lot of pictures of the same sign from different angles, and at different times of day or in different weather conditions -- you don't need to walk/drive around and physically find hundreds or thousands of different signs to take pictures of."

I hope this helps.

Amy
Science Buddies

[UNDEFINEDNULL]

Hi Amy, and thank you for your prompt reply. However, I will unfortunately be unable to implement this solution. It took me quite a long time(even with using internet photos and different angles) to capture my photos, and I will be unable to amass enough images before the deadline. I heard Teachable Machine's results are slightly different every time due to using only 85 percent of images each time(and using the other 15 as a validation dataset). Would that work, since Teachable Machine would not give the exact SAME results every time? Thank you very much!

Sincerely,
A Stressed UNDEFINEDNULL

[AmyCowen]

Hopefully you'll be able to tell based on the trials you run. We've seen reference to the same "it uses 85%" of the images that you mention, but we don't know if the 85% used are randomly selected each time. If they are, then doing the three trials serves the right purpose.

I don't see the information about the 85% come up in Teachable Machine's FAQ or support information. Make sure that you include what you saw in your writeup of the project as well as in any discussion -- especially if you don't know for sure "how" the 85% are selected when it runs.

Hope all goes well with your experiment!

Amy