Putting the McGurk Effect to the Test

Students interested in Virtual Reality can investigate how easy it is to trick the human brain. With a bit of video making, students can test the McGurk effect with friends and family. Are there limits to the effect? How will these limits influence the creation and success of Virtual Reality environments?

Do you ever doubt your ears? Without you stopping to think about it, your brain uses other available cues (or sensory input) in addition to sound to process what you "hear." When it comes to listening to someone speak, if you are also watching the person talk, your brain can be tricked and tripped up with mis-information! In other words, if you close your eyes and listen to something, you may find that you hear something completely different than you think you hear if you are simultaneously listening and watching someone speak the same words.

If you have ever been watching a television program or music video, and the audio and video tracks were slightly out of sync, you probably noticed the problem. It can be disconcerting to hear sound that doesn't seem to match up to what you are seeing. This is especially true when you are watching characters speak (or a singer sing), and the words you hear don't match what you see happening with the mouth. You may not be able to read lips, but your brain notices that things are out of sync. It can be really disturbing! (This is also the reason why people may quickly notice if an artist lip syncs a performance rather than singing live. Your brain takes in a whole lot of information from what it sees as you listen.)

Generally, our brain does a very good job processing sensory information and making sense of what is happening. It looks like a piece of pie. It smells like a piece of pie. It tastes like a piece of pie. It must be a piece of pie. For the most part, we trust the interpretation our brain offers. But sometimes the brain gets it wrong, and developers of Virtual Reality (VR) experiences are counting on that.

Virtual Reality and Illusion

Virtual Reality developers are creating immersive environments that are designed to make you think you are in a certain place or scenario, an online game world, for example, even though you know you are "really" not there. For VR to work, your brain has to receive sensory information that is convincing enough that you believe what you know isn't true. Successful VR will create an illusion that your brain accepts as real.

Students can explore questions related to human perception and VR with a simple real-world experiment that puts hearing and seeing to the test and demonstrates that our brains are not always perfect at sorting out received information. With a bit of video creation, you can trick your brain into thinking it is hearing something because of what it sees.

Testing the McGurk Effect

The McGurk effect describes what happens when the brain receives conflicting visual and auditory information. With the same audio being played of a sound or word being repeated over and over, the brain can be tricked into hearing the sound or word differently if there is visual information being provided that seems to indicate a different sound or word. Continuing our pie example, if you watch a video of a person saying "pie" over and over and also see the mouth forming the word "pie" over and over, you probably will say you "heard" the word "pie." But if you watch a video that has been altered so that you hear the word "pie" over and over but the mouth forms a slightly different word over and over, what will your brain "hear"? Close your eyes and watch the same video, and you will likely hear the sound or word that is being spoken. But with conflicting video information competing with the audio information, the brain will not always interpret the sound properly even if you know the McGurk effect is being tested!

In the Do You Hear What You See? science project, students experiment with the McGurk effect to see how the effect works and how strong the effect is. Students create their own McGurk effect audio and video and test volunteers to see the effect in action.

This is a fun hands-on science project for students interested in Virtual Reality, illusion, human perception and psychology, or the way the brain interprets and synthesizes information.

Making Connections

Students interested in the McGurk effect may also enjoy the following science projects and resources:

• Apparent Motion & Animation
• Are Your Eyes Playing Tricks on You? Discover the Science Behind Afterimages!
• Battle of the Senses: Taste Versus Smell
• Get the Scoop on Stroop
• Seeing Science: Weekly Science Activity
• Now You See It, Now You Don't: A Chromatic Adaptation Project
• Seeing Is (Not Always) Believing!
• Visual Illusions: When What You See Is... Not What's There?

Related Reading

If you are a Google Classroom teacher, give our Google Classroom Integration a try by assigning a related science news story! We have a complete walk-through to help you get started. The following science news articles are ones you might consider for a fun related reading assignment:

• Hearing: It's Not Just For Your Ears Anymore
• Scientists work their magic on 'shrunken finger illusion'
• An Optical Illusion As Seen By a Fish

You Might Also Enjoy These Related Posts:

• Plastics and Earth Day - Science Projects
• Arduino Science Projects and Physical Computing
• 10+ Robotics Projects with the BlueBot Kit
• 5 STEM Activities with Marshmallow Peeps
• March Madness Basketball Science Projects: Sports Science Experiments
• Women in STEM! More than 60 Scientists and Engineers for Women's History Month
• Explore Artificial Intelligence and Machine Learning with Student AI Projects
• 10 Reasons to Do the Rubber Band Car Engineering Challenge