Jump to main content

Can Humans Recognize ChatGPT's AI-Generated Text?

1
2
3
4
5
180 reviews

Abstract

AI (artificial intelligence)-generated text is a hot topic for many reasons. Computers can now generate convincing paragraphs or even pages of text that look like they were written by a human. How do you know if a news article you are reading was written by a human or an AI? How does a teacher know if a student's essay was written by an AI? How do you know this text was not written by an AI? In this science project, you will conduct an experiment to see if volunteers can correctly identify whether different passages of text were written by a human or AI.

Summary

Areas of Science
Difficulty
 
Time Required
Short (2-5 days)
Prerequisites
None
Material Availability
Readily available
Cost
Very Low (under $20)
Safety
No issues
Credits
Ben Finio, PhD, Science Buddies

Objective

Find out whether volunteers can identify which paragraphs of text were written by humans and which ones were generated by AI.

Introduction

Chatbots, or computer programs that can have a back-and-forth text conversation mimicking a human conversation, have been around for a long time. They have gotten better and better with the introduction of machine learning, which allows the programs to "learn" from massive amounts of training data, or text written by real humans. Machine learning programs are typically called models. Scientists "train" the model to improve its performance using the training data. ChatGPT is one such model that received a lot of public attention starting in late 2022.

ChatGPT is a computer program that is really good at understanding and answering questions. It was taught by a company called OpenAI using a special technique called "deep learning." This means that the program learned from a huge amount of information about how people talk and write in different ways. Now, when you type a question or a message, ChatGPT uses that information to try and come up with the best answer it can. It's like having a very smart friend who knows a lot about many different things and can talk to you about them!

Did you think that previous paragraph was written by a human? It was actually written by ChatGPT! That was its answer in response to the prompt "Write a paragraph about how ChatGPT works that is appropriate for a 5th grader to understand." Figure 1 shows a screenshot of the conversation.

screenshot of a chatGPT conversation

Prompt: Write a paragraph about how ChatGPT works that is appropriate for a 5th grader to understand. Response: ChatGPT is a computer program that is really good at understanding and answering questions. It was taught by a company called OpenAI using a special technique called 'deep learning.' This means that the program learned from a huge amount of information about how people talk and write in different ways. Now, when you type a question or a message, ChatGPT uses that information to try and come up with the best answer it can. It's like having a very smart friend who knows a lot about many different things and can talk to you about them!


Figure 1. Screenshot of a ChatGPT conversation.

ChatGPT can do many things—for example, offering the same explanation at a different reading level. Here is its response to "Write a paragraph about how ChatGPT works that is appropriate for a 1st grade to understand.":

ChatGPT is a helper that can talk to you. When you ask it a question, it tries its best to find the answer and tell it to you. It's like having a talking dictionary that can answer your questions!

You see how, in both cases, ChatGPT can produce text that makes sense and is free from grammatical or punctuation errors. This can make it difficult for a human reader to distinguish text written by a human from text written by ChatGPT. Can people tell the difference at all? Are some passages of text easier to spot than others? In this science project, you will conduct an experiment to find out. You will collect passages of text from different human-generated sources, and generate text on similar topics using ChatGPT. You will then present each passage of text to volunteers and ask them to mark whether they think the text was written by a human or AI. How good to you think the volunteers will be at this task?

Finally, note that ChatGPT is different from search engines like Google. When search engines first started, they would only give you a list of websites in response to a search. Now, they can use natural language processing to give direct answers to questions like "What is the diameter of the moon?" As of early 2023, ChatGPT cannot access the internet, so it can only provide responses based on its training data. So, for example, it cannot answer the question "What is today's weather near me?" Other virtual assistants that are connected to the internet (like Apple's Siri and Amazon's Alexa) are capable of answering such questions.

Terms and Concepts

Questions

Bibliography

Read the information on this page to learn more about how ChatGPT works:

Materials and Equipment

Experimental Procedure

Working with Human Test Subjects

There are special considerations when designing an experiment involving human subjects. Fairs affiliated with Regeneron International Science and Engineering Fair (ISEF) often require an Informed Consent Form (permission sheet) for every participant who is questioned. Consult the rules and regulations of the science fair that you are entering, prior to performing experiments or surveys. Please refer to the Science Buddies documents Projects Involving Human Subjects and Scientific Review Committee for additional important requirements. If you are working with minors, you must get advance permission from the children's parents or guardians (and teachers if you are performing the test while they are in school) to make sure that it is all right for the children to participate in the science fair project. Here are suggested guidelines for obtaining permission for working with minors:

  1. Write a clear description of your science fair project, what you are studying, and what you hope to learn. Include how the child will be tested. Include a paragraph where you get a parent's or guardian's and/or teacher's signature.
  2. Print out as many copies as you need for each child you will be surveying.
  3. Pass out the permission sheet to the children or to the teachers of the children to give to the parents. You must have permission for all the children in order to be able to use them as test subjects.

Important reminder: taking work written by someone else (whether a human or AI), and trying to pass it off as your own work, is plagiarism. In general, you should not copy and paste entire pages of text from a book or website. However, it is OK to provide short quotes as long as you cite your source. The same applies to AI-generated text from programs like ChatGPT.

  1. Plan out your experiment. You will need to decide:
    1. How many human volunteers you need. Try to have at least five.
    2. The general topic for your passages of text, such as "animals." You could choose something slightly more specific, like "dogs."
    3. How many passages of text you will test. 20 is a good start, 10 each for human-generated and AI-generated.
    4. What length you want your passages of text to be. A paragraph is a good place to start, although you could test shorter or longer lengths like sentences or essays. For this experiment, you should keep the length fairly constant (passages do not need to be the exact same number of words or characters). Comparing lengths is also an option for a separate experiment (see Variations section).
    5. How you will have your volunteers read the text (on a computer or printed paper).
    6. What sources you will use for samples of human-generated text.
      1. It is a good idea to stick to books and reputable websites with known human authors. Do not just copy and paste content from random websites, since it could have been written by an AI!
      2. You should try to choose samples at roughly the same level of complexity or reading difficulty. For example, you do not want to have some paragraphs from children's books and some from novels for adults. Different reading levels are another interesting variable you could explore in an experiment (see Variations section).
    7. What prompts you will use to generate text from ChatGPT. Make sure the prompts are related to the general topic you picked. For example, if you choose dogs as your topic, you could say "Tell me about golden retrievers in one paragraph at a 10th grade reading level."
  2. Create a data table for your experiment. Table 1 shows an example.
Original passage # Re-ordered passage # Human or AI? Volunteer 1 Volunteer 2  ...  % correct for each passage
1   Human    
2   Human    
...   ...    
10   Human    
% human correct for each volunteer       
11   AI    
12   AI    
...   ...    
20   AI    
% AI correct for each volunteer       
Total % correct for each volunteer       
Table 1. Example data table. Add columns for as many volunteers as you have.
  1. Find your 10 passages of human-generated text. Put them all into a single document, label each one with a number 1–10, and that it is human-generated. This document will only be visible to you.
  2. Go to the ChatGPT website and create an account. Experiment with ChatGPT for a bit to get a sense of its abilities so you understand how it works.
  3. Generate 10 passages of text with ChatGPT. You can specify requirements to make them similar to your passages of human-generated text, for example by specifying a length and reading level. Again, paste these passages of text into your document, and label them as "AI-generated" numbers 11–20.
  4. Create a separate version of the document for the volunteers to see. Delete the numbers and labels, then randomly move the passages of text around to mix them up. Label them 1–20 in the new order.
  5. In your data table, write the re-ordered passage number next to the original passage number. For example, if what was originally the first passage of text is now the fifth in the volunteer version of the document, you would write a 5 in the second column of the first row. This will make it easier to keep track of your results later.
  6. Optional: print the document if you prefer to have your volunteers read from paper copies.
  7. Have your first volunteer read each passage and tell you whether they think it was written by a human or an AI. Record your results in your data table. Be careful: remember that the volunteer document follows the "re-ordered passage #" column, not the "original passage #" column. Make sure you write each response down in the correct row. Try not to have any reactions or make any facial expressions that would indicate to the volunteer whether they are right or wrong. Make sure you do not give the volunteers any other information (for example, do not tell them that there are 10 of each type).
  8. Repeat step 9 for each volunteer.
  9. Now, calculate the following percentages and fill in your data table.
    1. For each passage of text, calculate the percentage of volunteers who identified it correctly. For example, if passage 1 was written by a human, you have 10 volunteers, and 6 of them said it was written by a human, the percentage is 6/10 = 60%.
    2. For each volunteer, calculate three percentages:
      1. The percentage of human-generated passages they correctly identified (out of 10).
      2. The percentage of AI-generated passages they correctly identified (out of 10).
      3. The total percentage of passages they correctly identified (out of 20).
  10. Analyze your data so far.
    1. Were certain passages of text easier for volunteers to correctly identify than others?
    2. Were certain volunteers better at correctly identifying passages of text than others? If so, were they better at identifying the human-written text, AI-generated text, or both?
  11. Calculate an overall average percentage correct for both the human-written and AI-generated passages. Do this by averaging the percentages in the last column for "original passage #" 1–10 (human) and "original passage #" 11–20 (AI).
  12. Analyze your final data. Overall, are humans good at identifying text that is written by an AI? Do you have any "false positives," where humans incorrectly identify text that was written by a human as written by an AI? Think back to the questions posed in the Background section. Overall, what benefits do you think AI-generated text could have? What problems do you think it could cause?

Extra: the procedure above was written by a human employee at Science Buddies. For the record, we asked ChatGPT to write the procedure for this project with the following prompt: "Write a step by step procedure for a science project where students ask volunteers to say whether they think paragraphs of text were written by a human or by ChatGPT. The students have access to ChatGPT to generate text and they can pull human-written text from other sources (books, the internet, etc.)." This is what it came up with. What are the differences between this procedure and the one above?

  1. Choose a topic: Choose a topic that you would like to create text on, for example, "Animals in the Rainforest."
  2. Generate Text using ChatGPT: Use the ChatGPT API to generate a paragraph of text about the chosen topic.
  3. Collect Human-written text: Gather several paragraphs of human-written text on the same topic from various sources such as books, the internet, etc.
  4. Create a Text Pool: Combine the ChatGPT-generated text and the human-written text into a single pool of text.
  5. Choose Volunteers: Ask classmates or friends to volunteer to participate in the experiment. Make sure to have a good mix of people with different levels of reading and writing abilities.
  6. Randomize the Text Pool: Randomly choose a paragraph from the pool of text and present it to each volunteer.
  7. Ask for Responses: Ask the volunteer to read the paragraph and say whether they think it was written by a human or ChatGPT.
  8. Record Responses: Record the volunteer's response for each paragraph in a spreadsheet or on paper.
  9. Repeat: Repeat steps 6 through 8 for each volunteer and for multiple paragraphs of text.
  10. Analyze Results: Once you have recorded the responses from all the volunteers, analyze the results to see how accurately they were able to determine whether the text was written by a human or by ChatGPT.
  11. Report Results: Write a report of your findings, including any trends or patterns you observed and any conclusions you can draw from your experiment.
icon scientific method

Ask an Expert

Do you have specific questions about your science project? Our team of volunteer scientists can help. Our Experts won't do the work for you, but they will make suggestions, offer guidance, and help you troubleshoot.

Variations

  • Re-run the experiment with passages of text of different lengths (for example, sentences and essays). Does the length of the passage affect whether humans can correctly identify it as human or AI-generated?
  • Re-run the experiment with passages of text at different reading levels. Does the reading level affect whether humans can correctly identify text as human or AI-generated?
  • There are other websites online that can generate text using AI. Can you run an experiment comparing different websites or programs? Are some better than others at generating text that looks like it was written by a human?

Careers

If you like this project, you might enjoy exploring these related careers:

Career Profile
Are you interested in developing cool video game software for computers? Would you like to learn how to make software run faster and more reliably on different kinds of computers and operating systems? Do you like to apply your computer science skills to solve problems? If so, then you might be interested in the career of a computer software engineer. Read more
Career Profile
Many aspects of peoples' daily lives can be summarized using data, from what is the most popular new video game to where people like to go for a summer vacation. Data scientists (sometimes called data analysts) are experts at organizing and analyzing large sets of data (often called "big data"). By doing this, data scientists make conclusions that help other people or companies. For example, data scientists could help a video game company make a more profitable video game based on players'… Read more
Career Profile
Mathematicians are part of an ancient tradition of searching for patterns, conjecturing, and figuring out truths based on rigorous deduction. Some mathematicians focus on purely theoretical problems, with no obvious or immediate applications, except to advance our understanding of mathematics, while others focus on applied mathematics, where they try to solve problems in economics, business, science, physics, or engineering. Read more
Career Profile
Statisticians use the power of math and probability theory to answer questions that affect the lives of millions of people. They tell educators which teaching method works best, tell policy-makers what levels of pesticides are acceptable in fresh fruit, tell doctors which treatment works best, and tell builders which type of paint is the most durable. They are employed in virtually every type of industry imaginable, from engineering, manufacturing, and medicine to animal science, food… Read more

News Feed on This Topic

 
, ,

Cite This Page

General citation information is provided here. Be sure to check the formatting, including capitalization, for the method you are using and update your citation, as needed.

MLA Style

Finio, Ben. "Can Humans Recognize ChatGPT's AI-Generated Text?" Science Buddies, 27 Mar. 2024, https://www.sciencebuddies.org/science-fair-projects/project-ideas/ArtificialIntelligence_p005/artificial-intelligence/ChatGPT-AI-generated-text. Accessed 14 Apr. 2024.

APA Style

Finio, B. (2024, March 27). Can Humans Recognize ChatGPT's AI-Generated Text? Retrieved from https://www.sciencebuddies.org/science-fair-projects/project-ideas/ArtificialIntelligence_p005/artificial-intelligence/ChatGPT-AI-generated-text


Last edit date: 2024-03-27
Top
Free science fair projects.