Can Humans Recognize AI-Generated Text?

This project follows the Scientific Method. Review the steps before you begin.

Important reminder: taking work written by someone else (whether a human or AI), and trying to pass it off as your own work, is plagiarism. In general, you should not copy and paste entire pages of text from a book or website. However, it is OK to provide short quotes as long as you cite your source. The same applies to AI-generated text from programs like ChatGPT.

1. Plan out your experiment. You will need to decide:
   1. How many human volunteers you need. Try to have at least five.
   2. The general topic for your passages of text, such as "animals." You could choose something slightly more specific, like "dogs."
   3. How many passages of text you will test. 20 is a good start, 10 each for human-generated and AI-generated.
   4. What length you want your passages of text to be. A paragraph is a good place to start, although you could test shorter or longer lengths like sentences or essays. For this experiment, you should keep the length fairly constant (passages do not need to be the exact same number of words or characters). Comparing lengths is also an option for a separate experiment (see Variations section).
   5. How you will have your volunteers read the text (on a computer or printed paper).
   6. What sources you will use for samples of human-generated text.
      1. It is a good idea to stick to books and reputable websites with known human authors. Do not just copy and paste content from random websites, since it could have been written by an AI!
      2. You should try to choose samples at roughly the same level of complexity or reading difficulty. For example, you do not want to have some paragraphs from children's books and some from novels for adults. Different reading levels are another interesting variable you could explore in an experiment (see Variations section).
   7. What prompts you will use to generate text from an AI chatbot of your choice. Make sure the prompts are related to the general topic you picked. For example, if you choose dogs as your topic, you could say "Tell me about golden retrievers in one paragraph at a 10th grade reading level."
2. Create a data table for your experiment. Table 1 shows an example.

3. Find your 10 passages of human-generated text. Put them all into a single document, label each one with a number 1–10, and that it is human-generated. This document will only be visible to you.
4. Choose an AI chatbot like ChatGPT, Gemini, or Copilot. Create an account if needed and experiment with the interface a bit so you understand how it works. Some services like ChatGPT may have their own separate website, and others may be built directly into search engines like Google or Bing. 
5. Generate 10 passages of text with the AI chatbot. You can specify requirements to make them similar to your passages of human-generated text, for example by specifying a length and reading level. Again, paste these passages of text into your document, and label them as "AI-generated" numbers 11–20.
6. Create a separate version of the document for the volunteers to see. Delete the numbers and labels, then randomly move the passages of text around to mix them up. Label them 1–20 in the new order.
7. In your data table, write the re-ordered passage number next to the original passage number. For example, if what was originally the first passage of text is now the fifth in the volunteer version of the document, you would write a 5 in the second column of the first row. This will make it easier to keep track of your results later.
8. Optional: print the document if you prefer to have your volunteers read from paper copies.
9. Have your first volunteer read each passage and tell you whether they think it was written by a human or an AI. Record your results in your data table. Be careful: remember that the volunteer document follows the "re-ordered passage #" column, not the "original passage #" column. Make sure you write each response down in the correct row. Try not to have any reactions or make any facial expressions that would indicate to the volunteer whether they are right or wrong. Make sure you do not give the volunteers any other information (for example, do not tell them that there are 10 of each type).
10. Repeat step 9 for each volunteer.
11. Now, calculate the following percentages and fill in your data table.
    1. For each passage of text, calculate the percentage of volunteers who identified it correctly. For example, if passage 1 was written by a human, you have 10 volunteers, and 6 of them said it was written by a human, the percentage is 6/10 = 60%.
    2. For each volunteer, calculate three percentages:
       1. The percentage of human-generated passages they correctly identified (out of 10).
       2. The percentage of AI-generated passages they correctly identified (out of 10).
       3. The total percentage of passages they correctly identified (out of 20).
12. Analyze your data so far.
    1. Were certain passages of text easier for volunteers to correctly identify than others?
    2. Were certain volunteers better at correctly identifying passages of text than others? If so, were they better at identifying the human-written text, AI-generated text, or both?
13. Calculate an overall average percentage correct for both the human-written and AI-generated passages. Do this by averaging the percentages in the last column for "original passage #" 1–10 (human) and "original passage #" 11–20 (AI).
14. Analyze your final data. Overall, are humans good at identifying text that is written by an AI? Do you have any "false positives," where humans incorrectly identify text that was written by a human as written by an AI? Think back to the questions posed in the Background section. Overall, what benefits do you think AI-generated text could have? What problems do you think it could cause?

Extra: the procedure above was written by a human employee at Science Buddies. For the record, we asked ChatGPT to write the procedure for this project with the following prompt: "Write a step by step procedure for a science project where students ask volunteers to say whether they think paragraphs of text were written by a human or by ChatGPT. The students have access to ChatGPT to generate text and they can pull human-written text from other sources (books, the internet, etc.)." This is what it came up with. What are the differences between this procedure and the one above?

1. Choose a topic: Choose a topic that you would like to create text on, for example, "Animals in the Rainforest."
2. Generate Text using ChatGPT: Use the ChatGPT API to generate a paragraph of text about the chosen topic.
3. Collect Human-written text: Gather several paragraphs of human-written text on the same topic from various sources such as books, the internet, etc.
4. Create a Text Pool: Combine the ChatGPT-generated text and the human-written text into a single pool of text.
5. Choose Volunteers: Ask classmates or friends to volunteer to participate in the experiment. Make sure to have a good mix of people with different levels of reading and writing abilities.
6. Randomize the Text Pool: Randomly choose a paragraph from the pool of text and present it to each volunteer.
7. Ask for Responses: Ask the volunteer to read the paragraph and say whether they think it was written by a human or ChatGPT.
8. Record Responses: Record the volunteer's response for each paragraph in a spreadsheet or on paper.
9. Repeat: Repeat steps 6 through 8 for each volunteer and for multiple paragraphs of text.
10. Analyze Results: Once you have recorded the responses from all the volunteers, analyze the results to see how accurately they were able to determine whether the text was written by a human or by ChatGPT.
11. Report Results: Write a report of your findings, including any trends or patterns you observed and any conclusions you can draw from your experiment.