Testing for Bias in a Photo Lineup

Note: There are special considerations when designing an experiment involving human subjects. ISEF-affiliated fairs often require an Informed Consent Form for every participant who is questioned. In all cases, the experimental design must be approved by the fair's scientific review committee (SRC) prior to the commencement of experiments or surveys. Please refer to the ISEF rules for additional important requirements for studies involving human subjects: https://www.sciencebuddies.org/science-fair-projects/competitions/human-subjects-regulations.

1. Instruct your witnesses that they will have, e.g., 20 seconds to examine a photograph and that they will then have to write a physical description of the person in the photograph.
2. Separately for each witness, show photo of "suspect" and get their written description.
3. Create a composite written description of the "suspect" using features identified in at least two of the written descriptions. This is the written description that your mock witnesses will use to try to identify the "suspect" from a photo lineup.
4. For each photo lineup you want to test, you will need 5 "filler" photos. Try to predict which of your lineups will be unbiased, and which will be biased.
5. Number the photos 1–6, so that your witnesses can select the "suspect" by number.
6. Do the following steps individually with each mock witness:
   1. Have them read the composite written description of the "suspect."
   2. Take the written description away, and then show them the photo lineup.
   3. Ask them to identify the "suspect" (by number) and record the results.
7. For each lineup, calculate the number of times each photo (1–6) was identified. Divide this number by the total number of choices made. If the "fillers" were chosen perfectly, you'd expect that each photo would have an equal probability of being chosen. That is, each photo would be chosen 1/6th of the time. How do your results compare to the ideal?
8. You can also calculate the frequency with which the "suspect" was chosen from the lineup, and calculate the probability that this frequency differs from chance. We'll work an example to show you how:
   1. Write down the number of mock witnesses, n, who viewed the photo lineup: 29.
   2. Write down the number of mock witnesses who chose the "suspect": 14.
   3. Write down the number of photos used in the photo lineup: 6.
   4. Calculate the proportion, p, of mock witnesses who chose the "suspect": 14/29 = 0.483.
   5. Calculate the proportion, q, of mock witnesses who chose "fillers": q = 1 − p = 1− 0.483 = 0.517.
   6. Calculate the standard error, s.e., of  p:
      .
   7. Expected proportion for choosing suspect by chance: 1/n = 1/6 = 0.167.
   8. Critical ratio for difference from chance: (p − chance expectation)/s.e. = 0.483 − 0.167 / 29 = 3.406.
   9. For a 95% confidence interval (i.e., only a 5% chance that the simulated lineup is biased), the critical ratio must be less than 1.96.
   10. For a 99% confidence interval (i.e., only a 1% chance that the simulated lineup is biased), the critical ratio must be less than 2.58.