Testing for Bias in a Photo Lineup (How to Analyze Results?)

[DBZfan29]

Hello. I am currently working on my eighth grade science fair project. It is due next Monday. I have completed the experiment and am now trying to analyze the results. My experiment is testing bias in a photo lineup, an idea I found on the site. Anyway, here is what the page says to analyze the data, with an example:

You can also calculate the frequency with which the "suspect" was chosen from the lineup, and calculate the probability that this frequency differs from chance. We'll work an example to show you how:

1. Write down the number of mock witnesses, n, who viewed the photo lineup: 29.
2. Write down the number of mock witnesses who chose the "suspect": 14.
3. Write down the number of photos used in the photo lineup: 6.
4. Calculate the proportion, p, of mock witnesses who chose the "suspect": 14/29 = 0.483.
5. Calculate the proportion, q, of mock witnesses who chose "fillers": q = 1 − p = 1− 0.483 = 0.517.
6. Calculate the standard error, s.e., of p: SQUARE ROOT OF P X Q DIVIDED BY N.
7. Expected proportion for choosing suspect by chance: 1/n = 1/6 = 0.167.
8. Critical ratio for difference from chance: (p − chance expectation)/s.e. = 0.483 − 0.167 / 29 = 3.406.
9. For a 95% confidence interval (i.e., only a 5% chance that the simulated lineup is biased), the critical ratio must be less than 1.96.
10. For a 99% confidence interval (i.e., only a 1% chance that the simulated lineup is biased), the critical ratio must be less than 2.58.

I have completed most of the steps. There were 21 mock witnesses, 16 of them picked the suspect, 6 photos were used in the lineup, the two proportions are 0.76 and 0.24 (I rounded to the hundredths), and for the standard error, I got 0.1. Steps 8 through 9 are throwing me off. How do I find the critical ratio? How do I find the confidence interval? Thanks in advance for any help!

[amyc]

I am sorry that your post did not receive a response from an Expert here in the Ask an Expert Forums in time for your project deadline. I hope you were able to sort out steps 8 and 9 and that the project turned out well.

Amy
Science Buddies

[rgoelmsft]

Hi there --

I'm also sorry you didn't receive a response sooner. But let me try and help anyway.

For step 6 (the standard error), I'm getting 0.093. Is that what you got? You said you got 0.1, which is close, but you probably need to keep some extra significant digits on this type of calculation.

For step 7 (expected proportion for choosing suspect by chance), I get 0.167.

For step 8 (critical ratio), I get:
(0.76 - 0.167) / 0.093 = 6.40
I can see why you got confused on Step 8 ... the example on the Science Buddies page was done incorrectly. The formula is:
(p - chance expectation) / standard error.
But when they actually did the example, they showed (.483 - .167) / 29, which is wrong. They should have shown the example as (.483 - .167) / .093. But in any case, they got the right answer --> 3.406.

For steps 9 and 10, since your critical ratio is so high (6.40), that means you have higher than 99% confidence that your lineup is biased. This makes intuitive sense since 16 out of 21 of your mock witnesses chose the suspect. So, what this says is that the lineup you chose would definitely NOT be a fair lineup, because mock witnesses overwhelmingly chose your mock suspect. If the lineup was fair, you would expect the mock witnesses to choose the mock suspect only about 1/6 (16.7%) of the time. Remember the goal of a lineup is to make sure that a mock witness (someone who wasn't actually at the scene of the crime) is NOT able to figure out who the criminal is. Only the real witness (the person who was actually present at the scene of the crime) would be able to identify the criminal because he/she actually saw him.

I hope this helps. I must say that the way this project is presented is very confusing. You may wish to read this article which has more detail, and is more correct:

http://eyewitness.utep.edu/Documents/DIY%20Kit.pdf

--Rajeev

[pfarley]

Hi: I’m having trouble analyzing my results.
Number of mock witnesses= 27
Number of mock witnesses who chose suspect =8
Total number of persons in lineup= 6

1) Proportion identifying the suspect (p) .296
2) Q= 1-p .704
3) N= 27
4) Standard Error of P= .4563
5) Chance Expectation CE= .167
6) Critical Ratio for diff from Chance: .2827


How do I interpret item 6) since it doesn’t match the given
(.05 requires 1.96) or (.01 requires 2.58)

Do I interpret item 6) as Confidence intervals?
1% confidence interval equals 0.243 to 0.722

thanks in advance

[rgoelmsft]

I think maybe you have made an error in your calculation. For #4, "standard error of P", you should have gotten 0.0879. It is calculated by sqrt( P x Q / N ).

Now, in #6, the critical ratio becomes 1.468

Now, since you're critical ratio does NOT exceed 1.96, what this tells you is that your lineup is probably pretty fair. That is, there is not a huge amount of bias toward the suspect. If you did have a large amount of bias in your lineup, then you're mock witnesses (who were supposedly not at the scene of the crime, and therefore would have no way of knowing who the criminal really is) would overwhelming have chosen the suspect merely because of the way he looks. This would be unfair. But in your case, only 8 out of 27 mock witnesses chose the suspect. That is pretty close to the probability of them choosing him purely by chance. This means that you've designed your lineup fairly, in a way that a random person (someone not at the scene of the crime) would not be able to identify the criminal.