Oh, the temptation to "fix" the data

[deleted-3443]

My daughter (5th grade) is doing a science fair project on sizing paper. One of the tests that she wanted to do was compare the texture of paper with sizing added to paper with no sizing. Since we couldn't figure out a way to measure this quantitatively, we figured we would do it qualitatively (by touching it, maybe drawing on it, then comparing).

Her big sister is doing the "Take a musical step back in time" project from this site, and she developed a survey form to have listeners rate the sound from different home made record players. Little sister copied this idea and made a survey form to have users test the paper. She decided to just have them touch the paper, rather than write on it, so she could use the same paper for all the volunteers. (Each piece of paper is about 20 minutes of work, so 6 samples times a survey size of 10 volunteers would be 20 hours of preparation time to make paper!)

Here's the problem: Little sister just didn't have the best survey design skills. Her first two volunteers (mom and dad) gently tried to tell her that her test was really hard to do, but she didn't really want to hear it. The form was really hard to fill out--keeping track of samples numbered one to six, and scores from one to five and put the right number by the right number. And, rating texture of paper is not something that comes natural to people who (unlike dear daughter) are not papermaking fanatics. From her point of view, the "fuzzy" paper has "bad texture" -- the fibers are falling out, loose fibers will scratch off as you try to write on it. From some users point of view (despite instructions to the contrary), the "fuzzy" paper feels like a soft cotton sweater, fun to the fingers, and thus has a "good texture."

Now that the data is in, we can see that the fuzziest paper (with no sizing added to keep the fibers from flying away) has been rated "best" by about a third of the users (contrary to instructions), and "worst" by the rest. It's clear that they filled in the survey scores backwards to what she was trying to ask.

I think the only solution is to either throw out this part of the experiment -- either redesign the survey, make it easier for volunteers, and re-do it, or to throw out the survey just use the quantitative data that she already has (she did strength tests and absorbency tests as well as another qualitative experiment where she folded origami with the different papers) and be done. It's a breach of scientific ethics to just "reverse the scores" on the bad surveys, even if we are sure that the data would make the most sense that way. This survey can't be saved? If we throw it out, do we put any mention of it on the science fair board?

[rmarz]

janet_425 - I think you have done an excellent job of answering all your questions (including your topic headline). The reason for using the 'scientific method' is to remove subjective bias from ones personal hypothesis or expectations and let the 'data' support the conclusion.

You correctly point out that perhaps the process was lacking (questions, instructions to the testers might have been flawed). I might suggest that to better get input on the tactile 'feel' or impressions of the surface qualities of the paper you have to include the perspective of the application, or what the paper is to be used for. If it is to be a writing surface for ink or even pencil, the qualities of smoothness to the touch might give you a positive response. If the use is to be of a protective covering for a camera lens or filter, maybe a fuzzy or furry touch might evoke a different response. Toilet tissue or facial tissue certainly are different than parchment or vellum, but each is uniquely designed for their intended uses. Just some ideas, the right answer is not to 'fix' the data to support your desired outcome. Run a new evaluation but give some guidelines as to the intended use. Your testers will likely give you highly correlated responses.

Rick Marz

[deleted-3443]

Thanks for the reply. My daughter is just "done" with experimentation, though. We're just going to write something saying that the results of her survey were very inconsistent and the results for this part of the experiment were inconclusive. She has lots of other things to talk about on her science fair board anyway.

I need to be firmer about getting her to cut back on her ambitions at the beginning of her science fair projects. She's got so much to write up before the fair, and she got a stomach flu last week and got very little done.