Page 1 of 2

Survey and the IRB, etc.

Posted: Sun Jun 03, 2007 6:37 pm
by methionine
Hi all,

I wanted to do a short study for fun during the summer. It involves a sensitive topic (depression and anorexia) in Asian adolescent females, and I planned to use statistics I found on data sets along with a survey.
The main problem I have is the survey. I had planned to distribute the survey online, through a blog group (titled "Asians with eating disorders" or something along those lines).

Would that even be a good way to distribute a survey? I know that this online community is very active, and I thought that the relative privacy provided by the way the survey was distributed would be a good thing. However, I guess I'd also have to take into account that the online population probably varies from the actual total population. So either I'd have to change my problem to something even more specific ("Online Eating disorder communities amongst asian american adolescent females"? haha, this is very tentative.) Furthermore, I need to know if that's infringing upon anyone's rights or something... I don't want to do anything illegal!

On a last note, do you think that the topic focuses on too small a group to even be a significant study? I know that Asian Americans constitue only about 4% of the US population. research on that group is generally underrepresented.... but there must be a reason for why it is, right?

thanks so much for your help!
-M

Re: Survey and the IRB, etc.

Posted: Mon Jun 04, 2007 8:12 am
by Louise
methionine wrote:Hi all,

I wanted to do a short study for fun during the summer. It involves a sensitive topic (depression and anorexia) in Asian adolescent females, and I planned to use statistics I found on data sets along with a survey.
The main problem I have is the survey. I had planned to distribute the survey online, through a blog group (titled "Asians with eating disorders" or something along those lines).

Would that even be a good way to distribute a survey? I know that this online community is very active, and I thought that the relative privacy provided by the way the survey was distributed would be a good thing. However, I guess I'd also have to take into account that the online population probably varies from the actual total population. So either I'd have to change my problem to something even more specific ("Online Eating disorder communities amongst asian american adolescent females"? haha, this is very tentative.) Furthermore, I need to know if that's infringing upon anyone's rights or something... I don't want to do anything illegal!

On a last note, do you think that the topic focuses on too small a group to even be a significant study? I know that Asian Americans constitue only about 4% of the US population. research on that group is generally underrepresented.... but there must be a reason for why it is, right?

thanks so much for your help!
-M



I think your concerns are valid. However, I'd like to argue that distribution of surveys online pretty much guarantees useless data. You have no idea if the people returning your survey are actually asian or have eating disorders. Pranksters also tend to enjoy filling out such surveys with misleading answers.

Studies based on self-reporting are always problematic. Self-reporting drawn from an unknown population with unknown motives seems like a recipe for disaster.

I see two routes:
1) find a mentor who does research on eating disorders or who works at an eating disorder clinic. This will require more work, and you undoubtedly will have to do an IRB, but you will get data that is meaningful, and perhaps even useful.

2) go ahead an do a survey on the internet for your own curiosity, but realize that the results will have no scientific validity. You could use this as preliminary data to convince a mentor to work with you, but you could not use these results for any project/paper.

You could also look and see if any large studies of eating disorders collected data on race or ethnicity. It may be that the data is there, but it just wasn't analyzed in this way.

Louise

Posted: Mon Jun 04, 2007 2:31 pm
by tdaly
Let me say this: EVERY SINGLE POLL THAT IS DONE ONLINE IS GIVES YOU USELESS DATA!

Second comment: ANY survey in which people choose to respond is ALWAYS biased.

Now, having ranted a bit (sorry), let me provide some advice. You're topic of interest is very relevant and interesting. The problem that faces you is gathering meaningful data. If it isn't feasible for you to design and conduct a sound survey (probably by working with a mentor in the behavioral sciences) the best thing to do is used existing data. There are many sites that have data on all sorts of differnt things. As Louise said, you can look at existing data and analyze it in a different way to answer YOUR question. If you are interested in using existing data, let me know and I can point you to some reputable sources.

Posted: Mon Jun 04, 2007 2:58 pm
by methionine
Thanks for your responses. :) If you do have any resources to point me to, I'd appreciate that greatly.

But-- the thing is, I wanted to study the ONLINE COMMUNITY itself. ... How else could I possibly know whether or not somebody is involved in the online eating disorder community if not by directly approaching them online? Plus, if I only gave out the surveys to people I knew who was involved in those online communities, that would almost certainly give an inadequate and biased amount of data-- after all, how many anorexic/online people could I know?

Do any of you have any more suggestions? I still am interested in this topic. Thanks for your help!

Posted: Mon Jun 04, 2007 3:05 pm
by methionine
Also, I found this article that stated that online research may be okay in the area of testing eating disorder/body image things:
This investigation compared online and paper administration of self-report measures of weight and shape concerns, mood, weight and height. The former was designed as a screening instrument for adolescent students to determine risk for eating disorders. Participants were female sophomores (N=74) from a San Francisco Bay area private high school. A counterbalanced designed was used to control for order effects. Four classes were randomly assigned to one of four conditions: (1) online-paper; (2) paper-online; (3) online-online; and (4) paper-paper. Participants completed self-report questionnaires twice, online and/or on paper, separated by 1 week. Agreement between online and paper assessment of weight and shape concerns and BMI ranged from 0.73 to 0.97 and 0.80 to 1.0, respectively. Agreement for positive mood ranged from 0.58 to 0.85 and negative mood from 0.59 to 0.82. Mean scores for weight and shape concerns and mood variables were significantly lower at the second testing. Online and paper assessment of weight and shape concerns was significantly correlated. However, there was a significant time effect. Mood was less stable and correlations between the two modes of self-report were less correlated. Online assessment may be beneficial and appropriate for many settings.


...?

Posted: Mon Jun 04, 2007 3:47 pm
by Louise
methionine wrote:Also, I found this article that stated that online research may be okay in the area of testing eating disorder/body image things:
This investigation compared online and paper administration of self-report measures of weight and shape concerns, mood, weight and height. The former was designed as a screening instrument for adolescent students to determine risk for eating disorders. Participants were female sophomores (N=74) from a San Francisco Bay area private high school. A counterbalanced designed was used to control for order effects. Four classes were randomly assigned to one of four conditions: (1) online-paper; (2) paper-online; (3) online-online; and (4) paper-paper. Participants completed self-report questionnaires twice, online and/or on paper, separated by 1 week. Agreement between online and paper assessment of weight and shape concerns and BMI ranged from 0.73 to 0.97 and 0.80 to 1.0, respectively. Agreement for positive mood ranged from 0.58 to 0.85 and negative mood from 0.59 to 0.82. Mean scores for weight and shape concerns and mood variables were significantly lower at the second testing. Online and paper assessment of weight and shape concerns was significantly correlated. However, there was a significant time effect. Mood was less stable and correlations between the two modes of self-report were less correlated. Online assessment may be beneficial and appropriate for many settings.


...?


In this study, the population was known (prep-school girls) and the mechanism of survey was different. (I would personally think this sample was too small to be valid (74 over 4 groups), but I am not a statistician.)

In your study, you have NO IDEA who is answering. Your survey population is completely undefined and unknowable. If you cannot appreciate the difference, then you should not do this project. I know this sounds harsh, but I see these types of surveys all the time online, and I agree completely with Terrik. They are useless. The problem is that people think they have meaning. People in the social sciences spend years learning how to do proper questionaires. You cannot just slap something online and expect it to have meaning.

You should probably try to find a reference on this (or on polling). How you state the question can influence the answers. The order you ask questions can influence results.

Louise

Posted: Mon Jun 04, 2007 6:00 pm
by MelissaB
If you are dead-set on doing an online poll, and if the place you want to do this at is Livejournal, make sure you friends-lock the post. This means that only members of the community can see it and respond--which doesn't negate the concerns that Terik and Louise have brought up, but it may at least mean that random people who are just watching the community for whatever reason can't respond.

I'm sure that other online places have similar options, and I suggest you take advantage of them if you decide to do this.

Re: Survey and the IRB, etc.

Posted: Mon Jun 04, 2007 11:03 pm
by ChrisG
methionine wrote:Hi all,

I wanted to do a short study for fun during the summer....

-M


I agree with the others that there are some problems to consider with an online survey if you want this to be scientifically rigorous. On the other hand, as Louise posed with option #2, if you are doing this just for fun, then I see no problem with using an online survey. It sounds to me like this might be more of an exploratory study where you are just trying to get some basic info, and that does not need to be a hypothesis-driven exercise to be a valuable educational experience.

I have a friend who is an M.D. specializing in eating disorders. She has Asian patients, and she is aware of some of the online activities. Of course, she cannot discuss any personal details of her patients. She might be able to provide some general info. I can ask if she can spare a few minutes to answer questions via email if you are interested.

Re: Survey and the IRB, etc.

Posted: Tue Jun 05, 2007 7:45 am
by Louise
methionine wrote:Hi all,

I wanted to do a short study for fun during the summer....

-M


I agree with the others that there are some problems to consider with an online survey if you want this to be scientifically rigorous. On the other hand, as Louise posed with option #2, if you are doing this just for fun, then I see no problem with using an online survey. It sounds to me like this might be more of an exploratory study where you are just trying to get some basic info, and that does not need to be a hypothesis-driven exercise to be a valuable educational experience.


Right. I just want to emphasize to the original poster that this is not a valid scientific study- a point which I wasn't sure got through, especially after the second post. This can be a useful excercise- to satisfy ones curiosity, to gather preliminary data to convince a mentor to help... Think of this as more of a "conversation" than a "study". You can learn a lot, but you couldn't present this at a science fair.

[Quote from Chris G]
I have a friend who is an M.D. specializing in eating disorders. She has Asian patients, and she is aware of some of the online activities. Of course, she cannot discuss any personal details of her patients. She might be able to provide some general info. I can ask if she can spare a few minutes to answer questions via email if you are interested.
[End Quote From Chris G]

Methionine- I highly recommend that you take Chris up on this generous offer. I'm sure you'd learn a ton from an M.D. who has worked in the field.


Louise

Posted: Tue Jun 05, 2007 2:59 pm
by methionine
Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much! :)

So it's okay if I just gather preliminary data in this way? If I did decide to use it in a future project though... how would that work out? (I might not, but it's always good to think about the possibilities..)

-M

Posted: Tue Jun 05, 2007 4:08 pm
by Louise
methionine wrote:Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much! :)

So it's okay if I just gather preliminary data in this way? If I did decide to use it in a future project though... how would that work out? (I might not, but it's always good to think about the possibilities..)

-M


You cannot use it in a future project. You can use these results to clarify what your hypothesis is for a real study. You can use the data to refine your questions (develop a good survey), but the data collected by this method is not scientifically (or statisitcally) meaningful. Plus, as you point out, you may need some sort of consent/privacy policy/additional paperwork/IRB review if you do a project intended for a science fair or publication.

So, this will help your curiosity and maybe give you some good ideas for how to do a formal study. But as I said before, this is more a "conversation" than a study.

I do "quick and dirty" (not rigorously planned) experiments every day that aren't intended for publication. I would never use these results in a paper, since they were not planned with the appropriate controls. However, the results help me plan proper experiments or assess the feasibility of experiments. This is an important part of science, but you have to realize the limits of the data you collect under these conditions. In your case, there are numerous weaknesses with the methodology proposed (some of which we have discussed above). As long as you realize the limits of what you are doing/ validity of the data you collect, this can be an interesting topic for you to explore. It may lead to a great project.

Louise

Posted: Tue Jun 05, 2007 5:33 pm
by ChrisG
methionine wrote:Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much! :)

OK, I'll check to see if she is available.

Posted: Tue Jun 05, 2007 5:49 pm
by Louise
I have been looking around for some references for you and haven't been able to find much. This page from sciencebuddies has good advice on designing questions for your survey.

http://www.sciencebuddies.org/mentoring/project_ideas/Soc_survey.shtml


I don't know what grade you are in, but this book got good reviews and might be very useful for you:
Real World Research: A Resource for Social Scientists and Practitioner-researchers (Regional Surveys of the World) (Paperback)
by Colin Robson (Author)
ISBN-10: 0631213058
ISBN-13: 978-0631213055


I haven't read it, but looking at the reviews, it seems like it might be very useful to you, though it is a college level text. I would see if you can get it from the library- it is probably too expensive to buy for a single study. If your library doesn't have it, see if they can borrow it for you from another library through inter-library loan.

Another possible choice is:
The Good Research Guide: For Small-scale Social Research Projects (Paperback)
by Martyn Denscombe (Author)
ISBN-10: 0335213030
ISBN-13: 978-0335213030

Again, I have not read it, but it looks helpful.

Louise

Posted: Wed Jun 06, 2007 12:18 pm
by ChrisG
My friend agreed to assist. You should be getting an email from science buddies administrators soon with the contact info. Please let me know if there are any delays or problems.

Help with eating disorders and online surveys

Posted: Wed Jun 06, 2007 2:37 pm
by rpeebles
Dear Methionine:

Chris G referred your question to me, and I am certainly willing to help in any way that I can. I research primarily on the health outcomes of young people with disordered eating and obesity, and I have done a fair amount of work with the Internet as well, researching how it can be a vector to both help and harm those pursuing a healthy weight.

In that line of research, I have researched the health outcomes and quality of life of pro-eating disorder website users, and published my first phase pilot study last winter, and have now submitted another paper based on, yes, an online survey. At the risk of being bashed by all of the people who have been helping you so far, online surveying is indeed an emerging field, and can be done successfully, but requires a lot of thought and care for the reasons expressed above by your other mentors.

In fact, creating a survey of any kind requires a lot of thought to be "valid" and publishable. It is true that you cannot generalize from a website designed to serve Asian people with eating disorders to the general population of Asians with disordered eating, as it is likely that a very specific subset visits that site. And while you cannot verify data, many argue that it is rare that people actually verify a great deal of published data in survey research - for instance, how many times do people who have published surveys on American Indian populations actually check ID, and then verify tribal status before enrolling subjects? Not often. There are ways to crosscheck validity and make it more thorough, and then you must also be honest when writing up findings that this was an online survey, and all that can truly be verified is that the people who answered had visited the website in question, and that is important in itself.

However, to study people via the Internet and publish it, you most certainly have to study them in an official manner, going through a human subjects review board through a major academic center. You can survey people online anytime you choose - it's a free country - but it won't be published without appropriate oversight.

I am happy to discuss this more with you if you are interested. Take care - Rebecka Peebles