Survey and the IRB, etc.

[methionine]

Hi all,

I wanted to do a short study for fun during the summer. It involves a sensitive topic (depression and anorexia) in Asian adolescent females, and I planned to use statistics I found on data sets along with a survey.
The main problem I have is the survey. I had planned to distribute the survey online, through a blog group (titled "Asians with eating disorders" or something along those lines).

Would that even be a good way to distribute a survey? I know that this online community is very active, and I thought that the relative privacy provided by the way the survey was distributed would be a good thing. However, I guess I'd also have to take into account that the online population probably varies from the actual total population. So either I'd have to change my problem to something even more specific ("Online Eating disorder communities amongst asian american adolescent females"? haha, this is very tentative.) Furthermore, I need to know if that's infringing upon anyone's rights or something... I don't want to do anything illegal!

On a last note, do you think that the topic focuses on too small a group to even be a significant study? I know that Asian Americans constitue only about 4% of the US population. research on that group is generally underrepresented.... but there must be a reason for why it is, right?

thanks so much for your help!
-M

[Louise]

Hi all,

I wanted to do a short study for fun during the summer. It involves a sensitive topic (depression and anorexia) in Asian adolescent females, and I planned to use statistics I found on data sets along with a survey.
The main problem I have is the survey. I had planned to distribute the survey online, through a blog group (titled "Asians with eating disorders" or something along those lines).

Would that even be a good way to distribute a survey? I know that this online community is very active, and I thought that the relative privacy provided by the way the survey was distributed would be a good thing. However, I guess I'd also have to take into account that the online population probably varies from the actual total population. So either I'd have to change my problem to something even more specific ("Online Eating disorder communities amongst asian american adolescent females"? haha, this is very tentative.) Furthermore, I need to know if that's infringing upon anyone's rights or something... I don't want to do anything illegal!

On a last note, do you think that the topic focuses on too small a group to even be a significant study? I know that Asian Americans constitue only about 4% of the US population. research on that group is generally underrepresented.... but there must be a reason for why it is, right?

thanks so much for your help!
-M

I think your concerns are valid. However, I'd like to argue that distribution of surveys online pretty much guarantees useless data. You have no idea if the people returning your survey are actually asian or have eating disorders. Pranksters also tend to enjoy filling out such surveys with misleading answers.

Studies based on self-reporting are always problematic. Self-reporting drawn from an unknown population with unknown motives seems like a recipe for disaster.

I see two routes:
1) find a mentor who does research on eating disorders or who works at an eating disorder clinic. This will require more work, and you undoubtedly will have to do an IRB, but you will get data that is meaningful, and perhaps even useful.

2) go ahead an do a survey on the internet for your own curiosity, but realize that the results will have no scientific validity. You could use this as preliminary data to convince a mentor to work with you, but you could not use these results for any project/paper.

You could also look and see if any large studies of eating disorders collected data on race or ethnicity. It may be that the data is there, but it just wasn't analyzed in this way.

Louise

[deleted-2131]

Let me say this: EVERY SINGLE POLL THAT IS DONE ONLINE IS GIVES YOU USELESS DATA!

Second comment: ANY survey in which people choose to respond is ALWAYS biased.

Now, having ranted a bit (sorry), let me provide some advice. You're topic of interest is very relevant and interesting. The problem that faces you is gathering meaningful data. If it isn't feasible for you to design and conduct a sound survey (probably by working with a mentor in the behavioral sciences) the best thing to do is used existing data. There are many sites that have data on all sorts of differnt things. As Louise said, you can look at existing data and analyze it in a different way to answer YOUR question. If you are interested in using existing data, let me know and I can point you to some reputable sources.

[methionine]

Thanks for your responses. If you do have any resources to point me to, I'd appreciate that greatly.

But-- the thing is, I wanted to study the ONLINE COMMUNITY itself. ... How else could I possibly know whether or not somebody is involved in the online eating disorder community if not by directly approaching them online? Plus, if I only gave out the surveys to people I knew who was involved in those online communities, that would almost certainly give an inadequate and biased amount of data-- after all, how many anorexic/online people could I know?

Do any of you have any more suggestions? I still am interested in this topic. Thanks for your help!

[methionine]

Also, I found this article that stated that online research may be okay in the area of testing eating disorder/body image things:

This investigation compared online and paper administration of self-report measures of weight and shape concerns, mood, weight and height. The former was designed as a screening instrument for adolescent students to determine risk for eating disorders. Participants were female sophomores (N=74) from a San Francisco Bay area private high school. A counterbalanced designed was used to control for order effects. Four classes were randomly assigned to one of four conditions: (1) online-paper; (2) paper-online; (3) online-online; and (4) paper-paper. Participants completed self-report questionnaires twice, online and/or on paper, separated by 1 week. Agreement between online and paper assessment of weight and shape concerns and BMI ranged from 0.73 to 0.97 and 0.80 to 1.0, respectively. Agreement for positive mood ranged from 0.58 to 0.85 and negative mood from 0.59 to 0.82. Mean scores for weight and shape concerns and mood variables were significantly lower at the second testing. Online and paper assessment of weight and shape concerns was significantly correlated. However, there was a significant time effect. Mood was less stable and correlations between the two modes of self-report were less correlated. Online assessment may be beneficial and appropriate for many settings.

...?

[Louise]

Also, I found this article that stated that online research may be okay in the area of testing eating disorder/body image things:

This investigation compared online and paper administration of self-report measures of weight and shape concerns, mood, weight and height. The former was designed as a screening instrument for adolescent students to determine risk for eating disorders. Participants were female sophomores (N=74) from a San Francisco Bay area private high school. A counterbalanced designed was used to control for order effects. Four classes were randomly assigned to one of four conditions: (1) online-paper; (2) paper-online; (3) online-online; and (4) paper-paper. Participants completed self-report questionnaires twice, online and/or on paper, separated by 1 week. Agreement between online and paper assessment of weight and shape concerns and BMI ranged from 0.73 to 0.97 and 0.80 to 1.0, respectively. Agreement for positive mood ranged from 0.58 to 0.85 and negative mood from 0.59 to 0.82. Mean scores for weight and shape concerns and mood variables were significantly lower at the second testing. Online and paper assessment of weight and shape concerns was significantly correlated. However, there was a significant time effect. Mood was less stable and correlations between the two modes of self-report were less correlated. Online assessment may be beneficial and appropriate for many settings.

...?

In this study, the population was known (prep-school girls) and the mechanism of survey was different. (I would personally think this sample was too small to be valid (74 over 4 groups), but I am not a statistician.)

In your study, you have NO IDEA who is answering. Your survey population is completely undefined and unknowable. If you cannot appreciate the difference, then you should not do this project. I know this sounds harsh, but I see these types of surveys all the time online, and I agree completely with Terrik. They are useless. The problem is that people think they have meaning. People in the social sciences spend years learning how to do proper questionaires. You cannot just slap something online and expect it to have meaning.

You should probably try to find a reference on this (or on polling). How you state the question can influence the answers. The order you ask questions can influence results.

Louise

[MelissaB]

If you are dead-set on doing an online poll, and if the place you want to do this at is Livejournal, make sure you friends-lock the post. This means that only members of the community can see it and respond--which doesn't negate the concerns that Terik and Louise have brought up, but it may at least mean that random people who are just watching the community for whatever reason can't respond.

I'm sure that other online places have similar options, and I suggest you take advantage of them if you decide to do this.

[deleted-71447]

Hi all,

I wanted to do a short study for fun during the summer....

-M

I agree with the others that there are some problems to consider with an online survey if you want this to be scientifically rigorous. On the other hand, as Louise posed with option #2, if you are doing this just for fun, then I see no problem with using an online survey. It sounds to me like this might be more of an exploratory study where you are just trying to get some basic info, and that does not need to be a hypothesis-driven exercise to be a valuable educational experience.

I have a friend who is an M.D. specializing in eating disorders. She has Asian patients, and she is aware of some of the online activities. Of course, she cannot discuss any personal details of her patients. She might be able to provide some general info. I can ask if she can spare a few minutes to answer questions via email if you are interested.

[Louise]

Hi all,

I wanted to do a short study for fun during the summer....

-M

I agree with the others that there are some problems to consider with an online survey if you want this to be scientifically rigorous. On the other hand, as Louise posed with option #2, if you are doing this just for fun, then I see no problem with using an online survey. It sounds to me like this might be more of an exploratory study where you are just trying to get some basic info, and that does not need to be a hypothesis-driven exercise to be a valuable educational experience.


Right. I just want to emphasize to the original poster that this is not a valid scientific study- a point which I wasn't sure got through, especially after the second post. This can be a useful excercise- to satisfy ones curiosity, to gather preliminary data to convince a mentor to help... Think of this as more of a "conversation" than a "study". You can learn a lot, but you couldn't present this at a science fair.

I have a friend who is an M.D. specializing in eating disorders. She has Asian patients, and she is aware of some of the online activities. Of course, she cannot discuss any personal details of her patients. She might be able to provide some general info. I can ask if she can spare a few minutes to answer questions via email if you are interested.
[End Quote From Chris G]

Methionine- I highly recommend that you take Chris up on this generous offer. I'm sure you'd learn a ton from an M.D. who has worked in the field.


Louise

[methionine]

Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much!

So it's okay if I just gather preliminary data in this way? If I did decide to use it in a future project though... how would that work out? (I might not, but it's always good to think about the possibilities..)

-M

[Louise]

Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much!

So it's okay if I just gather preliminary data in this way? If I did decide to use it in a future project though... how would that work out? (I might not, but it's always good to think about the possibilities..)

-M

You cannot use it in a future project. You can use these results to clarify what your hypothesis is for a real study. You can use the data to refine your questions (develop a good survey), but the data collected by this method is not scientifically (or statisitcally) meaningful. Plus, as you point out, you may need some sort of consent/privacy policy/additional paperwork/IRB review if you do a project intended for a science fair or publication.

So, this will help your curiosity and maybe give you some good ideas for how to do a formal study. But as I said before, this is more a "conversation" than a study.

I do "quick and dirty" (not rigorously planned) experiments every day that aren't intended for publication. I would never use these results in a paper, since they were not planned with the appropriate controls. However, the results help me plan proper experiments or assess the feasibility of experiments. This is an important part of science, but you have to realize the limits of the data you collect under these conditions. In your case, there are numerous weaknesses with the methodology proposed (some of which we have discussed above). As long as you realize the limits of what you are doing/ validity of the data you collect, this can be an interesting topic for you to explore. It may lead to a great project.

Louise

[deleted-71447]

Hi Chris G., I'd be very happy if you could refer me to your friend.... thank you so much!

OK, I'll check to see if she is available.

[Louise]

I have been looking around for some references for you and haven't been able to find much. This page from sciencebuddies has good advice on designing questions for your survey.

https://www.sciencebuddies.org/mentorin ... rvey.shtml


I don't know what grade you are in, but this book got good reviews and might be very useful for you:
Real World Research: A Resource for Social Scientists and Practitioner-researchers (Regional Surveys of the World) (Paperback)
by Colin Robson (Author)
ISBN-10: 0631213058
ISBN-13: 978-0631213055


I haven't read it, but looking at the reviews, it seems like it might be very useful to you, though it is a college level text. I would see if you can get it from the library- it is probably too expensive to buy for a single study. If your library doesn't have it, see if they can borrow it for you from another library through inter-library loan.

Another possible choice is:
The Good Research Guide: For Small-scale Social Research Projects (Paperback)
by Martyn Denscombe (Author)
ISBN-10: 0335213030
ISBN-13: 978-0335213030

Again, I have not read it, but it looks helpful.

Louise

[deleted-71447]

My friend agreed to assist. You should be getting an email from science buddies administrators soon with the contact info. Please let me know if there are any delays or problems.

[rpeebles]

Dear Methionine:

Chris G referred your question to me, and I am certainly willing to help in any way that I can. I research primarily on the health outcomes of young people with disordered eating and obesity, and I have done a fair amount of work with the Internet as well, researching how it can be a vector to both help and harm those pursuing a healthy weight.

In that line of research, I have researched the health outcomes and quality of life of pro-eating disorder website users, and published my first phase pilot study last winter, and have now submitted another paper based on, yes, an online survey. At the risk of being bashed by all of the people who have been helping you so far, online surveying is indeed an emerging field, and can be done successfully, but requires a lot of thought and care for the reasons expressed above by your other mentors.

In fact, creating a survey of any kind requires a lot of thought to be "valid" and publishable. It is true that you cannot generalize from a website designed to serve Asian people with eating disorders to the general population of Asians with disordered eating, as it is likely that a very specific subset visits that site. And while you cannot verify data, many argue that it is rare that people actually verify a great deal of published data in survey research - for instance, how many times do people who have published surveys on American Indian populations actually check ID, and then verify tribal status before enrolling subjects? Not often. There are ways to crosscheck validity and make it more thorough, and then you must also be honest when writing up findings that this was an online survey, and all that can truly be verified is that the people who answered had visited the website in question, and that is important in itself.

However, to study people via the Internet and publish it, you most certainly have to study them in an official manner, going through a human subjects review board through a major academic center. You can survey people online anytime you choose - it's a free country - but it won't be published without appropriate oversight.

I am happy to discuss this more with you if you are interested. Take care - Rebecka Peebles

[rpeebles]

I had one more thought to share - the concerns of survey bias are actually always true with survey research - both online and "in person". You NEVER force someone to fill out a questionnaire, so subjects ALWAYS choose to participate in your survey. So actually, there is always a selection bias involved. In the real non-online world, you cannot reduce that bias, but instead you keep track of how many people refuse to participate, and then publish your response rate, and that is how someone reading your study determines how large a role bias played. For example, if your response rate for a survey in a GI clinic was 90% of all patients visiting that clinic in a 2 month period of time, that's unbelievably good, and implies very little selection bias, in that everyone participated. You have other biases then, because you're surveying a clinic population that may not represent everyone with a stomach problem, but your survey selection bias is not as huge an issue. On the other hand, if your response rate is 15%, then there are a ton of people who didn't participate, and the question becomes "Why?" If you can explain that answer well, beyond the issue of bias, the survey may still be useful and publishable. If you can't, it's usually a big problem.

With online methodology, there are ways to measure hits to your survey, and completed surveys, and even track IP addresses, and all of these things can help. With small online communitites you can often find out how many active members there are as well. These things cannot give you a true response rate, but can help you approximate that for your analyses.

Hope that helps and clears that up. Basically: survey = selection bias, whether online or not. - Rebecka

[methionine]

Hi,
thanks for your help!

in response to what you 've written-- wouldn't it be sort of a privacy violation-- to track IP addresses and such? One of the main obstacles in my project (if i decide to carry on with it) is that I need IRB approval. While my school is willing to get an IRB together during the summer for me, it's probably difficult to actually get around the rules-- that is, since the ED community is classified as an "at risk" group, and each survey is required to have some sort of parent signed form, it'll be especially difficult, since a significant percentage of parents don't even know that their child goes on pro-ED web sites....
I may choose to email the people who run the competition direrectly.

Also, can you tell me or point me to any resources that can help me design an internet survey? I understand that surveys should be as short as possible (so the subjects don't stop caring in the middle of the test), but there are also many other things I'm probably not aware of.

-M

[rpeebles]

So, to answer:

1. It is not a privacy violation if they give informed consent and know you're doing it. We did not use IP address tracking for our larger study, but instead used hits, but for a one site survey (we were surveying 298 sites) it is not unreasonable. The key is that you don't do it secretly, but let people know that that is how you will be tracking to make sure no one is duplicating their responses. But you do not have to do it that way.

2. However you will do this, you will need informed consent and IRB approval.

3. Many people on these sites are 18 or older, so not everyone will need parental consent. It is certainly a lot more challenging to study people under 18 without parental consent.

4. I think the first thing you have to think about in designing any survey - web included - is that you need to think long and hard about what questions you most want to answer with your study. What are you most interested in learning? What is your hypothesis? What do you think this research will contribute? Once you have answers to these questions, or at least some direction, then you start to design your survey. It's always best if you include as many "validated" questions or measures as you can - those that have already been used in other studies, for example. Making up your own questions or phrasing should only be a last resort. If you let me know your questions, I can help. They don't always have to be short, although you don't want it any longer than necessary. You DO want it well-written and thoughtful.

5. With web research, just like you don't want people to lie to you, you need to be careful not to be deceptive with them. I would definitely not recommend "trolling" their site and asking a casual survey and THEN saying it was a research project.

So - catch me up as your ideas take form - R

[methionine]

Hi,
Sorry it took a while to respond -- I was away and studying for finals--

In response to what you wrote: although a significant part of people on the ED sites are 18 or older, there still are many that are under 18 years of age. Wouldn't that skew my results in a way?

Anyway, more about my ideas. I might still need some more time to sort them out, but ...

I wanted to learn about Asian-Americans' attitudes towards eating disorders, and whether or not that influenced the outcome/prevalence, etc of the disorder in any way. The reason why I was curious about this topic was because I had several friends who suffered from eating disorders (while they were not officially diagnosed, I'm fairly sure that it would have been considered anorexia judging by the criterion listed in the DSMIV).. and their parents did not "believe in" eating disorders themselves.

Sorry if this part is somewhat inconsistent to what I had been writing about earlier-- I'm still trying to get my ideas together. Part of the reason why I thought about doing an online survey was because I thought that this was a feasible way for me to contact those who had been afflicted with disordered eating. On second thought, however, do you think it would be better to simply survey the parents of patients instead? That presents a problem as well: the only way I can think of to contact parents of patients is from hospital lists (if it's even legal to get to those in the first place, that is). That means that the only parents I will be able to even distribute the survey to will have already had their child seen a psychiatrist and/or hospitalized, which would also skew the results drastically. ... I wanted to concentrate on those people whose parents might not even be aware of eating disorders in the first place. A significant number of those people now are online, etc. -- in fact, I know that there have been online groups called "Asians with EDs," etc. ... That's where it all started from.


I really appreciate your help.
-M

[rpeebles]

OK.

1. All surveys are skewed. There are limitations to every choice you make. That's why you develp your question carefully, and then design the best (and most feasible) study you can to answer it as responsibly as possible. Enough said.

It will be MUCH harder to study people under 18. Near impossible, if you want it to be published. This is because it will be very hard to get IRB approval to study them without parental consent, but to get parental consent you REALLY select out an abnormal group of kids for these sites - kids that are comfortable w/their parents knowing they're visiting such a site, which is a very unusual subset.

If you start with people 18 or older, you will start to get information, and potentially could get data that might support the need to study people under 18 - data you would really need if you wanted to convince an IRB that they should waive parental consent. But it does skew your data. It's just that I'm not sure how else you would practically go about this.

2. If you survey the parents, you will get different information. And you're right, you will only be able to access parents that have recognized that their kid has a problem and have sought care - again, an unusual subset. You could screen a random group of parents about their attitudes about their kids' eating and weight, but their kids might not have an eating disorder.

3. So, your question is developing, and that's fine. But right now, it's still not really a question yet. Take a stab at an honest-to-goodness question (ex: Do Asian parents' knowledge about eating disorders impact the severity of disordered eating in their children?) and then take an honest-to-goodness stab at a hypothesis, or a possible answer, to your question. I would recommend doing a literature review first to develop your question further - by looking at http://www.pubmed.org and at the psychologic/social science literature as well - to see what others have already published on your areas of interest. If you don't have access to any medical/psychology literature through your library, let me know and I'll see if I can maneuver some access for you.

4. I think you're right - with what you are expressing an interest in, I think online is a great way to go, and I think the over-18 group will give you a good start. Research is often done in stages. This would be your first stage.

OK, you're making progress.

Let me know when you have more.....-R

[rpeebles]

you might also want to check out the Pew Internet and American Life project - which surveys teens on the Internet - and also the kaiser Family Foundation work - and find out how they got consent for teens to answer their surveys....