?validity of postal survey with low returns

[cchang]

28,438 (about 10% of local population) postal survey was sent out and there were 781 correct returns. 781 is 2.7% of ramdomly chosen people for the survey. Therefore the return containing the views of the people is 0.27% of the local population in total. Is this low figure an "acceptable" return i.e. this figure of return is viewed to be an expected average? Can this result be considered to be a valid survey i.e. could one said the opinion of the 781 returns be representative of the whole population of 280,000? Thanks. Carl

[MelissaB]

There's an interesting discussion of response rates in surveys on Wikipedia: http://en.wikipedia.org/wiki/Response_rate . You might want to consider searching for response rates for surveys like the one you sent out; unfortunately, I am not sure what sort of response rates are typical of postal surveys.

As the link above and this one: http://www.dummies.com/how-to/content/s ... respo.html explain, a low return rate can indeed bias your sample. However, if you have done this survey there is little you can do besides explain what happened and include your response rate.

[cchang]

Dear Melissa B, thank you for all the info and the links. I have to come clean that I am not actually "preparing for science fair"(I do not know which button to click!). The 28,481 postal survey with only 2.7% (781) return was done by the local health authority. I am trying to find out if their results were biased or not as the patients' "wants" did not reflect the MORI poll nor any other surveys. So I wonder if this low return survey in fact is creditable or not. I am concerned that the health authority trying to use the result of their skewed survey to impose their demands and I am trying to find and prove if possible that there may be some flaws in their work. The first link in your reply is very interesting stating the non-responders may say more about the credibility of the survey. In their survey there are >97% non-responders! Is there any reference on postal survey expected responder rates anywhere? Thanks, Carl

[agm]

Hi cchang,

Some other concepts that it might be helpful to apply to this situation are self-selection and sample bias:

http://en.wikipedia.org/wiki/Self-selection
http://en.wikipedia.org/wiki/Biased_sample

Basically, the idea is that there can be a correlation between the type of responses provided by a person and his/her likelihood of responding. For example, if the survey consisted of a question like, "Do you like feature X of our service?", people who want it to be changed might be enthusiastic about responding because they see the survey as an opportunity to voice their opinions and get X changed, while people who like X might not be as strongly moved to respond because they are happy with the status quo. Thus, the average opinion of responders can differ quite a bit from the average opinion of the target population.

Amanda

[cchang]

Dear agm,

Thank you for your help and the links. They are very useful as there were also other surveys that the health authority did definitely with a "biased population". I just try to find out if the low return (2.7%) can in fact render their survey/result invalid! Anyone else out here can give advice ? Thanks Carl.

[ChrisG]

I just try to find out if the low return (2.7%) can in fact render their survey/result invalid! Anyone else out here can give advice ? Thanks Carl.

I think it is impossible to say whether the survey study is valid or invalid without knowing more about the methods and results. Even a very tiny sample % can give accurate results if it is a truly random sample, or if enough is known to be able to correct for observed biases. Also, without knowing the interpretation of the survey, it is impossible to say whether the interpretation is correct, or whether it warrants the position held by the health authority.

For more info, you might want to get a copy of this paper from a local university, or by sending a reprint request to the authors.
Title: Addressing Nonresponse Bias in Postal Surveys
Author(s): MacDonald SE, Newburn-Cook CV, Schopflocher D, et al.
Source: PUBLIC HEALTH NURSING Volume: 26 Issue: 1 Pages: 95-105 Published: JAN-FEB 2009

That paper includes references to several other relevant studies.

Good luck!
Chris

[cchang]

Dear Chris,

Thank you for your help and the info link. I just read the abstract of that article regarind "Non response bias" and will try to get the full article. I am most grateful for all the experts spending time to response to my question and giving me advice to find out more reading material. I found all the responses very helpful. Thank you once again. Carl