Scott Ketter's "Methods can matter: Where web surveys produce different results than phone interviews"
Pew recently revisited the question of how survey modality can influence survey responses. In particular, this survey used both web and telephone based surveys to ask participants about their attitudes towards politicians, perceptions of discrimination, and their satisfaction with life.
As summarized in the article, the big differences are:
"1) People expressed more negative views of politicians in Web surveys than in phone surveys."
"2) People who took phone surveys were more likely than those who took Web surveys to say that certain groups of people – such as gays and lesbians, Hispanics, and blacks – faced “a lot” of discrimination."
"3) People were more likely to say they are happy with their family and social life when asked by a person over the phone than when answering questions on the Web."
The social psychologist in me likes this as an example of the Social Desirability Bias. When speaking directly to another human being, we report greater life satisfaction, we are more critical of politicians, and more sympathetic towards members of minority groups.
The statistician in me thinks this is a good example for discussing sources of error in research. Even a completely conscientious research using valid, reliable measures may have their data effected based on how it is collected. It might be interesting to asks students to generate lists of research topics (say, market research about cereal preference versus opinions about abortion) and whether students think you could get "true" answers via telephone or web surveys. What is a "true" answer, how could we evaluate or measure this? How could we come up with an implicit or behavioral measure of something like satisfaction with family life, then test which survey modality is most congruent with an implicit or behavioral measure? What do students think would happen if you used face-to-face interviews or paper and pencil surveys in a classroom of people completing surveys?
Additionally, you can't call yourself a proper stats geek unless you follow Pew Research Center on either Twitter (@pewresearch) or on Facebook . So many good examples of interesting data!
As summarized in the article, the big differences are:
"1) People expressed more negative views of politicians in Web surveys than in phone surveys."
"2) People who took phone surveys were more likely than those who took Web surveys to say that certain groups of people – such as gays and lesbians, Hispanics, and blacks – faced “a lot” of discrimination."
"3) People were more likely to say they are happy with their family and social life when asked by a person over the phone than when answering questions on the Web."
The social psychologist in me likes this as an example of the Social Desirability Bias. When speaking directly to another human being, we report greater life satisfaction, we are more critical of politicians, and more sympathetic towards members of minority groups.
The statistician in me thinks this is a good example for discussing sources of error in research. Even a completely conscientious research using valid, reliable measures may have their data effected based on how it is collected. It might be interesting to asks students to generate lists of research topics (say, market research about cereal preference versus opinions about abortion) and whether students think you could get "true" answers via telephone or web surveys. What is a "true" answer, how could we evaluate or measure this? How could we come up with an implicit or behavioral measure of something like satisfaction with family life, then test which survey modality is most congruent with an implicit or behavioral measure? What do students think would happen if you used face-to-face interviews or paper and pencil surveys in a classroom of people completing surveys?