How Does Age Affect Polls?
This column was written by CBS News director of surveys Kathy Frankovic.
Last week, I wrote about question order, and why it sometimes matters. But I didn't say much about why it happens - or which groups it is most likely to affect.
A dominant view about why question order effects occur is that people who have the least well-formed and least strongly-held opinions can be affected by what they've just heard. That would be why we sometimes find that the assessment of one candidate is based on the context of who else was asked about immediately before. (This is why Bob Dole looked better, at least to some voters, when he was compared to Pat Buchanan than when he was compared to Bill Clinton). Contrary-wise, people who already have strong and consistent opinions theoretically won't be affected by what they are asked about beforehand.
In the early 1980's, we saw examples of how the subject matter of a questionnaire even affected the way people thought of themselves - e.g., when asked if they were Democrats or Republicans. When we asked for party identification (typically at the end of our surveys) we found that if many of the questions in the poll were about drugs or crime, more people called themselves Republicans; when they were asked mostly about poverty or other social problems, more of them called themselves Democrats.
We have long believed that the older you are, the more likely you are to have strongly-held opinions. A lifetime of thinking about political issues should make opinions part of your identity - if you think about politics at all.
But a new article by a team of American, Canadian and German researchers suggests a different reason for why question order has less of an effect on older respondents. In The Journal of Official Statistics 23 (2007) published by the Swedish national statistical office), Barbel Knauper, Norbert Schwarz, Denise Park and Andreas Fritsch make the case that older people holding strong opinions may only be part of the explanation. They suggest older people are less likely to be affected by question order because they process information differently - which is a polite way of saying that older people may not remember what they've just been asked as often or as easily as younger people do. So what has gone before is less likely to affect their responses later in the survey.
Now before you dismiss this as simply a stereotype of senior citizens, recognize that this research is part of a growing body of knowledge about the cognitive processes involved in answering survey questions. How do people react when they are asked a question? Cognitive theorists believe the mental process works this way: when people are asked a question, they first have to understand the meaning of the question, then recall the appropriate information from memory and then form a judgment and an answer which both reflects their judgment and fits into the options they have been offered. In other words, people don't necessarily have the right answer to a poll question immediately at hand - they have to FIND it.
When question order effects occur, we assume that something from the previous question or questions is one of the many things that people are thinking about when people are answering the next question. It is part of the respondent's "working memory" at the moment he or she responds. How much information is in working memory varies from person to person, and tends to decline with advancing age. Therefore, in theory, older adults are less likely to keep the details of the prior question in mind when they answer the next one. They have to go through the process of searching out the information to answer the question and make their judgment all over again.
The authors tested this hypothesis in an experiment that looked at differences between older adults with high working memory and those with lower working memory. [That difference in working memory was measured by asking people to answer questions based on a set of sentences read to them one by one, while at the same time keeping the last word of each sentence in memory - which they were asked to write down at the end of the sequence.] Sure enough, those older adults with high working memory were more subject to question order effects than were those with lower working memory!
The authors make another case for the role of aging (and the corresponding decline of working memory) in survey responses. Looking at the relationship of age to response order (as opposed to question order) the impact increases with age. Older people are more likely to be subject to what is called the "recency effect" - more likely to choose the answer option in a question that is offered to them last. In other words, the working memory for (at least some) older people may not be high enough to contain all the options. The longer the question, the harder it is to remember everything the question is about and all of what the answer options!
Thus, when designing survey questions and ordering questionnaires, it's very helpful to know that these effects exist. But we need to keep in mind that while the overall effect can be minimal, but their impact probably isn't random!
By Kathy Frankovic