A Holiday Break From Polls?

By Kathy Frankovic, CBS News director of surveys

Last year, according to the University of Connecticut's Roper Center, which archives questions from nearly all public poll releases, there was only one poll conducted between the Friday before Christmas and New Year's Day. That was the ABC News/Washington Post Consumer Comfort Poll, which uses continuous polling on three economic measures. Several polls started up again on New Year's Day 2007, to collect the public's expectations about the first Democratic-controlled Congress in more than a decade.

Four years ago, according to the same archive, the final pre-2004 election year poll ended its interviewing on December 22, and although Time Magazine and CNN started up again on December 30, Christmas week itself was still (except for those continuous-polling financial monitors, of course).

This year, nearly every polling organization continued polling until late in the season: at least one has polled in Iowa through the Saturday before Christmas. And it will be interesting to see what polls (if any) are released in the week between Christmas and New Year's Day. With the Iowa caucuses scheduled for just two days after New Year's Day, curiosity may get the better of pollsters this year. They may well be tempted to poll during the weekend before New Year's - and even on New Year's Day - to get one last measure of opinion before the caucuses.

There are two reasons why we typically do not poll during holidays. First (as last weekend's news reports reminded us), lots of people can't be found at their usual residences. Travelers on U.S. airlines were expected to approach 50 million during the holiday travel period. Add to that people traveling by car, bus and train and many, many people are simply unreachable.

But the second reason is that the responses people give during the holidays might not really be their best or their most thought-through. That is not just a holiday concern. Pollsters always need to worry about the kinds of questions they ask: are they clear? Are the choices mutually exclusive? Are these topics really ones that people have opinions about? There is a robust academic literature about how people interviewed in polls treat the questions they are asked: do they think through all the choices? Or do they "satisfice," and just retrieve as much information as necessary to give a reasonable answer? These questions are especially important when respondents are taking a holiday break from their day-to-day routines, and so may not want to spend much time contemplating their answers.

To illustrate, consider a recent article in Public Opinion Quarterly (POQ) (see below). Researchers Allyson Holbrook, Jon Krosnick, David Moore and Roger Tourangeau examined what happened in over 500 question wording experiments conducted between 1995 and 2000 by the Gallup Organization. Gallup rotated the order of the two possible responses in these two-choice questions (one randomly selected half of the total interviewed were read the options in one order, the other random half were read the responses in the other order). The authors note that while there were significant differences in the responses of each half sample in less than half of the experiments, there were patterns that recurred. Response order effects were most likely in longer questions (those where the choices were complete sentences and not just words or phrases), and in questions asked later in the questionnaire (after respondents may have been more tired from answering questions). Differences were reduced, the authors write, when the questions included phrases that suggested the respondents wait for the options being offered, or when they started with a phrase that made the question seem to be much broader than the final two options offered.

When there were significant differences, they mainly favored the option asked second, what in polling is known as a "recency effect." In these cases, respondents were more likely to choose an option when it came second, not first: when it was more recently in their minds. Other studies offer evidence that occasionally the opposite happens: another POQ article (by Monika McDermott) found that in early 2000 pre-election polls, at a time before voters may have made hard choices for that fall's election, in a choice between George W. Bush and Al Gore, the candidate whose name came first did better. That is known as a "primacy effect."

With only one week to go before 2008, we pollsters need to remember that we have a responsibility to make things easy for respondents - not to offer them complicated choices, and especially not to invent long questionnaires. We have to understand the people we interview - not force them to try and understand us. Think of this as our Christmas present to voters, and as our New Year's resolutions.

In the meantime, a peaceful and quiet holiday season to all!

Note: POQ is the official journal of the American Association of Public Opinion Research and is available online only to AAPOR members and to subscribers to JSTOR. The two articles referred to here are at here (Holbrook et al.) and here (McDermott) but one version of the Holbrook paper is available here.