More Polls Being Used In Problematic Ways
This column was written by CBS News director of surveys Kathy Frankovic.
We are nearing the 50-day mark - 50 days from the Iowa Caucuses. In that period, pollsters have been shocked at least once. But despite all of this year's questioning of polls' accuracy and of their value, polls still exert influence over both American journalists and the American public. They remain the only acceptable assessment of where the public stands.
But what have we really learned? And how are the polls being used and reported this year?
1. Far more polls are being conducted and reported than ever before. Prior to the Feb. 5 California primary, 10 separate polls were reported in the final last week, according to pollster.com. There were 24 New Hampshire polls released in the five days between the Iowa caucuses and the New Hampshire primary. The Gallup Organization now has the "Gallup Daily" which reports daily estimates of the Democratic and Republican horseraces.
2. Because there have been so many polls, many people choose to aggregate - or average - poll results. For an example, see CNN's "Poll of Polls," a concept borrowed from election coverage in Great Britain - here is one recent example. The political Web site RealClearPolitics.com's poll averages have been repeated on cable television as summaries of where the races stand in various states. RealClearPolitics.com also averages presidential and congressional approval ratings on a regular basis. And pollster.com tracks the daily changes in all pre-primary polls.
This reliance on aggregation, while simplifying the presentation of polling, is potentially flawed: It assumes that all poll results are of equal validity, including those with different and sometimes questionable methodologies. Some of the polls included on these sites are polls that do not meet CBS News' polling standards, and we would not report them individually.
3. Sometimes the polls that are more likely to get extensive coverage are those that fit a certain narrative. Prior to the New Hampshire primary, it was the Gallup Poll there that indicated a 13-point lead for Barack Obama that set the stage for the last few days of media coverage. Shrinking leads and changes in who's ahead - whether or not the lead is beyond a poll's margin of error - receive more coverage than continuing leads. This is probably analogous to some academic research that finds more coverage of declining presidential approval ratings than stable or even rising ratings.
On the Republican side, the national polls have reflected multiple changes in who is and who is not a front-runner. Rudy Giuliani was the clear national front-runner until mid-December, when Mike Huckabee and Mitt Romney started moving up. In fact, Huckabee had a lead in an early January USA Today/Gallup Poll conducted right after his Iowa caucus victory. But immediately after New Hampshire, John McCain came back, leading - with ever increasing margins - in every national poll of Republican voters conducted since.
Obama has tied Hillary Clinton in occasional polls since the start of the year, but only since Super Tuesday have national polls consistently showed a very close national race, with Obama slightly ahead.
4. However, the sometimes breathless narrative of poll reporting reflects the ongoing horserace, and is accompanied by criticism and skepticism when polls don't reflect the precision with which journalists imbue them. After Hillary Clinton won the New Hampshire primary, journalists speculated on the reasons the polls were wrong. Reporters, assisted by academics, touched on issues that were usually the purview of academics. And they are being helped by pollsters themselves.
This is not new. This kind of critique was also part of 2004 coverage. "The polls" were criticized in the early fall when there appeared to be little consistency in results and no obvious cause for that inconsistency. Reporters explored methodological questions, like the presence or absence of cell phone households in the samples.
This year, in The New York Times, Andy Kohut raised the question of whether respondents would over-report their willingness to vote for an African-American, when he referenced polls from Tom Bradley's races for Governor of California. The characteristics of the major Democratic candidates - their race and gender - became a key topic for discussing the limits of polling accuracy in many places, including this earlier column of mine.
5. Pollsters are confronting the problems of telephone surveys and rising costs. Thus far several modifications (like using recorded voices as interviewers and relying on Internet panels) have yet to achieve general acceptance. One planned change surely will make polling more expensive: adding a cell-phone only component to the standard random-digit-dial sample.
The American Association for Public Opinion Research, has taken a much more active role in this election. Among other actions, it has established an academic committee to look into pre-primary polling methods. In the last few years, AAPOR has also become more assertive in the media environment, hiring a communications director and making its president available to media on primary days. In addition, its website now links to other news Web pages, and columns about polls and methods by its members and others.
Fifty days from now (and that's still before the April 22 Pennsylvania primary), pollsters and the people who write about polls will know even more about polling problems and poll reporting. And they probably won't hesitate to tell us all about it.
By Kathy Frankovic