The Polling Challenges For 2009
This column was written by CBS News director of surveys Kathy Frankovic.
Did we learn anything in 2008 election campaign polling that will help us measure the public's assessment of the new administration in Washington?
I don't mean what questions we ask. There is a standard for evaluating presidents that is just as good as it was nearly 70 years ago!
Gallup first asked Americans if they approved or disapproved of the way the President (Roosevelt) was handling his job in December 1939, and the responses to that question let us assess the public's response at nearly every moment in individual administrations, and make comparisons between presidents.
Franklin Roosevelt had a 57 percent approval rating on that first Gallup approval measure. In contrast, according to the last Gallup/USA Today approval rating of 2008, only 29 percent of Americans approved of how George W. Bush was handling his job. Sixty-seven percent disapproved -- more than approved of President Roosevelt in 1939.
What I mean is how we get to that measurement and whether we believe what people say. For example, despite a lot of debate in 2008, there is still no agreed upon procedures for reaching all the population when we make polling telephone calls (and, in order to adequately represent the population, we still do make telephone calls).
Many organizations have moved beyond the (relatively) simple process of sampling from the area codes and telephone exchanges assigned to traditional land line phone numbers. That methodology became dominant in the 1970's. Now, many polling organizations, CBS News included, sample cell phones, too. But there is still much to learn about whether or not this helped or hurt pre-election estimates.
In fact, polling organizations that included cell phones in their samples tended to slightly overestimate the vote for Barack Obama last fall. See final pre-election poll estimates here.
The last pre-election CBS News Poll predicted a nine-point Obama lead over John McCain (as opposed to the 7-point edge for the Democrat in the final tally). The last ABC News poll showed a 9-point Obama lead, too, while Obama led by 11 points in the final Gallup measurement. A Pew Center analysis reviews these issues. Earlier in 2008 another Pew analysis suggested that Obama support was significantly higher among voters who had only cell phones than among voters who could be reached on landlines, even when demographic characteristics like age and race were controlled.
We do know something about those who ended up voting. And while the wireless-only voters were different, the differences that Pew found in the summer time didn't show up on Election Day. According to the NEP national exit polls, at least 14 percent of all voters claimed they could be reached only on cell phones, and their support for Barack Obama - by a 23 point margin of 61 percent to 38 percent -- made them very different from other voters.
However, much of that difference in cell-phone only voters was because (as we've known) wireless-only people are demographically different: three in four were under the age of 45, and more than four in ten were under 30. On Election Day, they were more likely than other voters to be minority: African-American and Hispanic.
Poll Database
Search recent CBS News campaign polls.
But among those young voters, whether they could be reached only by landline or only by cell phone or by both, there were hardly any differences in voting preference. The same was true for African-American voters. That's an interesting result, as it suggests that the old-fashioned correction for non-representation of those with only cell phones --weighting on demographic qualities like race and age -- would have worked just fine in the pre-2008 election polls when it came to predicting the vote. Special efforts to reach cell phone only households may only have served to exaggerate Obama's lead.
But those special efforts are still necessary. Our worries about reaching and polling people who only have cell phones won't go away. Because we strive for good polling methodology - and for the correct representation of the nation - we must give people who can only be reached by cell phones a chance of being represented, even if there are alternate ways (like weighting) to adjust the results.
It's like saying you'll conduct a national poll and leave out some states. For years many polling organizations did just that: not calling and interviewing residents of Alaska or Hawaii. Alaska and Hawaii combined make up only one percent of the U.S. population, and yet not including them in a poll seems undemocratic. Wireless-only households are a much bigger concern; they are creeping closer and closer to being one in five households (the latest estimate from the National Health Interview Survey had 17.5 percnet of households wireless-only; and about a third of those under 30 lived in those households).
The other 2008 polling discussion with implications for the Obama Presidency is the long-running debate about the so-called "Bradley Effect," the belief that Americans will lie about their support of an African-American politician, saying they will vote for him even when they won't.
Could it possibly affect assessments the public gives about a "President Obama?" There is a simple answer for this: the "Bradley effect" didn't show up in 2008, no matter how much people wanted to talk about it (there are 234,000 Google hits for "Bradley effect" and Obama).
In fact, it was probably the most over-hyped polling story of last year. If there ever was a "Bradley effect," Daniel J. Hopkins demonstrated, it was gone by the early 1990's.
That's good polling news for 2009 and beyond: Voters who were not voting for candidate Obama in 2008 were perfectly capable of reporting that to pollsters; Americans who disapprove of how President Obama handles his job should have no problem saying that, too.