With Polls, Trust But Verify

CBS' Kathy Frankovic: Putting Surveys Into Context Is Key To Understanding

(CBS)  By Kathy Frankovic, CBS News' director of surveys.

---

Polls say many things — but sometimes they don't — really.

I've been conducting public opinion polls for CBS News since 1977 — 30 years. That's a long time by anyone's standards, but examples of misread and misinterpreted polls don't only come from 30 years ago. Even today, they take place far too often.

One of the biggest problems in reading public opinion polls is placing too much confidence in them — taking numbers from a survey and reading them as a precise measurement of where the public stands. Some polls even insist on providing the results to a tenth of a percentage point, while admitting their results could vary by as much as three or four full percentage points just because of a sampling error.

Too many pundits (and pollsters) spend time trying to explain a one or two point change in ratings based on current events when there is a greater likelihood that the change really is due to small fluctuations in the sample — the margin of error.

Frankly, there is no practical — or political — difference between a presidential approval rating of 34 percent (the latest USA Today/Gallup poll), 33 percent (the latest Gallup poll) and 32 percent (the latest CBS News/New York Times poll). In every case, only about one in three Americans approve of the way the president is handling his job. It's bad news for the incumbent any way you look at it, with similar political impact.

There are other problematic and misleading interpretations. National pre-primary polls taken in the year before an election may say nothing about the outcome of even the first primary.

In March 1991, Bill Clinton wasn't even on the list of candidates proposed to the public in an NBC News/Wall Street Journal poll as possible Democratic nominees. New York's Gov. Mario Cuomo was Democrats' top choice. Of course, that was a different era. As late as that June, only Massachusetts Sen. Paul Tsongas had actually officially declared himself a Democratic candidate.

And in that same time period — one and a half years before the 1992 presidential election, then-President George H.W. Bush led the generic "Democrat" by nearly two to one in 1992 horserace questions. After the successful completion of the 1991 Persian Gulf War, the president's approval ratings were high. But by 1992, they were not. Pat Buchanan challenged Bush in the Republican primaries, and Bill Clinton easily won in November.

Current polls (including the CBS News Poll) suggest that a generic Democratic candidate would easily defeat a Republican candidate in the 2008 presidential election. But that may or may not happen. Polls taken now can't be used as a prediction of what will happen after more than a year.

All the poll reviews and all the academic studies of pre-election polls tell us that polls are very good predictors close to an election. Right now, they only reflect the public's current knowledge and perceptions, and the current state of the country. But things can change.

We've all seen poll reports of questions that ask Americans what they think other Americans think. Those are great for telling us whether people assume they themselves are in the majority of the minority on an issue.

For example, before an election we routinely ask voters not just who they plan to vote for, but who they think will win the election. Usually, one candidate's supporters are a lot more hopeful than the other candidate's, and that can be an indication of voter enthusiasm. In the days immediately before the 2004 election, 84 percent of Bush voters were confident that their candidate would be reelected, but only two-thirds of Kerry voters believed Kerry would win.

But don't assume these questions are always good predictions, even though Bush beat Kerry in 2004. The question was about each individual's optimism or pessimism about winning in 2004, but it also reflected media coverage and what "experts" are telling attentive voters. We sometimes ask people to tell us why an event happened, or what is in other people's minds, but their answers don't give us a definitive answer to those questions. The public isn't an "expert" on psychology, or on the motivations of politicians. They are experts on themselves!

Some poll questions tell us mainly what pollsters are thinking. The Roper Center at the University of Connecticut maintains a searchable database of nearly all public opinion poll questions, including their results.

My all-time favorite finding says a great deal about both the increasing numbers of polls and how professional pollsters judge what's important. There are nearly three times as many questions that include the name "Monica Lewinsky" than include the word "Watergate." Until just last year, Monica Lewinsky was the most frequent woman's name in public opinion poll questions, with more than 1,500 questions that included her name.

Last year, there was a new woman front-runner, and her name was Hillary Clinton.

[weareone2]

During and after Reagan's presidency, polls (I believe by Harris) kept comeing out about how everybody loved Reagan. But I noticed that there were no options that would allow me and many people I knew to register their deep feelings against Reagan. About the worse thing you could say was that you disagreed that he's was the greatest president of all time. A few months after Reagan left office, there was another one of those polls. So I finally e-mailed Harris and complained. I got a reply that they stood by their results. But a few months later, they ran another poll that allowed a full range of opinions toward both Reagan and Jimmy Carter. Carter and Reagan scored about the same on the percent of positive feelings toward them. But while very few people had strongly negative feelings against Carter, about as many people had very strong negative feelings against Reagan as had positive feelings for him. But this was the first time the so-called "liberal media" allowed the public to know that.

[realpatriot1]

GunownerDan,

It seems only fair that we Democrats get the left one and the Republicans the right. See how easy it is to find bipartison solutions.

[randalds]

I'll tell you since when. Since Congress (and especially Pelosi's) national poll numbers have fallen BELOW 29 percent approval (that's 3 to 6 percentage points LOWER than the Presidents approval rating.

That's why CBS is suddenly saying polls can't always be trusted.
Posted by SlipSter01 at 05:32 PM : May 16, 2007
+ report abuse

Strange. What polls are you using? Must be one of the Reich Polls because all the one's I've seen show the public liking the new congress and the fact that FINALLY someone is standing up to the Gestapo and the Little Nazi. I'll bet you've been using that Magic Swastika again haven't you? ROFLMAO
Posted by skyk at 05:42 PM : May 16, 2007

It's a fine example of how the right intentionly mis-reads polls. They say that the approval for Congress has fallen to 29% as if it was higher the last few months and is dropping, when in fact the opposite is true. Congressional approval IS indeed at 29%, but it is steadily rising and has been since the election last November, indicating not as they would have you believe a growing dissatisfaction with Congress, but rather in fact a growing approval of Congress in it's opposition to Bush. Whereas Bush's numbers are continuing, with minor fluctuations, on a downward trend.

[cbs_oliver]

Another problem is that people do not always answer questions truthfully.

Ask folks if they believe in God and the vast majority will say yes. Ask if they follow the ten commandments and the vast majority will say yes.

But they don't and they don't. We can tell by what we see them do.

So polls like that are not helpful in determining the truth - but they do help determine the prevailing fantasy - the "right" answer.

[realpatriot1]

I love the way the right reads what they want to into these polls. It isn't Democrats in Congress who register at 29% in this single poll,it's all of Congress. That includes the Republicans who are supporting Bush and the Dems who don't have the votes to override the Fuhrer. Was Pelosi rated individually in this poll? The answer is no.

Other polls have shown approval of Congress as high and rising, until this week.

I don't know what the margin of error is for any of these polls, but I do know that any poll with a margin of error of 4% or more are totally useless. Polls are often wrong because the methodology is screwed up and the wording of the questions often slanted toward a desired response.

That's why a mean of many polls is more telling. That's not a liberal slant, that's common sense.

[pakaal]

That's why people need to look at a cross-section of polls, not just hand-pick the ones they like. Not to mention looking at the numbers long-term, and not just a single one, but many.

Case in point, after Hussein's capture in 2004, Bush got the second major "bump" in his steady decline (the first being the lead-in to invading Iraq). He was at around a 50% approval in January 2004, and has slowly been creeping down ever since, to the lowest point (28%) noted last month.

As for Hawksprings' comment about Congress' ratings being lower than Bush's, the obvious reason is that people are really pissed at Congress for not yet being able to be effective in stopping Bush, despite being elected primarily on that promise, way back in November 2006....

[hawksprings]

Funny how this story is running the day other REAL news outlets are putting out poll numbers that show Congress' approval lower than President Bush's.

[forthepeopl1]

Last month, Republican Congressional leaders filed into the Oval Office to meet with President George W. Bush and talk about renewing the controversial USA Patriot Act," writes Doug Thompson for Capitol Hill Blue. "GOP leaders told Bush that his hardcore push to renew the more onerous provisions of the act could further alienate conservatives still mad at the President from his botched attempt to nominate White House Counsel Harriet Miers to the Supreme Court." Thompson reports the following exchange:

"I don%u2019t give a *******," Bush retorted. "I%u2019m the President and the Commander-in-Chief. Do it my way."

"Mr. President," one aide in the meeting said. "There is a valid case that the provisions in this law undermine the Constitution."

"Stop throwing the Constitution in my face," Bush screamed back. "It%u2019s just a ******* piece of paper!"

"I%u2019ve talked to three people present for the meeting that day and they all confirm that the President of the United States called the Constitution %u2018a ******* piece of paper.%u2019"

and america and our elected leaders do nothing about this matter....

[skyk-2009]

No jumbled, blurred accusations. No quotes from MoveOn.org or Micheal Moore, or John Kerry, or Hillary Clinton. What specific law or laws have been broken.
Posted by SlipSter01 at 05:29 PM : May 16, 2007

Sparky what the blue blazes does any of the Hate Targets you used have to do with the Complete and total FAILURE of Sir Lies-A-Lot. You fascist seem to be locked into this Blame them for our screw ups so badly that you can't even address what I'm sure most American's will want addressed. It's hard to imagine having a Government any worse than the one that you Nazi's gave us for 6 years. Better concern yourself with THAT and forget the Third Reich Garbage... you know Attack the Messenger? ROFLMAO

[skyk-2009]

I'll tell you since when. Since Congress (and especially Pelosi's) national poll numbers have fallen BELOW 29 percent approval (that's 3 to 6 percentage points LOWER than the Presidents approval rating.

That's why CBS is suddenly saying polls can't always be trusted.
Posted by SlipSter01 at 05:32 PM : May 16, 2007
+ report abuse

Strange. What polls are you using? Must be one of the Reich Polls because all the one's I've seen show the public liking the new congress and the fact that FINALLY someone is standing up to the Gestapo and the Little Nazi. I'll bet you've been using that Magic Swastika again haven't you? ROFLMAO

[slipster01]

And notice this article is about polls, and how they can't be fully trusted. Since when?

I'll tell you since when. Since Congress (and especially Pelosi's) national poll numbers have fallen BELOW 29 percent approval (that's 3 to 6 percentage points LOWER than the Presidents approval rating.

That's why CBS is suddenly saying polls can't always be trusted.

[slipster01]

OK, for your first response to the poll, I say No to impeachment.

You have nothing to base an impeachment on.

Tell me, and use quotes from the Constitution, Federal Law, etc, to prove your position and answer this question.

"What law or laws have the pr3esident and Vice President broken?"

No jumbled, blurred accusations. No quotes from MoveOn.org or Micheal Moore, or John Kerry, or Hillary Clinton. What specific law or laws have been broken.

[long_rider]

well there is one poll that should be placed before America:

Do you think that the president and vp should be impeached?

There no bias in that question, there is nothing more straight forward.