With Polls, Trust But Verify
By Kathy Frankovic, CBS News' director of surveys.
Polls say many things — but sometimes they don't — really.
I've been conducting public opinion polls for CBS News since 1977 — 30 years. That's a long time by anyone's standards, but examples of misread and misinterpreted polls don't only come from 30 years ago. Even today, they take place far too often.
One of the biggest problems in reading public opinion polls is placing too much confidence in them — taking numbers from a survey and reading them as a precise measurement of where the public stands. Some polls even insist on providing the results to a tenth of a percentage point, while admitting their results could vary by as much as three or four full percentage points just because of a sampling error.
Too many pundits (and pollsters) spend time trying to explain a one or two point change in ratings based on current events when there is a greater likelihood that the change really is due to small fluctuations in the sample — the margin of error.
Frankly, there is no practical — or political — difference between a presidential approval rating of 34 percent (the latest USA Today/Gallup poll), 33 percent (the latest Gallup poll) and 32 percent (the latest CBS News/New York Times poll). In every case, only about one in three Americans approve of the way the president is handling his job. It's bad news for the incumbent any way you look at it, with similar political impact.
There are other problematic and misleading interpretations. National pre-primary polls taken in the year before an election may say nothing about the outcome of even the first primary.
In March 1991, Bill Clinton wasn't even on the list of candidates proposed to the public in an NBC News/Wall Street Journal poll as possible Democratic nominees. New York's Gov. Mario Cuomo was Democrats' top choice. Of course, that was a different era. As late as that June, only Massachusetts Sen. Paul Tsongas had actually officially declared himself a Democratic candidate.
And in that same time period — one and a half years before the 1992 presidential election, then-President George H.W. Bush led the generic "Democrat" by nearly two to one in 1992 horserace questions. After the successful completion of the 1991 Persian Gulf War, the president's approval ratings were high. But by 1992, they were not. Pat Buchanan challenged Bush in the Republican primaries, and Bill Clinton easily won in November.
Current polls (including the CBS News Poll) suggest that a generic Democratic candidate would easily defeat a Republican candidate in the 2008 presidential election. But that may or may not happen. Polls taken now can't be used as a prediction of what will happen after more than a year.
All the poll reviews and all the academic studies of pre-election polls tell us that polls are very good predictors close to an election. Right now, they only reflect the public's current knowledge and perceptions, and the current state of the country. But things can change.
We've all seen poll reports of questions that ask Americans what they think other Americans think. Those are great for telling us whether people assume they themselves are in the majority of the minority on an issue.
For example, before an election we routinely ask voters not just who they plan to vote for, but who they think will win the election. Usually, one candidate's supporters are a lot more hopeful than the other candidate's, and that can be an indication of voter enthusiasm. In the days immediately before the 2004 election, 84 percent of Bush voters were confident that their candidate would be reelected, but only two-thirds of Kerry voters believed Kerry would win.
But don't assume these questions are always good predictions, even though Bush beat Kerry in 2004. The question was about each individual's optimism or pessimism about winning in 2004, but it also reflected media coverage and what "experts" are telling attentive voters. We sometimes ask people to tell us why an event happened, or what is in other people's minds, but their answers don't give us a definitive answer to those questions. The public isn't an "expert" on psychology, or on the motivations of politicians. They are experts on themselves!
Some poll questions tell us mainly what pollsters are thinking. The Roper Center at the University of Connecticut maintains a searchable database of nearly all public opinion poll questions, including their results.
My all-time favorite finding says a great deal about both the increasing numbers of polls and how professional pollsters judge what's important. There are nearly three times as many questions that include the name "Monica Lewinsky" than include the word "Watergate." Until just last year, Monica Lewinsky was the most frequent woman's name in public opinion poll questions, with more than 1,500 questions that included her name.
Last year, there was a new woman front-runner, and her name was Hillary Clinton.
By Kathy Frankovic
Copyright 2009 CBS. All rights reserved. Polls say many things — but sometimes they don't — really.
I've been conducting public opinion polls for CBS News since 1977 — 30 years. That's a long time by anyone's standards, but examples of misread and misinterpreted polls don't only come from 30 years ago. Even today, they take place far too often.
One of the biggest problems in reading public opinion polls is placing too much confidence in them — taking numbers from a survey and reading them as a precise measurement of where the public stands. Some polls even insist on providing the results to a tenth of a percentage point, while admitting their results could vary by as much as three or four full percentage points just because of a sampling error.
Too many pundits (and pollsters) spend time trying to explain a one or two point change in ratings based on current events when there is a greater likelihood that the change really is due to small fluctuations in the sample — the margin of error.
Frankly, there is no practical — or political — difference between a presidential approval rating of 34 percent (the latest USA Today/Gallup poll), 33 percent (the latest Gallup poll) and 32 percent (the latest CBS News/New York Times poll). In every case, only about one in three Americans approve of the way the president is handling his job. It's bad news for the incumbent any way you look at it, with similar political impact.
There are other problematic and misleading interpretations. National pre-primary polls taken in the year before an election may say nothing about the outcome of even the first primary.
In March 1991, Bill Clinton wasn't even on the list of candidates proposed to the public in an NBC News/Wall Street Journal poll as possible Democratic nominees. New York's Gov. Mario Cuomo was Democrats' top choice. Of course, that was a different era. As late as that June, only Massachusetts Sen. Paul Tsongas had actually officially declared himself a Democratic candidate.
And in that same time period — one and a half years before the 1992 presidential election, then-President George H.W. Bush led the generic "Democrat" by nearly two to one in 1992 horserace questions. After the successful completion of the 1991 Persian Gulf War, the president's approval ratings were high. But by 1992, they were not. Pat Buchanan challenged Bush in the Republican primaries, and Bill Clinton easily won in November.
Current polls (including the CBS News Poll) suggest that a generic Democratic candidate would easily defeat a Republican candidate in the 2008 presidential election. But that may or may not happen. Polls taken now can't be used as a prediction of what will happen after more than a year.
All the poll reviews and all the academic studies of pre-election polls tell us that polls are very good predictors close to an election. Right now, they only reflect the public's current knowledge and perceptions, and the current state of the country. But things can change.
We've all seen poll reports of questions that ask Americans what they think other Americans think. Those are great for telling us whether people assume they themselves are in the majority of the minority on an issue.
For example, before an election we routinely ask voters not just who they plan to vote for, but who they think will win the election. Usually, one candidate's supporters are a lot more hopeful than the other candidate's, and that can be an indication of voter enthusiasm. In the days immediately before the 2004 election, 84 percent of Bush voters were confident that their candidate would be reelected, but only two-thirds of Kerry voters believed Kerry would win.
But don't assume these questions are always good predictions, even though Bush beat Kerry in 2004. The question was about each individual's optimism or pessimism about winning in 2004, but it also reflected media coverage and what "experts" are telling attentive voters. We sometimes ask people to tell us why an event happened, or what is in other people's minds, but their answers don't give us a definitive answer to those questions. The public isn't an "expert" on psychology, or on the motivations of politicians. They are experts on themselves!
Some poll questions tell us mainly what pollsters are thinking. The Roper Center at the University of Connecticut maintains a searchable database of nearly all public opinion poll questions, including their results.
My all-time favorite finding says a great deal about both the increasing numbers of polls and how professional pollsters judge what's important. There are nearly three times as many questions that include the name "Monica Lewinsky" than include the word "Watergate." Until just last year, Monica Lewinsky was the most frequent woman's name in public opinion poll questions, with more than 1,500 questions that included her name.
Last year, there was a new woman front-runner, and her name was Hillary Clinton.
By Kathy Frankovic
It seems only fair that we Democrats get the left one and the Republicans the right. See how easy it is to find bipartison solutions.
That's why CBS is suddenly saying polls can't always be trusted.
Posted by SlipSter01 at 05:32 PM : May 16, 2007
+ report abuse
Strange. What polls are you using? Must be one of the Reich Polls because all the one's I've seen show the public liking the new congress and the fact that FINALLY someone is standing up to the Gestapo and the Little Nazi. I'll bet you've been using that Magic Swastika again haven't you? ROFLMAO
Posted by skyk at 05:42 PM : May 16, 2007
It's a fine example of how the right intentionly mis-reads polls. They say that the approval for Congress has fallen to 29% as if it was higher the last few months and is dropping, when in fact the opposite is true. Congressional approval IS indeed at 29%, but it is steadily rising and has been since the election last November, indicating not as they would have you believe a growing dissatisfaction with Congress, but rather in fact a growing approval of Congress in it's opposition to Bush. Whereas Bush's numbers are continuing, with minor fluctuations, on a downward trend.
Ask folks if they believe in God and the vast majority will say yes. Ask if they follow the ten commandments and the vast majority will say yes.
But they don't and they don't. We can tell by what we see them do.
So polls like that are not helpful in determining the truth - but they do help determine the prevailing fantasy - the "right" answer.
Other polls have shown approval of Congress as high and rising, until this week.
I don't know what the margin of error is for any of these polls, but I do know that any poll with a margin of error of 4% or more are totally useless. Polls are often wrong because the methodology is screwed up and the wording of the questions often slanted toward a desired response.
That's why a mean of many polls is more telling. That's not a liberal slant, that's common sense.
Case in point, after Hussein's capture in 2004, Bush got the second major "bump" in his steady decline (the first being the lead-in to invading Iraq). He was at around a 50% approval in January 2004, and has slowly been creeping down ever since, to the lowest point (28%) noted last month.
As for Hawksprings' comment about Congress' ratings being lower than Bush's, the obvious reason is that people are really pissed at Congress for not yet being able to be effective in stopping Bush, despite being elected primarily on that promise, way back in November 2006....
Funny how this story is running the day other REAL news outlets are putting out poll numbers that show Congress' approval lower than President Bush's.
"I don%u2019t give a *******," Bush retorted. "I%u2019m the President and the Commander-in-Chief. Do it my way."
"Mr. President," one aide in the meeting said. "There is a valid case that the provisions in this law undermine the Constitution."
"Stop throwing the Constitution in my face," Bush screamed back. "It%u2019s just a ******* piece of paper!"
"I%u2019ve talked to three people present for the meeting that day and they all confirm that the President of the United States called the Constitution %u2018a ******* piece of paper.%u2019"
and america and our elected leaders do nothing about this matter....
I'll tell you since when. Since Congress (and especially Pelosi's) national poll numbers have fallen BELOW 29 percent approval (that's 3 to 6 percentage points LOWER than the Presidents approval rating.
That's why CBS is suddenly saying polls can't always be trusted.
You have nothing to base an impeachment on.
Tell me, and use quotes from the Constitution, Federal Law, etc, to prove your position and answer this question.
"What law or laws have the pr3esident and Vice President broken?"
No jumbled, blurred accusations. No quotes from MoveOn.org or Micheal Moore, or John Kerry, or Hillary Clinton. What specific law or laws have been broken.