E-Mailbag: Poll Position

We got an email from Mark Blumenthal, the Mystery Pollster, concerning a point made by a commenter on his site about CBS News poll reporting standards. Blumenthal has been writing about a controversy over an American Medical Association-sponsored survey of young women about spring break which was covered by a number of news outlets, including the "Early Show." He pointed out that much of the reporting about the poll was flawed, in part because much of it relied on an AMA press release incorrectly identifying the respondents as a "random sample" of 17-34 year old women. (Mark has a lot more analysis here and here.)

The poll prompted one of his commenters, "JeanneB", to write the following:

More and more news outlets have hired ombudsmen in recent years. They also need to add someone to filter polls and ensure they're described accurately (if they get used at all)...

They have a responsibility to police themselves and set some kind of standard as to which polls will be included in their coverage and how they will be described. Please don't let them off the hook for swallowing whole any old press release disguised as a "poll".

I talked to Kathy Frankovic, CBS News Director of Survey, who emailed to me the updated standards for CBS News poll reporting. Here they are:

Polling is a complicated and often controversial method of research. Not everyone has the background to do it. CBS News does its own polling and reports on various outside polling efforts. Before any poll is reported, we must know who conducted it, when it was taken, the size of the sample and the margin of statistical error. Polling questions must be scrutinized, since slight variations in phrasing can lead to major differences in results. If all the above information is not available, we should be wary of reporting the poll. If there are any doubts about the validity, significance or interpretation of a poll, the CBS News director of surveys should be contacted. The CBS News Election and Survey Unit will maintain a list of acceptable survey practices and research organizations.

Whenever major poll results are reported on the air, we should include the pertinent facts about who conducted the poll, when it was taken and the margin of error. In the reporting the results of the poll, each question should be phrased accurately (verbatim is always best) immediately preceding the results.

In some cases, poll information may be shorthanded, such as, "In recent polls, the President has been getting high approval ratings from the American people." However, such assertions should only be made when the reporter can cite specific poll findings that support the claim. When the poll itself is the story, we should err on the side of providing too much, rather than not enough, information about its specific characteristics.

A word about the rapidly growing number of nonscientific polls: these are call-in polls, Internet and Web-site polls, and other surveys that inherently are not representative of the population. Such "pseudo-polls" should not be confused with scientific surveys and should never be reported as indicators of opinion.