Why Question Order Changes Poll Results

generic us american flag vote poll capitol building Frankovic polling positions CBS/AP

This column was written by CBS News director of surveys Kathy Frankovic.

Polling seems easy. Write questions. Make sure the options balance, and that the choices cover the range of options. Train interviewers to ask the questions as written, and tally the results.

But if you don't also ask those questions in the right order, things can get complicated.

In 1995, CBS News asked for opinions about the leaders of the Republican-controlled Senate; we learned that opinions of Senate Majority Leader Bob Dole could be quite different, depending on whom we had asked about just before we asked about Dole.

Republicans had won control of both houses of Congress the previous November. In a January 1995 poll, 20 percent had a favorable opinion of Dole; 21 percent an unfavorable opinion. But the next month, positive feelings about Dole shot up to 40 percent -- while unfavorable opinions were at 23 percent.

Yet we had asked the same question in both surveys: "Is your opinion of Bob Dole favorable, not favorable, undecided, or haven't you heard enough about Bob Dole yet to have an opinion?"

Something had changed, but we weren't sure what it was, until we considered question order.

In the January poll, the Bob Dole question followed a series of questions about Bill Clinton; in February, it followed a question about Pat Buchanan, who merited only a 10 percent favorable rating from the public. Even at Clinton's lowest points, after the Republicans' Senate takeover, he was far more favorably viewed than Buchanan. Some people were basing their judgment of Dole on how it contrasted with their judgment of the person we had just asked about. Dole wasn't as popular as Clinton was, but he was a whole lot more popular than Buchanan!

The survey world produces many examples of order affecting results, sometimes even for questions we think tap very basic, strongly-held opinions. Take abortion, for example. Responses indicating support for legal abortion in general can change if we first ask about support for legal abortion in specific circumstances. Once people allow for abortions in specific circumstances, they are less willing to say they approve of abortion in general.

Sometimes apparently minor differences in wording, made to accommodate changes in question order due to news events, will create large differences in responses. A telling example occurred in the summer of 1998. On August 17, President Bill Clinton gave a speech in which he finally admitted to a relationship with Monica Lewinsky that was "not appropriate." News survey operations quickly went into the field to determine the impact of the speech. (Even as the scandal grew, Clinton's approval ratings had remained fairly steady).

During that year, the Gallup Organization had been sounding out public perception of Clinton in the same way as it asked about other prominent individuals, using this question: "I'd like to get your overall opinion of some people in the news. As I read each name, please say if you have a favorable or unfavorable opinion of this person or if you have never heard of him or her. What is your overall opinion of Bill Clinton?" On the evening of Clinton's speech admitting responsibility, though, Gallup asked only about the President - and boiled the entire question down to "Thinking about Bill Clinton as a person, do you have a favorable or unfavorable opinion of him?" The question immediately followed the question that usually begins news polls: "Do you approve or disapprove of the way … is handling his job as President?"

In a poll taken before the speech (Aug.10-11), the CNN/USA Today/Gallup Poll reported a 58 percent favorable rating for Clinton, with 40 percent unfavorable. But in the post-speech Gallup Poll, opinion appeared to have nearly reversed: only 40 percent favorable and 48 percent not! It was tempting to think the public had finally turned against the President, once he made his admission.
  • CBSNews

Comments

CBSN Live

pop-out
Live Video

Watch CBSN Live

Watch CBS News anytime, anywhere with the new 24/7 digital news network. Stream CBSN live or on demand for FREE on your TV, computer, tablet, or smartphone.