Why Question Order Changes Poll Results

Kathy Frankovic: Change The Order Of The Questions In A Poll, You Change The Outcome

(CBS)  This column was written by CBS News director of surveys Kathy Frankovic.

---

Polling seems easy. Write questions. Make sure the options balance, and that the choices cover the range of options. Train interviewers to ask the questions as written, and tally the results.

But if you don’t also ask those questions in the right order, things can get complicated.

In 1995, CBS News asked for opinions about the leaders of the Republican-controlled Senate; we learned that opinions of Senate Majority Leader Bob Dole could be quite different, depending on whom we had asked about just before we asked about Dole.

Republicans had won control of both houses of Congress the previous November. In a January 1995 poll, 20 percent had a favorable opinion of Dole; 21 percent an unfavorable opinion. But the next month, positive feelings about Dole shot up to 40 percent -- while unfavorable opinions were at 23 percent.

Yet we had asked the same question in both surveys: “Is your opinion of Bob Dole favorable, not favorable, undecided, or haven't you heard enough about Bob Dole yet to have an opinion?”

Something had changed, but we weren’t sure what it was, until we considered question order.

In the January poll, the Bob Dole question followed a series of questions about Bill Clinton; in February, it followed a question about Pat Buchanan, who merited only a 10 percent favorable rating from the public. Even at Clinton’s lowest points, after the Republicans’ Senate takeover, he was far more favorably viewed than Buchanan. Some people were basing their judgment of Dole on how it contrasted with their judgment of the person we had just asked about. Dole wasn’t as popular as Clinton was, but he was a whole lot more popular than Buchanan!

The survey world produces many examples of order affecting results, sometimes even for questions we think tap very basic, strongly-held opinions. Take abortion, for example. Responses indicating support for legal abortion in general can change if we first ask about support for legal abortion in specific circumstances. Once people allow for abortions in specific circumstances, they are less willing to say they approve of abortion in general.

Sometimes apparently minor differences in wording, made to accommodate changes in question order due to news events, will create large differences in responses. A telling example occurred in the summer of 1998. On August 17, President Bill Clinton gave a speech in which he finally admitted to a relationship with Monica Lewinsky that was “not appropriate.” News survey operations quickly went into the field to determine the impact of the speech. (Even as the scandal grew, Clinton’s approval ratings had remained fairly steady).

During that year, the Gallup Organization had been sounding out public perception of Clinton in the same way as it asked about other prominent individuals, using this question: “I'd like to get your overall opinion of some people in the news. As I read each name, please say if you have a favorable or unfavorable opinion of this person or if you have never heard of him or her. What is your overall opinion of Bill Clinton?” On the evening of Clinton’s speech admitting responsibility, though, Gallup asked only about the President - and boiled the entire question down to “Thinking about Bill Clinton as a person, do you have a favorable or unfavorable opinion of him?” The question immediately followed the question that usually begins news polls: “Do you approve or disapprove of the way … is handling his job as President?”

In a poll taken before the speech (Aug.10-11), the CNN/USA Today/Gallup Poll reported a 58 percent favorable rating for Clinton, with 40 percent unfavorable. But in the post-speech Gallup Poll, opinion appeared to have nearly reversed: only 40 percent favorable and 48 percent not! It was tempting to think the public had finally turned against the President, once he made his admission.

[cyberus-2009]

Not news at all .. thats how polls are manipulated.
Changing the order of questions is the easiest .. asking the same question using different words with the same meaning can alter the results.
Thats why there are so many pollsters out there .. every one of them are looking to get a result that will sell to someone.

[it_oldtimer]

What an utterly nonsensical title for an article!

Without actually reading the article I wouldn''t have had a clue what the subject being presented in it was.

The total lack of basic literary skills shown by so many supposedly "professional" journalists these days has become truly appalling.

[joyous88]

All this is really indicative of is the simple fact

that in the United States the conservatives have

dumbed down education to the point that a high

school education is a waste of time.

Most people do not have the skill in statistics to

understand that they are in reality being manipulated

by the corporations.

You can not believe most of what you read from the

corporatist media, remember, One guy owns it all, and

he

is a greed driven conservative.

[veteran72]

You know that the Bush/Cheney Crime Cartel has crunched the numbers and knows they''re headed for the Little Big Horn come November. Even with the MSM giving McBushcain a complete pass, he barely shows on the radar......that''s what they get for responding to
all reports of their crimes and incompetence with: "So"???