Polling Continues To Raise A Weighting Issue

It seems that every time CBS News conducts a poll we get questions. When CBS News conducts a poll that shows President Bush’s approval rating at an all-time low of 34%, the questions become a whole lot more skeptical. We’ve looked at this issue from a couple different angles in the past and we’ll probably go through it many more times.

Since the questions we receive almost always revolve around the issue of weighting – how many Democrats, Republicans, Independents, etc. are included in the polling sample – we’ll take you through CBS’ methodology once again. Here’s how the CBS polling unit describes it:

At the end of our surveys, we find sometimes that we have questioned too many people from one group or another. Older people, for example, tend to be at home to answer the phone more than younger people, so there is often a greater percentage of older people in our surveys than exists in the American public.

When that happens, we take great pains to adjust our data so that I accurately reflects the whole population. That process is called “weighting.” We make sure that our final figures match U.S. Census Bureau breakdowns on age, sex, race, education, and region of the country. We also “weight” to adjust for the fact that people who share a phone with others have less chance to be contacted than people who live alone and have their own phones, and that households with more than one telephone number have more chances to be called than households with only one phone number.

So when we add up all the answers to our questions, we know that no one’s opinion counts for more than it should. When you see one of our poll results on TV or in the newspaper, you know that it does not show the opinions of only one or two groups of Americans.

Other pollsters use different methods of weighting than the U.S. Census breakdowns, but this is commonly accepted method of polling. There are plenty of complex and debated issues within the polling community about their craft, which is sometimes described as part science, part art. If you want to really delve into the issues, check out Mystery Pollster for a fantastic primer and in-depth analysis.

Update: Mystery Pollster has looked at the CBS poll results with a different weighting model:

Some will no doubt seize on the fact that the latest CBS News sample is a few points more Democratic on party ID (37%) than on their last three surveys (34% in late January, 33% in early January and 32% in December), although the Republican percentage (28%) is about the same as the last three surveys (27%, 29% and 28% respectively). However, the difference in the party results does not explain the drop in the Bush job rating, which occurs across all three categories.

In fact, even when MP recalculates the CBS job approval results for the most recent survey using the average party composition reported on their last three surveys (33% Democrat, 28% Republican, 39% independent or other), the Bush approval percentage still rounds to 34%. The reason is that my recalculation just increases the number of independents at the expense of Democrats. However, Bush's rating is now so low among both subgroups as measured by CBS that the adjustment makes little difference.

Update:The Anchoress has compiled a roundup of blog reaction to the poll.