I get a lot of questions about polls – which are the “good” ones, which are biased, etc. But I get more questions about Gallup than an other polling outfit. Which is why it was fascinating to stumble across this 2004 article that covers all the Gallupfreude – from allegations of partisan bias among the organization’s leaders, to questions about their methodology and sampling.
A clip from the archives:
Connoisseurs of polling – a group that includes this newsletter – have been confused by the presidential preference polls of the last few months. Some are portraying a tight race, and others a Bush runaway. Some of the strangest numbers have been coming out of our most famous pollster, Gallup. Recall that Gallup reported a one-point decline in Kerry’s popularity after the Democratic convention – a departure from historical precedent, since candidates usually have gotten, according to Gallup, an average of a six–point bounce after their nominating conventions. By contrast, Bush got a more normal bounce; the Gallup poll released right after the convention showed him gaining two points, widening his lead over Kerry to seven. That lead widened over the next few weeks to a peak of thirteen points in mid-September, which fell back to a mere eight at month’s end.
Other pollsters have reported a solid Bush lead in recent weeks (though not all – the IBD/TIPP poll has the race as a dead heat). But Gallup was out in front in both speed in size, and their polls showing Kerry’s negative bounce followed by Bush’s extended surge have contributed mightily to a sense that Bush is now the favorite by a fairly wide margin on November 2. Gallup has also been fairly consistently reporting higher approval ratings for Bush than other pollsters over the last year or two.
What’s going on? One clue can be found in looking at their reported party breakdown. In their late-September poll, they have 43% of repondents identifying as Republicans, and 31% as Democrats. We’ve never seen anything like a twelve-point Republican lead anywhere else before. A July New York Times/CBS poll reported the breakdown as 37% Democrat and 29% Republican, an eight-point Dem advantage. The exit polls in both the 2000 and 1996 elections reported the breakdown as 39% Democrat and 35% Republican, a four-point Dem advantage. (Independents were 25% of the Gallup universe, 30% of the Times/CBS universe, and 27% of the 2000 election’s.) Party affiliations change over time, for sure, but a sixteen- or twenty-point shift seems highly unlikely over the course of four years, much less two months.
Gallup provides a breakdown of preference by partisan self-identification (see table below). If you rerun their numbers using the Times/CBS party ID weights, you get a ten-point Kerry lead; if you rerun them using the exit poll weights from the last two presidential elections, you get a six-point Kerry lead. Most pollsters object to weighting raw results by party, since partisan identification is unstable over time, and is a reflection of opinion and not a demographic category. True enough, but if you get party IDs so at odds with other renditions of reality, then you should suspect something’s wrong with your techniques.
What could be wrong? There may be some mischief hidden in Gallup’s “likely voter” filters – a series of questions about voting history, knowledge of the polling place, and intention designed to strip out the stay-at-homes from the polling results. (Other polls report the preferences of registered voters or even all adults.) While it may be reasonable to narrow the universe to likely voters in the days before an election, it may not be so sensible a month or two before, when people just haven’t firmed up their intentions yet. And it may not even be reasonable in the final pre-election polls: a review of the last four elections by Ruy Teixeira shows that Gallup’s estimates for all registered voters were actually more accurate than those for likely voters in three of the cases.
Likely voter filters may be especially inappropriate this year, with record registrations of new black voters, intense motivation among Bush haters to send him back to the hellish climes of Crawford, and uncommon interest expressed by young voters, who are often discarded in likely voter models. …
Sound familiar? It’s like déjà vu all over again! To get the links in the article and read the rest, including an accusation that the big cheeses at Gallup are “all Republicans,” with one even a donor to the flameout Senate candidacy of one Herman Cain — you can’t make this stuff up — click here.
And of course, not to be missed is Nate Silver’s great post on calculating pollster ‘house effects’. (Though I wonder if Silver should re-run his data In the wake of that very hinky Pew poll…)
And if you want to understand Gallup’s world, check out the demographic breakdown of their likely voter screen. It tells you a lot, because almost no one believes that we had a 78 percent voting base in 2008, or that we will again.
UPDATE: Sully weighs in.