|
נייט סילבר שלח מייל בדיוק לגבי האמור מעלה ולמה (הדמוקרטים) לא צריכים לדאוג (אם כי לא קראתי, רק רפרתי בקושי)
Forwarded this email? Subscribe here for more
Why you should mostly ignore "internal polls"
Unless you’re very careful, they add more noise than signal.
Nate Silver
Oct 10
READ IN APP
I wasn’t planning on a newsletter today, but after a series of news accounts suggesting that Democrats are growing increasingly concerned about the state of the race, there seems to be a great deal of demand for this topic. So let’s dig in. You might consider it the third installment (part one, part two) in Should Democrats Be Freaking Out Week here at Silver Bulletin. If you appreciate this bonus newsletter, please consider becoming a subscriber.
Upgrade to paid
Most political observers find the idea of having access to inside information irresistible. Humans are fascinated by gossip. And reporters prize scoops. So the premise of much political coverage is that it promises to take you “Behind the Curtain” — that’s the title of the Axios column I linked to above.
So as long as I’ve been covering politics, people have been fascinated by internal polls (surveys that the campaigns themselves or closely affiliated partisan groups conduct). Especially given the high-profile public polling misses in recent years, surely the campaigns know the real story of the race?
Well, no. Or at least, not necessarily. If you really knew what the campaigns knew, that would probably provide valuable information. But unless you’re an Iranian or Russian hacker, you probably don’t. Instead, you know what the campaigns want you to know.
Although some people in every field are oversharers¹, for the most part campaign professionals are highly strategic in what information they disclose. You’re basically being spun, especially if you’re a friendly source who can be counted upon to pass along the spin uncritically. Internal polls are rarely made up outright, but they’re no different in this regard, strategically shared and designed to shape a narrative rather than to provide accurate information.
Indeed, recent elections have shown that you should be highly skeptical of reports about internal polls. Furthermore, as in a game of telephone, information fidelity degrades as it is further removed from the original source.
Level 1: The genuine article
Suppose that you’ve bugged both the Harris and Trump campaign HQs. You know every internal poll they’ve conducted all campaign long. How much of an advantage would this provide you versus public polling averages?
Probably some. Campaigns have access to proprietary information that public pollsters lack, like data mining they’ve done based on campaign contacts. And in general, the money in the polling industry is in the campaign side of the business, so they may be getting a higher grade of pollster talent. But there are also reasons to be wary. Sometimes the campaigns are just fooling themselves.
One issue is that campaigns have different incentives for conducting election polls than media or nonpartisan organizations. Rather than seeking to predict the outcome, they’re trying to make strategic decisions, usually about where to allocate resources or how to adopt more effective messaging.
For resource allocation choices — should we run more ads in North Carolina next week or in Wisconsin instead? — having a more accurate lay of the land undoubtedly helps. But one interesting property of these decisions is that precision matters more than accuracy. Because electoral shifts tend to follow a uniform swing, your polls can be biased so long as they’re biased in a consistent way. For instance, if all of the Harris campaign’s internal polls were biased by 5 points toward Harris, they’d still campaign in pretty much the same states if they wanted to maximize their chances of winning the Electoral College. Here’s some proof of that. When we turned the Biden-Trump forecast off after Biden exited the race, Trump led by 4 points in our national polling average. Now, Harris leads Trump by 3 points nationally. But the tipping-point states are almost exactly the same, with Pennsylvania ranked #1 with a bullet, then some combination of Michigan, Wisconsin, Georgia and North Carolina, then Arizona and Nevada trailing a little behind.
As for message testing, campaigns don’t care about the topline numbers at all. Instead, they care about directional effects: will Kamala Harris bragging about how she owns a gun sway more marginal voters than it detracts? If you want message refinement, trust the message testers; however, if you want accurate predictions, trust the people whose job it is to make predictions.
A second issue is that campaigns face principal-agent problems. Can the boss and her inner circle handle bad news? Most organizations have an optimism bias: for instance, internal estimates of project completion timelines are usually too rosy. And as I discuss in my book, collegiality (keeping the boss and other principals happy) matters a lot in Campaignland in particular. There isn’t a lot of objective feedback to tell you who’s good at their job and who isn’t when you’re only conducting one campaign every two or four years, it’s staffed by hundreds of professionals, and the result is often based on factors outside of the campaign’s control. It’s not a place for rogue contrarians who will stick to the data no matter what and deliver bad news.
Share
Third, campaigns mostly attract true believers, people who think their party is on the righteous path and who process information through a partisan filter. This year’s Biden campaign apparently sincerely believed that they would win, for instance — in some cases even after Biden’s disastrous debate.
So you’re selecting for people who are high on partisan motivated reasoning, who have organizational incentives to sugarcoat the data, and who won’t necessarily even harm their campaigns by having biased polling because campaigns aren’t really trying to make accurate predictions in the first place. There’s a host of literature on how it’s often better to have an unbiased “outside view” like the one public polls provide rather than an “inside view” colored by bias.
These aren’t just theoretical concerns. In 2012, a reporter for the New Republic got access to the Mitt Romney campaign internals after the campaign was over. The polls were biased toward Romney by an average of almost 5 points.²
Internal polls misled Romney in 2012
Level 2: Internal polls selectively shared with the media
On certain occasions, campaigns choose to share their numbers publicly. Sometimes, they go through the pretense that the data was “leaked” since this increases the air of mystery or they’ll have some tactical reason to want plausible deniability. But more often this is done in a relatively transparent way, on campaign letterhead with a memo spinning discussing the results.
When campaigns disclose the basic descriptive information that polling aggregators require — field dates, sample sizes, and so on — these surveys are included in polling databases, including Silver Bulletin’s. However, these polls need to be debiased. How come? Since these polls enter the public record, we can empirically track how biased they’ve been over time. (You can even do this yourself from publicly available Silver Bulletin data.) Specifically, presidential internal polls are biased by an average of about 3 points toward their candidates and a bit larger than that for Congressional and downballot races. The most infamous case is from the Republican primary in Virginia’s 7th Congressional District in 2014, in which House Majority Leader Eric Cantor lost to David Brat by 11 (!) points after a publicly released internal poll had shown Cantor ahead by 34 (!!). Cantor’s pollster? None other than John McLaughlin, who is now one of the Trump campaign’s chief pollsters.
Our polling averages account for this tendency through our house effects adjustment. For instance, the Trump campaign released a series of internal polls on Thursday. Here are the raw numbers, then how they entered the model after the adjustments we make:
How the model handles internal polls
Trump campaign internals released on Oct. 10 before and after our house effects and trendline adjustments
The magnitude of the adjustment differs slightly from state to state based on how much the Trump poll differed from public polls there, but that’s not particularly important. What matters more is that the adjusted numbers almost exactly match public polling averages in each state. That’s a sign that the bias of publicly released internal polls is highly predictable. Usually, campaigns won’t go Full McLaughlin; they’ll want to publish something within the range of credibility. And by the way, the Trump numbers are credible. With even a small systematic polling error in his favor, like in 2016, Trump could sweep every swing state and wind up with a map exactly like this. But they’re shaded in their client’s direction, as publicly-released internals nearly always are.
I also don’t doubt that these are real numbers. Internal polls may have house effects because of the factors I described in the first section. (If I were Trump’s pollster, I wouldn’t want to take a meeting with him with polls showing him losing North Carolina.) However, campaigns are also selective in when they share their data with the public. You’re in the field every week, and some weeks will look a little better than others just because of sampling error — polls can be noisy. Or campaigns may run different versions of the numbers, for instance, with different assumptions about likely voters. If one batch happens to be particularly favorable one week, you share it with Politico — otherwise, you don’t.
Level 3: Conventional wisdom based on supposed knowledge of internal polls
I suppose we could make some finer distinctions here:
Level 3.1: Journalists reporting on internal polls (without citing numbers directly) or the mood of the campaigns generally;
Level 3.2: Well-connected elites citing internal polls or sources within the campaigns;
Level 3.3: Randos on Twitter claiming to have seen internal polls.
In some instances, Level 3.1 can provide useful information. You need to know the journalists — how gullible they are and/or how likely they are to pass along spin uncritically, even if they recognize it as spin — and you usually need to do a bit of decoding. In the Axios report from today, for instance, the main named Democratic source is David Axelrod, Obama’s former chief strategist, who has often been critical this cycle of how Biden and Harris have run their campaigns:
David Axelrod, the operative behind Barack Obama's victory in 2008, says Harris "made steady, incremental progress in the 10 days after the [Sept. 10] debate, but now the race has plateaued."
Axelrod adds that Harris "had a great launch, right through the convention and the debate. But in these campaigns, every time you clear a bar, the bar gets raised. You have to lift your game and adjust your strategy."
Emphasis mine. Axelrod’s account is entirely consistent with publicly available polling averages, like those at Silver Bulletin! Harris gained ground in the ten days or so after the debate, and her numbers have since plateaued to a position that is improved but not really enough to give her a lead in the Electoral College. Here’s our data in Pennsylvania, for example:
Who’s ahead in the Pennsylvania polls?
An updating average of 2024 presidential general election polls, accounting for each poll’s quality, sample size and recency. Click the buttons to see the polling average in different contests Nat. AZ FL GA MI MN NV NH NC PA TX VA WI
Meanwhile, Axios’s unnamed sources say the race is a toss-up — which is also exactly what the public models say:
Reality check: Considering both polls' margins of error, it's all a big statistical tie — which top officials in both parties tell us is the true state of the race.
So the actual reporting in the Axios column is that both campaigns think the election is a toss-up. Well, no shit. And maybe Democrats are feeling more angsty about the race, but that’s because Democrats always feel angsty. Axelrod feels more at ease to lightly criticize the Harris campaign — maybe you should mix it up a bit, he’s saying — but that isn’t really news either, because Democrats have a healthier culture for handling internal dissent than Trumpworld does.
But Axios has an incentive to put a little spin on the ball — Democrats are panicking! — because that makes for a more interesting story. “Both campaigns think it’s too close to call” isn’t a good headline.
And then by the time we reach the next link in the game of telephone, Level 3.2, you’re getting the sort of people who read Axios, but don’t know how to parse a news story in this way. They’ll read the headline, and then talk to their friends who read the same headline, and before long, it will seem like all the clued-in people are thinking the same thing. There’s a feedback loop and the consensus is amplified.
The thing is, I know a lot of well-connected elites — not really in politics, actually, but in fields like finance and to some extent tech that are covered by the book. They tend to be highly attuned to social signals within their circles. When it comes to something like picking up on a hot new venture investment, that can be highly useful.
But in fields they don’t know particularly well, like politics, they’ll often uncritically repeat what their friends say. Some of the guys in my Wall Street poker game were absolutely convinced that Biden would be replaced on the ticket — but by Gavin Newsom, a rumor that never made much sense. Often, people like this will have some legitimate campaign sources, too, especially if they’re major donors. But they’re usually being flattered by these sources, and aren’t as skeptical of them as a reporter might be.
In some campaigns, this cascade can continue to the point where conventional wisdom about the race can become completely detached from objective reality. That was the case in 2022 with the “red wave” narrative, for instance — or in 2016, when insiders were convinced there was no way Hillary Clinton could lose when well-constructed models showed otherwise.
So long the data, and short the vibes.
Silver Bulletin is a reader-supported publication. To receive new posts and full access to the election model, consider becoming a subscriber.
Upgrade to paid
1
It can be hard to keep secrets, and reporters have a knack for extracting information from people that’s against their self-interest.
2
The public polls were biased toward Romney too in 2012, but not to the same extent.
_________________
״לי יש דרכון של מדינה מתקדמת ובעוד x שנים אני לא כאן.״ (נכתב בנובמבר 2022, על ידי יוזר שאיננו אני).
|
התחבר
הרשמה
בלוגים
שאלות נפוצות
חיפוש
