Mapping polling errors in US elections, 2016-20

Nov 6, 2022 9 min read politics, Data science, Academic

This post is from 2022, before the midterm elections. I have an updated post that incorporates the 2022 midterms, uses slightly better models, and has generally better fine-tuning. I’d recommend checking that out if you’re interested in the subject matter on this page.

If you follow US politics at all, you’ve probably heard a few things about polling in the past few election cycles:

  • They have been “wrong”
  • They have been biased towards Democrats
  • They have been worse in some places than others

And there’s some truth to all of this! But I think even for people who follow this stuff pretty closely, it can be hard to remember all the details. Some folks like to look at current polls and poll averages and make a mental adjustment to the results and see how they’d look if the polls missed the way they did in the recent past. My goal here is to help with that kind of thing.

Below are three types of maps:

  1. Average polling error for each state in the past 3 elections
  2. Average polling bias for each in the past 3 elections
  3. The 2022 Senate election polling averages

Some states seem to be easier to poll than others. The first map is meant to highlight the ones where polls are generally more on the mark, regardless of whether they tilt towards one or the other party. It goes without saying that high levels of bias also cause error, but some places may have little bias but the typical poll still misses by a good deal.

The second map shows polling bias instead. Here we are looking at how much the average polling margin tends to miss towards one party or the other compared to the actual results. I’ve shaded states blue for pro-Democrat bias (meaning the polling average overestimates the Democrats) and red for pro-Republican bias. Lighter shades mean the bias is closer to zero.

Finally, the third map lets you take the current (as of 11/6/22) FiveThirtyEight polling averages, which is shown in the default view, and see how the margins would change if you applied the levels of bias seen in that state in past elections. Let’s just say the Democrats will be rooting for a 2018 miss and the Republicans will probably be happy with anything else.

The polling data used to calculate bias and error is for President, Senate, and Governor only. No other statewide races, ballot issues, or Congressional races are used here due to data availability and other issues.

Hover over states (or tap on mobile) to see the exact number.

Important note about methods

These numbers are not derived from simply averaging all public polls and comparing them to the results. Instead, I’m using a statistical model to adjust for the quality and types of polls conducted in each race. So the estimates are ultimately quite similar to what you’d get if you compared the projected vote share from FiveThirtyEight (which is also based on polls with adjustments for pollster quality, partisanship, and some other things) to the actual votes.

See the “How this works” section for more info.

Polling Error