Mapping polling errors in US elections, 2016-22

Data

The data comes from FiveThirtyEight’s public data used for generating their pollster grades. I make no use of their pollster grades, but they conveniently include all late-cycle polls (i.e., within 3 weeks of election day) along with pre-calculated error and bias for each poll (so I don’t have to look up the results for every single race).

I do not use the data from primaries because they cannot have a partisan bias and generally have so much more polling error that it would swamp everything else. Combine that with the greatly differing methods for doing primaries (e.g., Iowa’s caucus system) and it just didn’t seem right to include even if just for the polling error part.

I did not include statewide races for the House, like in Montana, Wyoming, and Alaska. I may change that in the future. I also do not include Presidential results for the congressional districts in Nebraska and Maine that cast their votes according to their district vote rather than the statewide vote. I’m open to it, but can’t find a shapefile that has the 50 states and cutouts for those districts. 😅

The model

I’m a one-trick pony, so whether it’s calculating the value of field goal kickers or mapping polling errors, I’m reaching for a multilevel model. There are some serious advantages in this case. The data are grouped in a few ways:

• By state
• By pollster
• By election type (President/Senate/Governor)
• By year of election

Each of these has potential use as a grouping variable in a multilevel model. This is especially true because many of the pollsters and states won’t contribute much data so we need a principled way to use their data without overfitting. And multilevel models let me make adjustments for other things like pollster partisanship, time until the election, and sample size. (Note: I considered adding adjustments by method but there are so many thanks to pollsters combining many at once. It was a real mess and not adding much. I also considered using the election type, as I did for the previous version of this post, but it added a lot of model complexity for minimal gain and produced some odd estimates.)

With so many “groups,” many of which have little data associated, and a desire to produce useful preditions, the obvious option (for me, a one-trick pony) was Bayesian estimation via brms. I love this R package. With just a few lines of code I was able to fit a relatively complex model that includes multiple dependent variables (error and bias) that allows the residuals of those two dependent variables to correlate, as they should given the fact they are derived from the same information. With a few more lines of code, I could generate state- and year-level predictions via some of the conveniences in my jtools package.

Seriously, see below how brief the most intellectually challenging steps are.

Model fit:

brm(
  mvbf(
    # Bias model
    bias ~ partisan + log(samplesize) + days_till_election + 
             (1 | pollster_rating_id/year) + (1 | location/year) +
             (partisan | year),
    # Error model
    error ~ partisan + log(samplesize) + days_till_election + 
             (1 | pollster_rating_id/year) + (1 | location/year) +
             (partisan | year)
  ),
  data = d, iter = 3000, chains = 2, 
  control = list(adapt_delta = 0.9, max_treedepth = 10)
)

Making predictions used in the maps:

make_predictions(fit, "year", 
                 at = list(location = unique(d$location), 
                           days_till_election = 0, samplesize = 800), 
                 re.form = ~ (1 | location/year) + (partisan | year),
                 resp = "error")

There’s a good deal more code in the full R Markdown document used to create this post (which can be found in my website’s Github repository), but it’s mostly just data cleaning and futzing around with Plotly to make the maps.