Mapping polling errors in US elections, 2016-20

Data

The data comes from FiveThirtyEight’s public data used for generating their pollster grades. I make no use of their pollster grades, but they conveniently include all late-cycle polls (i.e., within 3 weeks of election day) along with pre-calculated error and bias for each poll (so I don’t have to look up the results for every single race).

I do not use the data from primaries because they cannot have a partisan bias and generally have so much more polling error that it would swamp everything else. Combine that with the greatly differing methods for doing primaries (e.g., Iowa’s caucus system) and it just didn’t seem right to include even if just for the polling error part.

I did not include statewide races for the House, like in Montana, Wyoming, and Alaska. I may change that in the future. I also do not include Presidential results for the congressional districts in Nebraska and Maine that cast their votes according to their district vote rather than the statewide vote. I’m open to it, but can’t find a shapefile that has the 50 states and cutouts for those districts. 😅

The model

I’m a one-trick pony, so whether it’s (calculating the value of field goal kickers)[/post/evaluating-kickers] or mapping polling errors, I’m reaching for a multilevel model. There are some serious advantages in this case. The data are grouped in a few ways:

• By state
• By pollster
• By election type (President/Senate/Governor)
• By year of election

Now there aren’t actually enough separate election years to accomplish much with treating it as a leveling factor in the multilevel model, but the others do have some value. This is especially true because many of the pollsters and states won’t contribute much data so we need a principled way to use their data without overfitting. And multilevel models let me make adjustments for other things like pollster partisanship, time until the election, and sample size. (Note: I considered adding adjustments by method but there are so many thanks to pollsters combining many at once. It was a real mess and not adding much.)

With so many “groups,” many of which have little data associated, and a desire to produce useful preditions, the obvious option (for me, a one-trick pony) was Bayesian estimation via brms. I love this R package. With just a few lines of code I was able to fit a relatively complex model that includes multiple dependent variables (error and bias) that allows the residuals of those two dependent variables to correlate. With a few more lines of code, I could generate state- and year-level predictions via some of the conveniences in my jtools package.

Seriously, see below how brief the most intellectually challenging steps are.

Model fit:

brm(mvbf(
  bias ~ year + partisan + log(samplesize) + days_till_election +
  (year | pollster_rating_id) + (year | location/type_simple), # first DV
  
  error ~ year + partisan + log(samplesize) + days_till_election +
  (year | pollster_rating_id) + (year | location/type_simple)), # second DV
  data = d, iter = 4000
)

Making predictions used in the maps:

make_predictions(fit, pred = "year", 
                 at = list(location = unique(d$location), days_till_election = 0, 
                           samplesize = 800),
                 re.form = ~ (year | location), resp = "bias")

There’s a good deal more code in the full R Markdown document used to create this post (which can be found in my website’s Github repository), but it’s mostly just data cleaning and futzing around with Plotly to make the maps.