All in all, we have made progress over the last 4 years, but the work is never really done.

In 2024, political engagement was related to vote choice, though there’s nuance in how, based on which measures you use.

We also do a lot to ensure that we don’t get too many respondents who are highly politically engaged, across a few different measures. We use vote history and turnout scores from the file, as well as a question that asks how important politics is to your identity.

We also looked at repeat responders and found very little evidence of winners bias. Most repeat respondents don’t switch who they said they voted for. Of course, stability last cycle is no guarantee of stability in the future, so we’ll continue to monitor.

Another concern about vote recall weights is that people will forget who they voted for, or misrepresent who they voted for, in ways that bias our estimates. We track responses to recalled vote over time. From 2021 to 2024, 2020 vote recall was very stable.

In all three, Harris lags behind Biden 2020 while the Dem House candidate is ahead of Biden 2020. Much of that over-performance is driven by 2020 Trump voters supporting the Dem House candidate.

There are a number of concerns that folks have about vote recall weights. One is that vote choice will be identical (or close) to recalled vote. That’s only true when the electorate is stable and no one changes their mind or splits their ticket. For example, here’s 3 House polls :

Another tricky question is who to weight. We weight on self-reported vote and allocate refusals & 3rd party voters by PID. That reduces random variation in partisanship from survey to survey, so we can be more confident that movement we see on other metrics is real.

Using vote recall weights is tricky, because electorates change. With help from calibrated past vote scores, we weight to in-cycle adjusted targets rather than what literally happened in the past election. That improves accuracy and is particularly important in midterms.

We fundamentally see vote recall weights as a tool for addressing attitudinal non-response. It helps us ensure that our surveys have the right mix of partisans. Our methodological choices follow from that.

We check the usefulness of our vote recall weight by comparing vote choice with and without it. It improves both error and bias when applied to this sample of pre-election polls. As you can see in the report, that remains true when we correct for turnout error.

We evaluate turnout error by reweighting a sample of our polls to match the actual electorate, once we’ve got confirmed vote history on file. Error stays about the same, but bias goes down in this sample of polls.

We think of error as coming from 4 broad sources: non-response, turnout, late movement, and sample error. This framework isn’t comprehensive, and there are other good ones, but we find it useful for identifying the big things that can go wrong.

There’s no one trick or decision that is responsible for our cycle-to-cycle improvement. The report focuses on vote recall weights, but as any pollster can tell you, there are many decisions to make about sampling, weighting, and measurement. It’s a constant process of evaluation and reevaluation.

I want to highlight some findings from our report. First, our polls were very accurate this cycle, building on improvement we saw in 2022. Error was down and there’s an average bias of +0.6 in our late ‘24 cycle polls. That bias is largely due to slight optimism in our LV models.

Happy Holidays, everyone

Next up, Acq Inc! #paxunplugged

Just what I needed