In the next New York Times/Siena poll, we’ll introduce the most significant methodological changes to our survey since it began a decade ago. All of the changes involve weighting, with the goal of making the survey more deeply representative of the population.
When we tried out these changes on Times/Siena polls from previous elections, they yielded modest but not miraculous improvements. Our polls from 2020, for instance, would have still overestimated Joe Biden by a wide margin (he won the election, but our polls had him winning by much more). Still, modest gains can be meaningful. If the changes had been in place in 2024, when the polls didn’t have a bad year, the poll’s underestimate of Mr. Trump — nearly three points — would have been cut nearly in half.
The changes are significant, but they’re a natural extension of the longstanding philosophy of the Times/Siena poll. It tries to marry the “gold standard” survey sampling of traditional pollsters with the more sophisticated weighting and modeling techniques used in the world of campaigns and analytics. As a consequence, the poll has always been weighted across more categories than other public polls. In the last election, every Times/Siena poll was weighted by at least 10 categories where applicable: race, age, education, gender, party registration, region, history of voting, method of voting in 2020, marital status and home ownership.
But as impressive as that list may seem, there’s actually a lot of data we don’t use in weighting. We know whether respondents are donors to a campaign, whether their neighborhoods are blue or red, whether they’re licensed doctors or lawyers, whether their spouses are Democrats or Republicans and more, based on voter file records. Nonetheless, we use none of this information in the typical Times/Siena poll: It’s hard or even impossible to weight a poll on more than a dozen or so categories, at least at typical sample sizes.
Even beyond the limited number of groups used in weighting, polls are also limited in their ability to weight on the intersections — or interactions — between different groups. Our polls, for instance, represent the right number of Democrats and the right number of white voters, but not necessarily the right number of white Democrats. While it’s possible to weight by interactions — we currently do so with race by education and age by gender — it’s challenging or impossible to do at scale.
