As we all know, polls in 2016 and 2020 were off to varying degrees and nearly always favored Democrats. The same was also true in 2018 if you focus on the battleground races where the contests were actually close. Pollsters have been embarrassed by this enough that they are trying to find some way to understand why it’s happening and also to correct for it.

The general consensus has been that some Trump voters don’t respond to pollsters, i.e. they literally don’t pick up the phone or hang up on them. And because Republicans seem more prone to do this than Democrats you can wind up with polls that don’t reflect the actual views of voters. In order to try to compensate for these “shy” voters, the NY Times tried something different. It offered to pay voters to complete a poll.

Poll questions were sent by mail, each containing a $5 bill and a promise to pay another $20 if the survey was completed and returned. The result was that instead of a 1% response rate that the Times gets on a normal phone poll, they got a 30% response rate. They did a phone poll at the same time so they could then compare and contrast the results.

Sure enough the mail poll did show slight gains for the GOP, though the differences were pretty slight. The left column is the mail poll results while the right column is the standard phone poll. All of this was done back in September so these results are pretty out of date by now.

As you can see, there is a difference but it’s not a very big one. So what does this mean? Nate Cohn says there are two ways to take these results:

On the one hand, the small to modest differences suggest that the Times/Siena poll and other low-incentive surveys remain competitive with a high-incentive survey with vastly higher response rates. On the other, it might be interpreted to mean that the “hidden Trump” vote remains out of reach — that $25 can’t reach a far more representative sample. (Of course, we won’t know which of these is true until the election is over.)

The good news is that we’ll have an idea how accurate the polls were this time around as soon as tomorrow. Meanwhile, Cohn did see some differences among the people who responded to the mail poll.

The share of mail respondents who said they were “moderate” was more than 12 points higher in the mail survey than in the Times/Siena poll, with both liberals and conservatives representing a smaller share of the telephone sample.

Similarly, the share of people who said they were absolutely certain to vote in November was higher in the telephone poll than in the mail survey. There are countless such examples in the data, from how often they take surveys to how closely they follow the news.
Another major difference between the two surveys was how much respondents appeared to like engaging and interacting with strangers.

The people who took the telephone survey, for instance, were likelier to say they wanted to have a job that involved working with people, as opposed to working with their hands or on a computer.

And the voters who took the mail survey were likelier to own or say they considered getting a “no trespassing” or “no solicitation” sign. The respondents to the typical telephone survey were 10 percentage points likelier to say they hadn’t considered putting up such a sign; Republicans had a commanding lead among those who had.

Perhaps relatedly, the survey found a wide gap on preferences about immigration: 47 percent of respondents to the mail survey said undocumented immigrants should be deported back to their home country, compared with 38 percent in the Times/Siena poll.

So we won’t really know if either the phone poll or the paid poll were accurate until tonight or tomorrow. But there is some indication that even the process of talking to someone weeds out a lot of people or causes them to downplay what they see as controversial opinions (like favoring deportation). To me this suggests there is probably something to the shy voter idea. After national Democrats have spent years accusing the GOP of being everything from deplorables to democracy-enders, it doesn’t surprise me that a lot of conservatives aren’t inclined to pick up the phone and talk to a pollster.