I’ll admit that when I saw the Iowa polls showing Mayor Pete with the highest percentage of all the top candidates at 25% in a recent Change Research poll, I knew I had to look at the cross tabs because no one would believe it. I’m not sure I even believed it, even though I do believe his support has been growing exponentially there and elsewhere.

First, I wanted to learn a but about the methodology of the polling firm. Looking at Five Thirty Eight’s ratings, the first thing of note is that it is an online collection poll. In theory it is just harder to verify who is completing the poll when it is conducted online vs by phone. However, as I have been reading through Five Thirty Eight’s pollster ratings, it appears they have not updated the ratings based on 2018 polls & races which is one area Change Research has proposed to have had a great deal of success.

Next, I went to Change Research’s website to see what their methodology really is like. Here is a synopsis on their website:

Change Research is a leader in polling innovation, allowing us to field polls quickly and accurately at a low price point. We recruit fresh participants online for each and every poll, meaning we aren’t affected by the dwindling response rates of landline polling. We use proprietary, patent-pending approaches to recruit participants, and we don’t use online panels of habitual survey takers.​

We collect survey responses by publishing targeted online solicitations via advertisements on websites and social media platforms. By finding a representative set of web and social media users to take a poll, we are able to cast a net that is wider than landlines. We reach twenty-somethings and seniors, rural and urban dwellers, and members of every gender, race, creed, and political persuasion.  

This is a review of their 2018 polls and success and further discusses their methodology. Here is a note they wrote about comparing to live phone polling:

Comparison to Phone Polling

We were excited to see the NY Times conduct dozens of public polls this year to shed light on polling methodologies. As all of their polls were conducted by phone and with the voter file, they serve as a good proxy for this methodology. In total, there were 16 elections (US House, US Senate, and Governor) that were polled both by the NY Times and by Change Research, 11 of them in the 3 weeks prior to Election Day.

For these surveys, Change Research had a mean absolute error of 3.6 percentage points. In these races, Change Research was more accurate than the NY Times, which had a 3.7 point mean absolute error. Change Research was 1% more accurate (2.5% vs. 3.5%) on polls conducted in the last three weeks.

This note indicates that perhaps they cross reference their poll takers with the voter file to confirm voter status, or at least start with voter data and use the available information to target those voters online instead of calling them.

Targeting the right respondents

Change Research uses a combination of proprietary geo-targeting technology and voter file data to invite respondents in a specific district, be it a district for city council or U.S. Congress.

Now, for the cross tabs! What was really in this poll? It looks like they polled approximately 400 likely voters in Iowa, New Hampshire and South Carolina but not nationwide. They polled Democrats and some independents. For this analysis I am only going to focus on the IA numbers so we can see if there were indeed trends that would help explain the poll.

Overall, I think the numbers that support the 25% would vote for percentage, are the “Would make best President” at 27% of Iowans in this poll, “Looked most presidential” at 28% of Iowans in this poll, and the fact that the largest number by far who gave to a candidate after the debate in Iowa, gave to Mayor Pete.

While of course, it’s just one poll, I think there is substantiated data to support the poll results. They may have just found a nice pocket of Pete supporters. But If you are going to believe the low percentage polls, then you should take a close look at the higher percentage polls too.