WEIRD SCIENCE
CNN: “CNN Anchor: You say it’s not a shakeup but you guys are down and it makes sense
Trump Spokesperson: Says who?
Anchor: Polls.
Trump Spokesperson: Says who?
Anchor: Most of them. All of them?”
Sparring over election polls is an American pastime.
CNN: Trump Spokesperson: Says who?
Anchor: Polls. I just told you. I answered your question.
Trump Spokesperson: Ok.
Sharyl Attkisson: Are we being fooled by polls?
John Johnson: There are people that are trying to purposefully mislead us, then there are times where people just don’t know better when they read part of the story.
John Johnson has a PhD in data from MIT. You could say he’s an authority on the Weird Science of polling. He says there’s good reason to be skeptical.
Sharyl Attkisson: “So someone like you, when you read it reported in the media that a poll says X is ahead by X points. What’s the first thing you think of?”
John Johnson: “I question it. I look to see what the sample size was, how many people? I look to see who they questioned and I look to see if we, actually if there’s a track record for that poll. Those are the three things I think about first.”
Sharyl Attkisson: “Tell us about the polls that intentionally mislead. How do they do that and what is their aim?”
John Johnson: “The broad category of polls are called push polls, and so what a push poll is is literally where you’re masquerading as a pollster from an independent organization, but you actually represent a party or political interest and you’re going to shape questions that are meant to influence the results.”
John Johnson: “There was allegations of push polling in Nevada, where, right before the primary calls to Bernie Sanders supporters, sort of framed in terms of, ‘Well do you realize that Bernie Sanders doesn’t actually support Obamacare, or this,’ or these kind of hot button issues, and then asking ‘So now, would you still support Hillary Clinton?’”
Phone Audio: “Hillary Clinton says Bernie Sanders is making big campaign promises that will cost up to 20 trillion dollars and as the Washington Post says, quote, realistically his plans are dead on arrival in Congress.”
John Johnson: “Now, both parties are equally guilty and there’s allegations of this, but that’s what a push poll is. So that’s just unethical.”
Sanders accused Clinton of being behind the Nevada phone campaign, but her defenders denied it met the true definition of a “push poll.”
Another nonscientific poll that’s popular is the online instant poll that lets anyone vote; sometimes more than once.
Donald Trump cited his ‘victories’ in a number of those polls following his first debate with Hillary Clinton.
Sharyl Attkisson: “When we see what we think are the reputable national polls, you say there are issues with those too?”
John Johnson: “Yeah there definitely can be. So first, I always think about are they asking questions in a way that might be misleading or might be shaping the debate.”
Sharyl Attkisson: “What are some examples?”
John Johnson: “And so for example, if you have a very long and complicated set-up question, you know, ‘Donald Trump’s position on Russia has been to show favor for Vladimir Putin’ dot dot dot. And then you ask a question about foreign policy and Trump, you’re kinda guiding it. Or, ‘Hillary Clinton’s email scandal continues to probe her or to haunt her. How do you think about that?’ Either of those type of things could guide the answer.”
MSNBC: “Mika Brzezinski: The latest NBC news SurveyMonkey online poll shows Hillary Clinton improving her lead on Donald Trump nationwide.
She now leads 50-45”
Joe Scarborough: “She’s up 5 points.”
Sharyl Attkisson: “I looked at a poll recently that had Hillary Clinton up by a couple of points, but when I looked at the sample, they interviewed 50% more Democrats, so suddenly, it didn’t look all that positive for her to have interviewed 50% more Democrats and she really wasn’t winning by very much.”
John Johnson: “At the end of the day, every poll you see has some underlying model of turnout, and when you described, well Clinton’s up by six but there’s a lot more Democrats in that poll, or you might see a different poll where Trump’s up by one but there’s a lot more Republicans in that poll or they’re weighted more, that’s what’s driving the results.”
Sharyl Attkisson: “For people who think that maybe a pollster gathers a hundred opinions and randomly publishes the results, there’s a lot more to it.”
John Johnson: “It is not the case that we simply go and pick a hundred people at random and they all answer the question and we tell you, ‘Oh out of the hundred, 50 voted this way, 45 voted this way, five voted this way, five are undecided.’”
Sharyl Attkisson: “Historically, is there a poll or an average that’s been right more often than the others?”
John Johnson: “Well 538 has sort of credit for sort of being the one that’s called most of the states correctly.”
FiveThirtyEight: A CNN Poll showed that people trust Trump more on terrorism, but that they trust Clinton more on foreign policy.
John Johnson: “But 538 does something completely different. Again, they’re an aggregator they’re not a pollster. So they take all the polls and based on those statistical precision make some predictions.”
FiveThirtyEight: “People whose top issue is terrorism tend to prefer Trump.”
John Johnson: “Reality is you and I could sit here and go through the 50 states, and we probably together, not knowing a lot, could tell you how 40 of the 50 are gonna work out, right? At the end of the day, the election is going to hinge at about 10 states. So then it’s, are there certain polls within those states that are particularly useful? Monmouth has a very good reputation, for example, that’s one. Now there’s this advent of a lot more Internet polling, Reuters Ipsos, YouGov, The Economist are doing some more daily Internet polls. So there’s different choices at least, and I always say, Well let’s look at it as totality, but at the end of the day, I’m telling you it’s a handful of states where it’s gonna matter: Ohio, Pennsylvania, Florida. Those are the first polls I look at.”
CNN: “The leave vote leading the remain vote in the UK, we are getting a Brexit and financial markets around the globe are in panic mode.”
Sharyl Attkisson: “Do any polling disasters stick out in your mind in recent times?”
John Johnson: “Brexit is a great example because that was really driven in part by difference between Internet results and phone results and averaging polls together.”
CNN: “The total number of votes cast in favor of leave was 82,000.”
Sharyl Attkisson: “What was the bottom line?”
John Johnson: “They weren’t representative enough of what the voters were thinking and also, they were too confident that people wouldn’t change their mind because a lot of people made a decision at the last minute.”
CNN/Nigel Farage: “Dawn is breaking on an independent United Kingdom
The UK has voted to leave the European Union.”
John Johnson: “The great example, and this is not recent, is always Truman, but that was sort of a situation where they stopped polling three weeks before the election, right? And that showed, Truman did sort of a bunch of whistle stop tours and really built up support.”
Sharyl Attkisson: “Are there any best polls and worst polls or organizations that have proven their accuracy or lack thereof over time?”
John Johnson: “ABC News Washington Post has done well historically although this year there has been sort of a Democratic lean to those. Usually the ones that do poorly don’t survive. There are as I said some notable examples of pollsters that have been a little bit umm dishonest, and usually get out of the way. There was one story I had read about 30% of Republican voters are in favor of bombing Agrabah. Agrabah is the imaginary town from Aladdin.”
Buzzfeed: “Public Policy Polling found 30% of 532 people who identified as Republican would bomb Agrabah and only 132% say nope, let’s not bomb the cartoon city.”
Sharyl Attkisson: “A comment on the LA Times polling unique way where they’re conducting the, by interviewing the same people. What are they doing that’s different and what does that tell you?”
John Johnson: “They literally take a same set of people, it’s called a panel, the exact same people and they interview them every day to try to measure the trends. Who’s changing, who’s changing their mind or not? The issue has been in part when they started, their poll sample, that integral group skewed a little bit more towards Trump. So frequently, well once you’ve said that you’re gonna interview the same people, that’s going to be inherent in every single poll. The day that that poll came out that Clinton was ahead, I said, ‘Wow that’s a really interesting data point,’ because there’s a group that has been predetermined sort of seems to be skewing Republican, now it’s sort of shifted back with the tightening of the race again and now Trump tends to be up.”
Sharyl Attkisson: “Would you go so far as to say those big headlines on the national polls are relatively meaningless without that extra data?”
John Johnson: “It’s inherently complicated thing we’re trying to do.
The ultimate poll is the election. So, I’ve got a hundred people I’m gonna survey, I’ve got a thousand people, it’s actually remarkable that statistics can even get it close. But as a result, that’s a pretty hard problem to solve. You’ve gotta treat it as what it is, it’s a pretty nuanced problem. Think hard about the polls.”
