In statistics, the more one repeats something, the less likely chance there is that one will end up with really unusual data.
For instance, flipping a coin only twice and having both flips land ‘heads’ would yield the not entierly unusual result of a 100% occurance of ‘heads’ versus ‘tails’. If, however, like Rosencrantz, you filp a coin 92 times and you end up with the same 100% result, you can rest assured that either something stupendously unlikely has happened or that maybe something ‘untowards’ has happened.
Now take polling: one would expect that the more polls one throws into the mix, the less likely it is that we will see a ‘Rosencrantzian’ result. Just for fun today (yes, I have a sick idea of “fun”), I took the polls from every polling firm conducted in all of 2009 and averaged them together.
I was, to say the least, surprised at what I found.
I have no solid explanation as to what could cause this kind of a discrepancy (at least none that would save me from the threat of litigation), however I invite my readers to critical consider for themselves which of the following two possible explanations is more likely:
- Occam’s razor is wrong and every single other reputable polling firm in the country (Ekos, Strategic, Angus-Reid, Environics, Decima, Nanos, Leger etc…) has flawed methodology while Ipsos Reid is the only firm with accurate methodology; or
- Occam’s razor — one of the most fundamental scientific precepts of the last millennium — is true after all and Ipsos Reid should explain themselves why their data so thoroughly contradicts everyone else’s.
Edit/Update:
Okay, some people have been asking some (legitimate) questions about my data and sources and I actually can’t believe that I forgot to include that information in this post.
First, the background:
There were five polls from Ipsos Reid and 24 polls from everybody else.
The MoE for five the Ipsos Reid polls (at an average of approximately 1000 respondents per poll) would be:
0.98/((1000*5)^(1/2))= approx. 1.39%.
The question was raised about how interrelated the other polls are and that’s a good question because of Bayes’ theorem.
I should have mentioned that the other 24 polls are not lop-sidedly produced by one polling firm or another. Each polling firm has its own methodology (some use random dialling – e.g. Decima – and some use lists – most others. Some prompt for party preference – e.g. Ekos – while others do not prompt – e.g. Nanos.. Et cetera)
The breakdown of polls from the other polling firms are:
Leger – 1 polls
Nanos – 5 polls
Decima – 5 polls
Angus-Reid – 5 polls
Ekos – 3 polls
Strategic – 5 polls
Ipsos – 5 polls
The sources of all polls conducted for this piece were each polling firm’s individual website. You can access an exhaustive list of all the 2009 polls (with links to said websites) here.
The following, for those interested, is a definitive list of all polls used for this piece.
Paulitics 
Look at who pays for the polls, nuff said.
Maybe it’s time to do what Nate Silver does over at http://www.fivethirtyeight.com/ and assign the pollsters different weights based on their historical accuracy. I doubt anyone can threaten litigation for comparing their track record to actual results, as opposed to other polls.
In between elections, Elections Canada posts the results of all of the public opinion polls released into the public domain.
They are all there–hundreds of them.
So, if you take all of the polls by all of the major firms and do some simple calculations, a couple things emerge based on fact:
First, the Conservative average vote percentage January 2008-June 2009 per polling company:
EKOS 37.5
Angus Reid 36.8
Ipsos Reid 36.2
Nanos 35.6
Strategic Counsel 35.2
Harris-Decima 34.8
The Mean: 36.02
This shows that regardless of the firm, every one of them produced the same results, on average, for the Conservative support. This should put to rest any argument that one firm or another is favouring the Conservatives by somehow jacking their support levels up. To state as such is simply and utterly untrue. To suggest that the methodology used by any of these firms in calculating such results is flawed infers that all of their methodology’s are flawed or all of them are not flawed. We prefer the latter view.
It also demonstrates that Ipsos Reid’s difference from the Mean is one fifth of one percentage point: 0.18
The second is the assertion that a firm (in this case, Ipsos Reid) is deliberately showing the Conservatives in the lead more often than others because of some bias or undue influence of the poll sponsor.
Here are the percentage of times the Conservatives are presented as leading the Liberals in polls released by polling firms during the time-frame examined:
EKOS 100%
SEGMA 100%
Ipsos Reid 96%
Strategic Counsel 91%
Nanos 90%
Harris-Decima 84%
The Mean is: 93.5%
The Ipsos Reid difference from the Mean: 2.5
This result based on fact shows that the second assertion is false and misleading as it applies to Ipsos Reid.
And if other firms (like Compas (at 100%) and others like Leger) were also added to the mix then the margin that Ipsos is from the Mean would be even less. The reason these firms were not included in the calculations was that they actually had so few polls released on Federal politics during the time-frame examined. Their results would have unduly skewed the results –actually to the benefit of Ipsos Reid.
Lastly, for some reason there is an alleged question presented that somehow infers that it was used to generate vote support numbers. In fact, it was part of a battery of agree/disagree statements that tested the veracity and impact of the rhetoric by the political leaders of the Federal parties at the time of the parliamentary crisis of December, 2008.
In conclusion, I think it’s safe to say that Ipsos Reid really won’t waste any time or effort on a legal suit because the fact actually trumps the conjecture easily–verdict: a slam dunk for Ipsos Reid.
And I didn’t bother to do the NDP % record because, yawn, it will probably produce similar results.
The author of the original false assertions need only fear being arrested and charged by the local police for being a moron–a much more serious offence because it’s obvious from these and previous raving’s that there is no chance of rehabilitation.
Best regards.
John Wright
Senior Vice President and Managing Director
Public Opinion Polling
Ipsos Reid
Mr. Wright: Why did Ipsos-Reid use the word “unholy” in a poll it conducted on the opinion of Canadians with respect to the coalition, back back in December, 2008?
The author of the original false assertions need only fear being arrested and charged by the local police for being a moron–a much more serious offence because it’s obvious from these and previous raving’s that there is no chance of rehabilitation.
Now that’s professional…
First, I wrote the question for the very reason noted above: it was an agree/disagree statement in an item bank to determine the veracity and traction of the PM’s statements. It was just one of many questions fully addressing all sides of the arguments being made at the time.
Second, what gives bloggers to make false and misleading assertions, and be able to use whatever language they want and yet when I reply in a mocking tone it is seen as “unprofessional”?
If you can’t take the heat, get out of the kitchen and be more respectful in the first place.
Wow. John Wright, the Senior Vice President and Managing Director of the Public Opinion Polling company Ipsos Reid is obviously a hypersensitive douchebag. I’m about as a far from socialist as you can imagine, but your response is beyond juvenile. Grow the fuck up.
“First, I wrote the question for the very reason noted above: it was an agree/disagree statement in an item bank to determine the veracity and traction of the PM’s statements.”
Indeed you did. Forgive me for not having read your response more carefully.
Veracity? No. Traction? Perhaps. I’d have to examine the entire battery of questions to comment further, but the events since the Parliamentary crisis have rendered that exercise rather pointless. The analyses since then have largely concluded that Canadians were not sufficiently knowledgeable about how their government works to have an opinion that is of substantive value to citizens (as opposed to political parties, the media and polling firms).
Second, what gives bloggers to make false and misleading assertions, and be able to use whatever language they want and yet when I reply in a mocking tone it is seen as “unprofessional”?
Because you are acting in your capacity as a professional. Bloggers and pseudonymous commenters most assuredly are not. More the point, it’s not really necessary. You can easily present your arguments without the emotional outbursts.
Really, it’s not that hard to understand
If you can’t take the heat, get out of the kitchen and be more respectful in the first place.
Or maybe it is.
Mr Wright, it’s great to see you back and commenting again here.
I’ve sorely missed you these last couple of years. Your last visit drove traffic up to my site by almost 10,000 hits.
You raise some important questions that deserve to be addressed seriously (irrespective of the manner in which they were delivered).
I have published Part I of my response to you. I hope to get the rest of my response published shortly as “Part II” and I hope to do so without having to spill over to a Part III.
You may read the first part of my response here:
Thanks for making my point.
So, what’s your e-mail? What’s your phone number? What’s your address??
What, you aren’t a coward are you???? Show us where you are…you have to have more than that “Paul Bernardo” lookalike picture, don’t you???
C’mon, fess up…
Please help me locate episode 3 by Michael Ponti in the Paulitics series. Thank you!
Ron