In statistics, the more one repeats something, the less likely chance there is that one will end up with really unusual data.
For instance, flipping a coin only twice and having both flips land ‘heads’ would yield the not entierly unusual result of a 100% occurance of ‘heads’ versus ‘tails’. If, however, like Rosencrantz, you filp a coin 92 times and you end up with the same 100% result, you can rest assured that either something stupendously unlikely has happened or that maybe something ‘untowards’ has happened.
Now take polling: one would expect that the more polls one throws into the mix, the less likely it is that we will see a ‘Rosencrantzian’ result. Just for fun today (yes, I have a sick idea of “fun”), I took the polls from every polling firm conducted in all of 2009 and averaged them together.
I was, to say the least, surprised at what I found.
I have no solid explanation as to what could cause this kind of a discrepancy (at least none that would save me from the threat of litigation), however I invite my readers to critical consider for themselves which of the following two possible explanations is more likely:
- Occam’s razor is wrong and every single other reputable polling firm in the country (Ekos, Strategic, Angus-Reid, Environics, Decima, Nanos, Leger etc…) has flawed methodology while Ipsos Reid is the only firm with accurate methodology; or
- Occam’s razor — one of the most fundamental scientific precepts of the last millennium — is true after all and Ipsos Reid should explain themselves why their data so thoroughly contradicts everyone else’s.
Okay, some people have been asking some (legitimate) questions about my data and sources and I actually can’t believe that I forgot to include that information in this post.
First, the background:
There were five polls from Ipsos Reid and 24 polls from everybody else.
The MoE for five the Ipsos Reid polls (at an average of approximately 1000 respondents per poll) would be:
0.98/((1000*5)^(1/2))= approx. 1.39%.
The question was raised about how interrelated the other polls are and that’s a good question because of Bayes’ theorem.
I should have mentioned that the other 24 polls are not lop-sidedly produced by one polling firm or another. Each polling firm has its own methodology (some use random dialling – e.g. Decima – and some use lists – most others. Some prompt for party preference – e.g. Ekos – while others do not prompt – e.g. Nanos.. Et cetera)
The breakdown of polls from the other polling firms are:
Leger – 1 polls
Nanos – 5 polls
Decima – 5 polls
Angus-Reid – 5 polls
Ekos – 3 polls
Strategic – 5 polls
Ipsos – 5 polls
The sources of all polls conducted for this piece were each polling firm’s individual website. You can access an exhaustive list of all the 2009 polls (with links to said websites) here.
The following, for those interested, is a definitive list of all polls used for this piece.