This is Part III in a three part series.
For Part I, click here. For Part II, click here.
A Response to Ipsos Reid, Part III: Comparing Ipsos Reid to the other polling firms using 2 methods and with control tests.
In Part I of the Paulitics response to Ipsos Reid, I showed two graphs of the level of support given to the Conservative and New Democratic Parties in all polls conducted in 2009 by all polling firms.
As you can see below, the graphs tell a startling tale of just how radically out of step Ipsos Reid’s estimations of the Conservative and New Democratic Parties seem to be when compared to other polling firms. I asked Mr. Wright to explain these data as I’m sure we would all be very interested in learning why we are seeing these trends. So far, the response has been: Crickets. Nothing. Deafening silence.
Now, a quick disclaimer: As I’ve mentioned before, these graphs don’t necessarily mean that Ipsos Reid has bad data. What these graphs mean is that either every single other polling firm in the country on average has bad data or Ipsos Reid has bad data. As always, I invite my readers to critically consider for themselves what Occam’s razor can tell us about this impasse.
Now, to be fair to Ipsos Reid, I started thinking about possible explanations for these graphs since Mr. Wright didn’t seem keen to offer his own explanations. One thing I noticed was that I was using the overall average of all polls conducted by all firms in 2009 and then calculating an overall average as if the movement up and down of the Conservative and New Democratic Parties wasn’t a factor. (Mr. Wright was doing this too, but keep in mind we’re being charitable here).
This method could possibly be a flawed method if, for instance, polling firm “x” released a whole slew of polls in January when Party “A” was enjoying high levels of support from all polling firms and then polling firm “x” didn’t release a poll in early February when Party “A” was down in support. Taking an overall average without taking into account differences over time could thus yield a favourable or an unfavourable view of polling firm “x”.
For instance, if we used this methodology in the hypothetical example of the “Purple Party” described below, we would wrongly conclude that polling firm “x” grossly over-estimated the support of the “Purple Party” when in fact, they look to be more or less right on the money.
So, I decided to analyse every 2009 Ipsos Reid poll against the 3 polls conducted before and after so that we’re taking into account what’s happening in party support right then and there and not several months earlier when things might have been different. I then calculated each 7 poll grouping’s standard deviation so that I could then calculate something called a “z-score” for each Ipsos Reid poll. For those of you not conversant in z-scores don’t worry, I have also calculated the “z-score percentile” as an easy way to understand what’s going on in each of the polls.
The way this works is that if something is exactly average, it would be in the 50th percentile (because 50% is right in the middle). If something is extremely unlikely at one end of the spectrum, you’ll get a very very low number (like something being in the 5th or 6th percentile) and conversely something extremely unlikely at the other end of the spectrum, you’ll get a very very high number (like something being in the 96th or 97th percentile). So the name of the game here is that Ipsos Reid wants to see medium range percentile numbers and wants to avoid really high and really low percentiles in the following tables because if the numbers are really high or really low that would suggest that Ipsos’s data is way off the mark.
Here’s what I found:
So what does this mean? Well, the data in the tables above show 2 outliers at the 95% confidence interval, 5 outliers at the 90% confidence interval (including the 2 at 95%) and 7 total outliers at the 80% confidence interval (including the 5 at 90% and the 2 at 95%).
On the face of it, that seems like a lot of outliers especially considering I only analysed a total of 10 Ipsos Reid data points. But, again to be fair to Ipsos Reid, maybe we would see roughly the same large number of outliers if we did this to any polling firm’s data in this way. I mean after all, 7 polls is not a particularly large population.
So, I decided to run the exact same calculations on Nanos’s numbers, Ekos’s numers and Strategic’s numbers as a control group. You can look through the tables of each of these calculations here, here and here.
As you can see, it’s not even close.
Nanos, which had 8 total data points, had zero (0) outliers at the 95% confidence interval, zero (0) outliers at the 90% confidence interval, and only one (1) outlier at the 80% confidence interval.
Strategic, which had 12 total data points (more than Ipsos and Nanos), still had zero (0) outliers at the 95% confidence interval, zero (0) outliers at the 90% confidence interval, and only one (1) outlier at the 80% confidence interval.
And lastly, Ekos, which had 6 total data points, had no outliers at either the 95%, 90% or 80% confidence intervals.
So, to put this another way, only 7.7% of Nanos, Ekos and Strategic’s 2009 Conservative and NDP data points are outliers at the 80% confidence interval.
Conversely, a whopping 70% of Ipsos Reid’s 2009 Conservative and NDP data points are outliers at the 80% confidence interval
Also, none (0%) of Nanos, Ekos and Strategic’s 2009 Conservative and NDP data were outliers at the 90% confidence interval whereas 50% (½) of Ipsos Reid’s data were outliers at the same interval.
If you are like me, and like to view findings like this in ‘pretty graph form’, here you go:
Oh, and just for fun, I also ran the same basic calculations for each Ipsos Reid poll compared to the entire set of 2009 polls and came up with almost the exact same results. You can see the tables for these calculations by clicking here.
The evidence supporting my initial contention is mounting and the responses we’ve gotten from Mr. Wright have been a mixture of snide, sardonic comments, fallacies and silence.
I believe we all deserve better than that.