This is Part I in a three part series.
For Part II, click here. For Part III, click here.
John Wright, the senior vice president of Ipsos Reid polling firm — and longtime avid Paulitics reader — recently made some comments here at Paulitics that called into question the methodology I used to examine a perceived discrepancy in Ipsos Reid’s polling data.
These comments deserve to be answered in a respectful and courteous manner irrespective of the manner in which they were delivered.
Mr. Wright explains in his comment:
“First, the Conservative average vote percentage January 2008-June 2009 per polling company:
Angus Reid 36.8
Ipsos Reid 36.2
Strategic Counsel 35.2
The Mean: 36.02“
Leaving aside for a moment the issue of where Mr. Wright got those individual averages for each polling firm (my numbers are completely different and I have meticulously documented all polls covered by the mainstream Canadian press here), Mr. Wright just committed a very serious error in the statistical world and it is a very amateurish one at that. Can you spot it? It’s not his math (again, leaving aside the issue of the individual polling firm averages). The problem is that, in his attempt to refute me, Mr. Wright took the averages of each polling firm and then took the unweighted average of the average. This, as anybody who deals at all with statistics will understand, of course, is a big “no-no”.
Consider if I did the same thing with the following data set:
Do you support the Metallic Metals Act of 1898?
Polling firm x (poll #1): Yes, 75%
Polling firm x (poll #2): Yes, 80%
Polling firm x (poll #3): Yes, 70%
Polling firm x (poll #4): Yes, 85%
Polling firm x (poll #5): Yes, 65%
Polling firm x (poll #7): Yes, 74%
Polling firm x (poll #8): Yes, 76%
Polling firm y (poll #1): Yes, 30%
Polling firm z (poll #1): Yes, 36%
Polling firm “W” (poll #1): Yes, 45%
Average of each polling firm:
Therefore, using Mr. Wright’s methodology for ‘rebutting’ my argument, the average of polling firms “x”, “y” and “z” = 46.7% which just goes to show that the result for polling firm “W” is both mainstream and accurate?
Then there’s the issue of each polling firm’s numbers. Readers can feel free to look up all of the polls published and reported on in the press for 2009 (here or here). Based on the data in these lists (which I believe to be exhaustive), the averages for each polling firm are as follows:
Now, in the interest of full disclosure, I discovered today that I had accidentally overlooked one poll conducted by Ipsos Reid on March 5th, 2009. I fully apologize for the error, however, as the math will demonstrate, this one omission on my part does not change matters much. With the formerly absent March poll, the Ipsos Reid sheaf of 2009 polls (that I have found) are as follows:
I am unsure of where Mr. Wright is getting his 2009 averages for each of the aforementioned polling firms, but I would be extremely interested in hearing directly from him on this matter. His number for Ipsos itself is close enough to mine that I would conjecture that he is probably just using the decimal values for each party obtained by his firm whereas I am using the rounded whole numbers published in the press. Unfortunately, Mr. Wright provides neither a source nor does he show his math. I have done both and am willing to change my position if he can show me a sheaf of polls that I’ve missed.
Even with this new missing Ipsos Reid poll, however, if we compare each 2009 Ipsos Reid poll to the three polls which directly preceeded and succeeded it, we still get these startling graphs:
Can Mr. Wright explain these graphs?
Stay tuned for Parts II and III of this response to Ipsos Reid, wherein the dénouement is reached and I will break out the z-scores, standard deviation calculations and other extreme geek methods to argue that this whole thing isn’t just a matter of attractive yet deceptive graphs.