This is Part I in a three part series.
For Part II, click here. For Part III, click here.
John Wright, the senior vice president of Ipsos Reid polling firm — and longtime avid Paulitics reader — recently made some comments here at Paulitics that called into question the methodology I used to examine a perceived discrepancy in Ipsos Reid’s polling data.
These comments deserve to be answered in a respectful and courteous manner irrespective of the manner in which they were delivered.
Mr. Wright explains in his comment:
“First, the Conservative average vote percentage January 2008-June 2009 per polling company:
EKOS 37.5
Angus Reid 36.8
Ipsos Reid 36.2
Nanos 35.6
Strategic Counsel 35.2
Harris-Decima 34.8
The Mean: 36.02“
Leaving aside for a moment the issue of where Mr. Wright got those individual averages for each polling firm (my numbers are completely different and I have meticulously documented all polls covered by the mainstream Canadian press here), Mr. Wright just committed a very serious error in the statistical world and it is a very amateurish one at that. Can you spot it? It’s not his math (again, leaving aside the issue of the individual polling firm averages). The problem is that, in his attempt to refute me, Mr. Wright took the averages of each polling firm and then took the unweighted average of the average. This, as anybody who deals at all with statistics will understand, of course, is a big “no-no”.
Consider if I did the same thing with the following data set:
Do you support the Metallic Metals Act of 1898?
Polling firm x (poll #1): Yes, 75%
Polling firm x (poll #2): Yes, 80%
Polling firm x (poll #3): Yes, 70%
Polling firm x (poll #4): Yes, 85%
Polling firm x (poll #5): Yes, 65%
Polling firm x (poll #7): Yes, 74%
Polling firm x (poll #8): Yes, 76%
Polling firm y (poll #1): Yes, 30%
Polling firm z (poll #1): Yes, 36%
Polling firm “W” (poll #1): Yes, 45%
Average of each polling firm:
x: 75%
y: 30%
z: 36%
“W”: 45%
Therefore, using Mr. Wright’s methodology for ‘rebutting’ my argument, the average of polling firms “x”, “y” and “z” = 46.7% which just goes to show that the result for polling firm “W” is both mainstream and accurate?
Then there’s the issue of each polling firm’s numbers. Readers can feel free to look up all of the polls published and reported on in the press for 2009 (here or here). Based on the data in these lists (which I believe to be exhaustive), the averages for each polling firm are as follows:
Now, in the interest of full disclosure, I discovered today that I had accidentally overlooked one poll conducted by Ipsos Reid on March 5th, 2009. I fully apologize for the error, however, as the math will demonstrate, this one omission on my part does not change matters much. With the formerly absent March poll, the Ipsos Reid sheaf of 2009 polls (that I have found) are as follows:
I am unsure of where Mr. Wright is getting his 2009 averages for each of the aforementioned polling firms, but I would be extremely interested in hearing directly from him on this matter. His number for Ipsos itself is close enough to mine that I would conjecture that he is probably just using the decimal values for each party obtained by his firm whereas I am using the rounded whole numbers published in the press. Unfortunately, Mr. Wright provides neither a source nor does he show his math. I have done both and am willing to change my position if he can show me a sheaf of polls that I’ve missed.
Even with this new missing Ipsos Reid poll, however, if we compare each 2009 Ipsos Reid poll to the three polls which directly preceeded and succeeded it, we still get these startling graphs:
Can Mr. Wright explain these graphs?
Stay tuned for Parts II and III of this response to Ipsos Reid, wherein the dénouement is reached and I will break out the z-scores, standard deviation calculations and other extreme geek methods to argue that this whole thing isn’t just a matter of attractive yet deceptive graphs.
Paulitics 
Read what I sent you in the first place: all of the polls
conducted by every company in 2008 and up to recently
in 2009 and all availavble on Elections Canada.
Next: it’s an average of the findings and doesn’t need to be weighted.
Regards.
Hi John:
With all due respect to Paulitics (and bravo to him and his blogpostings for getting your attention), you sure seem to be paying an inordinate amount of attention and detail to Paul’s site.
Granted, his feed gets aggregated on a few sites which gives him a bit more traffic and exposure, but still, it sure seems like you’re trying to swat a fly with a sledgehammer here. I can only imagine what you’d be saying or how many times you’d be posting if one of the bigger traffic blogs took you to task over something in your poll methodology.
Regards
Scott
Mr. Wright,
Taking an average of a group of averages (each populated by a different number of terms) is a good way of getting skewed data. That’s not an opinion, that’s a fact.
If you didn’t appreciate my proof above at how such a method can be used to generate an “average” of 46.7% instead of the actual average which is 63.6% then perhaps the following will do so:
Jimmy Sigma loves his stats courses but hates every other courses in his school. His mark breakdown is as follows:
Stats 101: 90%
Stats 102: 94%
Canadian History 101: 30%
British History 102: 20%
American History 103: 10%
Geography 101: 40%
If Jimmy’s school took his averages as you did, Jimmy would pass his grade even though he only passed 2 out of 6 classes (Stats average = 92%, History average = 20%, Geography average = 40%; unweighted average of the averages = 50.6%)
Also, I have been unable to find any such database that you speak of at Elections Canada. Since you seem to have access to this database, would you care to tell me which polls I’m missing or providing me a link?
I have a total of 33 polls conducted in 2009 (excluding perhaps any new polls that have been released in the last few days, I haven’t checked since last week). What is your number?
Wow.
First-time visitor and I must say that it has been informative. To see a VP of a polling firm calling for the arrest of bloggers and proudly owning the “unholy” question when its demonstrated that his firm has a skew was priceless. I am forwarding this exchange to several journalists in hope that someone might write about this firm and their interesting VP.
I will never look at an Ipsos-Reid poll again in the same light, if the VP can’t show a single number to prove his facts you know something isn’t Wright.
As for his comment about not needing to weight, I am speechless. The lack of knowledge it takes to make this statement is staggering and so poorly reflects on the company that I wonder if Mr. Wrights days are numbered.
I feel the need to elaborate on a point I made in the last article.
Paul, the argument you seem to have with Ipsos Reid reminds me quite a bit of something that occurred during the American election last summer. A poll released by GWU/Battleground showed John McCain ahead in the election when nearly every other poll was showing Obama with a large lead. As it turns out, the reason for the discrepancy was that the poll had not weighted for age. Therefore the responses included a disproportionately high number of older voters, who supported McCain much more than young voters. So the methodology had accidentally shown more support for one candidate over the other. I encourage Paul and anyone else who may care to read an indepth analysis here: http://www.fivethirtyeight.com/2008/09/whats-wrong-with-battleground-poll.html
Another important point I want to make is without any election or other definitive measure of voter support, we don’t know whose polls are correct at this time. It is just as likely that Ipsos is right and everyone else is wrong or vice versa. We simply cannot know at this time who is actually right. My argument last time (which I didn’t elaborate on at all) was that instead of look at average of polls right now, we should be looking at who got the result right in previous elections. We should be judging pollsters by their actual results, not their numbers without any election in sight.
To summarize, I would encourage either Paul or Mr Wright (or any Ipsos representative) to do 2 things. First, we should look at Ipsos-Reid’s methodology and compare that to the other pollsters to try and account for their recent differences. Second, we should compile results from previous elections to get a legitimate benchmark of Ipsos-Reid’s track record. To me at least this seems like a good way of discussing Canadian polling in general and avoids some of the nastiness in the comments that I feel may be accusing of bias when the real question should be what’s different and who seems to be getting more accurate results using their methodology.
Hey guys: Interesting argument. I too have noticed — with my somewhat untrained eye — a difference in the outcomes of surveys done by other companies and those from Mr. Wright’s company. I notice that for polls taken within a week or so Ipsos tends to show better numbers for the Tories and slightly lower numbers for the Liberals and NDP. And you know what? I don’t care. I like polls but I look at trends. I would argue Ipsos polls follow the same trends others do. For example, most polls I’ve watched in recent months generally show a slow but sure improvement in Liberal support with a corresponding drop for Conservatives and to a lesser extent the NDP. All this talk about averages and weights leaves me cold. Polls taken between elections have to be taken with a grain of salt. And I would suggest that’s why we’re seeing so much diplomacy in Ottawa between Ignatieff and Harper right now. That too must be taken with a grain of salt. The only poll that will count will be taken during the general election with all those who bother to vote. And I expect that will happen within the next year.
Until then, none of this stuff is worth getting one’s nickers in a knot.
Just a thought
Ken, you are a very observant and wise guy…
Ken, you are a very observant guy. The treadline is the important point. You take the Ipsos slant as granted, but you see if their numbers move. What I do, just automatically shave off a couple points from Ipsos. Ipsos says Cons at 35%, it’s more like 32-33%, which is better than previous Cons at 37%, which was more like 34-35%.
Mr Wright can say what he wants, but people aren’t stupid. It’s be noticed by pretty much every poll junkie online, apart from Conservatives, that Ipsos doesn’t jive. You can go back to the 2004 election, 2006 election, and you’ll find that Ipsos was the worst for overstating Con support, understating Lib support. Them’s the facts, and Paulitics here has merely shown the trend has continued recently.
One might also notice, that Ipsos only seems to have publishing gaps when it doesn’t look good for certain parties, the weekly onslaught somehow always wanes to here and there. There’s also the issue, and Wright needs to address this- why is that Ipsos is the only pollster that doesn’t publish it’s poll internals? Now, the obvious retort, is they demand money to see, but really shouldn’t a public presentation be available to public scrutiny??
Does Ipsos have a bias? Seems to me, somebody merely presenting something rational, like numbers, doesn’t quite equate to legal action. Wright knows this, or he wouldn’t run around blogs for years threatening this and that, never doing anything. Fact of the matter, the mere suggestion of bias in a court is bad, bad, bad for business, not to mention really embarrassing when you can present such striking graphs.
Yes, people aren’t stupid and don’t expect them to stop noticing or understanding basic math.
So, what’s your e-mail? What’s your phone number? What’s your address??
What, you aren’t a coward are you???? Show us where you are…you have to have more than that “Paul Bernardo” lookalike picture, don’t you???
C’mon, fess up…
Juust desire to say your article is as surprising.
The clerness for your puut up is simply excellent and i can suppose you
are knowledgeable in this subject. Well together with your
permission let me to grab ykur feed to stay up to date wth
approaching post. Thanks 1,000,000 and please carry on the gratifying work.
I have read some excellent stuff here. Certainly price bookmarking for revisiting.
I surprise how a lot attempt you set to create the sort of fantastic informative web site.
Hi there, I would like to subscribe for this blog to take newest updates, thus where can i
do it please help out.
whoah this blog is magnificent i love studying your articles.
Keep up the good work! You know, many persons are looking around for this info, you could help
them greatly.