A Long Time Ago, In A Scientific Field Far, Far Away . . .

by | March 31, 2015

Guest post by Sebastian Thaci

Somebody recently pointed me to this “comprehensive and in-depth new public opinion poll” on religion in Canada conducted by the Angus Reid Institute:


I have read all the study carefully, and I have to reject its results based on the selection bias and conceptual ambiguity.

The study was conducted “among a representative randomized sample of 3041 Canadian adults who are members of the Angus Reid Forum.” This means that the results of this study apply to the members of this online forum, which are not representative of the whole Canadian society. Only computer-literate people, with computers, Internet access, free time and interest in the subjects of the forum are members of the forum. Out of these, only some responded to the survey. Out of those, apparently, only some were considered “representative” (why?).

This is a huge selection bias. How did they go from this select group to “Religion and Faith in Canada”? A randomized phone survey would have had more statistical power.

For example, how many 55+ year old first generation immigrants are members of this forum, and how many of them responded to the survey? How many fundamentalist/conservative religious people do you think are on this forum and also took the survey?

There are also multiple conceptual problems or problems with the questions posed. Just a few examples

  1. The categories of embracing and rejecting religion do not make sense. Some deeply religious people could reject the present religious institutions because of perceived corruption or other reasons. Are they embracing or rejecting religion? Also, an atheist might go through the moves of a religion just to keep his minority community and culture alive, like many atheist Jewish people are indeed doing.


  1. The congregation attendance evaluation is subjective, and people can be wrong about this, or they can lie.


  1. The question “What’s right or wrong is a matter of personal opinion” – many atheists and irreligious persons will say no to this, but there is no alternative except the Ten Commandments. Very wrong. How about the third option of an innate human sense of morality developed during our evolution? Even some religious people who believe in evolution will choose this last option.


  1. Spiritual or religious – two dictionary definitions do not explain the meaning of these words. What do people really understand by ‘being spiritual’? Asked this question, I myself do not know what to answer. Am I spiritual if I light scented sticks and do Yoga meditation in the morning? Or am I spiritual because I read my horoscope every day?


  1. There is no ‘conversion’ question – I myself came to Canada an evangelical Christian, now I am an atheist. Interestingly, I rejected institutionalized religion in both cases. When I was a Christian, I thought that the institutionalized corruption of the Church can only be demolished by grass roots, small groups Christianity. Now, I reject any form of religion. According to this poll, I was not embracing religion back then as a Christian, and I am not embracing it now as an atheist. But my mind has completely changed. I take different decisions and vote vastly different now. Also, there are no questions in the study about how people come to be in any category: converted, born into a religion, etc. No question on how free they think they are to move into another category or how likely. We keep ignoring the fluidity of religious social reality. And we also ignore that people are sometimes harassed or even die for leaving a religion or other.


  1. The immigration factor is not studied enough. There are many variables there, and Canada receives a LOT of religious immigrants, some refugees, some voluntary. Some of them come here to lose their religion, though, or at least to loosen its grip a little. Also: “With greater immigration from Asian countries in particular, the greatest increases have been among Roman Catholics, evangelical Protestants, and other major world faith groups.” Really? I doubt that we receive mostly Roman Catholics and evangelicals from Asia. I may be wrong, but I suspect something is off here.

What is interesting about this study is our reaction to it – the results are already familiar to us. We consider the numbers close to what we knew about religion/irreligion in Canada from previous polls and the last Census of Canada. We can go straight to assuming that the study is well done, and that we can cogently comment on the results. The numbers can be intelligently discussed, as if they are right, just because they sound right, but that does not make the numbers right, unfortunately. Not for the whole of Canada.

How did the science of public opinion survey and statistics become so weak, subjective and muddy?

Will anybody in the Canadian public at large be aware of the shortcomings of this poll? And does the public even care about such details anymore?

Maybe we should do another Angus Reid forum poll to find out. That’s what Angus Reid would do!

6 thoughts on “A Long Time Ago, In A Scientific Field Far, Far Away . . .

  1. Indi

    I find the objections given in this post a little petty and small-minded, and overall wholly irrational.

    I mean, if you’re going to object to the survey because it used the forum population as its sample, how can you seriously offer a *phone* survey as an alternative with a straight face? First of all, one can make all the same kind of objections for a phone survey, you did for this survey: you’re only surveying people who a) have phones; b) will answer within the hours that the survey makes it calls (no night-shift people! and no one who screens their calls); c) are willing to take the time to do a lengthy phone survey (which is a pretty darn small group, really). And as Reid himself points out, phone surveys aren’t really all that effective anymore.

    Even more ridiculous, if you do really take these kinds of objections seriously, how can you then claim to take the “census” results seriously? Harper changed the census rules so the long form was optional, remember? That means the people who actually filled (which was only about 2/3 of people who got it) it out are also self-selecting.

    Angus Reid is not some hack partisan think-tank or fly-by-night operation; their surveys are quite well-respected, and generally produce very good data. If you’re going to just shrug off this survey for these reasons, then you are also effectively shrugging off every… single… survey… ever done… *including the last “census”*! Basically, you’re rejecting *ALL* surveys, because they all have the same shortcomings! That’s just a little ridiculous, i’d say. Surveys may not be *perfect* tools, but they’re hardly useless.

    As for the “detailed” objections: I’m sure among the ~1100 55+ participants in the survey, there were a substantial number of first-generation immigrants. Accounting for that sort of thing is kinda what the people at Angus Reid do for a living, ya know. I’m sure they’ve also recruited a fair number of fundamentalists, too… why would you assume they haven’t?

    1. That is all covered in the study. For the record, the number of people who are deeply religious people but reject present religious institutions seems to be around 18%.

    2. So what? Read it and you will find that the survey repeatedly spells it out as “*report* that they attend services”. They’re quite well aware of the chance that people are lying. It’s a freaking opinion poll – they’re not about to follow people around and make sure they go to church. But even if they are lying, so what? The results we have are for people who *report* they attend services once a month or more – that covers those who don’t, and those who are motivated to lie about doing so. That is not meaningless information. Even if it doesn’t tell us *actual* habits, it tells us what people *think* they *should* be doing… ie, it tells us how important going to religious services is to people, even if it doesn’t actually tell us who goes.

    3. And they’re all counted. The data’s right there, in table form.

    4. It doesn’t matter what those words mean, any more than it matters what “atheist” means in the NHS (where very few people identify as atheist, and many, many more probably are). What the survey tells is how people *identify* themselves. This is *extremely* useful information. If more people are calling themselves “spiritual but not religious”, that alone is a powerful indicator that religion itself is becoming increasingly marginalized.

    5. So what? No survey can cover everything, geez. Besides, complaining that the whole survey is so terrible you reject it out of hand, then saying you want *more* of it is a little like that old Woody Allen joke: “The food here is terrible!” “I know, and such small portions!”

    6. So the fuck what? Again, not every survey can cover every possibly angle, and just because *this* survey doesn’t stroke the particular little bugbear you are interested in does not mean the whole thing is worthless.

    The bottom line is that the data from this survey is extremely useful as a bellwether for the general Canadian public’s opinions. That’s why the Angus Reid Institute spends their limited cash – they are a charity, after all – doing it, and why people keep giving them money: because the information they produce is useful. Rather than rejecting the whole survey because of a handful of really silly minor objections, a rational response (to this, and to *EVERY* survey) is to learn what we can from the data while being mindful of its limits.

    The “criticisms” enumerated in this post are all absurd or irrelevant. This is not a legitimate critique. It is just anti-scientific ranting, apparently to flaunt the fact that you can spot weaknesses in experimental design. *All* real science is messy, and all real scientific data is collected under conditions where there is potential for biasing factors. Pointing that out is as shocking as pointing out that the sky is blue. *Rejecting* the results for that reason is patently ridiculous.

    All in all, a terrible post. It’s more about sensationalistic contrarianism for the sake of it than a serious critique of the survey – to the point of being completely irrational. It adds nothing intelligent to the discussion. It’s just grumpy, anti-intellectual shouting, like a bitter old man screaming his uninformed opinions about what’s wrong with everything at whoever will listen, regardless of the fact that there is no substance to them, just because he wants to be heard.

    1. Tim Underwood

      Not petty or small minded. A deeper criticism, of the ideas being surveyed, is healthy.

      To me, the survey’s results are intriguing and somewhat satisfying.

      I would align myself with those who are both non-religious and non-spiritual , but I probably couldn’t explain to anyone exactly what I mean by this.

  2. Shawn the Humanist

    Not defending the survey method or its results.

    I see your concerns. However, as I was reading some of your concerns it occurred to me they may approximate randomness by weighting participants.

    I used to be a member until this year and answered questions when I could. They always ask questions at the end about your age and gender, and sometimes political affiliation. General questions to find out what demographic group you are in.

    It’s possible they use these to weight your demographic to try to better approximate. They may not at all, but I always wondered if they were doing that.

    For example, if they find they have a lot of non-Liberal, non-NDP, non-Conservative, non-Green, not-Bloc voters who responded (maybe Christian Heritage Party?) and all thought one way, to lesson those beliefs to be representative of how many of those people there are.

    To clarify: If they are 50% of the poll but we know they are less than 5% of the population we can then fix it by weighting their answers differently.

  3. Diana MacPherson

    Sebastian, I think you raise some important questions about the survey and this sort of critical thinking should always be done when we see survey results.

  4. Indi

    > A deeper criticism, of the ideas being surveyed, is healthy.

    Deep criticisms are always welcome, but the “criticisms” in this post are an example of the most shallow kind of criticisms. It’s just: “Well, looks like there *might* be a problem… of course, I don’t really know what I’m talking about and can’t be arsed to actually *check* whether there really is a problem… so I’m just going to reject the whole thing out of hand.”

    I’m no an expert on Angus Reid’s methodology (especially with regards to ARI in particular, which is the latest incarnation), but even *I* know that most of the criticism above is not valid. The way it’s written, it almost reads like the writer thinks the Angus Reid forum is just a discussion forum on the Internet, where only people who share some particular interest sign up under their own motivation.

    The way Angus Reid works (at least, when they were ARS, before Reid retired and recreated it as a non-profit, but i can’t imagine the methodology has changed much) is that they attract people to sign up to their forums by offering cash rewards (among other things) for completing surveys. When you sign up, you fill out some sort of survey that gives Angus Reid the info to know where you stand, demographically, with respect to the general population.

    Using the reward incentives, they’ve grown their panel to well over 100,000 people. Even if they just polled their entire population each time, their results would be nothing to sneeze at. But they do something more clever than that.

    When it’s time to run a survey, they just use the information they have on each member to select a group from their membership that’s somewhat representative of the Canadian public, then email those members to ask them to do the survey. In other words, they do *better* than a blind random selection, because they get a group that’s at least somewhat representative of the population, demographically… which is the opposite of the selection bias the writer of this post accuses them of. (They may also use weighting, internally, but with this method, they usually don’t need to.)

    And they’ve generally produced fairly good results – which is why academics use them (and an academic was involved in this survey, too).

    Criticism is fine, criticism is welcome. But if the criticism is irrational, it’s just noise. *Real* criticism isn’t just a quick scan of something looking for superficial and potential issues, followed by a blanket dismissal without bothering to actually do any real digging to see whether the issues are actually real. This post is just a series of ungrounded, superficial criticisms on things that *look* wrong at a first glance, but of course Angus Reid wouldn’t have been around this long if they were as stupid as the writer of this post seems to think they are.


Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.