There is a survey getting a lot of hype in the media now, widely reported under headlines like “43 Percent of Canadians Say Science Is a ‘Matter of Opinion’”. Given that 100 percent of science headlines in the media are misrepresentions, what’s really going on here?
This survey was commissioned by the Ontario Science Centre for science literacy week. It seems to be an annual thing (here’s last year’s survey results). The survey was done by Léger (aka The Research Intelligence Group) using their forum.
I would have loved to do a really deep dive into the survey data, to suss out what’s really going on. Unfortunately, I can’t because no one has actually linked to the fucking survey data. Even the OSC source is just a summary, and I can’t find the data anywhere on Léger’s site.
So all I have to work with are various summaries (like this one)… but that’s enough to detect that there’s something rotten with the “43%” headline.
The first problem is that the various reported takes on the stories get the entire purpose of the survey wrong. Or at least the subsection of the survey that contains the 43% result. The survey covers a broad range off topics related to the public’s perception of science. One part of the survey does cover science literacy… just not the part that contains the 43% result. Other topics include trust in sources of scientific information – both media sources and sources in general (like scientists versus religious leaders) – trust in “scientific findings”, concern about “fake news”, belief in certain anti-science conspiracy theories, and more. The 43% result comes from the section on “scientific findings”.
But before we get to that, I have to explain the broad strokes of the survey. The survey basically categorized people into four groups (five if you count the 2% who answered “don’t know”… but they’re below the survey’s error threshold) based on the question “Which of the following best describes how you make decisions in your everyday life?”:
- Rely mostly on intuition and personal beliefs
- Facts are great but I often go with my gut feeling
- I often balance facts with my own personal intuition
- I tend to rely on data and hard facts
Here’s how the responses break down:
So right off the bat, we’re not looking at a country full of woo-swallowing suckers. The biggest group is the “balanced” group… not the least bit surprising; this is Canada, where we’re all about balance. But even the “balanced” group is hardly anti-science; it’s perfectly reasonable when confronted with new information to trust your gut when the facts are still uncertain. Heck, the first took in the skeptic’s toolkit should be the “smell test” – if something smells funny it warrants further analysis, but if it fits with everything else you already know, it’s probably good.
Now let’s get to the headline item.
The item in question was, “To what extent do you agree or disagree that Scientific findings…”, with the option “… are a matter of opinion”. Here are the results:
- Strongly agree: 7%
- Somewhat agree: 36%
- Somewhat disagree: 36%
- Strongly disagree: 19%
- Don’t know / prefer not to answer: 2%
The 43% comes from adding the “strongly” and “somewhat” agree, which is legit-ish. But what it does is obscure the real balance of opinion on the subject. It’s pretty much equal both ways with those who somewhat agree/disagree … but the number of people who strongly disagree is almost three times as much as those who strongly agree. In other words, the actual median – taking the strength of the agreement/disagreement into account – is much lower, probably somewhere in the mid/low-30s (I estimate just about 30%).
But even considering that, there are a couple of caveats to keep in mind here. The most important is that this particular item came as part of a group of items that all seem to have the theme of being about… NEW scientific findings. The other items mention “new research” and whether findings can be challenged.
If we’re just talking about brand new scientific findings, and particularly that we’re talking about things we’re reading in the media about brand new scientific findings… well, hell, that sure changes everything! You should certainly be suspicious that what you’re reading is not actually reflecting real science, and you’re just seeing some science writer’s spin on it.
Now, it’s not clear that’s how respondents actually interpreted the question. But it’s definitely within the realm of possibility. It’s even probable, which means the 43% number is a probably massive overestimate.
But even without all of those considerations, there is the simple matter of fact that the 43% number does not mean what most media sources are implying it means.
When you read something like “43% of Canadians think science is a matter of opinion”, it’s only natural to assume that means that no more than 57% think science is a matter of fact. That assumption would be very wrong, because the same survey actually asked that question, in the same section. Given, “To what extent do you agree or disagree that Scientific findings…”, with the option “… are objective facts”, these were the results:
- Strongly agree: 20%
- Somewhat agree: 61%
- Somewhat disagree: 13%
- Strongly disagree: 1%
- Don’t know / prefer not to answer: 5%
Well that is a very different picture!
Taken in isolation, completely stripped of its context, that 43% number looks really bad. But when you look at the context the question was asked in – with the implication that we’re talking about media reporting of new scientific findings – and with the surrounding data … it clearly doesn’t mean what it sounds like when you simply say: “43% of Canadians think science is a matter of opinion”.
What about some of the other headline results from the survey?
Well, there is a lot of talk about the “controversial theories” section, which covers climate change, genetically modified organisms, and the vaccine/autism “link”. These results are not good news:
But there’s really not much that’s surprising there. We already know that Canadians are shockingly ignorant about GMOs. And even the “[t]he science behind global warming is still unclear” results aren’t really as concerning as they seem at a cursory glance. You can argue that that’s completely true: scientists are not sure exactly how much the Earth will warm in the next few decades, or whether we’re past the point of no return or not. You don’t need to interpret it as “people don’t think global warming is actually real”.
The last item highlighted in most reporting on this survey is the “33% say they’re scientifically illiterate”. But that’s misrepresenting the result!
The actual question was “To what extent do you agree or disagree with each of the following:” with the item “I consider myself to be science literate (i.e. generally knowledgeable about the process that science follows, recent discoveries, etc.)”. The 33% number comes from those who strongly (8%) or somewhat (25%) disagree.
But here’s the thing: not considering yourself “scientifically literate” is not the same thing as considering yourself “scientifically illiterate”. “Scientifically literate”, in common understanding, means “well-versed in science”… but not being well-versed in science is the same thing as being completely fucking clueless about it! Someone who is very interested in science but not all that confident about their expertise could very well decline to identify as “scientifically literate”. They may still understand quite a bit, and of course they are keen on the topic, but feel more like interested spectators rather than one of the initiated elite.
And to back up that interpretation, another of the items in that same section was “I would like to know more about science and how it affects our world”. The results for that were 82% agreement.
Overall, the results of the survey really don’t support the dour, negative assessments most media outlets are spinning. A more honest summary would be:
- Canadians love science, and want more of it.
- Canadians very much trust scientists, educational institutions, and science centres/museums… but they don’t really trust anyone else (including most media).
- Canadians are particularly wary of media motivations in reporting science, and generally think they’re spinning it based on their biases.
- Canadians are having a hard time figuring out the truth behind “controversial” science topics (climate change, GMOs, and the vaccine/autism “link”). But for most, it’s not because they distrust science. It’s because with all the misinformation out there, they can’t figure out what the science really is.
- Canadians are actually very savvy about what science really is. They don’t believe it’s gospel, and they understand that it changes over time as we learn more.
- Despite liking science, and having a decent understanding of what it is, Canadians are thirsty for more. They want a better understanding of science, and more scientific knowledge. They support more science investment, and more science in schools.
The takeaway is that Canadians are fairly science literate, but don’t feel they know as much science as they could, but don’t know where to learn more; going back to school is impractical, and media sources are dodgy. That’s a very different takeaway from the “43% of Canadians think science is a matter of opinion” spin.
Reminder: My take on it is based on a very superficial analysis of the data… because I don’t have the data; I’m just going on summaries. I could very easily be way off base. But I think my interpretation both jibes better with results from previous surveys, and better fits the results of this survey than this silly idea that Canadians generally see themselves as clueless troglodytes who believe whatever shit comes out of Gwyneth Paltrow’s mouth.