Have you ever seen someone just totally twist a study to prove excatly what they want it to say?
Then you read the study and you just go–hold up. That’s not it at all!
We dealt with this back when we reviewed a post by Josh Howerton talking about how gender-roles-based religion leads to the best sex, despite the research he quoted showing… not that. (If you missed it, read it here.)
I (Rebecca) wanted to jump on the blog today by expanding on a newsletter I wrote back when we released that blog post. Here are 5 questions to ask to make sure you’re approaching research with integrity and with as little bias as possible.
This isn’t an exhaustive list, but it’s a start! So let’s go!
1. Does the study’s population match who I’m applying this research to?
Am I reporting on a different population than the study looked at in a way that changes the results? E.g., you may find a study that says 14-year-old girls are more emotionally volatile than 46-year-old accounting technicians, and then use that to say that all women are more emotional than all men. Not honest. The populations are far too different than what was originally studied.
Download Our Marriage Survey
Join 40,00 others and let's change the evangelical conversation about sex
2. Am I looking to validate my personal experience?
Obviously much research will validate personal experiences, but sometimes we don’t match the average so it’s important to address personal bias. For example, my experience giving birth is that it’s a horribly traumatic experience. If I were looking to prove an agenda, I’d get angry when any study found that for most women, birth is not traumatic. But that’s not fair or honest–that’s me working out of my own personal bias.
If we are to be honest when looking at stats, we have to leave room for the reality that we may not be “normal,” and the statistics can teach us rather than describe us.
3. Am I educated enough to properly understand this?
This is a tricky one, but it is important! Am I trained in this type of field enough to be able to understand not only what the article is saying, but also the limitations of what I know?
Am I willing to put aside my ego and acknowledge when someone else should be the one to work through this data? Or am I allowing my desire to “prove” something lead me astray?
And by the way, it’s perfectly acceptable to say to someone else, “I recognize you think that, but neither of us have the education required to really understand this complicated subject. I think we should both refrain from talking about this.”
Think about how much false teaching could have been prevented if the evangelical church had said that the first time someone with theological training tried to teach neuropsychology! (Not trying to disrespect theological training, it’s so important! But it does not a neuroscientist make.)
4. Does this pass the “sniff test”?
Some things sound believable on the first pass, but when you back up and actually think about it logically it falls apart. For example, if a Christian teacher tells you that sending naked photos to your husband will neurologically train his brain to not look at porn anymore, that doesn’t pass the sniff test. Sending sexy photos to a man struggling with pornography isn’t helpful or safe, so that’s likely a sign that this teacher is misusing statistics for their own agenda.
Similarly, anytime someone says that virtually all of men or women do something, that should be met with suspicion. Maybe the study’s question design wasn’t specific enough, maybe their population was self-selected, there’s all sorts of problems. But just know if someone tries to convince you that 90%+ of people are a certain way, that’s likely a sign that there’s something else going on. (Not always, of course! But it’s a sign to take a closer look.)
Now, I do want to say that this, once again, needs to be put through the personal bias filter. The “sniff test” doesn’t mean “Do I find this believable,” it means, “logically does this hold up?” It’s so, SO important to have humility and self-honesty when dealing with research to recognize the limitations of our own understanding as well as our areas of personal bias.
5. Am I being selective in which part of the research I report?
Focus on the Family famously quotes study after study about how divorce hurts kids, but fails to quote the parts of those same studies that state divorce protects kids when there is abuse present. No one is saying divorce doesn’t hurt kids! We’re just saying it’s not the whole picture: often divorce is a life-saving answer.
When we report research it’s important to ask: am I presenting all of the relevant information, or am I conveniently omitting the stuff that goes against what I’m trying to prove?
So there you have it. Five questions to ask. Using statistics is complicated, but research is such an amazing tool to help us find truth and set people free.
Do you have any helpful questions you like to ask about research? Like I said, this is not an exhaustive list! Let’s chat and give more tips in the comments!
The Research Deep Dive Series
- Does Sex Start in the Kitchen or Does It Die There?
- How Does a Couple's First Time Affect Libido?
- Is "All Men Struggle With Lust" a Primal Fear?
- Is She Dressing for Attention or is He?
- PODCAST: Ogling and Dressing for Attention
- 5 Questions to Ask That Minimize Bias When Discussing Research
- 10 Things to Know About Hormones and Libido
- Bring Back Vanilla Sex
‘The “sniff test” doesn’t mean “Do I find this believable,” it means, “logically does this hold up?” ‘
Such a good point. Because so many people think that what they find believable = what is logical!
Exactly! My former pastor used the “this doesn’t pass the sniff test” to shut down conversation and wouldn’t even let me give evidence for what I was trying to show. He just didn’t believe it, therefore it couldn’t be true and didn’t “pass the sniff test”.
SO FRUSTRATING!!!!
I find that a lot of people do not want to take the time to research things so they go with information that fits their beliefs (confirmation bias) and swear it’s the whole picture. Or they’ll say something like, “But that in the Bible or not in the Bible.”
Thank you for your hard work in the research Rebecca. Keep it up!
Regarding #4, I try to anchor things around other issues that have been repeatedly empirically tested (or debunked). For example, research on the physical differences between men and women (height, strength) are remarkably consistent. Those on the intellectual or psychological differences are all over the place. Even studies showing that men are “smarter” are limited – so what if they are slightly overrepresented amongst people who are four standard deviations above the norm? That has zero to do with whether or not women can handle college physics, which basically requires one to be a standard deviation above the norm. That context helps with the sniff test.
A lot of psychological research is performed on college students, because they are the ones who sign up for these studies. I appreciated the caveat regarding the orgasm at first intercourse/libido study – performed on college students. May see different results with people who first had sex later in life.
Some research is designed to get the result the researchers want. One example of this disproving a correlation between two events, one of which usually takes place long after the other, by swamping your population with people who are very close to the first event. You could “prove” that kids whose parents divorced are not more likely to get divorced by comparing (1) 18-20 year old kids of divorced parents with (2) 18-20 year old kids of married parents.
Sorry I could go on about this…
Yes exactly! Great, Jane.
I have a question.
Isn’t Point 3 an argument from authority?
No it’s not. Argument from authority, or appeal to authority, is a fallacy in which the argument is said to have weight by attaching FALSE authority to the person making it. The example I learned was, “God does not exist because Stephen Hawking says so.” Sure, Hawking is smart. But his authority in the realm of physics doesn’t make him an expert on whether or not God exists!
What I am suggesting is to recognize when we are NOT the authority and to listen to TRUE authorities. It’s not fallacious to say, “Susan Dubbs ran this study and she said the study was about XYZ, so I think you’re wrong when you say it’s about LMNOP.” It’s not wrong to listen to the actual authorities talk about things in their actual field of expertise! What IS fallacious is when we consider someone to be an expert in EVERYTHING simply because they’re an expert in SOMETHING. 🙂
I, a psychology grad, am not educated enough to understand the complex analyses of different asthma medication trials to figure out the best puffer to put my son on because I don’t have the necessary education. I am not educated enough to understand how to read my son’s echocardiogram results. As such, I am going to listen to to what my son’s cardiologist says about his results and follow his plan rather than challenging him on it based on what I read on wikipedia or some weird alt-medicine website.
Similarly, like I said on the podcast I did with my husband about neuroscience, I am not a neuroscientist so I need to understand both what I know, but also what I DON’T know so that I don’t make the same errors as Christian authors who also do not have degrees in neuroscience and fail to recognize where their knowledge limitations start.
We need to recognize that we don’t all know everything, and so we aren’t all equally qualified to speak on everything. And that’s OK! That’s why God made us to be a body–some of us are going to spend our time getting educated in engineering, some in developmental psychology, some in plumbing, some in neuroanatomy. And we don’t all need to pretend we’re experts at everything, and the evangelical world would be a lot better if when necessary, we just recognized when we’re out of our depth when it comes to research topics.
I found this explanation of appeal to authority and rather like it; it’s nuanced and gives good examples:
https://ses.edu/logical-fallacies-101-appeal-to-authority-ad-verecundiam/
That’s so good. Bookmarked! Thanks for sharing.
Hi Rebecca, I just found this website and podcast and have been enjoying reading/listening. Thank you for all your great work! I read your research deep dive post about “Is she dressing for attention – or is he?” and found it very interesting. https://baremarriage.com/2022/10/is-she-dressing-for-attention-or-is-he/ I do want to push back gently on a couple of things, however, because I think your discussion of the research illustrates some other points about minimizing bias. So here are two additional points:
1) I am always skeptical when people use leading language to make judgment about data instead of using statistics. You write, “Yes, women do engage in gaze-provoking behaviour. But it’s far less often than men.” We need to know how often women and men engage in this behavior. And then we need to decide what constitutes “far less often.” If say, 43% of women report engaging in this behavior often or very often, compared to 62% of men, is that “far less often”? What if it’s 43% and 52%? Someone reading “far less often” is led to imagine a significant difference when perhaps, if given the data, that reader would consider the difference to be less striking.
2) I always ask myself if the data could be interpreted in multiple ways, to avoid leaps in logic that may confirm my bias but not flow logically from the data. You write that when women engage in gaze-provoking behavior, “it tends to be for a set reason–to find a mate!” If I’m understanding correctly, this is based on the data showing that single women engaged in this behavior more than women in relationships. However, unless the women were asked questions about why they engaged in the behavior, then we are only speculating. It could be that the majority of women engaging in this behavior were looking for a hook-up rather than a mate, or felt pressured by friends to dress a certain way in certain social settings.
Hi, Margaret! I think these are some great comments, thank you for being so respectful in your notes! 🙂
In 1), I think you’re right that the “Far more” or “far less” is open to interpretation. I do believe, having looked at the numbers, that I can argue for the use of “Far less,” but because it is subjective I’m actually going to do a quick edit so that it simply says “less” rather than “Far less”! 🙂
And for 2), this is actually the study author’s perspective, that the relationship change actually does mean that the majority of women likely engage in gaze-provoking behaviours and body-gaze behaviours as a mate-seeking strategy. However, I will happily add “likely” to that sentence to make that more clear! 🙂
Glad you enjoyed the posts, Margaret.
Thanks for engaging with my comment, Rebecca! I appreciate and respect that. 😊
Another one to add is that a single paper rarely *proves* anything – you need to look at the totality of the evidence (as far as is reasonable given the significance of the claim you’re making and other circumstances). Especially true in the social sciences when there are far more factors that can’t be readily controlled than in the natural sciences, and when your study population is humans who are far more difficult to round up in very large numbers than atoms or bees or grains of sand. (Plus psychology has a really bad reputation for publication bias – whether fairly or unfairly I’m not sure).
Also, correlation does not equal causation! E.g. on a similar note to Margaret’s comment my initial thought on the post from about a week ago re women’s first sexual experience and influences on later libido were that it was much more likely that a third variable (or group of variables) was causing both effects, rather than that one was causing the other. Is it more likely that first experience of sex has an enormous impact on libido? Or is it more likely that women with a positive view of sex, good adult sex education, husbands who care about her pleasure etc etc are more likely to have a positive first experience of sex *and* higher libido later in life? I didn’t manage to read the full paper, but from your summary of the data I thought the second interpretation was the obvious one. Interested in your thoughts as you’ve obviously read it properly and I haven’t!
Slightly related: this is an amusing (if slightly bad taste and trigger-warningy) read on correlation va causation. https://www.tylervigen.com/spurious-correlations
On my first point, turns out someone’s done what looks to be a proper (and recent) systematic review on female sexual pleasure that might be worth a look for your series, on the off chance you’re not familiar with it already.
https://www.sciencedirect.com/science/article/abs/pii/S2050052121000019
Yep! And it depends what KIND of study, too! One meta-analysis is better than 4 smaller one-off studies on the same topic! I’m constantly having people send me 2-3 small studies on how spanking is beneficial in response to the massive meta-analyses out there showing that it is negative and they don’t understand why their small 28-case thematic analyses are not impressing me enough to change my mind.
And I understand the third variable idea, and I actually do agree with you, but the issue is that it seems the first time orgasm is the catalyst for the rest of the outcome–so you can have all the other factors lined up in your favour, but if sex starts off badly it’s much more likely to train your brain to think it’s not good for you. So I personally think most likely it’s:
Childhood/educational/relational factors leading to sex makes it more likely orgasm will happen –> orgasm happens –> high libido from the fact that she orgasmed, which was made more likely from other factors.
However, even if she had all the factors that increased her likelihood, if her sexual debut was a letdown or painful, the catalyst wouldn’t go off and her libido would find it difficult to recover. Additionally, there are many women who “should” have a hard time with sex who don’t–who had an awesome time right from the beginning. It might be that there’s a “sensitive period” like the first 3 times you have sex or the first few weeks or whatever it is, I would love if someone could do a more specific study, but I want to make sure we don’t minimize the importance of orgasm on first time for women’s sexual enjoyment, especially since it’s practically guaranteed for men and there is absolutely ZERO reason why it shouldn’t be guaranteed for her. The research shows it is really important, but also we don’t need to do either/or–we can do both/and! We can emphasize the importance of orgasm from the get-go AND increase the odds that it will happen easily, as well, through better sex education, less sexual shame in religious settings, more healthy relationship dynamics, addressing male sexual entitlement, etc. 🙂
Thanks Rebecca. Very interesting!