Have you ever seen someone just totally twist a study to prove excatly what they want it to say?
Then you read the study and you just go–hold up. That’s not it at all!
We dealt with this back when we reviewed a post by Josh Howerton talking about how gender-roles-based religion leads to the best sex, despite the research he quoted showing… not that. (If you missed it, read it here.)
I (Rebecca) wanted to jump on the blog today by expanding on a newsletter I wrote back when we released that blog post. Here are 5 questions to ask to make sure you’re approaching research with integrity and with as little bias as possible.
This isn’t an exhaustive list, but it’s a start! So let’s go!
1. Does the study’s population match who I’m applying this research to?
Am I reporting on a different population than the study looked at in a way that changes the results? E.g., you may find a study that says 14-year-old girls are more emotionally volatile than 46-year-old accounting technicians, and then use that to say that all women are more emotional than all men. Not honest. The populations are far too different than what was originally studied.
Become a part of the movement
Join 40,00 others and let's change the evangelical conversation about sex
2. Am I looking to validate my personal experience?
Obviously much research will validate personal experiences, but sometimes we don’t match the average so it’s important to address personal bias. For example, my experience giving birth is that it’s a horribly traumatic experience. If I were looking to prove an agenda, I’d get angry when any study found that for most women, birth is not traumatic. But that’s not fair or honest–that’s me working out of my own personal bias.
If we are to be honest when looking at stats, we have to leave room for the reality that we may not be “normal,” and the statistics can teach us rather than describe us.
3. Am I educated enough to properly understand this?
This is a tricky one, but it is important! Am I trained in this type of field enough to be able to understand not only what the article is saying, but also the limitations of what I know?
Am I willing to put aside my ego and acknowledge when someone else should be the one to work through this data? Or am I allowing my desire to “prove” something lead me astray?
And by the way, it’s perfectly acceptable to say to someone else, “I recognize you think that, but neither of us have the education required to really understand this complicated subject. I think we should both refrain from talking about this.”
Think about how much false teaching could have been prevented if the evangelical church had said that the first time someone with theological training tried to teach neuropsychology! (Not trying to disrespect theological training, it’s so important! But it does not a neuroscientist make.)
4. Does this pass the “sniff test”?
Some things sound believable on the first pass, but when you back up and actually think about it logically it falls apart. For example, if a Christian teacher tells you that sending naked photos to your husband will neurologically train his brain to not look at porn anymore, that doesn’t pass the sniff test. Sending sexy photos to a man struggling with pornography isn’t helpful or safe, so that’s likely a sign that this teacher is misusing statistics for their own agenda.
Similarly, anytime someone says that virtually all of men or women do something, that should be met with suspicion. Maybe the study’s question design wasn’t specific enough, maybe their population was self-selected, there’s all sorts of problems. But just know if someone tries to convince you that 90%+ of people are a certain way, that’s likely a sign that there’s something else going on. (Not always, of course! But it’s a sign to take a closer look.)
Now, I do want to say that this, once again, needs to be put through the personal bias filter. The “sniff test” doesn’t mean “Do I find this believable,” it means, “logically does this hold up?” It’s so, SO important to have humility and self-honesty when dealing with research to recognize the limitations of our own understanding as well as our areas of personal bias.
5. Am I being selective in which part of the research I report?
Focus on the Family famously quotes study after study about how divorce hurts kids, but fails to quote the parts of those same studies that state divorce protects kids when there is abuse present. No one is saying divorce doesn’t hurt kids! We’re just saying it’s not the whole picture: often divorce is a life-saving answer.
When we report research it’s important to ask: am I presenting all of the relevant information, or am I conveniently omitting the stuff that goes against what I’m trying to prove?
So there you have it. Five questions to ask. Using statistics is complicated, but research is such an amazing tool to help us find truth and set people free.
Do you have any helpful questions you like to ask about research? Like I said, this is not an exhaustive list! Let’s chat and give more tips in the comments!
The Research Deep Dive Series
- Does Sex Start in the Kitchen or Does It Die There?
- How Does a Couple's First Time Affect Libido?
- Is "All Men Struggle With Lust" a Primal Fear?
- Is She Dressing for Attention or is He?
- PODCAST: Ogling and Dressing for Attention
- 5 Questions to Ask That Minimize Bias When Discussing Research
- 10 Things to Know About Hormones and Libido
- Bring Back Vanilla Sex