There’s been a big uptick lately of men especially on social media accusing us of having biased research in our study of 20,000 women.

I don’t normally address these issues from the blog, because, quite frankly, our work speaks for itself.

1. We did the largest study of evangelical women’s marital and sexual satisfaction that has ever been done, and we followed it up with three other very large studies.
2. We determined to do the studies to academic standards, even though we have no backing from universities, no funding, and no ability to get grants.
3. We submitted our work to peer review, and we have three articles published so far, another accepted, and multiple other ones in various stages of the submission process.

In summary, it’s pretty much unheard of for citizen scientists, so to speak, to be published in peer reviewed journals, but it’s really unheard of for evangelical authors to write books and then get the secular world to recognize the quality of our research.

I’m very, very proud of what we’ve done, and in awe of my co-author and stats expert Joanna Sawatsky.

We’ve had these three articles published about our work:

• Sanctified Sexism: Effects of Purity Culture Tropes on White Christian Women’s Marital and Sexual Satisfaction and Experience of Sexual Pain (Sociology of Religion)
• The etiology of sexual pain in American Christian women: associations of sexual debut, maternal teachings on sex, and abuse history with pain severity (Journal of Sexual Medicine)
• It’s Not Just Fashion, It’s Sin: The Impacts of Modesty Messaging on Female Adolescents Within the White Evangelical Church (Journal for the Scientific Study of Religion

The first two are based on our dataset for our book The Great Sex Rescue, and the third on our dataset for our book She Deserves Better. We’re in three different journals with very different disciplines (Sociology vs. Medicine), which, again, is virtually unheard of. Our upcoming paper that isn’t out yet will be a qualitative analysis of our open ended questions around sexual pain, so we’ve done both qualitative and quantitative research. And again, we’ve done it all without university backing or funding (though our collaborators in two papers are affiliated with universities!). We’ve been able to do this because of the people who donate to us through the Good Fruit Faith Initiative of the Bosko Foundation (you can donate too to help our work!).

But at the same time, the critics have been coming out in full force, and I hate having to answer them in the comments. And so sometimes what I do in cases like that, where an issue keeps coming up repeatedly, is I write a response so that I can just drop the link in the comments from now on. Everything else is just too tiring and time-consuming.

I did it when people kept saying to me “you should have followed Matthew 18” instead of just critiquing other authors, or that I’m not doing things in the kingdom way.

So I’m going to address this again today, covering a few points quickly myself, and then handing it over to Keith (with Joanna’s input!) for a longer explanation before I wrap it up.

Accusation: “She got all her survey takers from her blog and they all believed just like she does!”

This is actually laughable on its face, and it shows how little they understand about the research we did.

First, the claim is patently false. Over 50% of our respondents for our largest survey didn’t come from my website or social media channels at all, but from others (including 87 influencers, some of whom we even critiqued in our book) sharing our link.

But second, and far more importantly: we couldn’t even have done our research if everyone thought just like me!

The whole point of our study was measuring the effects of beliefs on marital & sexual satisfaction, which meant that we had to compare two groups of people: People who did believe something, and people who didn’t.

If everyone believed the same way, we would have nobody to compare them to!

I was talking with my husband Keith (who is a physician, and the co-author on our peer reviewed papers) about this, and here’s what he said to me:

I think people saying “your sampling methods are biased” is an example of people not knowing enough to know what they don’t know.

Not only do they not know the scientific meanings of words like “bias” and “significant”, they don’t realize they are erroneously substituting the colloquial meanings for these words.

The word bias in regular speech means to have an assumed animosity against a certain belief or idea whether intentional or subconscious. This is also called implicit bias. For example, a person who believes in complementarian theology might be biased against a certain author because they have stated they are egalitarian.

Sampling bias in statistics, on the other hand, is a distortion of data based on the way it is collected. It is a technical term and does not have the same meaning as the colloquial use of the word.

Snowball sampling and convenience sampling, which is what the team used for the survey for the Great Sex Rescue, are prone to sampling bias (as are virtually all social science studies) and this is well known in the literature.

I suspect people are just googling snowball sampling or convenience sampling and seeing the word bias and then feeling they can dismiss your work as biased in the colloquial sense. (Ironically, they are thus showing their bias).

Snowball sampling’s bias, because it is referral based, comes from the fact that the participants all come from the same network. A similar problem happens with convenience sampling–the people who respond may not represent the population in some way–maybe the people who respond to a survey from an ice cream company about their favourite treat will be more likely to select ice cream. This means the group is likely to be relatively homogenous and lack diversity. This can be a problem as it limits the generalizability of the studies’ findings.

Hence the peer review process! Someone from outside looks at your work and determines if it meets statistical muster and they may even suggest ways to mitigate that bias.

For example, the sample produced for GSR was over 92% white. This is an example of (statistical) bias since obviously other groups were under surveyed. That doesn’t mean the results are invalid; it just means that our results are less likely to be generalizable to non-white women. Basically, you didn’t get enough non-whites to be sure what you found applied to them, too. The reviewers noted this and that’s why the title was changed to “Effects of purity culture tropes on white Christian women’s marital and sexual satisfaction”.

This kind of thing can also vary by discipline–we restricted our sample in our first paper to white women only because of advice from a tenured prof at a major research university in Sociology, but did not do the same for the Journal of Sexual Medicine. (We’ve also submitted to sociology journals including non-white respondents – all of this varies a lot and how to mitigate potential bias is a huge deal and people argue a lot.) But the broader point is that you note the bias and then experts in that area either decide if the sampling methodology and findings are good enough to merit inclusion in the journal in question. And we’ve been published. Repeatedly. And in the highest impact journals in their fields.

Side note: It’s interesting how you caught heat from this, too, because people assumed you thought only white women experienced bad effects of purity culture. So the one element of (statistical) bias in our work was addressed and people were mad about it. You can’t win, it seems.

Were you actually biased towards people who agreed with you?

The implication from the people claiming our studies are biased seems to be that not only did the sample lack demographic diversity (which we have admitted and cited as a limitation), it lacked diversity of thought (typically worded “you just got a bunch of people who agreed with you to do the survey.”)

Now with snowball or convenience samples (really any non-probabalistic sampling methodology), that can be a problem, but using the example of the obligation sex message we can see that in fact that this was very unlikely, especially for the kind of questions we are trying to ask.

First, you were not asking people’s opinions. Your critics make it sound like you asked a bunch of people, “Did the obligation sex message ruin your marriage?” and since they are all your followers, they said, “Yes.” But this is either misunderstanding or deliberately misrepresenting what was done. The study asked standardized questions about marital and sexual satisfaction then asked about belief in certain ideas then looked for correlations between belief and outcome.

If your critics are right, no one you surveyed would have reported believing the obligation sex message since they all agree with you that it’s bad. Now it really would have been (statistical) bias if the sample skewed toward unanimity of disbelief in the toxic teachings, but it did not. In fact, it was only because you had large numbers of people in the survey believing each of the toxic teachings that you were able to see a pattern between belief and result. To put it the way you have repeatedly said: “if there was no differences in belief we’d have had nothing to measure”.

Additionally, we have had people act as though our studies having limitations is a kindof “gotcha” moment when… all science has limitations.

The name of the game is to name them and work within the constraints that they create.

The other thing they don’t seem to understand is (statistical) significance.

Again I think the colloquial meaning of “importance” is getting mixed up in their minds with the mathematical definition. When we say a statistically significant correlation was found between people believing the obligation sex message and reporting lower satisfaction scores, it means people who reported believing the message tended to report lower scores, and that mathematically we calculated that the chance of that being random dumb luck was highly unlikely.

Imagine you surveyed two people. One believed in obligation sex and had a terrible sex life and another disbelieved it and had a wonderful sex life. You still haven’t proved anything because it could have been random. To achieve statistical significance, you have to have sufficient numbers of people in both groups to show they are different (this is called statistical power). So in our case of two people, the math would show you this could easily be random chance. Similarly, if you had a thousand people who believed one way and only one that believed the other, the math would still show you that could have been random chance. It’s all about having enough in each group to be mathematically convinced this is not random. If you don’t have that, the “math doesn’t math” and you can’t generate a statistically significant result.

So in this case, even if the vast majority of people in the sample “believed like you”, the math would only show a statically significant result if you also had enough people who “disagreed with you”. (Though in our sample, there were actually more people who disagreed with us than agreed with us!).

You did show statistically significant drops in marital and sexual satisfaction associated with these beliefs because you had the numbers. They just don’t like what you found so they want to attack it however they can.

In effect, what your opponents are saying is they believe you somehow magically managed to gather together in the same sample all the people who believe like you and by chance happen to have good marriages and all the people who believe like them who by chance happen to have bad marriages. Obviously, that’s ridiculously unlikely. But if you don’t want to change your mind about something, apparently you can convince yourself of practically anything.

Exactly. Thank you!

One other point that I want to make:

People misunderstand what “limitations” mean.

People have been quoting our limitations sections from our papers as a “gotcha” moment. I think they don’t understand what limitations mean, either.

One of the sections of most peer reviewed papers is “limitations.” Every paper has specific headings and sections–abstract, introduction, methods, discussion, limitations, suggestions for future research (and often there are a few more). This is standard. Every paper has to have these.

But people are pointing to the fact that we even have limitations and suggestions for future research to say, “see, you didn’t do a good job! You said that more research is needed!” Or they’re saying, “see, your work has limitations so we can’t trust it!”

But every single paper has these sections. Every single paper has limitations and suggestions for future resarch. This isn’t a signal that we’ve done something wrong, but rather this is how research is done. Scientific inquiry is an ongoing dialogue. Things are never done. That’s the whole point!

But I want to zero in on limitations for a second.

When we say that our results are not necessarily generalizable, that doesn’t mean they don’t apply to the popuation we studied.

We did not use a random survey of the general population, because our population of interest was not the general population. We wanted to know how these teachings affected evangelical women, not how they affected women in the general population, because that’s not who we were studying. We were looking at predominantly white evangelical women in the United States, with quite a few respondents also from Canada, the UK, Australia, New Zealand, and South Africa (and some from other countries as well).

We had huge numbers of white American evangelical women. When we say that our surveys weren’t necessarily applicable to other populations, what we’re saying is that we can’t say with certainty that the same results would be found for non-white evangelicals, or for non-evangelicals, or for secular Mexican women in high powered jobs, for instance.

We’re NOT saying that these results aren’t applicable for the white evangelical population. Just because results can’t be conclusively said to be applicable to everyone does not mean that they are not applicable to the population we studied. 

And how do we know they’re applicable for the population we studied? Well, we had a huge and diverse group take our survey. Our findings are completely in line with other peer reviewed papers (so we have external validation). But even more importantly: the peer reviewed process said we did this well.

Here’s how one commenter responded on Facebook yesterday:

It’s not about distrusting our research; it’s that they don’t want to believe it

There’s also blatant hypocrisy going on here. They say they can’t trust our results, but they continue to believe things  (like men need respect while women need love, or that complementarian marriages do best) when there isn’t actually backing for what they believe.

They believe authors who have not done studies, or authors whose studies would never pass the peer reviewed process, but they choose to say they can’t believe our research because…reasons.

It’s not about looking for good research; it’s about choosing not to believe us because they don’t want to.

Which leads me to my final question:

What are they so upset about in our findings?

When people try to discredit us, I have to laugh, because what is it that threatens them so much?

Our main findings from that survey are that:

• The obligation sex message plays a big role in sexual pain disorders and lowers orgasm rates and libido;
• Ditto with the idea that a wife should have frequent sex to keep her husband from watching porn;
• The “every man’s battle” message kills libido;
• Teaching teen girls that “boys will push your sexual boundaries so you need to be the gatekeeper” plays a big role in making arousal much harder later in life.

So what’s up? Do they LIKE the obligation sex message? LIKE every man’s battle? Do they believe that a husband needs to have sex with his wife or he’ll watch porn? Do they think the orgasm gap isn’t a big deal? Why do they want to keep these teachings?

Me thinks they protest too much.

Our Research Was Done Properly.

We have always said, from the very beginning, that we were going to do this to academic standards. We have always said that we were going to do this right. We were going to raise the bar for what counts as research in evangelicalism, and urge people to not just believe something because a pastor said it, but to demand that pastors and authors start using evidence-based advice.

If you ever see someone critique our research, just send them this page. And remember: they’re not upset about the quality of our research. They’re upset about the findings. And they don’t know enough about research to understand what they don’t know!

And if you want to help people see what good research looks like, how about gifting our books to someone who needs them? Get them for your pastor, your women’s ministry leader, your church library. They’re all based on proper studies, and they’re all evidence-based. And, quite frankly, they’re a breath of fresh air!