Everyday I get more questions about our survey, and about our data, and what we’ll be studying next!
I thought I’d try to answer that today in one central place.
We’ve done the largest and most comprehensive studies of marriage and sex on evangelical populations, and I know there’s been a lot of interest in our data.
Here are the surveys we’ve done:
- 20,000+ women that formed the basis of The Great Sex Rescue (at least 130 questions)
- 1,800 women (part of the 20,000) in a follow-up survey (20 questions)
- 3,000 men that formed the basis of The Good Guy’s Guide to Great Sex and The Good Girl’s Guide to Great Sex (along with the other survey data) (at least 90 questions)
- 7,500 women that will form the basis for our mother-daughter book, looking at how teenage experiences at church impact self-esteem, future relationships, and well-being (at least 110 questions)
We also are planning a completely different kind of survey for a marriage book coming up, but I’m going to keep that one under wraps for now!
I frequently get asked if we’ll look into other things--for instance how evangelical women or men compare with secular women or men–and while we’d love to, we have no specific plans for that. If I were to do more research, I’d really like to focus more on sexual debuts, and double down more on how the earliest experiences of sex affect one’s sexual satisfaction over the lifetime, so that we could teach couples better about how to start.
We likely will help a group of friends do a survey on divorce soon, but we ourselves don’t have a lot of plans to do too much more, because we’re already sitting on so much data that we haven’t mined yet.
How is the academic world receiving our data?
We have ethics approval from Queen’s University in Kingston, Ontario, to submit peer reviewed publications. We’re currently working on two big ones in collaboration with other academics: One with Physical Therapy professor Lori Mize from the University of Southern Arkansas regarding our findings on vaginismus, and one with Andrew Whitehead of Purdue University on our findings as a whole from our women’s dataset.
Those papers are on hold because we have a book deadline of April 1, plus Joanna is moving from the Arctic to Edmonton, but we’re hoping to get going on that in May!
In the meantime, we have presented at the American Physical Therapy Convention in February, and The Great Sex Rescue was favorably reviewed in the October issue of the Journal of Women’s Health Physical Therapy.
The American subset of our women’s dataset, scrubbed of all identifying details, is also available at the ARDA for other researchers and journalists to use. Joanna had to write 1500 lines of code to get it ready, so that’s pretty amazing!
“Can I See Your Dataset for Myself?”
We get this question a lot, too, and one of the authors that we have been critiquing has also said that she is planning on reviewing our data to make sure we did things properly.
I think, quite frankly, that this represents a deep misunderstanding of how academia works, and also the nature of the dataset that we have.
1. We aren’t ethically allowed to share the dataset.
Part of getting ethics approval to pursue peer review is that we keep our dataset under wraps, because there is specific identifying information in it. We had to promise that we would keep it secure, password protected, and on computers rather than in the cloud. We put only a subset of our respondents and variables inot the dataset that went to the ARDA to protect confidentiality. If anyone wants to look at our dataset, that subset is what we can share ethically and it is already available on the ARDA
2. Our work is a “study”, not just a “survey”, because our primary findings are about not just frequency stats.
Let me explain the difference using Shaunti Feldhahn’s work as a comparison. Shaunti surveyed 1000 people in different nationally representative surveys (so she did not survey only married people or only Christians) for her books For Men Only, For Women Only , For Young Men Only, For Young Women Only. Her survey was only frequency numbers. She’d ask a question, like “do you prefer to be alone and unloved or inadequate and disrespected?”, and then she’d report how many answered each way. But she never took people who answered a certain way on one question to see how that affected their other answers.
For instance, when we tell you that women who believe the obligation sex message have a higher risk of vaginismus, those aren’t frequency stats but odds ratios. We have to look at the chance of vaginismus overall, and then the chance if you believe the obligation sex message when you get married, and then the chance if you don’t believe the obligation sex message when you get married. So we’re looking at just a subset of the data to see how things affect one another.
That’s a whole different level of analysis. You have to know how to measure confidence intervals, and how to tell if something is statistically significant or not. That’s what makes it a “study”.
3. Without a full stats education, you wouldn’t even be able to do anything with our dataset.
Again, this is not a simple frequency survey. This is a full-blown study. And as such, we’re using high level stats software to get our results. Joanna has written thousands of lines of code.
When people ask to see our dataset, Joanna is a little flabbergasted, because here is what the dataset looks like (this is a snapshot of non-identifying bits):
Think of twenty-thousand lines that look like that, spanning over 130 columns.
To even get it usable, Joanna writes lines of code using our operational definitions, which we’ve decided on beforehand after a thorough literature review, and it looks like this:
Once Joanna writes some code, she can get it in this form:
What people want, I think, is to see the dataset in pretty tables with all of Joanna’s code. But you see–we’ve done that. It’s in the books. And so that honestly is the best place to see our dataset. Here’s a chart from The Great Sex Rescue where we found a way to make it easier to interpret those tables:
Women Who Feel Their Voice Matters in Marriage Report Better Sex
When people say that they want to evaluate our dataset, and they honestly don’t know how to use stats, they’re actually being quite rude to Joanna, to be frank.
I couldn’t do the stuff Joanna does. This is high level stuff. For instance, here is a quick tutorial from Princeton University on how to use the main software that would be required. In order to be able to do anything with our dataset, this would have to look intuitive and obvious to you.
I know that the authors who are critiquing us and saying that they will look at our dataset and explain what we have done wrong do not have the education to understand Stata. So when you hear people saying that, please remember that this is a whole new level of statistical analysis. The only people qualified to speak on our research are people who understand this level of analysis. And we have already passed those tests with our acceptance in the ARDA and our work on the presentations with vaginismus.
4. Putting our dataset in the ARDA was a huge deal
I’m not sure if people understand this, but having this amount of data that can be worked with is actually worth a fair amount of money. Any one of the three of us could walk into a university with this data and be virtually guaranteed to get into a Ph.d. program, with funding. In fact, we’d have funding for the rest of our lives.
But we put it up at the ARDA for others to use because we felt it was important enough that other people get to work with our data too.
However, it is up at a place where academics, who know how to handle this kind of data, can access it. That’s the proper way to do it.
Will we be looking for other partnerships?
We would actually love some partnerships, because we’d love to get more peer reviewed papers out, but we only have so much time, and Joanna, who is our expert, is also the stay at home mom to two toddlers. So time is limited.
We would specifically be looking for professors or grad students who are willing to take the lead on some of this, and who are familiar with STATA, SPSS, SAS, etc, or at least would want to do literature reviews. We also have qualitative analysis that could be done on our focus groups, but again, it would have to be someone affiliated with a university working towards a peer reviewed paper. If you know of such a person, just let me know!
What if you’re NOT the problem with your sex life?
What if the messages that you’ve been taught have messed things up–and what if there’s a way to escape these toxic teachings?
It’s time for a Great Sex Rescue.
I do get asked about all of this quite a bit, and I wanted to have a quick post that we could send people to in order to understand what we’ve done.
And can I just take a moment to say how utterly in awe of Joanna I am?
Like seriously, she is wickedly smart. Rebecca and I ask her questions on FaceTime all the time, and she’ll be getting the four year old a sandwich while bouncing the toddler, and she’ll start mumbling totally incomprehensible stuff about chi-squares and coefficients, and then ten minutes later she’ll have an answer for us.
She’s really good at this. And the reason is because she has a ton of education in it, and she has been involved in multiple other studies and authored papers using this kind of statistical analysis before.
God sent her to us at just the right time, and we’re very grateful.
Joanna’s not on the blog side much, though we do talk to her several times a day. But if you want to get to know her better and ask her questions, the best way is to support us on Patreon!
It was our Patreon supporters who paid for the STATA software this year (we have to renew that every year!). It’s our Patreon supporters who are paying for Joanna’s time while we get these peer reviewed papers ready. Because we’re not associated with a university, we don’t have a way to get funding like most researchers would. So it’s your support that keeps us going, since we can’t monetize writing peer reviewed papers or writing thousands of lines of code to get in the ARDA.
You can support us for as little as $5 a month! And our private Facebook group is awesome, and we have unique merch, and more! Keith even started giving us $5 a month just so he could join the Facebook group because Rebecca and I were always talking about it!
So thank you for your support! I hope that makes what we do a little bit more understandable. And if you have any other questions, leave them in the comments!
Very educational thanks!
I think your work is fantastic and honestly, conforms to what feels like common sense to me.
However, how do you respond to those who say “correlation does not prove causation”?
My understanding is that to DEFINITELY prove causation (these particular teachings cause these particular bad outcomes) you would have to do a prospective study.
I also realize that big tobacco avoided blame for lung cancer over many decades using the same “correlation is not causation” argument, yet eventually I think it was the Framingham study that eventually proved it. Thanks so much for all you are doing!
Great question, and maybe Joanna can jump in! Especially for our mother-daughter book we were looking at longitudinal issues–so we know what preceded what. But, yes, it’s difficult to tease out exactly what is correlation and causation.
We can say, however, that vaginismus is highly correlated with obligation sex message beliefs (which is what we do say), and this is really the nature of odds ratios. We’re not saying that one thing necessarily causes the other, but that if you do have belief A, you’re X times more likely to experience B, if that makes sense. So we can see that certain things are far more correlated with negative outcomes than other things. Again, that’s the beauty of odds ratios.
There are lots of ways to make causal inquiry and one is to have different time points – so if you know the exposure preceded disease and the exposure is correlated with higher disease burden then you’ve got a potential cause.
For some of our measures (belief now vs outcomes now) we can’t tell from the stats alone which way the causal pathway is moving.
However, when we look at high school beliefs (etc) and then at outcomes now, we do actually have evidence for causation because of the temporal separation between our putative causative agent and the effect in question.
Hope that helps
In my final term on my way to an undergraduate degree in engineering, I had to take one more non-engineering class in the college of arts and sciences. I chose an introductory statistics class, and it was fairly incomprehensible (and that’s compared to two full years of calculus, plus classes in Laplace transforms and some other higher math topic whose name I can’t remember thirty-five years later).
The only problem I remember from the stats class is how big a group of people you need to have a 50-50 chance of two people having the same birthday (just month and day, not including the year). Maybe Joanna would like to chime in to show how that particular problem is solved, and for everyone else, it’s NOT a group of 182 (i.e., half of 365).
So yeah, stats are complicated.
One thing I kept thinking as I read TGSR, it would have been nice to have a paragraph or two in a preface that explained exactly how to understand the numbers that were going to appear in both the text and the graphs.
So when the text says that there’s a 62 percent higher chance of X or when a graph shows that there’s a 9.3 times higher chance of Y, what do those numbers actually MEAN? Because if I understand those two examples correctly, “9.3 times” is equivalent to “930 percent.” And likewise “62 percent” is equivalent to “0.62 times.” Right??? Which means “9.3 times” is WAY higher than “62 percent,” even though “62” is a lower numerical value than “9.3.”
Or have I botched the meanings completely???
Hi Jo! It’s 23 people, if I did my math right. Simplified explanation:
Abby has a 22/365 chance of sharing a birthday with any other person in the room. Conversely, she has a 343/365 chance of not sharing a birthday.
Brian has a 21/365 chance of sharing a birthday with anyone in the room (we already counted the possibility of him sharing with Abby), and a 344/365 chance of not sharing a birthday.
Claire is a 20/365 chance of sharing with anyone who isn’t Abby or Brian, and a 345/365 chance of not sharing a birthday.
So you multiply the probabilities that no one shares a birthday (Abby doesn’t, Brian doesn’t, etc.), subtract from 100% to have the chance that a birthday is shared.
As I recall, it’s not the number of people in the group that you count (hence why it’s not 182—or 183 if you want to cover a leap year!).
What you need is 182 (or 183) PAIRS of people. (Because 182 divided by 365 gives you 50 percent.)
So you need however many people taken in unique groups of two. That’s a stat called a combination, and there are a zillion combination calculators on the internet.
There were maybe 40 people in our class, so if you use a combination calculator for 40 and 2, you get 780, which is way too high.
C(30, 2) is 435, so still too high.
C(20, 2) is 190, so getting close.
C(18, 2) is 153, so not enough.
C(19, 2) is 171.
Since we can’t take fractions of people 😁😁😁, then in a group of 20 people, there’s a (190÷365) = 52 percent chance of a shared birthday.
We went one by one around the room, saying our birthdays, and we hit a match less than halfway through. 🎉🎉🎉
My brain can barely follow the comments in this much less comprehend Joanna’s answers or all of what’s in the post, lol, but I sure am glad y’all are doing this work and sharing it with others! It takes many parts to make up the body and I’m thankful you are using your smarts to do good, godly work. Thank you.
Ok, wow!! That’s intense and amazing!! You are all amazing and Joanna is a superwoman!! 😁
Thank you for explaining-this makes a lot of sense.
I used STATA during undergrad to run econometrics and economic statistics. I had traumatic flashbacks from that one screenshot! 🤣 MAD respect to Joanna because STATA is no joke!
Hi, I was wondering if you ever saw the book “I Am a Church Member” by Thom Rainer. It is very brief; but I had a lot of unsettled feelings reading it. It seemed to push an abuse of authority; harmful ideas about why pastors have moral failure; and there was nothing about church being a safe place. I know you are busy but I feel like a troublemaker for having the views on it I have and I’d appreciate some feedback. Thanks.
I haven’t read that book myself, but The Wartburg Watch has written several posts over the years about Thom Rainer and his views on authority and hierarchy in church, and let’s just say there’s some major concerns:
Thank you, Sarah, I have read all these and they are very helpful!
Thank you for sharing this! I would love to be a part of this, I have basic knowledge of SPSS and analysing psychological research data and am currently studying at university for a degree in counselling, could we chat some more of this is something I could help with?
Do you have any experience in qualitative research? We have a treasure trove of interview and focus group data (as well as long form answers to our survey). I’m never going to be a qualitative researcher so we’re also really looking for collaborators on that side of things.
This is not pertinent to today’s post. I am wondering if you might do a reaction post to Francine River’s “Redeeming Love.” I read it years ago when it came out and disliked it, and I was surprised that it came out as a film this year. I have not seen the film and have no interest in seeing it, but I recently saw a series of videos talking about the story on youtube. In reading some of the comments, I started thinking about the less obvious places bad teaching has come from. I.e., Christian women’s fiction. Honestly, thinking back to some of what I’ve read (and I’m not even a fan of Christian chic lit) I realize that so many damaging concepts were presented as “fact” in the fictional universe. Francine Rivers generally, and “Redeeming Love” in particular has had an almost cult following (the Christian “Twilight”?) that I have never understood. But I think even disliking the book, some of the assumptions in the plot stayed with me. Do a series using your rubric on teachings to critique Christian fiction!
Hi Noel! I did write about Redeeming Love a while back! I had some concerns as well.
I have no idea how I missed that! Thanks!
When I rededicated my life to Christ 20 years ago, I devoured Christian romance books as the alternative to secular romance. I read Redeeming Love and thought it was a great book which made me more curious about the book of Hosea. Well, now that I look back on it and after some recent posts about this Francine Rivers’ novel, I realize the troubling content such as Michael Hosea being insistent toward Angel and trying to dominate her. How could I not miss this nearly 20 years ago, especially after I left an abusive marriage?
The thing about these Christian romance novels that now bugs me more than ever is the message that a woman is not complete until she marries the right Christian man. It doesn’t matter that the male character barely became a Christian, their dating relationship hasn’t been six months and they suddenly get married, one or both of the characters are not emotionally or spiritually healthy, but when “love” enters the picture, they are suddenly healthy, etc. I could get on my soap box about these “Christian” or inspirational romance novels which are the equivalent of secular Harlequin novels. Cheaply made with little substance.
Just want to say, I love the backstory. My educational background is in basic science and healthcare so I have some familiarity with how huge studies like yours are typically done. It sounds like you guys basically crowdsourced a passion project that could well have been a landmark sociological study if it was done in a university setting. That’s a HUGE achievement!
Thanks so much – that is a HUGE compliment and I’m really moved!
In college I took a Stats class and a Research Methods class for my BA. Both were fun (yes, even Stats – my professor was amazing) and I would have loved to go deeper into both, but that was not the direction I was going in my career.
One thing I know from taking those classes is that the kind of research you three have done is far above what most people understand. With the background I do, it boggles my mind. The research studies I did in college had so few variables, but it was still complicated and challenging.
You guys are 🏆🏆🏆
That’s really kind of you to say!!! Really appreciate it!
Thank you so much!
I just wanted to jump on the “praise Joanna” train…I love the fact that someone is doing this! Your work is SOOOOOOO important and, as someone who hates math to the core of their soul and sees numbers as a foreign language, you have my deepest respects! I’m the exact opposite…I just changed my major and am also now minoring in reading instruction; I like words and books as opposed to numbers and charts. I had no clue what all went into all these stats and I’m so impressed that you commit so much time to this when you are a mom and living life! Keep using the skills that God gave you to help others!
I don’t pretend to understand the more complicated stats, but I LOVE the respect for an evidence-based approach and respect for expertise.
I know that Joanna is super-busy now, but any thoughts to pursuing a PhD at some point?
HAHAHAHAHA. Let’s just say it’s under discussion again!
Argh!!!! SAS PTSD! Back in 1999-2000 I was using SAS on dial up and it was a non-windows based zoo as I recall. But it worked. Ya, generally speaking you want to avoid calling your audience and critics too dumb to understand, but when you start playing around with STATA and SAS stuff, its not a territory to get lost in if you don’t have the right education. Hats off to you Joanna!
Thanks, Chris! I can’t imagine doing all the stuff on dial up…but i remember those days!
You all did really amazing work.. Joanna’s statistics work is exceptional. As a survey respondent, I am very thankful that you are preserving our privacy by not putting all the data out there.
Even basic stats is something most people don’t understand, and it can be very counterintuitive. (See above about chances of people sharing a birthday.) Simpson’s Paradox turns people’s brains into mush. That’s before you get into all of the conditions probabilities and coding Joanna did. No, you don’t just look at this stuff and have a result pop out.
Now, Jane, you know at least a few of us will go look that up. All I can say is