During the COVID-19 pandemic, the most trusted survey of American life suddenly showed religion falling off a cliff. A new analysis says the drop was mostly an illusion: the survey went online, and the most religious Americans simply stopped taking part.
The paper, by sociologists Landon Schnabel of Cornell University, Sean Bock of Harvard University and Michael Hout of New York University, was published online in Sociology of Religion on April 5, 2024. An earlier version was posted on SocArXiv on October 16, 2022.
A sudden collapse in the 2021 data
The General Social Survey, or GSS, normally sends trained interviewers to people’s homes. Pandemic lockdowns made that impossible, so for 2021 it switched to a mail invitation pointing people to a web questionnaire. The response rate plunged to 17 percent.
The 2021 numbers looked dramatic. Strong religious affiliation and biblical literalism fell lower than ever measured in the GSS, and evangelical affiliation dropped to levels not seen in four decades. Frequent prayer held steady, and frequent attendance moved within normal year-to-year swings.
Inside the 2021 survey itself, the few people interviewed by phone were far more religious than web respondents. For example, 43 percent of phone respondents agreed that America’s success is part of God’s plan, compared with 25 percent of those answering online.
Who said no to the web survey
The key evidence came from a GSS panel. In August and September 2020, the survey’s organizers contacted people who had taken the in-person GSS in 2016 or 2018 and invited them to a web follow-up, using the same approach as the 2021 survey.
That made it possible to compare people who accepted with people who declined, using answers they had already given face to face. Among those who declined, 36 percent were biblical literalists in 2016. Among those who took the web follow-up, the figure was 25 percent, and it was still 25 percent in 2020.
In other words, among people who kept answering, intense religion did not decline. The apparent fall came from who was missing.
Nonresponders were not notably more Republican, older or more opposed to science. The authors describe them as more “disillusioned, distrusting, disinformed, disadvantaged, and disconnected.” They were more likely to be working class, renters, in poor health, lonely, and to think most politicians are corrupt. They had weaker vocabulary scores, read newspapers less and used the internet less.
Once the researchers accounted for those traits, the gap in intense religion between responders and nonresponders disappeared.
A warning for online polling
The authors argue that online surveys with substantial nonresponse will tend to underestimate intense religion, even when they use population sampling methods. As more surveys move online, they warn, the data could create a false impression of secularization.
They also note that earlier GSS response rates had already been slipping, from 69 percent in 2014 to 61 percent in 2016 and lower in 2018.
Study Details:
- Title: Switch to Web-Based Surveys During COVID-19 Pandemic Left Out the Most Religious, Creating a False Impression of Rapid Religious Decline
- Authors: Landon Schnabel, Sean Bock, Michael Hout
- Journal: Sociology of Religion
- Publication Date: April 5, 2024
- DOI: 10.1093/socrel/srad061