You Only Get Answers to the Questions You Ask

Understanding the Pew Report on Jewish Americans
June 4, 2021Ari Y Kelman

I was fortunate enough to have served on the advisory committee of the Pew Report on Jewish Americans in 2020, so I know it well. And I have great respect for my colleagues who served with me, but that does not mean the report is above criticism.

If Jewish leaders are to use its findings to shape planning and action, they will need to know how the survey itself – the instrument, the analyses, the reporting – are part of the apparatus through which the American Jewish community can become visible.  In other words, if we are to understand the meaning of the Pew report, we need to understand how it was made.

To help in this effort, I'm going to talk about three relevant social science principles and how they pertain to Jewish communities:

1.You only get answers to questions that you ask.

With respect to survey work, there are two things you have to know. First, you only get answers to questions you ask.  Second, those answers are usually selected from a list of options. Surveys are great at making order from the messiness of everyday life, but that process can distort the phenomena that the survey is supposed to represent.  For example, there's a question on the survey about whether or not a respondent keeps kosher. Coming from the Reform Movement, you understand that there's a variety of ways in which one can keep kosher, but on the survey, respondents are asked if they keep kosher at home.  They are given three response options:

  • No, I don't keep kosher.
  • Yes, I do because I'm vegetarian or vegan (incidentally, that's about 3% of American Jews).
  • Yes, I keep kosher and I separate meat and dairy.

Some of you may be thinking, “I eat kosher meat, but I don't separate meat and dairy,” or “why do you only care what I do at home?”  or any of the other combination or re-combinations of kashrut observance that people follow in their own way for whatever reason. There's no room for those in the construction of this question.

Highly structured questions like this one are great on surveys because it allows us to divide the Jewish population of America neatly into three groups: vegetarians and vegans, people who separate meat and dairy, and people who don't keep kosher. That's analytically useful, but it doesn't quite get at the complicated ways in which people live their actual Jewish lives.  People don't typically live their Jewish lives in ways that fall into neat buckets that can be framed out in a survey question.

2. The questions and provided responses define the terrain of Jewishness.

The questions shape the way we come to understand what Jewishness and Jewish people look like in the United States. Let’s consider how the Pew study engages with the complicated concept of religion. Pew relies on two big categories: Jews who say that Judaism is their religion and Jews who say they are Jewish, but not by religion. To be categorized as a “Jew not by religion” is to answer “no” to the question “Are you Jewish by religion?” If you answer “yes” to some other way of being Jewish -- ancestry, parentage, whatever -- you are classified as Jew, but not by religion.

Dividing Jews into these two categories is analytically useful but descriptively problematic when we take a look at how respondents answered other questions that might bear on their relationship to religion:

This I think is particularly important for the Reform Movement in which religious observance is not a qualification for belonging, affinity, or participation. 

  • 8% of Jews of no religion say that religion is either very or somewhat important to them.
  • 7% of Jews of no religion believe in the God of the Bible (which is a pretty stringent theological statement).
  • 48% of Jews of no religion say they believe in a higher power.
  • 33% of Jews by religion say they believe in a God of the Bible
  • 14% of Jews by religion don't believe in God or higher power at all.
  • 57% of Jews by religion explain their lack of attendance at synagogue by saying, “I’m not religious.”

So, you have Jews by religion who say “I’m not religious” and you have Jews not by religion who have theological and practical commitments to elements of Jewish life that could reasonably be described as “religious.”

As a result, the category of religion sits rather uneasily in the survey itself and does not capture the complexities of American Jewish life as something that is not easily defined as “religion.”  

3. The lens of the study helps brings some things into focus and distorts other things in the process.

Only by looking under the hood of the Pew report can we better understand how to use it. We have to know what it is and what it is not. Perhaps even more importantly, we have to understand how it tells us what it tells us. Only then can we look at the numbers for what they are and how they are as much a product of communal concerns as they are reflections of lived social realities. The process by which everyday life is translated into a survey question and then presented as tables, graphs, and data necessarily transforms the thing that it is trying to represent. Until we can account for that process, we risk misrepresenting the data and misunderstanding American Jews in the process.

 

Related Posts