Nonresponse bias is the error that arises when the people who do not answer a survey differ, on the thing being measured, from the people who do. Robert Groves’s 2006 synthesis found that “the nonresponse rate of a survey alone is not a very good predictor of the magnitude of the bias.” A low response rate raises the risk of bias but does not prove it; what matters is whether responding is related to the answers.
Definition
Groves explains that the bias of an unadjusted respondent average depends on two things: the nonresponse rate and the difference between respondents and nonrespondents on the measure.1 Under a second formulation, “nonresponse bias is a function of how correlated the survey variable is to the propensity to be measured”.1 Put simply: if the people who answer are, on the question asked, just like those who do not, a low response rate causes little bias; if answering is tied to the topic, even a high response rate can leave serious bias.
Origin and history
Concern about nonresponse grew as response rates fell. Groves’s article opens: “Many surveys of the U.S. household population are experiencing higher refusal rates. Nonresponse can, but need not, induce nonresponse bias in survey estimates”.1 Pew Research Center reported that “in 2017 and 2018, typical telephone survey response rates fell to 7% and 6%, respectively,” while earlier Pew studies “found little relationship between response rates and accuracy”.2 Pew’s summary of the evidence is careful: “A low response rate does signal that the risk of error is higher than it would be with higher participation”.2
Groves also posed the question that framed later debate: what advantage probability surveys retain “if they suffer from high nonresponse rates,” answering that it depends on design and on “the quality of the auxiliary variables” available for adjustment.1
How nonresponse bias is measured and reduced
Because nonrespondents did not answer, their views cannot be observed directly. Researchers therefore estimate bias indirectly, for example by comparing respondents with records available for everyone in the sample (such as administrative data), following up with a subsample of nonrespondents, or comparing estimates across levels of recruitment effort. Groves notes that “alternative designs to measure nonresponse bias exist, providing different but incomplete information” and that a synthesis of studies “shows the bias often present”.1
The main correction is weighting. It helps when the weighting variables are related to both responding and the answers, and fails when they are not. Pew gives an example where it falls short: for volunteerism, “standard weighting falls short, resulting in biased estimates”.2 A topic where willingness to answer is itself related to the behavior being measured is exactly where adjustment struggles.
Common confusions and neighboring terms
- Nonresponse bias vs. low response rate. The rate is a count; the bias is an error in an estimate. They are related but not the same.1
- Nonresponse bias vs. opt-in sample bias. Nonresponse affects samples where people were selected and some declined. Opt-in samples begin with self-selection.
- Nonresponse bias vs. sampling error. Sampling error comes from chance in who was selected and shrinks with larger samples. Nonresponse bias does not shrink as the sample grows.
- Unit vs. item nonresponse. Unit nonresponse is a person not answering at all; item nonresponse is skipping particular questions.
How My Vote Forward uses the term
My Vote Forward applies the logic of nonresponse bias to civic participation. People who take part in any civic process differ from those who do not, and the differences are often related to the subjects being discussed. The framework therefore reports who participated, and how many, rather than presenting participant results as the public’s views. See Measurement and Participation over time. The demo uses simulated participants.
What My Vote Forward can and cannot do here
My Vote Forward can disclose denominators, describe who is missing where that is known, and avoid representativeness claims. It cannot observe the views of people who do not participate, and it cannot remove the bias their absence creates. Encouraging broader participation helps only if it reaches the people who are missing.
Unresolved questions
- Which groups are least likely to take part in a civic priorities system, and how could their absence be measured without collecting sensitive data?
- Should participant results ever be adjusted toward population benchmarks, given the limits of weighting?
- How should institutions treat a priority strongly expressed by participants when there is reason to think non-participants differ?
Sources
- Robert M. Groves, “Nonresponse Rates and Nonresponse Bias in Household Surveys,” Public Opinion Quarterly 70(5), 2006, pp. 646 to 675. academic.oup.com/poq/article/70/5/646/4084443
- Pew Research Center, “Response rates in telephone surveys have resumed their decline,” February 27, 2019. pewresearch.org/…
