Weighting is the adjustment of how much each survey respondent counts, so that the sample matches known characteristics of the population, such as age, education, or region. If young adults are underrepresented, each young respondent counts for more. Weighting can reduce bias, but only for differences linked to the variables used, and Pew Research Center’s tests show it removed only part of the bias in online opt-in samples.
Definition
A survey weight is a number attached to each respondent that says how many people in the target population that respondent represents. Weights usually combine two parts: a base weight that reflects the chance of selection in a probability sample, and adjustments that correct for nonresponse and coverage by aligning the sample with population benchmarks from sources such as the census. In opt-in samples there is no known selection chance, so all of the work falls on the adjustment step.1
Origin and history
Weighting has become more central as response rates have fallen. Pew reported that “in 2017 and 2018, typical telephone survey response rates fell to 7% and 6%, respectively”.2 With so few selected people responding, adjustment became central to accuracy. The AAPOR Task Force on Nonprobability Sampling (2013) devoted a chapter to estimation and weight adjustment, noting that sample matching “has recently been advocated for use in surveys that use opt-in panels”.1
How weighting is done and evaluated
Pew’s 2018 study compared “three different weighting techniques, raking, propensity weighting and matching, both on their own and in combination,” on simulated opt-in samples ranging “from n=2,000 to n=8,000”.3
- Raking adjusts weights in turns until the sample matches population totals on several variables at once.
- Matching selects respondents who resemble members of a reference sample drawn from a high-quality source.
- Propensity weighting models each respondent’s likelihood of being in the opt-in sample and weights inversely.
The findings were sobering and specific. “The most basic weighting method (raking) performs nearly as well as more elaborate techniques based on matching”.3 The choice of variables mattered more than the method: adding political variables “reduced the average estimated bias by an additional 1.4 percentage points” beyond demographics alone.3 Yet across 24 benchmarks no procedure reduced average bias “below 6 percentage points,” down from 8.4 unweighted, and “even the most effective adjustment procedures were unable to remove most of the bias”.3
Weighting also has a cost. Unequal weights increase the variance of estimates, so a heavily weighted sample behaves like a smaller one. And Pew notes that even probability surveys can fail on some topics; for volunteerism, “standard weighting falls short, resulting in biased estimates”.2
Common confusions and neighboring terms
- Weighting vs. opt-in sample. Weighting is a correction; it does not turn an opt-in sample into a probability sample.
- Weighting vs. nonresponse bias. Weighting is the main tool against nonresponse bias, and it works only when the weighting variables are related both to responding and to the answers.
- Weighting vs. resource allocation. “Weight” in a budget exercise means importance; in surveys it means representation.
- Weighting vs. quotas. Quotas control who is recruited; weighting adjusts after data is collected.
How My Vote Forward uses the term
My Vote Forward does not weight participant results to stand in for the public. Its Measurement chapter treats participants as participants, and releases report raw denominators. If the framework ever reported weighted estimates, it would need to name the weighting variables, the benchmark source, and the method, and to label results as estimates with known limits. Demo data is simulated.
What My Vote Forward can and cannot do here
My Vote Forward can disclose exactly who answered and refrain from presenting weighted participant data as public opinion. It cannot remove self-selection bias by weighting, as Pew’s results show. It cannot weight by variables it does not collect, and collecting them would raise privacy costs.
Unresolved questions
- Would weighting participant priorities to census benchmarks make them more useful, or create false confidence?
- Which variables, if any, could a privacy-respecting system collect for weighting?
- How should weighted and unweighted results be shown side by side so readers understand the difference?
Sources
- Reg Baker, J. Michael Brick, and others, Report of the AAPOR Task Force on Nonprobability Sampling, American Association for Public Opinion Research, June 2013. aapor.org/…
- Pew Research Center, “Response rates in telephone surveys have resumed their decline,” February 27, 2019. pewresearch.org/…
- Pew Research Center, “For Weighting Online Opt-In Samples, What Matters Most?” January 26, 2018. pewresearch.org/…
