Public opinion polling: from straw votes to probability samples to the modern accuracy debate

Public opinion polling is the practice of estimating what a population thinks by asking questions of a sample of its members. Its modern form rests on probability sampling, formalized in statistics in the 1930s, and on a lesson learned from the 1936 Literary Digest poll, whose enormous mailed survey failed because both its sample and its pattern of responses were biased.⁠^1,2 Polls remain the main tool for measuring opinion between elections, but election polls in 2016 and 2020 showed systematic errors, which an American Association for Public Opinion Research committee attributed in 2016 mainly to a late shift in support and to failure to adjust for education.⁠³

In plain words

A poll asks some people a question to learn what many people think. It sounds simple. It is not.

In 1936 a magazine called the Literary Digest mailed questions to millions of people and predicted the wrong winner of the presidential election. A study published in 1988 found two reasons. The magazine’s list leaned toward some kinds of people, and the people who mailed answers back were different from those who did not.

The fix was a math idea: pick people at random, so everyone has a known chance of being chosen. A few thousand randomly chosen people can describe a country better than millions of volunteers.

Random sampling still has a problem. Today many people do not answer polls. If those who answer differ from those who do not, the poll can be wrong. That happened in some states in 2016.

There is a deeper criticism too. Some writers argue that polls do not just measure opinion. By choosing which questions to ask and how to word them, pollsters help decide what counts as public opinion at all. Any system that asks the public questions, including this one, faces the same challenge.

Defined terms

• Population. The full group a poll aims to describe, for example adult citizens of a country.
• Sample. The subset actually asked.
• Probability sample. A sample in which every member of the population has a known, nonzero chance of selection. Jerzy Neyman’s 1934 paper comparing stratified sampling with purposive selection is a founding statement of the approach.⁠² See opt-in sample.
• Selection bias. Error that arises when the sampling frame or method favors some kinds of people.⁠¹
• Nonresponse bias. Error that arises when people who respond differ from those who do not. See nonresponse bias.
• Weighting. Adjusting responses so the sample matches known population characteristics. See weighting.
• Issue poll versus election poll. An election poll estimates a vote; an issue poll estimates attitudes. They can fail differently.⁠⁴

Evidence

Research results: the failure that taught the field

The Literary Digest poll of 1936 is the field’s best known failure. Peverill Squire, writing in Public Opinion Quarterly in 1988, noted that no empirical study had examined why it failed. Using a 1937 Gallup survey that asked people whether they had taken part in the Digest poll, he concluded that both the magazine’s sample and its response pattern were biased, and that together they produced a badly inaccurate estimate. He also argued that if everyone sampled had responded, the magazine would likely have correctly predicted Franklin Roosevelt’s win.⁠¹

Squire’s finding matters because the usual textbook story blames only the mailing list. His evidence shows that nonresponse, the same problem that troubles polls today, was part of the failure from the start.

History: the statistical foundation

Neyman’s 1934 paper in the Journal of the Royal Statistical Society, “On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection,” set out the case for random selection over choosing cases judged to be typical.⁠² The principle is that randomness, not size, lets a pollster calculate how far an estimate is likely to be from the truth. Survey research methods are covered in depth in measurement.

History: the 1948 test

Polls also missed the 1948 presidential election. The Social Science Research Council commissioned a review, published as The Pre-Election Polls of 1948: Report to the Committee on Analysis of Pre-Election Polls and Forecasts, by Frederick Mosteller, Herbert Hyman, Philip McCarthy, Eli Marks, and David Truman, and reviewed in the American Journal of Sociology in 1950.⁠⁵ The episode established a practice that continues: when polls miss, the profession convenes a committee and publishes what went wrong.

Research results: 2016

After 2016, an AAPOR ad hoc committee led by Courtney Kennedy published its evaluation in Public Opinion Quarterly. It noted that forecasts based on polls had given Hillary Clinton roughly a 90 percent chance of winning, with individual estimates from 71 to over 99 percent. The committee found the strongest evidence for two explanations of why many polls, especially in the Upper Midwest, underestimated Donald Trump: a late shift toward Trump that many polls did not capture, and the widespread failure to adjust for the overrepresentation of college graduates, who leaned toward Clinton. It found mixed evidence that likely-voter models caused systematic error and “little evidence” that a “shy Trump” effect was important.⁠³

Research results: 2020 and issue polling

Pew Research Center reported in March 2021 that “most preelection polls in 2020 overstated Joe Biden’s lead over Donald Trump in the national vote.” It then asked whether errors that large would distort polls on issues. Comparing a “balanced” poll (Biden ahead by 4.4 points) with a “tilted” one (Biden ahead by 12), it found that such errors “would alter measures of opinion on issues by an average of less than 1 percentage point,” with the largest shifts 3 points for a single answer and 5 points for the gap between two answers.⁠⁴ The reason, the authors explain, is that issue views are strongly but not perfectly tied to party.

This is an important distinction for anyone using polls to measure priorities rather than predict winners. A poll that misses an election margin may still describe issue opinion within a few points.

Research results: who takes online polls

Many modern polls rely on online panels rather than random telephone or address samples. Eszter Hargittai and Gökçe Karaoglu, in Socius (2018), used a national survey of voter-eligible U.S. adults from summer 2016 to examine how age, gender, race, education, employment, and internet skills relate to taking part in online political polls, on the premise that people do not all use the same sites and services.⁠⁶ The concern is old: it is the Literary Digest problem in a new medium. See opt-in sample.

Competing interpretations

Polls as democratic listening. Supporters argue that polls give continuous, representative information about public views between elections, reaching people who would never attend a hearing or write a letter. The record of the Digest failure and its correction supports the claim that well-designed sampling can describe a large public.⁠^1,2

Polls as manufactured opinion. Walter Lippmann’s Public Opinion (1922), written before scientific polling, argued that people respond not to the world directly but to “a pseudo-environment” that stands between them and it, and that “the manufacture of consent is capable of great refinements,” having “improved enormously in technic, because it is now based on analysis rather than on rule of thumb”.⁠⁷ Critics in this tradition argue that asking a question can create an answer people did not hold, and that the choice of questions reflects the interests of whoever pays for them.

Do people have opinions to measure? Philip Converse’s 1964 essay “The Nature of Belief Systems in Mass Publics” argued that ideological thinking was confined to a small part of the public. Martin Wattenberg, revisiting the question in Critical Review (2019), argues that the share of Americans with ideologically constrained views has grown substantially, estimating “concept ideologues” at about 42 percent of the public by 2012 and about 38 percent by 2016, compared with roughly 12 percent in Converse’s data from around 1956.⁠⁸ The debate bears on how much weight any single poll answer can carry.

Accuracy debates. Some commentators concluded after 2016 that polling had failed. The AAPOR committee’s evidence points to specific, partly correctable problems rather than a collapse.⁠³ Pew’s analysis suggests that election misses overstate the error in issue measurement.⁠⁴ Skeptics answer that repeated misses in the same direction suggest nonresponse problems that weighting cannot fully fix.

Interpretation, this site’s reading. Each camp is right about something. Sampling theory is sound. Nonresponse is real and growing. Question wording and selection shape answers. A responsible use of polling states all three.

Application to this framework

These are design implications, not claims about the framework’s performance.

1. My Vote Forward is not a poll. Its participants choose to take part, so they are an opt-in group, not a probability sample. Results describe participants and should never be presented as “what the public thinks.” See is this just a poll.
2. Report the denominator and the profile. The Digest lesson is that size does not cure bias.⁠¹ A result from 100,000 self-selected participants needs its participation profile beside it; see measurement.
3. Weighting is a tool, not a guarantee. The 2016 education problem shows weighting helps only for characteristics you measure.⁠³ If results are ever weighted, the method and variables should be published.
4. Question wording is governance. Lippmann’s critique applies to any system that frames public choices.⁠⁷ See question design and who writes the questions.
5. Priorities are not predictions. Pew’s finding that issue measures are less sensitive than election margins,⁠⁴ is encouraging for priority measurement but does not transfer to opt-in data. Comparisons with probability surveys are a useful check; see evaluation.

Hypothetical example. A city’s priority tool collects 40,000 responses, and 58 percent rank road repair first. A probability survey of 1,000 residents run the same month finds 41 percent. The large opt-in number does not settle which figure is closer to the truth. Comparing participant profiles in both, and publishing the gap, is more honest than reporting either alone.

Unresolved questions

• How far can weighting and modeling correct for declining response rates, and how would anyone know when they have failed before an election tests them⁠³?
• Are issue polls as robust to nonresponse as Pew’s 2020 analysis suggests, across different topics and periods⁠⁴?
• How can opt-in participation data be validated against probability benchmarks without making participants feel surveilled?
• Does asking people to rank priorities create preferences they did not hold, as Lippmann’s critique would suggest⁠⁷ and how could that be measured?