Deliberation effects: what discussion changes, how it is measured, and what can go wrong in the measurement

Deliberation effects are the changes in opinions, knowledge and attitudes that result from structured, balanced public discussion. The best-known evidence comes from Deliberative Polls, which survey a random sample, bring members together to deliberate with balanced materials, and survey them again. Studies including a national U.S. experiment published in 2021 report knowledge gains and depolarizing opinion change, while other research warns that discussion among like-minded people can do the opposite. Measuring the effect cleanly is harder than it looks.

In plain words

When people talk through a public question with good information and people who disagree, do they change their minds? Do they learn? Do they start to dislike each other less?

Researchers test this by asking people what they think before and after a discussion. The hard part is knowing that the discussion caused the change. People who agree to spend a weekend deliberating may be different from people who say no. People may give answers they think the organizers want. Changes may fade after a few weeks.

The best studies compare deliberators with similar people who did not deliberate, and track them over time. Their results are encouraging, but they also show that the details matter: who is in the room, how the talk is run, and what information people get.

The evidence that fair procedures affect perceived legitimacy is in Evaluation. Real assemblies are covered in Ireland’s citizens’ assemblies, British Columbia and France’s climate convention. This page is about method.

Defined terms

• Deliberation. Discussion aimed at weighing reasons on a public question, usually with balanced information and moderation. See the glossary.
• Deliberative Poll. A design in which “a random, representative sample is first polled,” then deliberates and “is again asked the original questions”.⁠¹ See the glossary.
• Treatment effect. The difference between an outcome after deliberation and what it would have been without deliberation, for the same people.
• Pre-post design. Measuring the same people before and after.
• Control group. Comparable people measured at the same times who do not deliberate.
• Group polarization. The tendency of like-minded discussion groups to move toward more extreme positions, described by Grönlund and colleagues as opinions that “tend to become more extreme” when “like-minded people discuss with each other”.⁠⁶
• Enclave deliberation. Deliberation within a like-minded group.⁠⁶
• Affective polarization. Dislike and distrust of the opposing political side, as distinct from disagreement on policy.
• Intraclass correlation (ICC). The degree to which members of the same discussion group resemble each other after treatment, which affects statistical precision.

Evidence

The Deliberative Polling design

Stanford’s Deliberative Democracy Lab describes the process: a random sample is polled; members “are invited to gather at a single place for a weekend”; “carefully balanced briefing materials” are provided; participants question “competing experts and political leaders based on questions they develop in small group discussions with trained moderators”; and afterward “the sample is again asked the original questions”.⁠¹ The lab frames the result as “the conclusions the public would reach, if people had opportunity to become more informed and more engaged”.⁠¹ That framing is an interpretation, a counterfactual about the whole public, and it rests on the representativeness of those who attend.

Britain: the first Deliberative Poll

Luskin, Fishkin and Jowell’s 2002 article in the British Journal of Political Science reports “the first Deliberative Poll, in which a national British sample discussed the issue of rising crime and what to do about it”.⁠² The authors examined “the representativeness of the deliberative sample, the extent to which the participants acquired factual information,” “how much and how they changed their views,” and weighed “the extent to which such changes of view hinge on small group influences versus information gains”.⁠² That last question is a methods question at heart: did people change because they learned, or because of who they sat with?

The United States: “America in One Room”

Fishkin, Siu, Diamond and Bradburn’s 2021 article in the American Political Science Review analyzed “a national field experiment in which more than 500 registered voters were brought together from around the country to deliberate in depth over a long weekend on five major issues,” with “a pre-post control group” asked the same questions.⁠³ They report that “the deliberators showed large, de-polarizing changes in their policy attitudes and large decreases in affective polarization,” and they “contrast them with a literature which would have expected the opposite”.⁠³

Deliberation versus ordinary discussion

Jason Barabas’s 2004 article in the American Political Science Review used “two separate propensity score analyses using panel survey data from a deliberative forum and cross-sectional surveys” on Social Security reform.⁠⁴ Barabas found that “deliberation differs from ordinary discussion in that participants soften strongly held views, encounter different perspectives, and learn readily,” and that deliberation “increases knowledge and alters opinions, but it does so selectively based on the quality and diversity of the messages as well as the willingness of participants to keep an open mind”.⁠⁴

Propensity score methods match deliberators to non-deliberators with similar measured characteristics. In plain text, if e(x) is the estimated probability of attending given covariates x, comparisons are made among people with similar e(x). The method removes bias from measured differences. It cannot remove bias from unmeasured ones, such as openness to persuasion, which is plausibly related to both attending and changing.

Like-minded groups: a randomized test

Grönlund, Herne and Setälä, in Political Behavior (2015), ran “a population-based experiment with a pre-test post-test design” on attitudes toward immigration.⁠⁶ People with permissive or restrictive attitudes were “randomly assigned to like-minded and mixed small-n groups”; mixed groups had “four permissive and four restrictive participants”.⁠⁶ The main finding: “people with anti-immigrant attitudes become more tolerant even when they deliberate in like-minded groups,” with “similar learning curves” in both treatments, leading the authors to conclude that “deliberative norms can alleviate the negative consequences of discussion in like-minded groups”.⁠⁶

This design is stronger than most because group composition was randomized. It isolates one variable, who is in the room, while holding the deliberative procedure fixed.

Who wants to deliberate

Selection is not only a nuisance; it is a finding. Neblo, Esterling, Kennedy, Lazer and Sokhey, in the American Political Science Review (2010), studied “hypothetical willingness to deliberate and ... actual participation in response to a real invitation to deliberate with their member of Congress”.⁠⁷ They found that “willingness to deliberate in the United States is much more widespread than expected,” and that “it is precisely those people less likely to participate in traditional partisan politics who are most interested in deliberative participation”.⁠⁷ If true more broadly, deliberative formats may reach a different population than elections and comment periods do.

The core estimator and its threats

The simplest credible design compares changes:

effect = (mean_T_post-mean_T_pre)-(mean_C_post-mean_C_pre)

where T is deliberators and C a comparable control group. This difference-in-differences removes changes that would have happened anyway, such as a news event between waves. Its threats, each of which applies to the studies above to varying degrees:

1. Selection into attendance. Those invited who attend may differ from those who decline. Random sampling of invitees does not make attendees random.
2. Clustering. Deliberators are nested in small groups. If m people share a group and their outcomes have intraclass correlation ρ, the variance of the estimated effect is inflated by roughly 1 + (m-1)ρ. With groups of 15 and ρ = 0.1, that factor is 2.4, so a study of 500 deliberators has the precision of about 210 independent people. Analyses that ignore group structure overstate certainty.
3. Demand effects. Participants who enjoyed a well-run weekend may report the attitudes they believe organizers hope for.
4. Regression to the mean. People selected for extreme initial views will, on average, report less extreme views on remeasurement even with no treatment. Control groups with similar initial views are the guard.
5. Multiple outcomes. Studies measure many items. Without a preregistered primary outcome, some items will move by chance. See Replication and preregistration.
6. Durability. A change measured at the end of a weekend may fade. Follow-up waves are needed to know.
7. Bundled treatment. Information, balanced materials, moderation, expert access and meeting opponents arrive together. Without designs like Grönlund’s that vary one element, it is hard to say which element did the work.

Is deliberation a testable theory?

Diana Mutz’s 2008 review in the Annual Review of Political Science argued for “abandoning tests of deliberative theory per se and instead developing ‘middle-range’ theories that are each important, specifiable, and falsifiable,” replacing “vaguely defined entities with more concrete, circumscribed concepts” so researchers “may come to understand which elements of the deliberative experience are crucial to particular valued outcomes”.⁠⁸ A 2019 commentary in Science by Dryzek and colleagues takes the more optimistic view that “citizens can avoid polarization and make sound decisions” under the right conditions.⁠⁵ The two positions are compatible: the conditions are the research program.

Competing interpretations

“Deliberation is an antidote to polarization.” The America in One Room results,⁠³ and the enclave deliberation experiment⁠⁶ support the view that structured deliberation reduces policy and affective polarization, even in like-minded groups when deliberative norms are present.

“Effects are artifacts of selection and demand.” Skeptics note that attendees are self-selected among invitees, that participants know the event’s goals, and that most studies measure outcomes immediately. Propensity matching⁠⁴ addresses measured differences only.

“Effects are real but bounded.” A middle view holds that deliberation reliably increases knowledge, that opinion change is selective and depends on message quality and diversity, as Barabas found⁠⁴ and that whether effects scale beyond weekend events is unknown.

“The real value is not opinion change.” Some argue the purpose of deliberation is better reasons and mutual understanding, not changed positions. On this view, measuring opinion shifts misses the point, and stable positions with better reasons count as success.

Application to this framework

These are design proposals for My Vote Forward, not results. My Vote Forward does not currently run deliberative events.

1. Do not claim deliberative effects for asynchronous participation. Reading balanced information on a website is not a moderated weekend with opposing experts. Evidence from Deliberative Polls does not transfer automatically to online priority exercises.
2. If an informational step is added, test it with a randomized design. For example, randomly assign demonstration participants to see or not see a balanced briefing on tradeoffs before ranking, and compare rankings and a knowledge measure. Preregister the primary outcome.
3. Analyze clusters correctly. Any small-group feature, such as discussion rooms, would be analyzed with group-level variance, not as independent individuals.
4. Measure durability. Re-ask a subset after a delay, since the framework already supports revision over time (see Participation over time).
5. Report who participated in any deliberative component. Following Neblo and colleagues,⁠⁷ describe how participants differ from those who did not join, without implying representativeness.

Unresolved questions

• Do depolarization effects from intensive weekend events persist months later, and do they spread to participants’ social networks?
• Which component of deliberation does the work: information, moderation, diversity of the group, or meeting real opponents?
• Can any online, asynchronous format reproduce measurable deliberative effects at scale, and how would one test it without demand effects?
• How should deliberative results be weighed against larger, undeliberated participation when the two disagree? See Representation.
• If the people most interested in deliberation are those least active in partisan politics,⁠⁷ does deliberation broaden participation or create a separate, parallel public?