Replication and preregistration: how social science checks itself, and how a civic platform can too

Replication is the practice of repeating a study to see whether its findings hold; preregistration is publishing a study’s hypotheses and analysis plan before seeing the outcome data. Large projects found that only about a third to two-thirds of published experiments replicated, with replicated effects about half the original size, and a 2022 study gave 73 research teams the same data on immigration and social policy and got widely diverging answers. These findings apply to any organization that publishes numbers about public opinion, including civic platforms.

In plain words

Science is supposed to correct itself. One way is to redo studies and see if the results come out the same. When researchers did this on a large scale, many famous results came out weaker or not at all.

Why? Partly because exciting results are more likely to be published. Partly because researchers make many small choices when analyzing data, and each choice can nudge results. No cheating is needed. When dozens of teams were given the same data and the same question, they reached different answers.

One fix is to write down the plan before looking at the results, and publish it. Another is to share data and code so others can check. Neither is perfect.

A site that publishes what participants think is important makes many of the same choices: which rule ranks subjects, which participants count, how ties are shown. The same discipline applies. The evaluation plan for the framework itself is in Evaluation.

Defined terms

• Replication. Repeating a study with new data to test whether its finding holds.
• Reproducibility. Obtaining the same results from the same data and code. Usage varies across fields.
• Publication bias. The tendency for studies with strong or significant results to be published more often than studies with null results.
• Researcher degrees of freedom. The many choices in data collection, analysis and reporting that can affect results.⁠³
• Garden of forking paths. The multiple-comparisons problem that arises when analysis choices are “highly contingent on data,” even if only one analysis is run.⁠⁴
• Preregistration. Defining “the research questions and analysis plan before observing the research outcomes”.⁠⁶
• Registered report. A publication format in which a study’s plan is peer reviewed and accepted before results are known.
• Statistical power (1-β). The probability that a study detects an effect of a given size if it exists.
• Positive predictive value (PPV). The probability that a statistically significant finding reflects a true effect.⁠⁵
• Many-analysts study. A study in which multiple independent teams analyze the same data to answer the same question.⁠⁸

Evidence

Large replication projects

The Open Science Collaboration’s 2015 project in Science replicated “100 experimental and correlational studies published in three psychology journals using high-powered designs and original materials when available”.⁠¹ Its results: “Replication effects were half the magnitude of original effects”; “Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results”; and “39% of effects were subjectively rated to have replicated the original result”.⁠¹ The authors found that “replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams”.⁠¹

Camerer and colleagues, in Nature Human Behaviour (2018), replicated “21 systematically selected experimental studies in the social sciences published in Nature and Science between 2010 and 2015,” using analysis plans “reviewed by the original authors and pre-registered prior to the replications,” with samples “on average about five times higher than in the original studies”.⁠² They found “a significant effect in the same direction as the original study for 13 (62%) studies,” with replication effect sizes “on average about 50% of the original effect size”.⁠² They also found that “peer beliefs of replicability are strongly related to replicability,” suggesting “the research community could predict which results would replicate”.⁠²

Publication bias, measured directly

Franco, Malhotra and Simonovits, in Science (2014), studied “a known population of conducted studies, 221 in total,” from Time-sharing Experiments in the Social Sciences (TESS), a National Science Foundation program running survey experiments “on representative samples of American adults”.⁠⁷ Because they knew every study run, not just those published, they could measure the file drawer: “Strong results are 40 percentage points more likely to be published than are null results and 60 percentage points more likely to be written up”.⁠⁷ The main leak occurred before review: “Authors do not write up and submit null findings”.⁠⁷

Flexibility creates false positives without fraud

Simmons, Nelson and Simonsohn’s 2011 article “False-Positive Psychology” showed “that despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates,” and that “in many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not”.⁠³ They proposed “six concrete requirements for authors and four guidelines for reviewers” based on disclosure.⁠³

Gelman and Loken extended the point. Their 2013 paper argues that “researcher degrees of freedom can lead to a multiple comparisons problem, even in settings where researchers perform only a single analysis on their data,” because “there can be a large number of potential comparisons when the details of data analysis are highly contingent on data, without the researcher having to perform any conscious procedure of fishing or examining multiple p-values”.⁠⁴ The implication is uncomfortable: honest researchers who never “p-hack” can still produce unreliable results if their choices respond to what the data look like.

Same data, different answers

Silberzahn and colleagues gave one data set to “twenty-nine teams involving 61 analysts” with one question: whether soccer referees give more red cards to dark-skin-toned players.⁠⁸ “Estimated effect sizes ranged from 0.89 to 2.93 (Mdn = 1.31) in odds-ratio units”; “Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not”; and the 29 analyses “used 21 unique combinations of covariates”.⁠⁸ Neither prior beliefs nor expertise “readily explained the variation”.⁠⁸

Breznau and colleagues, in the Proceedings of the National Academy of Sciences (2022), ran the same experiment on a politically charged question. They “coordinated 161 researchers in 73 research teams” who used “the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public”.⁠⁹ Teams “reported both widely diverging numerical findings and substantive conclusions despite identical start conditions,” and “more than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow”.⁠⁹ The authors call this “a universe of uncertainty that remains hidden when considering a single study in isolation,” and they “call for greater epistemic humility and clarity in reporting”.⁠⁹

This result matters for political debate. A single published study on a contested question, however competent, represents one path through a large space of defensible analyses.

The arithmetic of false findings

Ioannidis’s 2005 essay “Why Most Published Research Findings Are False” formalized how field-level conditions affect reliability.⁠⁵ With R the ratio of true to null relationships among those tested, α the significance threshold and β the type II error rate, the essay gives:

PPV = (1-beta) * R / ( R-beta*R + alpha )

and notes that “a research finding is thus more likely true than false if (1-β)R > α”.⁠⁵ The essay argues that findings are less likely to be true “when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes”.⁠⁵

Worked example (arithmetic only). If one in ten tested hypotheses is true (R = 1/9, about 0.111), α = 0.05 and power is 0.8 (β = 0.2), then PPV = 0.8 × 0.111 / (0.111-0.022+0.05), about 0.089 / 0.139, or 64%. With power of 0.35 instead (β = 0.65), PPV falls to about 0.039 / 0.089, or 44%. Low power does not only miss true effects; it makes the significant results that do appear less trustworthy.

Benjamin and a large group of coauthors proposed in 2018 “to change the default P-value threshold for statistical significance from 0.05 to 0.005 for claims of new discoveries”.⁠¹⁰ In the formula, lowering α raises PPV, at the cost of requiring larger samples for the same power. The proposal has been debated; it addresses α but not the forking paths that make nominal α misleading.

How large must a study be?

The standard approximation for comparing two group means, with a standardized effect size d, two-sided α and power 1-β, is:

n per group = 2 * ( z_(1-alpha/2) + z_(1-beta) )^2 / d^2

For α = 0.05 and power 0.8, the z-values are about 1.96 and 0.84. For d = 0.2, a small effect, that gives about 393 people per group; for d = 0.1, about 1,570 per group. These are arithmetic, not findings. Many evaluations of civic interventions would need samples of this size to detect plausible effects, which is relevant to any claim that a participation feature “works.”

Preregistration and its limits

Nosek, Ebersole, DeHaven and Mellor argue in PNAS (2018) that “mistaking generation of postdictions with testing of predictions reduces the credibility of research findings,” and that “an effective solution is to define the research questions and analysis plan before observing the research outcomes, a process called preregistration”.⁠⁶ They note strategies for imperfect cases, “such as when the data are preexisting,” and that “services are now available for preregistration across all disciplines”.⁠⁶

Preregistration addresses forking paths and selective reporting. It does not address the many-analysts problem fully: a preregistered plan is still one path among many defensible ones, as Breznau and colleagues’ results imply.⁠⁹ Complementary practices include publishing results from several reasonable specifications, sharing data and code, and inviting independent reanalysis.

Disclosure in survey research

Survey research has its own transparency norms. AAPOR’s Transparency Initiative “is designed to promote methodological disclosure through a proactive, educational approach,” recognizing organizations “that pledge to practice transparency in their reporting of survey-based research findings,” while making “no judgment about the approach, quality or rigor of the methods being disclosed”.⁠¹¹ Disclosure is necessary for checking, but it is not a quality certificate.

Competing interpretations

“There is a replication crisis.” The 36% and 62% replication rates,⁠^1,2 halved effect sizes, and measured publication bias,⁠⁷ support the view that many published findings are unreliable.

“Replication rates are hard to interpret.” A failed replication can reflect differences in population, setting or implementation, not a false original. Camerer and colleagues’ complementary indicators ranged from 57% to 67%⁠² showing that the rate depends on the definition of success.

“Preregistration is the fix.” Advocates argue that separating prediction from postdiction.⁠⁶ addresses the core problem.

“Preregistration is not enough, and can mislead.” Critics note that many-analysts results⁠^8,9 show wide variation even without data-dependent choices, and that preregistration can lend unearned authority to one arbitrary specification.

“Stricter thresholds help.” The 0.005 proposal⁠¹⁰ would raise PPV. Opponents argue that thresholds of any kind encourage dichotomous thinking and that estimation with uncertainty is better.

Application to this framework

These are design proposals for My Vote Forward, not results.

1. Preregister evaluations. Any test of whether a framework feature changes comprehension, priorities or trust is preregistered with a primary outcome, sample size justified by a power calculation, and analysis plan, following the Evaluation design. Results are published whether or not the effect appears, so the framework does not build its own file drawer⁠⁷
2. Preregister release methods. Before a priority exercise opens, publish the aggregation rule, inclusion rules (for example, which unconfirmed answers are excluded), suppression rules and tie rules. Changes after data collection are logged with reasons. See Published releases.
3. Report a multiverse for headline results. Show whether the published order changes under alternative defensible choices: a different aggregation rule (see Ranked preferences), with and without flagged clusters (see Coordinated campaigns detection), and with alternative suppression thresholds. This responds directly to the many-analysts evidence.⁠⁹
4. Publish data and code for independent reanalysis. Within privacy limits (see Differential privacy and Small-group suppression), release aggregate tables and the code that produced them, and invite outside teams to reproduce them.
5. Label exploratory findings. Patterns noticed after the fact are reported as hypotheses for future testing, not findings, following the distinction between prediction and postdiction.⁠⁶

Unresolved questions

• Who should review a civic platform’s preregistered release plan before data collection, and how can that reviewer be independent?
• How many alternative specifications should a multiverse report show before it becomes unreadable for a general audience?
• If a priority result changes under a reasonable alternative rule, which version should be the headline, or should there be none?
• Can registered reports, common in some journals, be adapted for public civic releases?
• How should the framework handle preregistered plans that turn out to be wrong once data arrive, without opening the door to the forking paths preregistration is meant to close?