Who writes the questions, and how is wording kept fair?

In the My Vote Forward demo, the project itself writes the questions and proposals people give opinions on, and that is a real form of power: survey research shows that changing the options offered can move answers by more than 20 percentage points. The framework’s answer is to publish the exact wording and version of every item, tie every opinion to the wording it answered, and review questions against published guidance. Independent review of wording, of the kind the United Kingdom’s Electoral Commission performs for referendum questions, does not yet exist for this project.

The objection in its strongest form

The objection to My Vote Forward here is that the person who writes the question has already shaped the answer. A proposal described as “investing in public safety” will draw different responses from one described as “increasing the police budget,” even if the policy is identical. Options offered, options omitted, the order of choices, and the framing of costs all move results. A framework that publishes opinion results on questions it wrote is publishing its own editorial choices with a veneer of public endorsement. Publishing the wording does not help much, the critic says, because readers quote the headline number, not the fine print.

The objection is distinct from the question of who chooses the ten subjects. That page concerns the list of broad subjects people rank. This page concerns the specific questions and proposals people give opinions on, where wording effects are most direct.

Why the objection matters

Wording matters for My Vote Forward because opinions are tied to specific items, and an opinion release is only as fair as its questions. If items are systematically framed to favor one side, results will be biased in a way that looks like public opinion. That would damage the framework’s claim to be nonpartisan and would give anyone who controls the drafting a way to manufacture apparent support.

Defined terms

• Wording effect: a change in responses caused by how a question is phrased.
• Response options: the choices offered on a closed-ended question.
• Order effect: a change in responses caused by the sequence of questions or options.
• Neutrality review: an assessment of whether a question is balanced and not leading, usually by a party independent of its author.
• Item version: a recorded version of a question’s exact wording, so that opinions remain tied to what they answered. See opinions.

Evidence

Survey research documents large wording effects. Pew Research Center reports a 2008 experiment in which 58% of voters chose the economy as the issue that mattered most when it was offered, while only 35% volunteered it in an open-ended version of the same question.⁠¹ Pew summarizes the general rule: “the choice of options provided, how each option is described,” the number of options, and “the order in which options are read can all influence how people respond”.⁠¹

Professional standards respond with disclosure. The American Association for Public Opinion Research’s Code, revised in April 2021, requires researchers to disclose immediately, with any public report, items including who sponsored and conducted the research, the exact wording and presentation of the questions behind reported results, the population studied, how the sample was recruited, the dates of data collection, sample sizes, how data were weighted, and a statement acknowledging limitations.⁠² Disclosure does not make a question neutral, but it lets others judge.

Referendums show what independent review looks like. Aileen McHarg’s account of the 2014 Scottish independence referendum reports that the Electoral Commission found the Scottish Government’s proposed question easy to understand and unambiguous, yet had “substantial concerns about the neutrality of the question,” and recommended a revised wording: “Should Scotland be an independent country? Yes/No”.⁠³ The Scottish Government promptly accepted the recommendation.⁠³ A government drafted the question; an independent body reviewed it; the change was public. See the Scottish referendum case.

Deliberative bodies show another model. Ireland’s 2016 to 2018 Citizens’ Assembly had its topics set by a resolution approved by both Houses of the Oireachtas, and its 99 members were selected at random.⁠⁴ The legislature wrote the agenda; the members deliberated within it and produced recommendations for elected representatives to debate.⁠⁴ See the Ireland case.

How comparable systems have handled it

The comparable systems allocate wording power in three ways.

• Author plus independent review. The Scottish model: a government drafts, an electoral body reviews for clarity and neutrality, and the result is published.⁠³
• Author plus disclosure. The polling model: whoever sponsors a survey writes it, and professional codes require the exact wording to be disclosed.⁠²
• Accountable author. The Irish model: an elected legislature sets the questions and answers for them politically.⁠⁴

Each leaves a gap. Review depends on an independent body existing. Disclosure depends on readers checking. Accountable authorship depends on incumbents not using the agenda to their advantage.

Competing interpretations

One interpretation holds that there is no neutral wording, only disclosed wording. Every phrasing emphasizes something. On this view, the best a framework can do is publish wording, test alternatives, and report how much results depend on them.

A second interpretation holds that neutrality is achievable enough to be worth pursuing through review. The Scottish case shows that an independent body can identify and fix a leading phrase.⁠³ On this view, any serious deployment needs an independent reviewer, not just disclosure.

A third interpretation holds that participants should write or revise questions themselves, as some deliberative and crowdsourced processes allow. That spreads the power but gives it to whoever participates, and someone still has to decide which participant drafts are used.

What the framework answers

My Vote Forward’s provisional rules:

• Exact wording is published with every result. Every opinion release names the item, its exact wording, its version, the response options, and their order, consistent with AAPOR’s disclosure items.⁠²
• Opinions stay tied to wording. If wording changes, earlier opinions are marked as answering an earlier version and are not carried over. See opinions.
• Questions are checked against published guidance. Items are reviewed against the question design guidance, including balanced options and avoiding double-barreled items.
• Split-wording tests where possible. When a wording is contested, a pilot can randomly assign two versions and report both results, so readers see how much the answer depends on phrasing.
• Authorship is recorded. Each item records who drafted it, who reviewed it, and when. See governance.

Hypothetical example. A pilot item asks about a transit proposal. Half of participants see “a half-cent sales tax for expanded bus service,” and half see “expanded bus service funded by a half-cent sales tax.” The release reports both results side by side and notes any difference, rather than choosing one.

Where the objection still wins

The objection wins on independence. Today, the people who write the questions are the people who run the project, and there is no independent reviewer of the kind the Electoral Commission provides. Disclosure helps only readers who check it, and most will not. The framework also has no record yet of how its own items perform, because it has no real participant data. Split-wording tests show sensitivity but do not say which wording is right.

Unresolved questions

• Who should review wording in a real deployment: an election body, a panel of opposing advocates, survey methodologists, or participants?
• Should proposals written by a public body be used verbatim, even if their wording is leading, or rewritten for neutrality?
• How much sensitivity to wording should disqualify an item from publication?
• Should participants be able to propose alternative wordings, and how would those be selected?

Related

• Question design
• Who chooses the ten?
• Opinions
• Agenda capture
• Hard questions