Question design

In plain words

A question is not a neutral window into someone's mind. Change a few words and many people give a different answer. Ask one question first and the next answer can shift. Give five choices instead of an open box and people pick from the five.

This is not a reason to stop asking. It is a reason to write questions carefully, test them, and show everyone the exact wording next to any result. In this framework, a priority or an opinion is always tied to the question that produced it. If the question changes, the result is a new result, not a continuation of the old one.

This chapter covers wording, order, scales, double questions, "don't know" options, and false precision. How answers are counted and weighted is in Measurement. How many answers are combined into a group priority is in Collective priorities.

Defined terms

Question wording effect. A change in answers caused by changing the words of a question while the topic stays the same.

Question order effect. A change in answers caused by what was asked before. Survey researchers distinguish a contrast effect, where answers move away from the earlier question, from an assimilation effect, where answers move toward it¹.

Response order effect. A change in which option people choose depending on where it appears in a list. A primacy effect favors early options; a recency effect favors later ones¹.

Double-barreled question. One question that asks about two things at once, so a single answer cannot be read as a view on either thing¹.

Acquiescence. A tendency to agree with agree/disagree statements regardless of content¹².

Satisficing. Giving an answer that is acceptable with little effort instead of doing the full work of recalling and weighing what one thinks. The opposite is optimizing².

Labeled scale. A rating scale where each point has a word label ("not at all important" through "extremely important") rather than numbers alone.

False precision. Reporting a number with more exactness than the method can support, such as a difference of one point that lies inside the range of ordinary error.

See also the glossary for the framework's stable definitions of opinion, ranking, and ballot.

Evidence

Wording changes answers

Pew Research Center publishes worked examples from its own surveys. In January 2003, 68% of U.S. adults favored "taking military action in Iraq to end Saddam Hussein's rule." When the question added "even if it meant that U.S. forces might suffer thousands of casualties," 43% favored and 48% opposed¹. In 2005, 51% favored "making it legal for doctors to give terminally ill patients the means to end their lives," while 44% favored "making it legal for doctors to assist terminally ill patients in committing suicide"¹.

Category definitions matter as much as tone. In January 2002, when Pew asked whether domestic or foreign policy should be the main focus, 52% chose domestic and 34% foreign. When "foreign policy" was narrowed to "the war on terrorism," 33% chose domestic and 52% chose terrorism¹. That example is directly relevant to any priority list: how a category is named and drawn affects how many people choose it.

Order changes answers

Pew reports that in 2003, support for legal agreements for same-sex couples was 45% when asked after a question on same-sex marriage and 37% when asked without that context, a contrast effect¹. In November 2008, 81% said Republican leaders should work with the incoming president when the question followed one about Democratic leaders, compared with 66% otherwise, an assimilation effect¹.

For lists of options, Pew finds that phone respondents more often choose items heard last (recency), while people reading a self-administered survey more often choose items near the top (primacy). Pew rotates most option lists so that each item appears in different positions. Rotation spreads the bias across items; it does not remove it. Ordered scales are generally not randomized, though Pew has reversed scale order for half a sample to spread the effect¹.

Open and closed formats measure different things

In a 2008 question on the issue that mattered most, 58% chose "the economy" when it was offered in a closed list, and fewer than one in ten gave an answer outside the five listed. When the same question was open-ended, 35% volunteered the economy and 43% gave a response not on the closed list¹. A fixed list of subjects therefore shapes what people report as important, not only how they rank it.

Agree/disagree statements invite acquiescence

Pew reports that less educated and less informed respondents are more likely to agree with agree/disagree statements, that the effect is stronger with an interviewer present, and that offering a choice between two alternative statements produced a very different overall result¹. Krosnick and Presser's review of questionnaire research reaches a similar conclusion about agree/disagree formats².

Labeling scale points helps

Krosnick and Presser summarize studies finding that reliability is higher when all scale points are labeled with words than when only some are, that respondents report greater satisfaction with more fully labeled scales, and that reliability and validity improve when labels divide the range into roughly equal steps². Pew recommends keeping answer choices to about four or five in most circumstances, with more categories acceptable for factual and demographic items¹.

"Don't know" options are not automatically better

A common intuition is that offering "don't know" always improves data. The research is more mixed. Krosnick and Presser report that "don't know" filters do encourage some people without information to say so, but that "there is considerable evidence that DK filters do not improve measurement"². In one study they summarize, 15% chose "don't know" on a first nuclear power question that offered it, while only 3% and 4% failed to give a substantive answer on two later questions that did not; those later answers correlated with one another and predicted the respondents' later referendum vote². Satisficing theory explains part of this: a visible "don't know" can be an easy exit for people who do hold a view but would need effort to retrieve it². When asked why they said "don't know," people more often cite reasons such as ambivalence than a lack of information².

Social desirability shapes some answers

Pew lists behaviors that tend to be overstated in surveys, including church attendance, charitable giving, and likelihood of voting, and behaviors that tend to be understated, including drug and alcohol use and tax evasion. The effect is larger when an interviewer is present. Pew's turnout question offers a face-saving lead-in: "did things come up that kept you from voting, or did you happen to vote?"¹. The relationship between reported and actual voting is covered in Opinion and choice.

Margins of error describe only one kind of error

Pew notes that a poll's margin of error applies to each estimate, that the margin for the gap between two estimates is roughly double, that subgroups have larger margins, and that sampling variability "is only one of many possible sources of error," alongside wording, weighting, and nonresponse³. A result that shows two priorities one point apart is usually not evidence that one ranks above the other.

Competing interpretations

Wording effects show opinion is weak. One reading is that if a few words move answers by double digits, many people do not hold firm views. This reading has a long history in survey research.

Wording effects show opinion is specific. Another reading is that people are answering different questions. Supporting military action with no stated cost is a different proposition from supporting it with stated casualties¹. On this view, wording effects are evidence that people respond to substance, and the error lies in reporting either number as "the" public view. Research on measurement error in single survey items supports a version of this argument (see Opinion and choice).

Neutral wording is achievable. Practitioners often aim for a single best neutral phrasing.

No wording is neutral for every reader. Critics answer that category names, examples, and the presence or absence of costs always carry framing. The practical alternative is not perfect neutrality but disclosure, testing with varied audiences, and in some cases asking the question more than one way and reporting both.

Always offer "don't know." Some designers treat it as a respect for honesty.

Offer it selectively. The evidence summarized by Krosnick and Presser suggests it can lower data quality for some items². A reasonable middle position is that the choice depends on whether the question is about knowledge (where "not sure" is a true answer) or about a preference most people can form.

Application to this framework

These are design proposals for the framework and its demonstration. They are not findings.

1. Every published priority or opinion result carries its exact question text, option list, option order rule, and version number. If wording changes, the version changes and old and new results are not merged. This links to Published releases.
2. Category names are reviewed for scope. Pew's "foreign policy" versus "war on terrorism" example¹ shows that narrowing or broadening a category changes how many people choose it. Who decides the categories is a governance question, not a design detail.
3. Option lists rotate where the options have no natural order, and the rotation rule is disclosed. Ordered scales keep their order.
4. Opinion items name one proposition at a time. No double-barreled items. A proposal with several parts gets several questions, or a clearly labeled single question about the package. See Proposals.
5. Scales are fully labeled with words, using about four or five points for most attitude items¹².
6. Uncertainty options are chosen by item type. Knowledge items offer "not sure." Preference items may offer it, but the framework reports how many chose it rather than silently dropping them, because the denominator matters (see Measurement).
7. No false precision. Results that differ by less than their plausible error are shown as not distinguishable. Rankings near ties are shown as ties.
8. Priority defaults are not answers. If an interface starts sliders or tokens at a default position, an untouched default is recorded as "not confirmed," not as a choice. This mirrors the distinction in Priorities.
9. Questions are tested before use. Cognitive interviews and small pilots check whether people understand the question as intended. How that would be evaluated is in Evaluation.

Hypothetical example. A demonstration asks people to rank "Public safety" among ten subjects. One group sees the label "Public safety"; another sees "Policing." If the two groups give it different average ranks, the framework treats that as a wording effect to report, not as evidence that one label is correct.

Unresolved questions

• How should a framework choose between asking a short question many people understand and a longer one that states costs and tradeoffs? Longer questions may be more accurate about substance and less accessible to readers.
• When the same subject is asked several ways, how should results be presented without inviting readers to pick the number they prefer?
• Who reviews question wording, and how are disagreements about framing resolved? This is an open governance problem.
• How should translations be validated so that a question means the same thing in each language?
• Does rotating option order in a priority list create its own confusion for repeat participants?