Evaluation

In plain words

It is easy to build something and believe it works. It is harder to check.

My Vote Forward is an educational framework with a demonstration that uses simulated data. No one has yet measured whether it helps people understand public decisions, whether people find it fair, or whether they come back. This page says what should be measured, how, and what would count as failure.

Everything below about expected effects is a hypothesis. Hypotheses are ideas to test. They are not results, and this site does not report results it does not have.

Defined terms

Hypothesis. A specific, testable statement about an expected effect, written before data are collected.

Measured result. A finding from data collected under a stated method. As of this version, the framework has none to report.

Comprehension. Whether a person can correctly explain or apply a concept, such as the difference between an opinion and a ballot, rather than only saying they understand it.

Fair-process perception. A person's judgment that a decision procedure was fair, separate from whether they liked the outcome. Research on procedural fairness and legitimacy studies this distinction¹.

Perceived legitimacy. The belief that a decision or institution deserves acceptance.

Repeat use. Whether people return to the framework over time, measured in aggregate without building individual political profiles.

Correction handling. How the operator receives, decides on, publishes, and records corrections to content or releases.

Independent evaluation. An evaluation by people without conflicts of interest in the project's success and functionally separate from its operators².

Self-report. Information people give about themselves. It is useful, and it is subject to the biases described in Question design and Opinion and choice.

Evidence

This section reports existing research that informs the evaluation design. It reports no results about My Vote Forward.

Procedures can affect perceived legitimacy

Persson, Esaiasson, and Gilljam ran a controlled field experiment with 215 high school students who were subject to a decision on a collective issue, made either by direct vote or by the teacher, with or without prior deliberation. They found that "both voting and deliberation significantly increase perceived legitimacy" compared with a procedure lacking both, that combining voting and deliberation did not produce significantly higher legitimacy than voting alone, and that perceived influence over the process mediated the effects, while the epistemic quality of the decision did not¹. This is one experiment in a school setting. It suggests that perceived influence is worth measuring; it does not show that any online framework produces the same effect.

Evaluation of participatory processes is uncommon and often self-reported

The OECD's 2021 Evaluation Guidelines for Representative Deliberative Processes report that, in the OECD's 2020 review, the most common evaluation practice (67%) was self-reporting by participants, 17% had research-oriented academic analysis, and "only seven per cent have had an independent evaluation"². The guidelines recommend independent evaluation "as a gold standard," while recognizing it may not be feasible for small processes². They organize a comprehensive evaluation into three parts: process design integrity, the deliberative experience, and pathways to impact, with methods including member surveys, public surveys, document review, observation, interviews, media review, and policy analysis². The guidelines address representative deliberative processes such as citizens' assemblies, not online priority tools, so they are used here as a structural reference, not a standard this framework meets.

Self-reports need careful design

Survey research shows that satisficing, acquiescence, and social desirability can distort self-reports³⁴. Asking "did you understand?" invites agreement. Comprehension is better measured by asking people to apply a concept. Agree/disagree formats in particular should be replaced by choices between specific statements where possible⁴.

Participants are not the public

Opt-in samples overrepresent the civically engaged⁵. An evaluation drawn only from people who chose to use the framework will describe enthusiastic users. The measurement problems are covered in Measurement.

Competing interpretations

Satisfaction is the right outcome. Many civic technology projects report that users liked the experience.

Satisfaction is a weak outcome. The OECD finding that most evaluations are self-reported² and the known biases of self-report³⁴ suggest satisfaction alone cannot show that a process improved understanding or fairness. This framework treats satisfaction as secondary.

Perceived fairness is the goal. If people judge a process fair, they may be more willing to accept outcomes they dislike.

Perceived fairness can be produced without substance. A competing concern, offered here as interpretation, is that a process can feel fair while giving participants no real influence. The Swedish experiment found perceived influence to be a mediator¹, which makes the distinction between perceived and actual influence central. The framework's educational status means participants currently have no influence over official decisions, and evaluation must not imply otherwise (see Proposals and Representation).

Repeat use shows value. People return to useful things.

Repeat use can show habit or design pressure. Notifications, streaks, and social pressure raise return rates without improving understanding. The framework excludes those mechanisms, so repeat use would be a cleaner, though smaller, signal.

Application to this framework

All items below are proposed measures and labeled hypotheses. None has been tested.

Comprehension

• Hypothesis C1. After using the framework's explanation, participants will more often correctly distinguish an opinion from a ballot than people who read a short neutral text of similar length.
• Proposed measure. Scenario items that require applying the distinction (for example, deciding whether a described action reveals a vote), scored right or wrong, not "do you understand?" items.
• Proposed design. Random assignment between the framework explanation and a comparison text, with items written and fixed before data collection.
• Failure would look like. No difference, or a difference that disappears on items not shown in the explanation.

Fair-process perception

• Hypothesis F1. Participants who see the published question wording, method, and denominator for a result will rate the process as fairer than those who see the result alone.
• Hypothesis F2. That difference will be smaller among participants whose own priorities ranked low in the result.
• Proposed measure. Separate items on process fairness and outcome favorability, so the two are not confused, using fully labeled scales⁴.
• Failure would look like. Transparency having no effect, or increasing perceived fairness only among people who liked the outcome.

Repeat use

• Hypothesis R1. A share of participants will return to update priorities when a new proposal stage begins in a demonstration scenario (see Participation over time).
• Proposed measure. Aggregate counts of returning accounts per period, under the privacy rules in Privacy. No per-person political histories, and no political content in analytics.
• Failure would look like. Returns concentrated in a small, already engaged group, which would be reported as such.

Correction handling

• Hypothesis K1. Corrections submitted through the structured contact path will receive a recorded decision within a published target time.
• Proposed measure. Counts of corrections received, accepted, rejected with reasons, and time to decision, published in the changelog and release records (see Published releases).
• Failure would look like. Untracked corrections, silent edits, or missing reasons.

Who evaluates

Following the OECD's distinction², the framework's operators can run internal checks, but claims of effectiveness should wait for evaluation by people independent of the project, with conflicts disclosed. Evaluation questions themselves should be reviewed under Governance. The demonstration uses simulated election state; evaluations of the demo are evaluations of an educational tool, not of any voting system (see Security).

What will not be done

• No reporting of hypotheses as findings.
• No testimonials presented as evidence of effect.
• No evaluation that requires linking individuals to ballot content or building individual political profiles.
• No changing outcome measures after seeing data without reporting the change.

Unresolved questions

• Who would fund an independent evaluation without creating a conflict of interest? See Governance.
• What comparison condition is fair for an educational framework: a text, a video, a civics lesson, or nothing?
• Can fair-process perception be measured in a demonstration where participants know outcomes are simulated?
• How should negative or null findings be published so that they are as visible as positive ones?
• What sample sizes would be needed to detect modest comprehension effects, and can those be reached without paying participants, given the insincere-response risks described in Measurement?