An effectiveness assessment is a judgment of whether a program or policy achieves its stated goal, based on evidence. It is independent of priority: a subject can matter greatly while a specific program in it works poorly, or matter little while a program in it works well. My Vote Forward keeps effectiveness separate from priority and opinion so that “this matters” is never confused with “this works.”
Definition
The U.S. Government Accountability Office (GAO) defines effectiveness as “the extent to which a program or intervention is achieving its intended goals, as determined by a program evaluation”.1 It defines program evaluation as “an assessment using systematic data collection and analysis of one or more programs, policies, and organizations intended to assess their effectiveness and efficiency”.1 An effectiveness assessment, on this site, is any judgment about effectiveness, whether made by a formal evaluation or stated by a person as an opinion; the two must be labeled differently.
Origin and history
Program evaluation grew as a discipline alongside large public programs, and governments have increasingly written it into law and practice. GAO’s glossary, Program Evaluation: Key Terms and Concepts, published March 22, 2021, organizes the field’s vocabulary for federal decision makers.1 It distinguishes several types of evaluation by purpose:
- Process evaluation: assesses “the extent to which essential program elements are in place” and conform to requirements and design.1
- Outcome evaluation: assesses “the extent to which a program has achieved certain objectives, and how the program achieved these objectives”.1
- Impact evaluation: assesses the impact of a program on outcomes “by estimating what would have happened in the absence of the program”.1
That last phrase names the core difficulty. To say a program worked, one must estimate the counterfactual: what would have happened without it.
How effectiveness is assessed
GAO separates two tools that are often confused. Performance measurement is “the ongoing monitoring and reporting of a program’s accomplishments and progress, particularly towards its pre-established goals”.1 It tells you whether numbers moved. Program evaluation asks why, and whether the program caused the movement. A city whose traffic deaths fell after a street redesign has a performance result; an evaluation would compare similar streets that were not redesigned before concluding the design worked.
GAO also defines efficiency as the ratio of program inputs to outputs or outcomes.1 A program can be effective but expensive, or cheap but ineffective. Equity is a separate criterion again.1 A full assessment says which criterion it is applying.
Evidence quality varies. Many evaluators treat randomized studies as the strongest way to estimate a counterfactual where they are feasible and ethical, and treat comparison designs and before-and-after studies as weaker, though often they are all that exists. Evidence from one place may not transfer to another. See Evaluation for how the framework itself would be evaluated.
Common confusions and neighboring terms
- Effectiveness vs. priority. Importance is a value judgment; effectiveness is an evidence judgment. A person can rank housing first and believe the current housing program is failing.
- Effectiveness vs. opinion. A person’s belief that a program works is an opinion about effectiveness. A formal evaluation is evidence. Both can be recorded, but not under the same label.
- Effectiveness vs. resource allocation. Wanting more money for a subject does not mean the existing spending works, and an effective program may already be adequately funded.
- Output vs. outcome. Outputs are products or services delivered; outcomes are the desired results.1 Counting meals served is an output; reduced food insecurity is an outcome.
How My Vote Forward uses the term
My Vote Forward treats effectiveness assessment as one of the distinct judgments a person might make about public action, alongside priority and opinion. In the framework, people’s views on whether a program works are kept separate from their ranking of the subject and from their support for a specific proposal. Where published evaluations exist, the framework’s approach is to link the evidence rather than substitute participants’ views for it. The design is described in Tradeoffs.
What My Vote Forward can and cannot do here
My Vote Forward can keep effectiveness judgments labeled as judgments, link to published evaluations, and avoid presenting participant beliefs as evidence. It cannot evaluate public programs itself, and an aggregate of participant views about whether a program works is not a program evaluation. It cannot resolve disputes between studies; it can only show that they exist. Any effectiveness views shown in the demo are simulated.
Unresolved questions
- How should a public summary show participants’ views on effectiveness next to formal evidence without implying the two are equivalent?
- Who should decide which evaluations are credible enough to link, and how should that choice be made transparent?
- How should the framework handle programs that have never been evaluated, which is common at the local level?
Sources
- U.S. Government Accountability Office, Program Evaluation: Key Terms and Concepts, GAO-21-404SP, March 22, 2021. gao.gov/assets/gao-21-404sp.pdf
