Threat model: a structured account of what could go wrong

A threat model is a structured account of a system, the ways it could be attacked or fail, who might attack it, and what the system assumes is trustworthy. NIST describes threat modeling as “a form of risk assessment that models aspects of the attack and defense sides of a logical entity.” Any statement that a system is “secure” is only meaningful relative to a threat model, which is why My Vote Forward states its assumptions instead of promising security.

Definition

NIST SP 800-53 Rev. 5 defines threat modeling as “a form of risk assessment that models aspects of the attack and defense sides of a logical entity, such as a piece of data, an application, a host, a system, or an environment”.⁠¹ The Threat Modeling Manifesto, written by a working group of practitioners, defines it more simply: “Threat modeling is analyzing representations of a system to highlight concerns about security and privacy characteristics”.⁠²

A threat model usually has four parts: what is being protected (assets such as ballots, identities, or published results), who might act against it (adversaries and their capabilities), how they might do it (attacks and failures), and what is trusted (assumptions such as “the device is not compromised” or “officials follow procedures”).

Origin and history

Threat modeling is built into some software development processes; Microsoft, for example, describes its Threat Modeling Tool as “a core element of the Microsoft Security Development Lifecycle (SDL)”.⁠³ The Manifesto organizes it around four questions:⁠²

1. “What are we working on?”
2. “What can go wrong?”
3. “What are we going to do about it?”
4. “Did we do a good enough job?”

Microsoft’s STRIDE model, used in that tool, is one checklist for the second question. It sorts threats into six categories: spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege.⁠³ For example, tampering “involves the malicious modification of data,” and information disclosure “involves the exposure of information to individuals who are not supposed to have access to it”.⁠³

How threat models are used in elections and civic systems

In elections, threat models shape what evidence is required. The principle of software independence is a threat-model statement: it assumes software may be wrong or malicious, and requires that “an undetected change or error in voting system software should be incapable of causing an undetectable change or error in an election outcome”.⁠⁴ Coercion research uses explicit adversary models; Juels, Catalano, and Jakobsson consider an adversary who may demand that voters “vote in a particular manner, abstain from voting, or even disclose their secret keys”.⁠⁵

Applied to a hypothetical civic participation system, STRIDE prompts concrete questions. Spoofing: can one person act as many? Tampering: can stored opinions be altered? Repudiation: can an operator deny publishing a release? Information disclosure: can small-group totals reveal a person’s answer? Denial of service: what happens when the network fails during a scheduled release? Elevation of privilege: can a staff account read private data? Each question needs a stated answer or an admitted gap.

Common confusions and neighboring terms

• Threat model vs. risk assessment. NIST treats threat modeling as a form of risk assessment;⁠¹ a full risk assessment also weighs likelihood and impact.
• Threat model vs. security guarantee. A threat model limits claims. A system may resist every threat in its model and still fail to one outside it.
• Threat vs. vulnerability. A threat is a possible adversary or event; a vulnerability is a weakness it could exploit.
• Threat model vs. chain of custody. Custody procedures are controls chosen because the threat model includes tampering with physical materials.

How My Vote Forward uses the term

My Vote Forward does not claim to be secure in general. Its Security chapter states what the framework would need to protect, from whom, and under what assumptions, and it names threats that technology does not solve, such as household coercion. Privacy threats from small groups are treated in Privacy. The framework avoids “100% secure” and “zero fraud” claims because no threat model supports them.

Unresolved questions

• Who should review a civic system’s threat model, and how often should it be revised?
• Should a published threat model omit details that would help attackers, and if so, how is the public to trust what is left out?
• How should insider threats, such as a dishonest operator, be weighed against outside attackers?