Washington, D.C. 2010 internet voting test: a public trial that was breached

In September and October 2010, the District of Columbia Board of Elections and Ethics (BOEE) ran a public mock election on its Digital Vote-by-Mail system, an internet ballot return pilot meant for military and overseas voters. Researchers from the University of Michigan gained near complete control of the election server within about 48 hours, changed every vote, exposed almost every secret ballot, and were not detected for nearly two business days; the District then dropped electronic ballot return for the November 2010 general election.⁠^1,2

In plain words

Washington, D.C. built a website so voters living overseas could download a ballot, fill it out on their computer, and upload it back. Before using it in a real election, the District did something unusual and brave: it opened the site for a few days and invited anyone to try to break it.

A team of computer scientists from the University of Michigan did break it. They found one small mistake in how the website handled uploaded files. That mistake let them run their own commands on the server. They swapped every ballot for a fake one, set the system to reveal how people voted, and made the “thank you” page play their university fight song.

Nobody running the system noticed for about a day and a half. When officials found out, they stopped the test and decided not to let real voters send ballots back over the internet that year. Voters could still download a ballot, print it, and mail it.

Defined terms

• Internet ballot return. Sending a marked ballot back to election officials over the internet, by web upload, email, or an app. It is different from electronic ballot delivery, where only the blank ballot travels online and the marked ballot comes back on paper.
• UOCAVA voters. Voters covered by the Uniformed and Overseas Citizens Absentee Voting Act: members of the uniformed services, their families, and citizens living abroad. The 2009 MOVE Act amendments require states to offer electronic delivery of blank absentee ballots to these voters and to send requested ballots at least 45 days before a federal election.⁠⁴
• Shell injection. A software flaw in which text supplied by a user is passed to the operating system’s command interpreter, so that crafted input runs as a command.
• Public penetration test. A period in which outsiders are invited to attack a system so its weaknesses can be found before real use.
• Intrusion detection system (IDS). Equipment or software that watches network traffic or system activity for signs of attack.
• Ballot secrecy. The requirement that no one can learn how a particular voter voted. See the secret ballot and privacy.

What happened

The District’s pilot grew out of federal pressure to serve overseas voters faster. The 2009 MOVE Act required states to set up electronic transmission of blank ballots to UOCAVA voters.⁠⁴ Many jurisdictions went further and explored electronic return. The researchers note that as of 2010, according to Verified Voting figures they cite, 19 U.S. states used some form of internet voting.⁠¹

The District developed its system, officially the “D.C. Digital Vote-by-Mail Service,” as open source software with the Open Source Digital Voting Foundation’s TrustTheVote project.⁠¹ It then held a mock election open to public testers. According to J. Alex Halderman, who led the Michigan team, the public testbed launched on Tuesday, September 28, 2010.⁠² The researchers write that the test “ran for four days in September and October 2010”.⁠¹

The team found the decisive flaw after “a few hours of examination” of the published source code.⁠¹ Halderman’s account says the team had control of the server “within 36 hours of the system going live”,⁠² and the paper says “within 48 hours of the system going live, we had gained near-complete control of the election server”.⁠¹ The two figures describe slightly different milestones and are reported here as each source states them.

After completing the attack and leaving a musical “calling card,” the team waited. The paper reports “a delay of approximately 36 hours before election officials responded and took down the pilot servers”.⁠¹ Officials appear to have learned of the attack from a mailing list message asking what tune played for successful voters; they then suspended the public examination “five days sooner than scheduled, citing ‘usability issues’”.⁠¹

The Board then stated, as quoted by Halderman, that it “will not proceed with a live deployment of electronic ballot return at this time”.⁠² The testbed returned with electronic return disabled, and voters could still download ballots, print them, and return them by mail.⁠²

How it worked

The Digital Vote-by-Mail architecture was conventional for a 2010 web application. It used the Ruby on Rails framework on an Apache web server, with a MySQL database storing registered voters, hashed credentials, precinct to ballot mappings, and which voters had voted.⁠¹ A front end web server accepted HTTPS requests and passed them to an application server; firewalls blocked outbound connections from both servers.⁠¹

Each eligible voter received a letter by postal mail containing a voter ID number, registered name, residence ZIP code, and a 16 character hexadecimal PIN.⁠¹ The voter chose digital or postal return, logged in, affirmed identity, downloaded a PDF ballot, marked it in a PDF reader, and uploaded it.⁠¹ To protect secrecy, the server encrypted each completed ballot with a public key whose private key officials held offline. After the election, officials were to move the encrypted ballots to a non-networked “crypto workstation,” decrypt them, and print them for counting alongside mail ballots.⁠¹

The flaw sat in the code that encrypted uploaded ballots. The application built a gpg command as a text string that included the uploaded file’s path, then ran it through the shell. The file upload plugin preserved the uploaded file’s extension without filtering, and in the Bash shell, double quotes do not stop command substitution, so “a ballot named foo.$(cmd) will result in the execution of cmd with the privileges of the web application”.⁠¹ The researchers observe that the mistake “could have been prevented by using single quotes instead of double quotes”.⁠¹

Because outbound traffic was blocked, the team wrote command output into the server’s public images directory and fetched it over the web.⁠¹ The intrusion detection device sat in front of the web server and could not read the encrypted HTTPS traffic that carried the exploit.⁠¹

Evidence: measured findings

The 2010 D.C. findings come mainly from one peer-reviewed paper, presented at the 16th Conference on Financial Cryptography and Data Security in February 2012,⁠¹ and the lead author’s contemporaneous account.⁠² The District’s own after-action report is cited in the paper; this review could not open a copy of it, so the Board’s position appears here only as the researchers quoted it.

What the researchers report doing with their access:⁠¹

• Stealing secrets. They copied the ballot encryption public key and found database credentials in a shell history file.
• Changing past and future votes. They used the public key to replace every encrypted ballot then on the server with a forged ballot, and changed the ballot processing code so later ballots were copied out and replaced.
• Revealing ballots. Unencrypted copies of uploaded ballots remained in a temporary directory. Using login events and file names in the application logs, the team could associate ballots with voters, so they “could violate the secret ballot for past and future voters.”
• Finding real credentials. A temporary directory held a 937 page PDF containing the instruction letters, with real voters’ credentials, for the live election that was to begin days later. Because credentials were delivered by postal mail, the researchers judged that replacing them in time would have been infeasible.
• Hiding tracks. They edited the application logs to remove their uploads.

The paper also lists flaws the team did not need: test credentials identical to those in the public code repository; a session cookie encryption key left at the published default; session identifiers based on sequential database keys, which together allowed forged sessions for any voter; an application account allowed to rewrite the application’s own code; and an unpatched kernel with a known local root exploit.⁠¹

Separately, the team reached network equipment on the pilot network. A terminal server accepted the default password from its online manual, and through it the team logged administrators’ keystrokes and captured router and switch passwords.⁠¹ D.C. officials later told the researchers that those routers and switches were not part of the trial but were destined for the District’s core network.⁠¹ Two webcams with no password showed the server room.⁠¹ The terminal server’s logs showed password guessing from addresses in Iran, New Jersey, India, and China, which the researchers believe were routine internet scanning rather than attacks aimed at the election.⁠¹

On detection, the researchers report that officials found the code changes by comparing the server disk to an earlier snapshot, which “required several days of analysis,” that the attacks did not appear in IDS logs, and that officials “did not discover the attack until they noticed our intentional calling card”.⁠¹

Failures and criticisms

The D.C. failure was not a failure of encryption. The ballots were encrypted with a key held offline, yet the attackers never needed the private key: they substituted ballots using the public key and read plaintext copies left on disk.⁠¹ Encryption protected one copy of the data while the surrounding system leaked others. This is the general point made on security: encryption is necessary and far from sufficient.

The test design also drew criticism from the researchers themselves. The final start date was announced only three days in advance, the scheduled end was only three days before real voting, legal protection for testers was unclear, and the access logs showed only “a handful of attack probes from other testers”.⁠¹ The authors argue that “a lack of problems found in testing does not imply a lack of problems in the system”.⁠¹

The Board’s account of the cause differed from the researchers’. The researchers quote the after-action report as saying an untested new version of the upload plugin “had been released and included in the deployed software” and “did not perform filename checks as expected”; they argue instead that the vulnerable code was a custom copy of the plugin’s routine, and that a similar flaw in the plugin itself had been fixed months earlier.⁠¹ This page does not resolve that dispute; it reports both accounts.

Some observers suggested the team had an unfair advantage because the code was public. The researchers answer that motivated attackers can obtain source code through leaks, insiders, or compromised developer machines, so open code makes a test more realistic, not less.⁠¹

Competing interpretations

The 2010 D.C. test proves internet return is unsafe. The researchers conclude that “securing Internet voting in practice will require significant fundamental advances in computer security” and urge proponents to reconsider deployment “until and unless major breakthroughs are achieved”.⁠¹ The National Academies’ 2018 report later recommended that “Internet voting should not be used at the present time”,⁠³ and the 2020 federal joint risk assessment on electronic ballot return, summarized on security, reached a similar conclusion.⁠⁵

The D.C. test proves only that one system was badly built. A strong counterargument is that the specific errors (a quoting mistake, default passwords, an unpatched kernel) were avoidable engineering lapses, not laws of nature, and that better engineered systems, end-to-end verifiable protocols, and independent certification can reduce such risk. Proponents of remote voting for overseas and disabled voters also stress that mail ballots from abroad often arrive late or not at all, a real access cost. The researchers respond that such errors are “extremely hard to eradicate” because they can exist in a “multitude of potential places,” that a single miss can give “control of the entire system,” and that end-to-end verifiable schemes “may prove even more difficult for developers and election officials to implement correctly”.⁠¹

The test was a success. A third reading treats the episode as the process working: the District tested before real use, found a fatal weakness, and changed course before any real ballot was at risk. The researchers themselves say the Board “should be commended for running a public test”.⁠¹

Lessons

These are interpretations drawn from the record, not findings of the paper unless cited.

1. Testing is asymmetric evidence. A breach proves weakness; a quiet test proves little, especially a short test with few participants and unclear legal protection.⁠¹
2. Detection and recovery matter as much as prevention. The operators could not see the attack in their monitoring tools, and the researchers argue that recovery is difficult when the ballot processing code itself is suspect.⁠¹
3. Secrecy and integrity pull against each other in software. Logs and backups that help recovery can expose ballots; encryption that protects secrecy makes tampering harder to spot.⁠¹
4. Credential delivery is part of the attack surface. Mailed credentials could not be reissued in time once exposed.⁠¹ See identity and registration.
5. Ballot delivery and ballot return are different risks. The District kept electronic delivery with paper return, which Halderman described as “a lot less risky”.⁠²

Application to this framework

The 2010 D.C. test bears directly on the “My vote” part of the My Vote Forward framework. The framework treats private voting as a separate decision with requirements of secrecy and verifiability, and it treats official remote voting as a future research and requirements program, not a built feature (see private voting). The D.C. record is one reason for that separation.

The case also supports the framework’s insistence on three assurance layers: the educational website, the demo, and any future official election system. The D.C. pilot mixed a public test network with equipment destined for the District’s core network,⁠¹ which is the kind of boundary failure the layered model is meant to prevent.

Finally, the case informs the framework’s approach to evidence. The audits and verification chapter argues that trust should rest on evidence that does not depend on trusting the software. In D.C., the only independent record that survived the compromise was the paper path voters could still use.

Unresolved questions

• Would a longer, well-publicized test with clear legal safe harbor and paid testers have found the same flaws faster, and how should a jurisdiction weigh a “clean” test result?
• Can end-to-end verifiable internet voting protocols, now under evaluation for polling place use in the United States (see security), ever address the client device and coercion problems that remote voting adds?
• What is the right comparison baseline for overseas voters: internet return risk, or the documented risks of late and lost postal ballots?
• How should election offices separate test infrastructure from production networks, and who audits that separation?
• What did the District’s after-action report conclude in full? This review could not open it and relies on the researchers’ quotations.