Up for debate: Is Harris County's vote safe from the Russians?

Editor's note: Rice computer-science professor Dan Wallach's post has been updated to include a factual correction and a response from Harris County Clerk Stan Stanart. We've asked Wallach to respond in turn, and hope to post more updates from both in a rolling debate.

Last September, in the run-up to the election, we learned that Russians had attempted to attack 33 states' voter registration databases, later revised upward to 39 states. I was asked to testify about this in Congress, and my main concern was that the Russians might attempt to simply delete voters altogether, creating electoral chaos.

All the pieces were in place, but the election came and went without wide-scale problems. What happened? We know that the Obama and Putin had a "blunt" meeting at the G20 that same September, so it's possible that Obama was able to rattle Putin enough to make him pull back. Maybe Putin decided that leaking stolen emails was good enough.

We may never know the full story, but what is clear is that we need to adequately defend ourselves against future nation-state attacks on our elections, whether from Russia or elsewhere. As James Comey warned the Senate Intelligence Committee recently, "They will be back."

Computer security experts who deal with nation-state activities use the term "advanced persistent threats" (APT) as a shorthand to indicate that our adversaries have significant capabilities, including both engineering resources and spycraft, to quietly break into our computers, spread out across our networks, and avoid detection. It's common for APT attacks to last for months to years prior to detection.

Given these threats, we need to conduct a serious analysis of where our elections stand. Harris County's Hart InterCivic eSlate voting machines, for example, haven't had any major security updates following studies conducted a decade ago by the states of California and Ohio. (I was part of the California effort.) In short, an attacker need only tamper with a single voting machine. After that, the infection can spread "virally" to every machine in the county.

Compounding the problem, all of our vote-tabulating systems are running Windows 2000, for which Microsoft dropped all software support, including security patches, seven years ago 2007, which is two versions behind the latest release.

In the lead-up to the 2018 election, it may be financially infeasible for a complete replacement of our voting machines. We only just recently purchased our voting machines after a 2010 warehouse fire destroyed our original fleet of eSlate machines, so the funds aren't likely to be available so soon for replacements.

What's clearly necessary, since we know the Russians targeted voter registration systems, is a major upgrade to the way our voter registration systems are managed. A redesigned system would still, by necessity, require Internet connections so voters can verify their correct polling places, see sample ballots, and so forth. Most notably, during our early voting period, we need an online database to track which voters have cast ballots.

A modern design, intended to operate even if the entire Internet failed while the election was ongoing, would involve making local copies of the database at every voting center. Unsurprisingly, the needs of Harris County are essentially the same as the needs for every other county in our state, suggesting that a state-level procurement could be an efficient way to improve the voter registration security for every county's voters.

Another short-term recommendation will be for Harris County to upgrade its systems to the latest versions of Microsoft's operating systems, even though this will require a waiver from Texas's election certification requirements. Even though our vote tabulation systems are hopefully never connected to the Internet, they are nonetheless unacceptably weak in the present threat environment.

Likewise, Harris County needs to hire a professional security "penetration testing" firm to identify other soft points in its infrastructure and prioritize repairs; such consultants need to be brought in on a regular basis for check-up exams. We also need forensic security auditors to do a deep dive into our county's existing systems to make sure they're as clean as we hope them to be. This isn't just a matter of running some anti-virus scanner, since APT adversaries use tricks that automated scanners won't detect.

When we're talking about APT adversaries, we have to be clear about their motives and capabilities as well as our own defensive posture. As best as we can tell, the Russian goal was to influence who won the election, and they were prepared to take a variety of steps, ranging from subtle email "phishing" campaigns to outrageously visible attacks against our voter registration systems.

When faced with an adversary like this, we have to assume that they're capable of reverse-engineering our systems to build custom exploits. Given our limited resources, we need to be confident that we'll at the very least be able to detect adversarial actions, and we need to have concrete fallback plans in place, such as commissioning millions of paper ballots to be printed at the last minute.

BUT THOSE are only stop-gap measures, able to reduce only the most glaring holes in our defense.  We ought to re-engineer things properly, ensuring that no nation-state adversary will ever be able to tamper with our elections.

The most seemingly obvious solution would be hand-marked, hand-counted paper ballots. Unfortunately, there are a number of problems with this approach. For starters, in close elections, hand-marked ballots' ambiguities in voters' intent will lead to painful adjudications. We saw this problem in spades during the tight 2008 Minnesota Senate election between Norm Coleman and Al Franken. (Minnesota Public Radio hosts a great interactive website with many of the challenged ballots.)

Furthermore, manual tabulation would be far too slow, particularly when we have vast numbers of judicial offices on our ballots. Electronic scanning and tallying can be significantly faster, and is considered the best practice available with today's voting technologies. Doing it securely requires conducting compliance audits to make sure the electronic tallies correspond to the paper ballots, and the procedural details matter.

One of our big lessons from the recounts in Wisconsin and Michigan this year was that poor recount procedures can be used to mask underlying problems. In Michigan, if the seals on a paper ballot box are broken, then the contents of the box will never be recounted. This is great news if you want to tamper with an election, but it's bad news if you want to detect election tampering.

Back in 2011, Dana DeBeauvoir, the county clerk of Travis County, which uses the same eSlate systems as Harris County, decided that their voting systems were reaching the end of their service lifetime, yet nothing then on the election systems market was a suitable replacement for Travis County's needs.

Consequently, DeBeauvoir asked me and a number of other election experts to help design something completely new, from scratch, with the intent of being secure against a variety of adversarial models, transparent to election observers even in the face of possible computer tampering, auditable for recounts and testing purposes, and reliable in the face of all the things that can go wrong. And it also has to also be easy to use and cheap to maintain.

Our resulting STAR-Vote design features both securely encrypted electronic ballots and human-readable printed paper ballots. A touch-screen interface helps voters with their choices and eliminates issues of ambiguous intent. Electronic ballots can be tallied rapidly, while paper ballots can be sampled to make sure that the electronic ballots are accurate. Pragmatically, STAR-Vote is meant to run on off-the-shelf tablet computers, costing a fraction of the price of dedicated voting machines.

We then wrote a readable academic paper and a detailed request for proposals. Travis County's government allocated $4 million to start a procurement process and is currently evaluating 12 submissions. Travis County estimates it will take an additional $6 million to complete the process, yielding all the necessary software, and then another $2 million to $3 million to purchase the equipment.

It seems unlikely that we'll see an infusion of federal dollars to pay for new gear like we saw in 2002's Help America Vote Act. Consequently, we need the State of Texas to step up to the plate. By accelerating the funding for STAR-Vote, Texas could hasten a new option for every county in the state, many of which are locked into older insecure voting systems, while operating on limited budgets.

Briefly, I'll mention two seemingly attractive alternatives so I can explain how complicated voting security can be to get right. One alternative is to conduct the election entirely through the postal mail, which is how everybody in Oregon and Washington State, and a substantial fraction of California, cast their votes. The essential problem with postal voting is that it completely surrenders the privacy of the voting booth, enabling coercion (from a spouse, employer, union, or other community group), vote selling, and other sorts of local election fraud.

Some have even suggested we vote over the Internet, which combines all the disadvantages of postal voting with all the risks from voters' poorly secured personal computers. If we care at all about the integrity of our elections, then it's valuable to stick with voters going to polling places, where we can have well-designed systems, ensuring that, no matter what the Russians have in mind, they won't be able to tamper with our votes.

Dan S. Wallach is a professor in the Department of Computer Science and a Rice Scholar in the Baker Institute for Public Policy at Rice University. He has also served on the Air Force Science Advisory Board.

Harris County Clerk Stan Stanart responds:

First and foremost, the title of the article "Harris County's vote isn't safe from the Russians" is unproven, misleading, factually inaccurate, and highly inflammatory. (Editor's note: The headline has been changed to reflect the current nature of this article, including Stanart's response.  Wallach stands by the original headline.)

Apparently, the author is purposely misleading or is unfamiliar with the security of Harris County's voting system as there are many inaccuracies in the article. The comment in the first paragraph, "the Russians might attempt to simply delete voters altogether," does not take into account that voter registration does daily backups, that there are logs of every change made, and that the Secretary of State obtains copies nightly of any changes or updates made to the voter roll.

In addition, my office receives daily all updates to the voter roll. We back up these daily and store them on and offsite. There are many other processes in place to identify suspicious activity and we have the ability to restore a previous version, if it was ever needed. The copy of the voter roll on the voter registrar's website, or HarrisVotes.com, is a separate system that is used by the voter for finding their polling location is simply one of many copies and not the official version used to conduct an election. How can any rational person really believe that the Voter Registrar's copies and backups, the County Clerk's copies and backups, and Secretary of State's copies and backups could be hacked at the same time without anyone noticing?

The statement by the author that "an attacker need only tamper with a single voting machine. After that, the infection can spread "virally" to every machine in the county" is ridiculous.

Apparently, there is an attempt to deceive or a total lack of knowledge of the software on the Hart eSlate voting machines. The software is very specific in its functionality; and does not contain the functionality to download code from another unit so as to spread a virus to other machines when connected. In addition, even if the functionality was there, the connections are limited to a single string of 1-12 voting machines making it impossible to spread to "every machine in the county".

We run hash code tests to verify the integrity of the voting software code and the only thing that changes from election to election is the ballot specific items such as candidates and propositions. We also run a publicly held logic and accuracy test that is overseen by appointees from both major political parties for every election to ensure that the voting machines are accurately recording the vote.

The accusation that the voter tabulation system is running windows 2000 is simply false. All tabulation systems are running on Windows 7. In addition, these machines are never connected to any network, that includes the county network or to the internet. There is no access or opportunity for hacking, mischief or manipulation.

The author states that we need a major upgrade to our systems so that voters can verify their polling locations. Apparently, he has never visited Harrisvotes.com during an election cycle because I put this in place approximately five years ago. HarrisVotes.com runs on a separate copy of the voter roll so that the voter can see their polling locations, print their personal sample ballot, and obtain a Google map to their voting location.

The author once again is wrong when he states that we need an online database to track voters who cast votes during early voting. Harris County has been doing this for over a decade. Per Texas Election code, we provide the list of early voters the next day.

I understand the author has considerable time invested in the Star-Vote system that he worked on with the Travis County Clerk. Our understanding is the system is not complete and has not been certified by Federal Election Assistance Commission or the Texas Secretary of State making it illegal for us, or any county in Texas to use it. Harris County has a significant investment in our current Hart eSlate voting system. We should be able to easily get an additional five years of use from our existing technology after which, we will be evaluating the most current and secure election technology at that time to be the replacement product. As the author mentions, he was part of the team that evaluated the Hart eSlate voting system in California and attempted to manipulate it for eight weeks and was unable to change any votes, add any votes, or corrupt any votes.

The concept of taking off-the-shelf computers without them going through a full testing cycle is dangerous. Off-the-shelf equipment has firmware that's embedded within the machine that is more susceptible to hacking than the technology in the secure Hart machines used in Harris County.

In the earlier years of my career, I was the manager of BIOS Development and Test for consumer PCs at Compaq computer. I have extensive knowledge and experience in the internals of a personal computer. Without fully testing every piece of code that could actually run on a PC including the BIOS and all the firmware, and operating system of a computer, it is not possible to ensure the integrity of the whole system.

This is why the Federal Elections Assistance Commission has voting machines standards and a full suite of tests that a voting machine must pass to ensure the integrity of a voting system. The complexity of a complete operating system whether that be Windows, Apple's IOS, Android, etc., makes it much much more complicated to fully test all of the potential code that could interact with any voting software. That is why having a very small, special-purpose, dedicated operating system that does not have the ability to transmit viruses is a much more secure and safe system to have as the basis for our voting machines.

Stan Stanart was elected the county clerk of Harris County in 2010. He graduated from Oklahoma State University with a Bachelor of Science in electronic engineering, and worked 16 years in the aerospace industry. In 1994 he joined Compaq Computer Corporation, where he was a developer and advanced system architect and held numerous managerial positions. In 2007 he joined the Harris County Tax Office as Manager of Hardware Information Technology.

Dan Wallach responds:

Stan Stanart has disputed many of the things that I wrote about the security of Harris County's voter registration system and Hart InterCivic voting system. Before I get into a point-by-point response, I'll first note that this debate isn't new. Back in 2008, I testified about this issue before the Texas House Committee on Elections, where several of the election equipment vendors testified. I wrote a blog piece that responded to much of the misinformation they were saying at the time, and it's unfortunate that I need to reiterate many of the same issues again now, a decade later.

Voter registration databases. Stanart notes that Harris County does regular backups with the Secretary of State's office. That's great news, and it's a necessary step. Unfortunately, it's not sufficient if we consider what a nation-state/advanced-persistent-threat (APT) adversary is capable of doing.

For example, during our early voting period, we maintain an online database that tracks which voters have and have not cast ballots. Under the APT threat model, it's possible for our adversary to simply mark voters as having cast ballots, even though they didn't. This would manifest itself with voters arriving to do their civic duty and being turned away.

Alternately, our adversaries could slowly introduce errors into the database over time, such that our backups were every bit as tampered as the primary databases. All of these attacks would be categorized as "denial of service" attacks, which are generally the hardest class of attacks to defend against.

Prior to 2016, no voter registration management system in the country was explicitly engineered with this attack model in mind. Now, every voter registration management system must be engineered with APT adversaries in mind, and that's going to require a large engineering effort.

Stanart writes, "The author once again is wrong when he states that we need an online database to track voters who cast votes during early voting. Harris County has been doing this for over a decade. Per Texas Election code, we provide the list of early voters the next day."

When a voter goes to an Early Voting location, there's a computer that checks whether that voter has already cast their ballot. That computer must be connected to a database. If that database is online, then it's potentially vulnerable to attack. If that database is offline (i.e., a local copy at each polling location), then there's a risk that a voter can drive from one early voting location to another and vote multiple times.

A best practice would be to maintain both online and offline copies, preferably using the online database, but maintaining offline copies if the Internet becomes unavailable or if tampering is detected (i.e.., by the actions of an APT adversary). We also need to design suitable auditing procedures to carry out during the Early Voting period to validate the system.

In short, our practices were perfectly acceptable under our previous understanding of the threat. With what we learned of Russian activity last year, we need to step up and improve our game.

Windows 2000 vs. Windows 7, and general IT issues. I last visited the Harris County elections warehouse in 2012 as part of a recount effort. At the time they were running Windows 2000 on all their systems.

Since then, Harris County upgraded to Windows 7, which is commendable. Microsoft dropped "mainstream support" for Windows 7 two years ago but still provides security patches; newer versions of Windows would still be preferable from a security perspective.

The challenge for Harris County is maintaining a regular cadence of security patches for machines that must never be connected to the Internet. Microsoft does allow for security patches to be copied to CDs and installed offline, but it's a cumbersome process. Alternately, if these machines were even briefly connected to the Internet to run Windows Update, that would provide a window for external attackers to break in. APT adversaries make a point of watching for these sorts of transitory opportunities to attack systems.

In short, our IT management practices require a detailed review in light of the new threats that we face. Furthermore, we require a detailed forensic investigation to make sure that we weren't already compromised. Again, we need to improve our game.

Prior voting systems' security reviews. The states of California and Ohio commissioned security analyses in 2007 of all the major electronic voting systems then on the market, including the Hart InterCivic eSlate system that we use here in Harris County. I was part of the California team that analyzed the source code to the Hart InterCivic system. Our report describes a variety of vulnerabilities.

Notably, every eSlate has a network connection on its back panel. This network connection is used for a variety of purposes, including connecting the eSlates in a local precinct during an election, and also in the elections warehouse to load ballot definitions, extract copies of the votes, and do basic inventory tracking functions.

In conducting our analysis, we found that the commands we could issue, via this networking port, could overwrite the code and data within an eSlate. Even worse, when a compromised eSlate is connected to computer servers in the elections warehouse, it turns out that those servers have "buffer overflow vulnerabilities" that allow an eSlate to attack them. At that point, a compromised server can use the very same network interface to attack every subsequent eSlate machine that connects to it. This "viral" vulnerability was central to our 2007 report and led the State of California to issue serious restrictions (see the top paragraph on page 5), requiring the whole server to be rebooted after each and every eSlate machine was connected to it.

These sorts of attacks can easily defeat the "hash code testing" and "logic and accuracy testing" that Harris County regularly conducts. Hash code testing, as implemented by Hart InterCivic, is analogous to an airline employee asking you if you've packed your own bags. It's trivial for you to lie. Logic and accuracy testing has a handful of votes cast for each candidate, in a controlled environment, where the final totals should come out exactly as expected. Malware can generally detect when it's operating under such testing conditions, for example behaving correctly until a certain volume of votes have been cast, or waiting until an unusual write-in vote is cast before commencing with its misbehavior.

In short, our current security practices are not an adequate mitigation against the vulnerabilities of the voting machines that we use and the threats that they potentially face.

How did Hart InterCivic respond? In early 2008, a Hart InterCivic representative and I both testified at a hearing of the Texas House Committee on Elections. I subsequently wrote a blog piece picking apart their misinformation at the time.

Anyone can look at the Texas Secretary of State's web site, which shows all of the vendors' submissions of voting systems for certification in the state. On the Hart InterCivic page, the last submission by Hart InterCivic for the voting system that we use in our current elections was examined by the state in January 2008; this was the same version (6.2.1) that we considered in the California review. Hart InterCivic has issued no software updates since then!

Instead, Hart InterCivic has developed a new voting system, effectively discontinuing its software support for the voting systems that we use here. When Harris County bought our fleet of Hart InterCivic equipment in 2010, we unfortunately purchased a system that was already obsolete. We're running a voting system with vulnerabilities that have been known to the public for a decade. The hardware should continue functioning, but whatever problems exist in its software aren't going to be fixed.

Furthermore, the analysis we did in 2007 does not represent a comprehensive catalog of every possible vulnerability in Hart InterCivic's system. In the past decade, the toolbox available to attackers has radically improved. In the same way that Harris County was correct to upgrade its antiquated Windows 2000 systems to run the newer Windows 7, it similarly needs Hart InterCivic to conduct regular software maintenance and updates on its voting systems, which isn't happening.

In short, our vendor is no longer supporting our voting systems, leaving them in a vulnerable state. We need to have plans for moving on.

STAR-Vote. Travis County votes on the same Hart InterCivic system that we use here in Harris County. Facing the same obsolescence issues, they recruited me and a number of other experts in cryptography, computer security, statistics, and usability, to collectively design something better. Travis County is now in the procurement stage, looking for the right vendor or vendors to implement STAR-Vote to meet our detailed specifications. Once it's complete, it will go through the same testing and certification that all Texas voting systems are required to pass, and the resulting system would be suitable for use across the entire state.

Stanart specifically denigrates general-purpose PCs in favor of dedicated devices, from a security perspective. This is an outdated viewpoint. In recent years, our smartphones and PCs have incorporated "trusted boot" and "code signing" features that make it very difficult for malware to be installed on these computers. This specific issue came to a head in the dispute between the FBI and Apple over the "San Bernadino iPhone," wherein the FBI demanded Apple create a version of iOS specifically to overcome the security features preventing the FBI from accessing the contents of the phone. (The FBI subsequently paid almost $1 million to an undisclosed firm for an exploit to work around Apple's security features.)

As with Apple's iPhone security, the "trusted boot" security features of modern PCs are quite strong but far from perfect, which is why STAR-Vote uses sophisticated "end-to-end" cryptographic mechanisms that allow voters to take home a receipt that lets them verify that their vote was tabulated correctly while not being able to compromise the privacy of their vote.STAR-Vote even includes a "blockchain" feature that allows one voter's receipt to verify the integrity of other votes cast earlier in the day.

On top of all that, STAR-Vote requires the use of printed paper ballots, cast in a standard ballot box, which no computer can possibly tamper with once printed. These and other security features allow STAR-Vote to minimize its reliance on its software being free of bugs.

In short, we know how to build stronger voting systems, and we can do it at a reasonable cost.

Bookmark Gray Matters. As James Comey warned the Senate Intelligence Committee recently, "They will be back."