Senate Elections, Reapportionment & Constitutional Amendments

Are California’s Voting Systems Accurate, Reliable and Secure?

A Critical Look at the Federal Testing and Certification Process

Debra Bowen, Chair

Menlo Park City Council Chambers

February 16, 2006

VICE-MAYOR KELLY J. FERGUSSON: Good afternoon and welcome to the Menlo Park City Council Chambers. I’m Kelly Fergusson, the vice-mayor of Menlo Park, and I want to extend to you and to Senator Bowen and our speakers this afternoon a very, very warm welcome. We’re proud and happy to be hosting this event in the Menlo Park Council Chambers this afternoon addressing this vitally important topic.

Here in Menlo Park, we are known for our close elections. And in the past decade, a couple of decades, we’ve had some very, very close ones, one of which was decided by only two votes, so it is very important that every vote be counted. And as a citizenry, we must jealously guard this important right. With that, I’d like to turn things over to Senator Bowen with my warm welcome.

SENATOR DEBRA BOWEN: Thank you very much, and I want to acknowledge Nate Pinkston who’s here from Ira Ruskin’s office, right back here. Thank you very much for being here with us. And I want to thank everyone for joining us today.

This is the third informational hearing in this committee’s ongoing effort to look at the mechanical workings of our electoral process and how they can be improved and specifically how public confidence in the results of elections can be improved. Last week, we looked at the concept of using open-source software in our election systems. And in mid-January, we looked at where California’s counties are in terms of buying election equipment certified for use in an election that will be held on June 6 of this year.

Today we will look at the certification process itself—how it works, how it doesn’t work, how it might be improved. To everyone who is looking at the agenda wondering where Diebold and the other voting machine vendors—Independent Testing Authorities and the Secretary of State are today—all I can tell you is that they were all asked to attend and all refused. Clearly, each of them plays a significant role in this process, and their testimony is critical to helping the Legislature and the public determine whether they should or should not have confidence in the equipment that is used to cast and count ballots. Since asking nicely for participation hasn’t worked, it will be time to turn up the heat a little bit to get these parties to Sacramento for a hearing in March. I hope we can do this without the use of the subpoena power.

The three things that I want to focus this hearing on and want to keep us coming back to are as follows:

One, the question of transparency in the testing process and the fact, that once the machine is certified, it is not tested again.

Two, the relationship between the vendors and the Independent Testing Authorities, we will take testimony on the question of the relationship, question of conflict of interest, and the question of whether or not there is an incentive for the testing authorities to find bugs, holes, or problems, given who is paying the bill for the testing, and again, what we might do to change or improve on that situation.

Third, the very adequacy of the standards that the ITAs test against and whether it is really meaningful in terms of the actual conduct of an election. We could have the greatest, most transparent, and most independent test in the world. But if the standards are either too low or don’t test what needs to be tested, then what does it mean to pass a test?

We have a distinguished panel of experts on hand to help us answer these questions and address these issues. I’d like them to come forward at this point. As you will notice on the agenda, we also have a comment section at the end of the hearing. If you would like to testify, I would appreciate your signing in with the Senate Sergeant-at-Arms in the back of the room, not because we want to know your name, your social security number, or any other personal information about you, including whether you’re registered to vote or how you voted, but because we like to be able to manage our time so that everyone who wants to speak has an opportunity to do so.

So with that, let me ask Professor Avil Rubin, Professor Dan Wallach, Professor David Dill, and Professor Peter Neumann to the hearing today. And again, thank the City of Menlo Park. I was not aware of your history of elections. But having heard it, I understand why this is such an appropriate place for us to have a discussion about how every citizen can be registered, can vote, and vote only once, and then can have their vote counted as it was cast. So thank you for hosting us.

VICE-MAYOR FERGUSSON: Thank you.

SENATOR BOWEN: Let me ask Professor Rubin to start us off.

PROFESSOR AVIL RUBIN: Thank you, Senator Bowen, and Members of the Committee. My name is Avi Rubin, and I’m a computer science professor at Johns Hopkins University. I’m also the technical director of the Information Security Institute at Johns Hopkins, and I’m the director of the National Science Foundation ACCURATE Center. My research focuses on applied cryptography and system security, as well as network security. And since 1997, a great deal of my research has been on electronic voting.

This hearing is about testing. And so while there are many things that I would love to talk about and could talk to no end about, about electronic voting, I’m going to focus on testing today. And I think it’s important to understand, when we limit the discussion to testing, what the limitations of testing are. You cannot test for security the same way that you test for functionality. So while testing can be effective at determining whether a particular machine performs certain actions when it’s running under expected conditions, testing for security, which would be unexpected conditions or a malicious adversary, cannot be done in the same way.

And I thought of an analogy to give you to illustrate how security testing is very different from functionality testing. Imagine if you had a large vault that was protected with a large combination lock and you wanted to test how secure it was. There are all kinds of things that you can do. You could take a big power drill and see how hard it was to drill a hole into it. You can drop it and see if it breaks. You can look for worn out parts on the dials to see if after a while you could figure out what the combination was. But let’s say that this safe was set with a combination of 1-2-3-4. If you didn’t have a test in your testing plan that said, see if the combination is 1-2-3-4, you might be able to perform all kinds of tests and think that you can conclude that it’s secure. But if the combination is 1-2-3-4, nobody would agree that this safe is secure. It’s not a perfect analogy, but it kind of illustrates that, you know, an attacker might walk in, and the first thing they might try would be 1-2-3-4 and they got in, whereas a test could not possibly be designed for that circumstance.

The first step in certifying and testing a voting machine in California is for the machine to be federally qualified. The so-called Independent Testing Authorities test against federal guidelines. These can be the 2002 standards or the recently issued VVSGs by the EAC. And the vendor, after the federal qualification, is issued an asset number. When that process is over—and I’m not going to talk in my opening remarks about that process, but I’d very much like to during question and answer—then it goes into the state testing.

California recently instituted, and I believe it’s unique to this state, what’s called volume testing where they actually require the vendors to come to California and do the testing in the tester’s site as opposed to at the vendor’s site, and that eliminates many different problems that can arise when things are happening at the vendor’s site. California also performs parallel testing. This is not part of the certification process. This is something that happens on election day where voting machines are removed at random from polling stations and are then subjected to votes as though they were in a real election, and the idea is to see if the vote totals at the end match the votes that were inputted into them to test for wholesale fraud. I believe parallel testing is very, very important, and I’ve been hearing talk about the possibility that parallel testing may be stopped in California, and I think that would be a huge step backwards.

There are specific functionality tests that take place. Some are defined for DREs, and some are defined for optical scans. These are hands-on tests, testing all the features of the machines to make sure they’re correct. California also has a Technology Assessment Advisory Board which is chaired by David Jefferson. It’s an independent panel of computer scientists from primarily public universities. I think this is a tremendous thing to have, and I don’t think other states, any other states, have something like this, and I would encourage making really good use of this resource.

While I’ve described some of the good things that are going on in California, I believe there are areas of improvement. For security, it’s absolutely necessary to have penetration testing or so-called Red Team attacks where security experts can have a chance to try to find security flaws with the machines. That’s very different from doing functionality testing. You’re going to take people who know how to break into computers, who know how to break systems, who know how to take advantage of vulnerabilities and code, and you give them access to everything—to the source code, to the machines themselves, to any components of the machines, and to all the policies and procedures, and you let them go at it and try to come up with a security evaluation of these machines.

It’s important to incent ?? these Red Teams so that they’re rewarded for finding problems with them as opposed to be rewarded for not finding problems with them. If you give this to the security experts and you incent them to find problems, if there are problems, they’re more likely to find them, obviously.

The testing reports that result from the different tests that take place for security and for functionality need to be posted publicly, and it’s my understanding that that is the case in California, except that these postings have redactions in them, that sections of the reports are blacked out, and the public can’t see them. And I think it’s very important that testing results, all of the testing results, be published, and the debate can ensue after that. But the public should have the right to know what the testers found when they tested the machine.

There’s currently no testing of the audit process. So for example, in the case where there are DREs with voter-verified paper trails, the long ribbon, which I’m not a big fan of, they don’t test what it would be like to do a recount. If you’re going to test the voting system and you’re going to do 1 percent manual recounts, you need to test the manual recounts. There are no tests right now required of the procedures that are part of the certification, not just the voting machine, but they should test those procedures. And there’s no institutionalized code review. There’s not a requirement that software experts be able to analyze the code that’s running inside of the machines.

The California testing is fairly good at assessing reliability but not security or accessibility. Accessibility modules in California are tested by users who suffer from a disability that those accessibility systems address by having informal open houses where they come and try out the machines. Those accessibility features are not tested in the same rigorous manner that others are. So for example, the audio module in a DRE are not subjected to the same kind of testing that the counting of the votes is. The California volume testing has been successful in uncovering a very serious computer bug that could not have been found any other way. So I believe that the volume testing is very important.

Let me wrap up with some recommendations and some of the things that I skipped for the sake of time, I hope will go through, when you ask questions. I recommend making all the testing reports publicly available in their entirety, performing penetration and Red Team tests on all voting equipment, that testing by these qualified, independent security experts be done, such as the RABA team that analyzed the systems in Maryland, and that all of their test results be made public, testing the accessibility features with the same rigor as the others, and continuing the volume testing and the parallel testing.

Finally, I should say that I think California as a state probably has the best testing program in the country, but there’s still a lot more to do.

SENATOR BOWEN: All right. Thank you. It’s hard to know whether to start here or hear from all the panelists. But I think what I’ll do is go through the panelists because, that way, if you all have disagreements amongst yourselves, I will have a better idea what to ask. And my guess is that you won’t all be in total agreement on everything—that’s as it should be—or not.

Let’s go next to Dan Wallach, Professor of Computer Science at Rice University, and thank you very much for joining us from Texas, and please accept our wishes that our good California weather take whatever bug it is that you brought with you that it’s not a computer bug and dispatch immediately. (Laughter)

PROFESSOR DAN WALLACH: Thank you, Senator Bowen, Members of the Committee. It’s a pleasure to be here today.

So I am an associate professor at Rice University in Houston. I was actually a Cal undergrad, Class of ’93, so Go Bears. (Laughter) So I work on computer security which generally you can look at as breaking things, as building things. To me, it’s an engineering problem. How do you engineer a system to be robust? And in order to do that, you have to say, Well, what is the threat model? And voting is possibly the most engineering problem I’ve ever looked at because, from engineering, the threat model is the most convoluted. Every single person who touches the machine—every developer, every user—is potentially a threat. And that means that the engineering process is fascinating, and we can study this for years. But meanwhile, we have to have something that works because we vote every year.

So I’ve been working on voting since about 2001, when they first introduced these machines in Houston. And in 2003, Avi, myself, and two students wrote a report where we analyzed the Diebold voting system which we can talk about in more detail, if you’d like. And more recently, I’m the associate director of ACCURATE, the same center Avi mentioned earlier.

So let’s see. When you want to talk about testing, testing is always done with respect to some standard. So Avi spoke about testing. I want to speak about the standards that you test to. So recently, the EAC and NIST promulgated the 2005 voluntary voting system guidelines. We have a copy right here on the table…

SENATOR BOWEN: Let me stop you for just one moment. I am told, that with the microphones in this system, you have to get the mike very close to you and that there are people in the back who can’t hear. So let me do a little test. I’m going to talk; and if you can’t hear and you’re in the back, please raise your hand. If you can’t hear from the back, let me know.

Okay. Now we’re going to do a test of the microphones at the panel, and I will ask you to just repeat again that little part about this being an engineering challenge because I think it’s worth hearing again. And then I will ask people in the back, who you can’t see, to report. And you won’t know if I am accurately recording the results on this test (laughter) because it’s not transparent.

PROFESSOR WALLACH: Okay. So the engineering problem of microphones is not unlike the engineering problem of voting machines. It’s a different threat model.

SENATOR BOWEN: Okay. I’ve got an okay in the back, so I’m assuming that this has been a successful test and that now everyone who is here will have all of the testimony.

PROFESSOR WALLACH: Thank you.

So the 2005 standards are definitely an improvement over the 2002 standards. However, they have very little to say about critical issues that can affect vulnerabilities and security in voting machines. In particular, there’s no significant attention paid to the software engineering process used to develop these systems. When you want to build a system that you intend to be reliable, that you intend to be robust, that you intend to be secure, if you want it to actually be all those things, that has to be part of your design plan from the very beginning that affects how you write your software; it affects all the processes that you use; it affects how you hire people; it affects the tools that you use and generally makes the process much more expensive and much slower. But in return, you get a higher-quality result.

And when you look at the way critical systems, like, say, airline control software, you really don’t want your plane falling out of the sky. That would be bad, oops, sorry about that. And as a direct result, companies like Boeing invest huge amounts of money in their software development and qualification. None of this is done presently for voting systems. The ITA process, the VVSG standards have effectively nothing to say about the process behind the software. And if you get the process wrong, you’re guaranteed that the result will be broken. And even in California, we’ve seen plenty of evidence of this, probably the classic example being, finding uncertified versions of Diebold software running on Diebold systems in the state. That’s a process problem. That says they couldn’t even hang with the very simple process, such as it is—develop it, certify, and ship it. They somehow missed one of those steps.

SENATOR BOWEN: But one of the questions that I’m going to ask all the panelists is, If there is a way to test, given the number of polling places and the number of voting stations in California, if, realistically, there is a way to test whether or not every electronic piece of voting equipment is actually running the code that has been certified. So I’m going to ask all of you to address that.

PROFESSOR WALLACH: So one thing that you will often hear described as a possible solution—and this is what they call hash-code testing—where you ask the computer what it’s running, and it gives you a magic number. And if the magic number is what you expect, then you say, great. But that’s kind of like asking somebody who walks into a bank, Are you a bank robber? Why, no, I’m not. (Laughter) So, well, okay, then. Go right ahead.

So actually, the process of verifying that the software in the machine is the software that you wanted to have in the machine is a very interesting, technical problem. To me, it’s an open-research problem. Probably the only area where we have any traction on any similar problem in the computer industry is, of all places, in game systems. Sony and Microsoft are very, very concerned that you don’t run pirate games in your Xbox or your PlayStation, that you only run software that has the Microsoft stamp of approval and Microsoft gets the appropriate royalty payment. So I think we might actually be able to leverage the sort of technology that’s in, you know, a cheap Xbox. That same sort of technology may very well have a place in voting systems. But to date, no existing voting system does anything like that.

SENATOR BOWEN: Can you just describe a little more for us, how that works?

PROFESSOR WALLACH: Okay. So the way that game systems verify that they’re running official software is, that when the hardware first powers up, it begins to download the game from the CD. And that process involves checking that certain features of the disk are as they’re supposed to be. So with the original PlayStation, it checked that there were some blocks that had incorrect check zones. It turns out that a normal CD burner will refuse to write incorrect check zones. So if you burned a disk, it would be correct and that’s not correct. So Sony deliberately put errors on the CD and they checked those, sort of a funny trick.

In more recent systems like the Xbox, they used cryptographic techniques and made sure that they—well, what is the operating system is digitally signed such that—and then the game console actually has the appropriate cryptographic key materials to verify that Microsoft in fact blessed this particular game.

This hasn’t stopped enterprising people from rewiring their game systems to be able to play pirate games. But if someone to rewire a voting machine to run pirate voting software, that would be a physical modification that could be detected on physical inspection, if a voting machine were built the same way game machines were built.

SENATOR BOWEN: So there’s no way you can modify a game, an Xbox or a—these are all devices I’m not personally familiar with. (Laughter) You cannot modify the software or the firmware in such a way that it will run a CD that’s not authentic without modifying the hardware in a way that’s physically visible?

PROFESSOR WALLACH: So curiously enough, there was a particular game for the Xbox. It was a 007 James Bond-themed game. And this particular game had a vulnerability in it. And people were able to attack the game in order to highjack the system and then install Linux on their Xbox. It’s kind of a bizarre thing. So if you Google for the 007 exploit, you’ll see all these details. And that actually leads to an important point. Even if you use techniques, such as cryptographic signatures to authenticate that software is official, the software still has to be built to appropriate standards.

SENATOR BOWEN: Right. And I think that that’s a point that we want to pursue, as many people have asked me—and again, I will ask the panelists--many people have said to me, look, I use an ATM machine all the time and it is manufactured by Diebold, and it seems to be a fairly reliable piece of computer engineering. Why is it that the voting systems don’t function in the same, secure manner?

PROFESSOR WALLACH: So my stock answer to that is that there is nothing anonymous about an ATM. The ATM takes a picture of you when you use it and you put in your pin, and there’s a record and it gives you a receipt. Anonymity isn’t part of the problem. If anything, the last thing they want is anonymity. If I go to my bank and say, I didn’t make this withdrawal, they say, Well, what’s this picture of you standing in front of our ATM?

With voting, we seem to feel that anonymity if valuable. If you go back 150 years or so, people voted by standing up and say, I vote for Bob. And if we want to go to a world where votes are not anonymous, then that simplifies the engineering problem. But because we want to avoid voter coercion and bribery, our country and most other countries have moved to anonymous voting, and the anonymity is part of what makes the engineering problem more challenging.

SENATOR BOWEN: So the fundamental challenge from a software engineering standpoint—and if I can put this into terms that you can understand, even if you don’t own an Xbox—is that once you combine the desire for privacy with the desire for absolute security, it becomes much more difficult to build?

PROFESSOR WALLACH: Absolutely.

SENATOR BOWEN: I think that’s a really critical point for people because they do know that their airplanes fly on software and that that software works pretty well, and they do know that their ATMs work, and so they wonder, Why can’t this work for voting?

Another question, I think along that line, you alluded to earlier in your testimony, and that has to do with just plain old devotion of resources. What kind of resources are expended in developing and testing? Let’s use three examples. Again, I’ll ask any of the panelists to weigh in on this. One would be ATM machines. The second would be, since you raised it, airplane software. And third, how about nickel slot machines? What kind of resources do we expend assuring that the results in nickel slot machines, which have nickels at stake, not governance, are accurate?

PROFESSOR WALLACH: So David Dill probably knows more about airplanes than me. So I’ll focus on slot machines and ATMs. (Laughter)

So I’ve recently become enamored with economic game theory and incentives. And you can explain a lot by looking at the incentives behind things. In slot machines, regardless of their denomination, and with ATMs, all the parties have an incentive to look over their shoulder. Banks want their ATMs to be reliable and robust because otherwise people will steal money. Or, people will complain or maybe leave the bank because, Well, your ATM, I asked for $200 and it gave me $100. You know, forget you. I’m going to another bank.

So both the banks and the customers have an incentive. They both want accuracy. Of course, the customers would be happy to get free money, and the banks maybe wouldn’t mind if you got less. But that sort of averages out. And everybody’s watching the system, and everybody makes sure it works.

Slot machines are very similar in the sense that the casinos that run them tend to be for-profit ventures. And if the machine pays out more often than the odds that are printed in front of the slot machine, then the casino loses money. So if it’s one thing that casinos know how to do is count their money. And they can figure out exactly how much money every machine has paid out. And even if they can’t detect that one particular incident was erroneous, over time, over days or weeks, they can clearly determine that this machine or that machine has been paying out more or less. And then they’ve got those video cameras all over the ceiling, and they can figure out, Was there somebody who went and got an extraneous payout? Now they’ve got pictures; and now they’ve got all the evidence they need. And in fact, in 1998, in Nevada, some inspectors, whose job it was to check the slot machines, were actually tampering with the slot machines, such that, when you put in a particular series of bets, then you get a big payout.

So first, the inspector goes and dorks with the slot machine and then his compatriots come in later and do the funny bets and make the big payout. Why were they caught? Because they were trying to extract too much money too fast. So that’s a fine example of something where it was only caught because, well, you know, the parties have an incentive. The casino, the house wants to make sure that it’s not paying out too much.

In voting, it’s unclear where the analogous incentive is, and that’s part of the problem. In voting systems, the concern—and not that I would point the finger at any particular insider or developer or anybody—but you have to be concerned that any of them might be malicious, and you need a system engineer to work, despite the fact that any of them could be malicious.

SENATOR BOWEN: So in other words, you’re basically asking the engineering and the process to overcome the privacy limitations or the anonymity? You have to have a system that’s so robust, so many checks, that it doesn’t matter that it’s not…

PROFESSOR WALLACH: I would like the president of a voting machine company to be able to walk in and tell you with a straight face, even if I’m partisan, even if I want to throw the election, I can’t because my system is built in such a way.

SENATOR BOWEN: And can you test for that?

PROFESSOR WALLACH: Can I test?

SENATOR BOWEN: Yes.

PROFESSOR WALLACH: You can engineer for it; but all the way from the very beginning, you can’t slap that on as an afterthought.

SENATOR BOWEN: So you’re saying that has to be built into the engineering?

PROFESSOR WALLACH: From day one.

SENATOR BOWEN: How would that be different than what things look like right now, the way systems have been developed?

PROFESSOR WALLACH: So there are a lot of different proposals for how voting machines ought to be built. And the place to start as a baseline for a very well-designed, simple, and cost-effective voting system is mark-sense paper ballots. That’s where you fill in the bubble or connect the dots between two arrows and where you have the counter in the precinct, so there’s a scanner bolted to the top of the ballot box, this means that you have—the scanner can reject something if you vote for two candidates and you’re only allowed to vote for one, it can just say, Error. At least in Texas, you get three shots. I don’t know what the rule is here.

With a system like that, if the software in the tallying machine is messed up, then you have Plan B. You can go back to the original paper ballots.

SENATOR BOWEN: And how would you know if the software in the tallying machine is messed up or not functioning properly?

PROFESSOR WALLACH: So this is where you can either do statistical techniques, such as, you know, the 1 percent random audits, and you might furthermore, just randomly choose precincts and count everything again. Furthermore, you could—usually, the press wants to know who won the night of the election. But certification of the election happens several days later. In the interim, there’s no reason why you couldn’t have a separate mechanical system separately recount the ballots. And if the scanner in the precinct was accurate, you should get the same answer. And if it was different, then that’s interesting, and then you might want to investigate in more detail.

PROFESSOR DAVID DILL: So I wanted to comment on this and also on your question about what measures are taken with other things. You know, I have a lot to say about the certification process. And when I came in here, I was thinking, What’s the most important thing I can say in a few minutes? And I think the most important thing I have to say is actually going to be somewhat in conflict with what the other members of the panel are saying; although, maybe after more discussion, we’ll agree.

There’s a natural tendency for computer security people to think about tightening up the security of the machines.

SENATOR BOWEN: Let me stop you just for one moment to ask you to introduce yourself…

PROFESSOR DILL: I’m sorry.

SENATOR BOWEN: …audio only.

PROFESSOR DILL: Yes. I was just diving into an answer of a question.

SENATOR BOWEN: Yes, please.

PROFESSOR DILL: I’m David Dill. I’m a professor of computer science at Stanford University. I became involved in the e-voting controversy in 2003 by writing a petition that everybody on this panel has signed called The Resolution on Electronic Voting. Then I was asked by the Secretary of State to participate in a panel at the ad hoc’s Taskforce on Touch-Screen Voting where we did some of the first studying of touch-screen voting at the state level. I’m also the founder of VerifiedVoting.org and the Verified Voting Foundation which are organizations whose mission is concerned with election transparency and paper trail issues in particular.

So I’ve been worrying about this certification issue and looking at some of the things that have happened recently in states such as Pennsylvania and Florida where the certification process has worked against the adoption of the kinds of equipment that we prefer because computer security concerns in some sense have trumped what I think should be the real concerns which are the auditability of the machines. Ultimately, we can pull out all the stops and try to make these machines as secure as we possibly can. That would be an extremely costly and time-consuming process.

At the end of that process, we would still have machines that we couldn’t trust because we can all think of ways that they could have been corrupted by their manufacturers if by no one else. And so in that situation, you have to stop trying to make the technology better and start thinking hard about how can we make the technology so it can be double checked? So that’s really a focus on auditability of equipment. And not just the auditability of the equipment but the auditing procedures that are routinely invoked. So I think California—I think there’s lots of room for improvement in California. But I think that we have been a leader in having the 1 percent random audits and, of course, that process has been strengthened recently with SB 370. So I think really the priorities of testing, with the current system of certification, we’re really relying on the federal level process to do the job for us or most of the job. Although, as Avi mentioned, there have been recent innovations at the state level. But the current process at the federal level is ineffective, especially for security, but even straightforward bugs that have nothing to do with security and straightforward violations of the existing weak standards have slipped through the process and have been caught at the state level and other places.

But the flip side of it is it not only—you know, well, its lousy but at least it’s expensive and takes a long time. (Laughter) But the process is very costly and introduces a lot of delays, and I see this as harming voters because it has been a barrier to the deployment of improved equipment, equipment that is more auditable, more accessible, and more useable than the equipment we have on the market now. It’s created an oligopoly of a small number of major vendors who dominate the process. And I think that we need to resist the urge to just say we should make certification more stringent. In some ways, we do need to make it better and more stringent. In other ways, we also need to streamline the process and make it less costly and difficult for manufacturers to get through. Now this is a case of wanting to have my cake and eat it too, and I realize that what I’m setting up here is a very, very hard problem, but I think that we need to appreciate the difficulty of the problem and think it through carefully rather than saying, Okay, we just need to go take the processes that they use for safety, critical systems, such as airplanes, and use them in voting machines.

In fact, when I was on the IEEE Voting Standards Committee called the P-1583 Committee, one of the guys on the committee was an expert in software safety and hardware safety and in fact had worked on some of the networking apparatus inside the Boeing 777 which, you know, the certification of the hardware and software in that airplane, which is a completely different process, costs hundreds of millions of dollars. And he proposed that the same standards, which had already been written, be used for voting machines. And the reaction of other people in the committee was, Well, if we did that, the State if Texas would only be able to afford one voting machine. And, unfortunately, that’s probably true. I don’t think that that is a route to trustworthy voting. Well, we need to rely on for trustworthy voting is making sure that every voter can verify that their vote has been properly recorded and making sure that that record is properly used by both random auditing and by making it easy for candidates and inexpensive for candidates to be able to get recounts, manual recounts, on a routine basis.

SENATOR BOWEN: I guess the obvious question is, if it is that expensive to use an engineering and security process that are trustworthy or to certify a machine as trustworthy, should we be using machines?

PROFESSOR DILL: Well, that’s a very good question. I think it’s—you know, I’m a computer scientist, so it would be difficult for me to say, oh, just don’t use computers. (Laughter) But I’m willing to go that far if it’s the right thing to do. And I don’t think it’s really necessary. I think what we need are computers where you can double check everything that they do. Just treat those computers as people that you don’t know and give them an equal level of trust, which is basically none. You have to have in place checks and balances. Even if you have a completely manual process where people fill out the ballots by hand and count them by hand, you have trust issues because the people counting the ballots are just as untrustworthy as the computer. And so in that case, you need to rely on checks and balances in order to make sure that the ballots are properly counted. You would want to have them counted by several people of different parties looking at the ballots at the same time and put in place a lot of those other procedures. Essentially the same thing can be used to make computerized vote counting work, but the entire process relies on having a trustworthy record that has been verified by the voters, whether you do the process manually or whether your computers are involved in the process. There has to be some manual counting, but I don’t think—I’m not going to advocate that it needs to be all manual.

SENATOR BOWEN: I think the question that arises very often is not the question so much of—I think your point’s well taken about the fact that everyone is potentially untrustworthy. But the manual systems of counting rely on the fact that many people would have to collude in order, statistically, to change the outcome, and they also rely on the fact that one or many people can observe the counting process and the recounting process and the fundamental difficulty with having that level of trust established when a count or a recount is being done in a way that is not even potentially transparent. Response?

PROFESSOR DILL: I don’t know whether you’re saying this is only true in the case of 100 percent manual counting. Those are essential properties that you need to have with a trustworthy voting system, and I think that can be achieved with paper ballots, whether they’re counted by machine or by hand, so long as you have enough hand counts that you’re double checking the machines that are involved. But the principles you’ve stated are exactly right.

SENATOR BOWEN: You’re talking about at the central level as well because one of the concerns that we had is not just what’s happening at the polling place with an individual machine but what happens at the central tabulation point where votes from the various polling places are collected and assembled. And there, I think what you’re talking about, the checks and balances, just simple steps, such as posting the number of people who have voted at a particular polling place when the polling place is closed so that, if there is a polling place where it is reported that 320 people cast their ballots on that day at 8:01 p.m. and three days later it appears that 820 people cast their ballots that day, you know that something is wrong without even knowing what the count is.

PROFESSOR DILL: Yes. In many states this happens and nobody checks. So an election is a complicated thing. So from the point where the ballots go out to the voters, no matter what kind of ballot they are, till the final recount and whatever, every part of that process has to have checks and balances and has to be auditable. So we’ve been focused very much on electronic voting, but the same principles apply everywhere in the process.

SENATOR BOWEN: Okay. Peter Neumann. And I have more questions than I can even begin to know where to start. But, Peter, let’s turn to you. I’ll ask you all to reintroduce yourself again.

MR. PETER NEUMANN: Peter Neumann. I’m the principal scientist at Computer Science Laboratory at SRI. I’ve been in computers for over 50 years. As I mentioned to you last week, in security for over 40, and in the voting analysis and discussion of evaluations and certification and so on for close to 20 years now. I would refer the audience to the testimony, the written testimony, that I gave for you last Thursday, which is not on your website, and I went into considerable detail about why openness in the process is essential.

I’d like to begin by saying that there is no discrepancy between what Avi and Dan said and what Dave said but, rather, there is a sum of the two that is important. The voting process is an end-to-end integrity problem where essentially everything along the way is a weak link. We have nothing but weak links, whether it’s the registration process or the voter authentication process or the ballot preparation or the entering of the vote onto the screen or a punch card or whatever or a box-sense card, the counting of the ballots, the potential for manipulation and misuse and accidents exist in every single step. So auditing and oversight and openness are absolutely essential throughout the entire process, and any self-respecting computer security person is not going to say, that if we had a perfect voting machine, it would solve all the problems. There is no such thing as perfect security, especially when you consider the problems of insiders who are trusted to be able to do all sorts of nasty things or to make accidental mistakes that could alter the results.

So I think the important thing here for this particular testimony is that the federal voluntary standards are very weak. They are a little better than the 2002 standards which were a little bit better than the 1990 standards, but they are still enormously deficient. I have the stack of paper here which represents the current voluntary voting system guidelines, hundreds of pages. Each item is a sentence or two. And the level of detail is minimal. The amount of vulnerabilities that are not included is enormous.

I’d like to pick up on your previous question on the flight recorder, the ATMs, and the gambling. The $1.7 million Harris scam from many, many years ago was a progressive machine payoff that was triggered by some insiders. There’s one example. There are various other cases. But the gambling industry very quickly realized that they needed a tremendous amount of oversight; otherwise, they would be losing a great deal of money if there was in fact scams.

The ATM situation is a very interesting one. You’ve already heard how there are detailed audit trails and cameras and everything. Last night, I had dinner in San Jose in conjunction with the RSA meeting on security, and I’m on the advisory board for someone whose significant other happens to work for a bank. And he told me about really a monster security hole in the Diebold ATMs which has been recently discovered but not publicly known, and I’m not going to disclose it here. (Laughter) But here’s an example of where you have software that does have audit trails and is put on top of an operation system that is not secure. In fact, we’ve had cases where a Microsoft Windows 2000 prompt shows up when you go to the ATM—sorry—boy, there’s a Freudian slip (laughter)—to get some money out of the machine. But the point there is that even in the ATM world where you presume that there’s a great deal of oversight and audit trails, there are some security problems.

SENATOR BOWEN: I really want to stop and highlight that point because, while I may not own an Xbox or a Game Cube or whatever they are, I do spend a lot of time and do a lot of commerce and business online. And sometime ago, I started printing out pages of well-known websites. What I got instead, of example, the check-in page for an airline or an online auction site, instead of getting the item, I got a page of computer code. It started printing that, and I have a little collection of pages of computer code where I should have seen a chart to asking you which seat on an airplane I wanted or whether or not I wanted to bid another dollar for—I’m not going to tell you which option site it is or what I was buying—but I’m sure you can find out, really. (Laughter)

MR. NEUMANN: One of my favorite stories on that line was way back in 1964, I think it was, when in the MIT time-sharing system the entire unencrypted password file came out as a cookie, the message of the day. (Laughter) And it turns out that there was a shortsighted design flaw in the editor that was used. And the person who designed this system never assumed that two people would be editing two different files in the same directory at the same time. And it turns out that the temporary files got interchanged and out comes the password file as the message of the day, and the message of the day became the password file. (Laughter) Things like this happen all the time. And if you look on my website, you’ll see a list of literally thousands of cases of things where something was supposed to go right and in fact it went horribly wrong.

The third case, though, is avionics where in my lab in 1973 we built the world’s first fly-by-wire system prototype for NASA, and this was a system that was over-engineered enormously. It had a probability of failure of five orders of magnitude, better than the hardware that was used to develop it. And the point there echoes what Dan said, that you have to engineer it in. You have to build the system to be robust in the first place. Now if you do that, the cost is not that great. The problem is that most of the systems that we are forced to trust, even if they’re not trustworthy, were not designed with security in mind. And the problem then is you can’t retrofit security into something that was never designed to be secure in the first place. And the answer to the question, Does it cost more or is it massively prohibitively expensive is quite different from the question in the aviation situation. In the aviation case, the cost of the 777 mainframe, the airframe, is enormous. And the cost of the very redundant computer system is negligible by comparison.

In the case of the money machines, nobody really wants to sink a lot of money into the development because the marketplace is relatively limited, and there’s no real incentive to do it right. Now there may be a lot of reasons for that, but I’m not going to go into why one might want to design systems that could be easily rigged, for example. This is something I’m not going to get into. But it appears, that not only are the standards very weak, not only is the software engineering that goes into the system very bad, not only are the evaluation processes paid for by the vendor and proprietary, but subsequent to the evaluation, most of the vendors wind up changing the system in a way that is not audited and in a way that is not accountable in any sense.

The experience I had over a decade ago in New York City was to look at the source code under nondisclosure of a system that the city wanted to acquire and in fact had spent $17 million on. And the conclusion was, that even if the source code were perfect, there were a couple of dozen ways in which an election could be wrong, or it could maliciously defrauded, using that very system. And I think the lesson there is that looking at the disclosability of the source code is a piece of the puzzle, very important, but it’s by no means enough. In a situation in which the evaluation process is not adequate where the standards themselves are not adequate and where the development technologies that are being used by the developers are not only under wraps, but to the best of our knowledge, in how they will certainly back this up on the Diebold probe that he looked at, are appalling.

SENATOR BOWEN: Technical term, right?

MR. NEUMANN: Technical term, right. (Laughter) So my conclusion is very simple, that it is absolutely essential, as we said last week, to have a great deal of openness, but it has to be openness throughout the entire process, in that every step along the way is a potential weak link. So why don’t we subject ourselves to your questions. From last week, I want to applaud you and thank you so much for doing this. The questions you asked last week were very much indicative of the fact that you really understand what’s going on here.

SENATOR BOWEN: I’m not sure I want that responsibility. (Laughter)

PROFESSOR RUBIN: I want to interject something on the aviation analogy—

SENATOR BOWEN: Yes.

PROFESSOR RUBIN: --because I hear it a lot. David mentioned the hundreds of millions of dollars that would be required to develop software using the processes that are used for avionics, and it’s actually much harder because you’re not worried about one of the developers of the airplanes trying to make it crash, and there’s a big difference.

MR. NEUMANN: This is true. Good point. (Laughter)

SENATOR BOWEN: It’s very interesting. Let me actually go to a question that keeps coming up, which is, people are saying you have to engineer it in. How do you do that? What does that actually look like if you’re going to create a system where you have engineered in…

MR. NEUMANN: In my humble experience, I have several efforts. I live in a very high-end research world where, for most of my professional career, I’ve been involved in systems that were very trustworthy, that were survivable, that were very reliable, that were highly secure; we were human safe. And you might say they were over-engineered, and I would say, well, really, they were architected in such a way that the system had a possibility of being evolvable over a long period of time, so that as new technology came along, you could stick it in there compatibly in some way and that you were building something that had a long-term vision of the future rather than saying, hey, we’ve got this little widget that’s sitting on a desktop. It doesn’t have any networking. It’s a standalone desktop, personal computer, and we’re going to suddenly throw it onto the internet with no security in it. And maybe we’ll add a little security to make it okay. That is not the way you go about things, and that is pretty much the way the election seems to have been evolved.

So I think the answer is, that if you look at the research over the past 40 years on developing certain secure systems, there’s a very large number of papers and for other types and experimental developments that demonstrate how one could build things that are much more robust, much more predictably trustworthy. This is not a black art, but it’s made of black art by vendors and developers who don’t understand architecture and software engineering, testing, certification, building things to be auditable in the first place.

PROFESSOR WALLACH: So following up on Peter’s point, David Dill earlier discussed that it’s not clear that the right answer needs to look like a computer. It might look more like paper. And part of the engineering problem is also controlling cost. And if the cost of an engineering process is just out of control, then you need to engineer the process and say, well, if we can’t do, if we can’t build the perfect software artifact, what can we do to compensate for imperfect software artifact? And that’s where we get into the checks and balances. This is some form of a paper audit trail—and there are many, many different ways we can go into the details of, Should it be a continuous role; should it be individual cards? Those details we could get into. But the reason why so many computer scientists have stood up for the importance of paper is not that we like dead trees. If you see my office, you understand that I’m fighting with them all the time. (Laughter) It’s that paper is something that’s outside of the computer’s control. Once it’s been printed, it can’t be unprinted. A software bug or a software tampering can’t change the ink on the paper after it’s been printed, and that means that you now have something that’s redundant. You have a digital path and you have a paper path, and you can’t throw a lot of engineering at the problem and you can make the sum greater than either of the parts.

Paper by itself has a long history of election fraud. And computers, well, they don’t have a long history in elections, but we’ll see. When you can combine the two, the paper is a check against the computer, and the computer is a check against the paper.

PROFESSOR RUBIN: I want to kind of, sort of get back to my comment on the airplanes which is, I don’t know that we’ve ever faced a challenge of how do we engineer a security system that proves that the people who engineered it, the very same people, aren’t cheating? So it not only has to be secure; it has to carry with it a proof that it’s not doing anything that it’s not supposed to be doing, and I think that’s a much greater challenge. The two analogous challenges that I think we have are the anonymity and the privacy, and then the fact that you can’t trust the builder of the system or any other component. That doesn’t mean they’re not trustworthy, but we should build systems—and I think we can build systems—where it’s okay if they’re malicious because we’re not relying on them to be honest.

SENATOR BOWEN: Let me follow up on that with some other points you made in your first comment.

One of things that you suggested is a Red Team approach in which we deliberately set up systems for penetration. I have heard the criticism level that that is akin to testing a bank, bank-safety mechanism, by folding up 20 pieces of paper around the room and writing the combination to the save/safe ?? on file. In other words, it’s not something that ever would happen in the real world and that many of the security issues that computer scientists claim or should be a concern are not real-world concerns. So, please, gentlemen, defend your honor.

PROFESSOR RUBIN: Let me give you a quick counter to that.

It sounds to me like this would be Diebold saying, Our system is totally secure, and it relies on the fact that no one will see our source code. (Laughter) And then their source code happens to leak onto the internet. Now what about, instead of us publishing a paper and that got read by a lot of people—Bev Harris who founded, and a couple of other people, kept the knowledge to themselves, distributed that source code to a few of the wrong people—the assumption of security by obscurity, which is that the security mechanisms themselves will remain secret, is well known and has been for centuries. One of the mantras of security has been, We reveal how the system is in order to evaluate it. And if can still show that the system has security properties, then we can have confidence in them. But if our confidence is based on keeping things secret that may or may not actually remain secret, then I think we have a problem.

SENATOR BOWEN: Anyone else?

PROFESSOR RUBIN: And furthermore, penetration testing is done in banks. Military people have done this forever. You give somebody a get-of-jail-free card, and you say, Have fun. And the question is, Can they get in and do something they’re not supposed to do? And if the system is working and, you know, the security guards show up, they say, Okay. You’ve got me. I’ve got to get an get-of-jail-free card. I was doing my job.

MR. NEUMANN: I’ve seen Red Teams where one group came in and found a whole mess of problems. And a second group then came in and found another mess of problems. Red Teams and testing in general are inherently incomplete, but they’re useful in exposing what are perhaps the most obvious flaws.

In last week’s testimony, I mentioned some of the more obscure ways of breaking systems, Paul Kocher’s Differential Power Analysis, and Dan Bonet’s ?? Fault Injection, and various things like that which the Red Team normally would never even think of. And if a system is designed with the realistic threats in mind instead of requirements that spends all of those realistic threats, you get a very different result than if you tried to do a Red Team on something that was never designed to be secure in the first place.

So on one hand, Red Teams are useful. On the other hand, they’re not the best solution. They’re a useful addition. But again, I come back to having a good architecture and a good software engineering practice and open this in the entire development which would smoke out a lot of the problems before anybody has spent a lot of money building these systems, using them, Red Teaming them, and discovering that they are deeply flawed.

SENATOR BOWEN: Let me go back then to another point that Professor Rubin made which is that one of the improvements would be to publish the results of all of the testing that was done at the testing labs. And if you would spend a little more time on that, please—and in particular, I’m curious about let’s, first, just to make sure that everybody knows, I’d like to explain what happened with the Diebold source code and how it is that what was proprietary code became widely known because I think it’s important as background for people to understand why we’re concerned. And then let’s say that you have source code and a proprietary vendor whose proprietary code stayed proprietary, stayed secret. Of what use is it to publish the results if testing if no one knows what the underlying code is?

PROFESSOR RUBIN: Okay. I’ll address both of those.

Bev Harris was interested in Diebold. She was studying them. She’s very concerned about electronic voting. And she, through a search engine, found a web page—it was actually an FTP site, on Diebold’s own servers with all of their source code publicly accessible, and she downloaded it. My theory is that they had limited their thinking that no one would ever find it so that their engineers in the field would have access to it, would be able to look at it, although I don’t know that that’s the case. It could be they were just careless. Once that code was downloaded—it was archived in New Zealand—and then it was mired in many different places, and that’s how Dan and I and our two graduate students got our hands on it and were able to look at it. I think that’s everything you asked.

SENATOR BOWEN: