SENATE ELECTIONS, REAPPORTIONMENT & CONSTITUTIONAL AMENDMENTS
COMMITTEE
Informational Hearing on:
Open Source Software—
Does
State Capitol, Room 4202
Senator Debra Bowen, Chair
SENATOR DEBRA BOWEN: Good morning everyone. I’m Senator Debra Bowen. I would like to welcome you all to today’s meeting of the Senate Elections, Reapportionment & Constitutional Amendments Committee.
The
subject of today’s hearing is open source software. Many corporations now use open
source software systems, including Bank of America, Amazon.com, America Online,
Dream Works, Charles Schwab, IBM, and Merrill-Lynch, just to name a few. In
recent years, even federal and state agencies including, the state of
Today’s hearing will include a discussion of the Department of Defense and the California Air Resources Board’s experiences in moving towards open source computer systems.
My
goal with this hearing is to open the discussion about whether
And
so today, we will talk about the pros and cons and the challenges of going to
an open source model for
Last
year, the National Science Foundation awarded a $7.5 million grant to a team of
researchers from six institutions around the
I generally try to make hearings fairly interactive—that means that I may ask questions as we go along.
I just saw Senator Poochigian here. I don’t whether we will have other members here. I know there are other committees, and there’s a caucus retreat today for some members.
Anyone who has an interest in speaking during the public comments section, I would ask you to please let our sergeant at arms know. I don’t see any arms, but he is our sergeant at arms. He has arms. Maybe that’s it. He will hold them up and show you a signup sheet. We do that not because we wish to violate your privacy, and demand your social security number, or anything like that, but because it helps us know how many people are interested in addressing the body.
Our first panel is seated already. I want to thank you all for coming. Our first panelists include, Andrew Aitken, Founder and Managing Partner of Olliance Group; Michael Evans, the vice president of Corporate Development for Red Hat; Clark Kelso, a figure well-known to anyone who has attended hearings in which I have the gavel, chief information officer for the state of California; and, Anthony Hill, who is the chief technology officer for Golden Gate University.
I’d like to start out with Andrew Aitken. Thank you so much for agreeing to participate with us today. Please proceed.
ANDREW AITKEN: First I’d like to say, thank you for the invitation, Senator. I appreciate this opportunity.
I’ve been asked to provide some high level opening remarks on open source—the industry trends, dynamics, issues of that nature, which I will be happy to do so for a few minutes.
My media person here, Mike Evans, is going to be running my presentation for me. Thank you very much.
So a brief introduction: I’m here representing two organizations today. I am the founder and managing partner of Olliance Group, the leading open source management consulting firm. I’m also on the board of directors of the Open Source Software Institute, which is a nonprofit institution for providing advocacy and information for open source to the government.
So I’d like to start with some high level trends that we’re seeing around open source today. I will get into more details, get into more descriptions, advantages, challenges, and where the industry is going from our perspective.
A little bit more background about us, to put this presentation in context. Our firm, Olliance Group, has been around for four and a half years. We’re comprised of senior executives from firms like, Intel and Microsoft, and other vendors. And to date we’ve completed around 70 open source strategy engagements for most of the large OEMs and ISVs and also a number of smaller startups.
SENATOR BOWEN: I’m going to ask you to stop and tell the
listeners who are out around the state of
MR. AITKEN: Sorry. So we provided strategy consulting to large hardware and software vendors and also, both proprietary and open source startup companies and users—technology consumers.
SENATOR BOWEN: So your OEMs are your original equipment manufacturers.
MR. AITKEN: Yes. Sorry, I sometimes get lost in industry jargon.
SENATOR BOWEN: We all get lost there. But I will get complaints about it.
MR. AITKEN: Okay. So some interesting trends that we’re seeing today: The venture capital industry has gotten very interested in open source. They’ve made a number in the last 24 months. They’ve made more than 40 or 50 investments in open source companies, or companies that are beginning to sell open source solutions.
To date, one of the interesting aspects of that is, our primary client base are proprietary companies who produce proprietary products who want to understand how they can leverage open source in their technologies in their business models. That’s a very strong trend that we’re seeing today.
We’re also seeing a trend, most large technology vendors are beginning to move towards open source in some capacity, either consuming it internally, incorporating it into the products that they produce, or actually producing their own open source software and making it publicly available.
Today, open source is getting beyond infrastructure. Traditionally it’s very robust around the operating system, the web serving layers, application serving, email, technologies of that nature. Today we’re seeing a clear trend beyond that into more of the application layer, and I’ll talk a bit more about some companies that are in that space today.
One interesting item that is going to be very important for the entire open source industry is the newest version of the GNU public license, the GPL license, that governs more than 70 percent of all open source software, the first draft was released last month for public comment, and that process is going to go on throughout the year. And then they expect to complete the newest version of the GPL by Q-1 of next year. It’s a very important process for open source. There are some very interesting twists in the new license such as, how it treats digital rights management, and other technology issues.
The role of government: Government is important for the adoption of most technologies, but it seems to be particularly important, and especially on a global scale, for open source. It is being widely adopted by foreign governments around the globe, both at what we might compare to a federal and a state and a municipal level. There are a number of very, very wide spread efforts to do this. And I can talk in more detail later.
So, I think it’s important to also understand that open source or proprietary is not an either or equation. Today most large enterprises run both proprietary and open source technologies.
Some data points here: Open Office, which is comparable to Microsoft Office, is downloaded more than 800,000 times per month on average, and 60 percent of those downloads are on Windows.
SENATOR BOWEN: So that means you’ve got somebody who is running a Windows operating system who is downloading an Open Office software suite and probably has FireFox or Opera or some…
MR. AITKEN: In this case we’re talking about a Microsoft Office comparable product, so it has presentation, and has an Excel spreadsheet emulator, and it has a Microsoft Office Word offering. I mean, it’s a directly comparable suite. It’s about 90-95 percent compatible. And so they’re downloading this free in open source software and running it on top of Windows.
Postgress is the number two open source database. And approximately 60-65 percent of Postgress, again, freely downloadable, is downloaded and run on top of Windows.
Another interesting trend is that Microsoft is obviously becoming much more aware and involved in the open source community. They have open sourced three of their own very small applications to date. They’ve recently announced a partnership with J-Boss to make their solutions more interoperable.
So again, I would like to point out on this slide, that it’s not an either/or decision.
So these are some of the most cited advantages. Senator, if you have some specific questions, I’ll be happy to address them. But these are some fairly common advantages with open source.
SENATOR BOWEN: Actually, I think there are a couple of things that will be useful to explain in a little more depth. I know when people think open source, some people use the term “freeware,” or “free software,” and then the whole idea of a license for some people might seem strange. So, perhaps you can talk a little bit about open source software and the notion of licensing, and what the license does and why it’s needed.
MR. AITKEN: And I do have a slide on that, if you don’t mind.
SENATOR BOWEN: Sure. If you’re going to get to that, then…
MR. AITKEN: I have just a few more slides and then I’ll get right to that and describe it.
SENATOR BOWEN: Okay.
MR. AITKEN: So this is really one of the key questions—to open source or not to open source? I’d like to address the two key segments: technology vendors and technology consumers.
For technology vendors, it really boils down to two key issues: Are they able to reduce their SG&A (sales, general and administrative costs) by leveraging the dynamics of the open source community model? What this means is, are they able to open source their software to build a community around their software, which then helps drive revenue opportunities for them?
If you look at Oracle’s most recent financial statement, their license revenue did not cover their SG&A costs. So in essence, the people, you are paying for the privilege of an Oracle sales person coming and selling to you, not for the access to that license software—okay? Open source changes that equation, or, hopefully changes that equation.
The other reason for technology vendors to consider open source is, can they reduce their research and development costs and improve their time to market for release of their products by either incorporating open source or by implementing a more open development model?
For technology consumers such as…
SENATOR BOWEN: Can you talk a little bit about what the perceived advantage is in time to deployment? Why would it be any faster or cost any less for engineering?
MR. AITKEN: Sure.
SENATOR BOWEN: I’m going to ask you some questions that I know the answer to, or at least I think I do. By asking, I don’t have to demonstrate to you whether I’ve got it wrong or not, I just get to learn.
MR. AITKEN: Okay. So there is a concept of code reusability. There’s a lot of very, very robust open source code out there today and it allows organizations to take its existing code instead of trying to recreate it themselves. There is also the notion of the community conducts….the community doesn’t necessarily do your research and development for you. Where you can leverage open source for reducing R&D costs is, they tend to do a very good job of finding bugs, producing patches, and fixes, and things of that nature.
If you talk to Martin Mikos, CEO of MySQL, one of the premiere open source database companies, he likes to mention how he has the largest QA department in the world that no other company can compete with, and that’s his community and his customers, who go through the code and find bugs and provide fixes and patches and such.
SENATOR BOWEN: I actually had an email exchange recently with someone about that assertion, that there’s better ability to identify and correct bugs earlier on and it devolved into an argument about two leading internet browsers neither of which is without bugs. The assertion was made that the open source browser has as many bugs as the more traditional browser that probably more people are familiar with. And the assertion was then made that that proves that that’s a nice theory but it doesn’t actually happen that way. Your comment?
MR. AITKEN: I’ll name them. FireFox, I think is the open source browser you may be referring to and I have very, very personal experience with how buggy that can be. The browser that you download and run is very robust, very stable, and offers some wonderful features. The problem with open source, or the benefit of open source, depending upon how you look at it, is there are so many people who have written their own additions, new tools, utilities, and such, around FireFox, that if you want to incorporate those in your browser, they’re not all tested; they’re not all integrated. That’s one of the issues.
One day, I’m not quite sure why, but I went through a list of all the tools and utilities and added about 15 functions to my standard FireFox browser. And it took me about six hours to strip it all out and restart again. So that certainly can be one of the challenges. But over time, those tend to be, in fact, very rapidly, those tend to be addressed and fixed. You’ve probably heard the term “many eyes make for shallow bugs.”
SENATOR BOWEN: No, but it’s a good term.
MR. AITKEN: So for technology consumers I believe there are three primary questions to ask yourself when considering open source: Is there a compelling strategic reason? And I think that’s what we’re perhaps talking about today. I don’t know if it will save the state money to consider open source for their e-voting process. I’m not sure about the technological advantages, but there could certainly be a very strategic reason to do so. But this is what technology consumers need to ask themselves when they’re considering open source.
So briefly, I’d like to talk about some of the business models. Jenny had asked that I provide an understanding of how some companies are making money in open source, and these are some of the various business models and companies that are utilizing them.
So first is services—providing services and support around a particular open source application. J-boss, the leading open source application vendor, is a primary example. Red Hat, also, I’m sure Mike will talk in more detail about that. Compuair is an open source enterprise resource planning provider. It provides a core open source application, but makes their money from providing services around it. Then there are others that operate more on the traditional proprietary model. They provide a product and they provide a license to that product, and they may also provide services, but they generate the bulk of their revenue from a traditional license model. Sugar CRM is a prime example. They’re one of the open source success stories today. And Jasper Soft is a company that provides an open source business intelligence reporting tool.
A couple of others I want to point out briefly: Some people say that open source is more about commoditizing big, inefficient, existing proprietary vendors and that it doesn’t really innovate. I think this company Funambol, which provides open source from mobile applications, is a prime example of how the open source community can innovate, not just follow.
And a really interesting company is called Zimbra, and they’re doing both—they’re innovating, and they’re also commoditizing other vendors. They provide an email solution that competes with other proprietary email solutions. But they have some, frankly, cool technology in there that’s quite innovative.
Proprietary companies—how proprietary companies are making money around open source. Services, traditional consulting such as ourselves, such as large vendors, like, CSC is very obvious. Lost Leader, a company might create an open source application, put it into open source to generate interest, to generate the ability for more customers to see their actual proprietary revenue producing software. It actuates also in the business intelligence space. They provide an open source application that doesn’t generate too much revenue for them, but it drives revenue to their other products.
SENATOR BOWEN: Can you take a moment and tell people what the business intelligence space is?
MR. AITKEN: Sure. It’s reporting tools. So today technology is about generating information and data, well, you need to be able to access that data. You need to be able to extract usable information from it. And so they provide tools that will allow you to search large databases to create reports and make it usable for people like us.
SENATOR BOWEN: So they’re basically figuring out what’s in their own database largely and trying to put together what the information that is useful in business planning or in evaluating performance?
MR. AITKEN: Exactly. For making decisions based on data or information.
SENATOR BOWEN: Well, we use a lot of jargon in this world and I’m sure that I’m missing some of it, but will try to help as we go along.
MR. AITKEN: So there’s another strategy which is very important and it’s called “gifting,” or, “patronage.” IBM is one of the primary users of this strategy. What that means is, they actually will give code to the open source community to seed it to provide code that may not be core or critical to their proprietary offerings but will generate either a large community. In an example, they provided, I believe, $40 million in a number of resources to seed the Eclipse community. The Eclipse open source community which Eclipse is a nonprofit organization that has become the de facto standard for software development tools. And they did it to compete directly with Sun and Microsoft. And within two years it has gained tremendous, tremendous amount of attraction and interest. And so they did this by seeding the community, by giving resources and funds, and they’ve done that on a number of other areas also. I believe they have over 500 engineers contributing to Linux, is that right, Mike? Something like that? And so they’re working to make Linux, the operating system, better.
SENATOR BOWEN: And their return on that investment is, through their?
MR. AITKEN: Through the selling of their hardware. There’s a variety of advantages to them both what I would term as offensive advantages and defensive. So, putting a competitor in a down position, creating a new standard that happens revolve around their products and their solutions and their offerings.
Consumers….so those are examples of companies that consume or utilize open source technology but don’t necessarily produce or provide anything back to the community itself. So Google uses a tremendous amount of open source technology, but doesn’t provide or produce much of it’s own for the community. Oracle is an example; Linux, they base their products….their products run on Linux, and it’s the fastest growing of their products, so they run their products on top of various versions of Unix and Linux, and Linux is the fastest growing product suite they have there—product set.
And the last is, revitalization. So there are some companies there that would take what we might characterize as technologies that have lost their edge, losing their customer base, tired, as it were, and in some cases they believe that by open sourcing it, they may be able to revitalize that product and make it viable again.
Computer Associates is an example. They took a large database and have recently made it open source, formed a new company around it and are hoping that that will revitalize the technology.
To your license question: So, today there is an organization called the OSI that approves open source licenses. There are somewhere around 55 or more open source licenses to date. My favorite one is the one that says, “If you like my software, send me a beer.”
There is a movement afoot to reduce the number of licenses. A number of vendors who had their own open source licenses have retracted them.
SENATOR BOWEN: And you’re talking now about the legal language of the license—“Send me a beer,” as opposed to 40 pages of thou shall do this and thou shall not do that?
MR. AITKEN: Exactly. It doesn’t address the…
SENATOR BOWEN: So having 55 different licenses clearly increases the complexity for someone who is trying to use more than one piece of an open source…
MICHAEL EVANS: I think, Senator, one of your original questions was why you need a license if it’s free ____________ and the simple I give is that, the license is there to guarantee the freedom which is a higher level point than this.
MR. AITKEN: That’s a good point.
SENATOR BOWEN: In other words, you can’t take the open source software and close it?
MICHAEL EVANS: It depends. There are variations of licenses that allow you to do that and some don’t, and that’s the GPL being the one that tends to be the one that doesn’t allow you to do that.
MR. AITKEN: Most open source licenses fall into two categories, either permissive or restrictive (addressing exactly what Mike was saying). The permissive style of open source licenses on the right hand side of this slide, are the ones that allow you to take the open source software and do more with it what you will. More flexibility. The ones on the left, have more requirements to make sure that the open source code that you’re working with is, any derivative works you may create from that original code are also under that same license, so they are required to be made open source themselves.
The one that I want to know briefly is the one on the bottom left hand corner—the SPL. It’s called the sugar public license. It’s one that is becoming more and more popular with commercial open source companies because it allows for protection of your intellectual property. It allows for some branding of your software and provides some other protections that other licenses don’t. But it is not approved by the OSI, and that’s an important distinction.
SENATOR BOWEN: And what’s the practical impact of that? What does that mean…
MR. AITKEN: Of approval? By the OSI, the practical impact is that your software risks….if you publish a license that’s not approved to create your own license, the practical impact is that your software….well, first, you’ll be flamed to death, as they say through various open source license and discuss groups, which actually today impacts your revenue. So, there will be….today the open source community is no longer the geeks and the hackers who were working their second or third jobs at three in the morning. Today the open source community is comprised of developers working at a number of those firms that you mentioned today earlier in your opening remarks. One very pertinent example is, at Wells Fargo the head of their, or the director, of their infrastructure services is also an Apache committer, which means he’s at one of the highest levels of the Apache community, open source community…
SENATOR BOWEN: Can you tell us what the Apache community is?
MR. AITKEN: Sure. The Apache community is a nonprofit organization…
SENATOR BOWEN: When we talk about the layers of service, that’s…
MR. AITKEN: The Apache community is a nonprofit organization that manages a number of different open source projects today—probably 20 or 30 of them today. Apache itself is the most popular open source project that it manages, but there are a whole host of others that are under its umbrella. And Wells Fargo has made the determination that that technology is so important to them that they want to have someone whose job is to be involved with the community to provide their input and their requirements into the community, and hopefully impact the direction of the community, and also to learn and understand where the community is going itself. We are seeing that from more and more organizations.
SENATOR BOWEN: It’s interesting, it’s not all that different than what we see in the telecom and electric utility world, where if you take a look who is on a board of a local chamber of commerce, I challenge you to find more than a handful of local chambers of commerce in California who don’t have someone from either a telephone company or an electric utility serving somewhere in a volunteer position on the board.
MR. AITKEN: It sounds very similar.
SENATOR BOWEN: It’s interesting to see that moving into this area.
MR. AITKEN: So I wanted to conclude with one data point. Our firm recently completed a large research project for NOEM, which we did a lot of quantitative analysis. And one of the data points that we derived from that is that over the last three plus years, open source, and we looked at….this is comprised of…. probably 12 or 15 different open source applications were downloaded from the internet for free more than 450 million times globally. Most of those were downloaded and run on Windows, so I think it’s kind of a big data point.
SENATOR BOWEN: I’m responsible for some of them. I don’t have any family who is still running certain applications.
Let me ask you about the definition of open source and what you use. You know, there’s a running discussion in the open source community about what it means to be open source. What definition are you using? Or does it matter? Maybe we just shouldn’t discuss the head of that pin, I don’t know.
MR. AITKEN: Well, there are a lot of variables, certainly, to that definition whether you’re talking about individuals, organizations that are either commercial or governmental in nature. Personally, to be open source is, from a commercial perspective, means that you’re deriving revenue from open source solutions in some manner. As an individual it means that you’re contributing to the community however you can to the best of your ability. I’m not technical. No open source project would want me to contribute to their code. But I work with users groups and support them and put events together. That’s how I contribute to the community itself. So I think there’s some very wide latitude in that definition.
SENATOR BOWEN: Let me ask a few questions specifically about the process of determining what the appropriate solution is. When you’re working with a client and you’re evaluating whether to use open source software, proprietary software, or a mix, what process do you go through? Specifically, what kinds of things would you consider?
MR. AITKEN: So there is a very defined process to that. And the first thing is, the first question we may ask is, why are you doing this? What are you hoping to achieve? And it goes back to, is it a strategic imperative? Is there a financial imperative? Or do you expect there to be some technology benefit from this? Some of the issues that we look at when a customer gives us that response is, obviously you go through some of the basics, what is their IT organizational infrastructure? But more importantly, and what I think what many people miss is, what is their culture? Culture can be one of the key determining criteria’s in the successful adoption for open source or not.
I can give you a specific example of a retail organization that we helped develop an open source plan. We helped them select what applications they were going to use. And they were much more gung-ho about it than we were. And they wanted to adopt open source as fast they possibly could and we kept trying to put the brakes on that, because they were coming from a pure proprietary environment where the developers were very oriented to a certain technology set and they were enthusiastic about open source, but they really had no idea what that meant. And they deployed and began using a dozen, maybe 15, open source applications and it became a nightmare for them because they weren’t culturally prepared for the differences between open source and proprietary technology.
SENATOR BOWEN: And what do you mean, the cultural differences?
MR. AITKEN: It’s a different way of deploying and developing and managing your own infrastructure. It’s much more participative; it’s less hierarchical, and structural. Certain sets a proprietary technologies operate with very robust user interfaces. So it’s very simple to use. Open source technologies are what is called more command line driven, so you need a more sophisticated engineer to be using those technologies. And if you haven’t identified that, if you haven’t gone through that training, then that can become a significant issue. Additionally, open source might not be as well documented. You might have to provide your own support. And if you don’t examine these issues prior to begin using open source, it can become quite a challenge.
SENATOR BOWEN: You end up instead of paying for the software, you end up paying for someone to help you figure out how to use it.
MR. AITKEN: Yes. And that’s fine to get started with. You do that whether it’s proprietary or open source. But it’s important to identify those challenges; identify what your cultural organization is like to date, and then put a plan in place to address that rather than finding out after the fact.
SENATOR BOWEN: Okay. All right. Well, Clark Kelso has asked if he can have the second slot, since he has other engagements this morning. So, without any further ado, welcome. Glad to have you back again.
CLARK KELSO: Thank you, Madam Chair. Good morning. My
name is Clark Kelso. I’m the chief information officer for the state of
I can state the state’s general policy regarding open source relatively briefly. And this does appear in a memorandum that I released almost a year ago, February 16, 2005, that’s on my website. In essence, we view open source as an alternative that should be evaluated by departments and project owners and managers, and by the states IT professionals, as they are considering the appropriate technology solutions to support a department’s business needs and programs. There is no policy preference that we have established across the board in either project design, or in procurement for either open source, or proprietary solutions. We have instead tended to believe that the architecture of individual IT projects is really best determined by the project owner, and the analysis supporting those decisions typically is what should be appearing in feasibility study reports that are reviewed by the Department of Finance and by LAO and by the Legislature as new projects are moving along.
In considering alternatives, we look at a variety of factors. The most important of which probably includes alignment to the business needs, and the ability to produce value to business programs from a particular solution, security, reliability, performance, maintainability and sustainability over time, development and maintenance risks. We want to make sure that we’re going to not have a failure as we go through some of these projects, technology trends, and, of course, costs. Those are the major factors. There are others that are also considered, but that gives you a fair sense of the considerations that we ask departments to examine in determining an appropriate technology solution.
I think if you look at the history of the executive branch and IT, you can break it down into three or four big chunks. An early chunk, 30, 40 years ago, the state was much more itself, across the board, engaged in its own development activities, typically on mainframes. And the state was, I think, early on, very successful in developing and acquiring very stable platforms to support state programs. Nothing else can explain the fact that we have very stable 25-, 30-, 35-, year old legacy systems that are still serving the state very well. Beginning in the eighties and nineties, with the development of desktop personal computing and then midrange computing, much of which was done at the department level, we began to see an increase in the acquisition of commercial off the shelf products, cot solutions, as well as an increased usage of consultants to assist in the development of custom built applications.
I think today, when I say a commercial, off the shelf, it almost never is just commercial off the shelf, we almost always having to do a cot solution plus…
SENATOR BOWEN: There aren’t too many shelves that stock software for states of 37 million people.
MR. KELSO: Yes. And we, of course, government, does tend to have functions that are not replicated in the private sector. Many cot solutions are developed for really a private sector utilization. I think over time, also, what we’ve seen then, is depending on the department you’ve been in, the amount of attention paid to continuing to develop the ability, and maintain the ability, to do our own development, our own programming. That has varied substantially. Some departments made sure they maintained their workforce. They kept their training up. They kept them near the front edge of developing technologies. Other departments chose to go more the path of, let’s bring in consultants to assist in major development activities.
Part of the issue for departments, of course, has been the state’s HR systems. Something as seemingly mundane as the state’s classification systems for IT workers, which hasn’t been updated in 25 or 30 years itself. It does not recognize that the internet was created. That makes it sometimes difficult for the state to bring in and recruit folks directly out of college who have many of the skill sets that you do need to maintain cost effectively, a good development shop, which I think is one of the important prerequisites, not entirely, but an important aspect of the open source community and culture.
Now, what we have been doing is then several things. We do have several departments that early on made the decision, identified the trend in open source, and turned their systems towards open source. One of the leading departments that has done that is the Air Resources Board. This was a process that they began in the mid to late nineties. They managed to maintain a culture among their IT workforce and the skill sets to do development activities. So today, Air Resources Board, has reported to me that they run about 64 percent of their 88 servers on a Linux operating system. They have about 64 percent of their databases. They have 55 databases they’re managing using open source MySQL. Eighty-nine percent of their 61 web services are Apache based services. A very, I think, significant investment in open source technologies. And they report that they have seen in their operations cost savings that are substantial, exceeding half a million dollars or more. They think open source does provide best overall value to them and their solutions and their business needs. And echoing something that I think you heard before, it seems, although we haven’t quantified it, it seems to them that their development time is a little bit quicker. Now part of the reason for them, I think, that it’s a little quicker is, you are able to avoid, if you’ve got a good development shop and you’re able to pull things down off the web, you are able to avoid procurement cycles. And that, of course, can be a significant delay in the state in bringing up almost anything. That, of course, is a little bit peculiar to the state, but it is an important factor.
SENATOR BOWEN: Let me ask in a little more depth about that. How does procurement work for open source solutions? There is no procurement? You’re basically hiring people in your IT departments at the agency level who…
MR. KELSO: Well, it’s more complex than that, of course. Because the slides you saw from Mr. Aitken’s presentation, I thought it was, at least my understanding of open source community, a pretty fair picture of it. You can see that it’s much more complex than just, you pull it down off the internet; it’s free; you get to use it. So it depends, is my answer. To the extent that we’re going to be pulling down a portion of an open source solution, perhaps mixing it with something else, maybe we actually are going to be going to a vendor who is going to be helping us to develop something in part using open source, you still may have procurements, so it depends really on exactly what is it you’re developing; how small; how complex; can you do it entirely in-house? If you can, this maybe a way of simply avoiding much of the procurement activity. For big sorts of solutions and applications, I suspect there we still have procurements, open source can be a portion of that activity. So the answer isn’t, it depends.
And you can tell that the open source community is itself becoming somewhat more commercialized in a variety of very interesting ways. A development that has occurred as best I can tell over about the past three or four years significantly, the reference to venture capitalists coming in, major IT players like IBM suddenly showing up and saying, “We’re going to play in this space,” that’s fairly recent.
SENATOR BOWEN: And how do you work with departments, or how do departments determine where they’re going to use open source applications? How do we get to 64 percent running on Linux, 88 open source, 55 of the databases, (the web servers are easier because the dominance of Apache and that’s easier to understand) but still, that must mean you have 11 percent of your web servers that are running on something else, doesn’t that require a duplication of expertise and effort? How does that work?
MR. KELSO: I think the answer is, how do you get to that point; how doe they get to that point? It was primarily the leadership and commitment of their IT leadership. Bill Welty among others, who over the years decided that that was, for their department, a good strategic direction to go. And this relates to the things that we are now doing. And it relates to, I think, what Mr. Aitken referred to as that culture in the community that exists.
We did establish last year, a working group across agencies to track open source developments nationwide and within the state. We’re going to be conducting a survey very soon to see who actually has open source, to do some education across the state, because this is something that requires a bit of a retooling if you’re not ready to make the best value of open source solutions. And to create, really, a forum for those who are doing open source. And we have lots of people in different departments who are doing it at various levels, including people who are doing the open office sorts of applications, to give them a place to begin having a community building process so that the departments that are doing this don’t feel so much like they’re alone, and so that we can begin to interact a little more strategically with the open source community.
We also are very far along in discussions with the State Personnel Board, Department of Personnel Administration, and SEIU, to update the state’s IT classification system, to adopt an new testing methodology that would permit us easier access to college campuses, to help our recruiting of people who very likely will be very much in tuned with these types of developments. We have departments doing succession planning to try to see where our gaps are in the workforce. We’re doing statewide enterprise architecture including development of a service oriented architecture that, in a very small way, does some of the same things that the open source community is doing, I think, globally.
SENATOR BOWEN: Well, let me ask you a question.
MR. KELSO: And one final thing before I forget—we actually are looking pretty systematically, I think, as part of consolidation of our server infrastructure and management of our servers at the obvious swap out there. I think something can happen there fairly easily.
SENATOR BOWEN: Let me pick a wild example that I’m sure is
something you’ve never thought about before. Let’s imagine the possibility that
under the Real ID Act that we have to relicense, I think it’s 18 or 22 million
MR. KELSO: Well, and I think this goes back to what I’ve suggested before, but first let me say, that fortunately, I would never be in a position of actually having to make that decision. That’s why we have a Department of Finance that is well staffed to review project details. This would be a big enough project. I’m sure the Legislature itself would have an involvement. But, putting all those caveats to one side, I would be most concerned, I think, about sustainability and risk of a project’s success. Now everything that I think you’ve so far heard suggests that for something like a database development, and I suspect what we would be talking about with a Real ID implementation probably would be built on something that’s internet based, perhaps not, but it may well be. There’s clearly a trend in that direction. I would want to know that something as large as what will be required for Real ID has been implemented successfully somewhere else, either in the public sector or private sector.
SENATOR BOWEN: So you might look at a large fast food company and ask, do they have a way to track huge volumes of transactions with a lot of detail that they’ve implemented using an open source solution?
MR. KELSO: Sure. And I’m routinely getting advice from financial services industries which have similar sorts of database issues. So I’d want to make sure that the risk there is one that the state is comfortable in managing.
SENATOR BOWEN: So in other words, you don’t ever want to be the first, you’d rather be the second, or the third, or the fourth?
MR. KELSO: I think that’s a prudent approach when you’re using someone else’s tax dollars and you’re not just trying to do something with a quick return on investment. I would like to see that it’s been done elsewhere. Now my impression is, and I’m pleased to see with the open source community, that a lot of the major IT companies are beginning to invest there. We’re watching very closely of what other states and other governments are doing.
SENATOR BOWEN: I was just going to ask you about
MR. KELSO: Sure. I appeared on a panel at the Linux Conference last year with Peter Quinn. We were following their implementation with great interest. I mean, all state CIOs are following what is happening there. So we are trying to, through this working group in part, and through my own contacts, see how stable, sustainable, maintainable, open source solutions are. And the big question for me is, do we have a gap in our own capabilities.
SENATOR BOWEN: I should explain, because that question was
very much an insider question so I violated my own rule.
MR. KELSO: That’s why we thought it was worth tracking. I’m not sure it was all of their systems, but I believe someone here on this panel may know it. It certainly was going to be on their desktops, as I recall. Do we have anyone who knows for sure?
SENATOR BOWEN: I think we probably do on a subsequent panel.
MR. KELSO: I’m sorry. Bill.
UNIDENTIFIED: __________
MR. KELSO: It was an open document format across the board.
MR. EVANS: Could I say something just on this topic of the selection of open source that you addressed with Andrew.
SENATOR BOWEN: Please.
MR. EVANS: Because, six years at Red Hat, I’ve seen a range of governments, private enterprises, startups, to big corporations trying to say, “Should we be using open source?” And majoritively now it’s pragmatic versus idealistic decisions. And it’s just the normal stuff of, will the solution and utilization of it work for me short-, mid-, long-term and provide benefits? And then they start looking at the angles like, is there references of something similar? Is it going to be better, faster, cheaper? Is it going to be insecure and reliable? But then the smarter people, to me, are also looking at, now the solutions they’re looking at are likely to be 10, 20, 30, 50, and 100 years, as you described, a 25 year legacy already and starting say, “What will happen over that time period and, if that company that’s selling it goes away, what will I be able to do in 50 years to plug into it, extend it, get other people to work with it?”
SENATOR BOWEN: Let me ask you to identify yourself. We do have people who will be listening to this audio only, and when they start to get voices they’ve never heard before it’s even more confusing than the conversation about computer software already is.
MR. EVANS: Michael Evans from Red Hat.
SENATOR BOWEN: Thank you. And then, let me just ask Mr. Kelso one more question, because I know you’re going to have to leave.
MR. KELSO: I appreciate that.
SENATOR BOWEN: And then, I’m going to hear from Mr. Aitken on his comments, and we will get to Mr. Hill. This is the kind of interchange though, that I wanted to have. It’s much more interesting when we all have some input, than when we listen to 20, five-minute lectures.
Security, one of the things that I’ve heard touted about open source solutions is they are inherently more secure in part because of the shallow bug issue. Though others have said to me, “Well, they’re only more secure because they’re less of a target than some of the proprietary software that is….there are more users, so if you’re going to write a bug or write a virus, write one that’s big instead of one that’s little.” There’s also been the view expressed that open source solutions are inherently less secure because anybody can pull down the code (presumably, this is the code that someone is using) and change it. And I think some basic education about security and open source and where do you....if you are going to run an open source solution in a California agency, do you just Google the websites that have what you want and have everybody just download whatever it is they want, or do you have a more hierarchical kind of choice of deployment so that you know what’s running where?
MR. KELSO: Well, Mr. Welty, who is here in the audience, perhaps can address directly what they’ve done at ARB.
The open source community now is much more organized than simply Googling and grabbing things from here and there. That’s part of one of the benefits of that community, is that there’s a concentration of those solutions so that a lot of people can look at this one thing. It’s not completely decentralized. Mr. Welty, I’m sure, can suggest how they’ve done it. I can tell you from my own experiences as a computer programmer when I was younger, part of the culture of this is, you know, empowerment of the programmer who is able to go out there with the skills to bring in things that others have looked at and put them together in a way that serves your business needs.
SENATOR BOWEN: I think the concern is that, and specific to voting machines, I’ve heard the argument made that well, if you have open source software on a voting machine, anybody can come in and just put different open source….change the code on a particular voting….this presumes, of course, that they have access.
MR. KELSO: Sure. And on the security issue, I have no particular expertise. I’ve heard the same arguments on both sides. It seems to me, those arguments ultimately are inconclusive. I’d like to see something a little more quantitative and empirical. I would say, and I’m sure going to hear some of that, I would say, for me, anyway, the security problem of the day is the fact that we’re all networked. I’m much more worried about the number of connections that state governments networks has to the internet generally, and what are we doing to have defense in depth at all of those connections, knowing that the weakest one, I’m much more worried about that problem than I am about this particular dispute.
SENATOR BOWEN: Okay. Good. Let’s go to Mr. Evans. We haven’t given you a chance, so why don’t you proceed. If you want to start with this question and then do the rest of your presentation, that’s fine. But I think we’re not going to get into the meat of some of what we want to talk about.
MR. EVANS: Okay. So I’m Michael Evans of Red Hat. I just want to finish answering the security question, because the sound bytes that you mentioned are common ones about security.
But two points
that to me have evidence—one, as
SENATOR BOWEN: Can you explain locking down, opening. It inherently doesn’t make sense to some people, so you need to help us.
MR. EVANS: Because the source code is freely available in an open source manner under a GPL license or a BSD license, doesn’t mean that when it’s running on my system you’re able to come in and see my source code. That’s a leap, if you will, and it’s not true. I can lock down things that don’t allow you to see my source code. So that’s maybe a simple way to….although there are definitely a lot of people in the United States and in the world with expertise in this area. This topic has been debated back and forth with different agendas pushing different examples and scenarios and comparisons and coming up with results that often suit them. But we have deep experts in this topic at Red Hat and there’s deep experts in the open source community.
SENATOR BOWEN: A what person?
MR. EVANS: Deep experts.
SENATOR BOWEN: And would you define deep expert for us?
MR. EVANS: Someone who is a deep expert, is someone who fully understands all the elements of securing software and securing open source software.
SENATOR BOWEN: All right. And something that we have none of, by the way, in the Legislature.
MR. EVANS: No. I wouldn’t assume you would want them. Well, I would want them in this organization.
I’ll go ahead then.
SENATOR BOWEN: Please.
MR. EVANS:
It’s a pleasure to be here. I’m Michael Evans from Red Hat. I’m
doing my presentation in Adobe Acrobat because I had challenges with a
proprietary format in Microsoft Office getting translation, which is that
I’ve been six years with Red Hat, watching the evolution of open source over those six years and participating in it. I’m based in the Bay Area of California. And I’m going to talk just a little bit about Red Hat and our experiences in open source, as the company has been around a while.
The
company was founded in 1993; based in
So what do we do, actually? The simplest way I describe it is, we’re a business built on fueling and commercializing open source software. We do focus on infrastructure level software as was described, not application level like voting software, hospital software, accounting software. We are focused more at the infrastructure level. We’re considered one of the leaders…
SENATOR BOWEN: Could you explain….just take a minute again, we are talking here to a group of people who may have no sophistication. They may look at this slide and say, “What does it mean to fuel open source software, and what is the infrastructure level as opposed to the application level?” So if you could help us just with some basic stuff. We only have to do it once.
MR. EVANS: Sure. So fueling in terms of the business model around fueling and commercializing, fueling means helping to educate, propagate, and advance, and publicize that open source. That both the development model and the business model can work, are workable, are interesting, are providing value to the world and to people. Commercializing means, taking various free and open source technologies and packaging them up and offering paid services and options around them. In terms of infrastructure software it tends to apply to operating systems, or databases, or storage, things that run at the bottom level of the computing chain, if you will, that applications like, voting software or accounting software, or hospital management software, would run on top, that’s generally called the application level. And we are heavily focused on the infrastructure side.
We are considered one of the leaders, one of the best known names in open source. And there’s two parts to our model. There’s both an open source development model, which some call the supply chain. How we get the technologies to create that we then commercialize and offer to people to pay us for our services to provide to them. And the business model is how we commercialize that; how we charge for it; how we provide the services to governments, corporations, around the world.
We have a main focus at Red Hat, at least I’m providing things that are lower costs and provide higher value, for a couple of reasons. One, we believe open source allows us to produce better quality technology. And there’s examples of that. It’s the highest performing technology, or the most rich functionality. I’ll talk about some of those examples in a minute. And then lower costs of solutions, we believe allows a much wider use of technology around the world, which we are fans of propagating technology in more places for more people and reducing the silos usage or some of the constraints on financials.
We’re most well known for the Linux operating system, as I said. The Linux operating system has two main elements. There’s a server version that competes with the Unix vendors, with Novelle Netware, and Microsoft Windows versions in the server marketplace. And generally there’s a desktop or client’s side version that runs on desktop machines or devices like handheld devices, that tends to compete more with Microsoft on that side.
The next slide I have is, “Why open source works?” And these are the six elements we attribute to: Due to standards—true open standards which are royalty free and no restrictions; value—people pay for what they need. So there is an element of open source, at least at our level of technologies, where people can use things for free if they want, and then if the need the services they pay for them (Andrew mentioned some of that in his business model discussion); innovation—the unmatched speed of development where we have literally our programmers all around the world plus there’s literally hundreds of thousands of other programmers around the world contributing to some of our different projects like Linux, being a popular one; and the speed of development and the brain power you can tap, and that is unmatched of any one corporation trying to put people in one building or ten buildings; the quality; better software faster; choice, trying to provide choice—no vendor lock in; and full customization ability.
There’s a statement at the bottom I put there called “Software for the next 200 years,” which there is a paper by a gentleman by the name of Dan Bricklin who is from Massachusetts, a fairly famous person in the computer industry, having been the originator of VisiCalc, who wrote about software for the next 200 years, talking about that the software needs to be able to be built upon, modified, changed, adapted to, over time. And you can’t just have a two-year, five-year, ten-year vision of utilization, and if that company that sold you that software is gone in five years, what do you do then?
SENATOR BOWEN: You know, is that really possible? I can remember as a young lawyer, being probably one of the few people who ever went in the word processing room. I actually learned a little bit about how that stuff worked. We had something called a Wang. And we had floppy discs the size of dinner plates.
MR. EVANS: Right. You mean, is it possible that stuff will last for 200 years?
SENATOR BOWEN: Yes.
MR. EVANS: Well, but…
SENATOR BOWEN: That stuff didn’t last for 10.
MR. EVANS:
I just heard
SENATOR BOWEN: You know, government is unique in that. And
one of the things that I’ve learned, we laugh, but my first involvement with IT
was probably in 1994 in the Court Technology Taskforce, which is how I got to
know Clark Kelso. We were looking at providing the infrastructure for the court
system for aiming at the year 2020. And we have….everything we buy lasts
probably
MR. EVANS: And actually, my point is that especially as the internet infrastructure is exploding, you will have infrastructure that is built that is going to last similar to having the plumbing and the electrical and the sewage infrastructures of the world, that will be there for 100 years as the underpinnings. It will eventually be 100 years, but there is stuff that is going to be laid out as the utility infrastructure that is going to be just like when people dig up and find the 100 year old electricity that’s still working and plugging in. So to me, that’s a forethought that….there’s a great article if you haven’t seen this one, and a lot of it is about governments are going to be building their infrastructures around technology and the internet infrastructure.
SENATOR BOWEN: Well, that’s certainly been one of the major trends in the time from when I first created a Budget Subcommittee on Information Technology in 1995, because nobody was looking collectively at what any departments or agencies were doing. We didn’t know what we were spending; what we had. We didn’t even have an inventory.
MR. EVANS: And that analogy of, if you’re building infrastructure or certain technologies for the long haul to be modified, adapted, used….we use the analogy of a proprietary software being like buying a car with a hood welded shut. Open source is just like buying a car today. If you buy a car with a hood welded shut, there’s one dealer who can service it. He or she tells you when you’re able to end of life…
SENATOR BOWEN: I don’t think you have to have the hood welded shut. You just need to have a computer processing but only the dealer has the manual.
MR. EVANS: Yes. There’s validity to that, as well.
SENATOR BOWEN: Okay.
MR. EVANS: There’s another stack here, I’m not going to read through it, that talks about why open source software that goes down the….improved security though, audit ability, people can verify. There’s a transparency about open source software, which is a very common word around Red Hat. We like to believe we’re as transparent as possible in our business, as well as our technology. Cost reduction is an obvious winner. There’s stories of ten times, twenty times, performance increase with a quarter and half the cost that are rampant in terms of utilizing open source software in governments, in businesses, around the world.
As Andrew mentioned, there’s an entire ecosystem building around it. Almost every major vendor and minor vendor and small startups are looking at open source and how do they either work with it, work against it in some cases, or help propagate it? It’s a major impact on the global technology market. And then there’s better choice provided, we believe.
I just want to show one slide, just quickly, we sell free software, which is often a funny way to say what we do when I am at speeches. But this is our recent quarterly. We’re doing pretty good. We’re profitable, and sold $73 million of free software last quarter. So there is a real business to be had.
And this is an interesting data point, as well. There’s a CIO survey generally of corporate CIOs that we did not pay for a sponsor. And in the last two years, Red Hat has come out as the number one highest value provider to the IT industry _______ demonstrates the value of Linux and open source solutions.
There’s all sorts of market data on the expected growth of Linux market share compared to Windows and Unix in the server market. It’s obviously a worldwide force.
This is a story of open source benefits of a survey from Forester Group. Why do you like open source software? Some usual suspects up there—TCO, acquisitions, modifiable code, more choice, more hardware choice, better quality. There’s endless numbers of these surveys out there.
This next point I call “demand and range.” I just thought it would be useful to kind of describe what we see as we look around the world of demand in interest in Linux and open source software. It’s every country and region in the world, and in some places, very extreme. There are some governments that are going beyond what we advocate. We advocate choice, we don’t advocate that you dictate something.
Another interesting dynamic is in academia, where almost every technical school, university in the world is now creating people that come out of school with Linux and open source skills, because it’s a great teaching tool to be able to see the source code and evaluate the source code. And that’s a market force that’s just phenomenal around the world.
The range of customers, it’s everybody in the world. It’s financial markets, teleco, manufacturing, governments. Government market is actually one our largest and fastest growing. I’ve got every cabinet level department in the federal government uses open source. And the bottom one is, the range that open source technology is covering everything from servers, entire networks, and clients. Kind of the entire range of the technology infrastructure. It’s providing, due to the flexibility and allowance for people to modify, it allows solutions to be built by people all around the world.
And just my last slide then is what we advocate, which is, choice and options, free competition, and open standards.
SENATOR BOWEN: Okay, a couple of questions. You mentioned that you are focused on the infrastructure space. Who are the industry leaders in the application space in open source software?
MR. EVANS: Some of them were on Andrew’s slide. There’s Sugar CRM is currently one of the better known ones. Maybe you can help me.
SENATOR BOWEN: Should I toss that question to you?
MR. AITKEN: Sure. And obviously there’s a whole host of different application categories. And one of the points that I wanted to touch on briefly, was how a company like Sugar, has leveraged open source to grow, and I think they’re a good example for commercial open source. So they’re approximately 18 months old. Today they have over 500 paying customers. One of the most interesting data points, I think, is after they launched the 1.0 release of their software, their first real release of their software, they told me the story that they came back into the office the next morning and their software had already been localized in French and German overnight, in less than eight hours. In three weeks it had been localized in 17 different languages. And within, I think, 30 or 40 days, they had 45 partners that wanted to sell their software around the globe. And this was all because of the open source nature of the software itself.
So other categories—I mentioned business intelligence, there’s financial applications out there. There are a variety of different companies.
MR. EVANS: In my mind, it’s fair to state that the application wave of open source is just in its earliest stages, as well. The infrastructure sort of wave has proven things, and has been functioning for eight or nine years.
SENATOR BOWEN: How would you counter the argument, or do you think it’s accurate, that if we moved from proprietary software to open source software with regard to voting, that we will put the proprietary software vendors out of business because they’ll have to give away their product?
MR. EVANS: Well, I believe there’s business models that could be adapted, for one. That the proprietary voting software vendors should be looking at what kind of business model can they adapt to, or shift to? As Andrew described, there are several companies who are doing that—doing dual models or shifting things, and there’s ways to get paid for providing open source solutions.
SENATOR BOWEN: Anything to add?
MR. AITKEN: Yes, absolutely. Today most of the top technology vendors, both hardware and software, are moving towards providing open source based solutions, or driving revenue through open source. And typically it’s either they’re open sourcing some software and deriving services revenue from it—service, support, maintenance, things of that nature. Or, they’re actually providing it under a dual license. And what that means is, they create an open source version that maybe for free under an open source license, and they have another version which is either the same version of that software which they create under a commercial license and charge for. Or, they create two versions of their software, one they give out for free under open source, and the other that has a few more value added features to it, which they charge for. And those are becoming very proven models.
MR. EVANS: And there’s also open source to a limited group of people, as well, some people might have called shared source that Microsoft and others. So there are many ways to skin the cat, as they say.
SENATOR BOWEN: Actually we’ll get into this with the next panel, but voting software is somewhat unique because unlike, for example, drivers license databases or various other applications at the state level, or web services in any number of state agencies that run on Apache, we only use the voting application typically twice every other year. So it’s a sort of an odd beast, and then we don’t use it again. So that kind of intense but occasional use seems to be a place where the model of having someone else be there with the personnel that are required to deal with it, makes a whole lot of sense, so that we don’t have 58 counties having an IT department on staff year-round for something that is going to be used twice every other year.
MR. AITKEN: Do you pay them based on a license or a per usage fee for the software?
SENATOR BOWEN: Every county has their own arrangements, so there’s no standard. And some cities also have their own elections software. So, some cities use county software, county elections capability, and some run their own elections on their own equipment and software. So we don’t have much in the way of standards.
We had no
standards nationally until after
MR. AITKEN: This may be somewhat applicable. There are some companies that produce open source software that run kiosks for large corporations, HR related kiosks, and they charge, I think, on a per usage basis, which might be something similar. So a company produces and sells the software. It’s installed on a kiosk for XYZ Company…
SENATOR BOWEN: That might be a good idea to drive voter turnout, paid by the use of the kiosk.
MR. AITKEN: There you go. To help stimulate the vote. When that kiosk is used, and then there’s a fee for that usage.
SENATOR BOWEN: All right. Let’s go to Anthony Hill. Thank you.
I know we added you to this panel late yesterday afternoon and I really
appreciate you coming. You have gone to open source software at
ANTHONY HILL: Okay. Well, thank you, Senator Bowen, for the invitation, and to the panel for the invitation, to be here and to participate on this committee meeting on open source.
I
think as Senator Bowen was pointing
out,
So I’d like to talk today about why we chose open source solutions at Golden Gate University; how we have implemented open source solutions; what our experiences with open source solutions have been; and then to leave some parting thoughts about our experiences with open source, and parting thoughts that may guide some others in their own evaluation processes.
I’d
like to start out and very quickly give some business context to
So
the point of that is, is that information technology at a university like,
So
why did we choose open source at
Open source for us was a philosophy that we were optimistic about, and we’ve made our open source decisions at the product level. So we’ve selected open source products for various solutions in the university, and we selected these for cost advantages. Open source products are often a lot less expensive to own over the long term than proprietary solutions.
We also chose open source solutions for the breadth of support we get for those solutions. As many of the other panelists have pointed out, the open source solutions are often supported by global community of enthusiastic people who have strong interests in that particular subject area. And we have found in general that with our open source solutions, we get better support, better services, from this global community than we often get from a single vendor. So support and innovation was a big part of it.
And lastly, when we just looked at our technology strategy and what we needed to achieve as a university, looking at the different technologies that were available, looking at the technologies were going in the marketplace, we felt that we were best served by a mix of proprietary and open source products selected on a product by product basis.
I think a good example of that is Linux, the open source operating system that’s been mentioned many times today. We made the decision to go to Linux three and a half years ago, and to standardize on Linux as the core operating system for the university. We made that choice, choosing against the Unix operating system, which was very popular and very strong in the 1990s, and in the early part of this decade, and against the Windows operating system. We looked at the marketplace. We looked at where operating systems were headed. And most importantly in that decision, where the vendor community was headed in terms of which products they were going to write for which operating systems, and which operating systems the application vendors were steering their development efforts towards. And over the last three years, as has been pointed out already, the Linux marketplace for application vendors has been the fastest growing marketplace.
So, when we looked at where we wanted to be a few years in the future as a university, how we wanted to be positioned technologically, we wanted to be positioned with the broadest base of support in terms of the industry, and the best ability for us as a university to have a cost advantage, and to have a flexibility advantage, so we can continue to adapt our systems to the needs of our market.
So, Linux in that particular example, we felt gave us the best destination. Three and a half years ago we made that choice and the marketplace has proven that choice to be the right one in that example.
So those were the three primary reasons—cost, supportability, and the ability to enable a flexible technology strategy so we could adapt and grow in the future.
One of the key points there is reducing the concept of vendor lock-in. That’s been mentioned. The open source model does reduce or eliminate vendor lock-in. And with the open source platforms today, we don’t experience vendor lock-in. And as an IT manager, more and more that I work with open source, I’m appreciating the value of not being locked into vendors, and the flexibility that you get from that.
SENATOR BOWEN: Can you explain what that really means?
MR. HILL: Vendor lock-in?
SENATOR BOWEN: Yes.
MR. HILL: Vendor lock-in refers to the situation you get into in technology when you buy a technology product from a single vendor and then you can only get support from that vendor, you can only get product upgrades from that vendor, and essentially you become locked in. Your business strategy becomes married to that vendor’s product strategy, and that may or may not work to your best interests over time.
SENATOR BOWEN: And the difference with open source is?
MR. HILL: Well, open source, being supported by a global community and a broad base of vendors, you have the ability to migrate your business solution across multiple vendor products. And I can give you a very good example.
When we made the choice between Unix and Linux, we were choosing essentially between an operating system called Solaris, from a company called Sun Microsystems, and at that time the Sun operating system only ran on Sun’s hardware. The Linux operating system runs on a variety of vendors hardware, so we choose to run ours on computers from Dell Corporation, but we could choose IBM, Hewlett Packard, or the local computer vendor on the corner, and we could run Linux on those systems. So we eliminated hardware level vendor lock-in by choosing the Linux operating system as opposed to ensuring vendor lock-in at that time by choosing the Solaris operating system.
Just as a side comment, Sun Microsystems has since adapted. They ported their Solaris operating system to other hardware platforms, so that’s no longer the case. But it was the case three and a half years ago when we made our decision.
SENATOR BOWEN: Great. That’s a very good example. I appreciate that. It helps people understand. And I’m sure you get the same thing on the other side of it, where it’s the service. You decide that you don’t like the consulting team or the vendor you’re using and you don’t lose the investment you’ve made in the software and what you have, you simply have somebody else working on it.
MR. HILL: Yes, that’s right. We’ve got a broad community of support for the open source products, versus a single vendor as the general source of support for the product. If that answered your question.
SENATOR BOWEN: Yes, I think that’s very helpful.
MR. HILL:
Okay. So let me talk a bit about how
Well, you’ve heard the terms infrastructure versus application, and we’re using open source at both levels of our technology architecture. We’re using the Linux operating system from Red Hat as our core operating system for the university. That’s our infrastructure component. And then we’re using some other smaller open source tools from various open source communities to help us manage our network. So network monitoring—some of our security products are coming from the open source community, and working out very well.
MR. AITKEN: A quick question, do you mind? And for the audience it might be important to understand that I’m guessing that those tools that you’re using are noncommercial open source. They’re just open source projects but they meet and fit your needs. Is that right?
MR. HILL: Well, I think you brought up a good point, because in the case of the network monitoring, I couldn’t even answer your question, which I think is a good point. We use Nagios network monitoring. I couldn’t tell you if that’s an open source product or the outcome of an open source community effort.
MR. AITKEN: It’s an open source application to the open source community effort that the venture capital industry invested in about a year ago to form a company around.
So what you’re using is probably a open source version.
MR. HILL: I believe we’re using the open source version, but I think the key point there is as a consumer, I don’t always care exactly how some of these licensing issues, or the vendor side issues, have been structured. I mean, I do care about the licensing issues, we obviously need to comply, but whether what category of open source software would it fit in, I don’t always have to care about that on the consumer side.
SENATOR BOWEN: So if you were having problems, then you would need to find someone who you could work with and you might turn to Andrew or to someone else like that and say, “Help me figure out how to make this work better.”
MR. HILL: Sure. Well, yes. We know where we got the software. We’re familiar with the community of people working with Nagios. We’re familiar with the support vendors around the Nagios open source product, and we would turn to them and I’m sure we already do.
SENATOR BOWEN: What kind of cultural changes, if any, did you
have to make? We’ve heard a lot about open source requiring a cultural
shift—did you find that at
MR. HILL: I think you have cultural shift at two levels. You have to get the business people comfortable with the idea. And then you also have the cultural shift within IT itself.
Open source represents different products, and it really represents a way that products get built and people have product preferences. Some people are very, very emphatic about open source and they believe in open source philosophically. Sometimes that can cloud people’s vision. Other people latch onto vendors. I mean, Microsoft, they’re so big they make a global market in their products, so you can build an entire career around Microsoft products as an IT professional, so people get locked into that. So, you have cultural shifts at that level.
And at the business level, the business decision makers really need to be comfortable with the open source solutions, and I think that that’s an important component of any end user organization’s decision to adopt any product. There’s nothing really magic about open source there.
SENATOR BOWEN: Okay.
MR. HILL:
So for infrastructure at
At the application level though, we’re doing something a little different. We’re using open source components, software components, sometimes called software frameworks, and we’re building our solutions using these open source frameworks. These solutions have created our website and all of the online applications that drive our website, perhaps analogous to an online voting system if that were to be run over the internet versus on standalone servers at every voting location. So we have created our applications using open source components. So our website is the product that we build to service our customer base and we have built it entirely on open source components at most every level, but not at every level. So, I can provide some real world context for how this open source versus not open source might fit in an individual solution.
Our website runs on the Linux operating system which runs on Dell hardware—an open source operating system. We run it on an Oracle database—a proprietary database, not open source. And then everything else that we’ve built to create our website has been built on open source products, and we’ve integrated probably 15 different open source components to create a very powerful website and online application environment. We’ve written our own content management system very easily using open source components as opposed to having to purchase an expensive content management system, and we’ve had a great degree of success with open source components on the website. But the key point there is, is we didn’t buy a website that was open source, we bought open source components and we used those components to build our website. So open source products can be both consumer level products like the Linux operating system, and they can be building blocks that programmers would use to rapidly assemble the eventual solution.
SENATOR BOWEN: Let me ask you the security question. You have these components that you’ve gotten from various places, and you have a network of users around the globe who are using this website 24/7/365—why is that secure?
MR. HILL: That’s secure because of the environment the website runs in, and this has to do with the particular technical components themselves. We run the website on the Linux operating system using the Apache web server. We’ve heard those mentioned today. Those products, Linux and Apache, exist on a network that is secured with a variety of other vendors security products. Those are the layers that provide the security for our website, not the code inside the website. So we’ve used open source components to build our application, but the environment that the application runs in, is what provides the IT security.
SENATOR BOWEN: I think the fear is, or the concern is, that it’s open source. It’s open, therefore someone can get into your computer system and just change one of those components.
MR. HILL:
I’m glad you asked that because I’ve heard you ask that earlier
today and I don’t think it was really answered. Within the context of open
source, open source indicates the licensing model and the way that software
gets built. It doesn’t reflect, necessarily, how the product gets packaged and
deployed. Linux is an open source operating system that’s created by a
community of people worldwide. But when Linux is packaged and deployed in an
end user organization like ours, it’s highly secure. Nobody can get into the
source code that runs at
SENATOR BOWEN: Well, you’d have to have admin privileges and basically download another build or a patch.
MR. HILL: That’s right. Right. So, open source is a way to build software; it doesn’t mean that when the software runs, that it’s open. It can actually run highly secure.
SENATOR BOWEN: That’s the key. That’s a great way to put it. Would you say that again?
MR. HILL: Okay, I’ll say that again. Open source within the context of our discussion today, refers to how the software gets built, and how the software gets licensed. It does not refer to how the software runs at the consumer level. At the consumer level it can run highly secure, very tightly packaged, just like any other commercial software product, but it gets built by an open community of people.
MR. EVANS: What I was calling lock down. When you deploy it, it can be locked down, and so people can’t get to the source code.
MR. HILL: That’s right. Developing the software and deploying the software are two very different procedures, and two very different disciplines.
SENATOR BOWEN: So how it got built is irrelevant at the point when you’re loading it onto every PC in the Legislature. We don’t have the ability, I don’t have the ability, to go in, and if it’s an open source system, which I don’t think it is, and change the software simply because it was an open source building?
MR. HILL: That’s correct. The software on your personal computer would not be open. The software on your personal computer might be highly secure, but it was created by an open source community of developers and sold through an open source license.
Mr. Kelso earlier referred to a military term “defense and depth,” around your IT security. And just to support that, absolutely, you can provide security defense and depth through using open source security tools, and also other proprietary or commercial security tools, to secure your open source environment. There’s no real restrictions on that.
SENATOR BOWEN: So some of the security is inherent in the product in the way it’s built. Some of the security is, as in any other application, added on or another layer.
MR. HILL: Yes. That security is often a function of the practices around how you manage the technology. You know, a secure application in terms of an end user application, a business level application, a voting application, will come largely from the environment that that application runs in.
SENATOR BOWEN: And now you’re going to have to explain that because around this building “environment” means the temperature, and whether it’s raining, and whether it’s a woodland, oak, or switch grass.
MR. HILL: I think context would be the technology equivalent of what you’ve just mentioned. The physical environment being the buildings and the electricity and the plumbing and that sort of thing. And in the technology space, the environment is the servers that the application runs on, the operating system that the application runs on, the networking environment that the application runs on, that’s where the bulk of the security comes from.
SENATOR BOWEN: So this is part of why the discussion about bugs and whether what the flaw rate is, and how quickly things get corrected, comes into play.
MR. HILL: It does. For the types of products that provide security. And it would also apply to end user applications. I mean, applications need to be written in a secure way.
SENATOR BOWEN: No, I’ve certainly applications that inadvertently open holes in the underlying operating system.
MR. HILL: That’s correct. And that can happen with open source; that can happen with proprietary software; and I think the real issues comes down to, at the product level, be it open source or proprietary, how is the application written? How robust is it? How thoroughly was it tested? How mature is it in the marketplace in terms of an installed base which provides a larger test bed, the larger the installed base? And it’s really a product level decision. And there’s nothing really magic between open source and proprietary. And there’s the ongoing debate, can the open source community react more quickly than an individual software company? And I think you could come up with examples on either side to try and prove the point. I don’t think there is any absolute answer to that.
SENATOR BOWEN: Good. If any of the panelists have anything additional, that’s great. If you can be here, that’s terrific. We’re going to turn specifically in the next panels to open source systems and voting, because the business case is arguably different, and the challenges are unique in the voting environment. When you vote there’s no counter party. When you do an ATM transaction there’s somebody on the other side of it who could say, “Hey, that’s the wrong amount.” So, very different kind of context.
But I want to thank you. I think we got a much better understanding of many of the issues that are bandied about in the open source space. Thank you.
Let me call up the next panel—Deirdre Mulligan, Peter Neumann, and Joe Hall.
As we’re getting ready, another way of addressing this has come up from the committee consultant who says, “Just because everyone knows the recipe to the soup, doesn’t mean you can’t lock the door to the kitchen when you’re making it.” So, I think that’s perhaps even an easier way to understand it. And I challenge the members of the audience to come up with the simplest, most understandable way to understand this security concern.
All right. First of all, let me reintroduce Peter Neumann, who is the principal scientist at computer Science Lab at SRI International.
I’m going to start
this panel with Deirdre Mulligan, who is the director of the Samuleson Law,
Technology & Public Policy Clinic at Boalt Hall,
DEIRDRE MULLIGAN: It’s a pleasure to be here. Thank you for the invitation. I’m here today, as you mentioned earlier, with Peter Neumann and Joseph Lorenzo Hall, who is a Ph.D. student at our School for Information Management. All of us are here, and we’re very pleased to have the opportunity to talk with the committee, and we hope that we get to speak with you in the future as you continue to look at issues around electronic voting.
As you mentioned, we’re all collaborators on recently funded NSF center to look at the development of electronic voting systems that are accurate, reliable, transparent. And we believe that this conversation about open source software is an important component of that particular discussion.
So I wanted to begin by talking about what’s happened to our voting system. We’ve had what we like to call an enclosure of transparency. Voting systems used to be something that were observable, not just by experts, but by the public. And as we’ve moved to electronic voting, we have seen real barriers placed on the ability of election officials and the average member of the voting public to oversee election technology and ultimately have some confidence in that their vote is being captured and counted as they intended. And as computers replace the pens and the paper, these previously transparent and familiar processes, are becoming really, really difficult to evaluate. This enclosure of transparency is happening on a whole host of levels. And it’s incredibly important to consider what this enclosure means for the ability of the chief election officer of the state, for example, the secretary of state, to assess, validate, and test the voting systems that they are charged with evaluating.
And so I think a logical starting point for the conversation about the degree of openness required of voting technology is, to ask what level of access, review, and openness of code is necessary to ensure that the secretary of state can establish, with certainty, that election technology supports election values. And for me, that’s kind of the beginning of this inquiry. What’s necessary for us to basically have faith in the technologies that we choose?
SENATOR BOWEN: Let me stop you for one moment if I might, because your formulation of the issue is, and I probably didn’t catch it entire accurately, is what do we need so that the secretary of state can establish with certainty that….and I think some people would argue that the role of establishing with certainty should not belong to any single person or group of people, but rather should belong to all citizens.
MS. MULLIGAN: For me, it’s the beginning of the inquiry. It’s not necessarily under the inquiry, but I think it’s an important place to start. And I think given that the secretary of state, in their charge as chief election official of the state, has a particularized duty and set of obligations. And as beginning point of an inquiry, I think it is important to consider what must they have, what access to code, to testing procedures, to kind of the whole range of things that dictate how elections occur if they’re going to be able to independently evaluate whether or not systems are appropriate for use in this state.
So what’s required for effective oversight of voting systems? I think there are several prerequisites?
First, I think the secretary of state requires full and unfettered access not only to the source code of electronic voting systems, but to all the materials that are relevant to an exhaustive evaluation which includes, source of a code but goes far beyond its system documentation. It’s changed laws; it’s manuals; it’s procedures; it’s training documents.
Second, and incredibly importantly, the secretary of state has to have the resources. They have to have the expertise. They have to have the time necessary to understand and test the materials and machines that are seeking approval.
Third, they have to have an appropriate testing method. And this evaluation has to include security ratings along multiple axis. It has to include thread analysis and code review and architectural review, which is something that was touched upon earlier, penetration and parallel testing. It has to include usability, including accessibility testing. We have to have methods for identifying risk to voter privacy and equal participation, as well as security.
And fourth, which I think is where you were headed, Senator Bowen, is that the public has to be provided with information that allows them to not just to turn to the government and say, “Well, we trust you,” but the public has to be able to make, at some level, a decision about whether or not they trust the systems. We know that this is core to voter confidence.
And I think there’s an open conversation about what level of code disclosure is important for public trust. I would actually suggest that for the vast majority of the public it’s not going to be the disclosed code that’s going to give them confidence, it’s several other things that California has moved to provide such as, that verified voter paper trail because that’s something that the average person can actually look at. So I think you can look at open code as one component of how we provide transparency, but certainly not the only mechanism, and perhaps not the best mechanism depending on who it is we’re seeking to empower to provide some kind of oversight.
So, the next
question is, what is the potential role of open source software? It’s certainly
one mode for increasing code transparency. There was some conversation earlier
about what is open source software. For a lawyer, open source software is,
software that is disclosed under an open source license. And open source
licenses typically have several core terms. There are two general flavors. One
is the general public license, which you heard discussed earlier, and the other
is what’s called the BSD family of licenses, which actually have their home in
Where the GPL and the BSD families diverge is on what’s known in legal circles as whether or not they’re viral licenses. So the GPL, if I take code that is GPL code and I modify it and then I redistribute it, I have to redistribute it under the same terms under which I got it. And BSD licenses allow for more flexibility in the distribution of the code.
SENATOR BOWEN: I think this discussion about security really comes from that licensing requirement that software be freely distributable, and that is what, I think, makes people…
UNIDENTIFIED: __________
SENATOR BOWEN: Okay, good.
MS. MULLIGAN: I am going to leave. I don’t want to preempt either Peter or Joe’s comments.
So, open source code is one version of disclosed code. Public access to code can be achieved without an open source license. So we could have a requirement that the source code of voting technology is disclosed, but we wouldn’t necessarily have to have that code disclosed under a license that was an open source license. There are different models to think about providing either the secretary of state and some pool of experts or the public with access to the code on which these machines run. So, I think it’s important to understand the distinction between open source code and code disclosure.
And a third thing that I think is also very, very important and it came out in the discussion of Red Hat and Linux in particular, is that open source code is typically thought of as an open source movement, which is that there’s not just the code that at the end ups distributed under a particular open source license, but we also have an open source development project.
And so the process is one of millions of eyes looking at all the bugs, looking at the programs, running lots of scripting programs over to identify bugs, doing all different kinds of testing and modifications. And so we have this very active lively community of users who invested. They may not be paid, but don’t let that distract from the fact that these are highly motivated individuals who are invested in the improvement of a particular piece of code. And this base of knowledgeable experts is very much a part of the open source model of development that leads to, as others pointed out, very nice code that tends to be quite elegant, that tends to be quite well reviewed. In the academic world we like to think about peer review as being kind of the highest standard, and open source is this kind of wildly energetic peer review model. And it certainly is very deeply aligned with the transparency goals of our voting system.
Now that said, open source or disclosed code is certainly a necessary component of transparency, but I wouldn’t want you to believe that it was sufficient to provide transparency. Transparency in our voting process over voting technology requires much more insight into what happens at the federal level with both certification and testing. Right now in the independent testing authorities review technologies, we don’t know what tests they’re running; we don’t know what the results are; we know nothing other than whether they passed or failed. So there are many black boxes, so to speak.
SENATOR BOWEN: Well, we’ll peel that onion next week, I hope,
in
MS. MULLIGAN: I’m helping you tee it up. So there are many black boxes that currently limit our ability to make judgments and assessments about the systems on which we run.
So in closing….I’m sure you’re going to have many questions….but in closing, I’d like to say that ACCURATE, the group of us….and today I’m really speaking here on behalf of myself and not for everyone on the ACCURATE team….but we submitted comments to the EAC on the voluntary voting system guidelines, which are the voluntary guidelines that voting systems are tested to at the federal level. And in those comments we certainly stated that we believe ultimately all voting system’s source code, design documents, and security analysis, should be available to the public. So this is not a call for open source code, but it is a call for the disclosure of code on which these very vital systems run. But to be clear again, it calls for more than just the disclosure of the code. It calls for kind of transparency along a whole line of processes that will enable us to evaluate the functionality and suitability of voting systems.
I think in that document we also say that as an incremental step towards full public oversight, we believe the source code and related information must be available for review by independent experts. And I would have to say that California, under Secretary of State Shelley, under Secretary of State McPherson, with the help of the Legislature, and frankly, a very lively and engaged community of computer scientists and good government activists, has made enormous strides compared to the rest of the country in moving forward to developing models where we can have more certainty that the technology that we’re relying on meets public values.
SENATOR BOWEN: I do have a number of questions. First, let me
ask you a very simple question. You referred to a community of people who are
“invested in the code,” and someone has asked me in the past, well why? Why are
they invested in the code? Why are they up at
MS. MULLIGAN: It’s interesting. I think there’s a wealth of literature on kind of what motivates different individuals to give back to communities in ways that they’re not necessarily economically rewarded for. Certainly the open source community is a community full of status. Just because there’s a distributed model of development doesn’t mean that there aren’t a select group of individuals who serve as kind of the editors and managers of a given code base. I think that people view this both in a very kind of public spirited way, but I think it’s also something that you can see in the hacker, and I mean that in the positive way area too, where people are really interested in showing their skill and testing out their chops.
SENATOR BOWEN: That’s like getting your name on a movie credit when the movie rolls.
Does it make any
sense at all to you that we would have hundreds of different voting systems in
the
MS. MULLIGAN: I think that some of the divergence….I
mean, certainly the way in which our elections are run at the state and local
level means, the shape and size of ballots, there’s a whole host of issues that
are so localized at this point, that it’s hard to imagine a unitary federal
system. At the same time, I think that part of the reason that
SENATOR BOWEN: Yes, we’ve actually been through this
discussion in other context, where there’s dual federal and state jurisdiction.
And I don’t want to equate voting and pesticides, but the one where I’ve had
the most experience is in the licensing of pesticides, where the state has a
set of standards and the federal government has a different certification
process. And it took many years for a process called, harmonization to be done
that maintained public confidence, because that was really the issue in
Let me go to Mr. Neumann. I know there are going to be many more questions, but on this panel I really want to lay out the differences and then get into the subject matter.
Thank you very much for coming up. You are described to me as an open source guru and systems security expert, so, now you have to live up to that introduction.
PETER NEUMANN: Wonderful. I want to thank you for bringing us in, and for having this hearing in the first place. I think this is a rare opportunity to look at some of these issues in this kind of a setting.
I need to point
out that my background is very different from Deirdre’s and there’s going to be
a complete shift from some policy issues into technology. I’ve been in computer
related research for over 50 years, in security research for over 40 years, and
involved in voting machines for something now approaching 20 years. And I would
echo what
Strength and depth is a concept that is very old in the security community. But what we have is in fact, weakness in depth. Everything is a weak link in the voting process, from registration to the authentication of voters, in some places requiring three pieces of identification, or none, depending on precinct and perhaps the color of the voter.
I very briefly want to go through the state of the art as it exists today and say how open source can change that.
We’re dealing with very weak requirements. The 1990 federal standards were voluntary. They were replaced by the 2002 standards which were voluntary. They’re about to be replaced by new standards. Those standards are exceedingly weak. You can drive entire trucks through them with no trouble at all and completely vitiate the kinds of security that you’re asking about when you say how easy it is to put subversive changes into open source. Well, it’s easy to do it in proprietary code because the vendors have, first of all, proprietary code, proprietary evaluation standards, they’re paying for the evaluations, and in many cases they have the ability to change the system. In one vendor case I’m the fly during the election. In another case a vendor completely modifies it’s certified software to prepare the ballot face for each precinct that has a different face. They have a so-called back room operation, where they go in and they change the software despite the fact that it’s been certified, and despite the fact that the regulations supposedly say that this isn’t supposed to happen. It does happen.
So we have a situation…
SENATOR BOWEN: And that’s basically designed in. That’s just the way that software works.
MR. NEUMANN: That’s the way it’s built.
SENATOR BOWEN: That’s not a bug. That’s the works.
MR. NEUMANN: That’s the feature. That’s the way it’s supposed to be. And so there’s no configuration control; there’s no patch management. There are cases of dynamic fixes, so-called, to software during elections where, first of all, the system doesn’t boot in the morning for hours and a vendor agent comes in and he actually changes the software on the fly. So there is in today’s world very little configuration control; very little scrutiny of the software.
I’ve actually had
the pleasure, if you call it that, of being under deep nondisclosure to the
city of
And in looking at that source code, I came to the conclusion that even if the source code were perfect, there were still a couple of ways an election could be erroneous, undetectably, or maliciously altered without any evidence that it happened.
And I think the question of what can open source do….I’m going to try to answer some of the questions that you’re already asked in different contexts. I think the question that keeps getting asked is, is open source inherently more secure? In my written statement, which is on your website, I go into some length of pointing out that it’s not just whether the source code is open, it’s whether it’s been developed with good architecture, with good software engineering, with open standards, with open interfaces, with disclosable software, and you’re maintaining provenance or pedigree over every piece of software so that every change is in fact monitored in some way, and that the evaluations are open and that the system in fact provides some voter verifiable assurance that your vote actually is correctly recorded. And in essentially all of the electronic systems that we’re dealing with today, this is not the case. In fact, none of those things is the case.
So what the thrust of my written statement is not strictly open source; it’s openness, and openness throughout the entire process.
SENATOR BOWEN: I think that’s a good point. And this hearing is focused on open source software, but other hearings and discussions have been focused, and will be focused, on transparency in other parts of the electoral process. And we all like to use examples to help people understand, and mine is the third grade class election where everybody takes a piece of paper and marks a ballot, puts it in a hat and then two or four students and one or more teachers disappear into the classroom next door and they can observe the count of the ballot. Once we start putting machines, whether they be lever machines, which is how I cast my first ballot, or computers into the mix, we make it much more difficult for the public to observe. So this open source software discussion is a component, but not the only component, of an effective system.
MR. NEUMANN: One of the things that came out of
research that I have been involved in….and again I’m going to stress the
importance of research here. Way back in 1965 when I was working at Bell Labs
and commuting to MIT to work on the Multex System, Multex solved buffer overflow
problems. It solved the virus problem. It solved the Y2K problem in 1965. It
provided a level of security that was unavailable in any other commercially
available system for the entire century, basically. And, the system that I did
work on that I led the design of, SRI, in the seventies, was effectively the
basis for
My research over the past 40 years in security has dealt with, how do you design and implement systems where you don’t have to trust every piece of the system? Where, in fact, there is a one very small component where you can demonstrate using mathematical logic and formal methods and things like that, that this particular piece is in fact incorruptible in some sense. Nothing is perfectly incorruptible, when you have insiders and you building these systems on top of a Microsoft operating system for which anybody who has access to it can, in fact, undermine the entire voting process. But the point there, is that if you design a system in which only a small piece has to be trustworthy, and you can prove somehow, mathematically or through testing or whatever makes you happy, that this particular system, this piece, cannot be compromised, and that it cannot be altered from the outside, then you have an assurance that the voting process is, in fact, going to have higher, much higher, integrity than it could if there has been no architecture, no good software engineering, no architecture that would allow you to mix and match.
Ideally, in answer to your question are we going to put the vendors out of business, there’s a tremendous amount of need for policy support and what might be called logistic support in arranging an election, organizing it. But if you have to trust a proprietary vendor to do all of that, because nobody in the election process of the state or the government or the county, whatever, is competent to do it, then you are at the mercy of a supposedly trusted third party that may in fact not be trustworthy.
So the idea of mixing and matching with open interfaces is that you could take most of an existing system and provide this little piece that would provide the voter verification in one way or another—it could be paper; it could be an electronic system that was independent of the other system.
And I think one of the main things that we’re looking at in the NSF project across five universities and my lab is, how can we build systems in which you don’t have to trust absolutely everything; in which it is possible to have, if you wish, 100-year survivability of the system because you can mix and match. You can take a piece out and replace it with something else, because there is an evolvable architecture. And that’s something I’ve worked on for at least the last 40 years, on how do you build systems of that nature? So I think there’s a very important part here that has to do with the research issue.
SENATOR BOWEN: And does that exist in any context, whether it’s voting or otherwise?
MR. NEUMANN: It is very hard to find that in the commercial world, particularly in proprietary systems. The software engineering is perhaps also a victim of not very good education. We have some problems in this country of universities that don’t teach security reliability, survivable systems, good software engineering practice, and so on. So I think there’s a very fundamental problem, that a lot of the programmers who are going into industry today are very inexperienced. They’re building life critical systems. They’re building perhaps, voting systems. They’re building systems that are supposed to be human safe, and in fact, those systems aren’t. So this is not just a problem that arises in the voting area, but I think what I keep saying in this context is, that the voting system is really a paradigmatic example of one of the most complex security problems, because we’re trying to have privacy, not vote selling, no vote buying, privacy of how you voted, no coercion of being forced to vote a particular way, which is a problem with internet voting, for example, where you have no idea what circumstances the voter is actually engaged in voting. So I think the question…
SENATOR BOWEN: Much less if the voter who’s voting is actually the voter.
MR. NEUMANN: This is the question, sure. This is one of the many questions…
SENATOR BOWEN: If you give your code or your access point to someone else, sell it…
MR. NEUMANN: When I say voting is an end to end security problem, it is in fact a very complex problem. And I remember one professor in a doctoral exam who said, “I don’t understand why there is a problem here. I can write a program that will add up a bunch of numbers.” And the answer to, are the vendors going to go out of business if it were open source, I think the answer is clearly, no. I also say at the very end of my testimony, that I believe, and let me read this last sentence, “In the long run, effective technology transfer seems more likely to happen in systems that can be subjected to independent scrutiny.” And I point out that in any event, openness will remain the ultimate need if we are to attain correct, usable, reliable, auditable, and transparent elections. If nothing else, openness successes may have a forcing function on the closed proprietary system developers because competitively they’re going to have to do a much better job than they’ve been doing.
SENATOR BOWEN: Great. Would you explain, I was very interested in reading your material that you submitted, your formulation of the difference between security by obscurity and security by open design? I think it’s worth having…
MR. NEUMANN: Security by obscurity is a fraud in some sense.
SENATOR BOWEN: First of all, would you explain what you mean by security by obscurity?
MR. NEUMANN: It’s the putting a black box around something and saying, “Now nobody can guess what’s in it because they can’t see it.” And in fact, there is wonderful research on trying to derive what is inside the black box from the outside, effectively either reverse engineering or doing experiments on the black box and figuring out exactly how it was developed, and how it was designed, and how it was implemented. And of course from the security point of view, there are some colossal security vulnerabilities that have resulted from people with no knowledge of the inside of the black box, discovering a way of by, say, injecting a little bit of noise into the box. Dan _____ was able to derive the private key of a public key crypto system. Paul Kotcher was able to derive the private key by examining the power consumption of the black box. So these are very obscure and advanced techniques. But even much simpler is the notion that we’re going to stick our head in the sand and pretend that nobody will be able to figure out that our head’s in the sand. This is the paradigm of security by obscurity.
Security by constructive design is that you’ve designed a system and implemented in such a way that you can actually demonstrate that for any particular set of vulnerability potentials that is threats to the system, that none of those can succeed. Now there are always other schemes, like, for example, the two ways of breaking the crypto system that I just mentioned, that typically are not thought of by the designers. But ignoring that kind of a problem design security or security by architectural design, says that we’re going to allow anybody who wants to, to look at the system. And the best example of that is cryptography where we have extremely strong cryptographic algorithms which have been subjected to open analysis for years and years and years, and yet the analysts are totally unable to maybe take a few binary orders of magnitude. Maybe they can simplify a 256 bit key down to an exhaustive 224 bit, or something like that exhaustive search. But the idea that you can analyze a system of that nature and demonstrate again, in a mathematically robust sense, that the thing is sound, is vastly stronger than the security by obscurity, where you put the black box around it and you pretend that it’s secure and that nobody is going to be able to find out.
I have one favorite quote which comes from, I think, three different hearings. It’s the same quote from three different vendors, all whom have said, “No, if anybody were to be able to look at the software, it would diminish the security of our system.” And this is absolutely fallacious.
SENATOR BOWEN: Good. Interesting. I have questions, but I want to go to Mr. Hall first.
MR. NEUMANN: Let’s get Joe in first, and then you should have questions for all three of us, perhaps.
SENATOR BOWEN: Joe Hall, thank you very much for being here.
JOE HALL: Thank you for inviting us. My name is Joseph
Lorenzo Hall. I’m a Ph.D. student at the
Anyway, I’d like
to talk about three things today. Specifically, barriers to using open source
in the voting systems market, which open source business models may actually be
viable. And this to emphasize these are all sort of open research questions. No
one’s attempted to do some of this stuff. And then finally, the Australian experience
with Evex and Canbera, in
So first of all, there are some real barriers to businesses or nonprofits or whomever using open source software in the voting systems market. I think the first and the most significant is the regulatory barrier. That is, any changes, any software or whatever, has to be passed back through various levels of certification. That’s a very necessary part of evaluating voting systems. But at the same time, it means that every time you change the software, it should go back through the federal and state and local certification and acceptance testing processes.
The second barrier that I’d like to highlight is the economic barrier that organizations and institutions using open source would face. Specifically, it costs a lot of money to go through various levels of certification. You can imagine the federal testing is estimated to cost hundreds of thousands of dollars for a brand new developed system, and that’s probably on the low side, at least the low hundreds of thousands of dollars. For sure, it’s going to cost money for marketing integration, support, things like that and having that in place.
And closely connected to that is another barrier, which is the organizational barrier. Most open source projects, for example, are just that. They are a web space. A place where people can discuss things, and post code, and that sort of stuff. Whereas, open source businesses are much more. They require quite a bit of organizational coordination to ensure that their product is viable in the market they work in.
Finally, there’s a process related barrier that is if the code is open and there’s a vulnerability found open or disclosed, I should emphasize by whatever means, and a bug or vulnerability is found close to an election, we don’t currently have a policy mechanism for deciding at what level you decide to postpone an election, for example. I can think off the top of my head that it wouldn’t be that complicated, but as we all know, postponing elections is not trivial, and something to be avoided by all means.
There are some pitfalls of unilaterally disclosing the source code, and I don’t imagine anyone is going to do that. But, I can talk later about those kinds of pitfalls.
Now on to what kinds of business models. So as the presenter earlier today mentioned, there are a whole lot of ways companies are using to make money off of open source software.
Certain business models that are being used will decidedly not translate well to the voting systems market. For example, Google, runs a lot of open source software to provide search services. Given the problems with remote voting, there’s just no way you could do something like that.
There are other types of models that just wouldn’t be viable, such as subscription based software update models where you have to buy a subscription in order to get software updates. It would be horrible for a county to not have access to the latest software because they didn’t pay their subscription.
There are other things like, optimization and customization that a lot of companies that were spoken about earlier do where they take a piece of open source software and either build a component that works with that, or they improve the open source product itself. That would of course be passed back through the certification process at all levels in order to be useful. So it would have to be that much profitable.
Certain others
business models could be viable, and not being a business person, I’m just
thinking in terms of inputs and outputs and might it work. Things like system
integration, where you’re not really concerned with the various products you’re
putting together but what you’re delivering is a total package. You could
imagine if there was a body of open source software that worked on commodity
hardware you could purchase, you could contract with someone like
There are other things
like targeted development. If there was an open source software voting system
in a project, to get that certified, you could specifically hire developers to
enhance features, to fix something that didn’t work, to implement, for example,
IRV (instant runoff voting) which is something we’ve had trouble with in
There’s the dual licensing approach which was mentioned this morning. You could offer sort of a GPL version or whatever of the software to jurisdictions that didn’t want to pay for it and were going to take care of the rest themselves. Or you could provide a commercially licensed version of the software to commercial integrators or vendors or people that wanted to actually not have to release their source.
SENATOR BOWEN: And you know, in a lot of places in
MR. HALL: That actually speaks well to my last point under business models, which is, there are hybrid business models that are more along the nonprofit. For example, there’s a project called the Psychi Project that UC Berkeley, Stanford, MIT, and a bunch of other institutions decided one day that they were sick and tired of paying licensing fees for course website management software and that they were going to form a consortium and do it themselves. And in order to be a part of this consortium and to actually contribute features and code, you have to agree to a number of requirements. For example, donating two fulltime coders, paying a yearly due, and then agreeing that your intellectual property will be released under the terms of this license, which in their case is the BSD license, a very permissive license. That’s actually working pretty well for them. And I can imagine that working really well. I think it would take a good feasibility study to definitively say this, but my naïve….I think that it could work very well in voting systems if you had the yearly due, for example, cover sort of the administration and the certification costs, which are a big sunk cost in this part of the market. And so I think a community source model could work really well. And you could, and example, a consortium of states, not even counties, for example, but other people realizing it’s in their interest to do this. There’s open questions that need to be answered such as, some of the market, and if you’re going to try and sell the product to other people, we may have marketing. There’s things like hardware integration. Are you going to use commodity hardware? Are you going to actually design hardware yourself? That’s not something that an open source really does a lot of, but it could be handled with, if you spent enough time thinking about it.
SENATOR BOWEN: Yes. And the hardware is a little bit different for voting systems than it would be for other things, because you actually have to take whatever the hardware is to a whole lot of different polling places.
MR. HALL: Yes. One vendor, for example, Populex, actually has all stuff you could go buy at a Staples in a box, and the box makes it look like it’s a really fancy voting machine, but in the essence it’s essentially….so they, for example, are the least interested in open source because all of their development investment is in ________ source of their code, but they’re also interested in exploring in this as well.
MS. MULLIGAN: You asked a really important question, and it’s a broader question about kind of the sustainability of the way in which we currently regulate and conduct elections.
And I have to state, I’m on the Board of the California Voter Foundation and Kim Alexander, I know is in the audience, and I’m sure is dying to answer this question, so I’m going to let her do most of it. But, I think certainly election technology has changed at a rate that election, kind of, regulation and conduct hasn’t. And it used to be that we could have the retirees come down and do a credible job of making sure the papers and then pens got handed out and they got back in the box. But we’re now putting these really complicated pieces of technology into polling places all around the country and training issues, and maintenance issues, and service issues, and trouble shooting issues, I mean these are things that all of a sudden they don’t look like what I want to do in my garage, which is where many people vote. And so I think that the transparency issues and the oversight issues are a good point to say, “Well, should we reflect upon kind of the administration of elections, generally,” and whether or not the technology that we’re currently using is something that needs a different level of professional support. This is not the pen and paper that we had in 1902. We’re looking at radically different, and what are the requirements for this to deliver on things like our promise to have every vote counted equally. And if we have differences in poll worker training, and differences in poll worker expertise, and it radically influences the ability of me to cast my vote, I think we begin to get into an area where we really need to think about some level, to use your word, of harmonization.
MR. NEUMANN: Let me add something. In some contexts I am known as the designated holist. So let me take a broader view of this question. Years ago you voted on a lever machine. I did also in 1950. And lever machines can be rigged, but you can only rig…
SENATOR BOWEN: I’ve learned a really great way involving a really sophisticated device called a number 2 pencil. You just jam it into the lever.
MR. NEUMANN: Yes. Okay, now the designated holist
says, “Well, okay, but then we went to punch cards.” In lever machines you can only rig one
machine at a time. In punch card machines and punch card systems you can rig an
entire ballot box, or deck of cards, irrespective of the machine they came from.
This happened in
SENATOR BOWEN: You’re reading the same alternative press I am.
MR. NEUMANN: I have another case of that happening in the House years ago where they had significantly more than the number of votes that had actually been tabulated. And of course the machine died in the middle and they rebooted it, and all of the original votes were left over. But the point is, with the e-voting you can, in fact, effect statewide, nationwide, in the sense that there are vendors who are nationwide. And whether or not it’s accidental or intentional or not is not really the point. The point is that there is, in today’s direct recording systems, no evidence that the vote did go in correctly. And if something does go wrong, and you discover that there are thousands more votes than people who voted, that there’s nothing you can do about it because there’s no audit trail there’s no accountability there’s no evidence that you can take into court if there’s a protest. And so I think in the question of scale, we’ve gone from lever machines to optical scan, to all electronic machines with absolutely no paper and no audit ability. The scale of the problem has become dramatically different.
So again, I started out by saying that voting is an end to end paradigmatically complex security problem. No matter how you slice it, we have problems. And the only sensible strategy here is openness, I think.
SENATOR BOWEN: Your argument though, and your discussion
about the vulnerabilities of optical scanners suggests that we go back to just
using a paper ballot and marking it with a pen and then having human beings
count everything. Now, I can only imagine that on the 2002
MR. NEUMANN: With 20 propositions and 14 candidate
issues and all that, yes, it’s impossible. In
But I think the point is, that there needs to be accountability and oversight and openness throughout the entire process. And here we’re just focusing on the little piece of it that is the all electronic machine that at the moment has almost no accountability.
SENATOR BOWEN: Although I think to be fair, we need to be focused not just on the DREs, but at some point also on the tabulation which also is a computer driven process.
MR. NEUMANN: Absolutely, I could not agree more.
SENATOR BOWEN: Which is not even transparent to the extent that the last time I went and watched a vote count I saw trays of ballots going into the plexi glass enclosure, but I don’t know if they actually went into the machine or not, nor do I know what happened after they did. And I don’t know if they were actual ballots. I had to assume a whole lot of things.
MR. NEUMANN: And that is exactly the point. That there are so many assumptions that we have to make, and we need to have transparency on every one of those assumptions to see whether it is in fact being satisfied.
SENATOR BOWEN: That’s software too, that runs the optical scan machine. I think the difference there is that optical scan is a technology that’s used in a lot of places for a lot of things, so we’ve got some idea about that software. And you also do have a piece of paper that you can manually recount.
MR. NEUMANN: That’s the key point. Good. Hey, this has been an absolutely wonderful hearing, I think. You’re getting at some very complex issues and recognizing that they are complex.
SENATOR BOWEN: They are complex. And you know, we just had,
yesterday, a couple of elections in other countries and was hearing this
morning on the radio that ballots are going by donkey to the central counting
location. And I was reminded of the last
MR. NEUMANN: And it would be nice to get it right demonstrably, rather than just come out with a preliminary result.
SENATOR BOWEN: Let me ask another question to the entire
panel. As you know, that we have a large number of voters, the number depends
on the county, who have registered as permanent absentee voters and vote by
mail in every election. Some of them bring their ballot into the polling place
on election day because they aren’t sure that, since they forgot to mail it
until that day, they know that it won’t get there if they put it in the mail
box that day. But that poses another question with counting and auditing. It
poses issues dealing with knowing whether or not the ballot has actually
arrived. That’s a solvable issue.
MR. NEUMANN: This should be independent of that. Joe showed me the legislation that you introduced yesterday, which makes a great deal of sense.
MR. HALL: SB 1235.
MR. NEUMANN: Let me point out that there serious
problems with not counting all of the absentee ballots.
Here, there seem to be entire counties and precincts that don’t bother to count them unless their election is close, and that is a, I think, very serious problem. As we get to the point where voters don’t trust the electronic systems and are increasingly apparently going to absentee ballots, if those votes aren’t in fact being counted, there’s a real problem.
Now one of our affiliates with our NSF ACCURATE project, in fact, has some schemes which are cryptographically based where a voter can verify if the system is correct that his or her vote was correctly recorded but can do it in a way that he can’t prove to somebody else that he voted the way he thinks he voted, and yet he has some assurance that his vote did go in correctly. It’s a fairly complicated scheme, and it’s not clear that it passes the voter trust issue at this point. There is another vendor who has a similar but much more cryptographically oriented approach.
SENATOR BOWEN: I think that’s actually an important thing to move towards as we go along because it’s very difficult with absentee ballots for a couple of things. One is to see how the ballots tied back to the precincts so that people who are actually volunteer working in the political process not on this end of it, but actually on the ground, can’t really tell what happened, and they have a hard time knowing whether or not somebody has voted absentee in a way that’s timely in terms of where their efforts should be.
MR. NEUMANN: One of the things we are going to be researching in this project is the kind of scheme where the voter can, in fact, prove, in some sense, that his vote did go in correctly.
I wanted to throw in one comment. Gambling machines are held to a much higher standard than voting machines, and that’s very ironic, I think.
SENATOR BOWEN: Thank you. Yes, I’ve actually asked somebody to look for me an analysis, the Legislative Analyst’s Office, of the difference between how we certify slot machines and how we certify electronic voting machines.
MR. HALL: It’s appalling. Send it our way. We’d probably love to see it.
SENATOR BOWEN: We probably don’t have the technical capability to do what’s really needed to be done.
But let me ask the
panel, each of the panelists a few more questions. First is, how should
MR. HALL: Well, I would say that it would be ill advised to just turn on a mandate. It would be ill advised to just automatically flip a switch of a bill and pass it into law that said this is how it’s going to be. I think we need….for example, the secretary of state recently passed, or, promulgated requirements that say that in addition to storing the code with an approved escrow facility, which is in San Diego, you have to deposit the source code with the secretary of state and he reserves the right to allow that code to be reviewed by independent parties.
SENATOR BOWEN: But can I look at it?
MR. HALL: He reserves the right to allow the code to be reviewed by independent parties. So if he chooses you as an independent party, then you can look at it.
SENATOR BOWEN: Why should I have to ask?
MR. HALL: Well, what I’m trying to advocate for are small steps. And the first thing you can do to increase transparency is, have an independent publicly available review so the result documents from that will be available and people like, Peter, and people who know a lot about this stuff, would be on such a panel and could say, this meets this set of requirements; it doesn’t meet these, and things like that. And I think that’s probably a first good step in that direction.
SENATOR BOWEN: I have real questions about that approach. It seems to me that if you’re talking about national security risks, there’s a justification for secrecy. But other than that, I’m really hard pressed to understand what the case is for not allowing anybody who wants to, to review the software that’s used to record and tally votes.
MR. NEUMANN: Well, again, it’s get back to the licensing agreement. There is the competitive interest of one company against another. That’s not the only argument for security by obscurity. There are other arguments, as well. But I think that is one where the vendors, in particular, are concerned. Now if you were to enforce not a particular open source licensing arrangement, but the assertion that all evaluations must be open, and that all interfaces, all internal interfaces within the system, must be not only open, but nonproprietary. At the moment, even if they have an interface, there lots of software vendors who say, you cannot even use our interface. You can’t even use our API (application interface).
And if in fact, as I said earlier, if we had a system in which the interfaces were nonproprietary and open and published, you could take 90 percent of the vendor system and then tie this little 10 percent thing that was the thing that provides the voter verified assurance, or provides a demonstration that the vote has been correctly recorded, independent of all of this other 90 percent, then you would have achieved something, I think, very spectacular in the sense that you’re not taking anything away from the vendor’s ability to do their thing.
Now, I think it’s difficult to demonstrate in a proprietary system that the system is not subvertable. And again, if you put it on an operating system where the keys are stored in memory, basically, for all of….the crypto keys for the voting system, as one of the vendors did, then the system is in fact subvertible. But the point here is, if the interfaces are open and accessible, and the specification…
SENATOR BOWEN: Describe for us what you mean by the interfaces because I’m not sure…
MR. NEUMANN: Well, suppose you build a system in which one piece says I’m entering my vote, and it comes back and it gives me all sorts of guidance and says “You’ve just tried to over vote, we’re going to let you do that. Did you really mean…
SENATOR BOWEN: To vote for Pat Buchanan and Al Gore.
MR. NEUMANN: Yes, exactly. Or you voted for one, but there are five candidates—the under vote problem. You can have one system that does that—the preparation system. The entry. And then another system that actually records the vote. And at that point you could say, well with the voter verified audit trail the voter is given the choice, does the vote you are about to approve agree with what’s on the screen? And if it doesn’t, you take that machine out of service because it’s obviously doing something wrong.
We had one case of that kind of a system where the election commissioner said “The voter verified audit trail is not agreeing with what’s on the screen.” What’s on the screen must be right, so let’s turn off the audit trail. That was obviously the wrong solution.
We had another
case in
So I think the question of the open interface is that if you can mix and match different pieces of different vendor systems…
SENATOR BOWEN: Are you arguing, essentially, that we have a separate vendor provide the piece of the system that does the voter verified paper trail?
MR. NEUMANN: You could have an open source highly evaluated trusted system because it had been proven correct, for example again, by extremely advanced mathematical techniques by two different teams, for example, so that you can trust that they’re coming up with….they’re independent and they probably are not colluding.
And that, in fact, the rest of the system is only for, say, ballot preparation, and it’s doing all this fancy stuff of getting the ballot face right, and making sure that when you press for this guy, you’re actually getting the right result, but that the actual verification is a separate entity. And that could be open source. It could be open interface. It could be highly subjected to scrutiny, and you can demonstrate that this was not comp….that even if something went wrong in the other one, it would not compromise the results of the election.
SENATOR BOWEN: And then you need a good audit on top of that.
MR. NEUMANN: Yes. Right.
SENATOR BOWEN: But that alone isn’t sufficient.
MR. NEUMANN: Yes. So the accountability and the auditing is something that perhaps could be done independently. And if you could do that, and that was an open interface to that system from the rest of the system, you could in fact mix and match your different vendors.
Now, it may be in the long run, you asked, are we going to wind up with 100 different systems? On my website is a two and a half year study that I did for the Department of Defense on how to design and build 100-year systems, evolvable systems, where the architecture is such that you can take out pieces, you can replace them. It relies heavily on open interfaces and on an architectural concept that is engineered from the very beginning to address security reliability, availability, survivability, interoperatability and all these other things. The problem is, when you try to add that onto a system that was black box designed by people who didn’t have the requirements that are needed, you’re never going to get there. Does that sort of answer your question?
SENATOR BOWEN: But that seems to conflict with what you just said about having an open interface and openness in the verification process. You’re really talking about two different goals.
MR. NEUMANN: Well, I’m saying that it is conceivable, yes. It’s conceivable that you could have a hybrid solution, where a piece of the system, in fact, the critical pieces are open source.
MS. MULLIGAN: I think the answer to your question is part to probe a little bit more deeply about what are the state’s goals—right? I don’t think the state’s goals are necessarily tied up with having code that can be freely modified, freely distributed, and has kind of a viral licensing effect. I do view the state’s goals as deeply tied into the ability of experts, the public, to review the code. And so I think mandating open source gets you transparency, but it also gets you a whole bunch of other things, which the state may or may not have a deep desire for. Now, if the state decided that it was interested in perhaps a business model around open source development, and we had folks from Red Hat and others earlier talk about some of the pragmatic considerations, the state IT officer talked about, that go into decisions about how to use investment dollars. And I think if you waved your wand tomorrow and said, well, we’re only going to use open source products, you would have a problem in that there aren’t any yet for you to buy. And there are some very, very serious issues present in the form of regulatory barriers and sheer economics that are going to really dictate whether or not those products come to market.
MR. NEUMANN: And certification as well.
MS. MULLIGAN: Well, that’s part of the regulatory barriers, can they get through the certification and testing process? I think those can clearly be remedied. So right now, the states and the federal government invested an enormous sum of money, particularly with HAVA funds available right now, in the procurement end of technology. Certainly states and the federal government could choose to spend their voting technology dollars in a different manner. They could choose to invest in open source strategy. They could choose to engage in some kind of pooled resources, as Joe talked about, this kind of community pool of building this technology. And perhaps instead of investing in purchasing, investing in designing software and doing some other kind of collaborative model. But I think that open source code isn’t necessary to gain transparency. There are other methods of gaining transparency. So part of the answer as to, should the state does this or not, depends upon what the state’s goal is in a much broader perspective than just on transparency.
MR. NEUMANN: I think your primary goal is to level the playing field.
SENATOR BOWEN: Well, the primary goal is transparency. But so far what we’ve seen from most vendors of proprietary code is, a refusal to release the code to the public and that really forces the decision. Because, if you have five proprietary vendors and none of them will release their code, and your only way to achieve transparency is with open source code.
MS. MULLIGAN: Well, we didn’t have as full a discussion as we might have of some of the benefits of open source code. So, clearly it aligns very well with the transparency goal—no doubt about that. Clearly, where you have enough eyeballs, there are, kind of, living, breathing, proof that we can develop secure systems. And I think it’s fairly clear right now that the development of secure systems, trusted systems, systems where worthy of trust, has been a problem in the voting technology area. And so you can say well, we should at least want to explore whether or not we can reap some of the benefits that seem if not inherent, then at least evident, in the open source development process.
I think it’s also very attractive for the intellectual property issues that you just raised, I mean, intellectual property has been raised as an objection to source code review, source code escrow requirements, independent code reviews, independent testing, testing of add-on products to address accessibility concerns, and a whole host of other obligations, where the state has a duty and desire to oversee this process and intellectual property concerns are being raised as a barrier. Clearly, open source code is going to remove those barriers. I would suggest that nondisclosure arrangements, you know, there are other ways that you can seek to mitigate the ways in which those intellectual property issues are frustrating oversight over our voting systems.
SENATOR BOWEN: And I think it’s really a philosophical matter. But for me, the idea of having a panel of experts reviewing the code, just doesn’t even start. I just will never…
MR. NEUMANN: Yes, I agree. It’s not enough—nowhere near enough. It’s too late.
MS. MULLIGAN: But they don’t have to be limited. I mean, you can have disclosed code models that allow anybody to look at the code. It’s doesn’t necessarily have to be limited to a pool of experts. I mean, there’s a whole range of models here.
SENATOR BOWEN: But disclosed code means you have to have vendors who are willing to disclose the code.
MS. MULLIGAN: Or it means that the states have to come together and say that that’s part and parcel of participating in this market.
SENATOR BOWEN: But we are about to spend a whole lot of money on voting systems now. And so I think we are at a critical time in this decision making because we know that whatever the state buys we tend to use for much longer than anybody said was the usable life of it. So if we purchase, if we spend a lot of money purchasing systems where the code….and I also want to….we’ll do the testing hearing next week, but there are concerns that even if you disclose the code, that you’re still going to have potential bugs and security issues that are just not going to be found in any certification.
MR. NEUMANN: My answer to that is, stick with the optical scan for the time being for those counties that are using it. It’s a perfectly adequate solution, and it certainly obviates the need to spend thousands of dollars on a bunch of machines that don’t have any transparency whatsoever.
SENATOR BOWEN: Right. And I think that’s the concern about going to a model that’s disclosed code. We have vendors who don’t want to do that. We have vendors where purchases are imminent. We have elections to run this year, where we don’t know what’s going to happen. And so there’s a real lot of pressure to buy equipment right now, and then we have many county officials saying, “Well, I don’t want to be out there spending $8 million on something that I have to put in a warehouse because the vendor won’t disclose the code.”
MR. NEUMANN: I think it’s fascinating that one of the vendors you invited today decided not to operate in North Carolina because North Carolina has required disclosure of code, and they don’t want to do that, and this is a company that has already had some of its code analyzed, and discovered that the crypto key was the same one that Doug Jones had found seven years ago and reported it to all the authorities, and that crypto key was still in the code.
SENATOR BOWEN: Right.
MR. HALL: And it’s important to notice that for your position, Senator Bowen, it’s a nonstarter to not have transparency and disclosed code. But for the vendor’s position, it’s nonstarter to have it. And so, there really needs to be some sort of compromise in the immediate term in order to be able to at least move forward and increase transparency.
SENATOR BOWEN: It seems like in the immediate term the solution is to vote on paper and use an optical scan system that doesn’t require an investment in a lot of equipment that we may or may not be able to use. And we already had a situation with some touch screen equipment, where they were used in one election and there were so many problems that they were immediately stuck in a warehouse, and where some of them are still gathering dust. So that’s not what we want to see as how we deal with taxpayer dollars.
All right.
MR. NEUMANN: Thank you.
JOE HALL:
Thank you.
SENATOR BOWEN: Let me ask for public comment at this point. We did invite all four vendors. I don’t think it’s surprising. We had Hart InterCivic decline on Thursday. Diebold declined yesterday. And despite repeated requests to ES&S and Sequoia, they have yet to respond to the committee with an answer about being here, although they’re not here, so I suppose that is an answer. I asked the vendors to attend because they’re in the business of making and selling voting machine equipment, which means they’re an important part of this discussion. And I am concerned about their stand.
And I do want to read one paragraph from a paper that the vendors trade association, the Information Technology Association of America presented yesterday. It says, and I quote: “Review by, or disclosure to, the general public will not improve the efficiency or effectiveness of voting system software inspection. Additional inspection review of code by technical laypersons with no ability to provide regulated feedback into the state election management process, is unlikely to improve the quality or security of the software.”
Now there are any number of reasons to be concerned about, or opposed, to the open source code concept, and I respect that, but to assert that open source means there’s no ability to provide regulated feedback is just not what anyone’s talking about here. And it appears to me at this point, that if we want to have voting machine vendors appear before this committee to talk about security issues, talk about disclosure, we’ll have to do that by subpoena. So the committee will evaluate that option.
Let me ask for public comment at this point. We have six people, I believe, starting with Kim Alexander. But, Jerry Berkman, you’re down here in the front, so why don’t you come on up and Kim can follow you, and then we’ll go to Ronald Crane, Jim March, Jon Barrilleaux, and Jim Super. Kim, welcome. Thank you.
Anyone who has written testimony is welcome to submit it to the committee. We will get it to the other committees and post it to our website. Let me apologize to people who are trying to use the committee’s website. It is not a model of what a website should look like. You have to go under the information tab on the committee website, or on the hearing tab for this, I think—information then hearings. We’re trying to get the things moved to the front page, but the wheels of disclosure and progress move slowly here. It is there. And if you have trouble, send an email to the committee and we’ll help you find the documentation.
Kim, welcome.
KIM ALEXANDER: Thank you, Senator Bowen, for holding this hearing today. I’m Kim Alexander, president of the California Voter Foundation online at calvoter.org.
My view about open source and voting systems is good but not good enough. There’s no guarantee that the code that’s been inspected is the same code that’s running everywhere. This is the chain of custody issue that’s important to flag as a security issue to consider in this.
Security in county election offices is not air tight, as Clark Kelso pointed out during his remarks. His comment was that we have poor network security. Every computer network to the internet in state government offices that creates all kinds of security risks that we have to take into consideration when we think about open source approach.
And as was also mentioned earlier today, patches are installed at times without authorization, or with limited scrutiny, so that also can negate the scrutiny that’s been placed into the open source software. If you study that software but then it can later be patched or amended without further scrutiny then….
SENATOR BOWEN: Although it seems to me that what your comments really get to my earlier question about how many systems we have. Because if you have a patch and you’ve got one system, you only have one thing to review if you have a late problem. If you have ten different systems and each vendor is submitting patches or changes, it’s a lot harder to get it done on a timely basis.
MS. ALEXANDER: Yes. I mean, whether you’re talking about open source or proprietary source, there’s no doubt that complexity is probably one of the greatest security weaknesses that we have in our current voting systems. I’m not a technologist myself, but in the technology world there’s what’s called the kiss rule (keep it simple stupid), and we have done anything but that with our voting systems. It is enormously complex.
So, we have to look at this issue of moving toward open source in the context of the reality of our voting systems today, which is that they’re being run in county government offices that themselves may not be secure or may have networks connected to the internet, create a whole host of new vulnerabilities and weaknesses that we have to consider.
And I think it’s important, and I think you’ve touched on this, that we need to avoid finding ourselves in the situation where we are asking the public to trust the experts. You should not need to be able to read computer code to have confidence in the integrity of the vote count. And this is why the paper trail is essential.
And Peter Neumann earlier today, in his remarks, talked about voter verified audit trail. I, and a lot of people who work on this issue, are very explicit about requiring a voter verified paper audit trail because we do not want to have to rely on computer experts to give us the confidence that we’re looking for in the accuracy of the vote counts. Having that paper trail and using it to publicly verify software vote counts provides broad public access to meaningful verification of the software’s performance. And open source software cannot do that. What it can do, is give more people a reasonable degree of confidence that the voting systems will work as expected. But open source software alone cannot verify the accuracy of the vote count; it cannot verify what actually happened on election day.
So that said, it is simple common sense that there is something intuitively wrong with using proprietary software to conduct our elections. Perhaps a private sector vendor will make a public source code voting system, and perhaps some counties will buy it, but if we really were to be serious about integrating public code into our voting systems, we would need to create our own voting system. We would need to invest public money into developing a new system from scratch.
And I was very
interested in what you said about the early systems and what was said earlier
in the hearing about the beginnings of
SENATOR BOWEN: Thank you. Yes, I don’t think anybody is talking about open source software as being a replacement for transparency.
MS. ALEXANDER: They actually do. There is often that discussion. I know you appreciate the distinction here, Senator Bowen, but I often hear people in my work saying, “Well, let’s just make everything open source and then we don’t need a paper trail.” So I’m constantly trying to address that issue, as well.
SENATOR BOWEN: Well, you know, just to put that one….I hope we can put that one permanently to rest. No matter what kind of machine or software you’re running, you have times when the power fails, the system crashes, somebody feeds it a bowl of oatmeal, God only knows what happens out there, or you just have a simple human error. You have somebody who sets a server in a precinct to hold 3,200 votes and 7,000 people vote there, and it doesn’t occur to anybody until later that, gee, the server only recorded the first, or the last, we’re not entirely sure which, 3,200 votes. So unless there’s a way to solve all human error problems in the use of computing, I do not see any way in which we can eliminate the paper from voting. Some things work and work well.
And you know if you look at redundancy in the ATM systems, and I’m going to go on a little soap box here, because people say to me, “Well, gee, you know, we rely on ATMs all the time.” And I will tell you that the software systems in ATM machines are so redundant, that if we all went downstairs and used the various ATMs here, and we had a terrorist attack on this building five minutes later, we probably wouldn’t be able to reconstruct anything on the legislative counsel’s data center; we wouldn’t be able to reconstruct any of my….we could get the Blackberry email, I bet, but I have no doubt that we would be able to reconstruct every transaction on the ATM machine, because the level of redundancy that’s there from the very first moment of the transaction, allows that. And we don’t have that with voting, and we’re not going to have it in the same way as we have it with ATMs because of the network and internet connection problems which are magnified by the fact that there’s no counterparty. There’s nobody on the other side of the voting transaction to say, “Wait a minute, my vote is not recorded properly.” And there is somebody on the other side of every financial, stock, proxy vote, all of the other kinds of things. Where we routinely use computers there is a counterparty. And while every problem may not be caught, over time, even the clever programmer who does the roundup fraud, where you basically roundup interest or round it down and take the fraction of a cent and put it into another bank, that gets caught over time because there’s a counterparty. There’s no counterparty; there’s nobody on the other side of a voting transaction. So I think the paper trail is here to stay. And certainly over my dead body will it disappear.
MS. ALEXANDER: Well, I’m glad to hear that. And I appreciate the fact that you and many others have made this distinction between the ATM transactions and the voting transactions. I think it’s a subtle but important point that voting is unlike any other transaction. And unlike the ATM transaction, the content of your transaction is secret. With the ATM transaction, your bank knows how much money you deposited or you think you deposited. With a voting transaction, ideally, the county election officials don’t know exactly how you voted.
SENATOR BOWEN: Not only does your bank know, but whoever it is that the central bank is that’s doing the processing, it may go through a federal reserve, I mean, there are all kinds of places where it might go. Any bank examiner has the ability to look at those records. They’re not secret. I mean, they are nondisclosed, but they’re not secret in the way that we demand that our votes be kept secret.
MS. ALEXANDER: And the federal regulations aren’t voluntary, I’m sure.
SENATOR BOWEN: No, the federal regulations of banks are not voluntary. That’s a good point.
MS. ALEXANDER: Thank you.
SENATOR BOWEN: Thank you. I bet you the banks wish they were.
Jerry.
JERRY BERKMAN: Hello. I’m Jerry Berkman
from
As for open source, I’ve worked extensively with open source, especially with SendMail, the program that’s used for 60 to 70 percent of the email. It operates server to server. It’s not a desktop client.
And you were asking before about why people contribute code. Well, in my case at one time, I don’t even remember what it was exactly, but something didn’t work in SendMail. And so since I had the source, I spent a few hours and figured out what it was that didn’t work, and it turned out that they had named a function VS print instead of VSN print. They were calling the wrong function, and so, I fixed that and I got my stuff to work, and I sent that to SendMail. And the next release that was put in and I’m listed as the contributor for adding one letter to the source. But that’s one of the motivations.
Another is the, you really want your program, the reputations of people, and they want to do good for the world or whatever, and people, there’s a core, you can look at the SendMail distribution notes and see hundreds or thousands of people have contributed to SendMail. And this was all online with the source, the release notes.
SendMail was started in 1981 by Erik Allman, and then it caught on. And it was just a hobby originally, and then eventually it became….he founded a company 17 years later. And they provide support, especially for large sites which want to have really secure and redundant systems. And I think a similar system would work well for elections. I can see an open source solution in several companies springing up to supply services to election….communities for their elections, helping them configure the system, etc. The advantage, of course, if you don’t like your supplier you can fire them and get a new one and they can use the same source.
There are a lot of problems with the proprietary systems. They include, being locked into one vendor, lack of confidence by the public in the system, lack of transparency, proprietary rights to data in addition to source, and fear.
I would say what
happened in
One compromise you could have instead of immediate open source is, the vendors currently restrict the data, and I just don’t understand how we allow that. They claim proprietary rights, a trade secret protection not only to the source, but to any and all data files created by their systems.
At the Voting
Systems Testing Summit in
The following are reasons we may want access to the files:
We may want to make overnight backups. Right now, if they have proprietary rights, I’m not even sure you can do that with the files without permission. We may want to get snapshots during the elections, just copying the file rather than having to print it out each time every 10 minutes on election eve. Use it in computer forensics. Modern high speed electric optical scanners save pictures of the ballots. These could be recounted, like with Hari Hursti’s software. You could have two counts of the same ballot image, ballot picture….actually, I don’t know. Am I talking all the things Jim is going to say, or what?
UNIDENTIFIED: There is breaking news on several fronts on what you’re talking about.
MR. BERKMAN: Okay. The ballot definition files, it would be nice if we could look at those and see if they’re correct. See generating reports that the vendor may not have provided for in their systems. So all of these things are reasons we may want access to the data. Not the source, but the data and the format of the data. And right now the vendors are unwilling to let us do that. This can get really ridiculous.
The Democrats in
Some additional notes I made today: Certain things would happen with open source. First of all, if we had open source I think IRV would happen in a few months. Somebody would just sit down and write the IRV components, as long as you got all the other components. It can’t be that hard.
Second, executable
on a memory card would never fly. Any security person that taught executable
would just throw up. There are 1,000 people in
I’m really sort of disturbed that we have to argue about legalities and the others.
Okay, I agree with the other speakers—we have to have openness throughout. The audit should include chain of custody, etc., etc., not just the ballot tally.
And also, I’d agree that even with open source you can’t guarantee what’s on the machine at the time of the election, so you need a lot more.
Thank you.
SENATOR BOWEN: Thank you. Ronald Crane.
RONALD CRANE: Thank you. Good afternoon. I’d like to address transparency in more depth. But first I’d like to just take a little sidelight on the question of whether vendors could make money on an open source or a disclosed source model. And basically there shouldn’t be any reason that vendors could not create a license where the sources open to inspection by anyone in the world, but only the vendor can produce a system that uses that source. So, anyone can review it, but the right to commercially exploit it is still proprietary.
But anyway, going on to transparency. I would say that open source, it’s a step forward from what we have, but it’s not what we need. Basically, it will not make elections secure and it won’t make them supervisable by ordinary citizens who don’t have special training. And I’m saying this as someone who does have special training. I’m a software engineer and a lawyer.
Elections, they’re not just another governmental service. They’re unique because they determine who governs. And if we transfer their supervision from the general public to vendors and officials and experts, as e-voting does…
SENATOR BOWEN: What do you mean, if?
MR. CRANE: Right. Well, we have. Exactly. What we’re doing is transferring the power to determine who takes office, and we can’t assume that that power is going to be exercised properly. Governments spend trillions of dollars every year, and those who determine who takes office determine in part how that money is spent. Do companies hire lobbyists just for fun because they pity them? No, they hire them to gain power. And as Jack Abramoff shows us, it’s not always done honestly. With these stakes we can expect honest elections only if we the citizens supervise them ourselves.
And I have a few other points that I’d like to go on.
Now let’s assume that a vendor’s voting program is honest, as open source seeks to guarantee and that someone has solved the chain of custody problem and it’s correctly installed in every machine. Even then a crooked vendor, or a few well placed crooked employees of an honest vendor, who doesn’t have sufficient internal controls, could cheat. How? They could just program the machines firmware to replace the voting application with one that cheats, and now we’re right back in the “we don’t know what’s going on in the machine” category.
Now, the secretary of state recognized this in his recent open source report. I believe it was page-8 when he said, “A thorough examination of the system for security vulnerabilities must include a perusal of each and every one of these components,” and the components he had listed previously were not just the voting application, but the operating system, the hardware, and the firmware. But most experts can’t, even most computer experts can’t, do that review, let alone ordinary citizens.
SENATOR BOWEN: That’s the testing hearing we’ll do next week is going to focus specifically on some of those challenges.
MR. CRANE: Yes, and here we are. We’re told to trust the vendors, trust the testers, trust the officials, trust, but not verify. It’s like Vegas, and someone brought this up a little earlier. I’d like to expand upon it. A gambler puts their chips in a gambling machine, she pulls the handle, and she gets a chance at a jackpot. A voter goes up to an electronic voting machine, selects her candidates, clicks cast vote, and gets a chance at having her vote counted. But the gambler, it sounds silly, but the gambler is actually better off. And the reason the gambler in Vegas is better off is, the Nevada Gaming Control Board which supervises those systems, and on a gambler’s complaint, or on its own, can go into a casino, rope off the machine, and rip it to shreds to determine what’s inside of it (the hardware, firmware, and software) and what it’s doing. There’s nothing remotely like this for voting machines. Again, it’s simply, trust the vendors, trust the testers, trust the officials, trust, but don’t verify.
Basically, the problem is that ordinary citizens can’t supervise these machines. Voters don’t check the paper trail sometimes. Officials don’t do recounts properly. Officials say that mismatches between the hand recount and the machine counts are glitches, or they don’t change the outcome.
And again, instead of investigating why there were any mismatches and recounting all the precincts, again it’s just trust. Trust but don’t verify.
Finally, to address something that Mr. Neumann brought up, the separation of machines into components that can be produced by different vendors and have open interfaces. I would point out that there’s actually a problem with that in terms of the portion of the system that solicits the votes from the voter, and that is that that portion of the system itself can defraud voters. And how it can do that is, undecided voters, basically. What it can do is make it more difficult for a voter to select a specific candidate or slate of candidates. This is not something that’s going to deter a voter who knows whom she’s going to vote for, but it is something that will change the votes of some voters who go into that voting booth saying, “I don’t know. I’m undecided.” Maybe that one didn’t register. No, that one. That’s the kind of fraud that even that advanced system, assuming that it’s implemented properly; assuming that the vendors who make it are not in collusion, would still not catch. And in fact, the symptoms of this kind of fraud, we don’t know whether it’s actually happening, but the symptoms of this kind of fraud were reported in 2004. A lot of voters nationwide reported what they called “vote jumping,” where they would try to vote for one candidate; they would press the button for one candidate, but it would register for a different candidate. Now we don’t know what’s going on there, but that’s exactly the problem—we’re still in a state of, we’ve got to trust the vendors, we’ve got to trust the testers, we’ve got to trust the officials.
I’m going to wind up.
Basically, I’ve written software for the intelligence agencies--______, actually, which Mr. Neumann may know of, headed by Steve Lipner over at Digital Equipment. And I’ve written software to protect nuclear power plants. I might conceivably be asked to become one of the experts whom the public would be forced to trust. And I say, don’t trust me—trust yourself and use a voting system that anyone can verify, whether it’s just an ordinary citizen or an expert. And that’s, hand marked paper ballots, counted by the public under public supervision in the places where they’re cast and non-computerized assistive devices for the disabled, like the VotePad that Yolo County just started using.
We’ve been told to trust, to trust the vendors, the testers, and the officials, to trust the experts—trust but don’t verify. It’s time to trust ourselves instead.
Thank you.
SENATOR BOWEN: Thank you. Jim March.
JIM MARCH: Hi. Jim March, representing Black Box Voting. It was my boss that came up with the term “black box voting machine,” in the book she started in 2002.
Everything that
we’re discussing here is about transparency. This isn’t about open source
versus anything else. Open source is actually a subset of something called
public source software. Where, the public at large can see the source code for
the program in question. Now whether that software is free or still sold as a
commercial product is a separate question. For example, Microsoft sells a
product called Windows CE Compact Edition (they’ve upgraded to something else
now), but CE, because it was not meant to be quite finished by Microsoft to
speed up the production cycle and make it available to hardware vendors, they
published the source code to it on the Microsoft website. Anybody in the world
could download it and look at it. Yet, to actually use it in a product you had
to pay $10 a seat licensing fee. So that kind of a model could still work for a
Diebold, Sequoia, or Hart, or whatever they want, if
they wanted to do that. In other cases, if this body mandates public publication
of source code, we’ll probably see system integrators that will provide a
complete package solution to counties that, in my experience in looking at
county IS departments, and especially county elections departments, they don’t
have the expertise on staff to integrate a solution the way
A lot has been discussed….well, first of all, I’d like to address two things my compatriot here mentioned. First of all, the Nevada Gaming Commission, there’s actually interesting interaction between them and voting systems.
In 2003, when the Diebold controversy first broke, the
On a more serious note, publication of source code, I believe, cross references, or should cross reference, two current California Election Code statutes that need to be at least subtly tweaked to meet the demands of open source.
One, a question keeps getting raised of, okay, it’s two months before the election, or two weeks, and we find a bug in the software and we find a big old hairy security hole. What do we do about it? Well, California Election Code 2300 establishes a civil right of the people to observe the counting of their vote. Election Code 15204 starts with the words “all proceedings at the central counting place, or counting places if applicable, shall be open to the view of the public.” And then it goes on to say “but no person except authorized shall touch any ballot container.” That’s fine. This needs to be read properly to allow enough oversight from the people to make sure that a particular known security hole doesn’t get used. So that’s a lot of oversight needed. That means if you’ve got a monitor at a screen tabulating the vote that’s 20 feet away from the public, you’ve got to split that video signal into a projector, just like this one, or a second monitor or two, and let people see those monitors. You’ve got to really tighten up on the oversight, because without it, you don’t have a trustworthy system.
Another key question is, well how do we make sure that the public source code published on the California secretary of state’s website is the same as what gets used out in the county? How do you make sure it’s the same stuff?
Well, Election Code 15004 says “the county central committee of each qualified political party may employ and have present at the central counting place or places not more than two qualified data processing specialists or engineers to check and review the preparation and operation of the tabulating devices, their programming testing, and have the specialists or engineers in attendance to any or all phases of the election. Now, with a little tweaking, that must be explicitly stated to be powerful enough for those engineers to make sure that that code is the legit stuff. Now you’re starting to get somewhere.
The other thing I would suggest in 15004 is, it’s now too limited to county political parties. It should include state political parties as well, number one, and I would argue it should be opened up to 501(c)(3) and 501(c)(4) nonprofit political organizations. And those kinds of eyeballs, looking at the machine at that level of detail….okay, now we’re starting to get somewhere. Now, this is all pretty complicated and it needs to be enforced against the counties.
The last thing I would suggest is, that public records, and these public oversight provisions where elections are concerned, if they’re violated by county election officials, they can be taken to court and made to pay penalties for it. There’s got to be a stick on top of a carrot on this one.
There are a lot of
states like,
One of the battles
going on is the fight over public records related to elections, and especially
the data files created by these computerized voting systems. Jerry mentioned a
big brawl going on in
SENATOR BOWEN: Let me just say that with a lot of these comments, we’re way off on open source software questions, but because we’re here, I’m going to go ahead and we’re going continue to take the testimony. You’re hear. I don’t want to make you come back and make these points.
MR. MARCH: This won’t take long. I understand. Well, it’s all tied in though. It’s all about transparency and it’s all….
SENATOR BOWEN: But I have to divide up the subject matter in order to make it manageable. I can’t eat the whole rhinoceros in one bite. I’m too small.
MR. MARCH: Well, okay.
UNIDENTIFIED: ________
SENATOR BOWEN: But there’s a dispute about that. But anyway, this hearing is on the open, next week we’ll do….we’ll do another hearing on auditing.
MR. MARCH: I know.
SENATOR BOWEN: So be not concerned. Thou shall have an
opportunity to criticize, to pull apart, to peel the onion and cry prolifically
over the state of elections in auditing in
MR. MARCH:
I understand. Okay. I’ll just point out that Diebold
appears to be capitulating in
I’ll close by saying this: You need transparency when things go wrong, not when things go right. It’s when things start coming unglued, when vendors sell software that was written by a multiply convicted criminal. You need to start looking at the software.
If you don’t have
transparency, democracy starts to crack. In 2004, we had some of the first
cracks in our system as the level of election related violence in vandalism
rose. In 2004, we had campaign headquarters on both sides raided right within
100 miles of where sitting. I believe it was the Democratic Party right down
here in downtown
Anyway, just the fact that we’re thinking about increasing transparency in this manner, the fact that somebody is at least listening, matters a lot. And I thank you very much, Senator Bowen, for holding this and the next hearing. And yes, I’ll be there.
SENATOR BOWEN: Thank you. I do think what we’re trying to get at is transparency—how do you change the numbers from the ____ poll that 48 percent of Americans had confidence that their vote was counted the way it was cast? We can’t have a democracy where that’s the result.
I have John Barrilleaux, Jim Super, and then Alan Deckert.
JON BARRILLEAUX: My name is Jon Barrilleaux.
I’m billing myself as a concerned citizen from
And to tie into what some of the other people were saying, as a small business I’m using open source software. I use Modzilla Suite, as well as Eclipse, for software development. And the interesting thing that I wanted to add was, on this particular contract that I with the government, the issue came up, because I had started an open source software project, and the question that came up with them as, “Well, gee, if we’re paying for it we want to own it.” So after some haggling over this thing and some discussions, they began to see the wisdom, and it has to do with lock-in. They realize it because I was such a small vendor that if they made it proprietary then I was going to be the only one that was going to be able to touch it. So they saw the wisdom of it and they decided to go ahead and require, as part of the contract, that the software I developed be open source.
And just one other point, and I guess other people have pointed this out in other ways, besides the whole security issue of open source, getting many eyes to see it, I think there’s just the whole issue of, gee, if a lot of people are looking at it, you can’t get away with sloppy engineering practices. And I think that, probably more than anything else, will help beef up this whole election system.
And so to close, I just want to say that as a user of technology, as a developer of open source software, this is a very, very good start. A very necessary part of trying to reform this overall election system.
Thank you.
SENATOR BOWEN: Thank you. Thanks for making the trip.
Jim, welcome back.
JIM SUPER:
(off mike) _______________. My name is Jim Super. I live _______
By the way, I’m a software programmer for over 20 years, and I was at one point a senior consultant at Digital Equipment Corporation back when it was the number two computer company in the world.
SENATOR BOWEN: But it didn’t have a 100-year product, I take it.
MR. SUPER: Some of it is still in use. Not 100 years, no. I stated this before, before the record I want to it state again. I sent an email to the secretary of state’s office asking, how do we know that the code that has been tested and reviewed nationally and state level is the code that is in the machine? And the response was, “We’re working on it.” This is important because we do not know what’s going on in those machines. So it really gets scary when people are working with closed source, where they can do whatever they want as easily as they want, and then put it on machines when they want, and we don’t know what’s going on. So we need to have open source software. We need to have a very tight chain of custody on that software to know that it’s in the machines, all the way down to election day. And, we need to have then paper audit trails and things like that. All of that tighten up afterwards. There are three major areas, the software, the procedures, and the audits afterwards.
A couple of technical comments here: There were some statements that with security issue, the code didn’t really matter whether it’s open or closed, and I beg to differ. It matters. Because, people even with quite a bit of security, they can penetrate into the code using things like viruses. This in my conversation with registrars. They’re very proud. And the registrars that I’ve talked to, they’re very proud of their staff and have a great deal of trust in them. But they don’t understand that somebody handling these thousands of electronic cards, can pass a virus all the way back into the tabulator and flip a vote. I mean, this is something that they don’t get. And it’s important that the code inside be very good.
Another point, there was talk about open interfaces and I’m going to try to use a slightly different example to make that more clear as to how open interfaces can be useful.
You have a machine like the AutoMark that prints a ballot. And then you take that ballot and you carry it over and you put it into a scanner. You could have two different systems, one from the AutoMark from ES&S print the ballot and your ballot then is part of the open interface. It follows a certain standard. And then you can carry it over and have a Diebold scanner read it, or somebody else’s scanner, or even better, have two different scanners from two different companies read it. You get more security that way. So when they’re talking about using open interfaces, it’s making the interchangeability easier and more reliable, because you can see what’s going on, at least in this case where the ballot _____ computer people we could see what’s going on and things become more obvious as to what’s happening.
Thinking about this from a business angle, I am convinced that one, the registrars and the people making the purchases want some kind of minimum reliability and assurance that the votes will count, but I think what’s really gotten them interested in machines is that, one, they were hoping for paperless. They wanted to get rid of the millions of pieces of paper flying around, and well, that didn’t happen. So, we’re going back to paper, and that’s good.
The other thing that they are looking for, almost more than machines, is the service components of these companies. That they want people to come in and manage these computers for them. And even if you go open source and the companies are not making money off their software, there is going to be a huge business on the service side. It’s a tough business, but it will be there. And that needs to be even helped along to make sure that that business appears, because otherwise, just having the software won’t work. You have to provide an entire solution. Along with the software, you’re going to have to make sure that there’s people providing the service.
Talking about these companies still: When the companies get locked into one vendor, or when the counties get locked into one vendor, it’s really dangerous for them from a business point of view, because the vendor sometimes when you want an improvement on your product like instant runoff voting in Alameda County, the vendor is going to come back and charge you up the yin/yang, to quote one vendor, or charge you a million dollars to have instant runoff voting. And you’re locked into them. And they talk about, well, stability and open source software is all sort of nebulous and that stability. Well, Diebold may close down its voting machine system business. This was announced a few days ago, that they’re thinking of closing it. This is not stability. So there’s no advantage there.
I’ll finish up with a couple more comments. One, closed source software, from what I’ve seen in the evidence, is sloppy—really sloppy. Because nobody outside is looking at it, they don’t have to worry about it. When you do open source, you get a little bit more pride involved and you try to do it right. Because you’re going to get criticized otherwise if you don’t do it right. So, there’s advantages to having all those eyes looking at it. Among other things, you’re going to do it the right way, and that’s good.
We have to have open elections. Openness is fundamental to the whole election process. I saw just a union election where they had county sheriffs accompanying the ballot boxes and staying in a certain room overnight, and there were both sides sitting outside that one room all night to make sure that nothing funny was going on. And then the next day they all brought it in the room and they held up the ballots one by one and showed the whole room. And this is just a union election. We have the presidency of the United States at stake at some times and we’re letting this go on with machines we have no idea what’s going on, and the comparison is scary.
And one final
note: In thinking about this business,
and you’ve heard about how quickly the open source software has spread
internationally. The next day it’s translated in French and German and a week
later it’s in 20 languages. This country is spending hundreds of billions of
dollars and thousands of lives to try to spread democracy. If
I thank you.
SENATOR BOWEN: Thank you. Alan Deckert, welcome back.
ALAN DECKERT: Thank you very much. I’m Alan Deckert. I’m president of the Open Voting Consortium. Sometimes we’re mischaracterized as the Open Source Consortium, and as if that’s all what we’re about. And of course, you made clear today that transparency in elections is more than open source and we definitely concur with that.
I coined the term “open voting” only to find….well, I coined that in 2003, only to find that it had been coined 10 years earlier by Irwin Mann in a paper called “Open Voting Systems.” And that definition is consistent with the definition that you are going for here. That is, that all aspects of election administration should be publicly observable. So we definitely are in support of that goal.
Our organization
sponsored a bill to get a resolution through the Legislature in 2004, asking
the secretary of state to investigate using open source software for elections
and issue a report on that. The due date was
The report claims that they have looked into this and this is what their findings are, but they don’t really tell us what it was. There is no data. There is nothing about their methodology for coming up with these conclusions. And the report, unfortunately, is inconclusive. It basically, you know, we asked the Legislature….the Legislature asked them to investigate it, and he came back and said, “Well, why don’t you do it?”
So, I do want to point out that Michael Shamus, who is one of the well known computer experts who has been a voting machine examiner, one of the highlights of this report is that he states unequivocally, all voting system software should be disclosed to the public.
We had some excellent testimony today and I thank you for pulling that together, especially Deirdre Mulligan, Peter Neumann. Great testimony. There was some equivocation about the level of disclosure of the source. And I concur with what you said, Senator Bowen, that what’s the reason for not making this publicly available to anybody that wants to see it. I see no reason.
And I think that we need to forge ahead on this. Joe Hall made some good points about barriers that we face. But the fact is, there have been no resources devoted to this particular endeavor for voting systems. I have, and my colleagues have, worked on proposals with the state legislature, with the former secretary of states, both Bill Jones and Kevin Shelley. The bottom line is, that no resources were ever devoted to really making this happen. And I think that those barriers that are preventing open source in elections, I think with some devotion of resources to this issue, the barriers will come down pretty quickly.
We spent five years since the election mess in 2000. And I’ve been lobbying this building since practically day that the vote was taken, or the decision was made, actually, in the Supreme Court. And we don’t have a more secure system. And part of the reason we don’t have a more secure system is, what exists in other states? And I think that we need to look at the role of California because California actually has one of the more secure voting systems in the country, but when it comes to a national election, the election result is as good as its weakest links and there are many weak links around the country, especially where we still have paperless voting.
I think it’s up to
I’m disappointed that the vendors did not show up for this today. I wanted to hear Hart InterCivic, for example. They complain about the prospect of having unfettered access to their source code claiming that it might lead to specious claims about their software. I would be very interested in to have them on the stand and ask them why they think that’s so.
I also want to mention that we are co-sponsoring a bill with Assemblywoman Jackie Goldberg that will, if signed into law, would require full public disclosure of the source code by 2007.
Thank you very much.
SENATOR BOWEN: Thank you. Jeffrey Jee, our final speaker for the day. You have the last word.
JEFFREY JEE: Hello. My name is Jeffrey Jee. I decided to add a few words ______ because I feel that there is a need to address this issue regarding voting. I hear a lot of rhetoric regarding voter fraud and how bad the Diebold machine is, and open source coding, and things like that, in the past six to eight months. I’ve attended a couple of meetings here and I’ve gotten a lot of information on this subject.
I believe that much of the rhetoric started in the belief by some that there was cheating in the elections and that our voting system needs to be open and fair and things like that so that it won’t happen. However, in my research of this subject after I got interested in it, I found that there are areas in California’s election system that I believe is deliberately setup for voter fraud, and this is done by the party that is yelling the most about voter fraud.
The two issues that I am looking at is, one, the lack of voter ID at the polling places. And two, the putting in a voter registration application into the DMV drivers license and then trying to pass a bill that will allow illegal aliens the right to drive and at the same time letting them have that application to fill out and fraudulently vote in our elections.
There have been
several mentions about situations where there has been a voter count that is
higher than what the registration is for that particular area, and I find that
is not amazing based upon what California has setup for itself. By not requiring a voter ID a person could
actually, or a person’s name, could actually be used for at least two votes,
and that in itself would double the amount of people that is registered in
I have talked to
several senators and assemblymen in
I just find it very disingenuous that there’s so much flack being brought up by supposedly by the Bush people cheating on the elections, and how we need to clean up the election voting system, and yet they’re allowing this particular item to stay available to them to cheat, and now they’re trying to open up a new one where they allow voter registration to be easily taken up by a voter that is not entitled to a vote in California.
SENATOR BOWEN: Mr. Jee, I know you’ve been here to talk about this issue before. I appreciate you coming back. This hearing is focused on open source software. But I just want to state that only eligible citizens should be allowed to vote, and no one should be able to vote more than once. That’s a fundamental portion of elections. But that’s not the topic of this hearing. So, if you have more to say about open source software, I’d be glad to hear it, but I’ve been trying to get people to focus on that issue.
MR. JEE: I do support whatever all these speakers are saying about open source voting software only because if it does the job, it would be great. However, since you are running for secretary of state and I believe voter fraud is the issue that is supposed to be important to you, I think that this is….I call your office many times and it seems like there’s not an interest in trying solve this part of the voter fraud issue.
SENATOR BOWEN: Actually, we have more prosecutions for
falsifying registrations than in any other part of the voter system in
But that is not
the issue here today, so unless you have something else to say about open
source….it’s 1:30. We’ve been here since a little after
MR. JEE: I’m sorry for taking up the time, but I think it’s an important issue. I would like to find out though if you would, in the future then, have any kind of hearing that would address these two important issues.
SENATOR BOWEN: Mr. Jee, I’m working my way through transparency and the integrity of the election process one segment at a time, and the requirement that we have a list of registered voters that deals with the dual possibility of somebody voting twice, is already there. There’s actually not much evidence to suggest that people go to vote twice. It would be far easier actually for somebody to vote absentee in someone else’s name because you don’t have to show up in person, which for many people is fairly scary to come into a polling place and think, gee, the neighbors might actually know that that’s not the person who’s signing in. So, I’m actually more concerned about….and HAVA, the list is to clean this up….the history of people who are no longer present with us on the planet voting, which was not uncommon in Illinois where I grew up. But again, that’s not the topic of this hearing. And we are going to close the hearing now.
Again, I will state the basic principles: Every citizen has a right to vote. No one has the right to vote more than once. And every vote should be counted as it was cast. And the open source software discussion is one portion of that.
I want to thank the witnesses, particularly who came from Berkeley, from SRI, for their time, and for contributing to the discussion, as well as those who came from the private sector to talk about their experience with open source software. Clearly we’ve got issues in the confidence of the count with 52 percent of Americans indicating that they are not confident that their vote was counted as it was cast. And the purpose of this committee’s hearings and our work is to work our way through to a situation where we have a different result when that question is asked.
So, thank you all for coming; for caring about your democracy; and this hearing is adjourned.
###