Linux in Government: California Air Resources Board's Secrets Revealed
In a recent article, I criticized an evaluation of the Linux desktop made by a member of California's Air Resources Board (ARB). Knowing how public sector vendors work, I wanted to head off a possible sabotage of California's open-source initiative based on the evaluation of a non-enterprise Linux desktop. That prompted some communication between Jim Welty, the CIO of ARB, and myself, which resulted in a conference call with key members of his staff.
In speaking with Bill, I discovered a model state agency that has taken advantage of Linux and open-source software extensively for over a decade. The team believes ARB is first in the country in air quality management and first in the state in open-source IT solutions. When I first spoke to Welty, he immediately pointed out that his team is responsible for the agency's IT success. He points to Bill Fell, Harry Ng and Narci Gonzales as the proponents, visionaries and programmers who make open-source systems work at ARB.
The California ARB has documented both the effectiveness and the cost savings of open-source software, proving that the open-source model saves money; provides comparable or better performance than proprietary software; offers reliability, flexibility and freedom from licensing hassles and violations; and provides support options from a rich variety of suppliers and user groups. As Bill states, "Management tends to believe that not all great or elegant solutions, IT or otherwise, need to be expensive, must come pre-packaged or shrink-wrapped or include every bell and whistle. The goal is to facilitate and enhance individual productivity, albeit at a reasonable cost."
Bill also openly discusses having his agency be in control of the software. He states that open source provides his agency with control over upgrades and source code. He also believes that the agency's software allows users to access data without requiring them to stay current with proprietary solutions.
In the same breath, Bill begins discussing Metcalfe's law, the law developed by Robert Metcalfe of PARC, who also is known as the Father of the Ethernet. Metcalfe stated that the value of a system equals the square of its nodes. The ARB team sees that value because its Internet sites are organized and supported organically. The knowledge of the organization increases exponentially as everyone participates in sharing their knowledge.
At ARB, every employee can contribute to the Web sites every day. Bill's team calls the process organic because its sites refresh from the bottom up. Bill Fell says, "We've kept our model open: anyone can contribute. We have at least one page for every program, and we empower the program staff to work in [its] own best interest to keep the pages up-to-date."
ARB's first use of open-source products was to address the delivery of information. Welty describes the process like this:
This [made] sense, because the Internet was developed to support the exchange of information between disparate systems worldwide. We addressed the development of systems using open-source products, such as the LAMP suite.
We joined the Internet movement in 1991. Working with Teale Data Center, we built the Ethernet infrastructure needed to connect our air quality modelers with the San Diego SuperComputer. We also introduced Internet-based e-mail to the Board, using products such as Eudora and Pegasus. Those were primal days of the Internet. Some of you will recall using Internet search engines with names like Gopher, Archie, Veronica; names taken from Archie comic books.
Our World Wide Web services program began in 1994, when the Web sported only 50 servers. Today, there are over 35 million.
In 1995, ARB purchased a distribution of Red Hat for $50 to support proxy services to protect NT 3.51 servers falling prey to crackers. The agency subsequently implemented Linux to support its list serv program, FTP server, the network DNS and an Internet search engine.
Beginning in 2000, ARB began developing Web-based applications using Linux as the OS, Apache as the Web server and PHP as the scripting language. According to Billy, "Harry Ng has initiated nearly all OS programs using LAMP."
ARB migrated its NT Web servers to Linux in 2000. According to Bill, the net result of IT's efforts was measurable in cost savings. He also claims that the team greatly increased its understanding of Internet systems and benefited from inexpensive redundancy, systems reliability, freedom from vendor licensing strategies and increased control over operations. In a presentation he gave, Welty said, "Our experiences confirmed what the trade magazines had been saying about these open-source products."
In a case study titled "ARB's Open Source Initiatives, Bill writes:
I'd like to share our experience in migrating an application from a commercial platform to one based largely on open source products. In 1994, our emissions inventory systems group was developing an Oracle database on a local HP/UX server, and on a Sun Solaris server hosted at the Teale Data Center.
The application was being built using Oracle's PL/SQL along with the Oracle Developer suite of tools. The design called for users to telnet to the appropriate server to access character mode menus, data entry forms, and reports. By 1995, it appeared that the Web and Web browsers offered a better way to deliver applications to clients as well as provide information to the general public.
So, we bought Oracle's Web Application Server, and used it to build ARB's first Web-based application, replacing the telnet interface of the emission inventory system.
Unfortunately, we found that Oracle's early tools for developing Web applications had, as they say, "issues". Each new release added significantly different features. This led to stability problems: Staff had to relearn Oracle's ever-changing Web development tools and rewrite the applications. Costs were also becoming a factor: each new "bigger and improved" version of Oracle's Web server and tools became more expensive.
As server licensing costs increased, we found it more difficult to budget for second or third Web application servers; servers needed for testing, development, or fail over purposes. Using the Oracle Web Application Server also meant that the application server worked only with Oracle databases. ARB had other databases in use, like DB2, SQL Server, and Microsoft Access. Oracle s Web application server was inoperable with these other systems.
By 1999 we started looking for replacement software which would: 1) be cheaper to acquire and maintain, 2) provide greater stability, 3) provide better programming tools, and 4) provide greater interoperability. Since we had experience using Linux/Apache to support our proxy servers, ftp servers, DNS, and list servers, we looked for open source products to replace the Oracle Web Application Server.
We found that Linux and Apache met our performance requirements, again. As an aside, Apache software is now used on over 63% of all Web servers. We still lacked an application scripting language, however, one to handle transactions.
Actually, there were many open source scripting languages we could have used, like Perl, C, Python. We happened upon PHP. PHP is a language developed specifically for developing Web applications. It also has connectors to a wide variety of databases, including Oracle, DB2, and others.
PHP also integrated well with Linux and Apache, and contained many functions which simplified building Web applications. We've since found that the PHP user/developer community refines and incrementally adds functionality to the language on a regular basis. Our initial trials using Linux/Apache/PHP instead of Oracle's Web Application Server progressed smoothly.
There were some telecommunication issues, but we found user groups on the Internet to be a rich source of information and assistance in resolving them. In the end, while we kept the data base itself in Oracle, we replaced the other components of the emissions inventory application with the Linux, Apache, and PHP suite.
Our programmers were able to integrate the functions easily, and they were impressed with the breadth of functionality this combination provided for developing Web applications. The change to open source products was transparent to the users. And security was not an issue since the data base and web application servers remained behind our Fire Wall; all public inquiries to the system are handled by the Proxy server.
We estimate that our costs would be about $27,600: $1,600 for the OS; $20,000 for the Web App Server and $6,000 for the Development Software, for 6 programmers. In contrast, by using the open source products we only paid $59. And we have no ongoing required Annual Maintenance Costs.
Further, with the open source solution, we can replicate as many implementations of the system as necessary, not just for production and testing, but also for admin. training, spares for operational recovery, and so on, at no cost, and with no worries over licensing violations.
Since the migration to the Linux/Apache/PHP suite, we discovered MySQL (my sequel), an open source product for developing data bases. This product integrates well with the Linux/Apache/PHP suite. We've built several small to medium sized systems with MySQL and find that it easily competes with proprietary products on the market, like MS ACCESS, MS SQL, even Oracle in some cases. It handles JOINS quickly, as well as transactions and queries.
Here's something else I learned from Bill Welty: ARB has an outstanding management culture. None of the achievements of the IT group could exist without the environment. This is something from which other agencies across the globe can learn.
As Bill stated during our conversation, "ARB's IT program is conservative. The systems we develop must be useful, reliable, timely, accurate, yet reasonably easy to integrate with the outside world. Given the nature of today's budget, the systems also must be durable and efficient. Secondly, ARB's IT program has a high level of executive visibility. As CIO, I report directly to the executive office. Our Office of Information Security is centralized, as is our project management office. The executive office strongly supports effective project management for IT as well as for non-IT efforts."
Governor Schwarzenegger, I hope that you'll spend more time with Bill Welty and his group. I am convinced he can help you meet and exceed your goals of making California a better place to do business.
