Eric Raymond has played a key role in the history of open source. In 1997, he published The Cathedral and the Bazaar (CatB), his seminal analysis of why the open-source development approach works so well. He was one of a group who came up with the term open source and, until 2005, was President of the Open Source Initiative (OSI), which he co-founded. After years as one of computing's most vocal and colorful characters, Raymond has been conspicuous by his absence recently. He tells Glyn Moody why, looks back over the ten years since CatB was published, and forward to the future of open source and its ideas.

GM: I can't quite remember the point when you were there and then you weren't, but things have been rather quiet from you recently on the open-source front. What have you been up to?

ESR: I've been keeping a low profile for a number of reasons, some of which I can't talk about but expect to go public with within the next few months. Among the reasons I can talk about it is that I made a strategic decision ten years ago not to try to be the indispensable man. That meant at some point I was going to have to step away from various leadership roles deliberately, and I did that (notably by resigning from the presidency of OSI).

I was a historian before I was an activist, and one of the things I've paid attention to are patterns of success and failure in reform movements. One of the things I've observed is movements that remain dependent on the talents and charisma of key individuals don't survive those individuals. In fact, if your charismatic founder doesn't step offstage, healthy reform movements often end up having to ritually execute, banish or disgrace him/her in order to build themselves properly into sustainable institutions.

So one of the things I've been up to is doing nothing, letting myself slip out of the public eye and watching the movement mature. I still do a lot of programming, as I always did. Two of my recent projects have been gpsd (a GPS monitoring dæmon) and Battle For Wesnoth (a rather spiffy fantasy combat game). I still occasionally talk to reporters, and I've done a fair amount of behind-the-scenes work that's important to the community, including some things I still have to be mysterious about for a bit.

GM: Do you have any plans to add your voice to the debate more frequently in the future?

ESR: I've un-stealthed a little in the last few months. I haven't yet decided how much more visible I want to get. In truth, I never really liked being famous, which is why I find it more entertaining than infuriating to be accused (as I sometimes am) of being a monster of ego or an attention junkie. Fame was never more than an instrument for me, and it's ridiculously easy to get; if I decide it's tactically necessary again, I'll just fire up the same techniques I used a decade ago and take it.

GM: Looking back over the ten years since CatB was published, what surprises you most about your analysis and its effects?

ESR: There was a day in early 1998 that Netscape open-sourced Mozilla, and I thought out the strategy I've been following ever since. Nothing since then has surprised me much. Details, yes, like IBM being the major Fortune 100 ally that stepped up, or the fact that our best shot at the end-user desktop came out of the brain of a former space tourist from South Africa.

But the broad trends, not. For example, I wrote in 1999 that I expected us to have basically won our technical argument for the superior quality of open source by about 2003, and that happened pretty much on schedule. Since then, the arguments you hear against it are mostly about transition costs and lack of a road map. That's a heck of a victory.

The reason I haven't been surprised much is that I understand the rise of open source as being driven by long-term trends that go much deeper than hacker ideology or intercorporate rivalries or any of the other colorful stuff that tends to preoccupy people. Even the dot-com boom and the following bust didn't perturb those trend-lines noticeably.

The deep drivers come from things like scaling laws, the way that various different costs and payoff functions in software design shift in response to hardware getting ever less expensive. Linus Torvalds and RMS and I are chips on that wave; we didn't create it, we rode it.

Here's an example of the sort of thing I mean: the Vista flop. Completely predictable, didn't surprise me for a nanosecond, and not because I think Microsoft is staffed by incompetents either. It's not; it hires some of the brightest programmers in the world. But, as I've been explaining for ten years, there's a scale regime above which closed-source development is unsustainable as the ratio between productive work and complexity-management overhead rises. Microsoft was bound to reach it; the only question was when.

GM: What do you think are the most important new lessons we have learned in the last ten years about how open source works?

ESR: I'm not sure we've learned anything fundamental, which actually disappoints me somewhat—I expected my original theoretical models and language to have been thoroughly superseded by now with better ones, and they haven't been. Actually, that might count as my biggest surprise since 1998.

That being said, there have been a lot of evolutionary developments that really matter. Our technical toolkits have improved markedly, and so has individual and social consciousness about how to use them. One good example is the rise of project workbenches like SourceForge and Gna and Berlios; another is the shift now underway from centralized to decentralized version-control systems; and a third is the clever ways development groups are finding to use IRC as a complement to and replacement of traditional e-mail channels.

GM: How do you think the open-source tribe differs today from the one you wrote about ten years ago, or the one that existed even further back? Do those differences matter?

ESR: Oh yes, they certainly do. The two big ones are that we're conscious and unified now in a way we weren't before. That sounds like a fuzzy feel-goodism, like political rhetoric, but it's actually a sharp fact with hard, measurable consequences, and it may actually be the most interesting thing I can talk about in this interview.

Before The Cathedral and the Bazaar, open-source development was a folk practice, a set of working methods evolved unconsciously by hackers who had no theory about why the things they were doing actually worked. It didn't have a name—and no, “free software” wasn't it, because that label was about ideology and goals rather than working methods and communications structures.

After The Cathedral and the Bazaar, we got conscious. We woke up. We started reflecting on what we were doing and deliberately trying to improve our process efficiency. One measurable consequence of that is the toolkit changes I just listed. These are things that happened because hackers reflected on the open-source process with the conscious intention of improving it.

The “unified” part—the Open Source community has a sense of itself now that has become steadily more important relative to membership in what used to be more loosely connected subtribes—Perl programmers, Emacs fans, Linux users, BSD users, pro- or anti-GPL zealots, whatever. While those affiliations are still important and spawn way too many flamewars, the overarching “open-source” affiliation is now much more important for almost everyone in those subtribes.

There are two ways to look at that shift. One truth is that I engineered it as part of my sinister master plan for world domination, bwahahaha. There are specific things I did, like founding OSI and some choices I made in my early propaganda and people I chose and groomed for leadership positions, that were designed to unify the community and did, in fact, have that effect.

One insignia of the success of my meme-hacking (insert more demented mad-scientist-type cackling here) is the effect on tribes like the BSD people and the X developers, who weren't part of the Linux community to which I addressed my original propaganda. They have not only enlisted themselves into today's Open Source community, they've also reinterpreted their own pre-1997 histories into open-source language and categories!

The equal and opposite truth is that this was bound to happen sooner or later; you can only sleepwalk for so long before you stumble over something and wake up—and the open-source process is not just more important than any of the specific tools or languages or operating systems we apply with it, it's obviously more important. So it's not surprising that people's allegiances and self-identifications have shifted; if I hadn't social-engineered that, someone else would have. Later and more slowly, maybe, but I'm certain it would have happened without me eventually.

Significantly, the only subtribe that has even tried to maintain a strong identity in opposition to open source is the more extreme wing of the FSF crowd—and that is precisely because they believe ideology and goals are more important than working methods.

For me, the working method is the ideology. Our practice is more powerful than our preaching. What really persuades are deeds and results. Or, as I've sometimes put it: talk is cheap, shut up and show them the code.

GM: One remarkable change in the last decade has been the rise of Google, a company built almost entirely on top of open-source software, which employs some of the top hackers and promotes the training of a new hacker generation through its Summer of Code. What do you think are the good and bad aspects of Google as far as open source is concerned?

ESR: I have the same concerns everyone else does. On the one hand, Google is an important patron of the community and has been on the right side of a lot of battles. On the other hand, the concentration of power it's acquiring is enough to make anyone uneasy, and I'm disturbed by the way the founders' political slant has sometimes compromised the neutrality of the channel—the generally leftish tilt of Google News' source selection, for example, or the fact that it won't let you sell firearms through Google Ads. And yes, I'd be just as unhappy if it were conservative dogma apparently manifesting, or even my own off-the-spectrum radical libertarianism. For a company in Google's position, any political partisanship outside narrow issues connected to free speech is an abuse of trust.

I'm optimistic, though. Markets correct this sort of thing. One consequence of the Internet is that they're doing it faster all the time, especially for information goods. Microsoft will probably have been the last of the long-lived information monopolies; if Google is seen to emulate, say, the New York Times' combination of left-wing partisanship with unconvincing denials of same, I have no doubt that we'll see some countervailing equivalent of Fox News spun up in fairly short order.

GM: The rise of Google has added to Microsoft's difficulties, since it must now fight on two fronts....

ESR: Exactly...and an important subtlety here is that the desktop-Linux and Google-Web-services attacks are different. They're not just in defense on two business fronts, they have to beat two strongly divergent technical approaches with one platform. Tough lines.

GM: Microsoft's response to open source has been highly schizophrenic. How do you expect all this to pan out: what will Microsoft do, and what will it become?

ESR: If I had good answers to that, I'd go play the stock market and be rich. I can see the underlying trends like scaling laws clearly, but surface phenomena, like the rise and fall of individual corporations, have too much noise and time-jitter in them.

The only thing I'm sure of is that Microsoft's days of being able to ship competitive software from closed source are numbered, let alone its days of maintaining monopoly lock-in. The Vista stall-out, and the scaling phenomena beneath it, guarantee that.

GM: What do you see as the big trends in open source at the moment?

ESR: My friend Rob Landley and I wrote a paper a year ago (“World Domination 201”, catb.org/~esr/writings/world-domination/world-domination-201.html), on how the transition to 64-bit hardware opened a critical window of opportunity for mass Linux adoption that is likely to close sometime in 2008. The two most interesting things to happen since are that 1) the hardware trend curves we looked at have been tracking our predictions like they were on rails, and 2) the two major opponents we were worried about have both been falling out.

Vista is a flop, and Apple is steaming away from the desktop market as fast as it can—it took “Computer” out of its corporate name a few months ago and now seems to want to be all about iPods and cell phones and media, oh my. Can't blame Apple; profit margins and volumes are both higher in those markets.

This means there's a huge Ubuntu-shaped power vacuum opening up in the desktop market. (A year ago, I thought it might be Linspire, but Linspire blew it big-time on several levels. One of those was soliciting strategic advice from me, swearing to follow it, and then doing the exact opposite.) If Mark Shuttleworth and his merry crew at Canonical don't blow it, we're going to win. And Ubuntu seems to me to be doing the right things, like hassle-free support for the evil proprietary multimedia codecs that nontechnical end users actually want.

Unfortunately, not blowing it requires not just doing the right things but doing them fast enough. Time is getting tight. We have at most another year before the market settles on the dominant desktop OS for the 64-bit era. That's not a lot of release cycles even at open-source speed.

We're so close. If we had even another nine months on the window, I don't think I'd be worried. As it is, it's going to be a damned near-run thing either way.

GM: As open-source commoditizes more and more of the software stack, do you think that free software will run out of big challenges and will eventually turn into a kind of mopping-up operation, filling in the holes?

ESR: No more so than software in general. One clue as to why is that “commoditizing” turns out to be an extremely misleading term for what's actually going on. I'm writing a paper on that; it's called “The Art of Commodity”. I think it may surprise people as much as my last one, “The Magic Cauldron”, did.

I'm not going to go into this in a lot of detail right now except to point you and your readers at the same source that started me thinking: Brent Williams' “Open Source Business Models: A Wall Street Look at a Wild 2006 and the Prospects for Even More Fun in 2007” (stephesblog.blogs.com/presentations/BrentWilliamsEclipseConV02.pdf).

This guy is a stock analyst who noticed something very interesting about the open-source software market, but lacked the tech background to understand the true significance of the anomaly he spotted.

GM: On a related note, do you think it is possible that the most exciting open-source projects will be those outside software—that is, the application of open-source ideas to content, business, science, politics and so forth? Do you see evidence of that happening already?

ESR: Well, there's the open-access movement in scientific publishing. The principals in that one are very explicit about their debt to the open-source movement in software, which kind of closes a circle, because one of my key insights was that open-source development harnesses the effects of decentralized peer review the way scientists are supposed to do it.

Politics: I recently read that New Zealand is inviting the public to rewrite a fundamental part of its legal code on a wiki. Business on content: the company that owns Dungeons & Dragons released its stuff under an open-content license a few years back. (The funny part about that is it credited me with inspiring the move but probably had no idea that I was a D&D player from waaaay back, like the 1974 first edition. I still have that original rule set in my basement.)

There are polyps of open-source thinking sprouting all over the place. But, most exciting? Not to me. I'm a programmer. Software is what I do, and the increasing pervasiveness of the Internet means that the effects of the software open-source movement will be felt everywhere.

There's also the problem that, unfortunately, most of the people who work themselves into a lather about applying open-source principles elsewhere are...well, let's be gentle and simply note that their idealism tends to greatly exceed their grasp of consequences.

But ask me again in another ten years; my answer might change.

GM: What do you think the open-source approach can offer to help solve the big challenges facing humanity and the planet today—things like climate change, sea acidification, water shortages, resource depletion and so on?

ESR: It's obvious. Planned economies, rigid authority hierarchies, closed and secretive decision procedures—they don't cope with situations of ambiguity, uncertainty and rapid changes in requirements well, if at all. They're maladaptive because their decision processes get captured by what political economists call “agency problems”.

Every single one of the problems you listed has market-based, decentralist, open-system solutions that are superior to anything a top-down planner or bureaucrat would ever come up with. The danger isn't that we'll fail to respond, it's that we'll get locked in to “solutions” that do more harm than good simply because they fit somebody's centralizing, authoritarian political agenda.

If you want a largest lesson from open source, here's mine: trust decentralization over centralization, voluntarism over coercion, bottom-up over top-down, adaptation over planning, openness over secrecy, practice over ideology, and markets over politics. Freedom works. Now go do it!