When I started to develop Web applications in the early 1990s, I sometimes found myself needing to store persistent data. At the time, the main options available were files—plain-text files, binary files or DBM (now Berkeley DB) files.

Fortunately, the languages I was using for Web development at the time (mainly Perl and Tcl) made it fairly easy to use any of those options. However, Web developers like myself quickly discovered the problems inherent in these technologies, particularly on large sites. The lack of intelligent, fine-grained locking meant that a popular application would be forced to wait before it could read from, or write to, an external file. Moreover, storing the data in this way prevented us from retrieving it intelligently or selectively. You could, of course, read the data into your program and produce your own reports, but each report required not just its own query, but its own program.

So, you can imagine how my life changed dramatically when I was introduced to relational database systems. Simply put, a relational database is built out of many two-dimensional tables, with each table containing one or more columns (the attributes) and one or more rows (the records). You could relate these tables to one another (thus, making the database “relational”) with foreign keys, such that a column in one table would represent a record in another table.

When you store data in a relational database, you suddenly are able to stop thinking in terms of how your data is stored and more in terms of how it is structured. Is the database server keeping information in binary format, in ASCII or in something altogether different? Who knows? And furthermore, who cares? So long as I can store and retrieve my data easily and efficiently, I'll treat the database as a black box, concentrating on writing my application.

SQL, the query language that works with relational databases, encourages this kind of thinking. When you use SQL to query a database, you describe the data you want to retrieve, rather than telling the computer how to retrieve it. True, now there are procedural extensions to SQL, and you also can write programs in non-SQL languages that manipulate the data in further ways. But, even they allow you to loop over, manipulate and perform calculations on the retrieved data. The data itself is still in the database, and you still describe how and what you want to retrieve in the declarative language of SQL.

Relational databases became popular because they offered so many advantages to programmers. They offered stability and security. They offered transactions, allowing programmers to ensure that a set of operations either all happened atomically or that they were rolled back to their state before the query began to execute. It was fairly easy to map many types of data onto two-dimensional tables. Databases gained all sorts of capabilities, including the ability to check the integrity of the data they are storing. And, they even gained the ability to distribute the load across multiple computers, albeit with some degree of difficulty.

Open-source relational databases, such as MySQL and PostgreSQL, began to challenge their commercial competitors. When I began writing this column in 1996, MySQL had not yet been released under an open-source license, and although PostgreSQL was fully open source, it had a number of limitations. So, it was expected that if you were interested in using a relational database, you almost certainly would fork over money for one, paying Oracle or one of its smaller competitors many thousands of dollars. Today, the opposite is the case. Open-source databases are more than adequate for Web applications. Moreover, it is basically expected that a Web application will use a relational database as a back end. If you say you are a Web developer, it is basically expected that you have some experience with SQL, as well as the care and feeling of a relational database.

I've been using relational databases for years, and I'm quite comfortable with them. Indeed, as I've sometimes told my clients, I see the world through two-dimensional tables. Breaking problems down into tables and the relations among them is something I do nearly every day. Yet, even I recognize that a database is a tool, and that not every tool is appropriate for every task. Saying every Web application should use a relational database is like saying all programs should be written in C (or whatever your favorite programming language is at the moment). As the old adage goes, if your only tool is a hammer, every problem looks like a nail.

And, there are indeed some issues with relational databases that have come to the foreground during the past few years. Perhaps the biggest problem many developers face is the “impedance mismatch” between objects, with which we enjoy writing our programs, and tables, in which relational databases function. If you want to store your object in a database, you have several options, none of which are particularly good, and all of which lose some of the semantic value that is the whole point of objects. True, you can treat each object attribute as a column, each instance as a row and each class as a table, but many corner cases don't map well to this, even with an excellent object-relational mapper (ORM) library. One of the reasons I enjoy working with Ruby on Rails is the large degree to which ActiveRecord is able to paper over these differences, but even with all of ActiveRecord's smarts, some problems can't be solved easily with objects, where it's required to massage my data model or even to drop down into SQL.

This problem has become bigger in the past few years, with the growth of documents and objects that contain many different types of data, some of which are unpredictable or contain highly variable content. In response to these needs, database servers have begun to incorporate functions that previously were considered to be outside the database domain, such as full-text search (which is extremely useful) and XML handling (which I still find puzzling and somewhat inappropriate). Regardless of your attitude toward such features, it does seem a bit strange to set up a database server just so you can store and retrieve files. There must be better ways to store such files, without all the administrative and computational overhead associated with a database server.

Another problem has to do with the greater and greater performance requirements of Web applications. I tend to dismiss talk of “scalable” languages and frameworks, in part because I believe that any language or framework can scale, given sufficient hardware. Instead of worrying about the scalability of a particular language, I would concentrate on the scalability of an architecture—and that is certainly a weak point for relational database servers. Modern Web applications using a “share-nothing” approach easily can scale up to hundreds or even thousands of Web servers, each handling part of the load. But if each of those servers needs to communicate with a database server on the back end, the database might become overwhelmed at some point. There are solutions to this problem, even among the open-source databases, but they are far from dead simple to install and configure.

The rise of increasingly variable document types, combined with the scalability issues associated with Web applications, has led many in the Web developer community to push for NoSQL, a variety of solutions that are united only in their rejection of the relational database model. The term itself was coined in 2009 by Eric Evans, who wanted to describe the collection of non-relational tools that were rising in popularity. As I write this in early 2010, we are seeing a veritable NoSQL revolution, with a number of prominent Web developers and sites switching away from relational database servers to other systems.

The reasons for the switch to these various NoSQL systems vary with the site and developer, but they echo the reasons that I outlined above: a good match with the objects used to program the application, the flexibility that a document-oriented database provides and the easier replication that a NoSQL system offers. These systems are far from identical in approach, communication or performance, but they each have their own backers and are released under an open-source license. Besides, many of these systems are currently in use on high-performance Web sites, such as Facebook and LinkedIn, so if your site is smaller than those, the odds are good that these solutions will work for you too. Another advantage is that the most popular of these systems offer a number of language bindings, such that it doesn't matter whether you're using Ruby, Python, Java or PHP.

The names I hear most often when discussing NoSQL solutions are CouchDB and MongoDB, with others (such as Tokyo Tyrant) gaining a great deal of mention. CouchDB came to prominence as much for its implementation in Erlang and its use of JSON as a query and response format, as for its database capabilities. CouchDB allows you to use the famous map-reduce algorithm on your database, write functions in JavaScript and send queries over HTTP using REST. MongoDB, by contrast, is written in C++ and uses its own object-oriented system (BSON) to handle queries. I'm still mulling over the differences between the two and trying to understand where each of them might have an advantage.

There is no doubt in my mind that these sorts of NoSQL solutions might be very useful and do indeed solve many problems. But, a large part of me wonders if we really should be throwing away 30 years of accumulated knowledge about relational databases. Moreover, relational databases were designed to ensure data integrity, not just the fast storage and retrieval of data. And, although these modern NoSQL solutions might indeed be fast and flexible, their lack of integrity checking does worry me somewhat. Maybe I'm old-fashioned, but I like the fact that I can define a relational database table to be NOT NULL or UNIQUE and, thus, know that the data contained within exists, is unique or both. I like knowing that each record in my database has a unique identifier. I like being able to traverse foreign keys, such that when one record points to another, I know there will be something on the other end, rather than the database equivalent of a null pointer. I'm also curious to see how such databases avoid duplication of data, or what the NoSQL equivalent of “normalization” might be.

So, I believe the people who think that NoSQL is a complete solution for all application storage needs are exaggerating quite a bit and are ignoring a great deal of wisdom associated with relational databases. Databases work so well, and are so easy to work with nowadays, that it seems a shame to put them out to pasture because they are having trouble handling loads of the top tier of Web applications.

Indeed, there are a variety of problems for which NoSQL offerings seem to have no solution, at least for now. They don't pay any attention to data integrity, be it a lack of null data or ensuring that only certain values will be allowed to be stored. They don't offer any way of associating objects to one another on multiple dimensions, which I find to be one of the most attractive elements of the relational model. And the query languages are far from standardized, meaning that moving from one NoSQL solution to another requires translating the query from one language to another.

That said, there is a growing class of Web applications for which relational databases are a poor fit and for which a document-oriented approach might indeed be superior. My dissertation software, which works with a large number of text documents, might well be a good example of a program that would benefit from a document-oriented approach, and I already have begun to explore such alternatives. However, even if I switch part of my software to use a NoSQL solution, it will be for a portion of the data storage, not for the entirety. In this way, I hope to benefit from the best of both worlds, using a relational database where it makes sense and a document database where it provides the superior solution.

My feeling is that the NoSQL movement, at least in the image its name projects, is a bit extreme and fails to appreciate the advantages and sophistication of many relational solutions. I would prefer a more nuanced approach, which might be called SomeSQL, in which non-relational databases are seen as additional tools, which can (hopefully) be integrated into a larger data-storage solution. After all, memcached and relational databases have managed to live in harmony for a number of years already. With a bit of planning, and with the right tools in place, I would argue that a combination of SQL and NoSQL could be extraordinarily powerful.

There is, of course, the chance that the NoSQL people are completely right, and that we are seeing the decline of one technology in favor of another. Perhaps we're at the leading edge of a new revolution, the database equivalent of the growth of high-level “scripting” languages in the 1990s. But, I'd prefer to see this as a new technology that Web developers will have to fit into their toolboxes, and which they will need to understand when designing and writing new Web applications.

Over the coming months, I plan to explore some of the better-known NoSQL solutions. I will look at them from the perspective of a developer—what do they offer, and why would I want to use this over a relational database or its NoSQL rivals? I cannot yet say which of these technologies will be my favorite or which would be appropriate for your applications. But, I do believe that the SomeSQL approach is one that has the potential to improve our applications' flexibility and performance dramatically, at the expense of the data-storage simplicity to which we have grown accustomed during the past few decades.