Seven Criteria for Evaluating Open-Source Content Management Systems
Using a content management system (CMS) to manage a Web site used to be a luxury within the reach of only those who had the technical resources to build a custom solution or the financial resources to afford a proprietary one. In recent years, numerous pre-built open-source CMSes have emerged as viable alternatives to costly proprietary products or custom-built solutions.
Cost savings are not the only reason why open-source CMSes are gaining in popularity. Unlike proprietary products, the source code for open-source CMSes is freely available so it is possible to customize the CMS to match your project's requirements. Customized versions also can be redistributed according to the terms of their licenses. In addition, the benefit of free technical support provided by the Open Source community cannot be discounted.
Development and long-term maintenance costs of a custom-built CMS can be high. Keeping up with new requirements or security issues can take a lot of time and effort. Open-source CMSes combine the advantages of the proprietary and custom-built approaches--a pre-built CMS that is ready to use but can be customized if needed.
In his April 2005 column, Linux Journal columnist Reuven Lerner observed:
It used to be that you needed to search high and low for an open-source application that would suit your needs. Nowadays, it still takes time to find the right application, but that's because you need to sort through so many bad or inappropriate ones before finding the one that is right for you.
Choosing a pre-built CMS from the large number of alternatives can be an overwhelming task. They come in several shapes and sizes, employing different platforms, programming languages and content management approaches. Complicating the decision-making process, open-source CMS projects are at varying stages of development, from stable and reliable applications to cutting-edge experiments. How do you decide which one is right for you?
The CMS Matrix (see Resources) is a community-maintained list of over 300 CMSes, both proprietary and open source. This list may not be comprehensive, but it is a good place to start. This Web site makes it easy to review CMS details such as system requirements, availability of technical support, ease of use, performance, security, interoperability, flexibility and commerce. It also features a convenient matrix to conduct a side-by-side comparison of up to 10 CMSes at a time. But you still need to decide which ones to compare and evaluate. I use the following seven criteria to evaluate potential CMS candidates:
Web application platform
Software license
Stability and development activity
User community
Documentation and source code
Web standards, accessibility
Suitability and usability
Most CMSes use a database, at least one programming language and a Web server. If you have experience with a particular programming language, database or Web server, you might want to consider an open-source CMS that uses those components. If you plan to customize the CMS, familiarity with the various components used by the CMS is especially beneficial.
If you do not have a preference or technical limitations, consider CMSes built using the popular open-source LAMP platform. The first three letters in LAMP stands for the GNU/Linux operating system, the Apache Web server and the MySQL database. In addition, although it does not fit neatly into the LAMP acronym, the open-source database PostgreSQL also is used in this combination. The P in LAMP stands for a programming/scripting language; the most common choices are Perl, Python and PHP. The LAMP platform is stable and regarded as "a mature alternative that has a lot to offer, particularly for projects on a tight budget" (McAllister, 2005). It also is easy to find Web hosting services that provide LAMP hosting.
The flexibility offered by a CMS that can run on multiple operating systems is an advantage, because it doesn't tie down your content management solution to a single operating system. Open-source CMSes built on the LAMP platform often run on operating systems other than GNU/Linux. Some even allow you to use alternate databases. Check which operating systems and databases are supported by the CMS you are evaluating.
The key distinguishing feature and value of open-source software is the license under which it is offered. It is important to check the license of the CMS you are considering, particularly if you are going to modify the software and plan to redistribute it. Simply because it claims to be open may not mean the CMS offers the benefits and protections that you might be expecting. The Open Source Initiative has a list of approved licenses. The Free Software Foundation (FSF) also provides a list of licenses along with a commentary. Check the license of the CMS you are considering against both of these resources. One license that you are most likely to encounter is the GNU General Public License (GPL). Invest the time to understand the GNU GPL, as it is used by many of the leading open-source CMS projects.
Open-source CMSes frequently start out as an experiment by a single programmer. Promising CMSes may attract other programmers, while less interesting projects are abandoned. CMSes that continue to grow may evolve into mature applications. One indicator of a healthy open-source project is constant development activity. Check how often new versions have been released. Regular or frequent releases are a good sign.
Do not base your decision solely on the version number of the open-source CMS, however. Some projects tend to inflate version numbers while others are modest. For example, a CMS sporting a 5.3 version number may not be necessarily more stable than one whose latest release is 1.2. Look for the project's Changelog or Release Notes documents. These usually include a list of new features, fixed bigs, improvements and known issues for each release. Scanning through this list may give you an idea of the pace of development activity and the project's stability.
Don't be discouraged by open-source projects that display a list of software bugs on their Web site. Good open-source projects acknowledge bugs and work openly to fix them. Projects that work quickly to address reported bugs are likely to be a good choice. Some may use bug-tracking software that allows developers and users to track the status of each bug. Try to get a sense of how long bugs stay open before they are addressed by looking at recently closed bugs. Also, assess if known issues or currently open bugs may make the CMS unsuitable for your needs. For a closer look at the development activity behind an open-source CMS project, take a look at the development forums or e-mail list archives, if they are available. This level of transparency is a strength of the open-source development process and enables you to make more informed decisions.
Repositories of open-source software try to make it easier to evaluate an open-source project by displaying indicators of its development activity and popularity. Figure 1 shows statistics compiled by the Freshmeat.net repository for Moodle, an open-source CMS. It includes scores and rankings by popularity, vitality and user ratings. Figure 2 shows similar statistics compiled by the development Web site Sourceforge.net, for the Plone CMS. Sourceforge.net also provides an activity percentile along with projects statistics that include downloads, page views, bugs, patches and project rank.
Figure 1. Statistics for the Moodle CMS
Figure 2. Statistics for the Plone CMS
The size and involvement of the user community is a good indicator of interest in the CMS. Look for active user discussion forums and/or mailing lists. Some projects proudly display a list of Web sites that are using its CMS. Visiting some of these Web sites can give you can idea of how the CMS is being used and how it can adapted.
The functionality of some open-source CMSes can be extended with the help of plugins. Others offer the ability to change presentations and appearances easily by using different themes. A large number of plugins and themes created by the user community demonstrates how easy it is to the customize the CMS. Also, scan user discussion forums and mailing lists to get a sense of how the developers respond to users' concerns and suggestions. Some projects also may offer user-contributed tips and tutorials. All are indications of an involved user community that may be able to help you. As open-source offerings have matured, commercial support for open-source CMS also is becoming available. Search for consultants that may be able to provide commercial technical support if you need it.
Successful open-source CMSes always are evolving, and it is a challenge to keep documentation up-to-date and useful. The Open Source user community often lends a helping hand in creating and updating documentation. Well-developed and current documentation would indicate an organized open-source CMS with an active community. Documentation may come in many forms; review changelogs, release notes, installation instructions and user manuals. Does the documentation address your concerns and questions?
Check if the open-source CMS has a developers' guide, coding standards and information on how others can contribute to the project. Established coding standards can help maintain consistent code even when a large number of programmers contribute to its development. Even if you do not have any programming experience, it may help to glance at the source code for the open-source CMS. Look for comments within the code written in English. Well-commented code can be easier to maintain and customize. If you are familiar with the programming language, you may be able to gain valuable insights into how the CMS works and the quality of its code.
Ignored in the past, compliance with the Web standards set out by the World Wide Web Consortium (W3C) and other standards bodies is an important requirement for any software that generates content for the Web. Many open-source CMSes are committed to Web standards. But you don't have to take their word for it. With a quick visit to W3C's Markup Validation Service you can check for yourself if the CMS you are evaluating generates valid HTML or XHTML.
The W3C's Web Accessibility Initiative (WAI) has published Web Accessibility Guidelines to ensure that Web content is accessible to people with disabilities. Also, according to Section 508 of the Rehabilitation Act of 1998, Web sites of organizations that receive US federal funding must be accessible to individuals with disabilities. Check if the CMS you are considering implementing these guidelines. Although a complete Web accessibility assessment should be conducted by an expert, you can conduct a preliminary assessment using an on-line accessibility validator such as the Cynthia Says portal, which checks a Web site's pages against both Section 508 and the WAI guidelines.
List the features you need and how critical these features are to ensuring that your content management project meets its goals. Then, try to gauge if the CMSes you are evaluating meet the project requirements in part or in full. Also, how easily can it be adapted or customized to fulfill project objectives?
CMS-based Web sites can be complex to use. Web usability is defined as the quality that makes a Web site easy to use. Assessing and improving Web site usability is a task for usability experts, but you can conduct a informal usability assessment by gaining familiarity with the CMS you are evaluating. Open-source CMSes make this convenient because you can download and try them out without any restrictions. The Web site Open Source CMS make this even easier by providing complete installations of over a hundred open-source CMSes that you can try out (see Resources).
If you choose wisely, an open-source CMS can provide a stable, flexible and cost-effective system that is well-suited for your content management needs. More importantly, open-source CMSes give you the freedom to stay in control of your content management solution.
Free Software Foundation: various licenses and comments about them.
Free Software Foundation: the GNU general public license.
"Column 100", Reuven Lerner, Linux Journal, April 2004.
"Shining a Light on the Open Source Stack", Neil McAllister, Infoworld, April 4, 2005.
Open Source Initiative, the approved licenses.
"Content Management", Reuven Lerner, Linux Journal, April 2003.
Abhijeet Chavan is the Chief Technology Officer of Urban Insight, Inc., a Web development consulting firm. He also is the co-founder and co-editor of Planetizen.
