Recently, I was asked to reorganize some of our Web applications to improve their stability. The major push was to get each of our applications running in its own instance of Tomcat. These applications all are in various stages of development, and if a single instance allocates all of the memory available, the entire Tomcat server must be restarted. This, in turn, brings down all of the other applications.

An obstacle to running multiple instances is each instance would have to run on a unique port. Enter Apache's Web server. With the use of mod_jk, we were able to forward requests to specific hosts and contexts to the respective running instance of Tomcat.

Pulling all of this together was not the hardest thing I've done; however, it did prove challenging due to a lack of documentation. By no means, though, is this statement a stab at any of the development teams. In fact, the developers themselves helped me on numerous occasions when I found myself in over my head. When the documentation had me confused or I could not find exactly what I was looking for, I would head over to #tomcat or #apache on irc.freenode.org and pose my question there. Usually, I received a response within minutes.

The following software was used for the purposes of this article. See the resources section at the end of this article for download locations:

I am certain some of you are wondering why I used the Java software development kit (SDK) rather than the Java runtime environment (JRE). The answer is simple: Tomcat requires tools.jar to compile JSP pages, and tools.jar is provided in the SDK. If you do not wish to use the SDK, you need to place tools.jar in $CATALINA_HOME\common\lib.

For the purposes of this article, I assume that you already have Apache, Java and Tomcat installed. With all of the software listed above installed, except mod_jk, let's set up our environment.

In order for Tomcat to start, you need to set two environment variables, JAVA_HOME and CATALINA_HOME. JAVA_HOME should point to the J2sdk installation directory, and CATALINA_HOME should point to the installation directory for Tomcat. To make life easier, I have placed the following lines in /etc/bashrc:

export JAVA_HOME=/usr/java/j2sdk1.4.2_09
export CATALINA_HOME=/opt/tomcat 

Notice that I have set CATALINA_HOME to /opt/tomcat. Although this is the case, /opt/tomcat actually is a symlink to /opt/jakarta-tomcat-5.0.28. Although you can tell which version of Tomcat you have installed by running $CATALINA_HOME/bin/catalina.sh version, I prefer to be able to note the version immediately by looking at the installation directory.

Before we dive into compiling mod_jk, I would like to dispel some confusion I came across while working on this project. Specifically, the confusion centers around the myth that bigger is better. In the world of Tomcat connectors, there exists mod_jk and mod_jk2. In my mind mod_jk2 was the logical choice, as it had a higher version number. After a bit of probing and inquiring, however, I discovered that mod_jk2 is deprecated--it is no longer being developed. Unfortunately for me, I had invested several hours in getting mod_jk2 to work when I discovered this fact. Hopefully, you will have saved yourself some time by reading this article.

Open a terminal and change directories to the location where you have saved the mod_jk source. See the Resources section at the end of this article for the download link. In the terminal window, extract the source archive and install with the following four lines:

tar -xzvf jakarta-tomcat-connectors-1.2.14.1-src.tar.gz
cd jakarta-tomcat-connectors-1.2.14.1/jk/native
./configure --with-axps=/usr/sbin/axps 
make && make install

The --with-axps=/usr/sbin/apxs configuration option allows us to build Apache modules without requiring the Apache source. If all goes well, you should see a message near the end informing you of the location of the libraries. On my distribution, it is /usr/lib/httpd/modules.

With mod_jk installed, we must now configure Apache to load the module by editing httpd.conf. httpd.conf is located in /etc/httpd/conf on my system. Configuring Apache to load mod_jk is a simple two-line step:

#Load the mod_jk connector LoadModule jk_module
/usr/lib/httpd/modules/mod_jk.so

A mistake I initially made was naming the module mod_jk, as in LoadModule mod_jk /usr/lib/httpd/modules/mod_jk.so. The above command is certain to make Apache complain and refuse to start. With that said, it is a good idea to start or restart Apache now to verify that it can load the newly compiled module. If Apache starts without a problem, it is safe to continue.

Multiple Tomcat instances are possible to create with the use of the CATALINA_BASE environment variable. Each instance uses a common binary distribution but uses its own conf, webapps, temp, logs and work directories. Each instance also has its own JVM and, thereby, its own memory pool. If you have defined the maximum memory to be 512MB via JAVA_OPTS, each instance will attempt to allocate a maximum of 512MB.

Let's proceed now to set up these directories. As I mentioned before, Tomcat is installed in /opt/tomcat. To keep things somewhat organized, I created the following folders in /opt: /opt/tomcat_instance1, /opt/tomcat_instance2 and /opt/tomcat_instance3. It probably is more appropriate, however, to name these folders based on their purposes or applications. Remember that each of the three folders will contain conf, webapps, temp and work directories.

Tomcat uses a server.xml configuration file to determine the ports, connector engines and various other "server" configuration options. We are going to copy the installed server.xml from $CATALINA_HOME/conf/server.xml to /opt/tomcat_instance1/conf/server.xml. While we are at it, we might as well copy $CATALINA_HOME/con/server.xml to /opt/tomcat_instance2/conf/server.xml and /opt/tomcat_instance3conf/server.xml as well.

Tomcat also uses a global web.xml file. By global, I mean it is used for each instance. The web.xml file provides the default configuration for each Web application running under the given instance. If an option is not defined in the individual Web application, the default web.xml option is used. We can copy $CATALINA_HOME/conf/web.xml to /opt/tomcat_instance1/conf, /opt/tomcat_instance2/conf and /opt/tomcat_instance3/conf.

Now we must make some edits to server.xml. First, we need to disable the Coyote connector. To do this, we comment out the Coyote connector information. This is an XML file, so it uses the same comment syntax as HTML. After we are done commenting out the the connector, it should look something like this:

 <!-- 
  <Connector port="8080"
               maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
               enableLookups="false" redirectPort="8443" acceptCount="100"
               debug="0" connectionTimeout="20000"
               disableUploadTimeout="true" />
 -->

Because this is the first running instance, we do not need to modify any more of this file. For subsequent instances, we are required to change the shutdown port and the AJP connector port. The AJP connector port is the port that Apache uses to forward requests.

Next, copy the servlets-examples file provided with the installation of Tomcat from $CATALINA_HOME/webapps/servlets-examples to /opt/tomcat_instance1/webapps/servlets-examples. Again, copy the sample application to /opt/tomcat_instance2/webapps and /opt/tomcat_instance3/webapps as well. At this point, the set up of tomcat_instance1 is complete. We now need to set up the second and third instances before we pull it all together.

We already copied server.xml from the installation directory. We now need to make the same edit to /opt/tomcat_instance2/conf/server.xml as we did for the first instance. That is, comment out the Coyote connector exactly as we did above.

Additional required edits are to change the SHUTDOWN port from 8005 to 8105. We must change the port from 8005 because the first instance already is using it. You can change the second instance's port to be any unused port above 1024, but for simplicity and organization's sake, let's use 8105. Here is the line as it should be in the file:

<Server port="8105" shutdown="SHUTDOWN" debug="0">

Now we must change the AJP connector from 8009 to 8109. Again, this is required because the first instance already is using 8009. Below is the line to change with the required edit:

    <Connector port="8109"
	enableLookups="false" redirectPort="8443" debug="0"
      protocol="AJP/1.3" />

If you are using SSL, you also should change the redirectPort to the appropriate SSL port that Apache is listening on, normally 443.

We need to make the same edits to /opt/tomcat_instance3/conf/server.xml as we did for the second instance, except we substitute 8205 for 8005 and 8209 for 8009. In case this does not make sense, here are the respective sections of server.xml:

<Server port="8205" shutdown="SHUTDOWN" debug="0">

    <Connector port="8209"
	enableLookups="false" redirectPort="8443" debug="0"
      protocol="AJP/1.3" />

mod_jk uses a file named workers.properties. I recommend placing this file with the rest of your Apache configuration files. workers.properties is used to define where Apache looks for the Tomcat instances. Here, I cover only the items we are going to use for the three instances we have set up. Below is the workers.properties file we are going to use, followed by an explanation of the options:

worker.list=worker1,worker2,worker3
# Set properties for worker1
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
# Set properties for worker2
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=8109
# Set properties for worker3
worker.worker3.type=ajp13
worker.worker3.host=localhost
worker.worker3.port=8209

worker.list is a comma-separated list of worker names. You could have Tomcat workers defined later in the file that will not be used. Any worker defined is not used unless the worker is listed in the worker.list value. Workers are defined in format of worker.NAMEOFWORKER.type, with the value being the type of connector. All of our workers are of type ajp13. In the above example, we have defined three workers: worker1, worker2 and worker3.

You may have noticed the host portion of the configuration. This can be used to configure Apache to forward to Tomcat instances on separate machines. In fact, this is an option that one might choose to employ in order to make a site more secure. Because Tomcat and Apache both reside on the same machine, we use localhost.

Each worker also needs to define the port on which the connector is configured to work. If you remember, earlier we configured instance1 to listen on port 8009, instance2 to listen on port 8109 and instance3 to listen on port 8209.

To get all of this started, we need to tell Apache where to find the workers.properties file and where to log mod_jk requests. We also need to specify the format of the log files and the options specific to mod_jk. I did this by adding the following lines to httpd.conf; I placed all mod_jk configuration directives just before the virtual host declarations:

JkWorkersFile "/etc/httpd/conf/workers.properties"
JkLogFile     "/var/logs/www/mod_jk.log"
JkLogLevel  info
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions     +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat     "%w %V %T"

The above options tell Apache to use /etc/httpd/conf/workers.properties for the worker definitions and to use the /var/logs/www/mod_jk.log log file. If you are experiencing trouble with mod_jk, adjust the JkLogLevel to "debug" in order to get more verbose messages. JKLogStampFormat and JkRequestLogFormat define the logging formats. The option ForwardKeySize instructs mod_jk to forward the SSL key size along with the request. ForwardURICompat instructs mod_jk to forward the URL to Tomcat normally. -FowardDirectories instructs mod_jk not to return a directory listing from Tomcat.

We use domain1, domain2 and domain3 as our virtual hosts. To pull this off, we also have to edit /etc/hosts to ensure that domain1, domain2 and domain3 are resolved properly. To do this, we make three VirtualDirectory declarations, each corresponding to a worker defined in workers.properties. Below is the VirtualHosts section, followed by an explanation of the options.

<VirtualHost *:80>
    ServerName domain1
    JkMount  /servlets-examples/* worker1
</VirtualHost>
<VirtualHost *:80>
    ServerName domain2
    JkMount  /servlets-examples/* worker2
</VirtualHost>
<VirtualHost *:80>
    ServerName domain3
    JkMount  /servlets-examples/* worker3
</VirtualHost>

Now that we have most of the configuration complete, it is time to set up the instances to start at boot. This partially is done by creating a bash script and placing it in /etc/init.d. I used the start-up script found here.

#!/bin/bash
#
# tomcat        
#
# chkconfig: 
# description:  Start up the Tomcat servlet engine.

# Source function library.
. /etc/init.d/functions


RETVAL=$?
export CATALINA_BASE="/opt/tomcat_instance1"
export CATALINA_HOME="/opt/tomcat"

case "$1" in
 start)
        if [ -f $CATALINA_HOME/bin/startup.sh ];
          then
            echo $"Starting Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/startup.sh
        fi
        ;;
 stop)
        if [ -f $CATALINA_HOME/bin/shutdown.sh ];
          then
            echo $"Stopping Tomcat"
            /bin/su tomcat $CATALINA_HOME/bin/shutdown.sh
        fi
        ;;
 *)
        echo $"Usage: $0 {start|stop}"
        exit 1
        ;;
esac
exit $RETVAL

Copy the contents above and place them in /etc/init.d/tomcat_instance1, /etc/init.dtomcat_instance2 and /etc/init.d/tomcat_instance3. Be sure to change CATALINA_BASE to the appropriate directory for each script. Now, we need to link to these in /etc/rc5.d. To create the symlinks, issue the following as root:

cd /etc/rc5.d
ln -s /etc/init.d/tomcat_instance1  S71tomcat_service1
ln -s /etc/init.d/tomcat_instance2 S71tomcat_service2
ln -s /etc/init.d/tomcat_instance3 S71tomcat_service3

With all of our configuration complete, it now is time to test our setup. We begin by starting Apache or restarting if it already is running. Next, bring up the first instance:

/etc/init.d/httpd stop
/etc/init.d/httpd start
/etc/init.d/tomcat_service1 start

Now, open a browser window and go to http://domain1/servlets-examples/. If all goes well, you should see something similar to Figure 1.

Figure 1. Checking Your Setup

If you do not see a page similar to the above, look in the log files. Specifically, check /var/logs/httpd/mod_jk.log and /opt/tomcat_instance1/logs/catalina.out for any errors that may have occurred.

If everything looks correct, go ahead and start the remaining two contexts:

/etc/init.d/tomcat_instance2 start
/etc/init.d/tomcat_instance3 start

Point your browser to http://domain2/servlets-examples and http://domain3/servlets-examples. You should see the exact same page for all three instances.

To make things a bit more clear and to verify that we are hitting the correct instance, we can modify a line in /opt/tomcat_instance2/webapps/servlets-examples/index.html and /opt/tomcat_instance3/webapps/servlets-examples/index.html. The line to modify in both files is:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Servlet
Examples with Code</font></font></b>

For the second instance, we want the line to read as follows:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Domain 2 Servlet
Examples with Code</font></font></b>

Make a similar modification to the third instance:

<b><font face="Arial, Helvetica, sans-serif"><font size=+2>Domain 3 Servlet
Examples with Code</font></font></b>

If you now point your browser to http://domain2/servlets-examples and http://domain3/servlets-examples, you should see Domain 2 and Domain 3 at the beginning of the respective page.

Hopefully this HOWTO has given you a step-by-step understanding of what it takes to get multiple instances of Tomcat running when Apache is the front-end. Many more options are available that would produce similar setups. You could, for example, use Apache as a front-end and load balance between several servers running Tomcat. See the documentation for mod_jk for more information about the available options.

The Apache Jakarta Tomcat Connector

Jakarta-Tomcat Connectors Source Files

Apache Tomcat

J2SE

Apache