Organizations supporting Linux operating systems commonly have a need to build customized software to add or replace packages on production systems. This need comes from timing and policy differences between customers and the upstream distribution maintainers. In practice, bugs and security concerns reported by customers will be prioritized to appropriate levels for the distribution maintainers who are trying to support all their customers. This means that customers often need to support patches to fill the gap, especially for unique needs, until distribution maintainers resolve the bugs. </EndSoapBox>

Customers who desire to fill the support gap internally should choose tools that the distribution maintainers use to build packages whenever possible. However, third-party software packages often present challenges to integrate them into the distribution properly. Often these packages do not follow packaging guidelines and, as a result, do not support all distribution configurations or procedures for administration. These packages often require more generic processes to resolve the improper packaging. </EndSoapBoxAgain>

From this point on, the tools and methods discussed in this article are specific to Red Hat Enterprise Linux (RHEL). These tools and methods also work with derivative distributions like Scientific Linux or Community Enterprise OS (CentOS). Some of the tools do include support for distributions based on Debian. However, specifics on implementation of the process focus on integration with RHEL-based systems.

The build phase of the process (described in "A Process for Managing and Customizing HPC Operating Systems" in the April 2014 issue of LJ) requires three pieces of software that can be filled by Trac, Cobbler and Jenkins. However, these pieces of software do not fill all the gaps present from downloading source code to creation of the overlay repository. Further tools and processes are gained by analysis of the upstream distribution's package management process and guidelines.

The application of the Fedora Packaging Guidelines and its counterpart EPEL Packaging Guidelines are good references for how to package software for RHEL-based systems appropriately. These guidelines call out specifics that often are overlooked by first-time packagers. Also, tools used in the process, such as Mock, work well with the software mentioned previously.

Fedora uses other tools to manage building packages and repositories. These tools are very specific to Fedora packaging needs and are not general enough for use in our organization. This is primarily due to technical reasons and features that I go into in the Jenkins section of the article.

The rest of this article focuses on implementing Trac, Cobbler, Jenkins, and the gaps between the three systems. Some of the gaps are filled using native plugins associated with the three systems. However, others are left to be implemented using scripts and processes requiring human interactions. There are points where human interaction is required to facilitate communication between groups, and other points are where the process is missing a well implemented piece of software. I discuss setup, configuration and integration of Trac, Cobbler and Jenkins, along with some requests for community support.

Trac

Trac consists of an issue-tracking system and wiki environment to support software development projects. However, Trac also works well for supporting the maintenance of administrative processes and managing change on production systems. I'm going to discuss the mapping to apply a software development process to the process by which one administers a production system.

I realize that talking about issue tracking and wiki software is a religious topic for some. Everyone has their favorite software, and these two kinds of systems have more than enough open-source options out there from which people can choose. I want to focus on the features that we have found useful at EMSL to support our HPC system and how we use them.

The ticket-tracking system works well for managing small changes on production systems. These small changes may include individual critical updates, configuration changes and requests from users. The purpose of these tickets is to record relevant technical information about the changes for administrators as well as management. This helps all stakeholders understand the cost and priority of the change. These small changes can be aggregated into milestones, which correspond to outage dates. This provides a starting framework to track what change happens and when on production systems.

Trac's wiki has features that are required for the process. The first is the ability to maintain a history of changes to individual pages. This is ideal for storing documents and procedures. Another feature is the ability to reference milestones from within pages. This feature is extremely useful, since by entering a single line in the wiki, it displays all tickets associated with the milestone in one simple line. These two features help maintain the procedures and outage pages in the wiki.

The administrative procedures are documented in the wiki, and they include but are not limited to software configuration, startup, shutdown and re-install. The time required to perform these administrative procedures also should be noted in the page. We also make sure to use the plain-text options for specifying commands that need to be run, as other fonts may confuse readers. In many cases, we have specified the specific command to run in these procedures. For complex systems, creating multiple pages for a particular procedure is prudent. However, cross links between pages should be added to note when one part of the procedure from each page should be followed.

Trac's plugin infrastructure does not have plugins to Jenkins or Cobbler. However, what would be the point of a plugin going from Trac to continuous integration or provisioning? Most software development models keep ticket systems limited to human interaction between the issuer of the ticket and the people resolving it. Some exceptions are when tickets are considered resolved but are waiting for integration testing. Automated tests could be triggered by the ticketing system when the ticket's state is changed. However, mapping these sorts of features to administrative procedures for managing production systems do not apply.

Cobbler

Cobbler works well for synchronizing RPM-based repositories and using those repositories to deploy systems. The RPMs are synchronized daily from Jenkins and distribution maintainers. The other important feature is to exclude certain packages from being synchronized locally. These features provide a platform to deploy systems that have specific customized packages for use in the enterprise.

The initial setup for Cobbler is to copy the primary repositories for the distribution of your choice to "repos" in Cobbler. The included repositories from Scientific Linux are the base operating system, fastbugs and security. Other distributions have similar repository configurations (see the Repositories and Locations sidebar). The other repository to include is EPEL, as it contains Mock and other tools used to build RPMs. There are other repositories that individual organizations should look into, although these four repositories are all that is needed.

Repositories and Locations

• Extra Packages for Enterprise Linux: http://dl.fedoraproject.org/pub/epel/6/x86_64

• Scientific Linux 66 Base: http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/os

• Scientific Linux 6 Security: http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/updates/security

• Scientific Linux 6 Fastbugs: http://ftp1.scientificlinux.org/linux/scientific/6/x86_64/updates/fastbugs

• CentOS 6 Base: http://mirror.centos.org/centos/6/os/x86_64

• CentOS 6 FastTrack: http://mirror.centos.org/centos/6/fasttrack/x86_64

• CentOS 6 Updates: http://mirror.centos.org/centos/6/updates/x86_64

• RHEL 6 Server Base: rhel-x86_64-server-6 channel

• RHEL 6 Server FasTrack: rhel-x86_64-server-fastrack-6 channel

• RHEL 6 Server Optional: rhel-x86_64-server-optional-6 channel

• RHEL 6 Server Optional FasTrack: rhel-x86_64-server-optional-fastrack-6 channel

• RHEL 6 Server Supplementary: rhel-x86_64-server-supplementary-6 channel

The daily repositories either are downloaded from the Web on a daily basis or synchronized from the local filesystem. The daily repositories get the "keep updated" flag set, while the test and production repositories do not. For daily repositories that synchronize from a local filesystem, the "breed" should be set to rsync, while daily repositories that synchronize from the Web should set their "breed" to yum. This configuration, through experience, has been chosen because some RPMs do not upgrade well with new kernels nor do they have standard update processes normal to Red Hat or Fedora.

An example of a set of repositories would be as follows:

• phi-6-x86_64-daily — synchronizes automatically from the local filesystem using rsync once daily.

• epel-6-x86_64-daily — synchronizes automatically from the Web using reposync once daily.

• phi-6-x86_64-test — synchronizes manually from phi-6-x86_64-daily using rsync.

• epel-6-x86_64-test — synchronizes manually from epel-6-x86_64-daily using rsync.

• phi-6-x86_64-prod — synchronizes manually from phi-6-x86_64-test using rsync.

• epel-6-x86_64-prod — synchronizes manually from epel-6-x86_64-test using rsync.

To exclude critical packages from the upstream distribution, the "yum options" flags are set on the daily repository to remove them. For example, to exclude the kernel package from from being synchronized, add exclude=kernel*. It's important for administrators to consult both the Cobbler and yum.conf man pages to get the syntax right.

Setting up Cobbler in this way allows administrators to deploy systems using customized critical packages. Cobbler also is used in future phases where the repositories are used to deploy the test and production clusters. The repositories and their relationships are all Cobbler needs to support package building, the test cluster and the production cluster.

Jenkins

Jenkins is a very powerful continuous integration tool used in software development. However, from a system administration view, Jenkins is a mutant cron job on steroids. Jenkins handles periodic source code checkout from source code management (SCM) repositories and downloading of released source code, via HTTP or FTP. It then runs a series of generic jobs that build, test and deploy the resulting software. These generic interfaces work well for building and distributing RPMs to be included by Cobbler.

The use of Jenkins in a software development role is not all that different from building RPMs (see Table 1 for a comparison of the two processes). The first step in the two processes differs in that (hopefully) the software development code required for the build step is in one place. Package developers need to have, at a minimum, two locations to pull code from to continue with the build. The first location is for patches and spec files, normally kept in an SCM. The second is for released source code packages. Source code is released in a single file and usually in some container format (such as tar, rar or zip). These files do not normally belong in an SCM and are more suited to an S3 (http://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html), swift (http://docs.openstack.org/api/openstack-object-storage/1.0/content) or blob store-like interface.

Table 1. Packaging vs. Development

Software Development	RPM Packaging
Download source code from SCM.	Download released source, spec file and patches.
Run the build process.	Build the RPMs using Mock.
Run the testing suite.	Validate the RPMs using rpmlint.
Publish test results.	Save validation output for inspection.
Save source code package to repository.	Save built RPMs for later download.
Send notification to pertinent developers.	Send notification to pertinent packagers.

Jenkins is built primarily for downloading code from one and only one SCM. However, you can work around this issue by adding another build step. This means that the SCM plugin is used to download the spec file and patches while the first step in the build process downloads the source code package. After these two steps are done, the source code, patches or spec file can be patched with site-specific customization.

The next step is to build RPMs using Mock. This involves several tasks that can be broken up into various build steps (see the Mock Build in Jenkins sidebar). All these steps are done using the Jenkins execute shell build steps. Some of the Jenkins jobs we use are multi-configuration jobs that contain one axis defining the Mock chroot configuration. That chroot configuration should be generated from the daily repositories defined in Cobbler. Following these tasks can get you started on using Mock in Jenkins (Listing 1).

Listing 1. basic-mock-jenkins.sh

#!/bin/bash -xe

# keep in mind DIST is defined in multi-configuration axis
MOCK="/usr/bin/mock -r $DIST"
PKG=${JOB_NAME##*/}
# keep in mind VER could also be a multi-configuration axis
VER=${VER:-1.0}
# if you are ripping apart an RPM might have this one too
REL=${REL:-4.el6}

OUT=$PWD/output

wget -O $PKG-$VER.tar.gz 
 ↪http://www.example.com/sources/$PKG-$VER.tar.gz
rm -f $OUT/*.src.rpm
if ! $MOCK --resultdir=$OUT --buildsrpm --spec=$PKG.spec 
 ↪--sources=$PWD
then
    more $OUT/*.log | cat
    exit -1
fi

if ! $MOCK --resultdir=$OUT --rebuild $OUT/*.src.rpm
then
    more $OUT/*.log | cat
    exit -1
fi

rpmlint $OUT/*.rpm > rpmlint.log

Mock Build in Jenkins

1. Prepare the source and specs.

2. Run Mock source rpm build.

3. Run Mock rpm build.

4. Run rpm validation.

Once the RPMs are built, it's important to run rpmlint on the resulting RPMs. This output gives useful advice for how to package RPMs properly for the targeted platform. This output should be handled like any other static code analysis tool. The number of warnings and errors should be tracked, counted and graphed over a series of builds. This gives a good indication whether bugs are being resolved or introduced over time.

The generated RPMs and rpmlint output need to be archived for future use. The archive artifacts plugin works well for capturing these files. There also is an artifact deployer plugin that can copy the artifacts to directories that Cobbler can be configured to synchronize from for its part of the process.

There is some room for improvement in this process, and I outline that in the conclusion. However, this is the basic framework to start using Jenkins to build RPMs using Mock and rpmlint. This part of the process needs constant care and attention as new updates are pushed by the distribution and package developers. Jenkins does have plugins to Trac and other issue-tracking systems. However, they are not included in this process, as we find e-mail to be a sufficient means of communication. The outlined process for building RPMs using Jenkins helps us track the hacks we use to manipulate important packages for our systems.

Table 2. Software

Role	Software Choice
Continuous Integration	Jenkins
Repository Management	Cobbler
Provisioning	Cobbler
Ticket Tracking	Trac
Wiki	Trac
Package Building	Mock
Package Guidelines	Fedora Packaging Guidelines

Conclusion

I have discussed a method for setting up tools to develop RPMs against a custom distribution managed by Cobbler. Along with Trac, package developers can maintain updated RPMs of critical applications while managing communication. However, this process is not without gaps. First, I'll go over the gaps present in Jenkins, discussing core and plugin gaps that were not found. Then I'll discuss the gaps in Cobbler regarding repository management. These two systems are lacking in integration, although that can be worked around.

MultiSCM is a functionality in Jenkins that would simplify the package building process. There is a MultiSCM plugin; however, it is advertised as a proof-of-concept code. The hope is that the radio button selection for SCM would turn into a set of check boxes. There are related bugs, but they have not seen traction in years. Package development is another good example of the need to download and poll for updates on code from multiple places.

Here are links to information on the Jenkins Multiple SCMs Bugs:

• https://issues.jenkins-ci.org/browse/JENKINS-7192

• https://issues.jenkins-ci.org/browse/JENKINS-9720

Static code analysis tools are available as plugins for Jenkins, although these plugins do not include rpmlint. These plugins create graphs to track the number of warnings and errors in code over time. To perform the same task for packaging would be very helpful. However, you can work around this gap by using the generic plot plugin and another build step for each job.

Mock has a very well defined interface and workflow. A generic plugin to use Mock in Jenkins would be very useful. The plugin should include configuring the chroot configuration. Two kinds of build jobs also could be created, one using spec and source files, the other using source RPMs. A test also would need to be created to verify that Mock can be run without prompting for a user password. This plugin would be very helpful for automating this process, as we currently have to copy scripts between jobs.

There are some additions to Cobbler that would be useful for this process as well. There are no per-repo triggers. The ability to tell Trac that packages went from repo test to repo prod would be useful. Furthermore, the ability to tell Jenkins to build a package because a dependent package updated also would be useful.

The other useful addition to Cobbler would be the ability to remove older RPMs in the destination tree while synchronizing from the remote mirror. Cobbler repositories, if the "breed" is yum, build up in an append-only fashion. Processes for managing the space may be run periodically by removing the RPMs and then synchronizing the repository again. However, this leaves the repository in a broken state until the process is complete. This feature could be useful in any Cobbler deployment, as it would make sure repositories do not continue to take up space when RPMs are not needed.

Trac does not need any additional plugins to integrate better with Cobbler or Jenkins. We have found some usability issues with manipulating large tables in the wiki format. Some plugin to make editing large tables easier in the wiki format would be useful for us. Also, editing long pages becomes an issue if you cannot put comments throughout the page. We validate our procedures by having members of the group who are unfamiliar with the system read through the procedure. The reader should be able to comment on but not edit parts of the page. We have worked around or found plugins on the Trac Hacks page to resolve these issues.

The final request is for some level of certification from distribution maintainers to certify third-party packages. Many of the third-party packages we have applied to this process to do not support all distribution configurations. A certification from distribution maintainers validating that software distributed by third-party vendors have packaged their software appropriately for the distribution would help customers determine the cost of support.

This is by no means a complete solution for organizations to build customized critical applications. There are still gaps in the system that we have to work around using scripts or manual intervention. We constantly are working on the process and tools to make them better, so any suggestions to improve it are welcome. However, these tools do fill the need to support customization of critical applications for HPC at EMSL.

Acknowledgement

The research was performed using EMSL, a national scientific user facility sponsored by the Department of Energy's Office of Biological and Environmental Research and located at Pacific Northwest National Laboratory.