Build and release is a complicated process. I really don’t need to tell anyone that…but I did anyway. But a rapid, precise feedback cycle that notifies whether the latest build of software passed successfully or not, and one that tells you WHAT failed specifically instead of just “operation failed,” can mean the difference between being able to quickly fix defects, or not discovering those defects until later in the project’s lifecycle, thereby increasing exponentially the cost and difficulty of fixing them. IncrediBuild has been addressing precisely this with Microsoft TFS for years, and is now reinventing their tool for Linux developers.

The secret, according to Eyal Maor, IncrediBuild’s CEO, is what they call Process Virtualization. In a nutshell, Process Virtualization helps to speed up the build process by allowing the build machine to utilize all the cores in ALL the machines across the network. Maor says that in most cases, “cycles can be reduced from 20 minutes to under 2 minutes, and there are many more cycles available.”

For Linux Journal readers, perhaps the most interesting part of this is how it works. Let’s start at the beginning. Standard build processes usually allocate a single build agent, which often utilizes a single CPU core in order to run the entire workflow. Most of the workflow is done sequentially, and very few parts run in parallel. This means that in order to gain an improvement in performance, it is necessary to increase the speed of the build agent’s CPU for computational processes, and the IO capabilities for IO-bound processes, such as reading and writing to the file-system or network.

Since Moore’s Law was essentially “repealed” around 2003, the idea of just throwing more hardware at speed-related performance issues no longer works. Using multi-core machines, of course, is a start, but one cannot simply take a dual-core machine, run a process, and expect it to automatically run twice as fast as a single-core machine with the same clock speed. A dual- core CPU CAN run twice as many operations as its single-core counterpart, but it must run them in parallel, two at a time, in order to achieve the goal of doubling the process speed. The key, then, is to break down processes into independent processes, and run each on a separate core to realize the multi-core benefit. It’s critical, therefore, to break down and parallelize as many of your CPU bound operations as you can, in order to maximize the speed of your processes and not under-utilize all that shiny, expensive hardware.

So obviously, operations need to be done in parallel, but the devil is in the details for parallel processing. Remember that, if improving speed by more efficient usage of multi-core CPUs is the goal, attention should be focused on parallelizing CPU-bound operations, not I/O-bound operations. Beyond that, there are a couple of possible parallel processing solutions:

• Clustering - Essentially parallelizing the build process workflow level, running one complete workflow on each agent, so that multiple build requests can be handled in parallel.
• HPC - Computers (physical or virtual) that can aggregate computing power in a way that delivers much higher performance than that of an individual desktop computer or workstation. HPCs make use of many CPU cores in order to accomplish this. Ubuntu-based machines, for example, can support up to 256 cores.

While either of these solutions provides opportunity to speed up processes, both have limitations and drawbacks. The clustering option gets ridiculously complex very quickly, and it is also expensive and time consuming to force the cluster to be optimally utilized. The entire build workflow would need to be broken down into multiple workflows, requiring a separate workflow for compiling each project solution, running each test, etc. Then it would still be necessary to synchronize the environments and outputs of each such workflow, send the source files to each compiling workflow, retrieve and organize the compilation results, and then send them to new workflows for testing, and synchronize the results of those workflows. Ugghh. It would also be necessary to set up the required software on each participating build agent, so parallelization efforts would be capped at the number of machines that are dedicated to the build process. This parallelization of numerous discrete projects is, as I mentioned, complex, expensive, time consuming and ultimately not cost-effective.

The HPC solution also has drawbacks. It is limited by the number of available cores, which in a complex build can get eaten up very rapidly. Granularity is also a concern. Without spending a great amount of time and effort modifying the way software is compiled, the project-solution (or top-level project) is atomic, and building it cannot be done in parallel. Finally, the cost of the system itself can be prohibitive; usually tens to hundreds of thousands of dollars, and many companies cannot afford that type of capital outlay and then devote those assets exclusively to the build process.

IncrediBuild, on the other hand, transforms any build machine into a virtual supercomputer by allowing it to harness idle CPU cycles from remote machines across the network even while they're in use. No changes are needed to source code, no additional hardware is required, no dedicated cluster is necessary and it works on both local and WAN so one can scale limitlessly into the cloud. It does not disturb the normal performance of existing operations on the remote machine, instead making use of idle CPU cycles on any cores on the network.

IncrediBuild can be incorporated into your existing build workflow, and accelerate your build engine, by replacing the default compiler. When activated, it intercepts the calls to the compiler and detours these calls to run remotely on another machine. IncrediBuild makes sure that the remote executor has what it needs for running the compilation process, and the remote machine sends the compilation results back to the initiator (the build agent). Compilation performance benchmarks have shown IncrediBuild’s ability to accelerate processes by at least 8x plus. This speed is accomplished using process virtualization.

Process virtualization consists of two actors – the initiator machine and the helper machines. The initiator distributes computational processes over the network as if the remote machines’ cores were its own, while the processes executed on the helpers (remote machines) see the initiator’s file system and registry instead of the helpers’ own. Process virtualization is based on the following concepts: local interception of requests to execute processes, delegation of these processes for remote execution, interception of file system requests performed by the remotely executed processes, on-demand synchronization of these resources from the initiating machine to the helper machine, and rerouting of the remote process file system calls to the files that were synched and cached from the initiating machine.

When IncrediBuild is set to compile a project, it activates the default compilation tool. When the compiler attempts to create a compilation process (e.g. compiling C# with csc.exe), it intercepts the call to the operating system, and sends the request, along with the necessary inputs (the compiler executable, source code, and environmental settings) to an idle remote helper core for execution. This information is compressed and cached in order to optimize and reduce network traffic between the initiator and helpers. The idea is to remotely execute processes which are expected to be computationally expensive.

Upon receiving a process, the helper (remote machine) begins to execute it, using the parameters it received from the initiator. The helper retrieves any file-system or registry resource that it needs (and is missing in the cache) from the initiator, in order to complete the process. This is done by intercepting any I/O or machine-specific information, and forwarding the request back to the initiator. In this way, the helper has access to the initiator’s registry, file-system, and other machine-specific information. The helper’s core becomes, virtually, a core of the initiator machine.

Finally, when the computation is complete, the output, i.e. the compilation result of this remotely run process, is returned to the initiator where it is stored as though it were run on that machine (any StdOut, StdErr, etc. are transferred as well). In this manner, every lengthy CPU bound process is delegated to another machine, allowing the entire network to work on them at the same time, rather than having one CPU run them all sequentially. All this is achieved without changing the way the software is written, without any need to install anything on the remote machine, and without changing how build workflows are run.

IncrediBuild is a product of Xoreax Software which launched in 2002 and has since become the de facto standard solution for code-build acceleration. Today it boasts more than 100,000 users at over 200 companies who depend on it for application and build acceleration. The Linux version will be announced at LinuxCon in Seattle on 8/17.