In advance of the release of the AMD64 Opteron processor, we had the opportunity to test a dual-processor SMP system with a 64-bit Linux distribution. As far as most Linux users are concerned, Opteron is the most significant new hardware introduction so far this decade. This early look covers the following key areas:

The Opteron is a new device family based on a new 64-bit architecture that is compatible with the pre-existing x86 32-bit architecture. AMD's choice to preserve compatibility has positive implications for transmigrating 32-bit workloads.

The Opteron architecture supports four application programming models. The first is the General-Purpose model, which performs basic operations like memory access, control flow, exception handling and I/O. The General-Purpose model also sets up memory optimizations that are used by the other application programming models. The next model is 128-bit Media Programming, which uses 128-bit XMM registers. Operations supported under this model include integer and floating point on vectors and scalar data. Similar capabilities are supported in the 64-bit Media Programming model. The last programming model is called x87 Floating-Point Programming, which uses x87 registers for 80-bit floating-point and scalar operations.

The code name for the processor core is Sledgehammer, and the device ships in a 940-pin ceramic micro PGA package. The current Opterons use nearly 106 million transistors in a 130nm Silicon on Insulator (SOI) process. The devices were fabricated at AMD's Fab30 in Dresden, Germany. The L1 cache has a 128KB capacity, split into a 64KB instruction cache and a 64KB data cache. An on-chip L2 cache has a 1MB capacity. The processor runs at 1.55V and provides a die size of 193mm2. The Resources section has more information on AMD64 architecture and open-source software support for this processor.

The Opteron processor is a highly integrated processor with features designed to attain balanced system performance. As such, it contains an integrated high-performance coupling link called HyperTransport, which offers 6.4GB/sec full-duplex data exchanges between processors or other HyperTransport nodes. Support is provided for up to three HyperTransport links, for a total of up to 19.2GB/sec peak bandwidth per processor. In addition, each Opteron contains an integrated memory controller, which offers very high bandwidth and error control capabilities. ECC (error correcting code) protection is provided for L1 cache data, L2 cache data and tags, and in external DRAM with hardware scrubbing of all ECC-protected arrays.

AMD has a three-digit part numbering scheme for the Opterons. The first digit indicates the intended SMP scalability, which is two-way in the 1.8GHz Opteron Model 244 and the 1.6GHz Opteron Model 242 covered in this article. The second digit indicates relative performance within the scalability family. As chip manufacturing costs decline and process technology improves, other model numbers of a given scalability class will emerge at higher frequencies and lower costs. AMD also will be manufacturing a one-way Opteron, which is intended for high-performance lower-cost systems.

Models 240, 242 and 244 are available at the time of this writing. The eight-way capable models 840, 842 and 844 are scheduled to be available in May 2003 and model 144 in Q3 of 2003.

The Opteron extends the x86 architecture, allowing customers to run existing 32-bit applications on a 64-bit OS. Customers who run a 64-bit OS will be ready to support future 64-bit applications and migrate at their own pace, while maintaining the usefulness of their 32-bit applications.

The reference platform is called the 2100 and was realized by Newisys, Inc., a technology provider that now has two years of experience with Opteron (see Resources).

Figure 1. The Newisys 2100 puts two Opteron processors in a 1U enclosure.

The 2100, a 1U, two-processor, rackmountable system, is superbly engineered. The mechanical and electrical design offers reliability in a dense package (Figure 1). For example, the power supply actually is designed for 500,000 hour MTBF. If you work a typical 2,080 hours a year, this level of reliability would be like working for 240 years without making a mistake.

With Opteron-balanced chipsets and excellent board-level integration features, this system has improved memory performance and capacity significantly. The result is a high-performance balanced server design with robust I/O. The evaluation system came with 6GB of PC2700 memory, but the server supports 16GB.

Figure 2 is a top view of the system. The two copper heatsinks are the processors. Two speed grades are supported: the 1.6GHz Opteron model 242 or the 1.8GHz Opteron model 244. The Opterons link to each other and to the chipset using HyperTransport. The CPU-to-CPU bandwidth is 3.2GB/sec—in each direction (full-duplex). The Opterons each have an internal memory controller that supports ECC DDR SDRAM at a bandwidth of 5.33GB/sec (each). There are two banks of memory, one next to each processor.

Figure 2. A Top View of the Newisys 2100

The AMD-8000 HyperTransport chipset includes an AMD-8131 HyperTransport PCI-X chip, as well as an AMD-8111 I/O hub chip. The AMD-8131 is configured to drive a full-slot PCI-X 64-bit/133MHz at 1GB/sec data rate, as well as a half-slot PCI-X 64-bit/66MHz, at a 0.51GB/sec data rate. The AMD-8131 also provides a pair of triple-mode NICs (10M/100M/1GB), plus a dual Ultra-SCSI RAID controller. Our test system had dual hot-swappable Ultra-SCSI drives in a RAID configuration in the front of the case. The AMD-8111 chip provides a VGA port, IDE CD-ROM and a USB port. Separately, a SuperI/O chip provides a floppy, keyboard, mouse and conventional serial port.

Figure 3. The server management processor has its own LCD display, under the CD-ROM drive.

The system also contains a separate embedded server management processor and it runs Linux. This subsystem is based on a Motorola XPC855T PowerPC processor, running kernel 2.4.18. In addition to a small front-panel control console, the server management processor provides a pair of isolated 10/100 Ethernet interfaces to connect to an independent management subnet. Thus, system management can be done without a keyboard and monitor, or even a serial console access server, and using it does not take up one of the PCI slots.

This system's management facility is quite advanced. The management processor supports SNMP, CIM and IPMI protocols. NIS, Microsoft Active Directory and LDAP authentication support all are provided. Cloning of the service processor configuration can be done peer-to-peer. In addition, one service processor can be designated as the controller for an entire farm of servers. The management processor also provides zero-footprint diagnostics. Machine check analysis, such as access to memory and processor scans, can be done independently.

Newisys has announced they will supply the core technology integration and packaging engineering expertise, but leave the manufacturing and distribution to external OEMs and licensees. They have contract manufacturing arranged though Sanmina SCI and distribution through Avnet.

Newisys partners include some well-known names in IT: Angstrom Microsystems, APPRO, RackSaver, M&A Technology, Microway, New Technology Solutions, Inc., and ProMicro. At the time of this writing, some 600 systems have been fielded for development and evaluation among OEMs, Fortune 500 companies and, of course, Linux Journal.

Because this is a new fast system, after typing uname -a at the bash prompt, the next thing you do is cat /proc/cpuinfo:

processor   : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 5
model name      : AMD Opteron™
stepping        : 0
cpu MHz         : 1594.286
cache size      : 1024 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce
    cx8 apic sep mtrr pge mca cmov pat pse36 clflush
    mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext
    3dnow
bogomips        : 3178.49
TLB size        : 1088 4K pages
clflush size    : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts ttp

Processor 1's information is the same but with a BogoMips of 3185.04.

The main GUI environment supported on SuSE Linux Enterprise Server 8 is KDE 3.0, and the system comes with all the regular tools and applications usually expected with a SuSE distribution. The SuSE Enterprise Linux implementation on the Opteron itself is the result of a lot of hard work and a long history of 64-bit Linux hacking, dating back to when Jon “maddog” Hall first gave Linus Torvalds a Digital Alpha.

The GNU environment simply works—only at 64 bits. All of the usual compilers and the GNU build environment work right out of the box. One of the first things you notice with this system is that Emacs starts immediately, faster than you can blink.

The first test was to compile and run the GNU Scientific Library (GSL), which is a numerical computing library. This library itself is rather robust, and we compiled version 1.3. Then we compiled some sample code from the GNU Scientific Library Reference Manual to generate random numbers. This really simple program, which was compiled without any optimization, revealed that this system was about three times faster than the same program on x86 at 800MHz.

However, we learned right away how often programming is done with the unconscious thought that it's a 32-bit system. One part of the test program left-shifted an integer 1 by sizeof(int) * 8 - 1 to obtain the number of bits, based on the bit size of the machine's native integer. On this machine, in contrast to typical x86 PCs, compiling the program with that nonportable hack caused immediate integer overflow. Although common tools are now 64-bit clean, your locally developed code may need some work.

We performed the next test with the Icarus Verilog compiler. Icarus Verilog is a GPLed Verilog HDL compiler for electronic design automation (EDA), especially HDL simulation and synthesis. Linux Journal has interviewed Stephen Williams, the author of Icarus, two times now [see “Open Source in Electronic Design Automation”, LJ, February 2001, www.linuxjournal.com/article/4428 and “A Conversation with Stephen Williams”, LJ, July 2002, www.linuxjournal.com/article/6002]. Over time, we have documented the steadily increasing advances this compiler has made as a serious industrial-strength EDA tool for logic and FPGA design. Because Stephen has been building Icarus as 64-bit clean code for the past five years under Linux for Alpha, compiling the compiler was relatively easy. What was of interest was learning how fast it was and how well it responded to large workloads and many more cycles of operation in terms of runtime.

We tested a large multiplier logic model and testbench from the Icarus test suite, and it was almost twice as fast as a 1.5GHz Athlon processor running the same binary. For the large workload test, we compiled a logic model consisting of 1,720,648 lines of Verilog code. This also was breathtaking. In 61 minutes, the machine compiled a model with a memory footprint larger than the largest user space in 32-bit Linux—3.6GB.

Because Icarus uses its own internal threading system, these particular EDA tests used only one-half of this computer. Another processor, with scalable memory bandwidth, runs more EDA simulation projects without even affecting the first. Opteron clearly is well suited as server technology for engineering work.

The last major test compiled was for the Linux Test Project (LTP). The LTP is a GPLed test environment for Linux, available for download on SourceForge (see Resources). The version tested was ltp-full-20030404, which is from early April 2003. It compiled and ran the default tests with no problem whatsoever.

A qualitative summary of the LTP tests would be that 64-bit SuSE Enterprise Linux is squeaky clean—numerous picayune Linux system calls are tested in LTP. There were extremely few failures, and of these, none were of any substantive impact. In our experience, this same LTP on some other recent 32-bit Linux distributions has many more test fails, including actual crashes.

Although the LTP is well geared for testing Linux, some other more quantitative data about the Opteron is necessary to begin to glean numerically the limits to performance for this processor and system architecture. We turn now to some preliminary benchmarks.

These processors are wicked fast, and it's still early in the system characterization cycle. For example, new applications need to be compiled for a 64-bit operating system to attain the full measure of the available performance. For now, some early industry-standard benchmarks available at press time are listed in Tables 1 and 2. These are for a dual-processor SMP Opteron 244 with PC2700 memory configuration, for 32-bit applications. In Table 1, the benchmarks infer relative integer and floating-point processor performance. The benchmark in Table 2 is designed to evaluate the performance of a server's ability to run Java applications.

Your intended application determines the best benchmark. Programs can appear similar but have unique peculiarities that are stressed differently under system loads.

Table 1. Opteron 244 Benchmarks

Table 2. Opteron 244 Java Middleware Benchmark

The Opteron is breakthrough 64-bit processor technology, one that seems destined to provide high performance and cost savings for cleanly migrating 32-bit architecture programs into a 64-bit application space. In the near future, the upside for 64-bit “long mode” applications running on Linux seems high indeed, because eight-way SMP processors are coming. Linux Journal will be covering this emergent technology with additional articles in the future.

Resources