Building A High-End Workstation February 03, 2003 Alan Dang

Introduction

Page:: ( 1 / 8 )

We have built or tweaked custom gaming systems -- if you didn’t, you wouldn’t be reading FiringSquad. Your custom-built gaming system is probably an equally impressive system for your non-gaming tasks, and so if you ever needed to build an office PC for someone else, your experiences will serve you well.

However, what happens when someone asks you to advise them on building a high-performance PC where the focus is not gaming or even office applications, but honest-to-goodness computation? In these cases, you need to throw out everything you know about gaming PCs. Building a workstation PC takes a different mindset. Recently, I had an opportunity to do just that for a medical lab that relies heavily on finite element analysis. The details are unimportant, just think of it as being similar to the crash-test studies a car manufacturer will do, only on a smaller scale.

Our lab’s current computation system is an SGI Octane2 with dual R12000 400MHz CPUs, 4GB of RAM, and V10 graphics. These systems were around $40,000 when first released. Each R12000 400MHz has a SpecFP2000 of around 350-360 and so it’s approximately equal to an Athlon 1.2GHz. The caveat is that the SpecFP2000 benchmark is actually made up of a bunch of other, smaller, tests. For computational fluid dynamics or neural network image recognition, the 400MHz SGI CPU is 2.5 to 5 times faster than the Athlon! For crash test simulations, the Athlon is almost 2 times faster than the MIPS CPU. From the SpecFP alone, it’s hard to tell whether a PC or SGI machine would be faster for our own research. You have to recall that there’s the issue of operating system overhead, the crossbar architecture of the SGI, and other little details such as the fact that the MIPS CPU has 2MB of L2 cache. In addition, it’s hard to know exactly how close a custom app will be to the synthetic benchmarks.

That said, our lab still needed to move to x86. SGI equipment was too expensive, and although we have some custom written software for IRIX, a good amount of our work involves MATLAB; In January, the developers of MATLAB announced that they were abandoning the SGI platform. That gave me the green light to build a high-end PC.

In this article, I’ll describe the system I built. Along the way, I’ll explain why I picked certain components and why it’s different from what I’d pick in a gaming system.


SIDEBAR: Thanks to software such as Inferno, SGI systems are still the leaders for cinematic video compositing.

The CPU

Page:: ( 2 / 8 )

Pentium 4 versus Athlon

No one gets fired for buying Intel…no one gets fired for buying Dell… but at one point in time, no one got fired for using i820 Rambus.

If you are being asked to build a system for someone else, don’t simply take the easy road out. Someone has entrusted you to determine the best system spec for their project. You’re a knowledgeable reader – you know what you’re talking about. In other words don’t blindly go with Intel or AMD. Look at your project needs and then pick the right product.

Pentium 4 based CPUs are great whenever you can use the SSE2 instructions. Since the SpecFP benchmarks can be auto-vectorized with Intel’s compiler, the Pentium 4 has superb benchmark ratings. However, not all code can be auto-vectorized, and we know that the Pentium 4’s FPU performance is not very impressive. Since we’re building a workstation, you probably won’t even bother looking at P4’s – XEON is what you want. While both the P4 and XEON are based upon a similar cores, the XEON offers multiprocessor support and larger L2 caches. Intel CPUs are a good choice if you need good real-time 3D graphics or if you’re doing a lot of media encoding that is SSE2 enhanced. Large L2 caches are important when you’re dealing with repetitive data and repetitive tasks. Dynamically changing data won’t work as well.

For our scientific research, it’s important to have extreme precision throughout the entire calculation. Although SSE2 is incredibly fast, it deals with numbers in 64-bit precision. The standard x87 FPU maintains 80-bit precision. In case you’re wondering about how that compares to the GeForce FX’s 128-bit precision, you need to remember that that’s only 32-bits per channel (red, green, blue, alpha). The FPU is dealing with a single number.

The other funny thing about research is that while, by definition, we’re at the cutting-edge of information, the tools we use are usually old (and reliable). So, most academic computing software is not optimized for SSE2 instructions, even if there are portions where 64-bit precision is sufficient. This is where the AMD Athlon CPU comes in handy. The Athlon FPU has three parallel floating point execution units – each which can add, multiply or store numbers. The Pentium4 has one full FP execution unit, and one that can only move or store numbers.

Clearly, in the case of pure high-precision FPU tasks, such as our project, the Athlon architecture is superior to the Pentium 4. The system you may need to build in the future may have different requirements. You might not care about FPU performance, and instead need a very large L2 cache. Choose wisely.

Athlon MP versus Athlon XP

Techies all know that these two CPUs are the same, right? They’re wrong.

It’s true that the Athlon MP and XP are built as identical cores at the fab. In fact, you can tweak your L5 bridges on your Athlon XP to enable support for dual processors. Nonetheless, this still doesn’t mean they’re the same thing. Have you ever wondered why the Athlon MP lags behind the Athlon XP in megahertz? The flagship Athlon MP is only at 2400+ while Athlon XP is at 2800+…

Athlon MP’s are binned Athlon XP’s. No two CPU cores fabricated are absolutely identical, and the Athlon MP represent AMD’s best product. The goal for the MP line of chips is to have lower temperatures for the same megahertz. This makes their clock ramp-up fall behind the Athlon XP line. Stability is the obvious benefit, however recall also that rackmount servers don’t have the same exotic cooling solutions that your desktop may have. Binning is one way AMD ensures a superior product for multi-processor systems. The other way AMD ensures the MP lineup is reliable is that the first Athlon MP 2400+ is going to have a later CPU stepping than the first Athlon XP 2400+.

For our system, we went with two Athlon MP 2400+ CPUs, AMD’s fastest MP processor.



SIDEBAR: Fortran is the programming language of choice for scientific computing. It’s really fast.

Choosing the motherboard

Page:: ( 3 / 8 )

Dealing with power

We have our CPU. Now we need a motherboard. Before I go on, let me side step a bit and talk about stability. Recall from December’s article that the power supply plays a huge role in stability. For the workstation I was building, I wanted to go with an EPS12V power supply as opposed to ATX12V.

EPS12V is a standard used in high-end XEON and AMD motherboards in which there are three +12V outputs. This provides added stability. Our power supply of choice was the $200 PC Power & Cooling Turbo-Cool 475 XE. Though 475W may not seem like much (our Octane2 features an 800W power supply), the true performance of the 475XE is excellent. The rated specs are:

+5V @ 40A
+12Vio @ 16A
+12Vdig @ 18A
+3.3V @ 45A
+5VSB @ 3.5A
+12Vio/dig<30A
+5 & +3.3V<300W
power@50C<460W
power@40C<510W
peak power <600W

Notice that the power supply is appropriately rated for differing temperatures. The Turbo-Cool is probably one of the best power supplies in its class.

Workstation/Server Motherboard

For our motherboard, we went with the Tyan Thunder K7X Pro. We did this for a few reasons. When it comes to EPS12V based dual AMD systems, Tyan had the most experience of the motherboard manufacturers we considered. Google and Yahoo run on Tyan motherboards. Specifically, the Thunder K7X Pro gave us two on-board Ethernet controllers, one being Gigabit Ethernet. This was important to us since the system may become a node in a larger cluster in the future.





SIDEBAR: The difference between a Beowulf cluster and a “real” supercomputer is latency.

Memory

Page:: ( 4 / 8 )

I went with PC2100 memory, not PC2700 or PC3300. Why? It’s not because I only needed PC2100 DDR RAM. If you were building a gaming or office PC with 266MHz DDR-RAM, you’d still shop for the memory with the highest rating. What makes a workstation or server different is Registered ECC memory.

Two to twelve times each year, a bit in memory gets inappropriately flipped. This can be caused by cosmic rays flying through your RAM or a decay of the minute radioactive isotopes found in your RAM – the impurity need only be a single atom. Most of the time, this flipped bit is unimportant. Maybe it’s a flipped bit in unallocated memory, or maybe it just altered the position of a pixel for a fraction of a second. If you’re unlucky though, this flipped bit can alter critical data and cause your system to crash. In our situation, a flipped bit could potentially alter our results significantly.

ECC memory provides error checking and correction facilities. By adding an extra memory chip, ECC memory makes your memory act in a similar way to a RAID array. In other words, when that lone bit is inappropriately altered, the memory is able to detect this failure and correct it.

FiringSquad has always had good experiences with Corsair memory, and so I went with two 512 MB sticks of registered DDR memory. In general, it’s not a good idea to rely on a single DIMM for a high-end workstation. Even though Corsair offers a lifetime warranty, replacing of a bad stick of memory still requires an RMA (shipping time), or a trip to Fry’s. Having a system built with two memory sticks means that you’ll still have a running system while you wait for the part to be replaced.

But I made mistake in selecting the memory. There is no such thing as “cost-is-no-object” -- you are always on a budget. In selecting the memory, I decided that a gig of RAM would be plenty. Of course, once the system was built and we started running some more test applications, I realized that 1GB was just enough to provide data to one CPU. (Some of our datasets contain over 800MB of data). Our system needed at least 2GB of memory. By going with two 512MB DIMMS instead of two 1GB DIMMS, I have limited the system upgrade potential to 3GB rather than 4GB (unless we are willing to sacrifice the two 512MB DIMMS). Two lessons:
1. Make sure you know your requirements when you build the PC
2. You can never have too much memory … really.


SIDEBAR: Overheard: 1GB is enough for everyone.

Quadro FX

Page:: ( 5 / 8 )

What video card would you choose? In our system, we decided to go with the Quadro FX 2000. This card alone would double the cost of the system. Did we need the QuadroFX instead of a slower Quadro4-class card? Probably…but we certainly needed a Quadro instead of a GeForce or a FireGL instead of a Radeon. Do you know why?

[image]

Differences

Admittedly, a fair amount of the Quadro’s performance comes through workstation-optimized drivers, but a hacked GeForce isn’t the same thing as a Quadro. You might read that the hacks work, and that performance is “pretty similar except for a few benchmarks.” The problem is that the exceptions are important in true workstation applications, and I’ll prove this using real world data. The Quadro line of cards adds hardware:
      1) Line Anti-Aliasing
      2) Accelerated Clip Regions
      3) Clip Planes
      4) Two Sided Lighting
      5) OpenGL Logic
      6) Overlay Planes

I won’t say too much about these features. OpenGL logic or overlay planes since they will be likely become obsolete as applications move to shader paradigms. Two-sided lighting is exactly what it sounds like. Clip planes allow you to arbitrarily slice models in hardware, thus maintaining full performance. For our work, line anti-aliasing, accelerated clip regions, and sub-pixel accuracy are the important features.

Line Anti-aliasing

In workstation applications, you’re often working with lines rather than polygons/textures. Traditional multisampling or supersampling anti-aliasing algorithms may not be the best algorithm for rendering points or lines. The Quadro lineup features hardware line anti-aliasing. Unfortunately, it’s not a feature that can be forced on.

Accelerated Clip Regions

Workstation 3D is almost always windowed 3D and therefore, it’s not uncommon to have multiple windows open. During a typical use, applications may pop up many windows containing other 3D scenes or menus. On regular gaming cards, overlapping windows can noticeably affect graphics performance.

When a window has no overlapping windows, a Quadro board will copy the entire contents of the color buffer into the frame buffer as a single continuous rectangular region. If other windows overlap the 3D region, the data will be transferred in multiple, smaller chunks of region called “clip regions.” Put in a different way, hardware clip regions acts like early Z-test visibility subsystems. In 3D rendering, you don’t want to process pixels that will not be visible. With multiple hardware clip regions, you won’t be losing as bandwidth to regions of the screen that will not be visible. The Quadro supports 8 clipping regions whereas the GeForce only supports 1. Again, this is a useless feature for full screen gaming, but a very important performance-enhancing feature for typical workstation use.

These differences are present between the Quadro and GeForce versions of the the NV17/NV18 class cards (GeForce4 MX). The Quadro FX offers even more differences from the GeForce FX. Quadro FX allows users to run more complex pixel shaders (2048 instructions versus 1024), has 12-bit sub-pixel precision, and a cooling system more appropriate for workstations.

Subpixel Precision

The advantage of subpixel precision is just like the advantage of having floating point color buffers. Lines are more precisely drawn in 3D, reducing artifacts. Imagine if you had to draw a line from (0.5,0.13) to (35,50). You cannot draw a pixel at non-integer locations; subpixel precision provides the ability to keep geometry calculations at a higher precision. The nice thing about this is that it is automatically turned on – you don’t need to wait for new software. That’s all I need to say about this feature because the pictures will explain everything. In these examples, I am using my real data sets and not a mythical “worst-case scenario.” The blue lines on the red surface represent the angle of the myocytes at that location and subpixel accuracy is critical for positing the lines on the surface, and not below the surface.

[image]

Quadro vs. FireGL

Page:: ( 6 / 8 )

There are a number of reasons why people would want a Quadro FX. It’s unmatched when it comes to subpixel accuracy and the card offers incredible performance. On my Athlon XP 2000+, my SpecViewPerf 7.0 scores are
      3dsmax-01 Weighted Geometric Mean = 18.13
      drv-08 Weighted Geometric Mean = 80.05
     dx-07 Weighted Geometric Mean = 78.35
     light-05 Weighted Geometric Mean = 20.51
     proe-01 Weighted Geometric Mean = 28.27
     ugs-01 Weighted Geometric Mean = 28.26

The Unigraphics viewset (ugs-01) has been one of the FireGL X1’s strongest benchmarks (more than doubling the Quadro4 900), and yet the score of 28.26 on my XP2000+ handily beats published results of 19.10 on a FireGL X1 on a Pentium4 2.2GHz.

The CineFX architecture is also a big deal. Perhaps you have an application needing 128-bit color, our Shader 2.0+ technology. The QuadroFX is the only way you’ll have shaders with 2048 instructions long! But wait you think, what good is having a card that does 2048 when the shaders are going to be “dumbed down” so that they’ll run on regular consumer cards? Two answers. Cg is a just-in-time compiler which means that high-level code can be compiled differently for the QuadroFX than it is for a consumer card. The second answer is that a lot of people will never even use the QuadroFX for game development. Shader technology is already being used by 3D CAD applications such as Solidworks to improve visualization.

Why a FireGL

Suppose you don’t need the performance of the QuadroFX or the added subpixel precision, would the ATI Radeon 9700 based FireGL X1 be a good choice? Sure – in fact, there are a few obvious reasons why someone would want a FireGL.

Suppose you needed DirectX9 shaders but also needed a massive 256MB of RAM for texturing? The Quadro FX is only 128MB. Suppose the Quadro4 XGL is just shy of your performance targets, but you don’t have the budget to step across into the QuadroFX?

Our decision

Everything comes back to the lesson of the day: know your task. Professional 3D graphics isn’t just about hardware – it’s also the software. Quadro cards come bundled with a number of workstations applications, one being QuadroView. QuadroView is an SGI Open Inventor viewer that I can use to visualize my data sets…an utterly useless application to the gamer, but critical to me. The FireGL products offered no such bundle. In addition, QuadroView was twice as fast as the demo of TGS 3Space Assistant (a commercial Open Inventor viewer for Windows), and a freeware Open Inventor viewer from dev-gallery.com wouldn’t even open up my data sets produced from the SGI.

Some professionals will never even touch QuadroView. Others may need something more complex. Suppose you want DirectX9 support for building game art, but can’t afford the Quadro FX? A FireGL would be a smart choice. Some professionals may use an Unreal Engine application (it has been used for commercial real estate applications). In that case, you’d really would be better off sticking with a regular gaming video card. For our needs, the Quadro fit the best.

TGS fans will probably be quick to point out the added functionality of the commercial 3Space Assistant, but our lab didn’t need the ability to decimate models, or add 3D text. When you’re working as an artist, your goal is to develop the best looking model. “Inefficient” polygons can be removed and budgeted elsewhere to improve the model. Imperfections caused by lack of subpixel precision can be tweaked by moving the objects. In the case of scientific visualization, you’re looking at hard data. You can’t change the data just to make it visualize nicer – you need to display all the data as is. For our lab, QuadroView was exactly what we needed. If we had gone with a FireGL, we would have had to spend the extra money buying a commercial application that may not even offer the same level of performance.

This page is here to reiterate the point that knowing your task at hand is critical in helping you decide components. For gaming, it’s easy to play favorites, but when you’re building a system for someone else, you want to think about every possibility.

Now, let’s take a quick detour to gaming…


SIDEBAR: I have an SGI Indigo2 Extreme graphics board as part of my museum collection. It is composed of three full-length cards.

Benchmarks

Page:: ( 7 / 8 )

Test notes

Even though this card will ultimately be used for research work, I couldn’t let it stay idle over the weekend, and so I decided to try out the card in my home system. Since workstations have better case cooling, NVIDIA decided that the FX Flow was not necessary. The Quadro FX 2000 cooling solution is still dual slot, but it is closer to a traditional heatsink and fan setup, and is very similar to the GeForce4 Ti cooler. The difference is that 12 inches away from the case, the A-weighted sound level is only 52 dB. My case features six fairly quiet temperature-controlled cooling fans. 50 dB is the same intensity as very soft music. The background noise outside my apartment was 55dB. Suffice it to say that when I first booted my computer with the Quadro FX in it, I was worried that something had failed – it was just too quiet. In idle, the GPU was only 51-52 degrees C with ambient temperature at 47.

After 1 hour of complex DirectX8 shading, temperatures reached only 61-63 C with ambient temperature of 52C. By default, the Quadro FX will not throttle down its core speed unless a temperature >140 C (or 248F) is reached! That must be the .13 micron in action! Seriously though, I suspect these temperatures are a bug in the drivers since it seems like the rest of the PC would be at great risk once a component was that hot. On my next reboot, I was able to hear a loud (60 dB) whir of the Quadro FX cooler running at full speed, but within 3 seconds it returned to the quiet operation. It never sped up again so long as I didn’t reboot.

[image]

Q3A 1.17

No AA, No AF
Demo 001
1152x864 – 212.0fps
1280x1024 – 200.9
1600x1200 – 169.1
2048x1536 – 113.8

Demo 002
1152x864 – 219.6
1280x1024 – 212.8
1600x1200 – 186.8
2048x1536 – 123.3

Clearly I am CPU limited at the resolutions at below 1280x1024. A 3 megapixel resolution is gorgeous. Since it’s beyond the resolution of the aperture grille of a monitor, you end getting an analog anti-aliasing that’s looks great.

No AA, 8x AF
Demo001
16x12 – 153.8
3MP- 105.2
Demo002
16x12 – 154.3
3MP- 104.6

Anisotropic filtering does not appear to take a significant hit. On the QuadroFX, the quality is fixed at “Balanced” and cannot be altered by the user.

VillageMark

10x7 - 185
16x12 - 103

FableMark

10x7 - 60.9
16x12 - 26.5

What is interesting

I’m getting solid performance with a GPU that never runs past 63C and enters into the “high fan speed mode.” This brings up the question of the headroom available to the NV30 GPU itself, and how much the GPU is being held back by memory bandwidth. There’s also the question about how an NV30 based card is going to perform in the future, when 3D applications start becoming shader limited rather than memory bandwidth limited. Why is it that I have a Quadro FX 2000 in a quiet system running incredibly stable without an FX Flow? Maybe I was lucky and got a golden sample or are we seeking some of the advantages of Quadro FX having longer development times? Time will tell.


SIDEBAR: The weary sparrow

stretched its crimson wings and flew

off into midnight

Other Components

Page:: ( 8 / 8 )

So what else did we choose?

Western Digital 80GB Caviar with 8MB Cache

Sure, in an ideal world, we’d run Ultra SCSI 320. We’ve already got plenty of SCSI devices (from our SGIs). Again, price is always an issue. One thing to consider is RAID (even software RAID). Our backup policies are good enough, so it was OK to skip the RAID.

Of course, we didn’t completely ignore HDD stability. I added a Cooler Master Hard Drive Cooler to the system. A HDD running for an extended period of time at 5 degrees above the recommended temperature can experience an increase in failure rate of 10-15%! I liked the Cooler Master design since it was a combined heat sink and fan setup. The fans draw cool air from the front of the case and run it through a thin mesh. This is superior to the ones that screw onto the bottom of the hard drive since it prevents the amount of dust that builds up.

Lite-On 48x24x48 CD-RW

My Plextor is king but I’ve had great success with my own Lite-On 48X CD writer. DVD-RW and DVD+RW drives are now affordable, however media is still expensive.

Teac 1.44 floppy

Don’t forget the floppy! Teac has been making 1.44 floppies since they were invented, and Teac has historically produced floppy drives with the best reliability.

Philips Seismic Edge 5.1

You always need a sound card for Windows systems. Speakers are optional. A handful of Windows applications demand the presence of a sound device. If no sound card is installed, the application will just quit out with an error message (bad programming). The sound card is there to preemptively address this problem. The Philips Seismic Edge came to me recommended by Tyan. Since my sound card was primarily for compatibility reasons, stability was key. Philips has had a stellar record with their current Windows 2000/XP drivers and dual processor systems. One significant caveat is that Linux isn’t supported, but we can get by. With the increased popularity of Linux among gamers, we certainly hope that Philips will at least produce basic drivers that support MP3 playback.

Logitech Keyboard/Mouse

Don’t forget the keyboard and mouse either!

Chieftec DX-01 Chassis

I needed a large case to support the extended ATX motherboard. One mistake I made here was forgetting that I would need to buy longer IDE cables.

Closing Thoughts

When you’re building a system for someone else, always think about what it is going to be used for and make sure you spec out the system appropriately. Is ECC memory something that’s going to be important? Will professional 3D accelerators be needed instead? Is the Athlon or Pentium4 more appropriate for the task? Do your research, be confident, and spend the money when it’s necessary.

Special thanks to AMD, NVIDIA, TYAN, and Ryan Ku at Rage3D.com for helping me with this project.




SIDEBAR: Questions or comments? As always, feel free to voice your thoughts in the news comments!