Linux Cluster HOWTO

Ram Samudrala (me@ram.org)

1. Introduction

2. Hardware

• 2.1 Node hardware
• 2.2 Server hardware
• 2.3 Desktop hardware
• 2.4 Putting-it-all-together hardware
• 2.5 Costs

3. Software

• 3.1 Linux, of course!
• 3.2 Costs

4. Configuration

• 4.1 Disk configuration
• 4.2 Package configuration

5. Acknowledgements

6. Bibliography

1. Introduction

This document describes how I set up my Linux computing clusters for high-performance computing which I need for my research.

Use the information below at your own risk. I disclaim all responsibility for anything you may do after reading this HOWTO. The latest version of this HOWTO will always be available at http://www.ram.org/computing/linux/linux_cluster.html.

Unlike other documentation that talks about setting up clusters in a general way, this is a specific description of how our lab is setup and includes not only details the compute aspects, but also the desktop, laptop, and public server aspects. This is done mainly for local use, but I figure I might as well put it up on the web and perhaps someone else will find it useful. The main use as it stands is that it's a report on what kind of hardware works well with Linux and what kind of hardware doesn't.

2. Hardware

This section covers the hardware choices I've made. Unless noted, assume that everything works really well.

2.1 Node hardware

32 machines have the following setup each:

• 2 Pentium III 1 GHz Intel CPUs
• Supermicro 370 DLE Dual PIII-FCPGA motherboard
• 2 256 MB 168-pin PC133 Registered ECC Micron RAM
• 1 20 GB Maxtor ATA/66 5400 RPM HD
• 1 40 GB Maxtor UDMA/100 7200 RPM HD
• Asus CD-S500 50x CDROM
• 1.4 MB floppy drive
• ATI Expert 98 8 MB PCI video card
• Mid-tower case

2.2 Server hardware

1 external server with the following setup:

• 2 Pentium III 1 GHz Intel CPUs
• Supermicro 370 DLE Dual PIII-FCPGA motherboard
• 2 256 MB 168-pin PC133 Registered ECC Micron RAM
• 1 20 GB Maxtor ATA/66 5400 RPM HD
• 2 40 GB Maxtor UDMA/100 7200 RPM HD
• Asus CD-S500 50x CDROM
• 1.4 MB floppy drive
• ATI Expert 98 8 MB PCI video card
• Full-tower case

2.3 Desktop hardware

4 desktops with the following setup:

• 2 Pentium III 1 GHz Intel CPUs
• Supermicro 370 DE6 Dual PIII-FCPGA motherboard
• 4 256 MB 168-pin PC133 Registered ECC Micron RAM
• 3 40 GB Maxtor UDMA/100 7200 RPM HD
• Ricoh 32x12x10 CDRW/DVD Combo EIDE
• 1.4 MB floppy drive
• Asus V7700 64mb GeForce2-GTS AGP video card
• Creative SB Live Platinum 5.1 sound card
• Microsoft Natural Keyboard
• Microsoft Intellimouse Explorer
• Full-tower case

2 desktops with the following setup:

• 2 Pentium III 1 GHz Intel CPUs
• Supermicro 370 DLE Dual PIII-FCPGA motherboard
• 4 256 MB 168-pin PC133 Registered ECC Micron RAM
• 3 40 GB Maxtor UDMA/100 7200 RPM HD
• Mitsumi 8x/4x/32x CDRW
• 1.4 MB floppy drive
• Jaton Nvidia TNT2 32mb PCI
• Creative SB LIVE Value PCI
• Microsoft Natural Keyboard
• Microsoft Intellimouse Explorer
• Full-tower case

2 desktops with the following setup:

• 2 Pentium III 1 GHz Intel CPUs
• Supermicro 370 DLE Dual PIII-FCPGA motherboard
• 4 256 MB 168-pin PC133 Registered ECC Micron RAM
• 3 40 GB Maxtor UDMA/100 7200 RPM HD
• Asus CD-S500 50x CDROM
• 1.4 MB floppy drive
• Jaton Nvidia TNT2 32mb PCI
• Creative SB LIVE Value PCI
• Microsoft Natural Keyboard
• Microsoft Intellimouse Explorer
• Full-tower case

Backup:

• 2 Sony 20/40 GB DSS4 SE LVD DAT

Monitors:

• 4 21" Sony CPD-G500 .24mm monitor
• 2 18" Viewsonic VP-181 TFT-LCD monitor

2.4 Putting-it-all-together hardware

We use KVM switches with a cheap monitor to connect up and "look" at all the machines:

• 15" .28dp XLN CTL Monitor
• 3 Belkin Omniview 16-Port Pro Switches
• 40 KVM cables

Networking is important.

• 1 Cisco Catalyst 3448 XL Enterprise Edition network switch.

2.5 Costs

Our vendor is Hard Drives Northwest ( http://www.hdnw.com). For each compute node in our cluster (containing two processors), we paid about $1500, including taxes. Generally, our goal is to keep each node to below $2000.00 (which is what our desktop machines cost).

3. Software

3.1 Linux, of course!

Specfically we use 2.2.17-14 kernel based on the KRUD 7.0 distribution. We use our own software for parallising applications but have experimented with PVM and MPI. In my view, the overhead for these pre-packaged programs is too high.

3.2 Costs

Linux is freely copiable.

4. Configuration

4.1 Disk configuration

This section describes disk partitioning strategies.

desktops (without windows):

hda1 - swap  (2 * RAM)
hda2 - /     (4 GB)
hda3 - /home (remaining disk space)
hdb1 - /maxa (total disk)
hdd1 - /maxb (total disk)

desktops (with windows):

hda1 - /win  (total disk)
hdb1 - swap  (2 * RAM)
hdb2 - /     (4 GB)
hdb3 - /home (remaining disk space)
hdd1 - /maxa (total disk)

laptops (single disk):

hda1 - /win  (half the total disk size)
hda2 - swap  (2 * RAM)
hda3 - /     (4 GB)
hda4 - /home (remaining disk space)

farm machines:

hda1 - swap  (2 * RAM)
hda2 - /     (remaining disk space)
hdb1 - /maxa (total disk)

4.2 Package configuration

Install a minimal set of packages for the farm. Users are allowed to configure desktops as they wish.

5. Acknowledgements

The following people have been helpful in getting this HOWTO done:

• Michael Levitt ( Michael Levitt)

6. Bibliography

The following documents may prove useful to you---they are links to sources that make use of high-performance computing clusters:

• RAMBIN web page
• RAMP web page
• Ram Samudrala's research page (which describes the kind of research done with these clusters)