x86 with Pentium-class CPUs

Three (to guarantee adequate recovery space in the event of node failure)

Nodes must connect to switches configured for multicasting

To accept incoming syslog messages

USB flash drive or PXE boot

One Gigabit Ethernet NIC with one RJ-45 port *

One hard disk

2 GB + ( 0.5 GB × number of volumes (light loads and no erasure-coding) )

To synchronize clocks across nodes

x86 with Intel Xeon or AMD Athlon64 (or equivalent) CPUs

Three or more

Nodes must connect to switches configured for multicasting

To accept incoming syslog messages

USB flash drive or PXE boot

Two Dual Gigabit Ethernet NICs with two RJ-45 ports for link aggregation (NIC teaming)

Important: Mixing network speeds among nodes is not supported. Do not put a node with a 100 Mbps NIC in the same cluster with a node containing a 1000 Mbps NIC.

One to four standard non-RAID SATA hard disks

4 GB + (0.5 GB × number of volumes)

To synchronize clocks across nodes

4 GB

33 million

16 million

8 GB

66 million

33 million

12 GB

132 million

66 million

In general, plan to support EC with more and faster CPU cores.

Scale up for larger objects: Larger objects require more memory to manage erasure sets. How many EC objects can be stored on a node per GB of RAM depends on the size of the object and the encoding specified in the configuration. The erasure-coding manifest takes two index slots per object, regardless of the type of object (named, immutable, or alias). Each erasure-coded segment in an erasure set takes one index slot. Larger objects can have multiple erasure sets, so multiple sets of segments exist. In k:p encoding (where integers for the data (k) and parity (p) segment counts), there are p+1 manifests (up to the ec.maxManifests maximum). That means 3 manifests for 5:2 encoding. With the default segment size of 200 MB and a configured encoding of 5:2:

• 1-GB object: (5+2) slots for segments and (2+1) for manifests = 10 index slots

• 3-GB object: 3 sets of segments @ 10 slots each = 30 index slots

Increase for Overlay Index: Larger clusters (above 3 nodes by default) need additional RAM resources to take advantage of the Overlay Index. For erasure-coded objects, allocate 10% additional RAM to enable the Overlay Index.

Network use by the requesting SAN is an important factor in EC performance. The requesting SAN must orchestrate k+p segment writes (for an EC write) or k segment reads (for an EC read). Because these segment reads and writes must occur at the same rate, the slowest one slows the overall request. Consequently, when a cluster experiences a lot of EC requests, nodes can play different roles and slower nodes can affect multiple requests.

Enterprise-level

The critical factor is whether the hard disk is designed for the demands of a cluster. Enterprise-level hard disks are rated for 24x7 continuous-duty cycles and have time-constrained error recovery logic suitable for server deployments where error recovery is handled at a higher level than the onboard controller.

In contrast, consumer-level hard disks are rated for desktop use only; they have limited-duty cycles and incorporate error recovery logic that can pause all I/O operations for minutes at a time. These extended error recovery periods and non-continuous duty cycles are not suitable or supported for Swarm deployments.

Rated for continuous use

The reliability of hard disks from the same manufacturer vary, because the disk models target different intended use and duty cycles:

• Consumer models targeted for the home user and assume the disk is not being used continuously. These disks do not include the more advanced vibration and misalignment detection and handling features.

• Enterprise models targeted for server applications and tend to be rated for continuous use (24x7) and include predictable error recovery times, as well as more sophisticated vibration compensation and misalignment detection.

Large on-board cache

Independent channels

Fast bus

Optimize the performance and data throughput of the storage subsystem in a node by selecting disks with these characteristics:

• Large buffer cache: Larger onboard caches improve disk performance.

• Independent disk controller channels: Reduces storage bus contention.

• High disk RPM: Faster-spinning disks improve performance.

• Fast storage bus speed: Faster data transfer rates between storage components, a feature incorporated in these types:

  • SATA-300

  • Serial Attached SCSI (SAS)

  • Fibre Channel hard disks

The storage bus type in the computer system and hard disks often drives the use of independent disk controllers.

• PATA: Older ATA-100 and ATA-133 (or Parallel Advanced Technology Attachment [PATA]) storage buses allow two devices on the same controller/cable. Bus contention occurs when both devices are in active use. Motherboards with PATA buses only have two controllers. Some bus sharing must occur if more than two disks are used.

• SATA: Unlike PATA controllers, Serial ATA (SATA) controllers and disks include only one device on each bus to overcome the previous bus contention problems. Motherboards with SATA controllers typically have four or more controllers. Recent improvements in Serial ATA controllers and hard disks (commonly called SATA-300) have doubled the bus speed of the original SATA devices.

Avoid highest capacity

Improve the failure and recovery characteristics of a node when a disk fails by selecting disks with server-class features but not the highest capacity.

• Higher capacity means slower replication. Consider the trade-off between the benefits of high-capacity disks versus the time required to replicate the contents of a failed disk when choosing the disk capacity in a node. Larger disks take longer to replicate than smaller ones, and that delay increases the business exposure when a disk fails.

• Delayed errors mean erroneous recovery. Unlike consumer-oriented devices, for which it is acceptable for a disk to spend several minutes attempting to retry and recover from a read/write failure, redundant storage designs such as Swarm need the device to emit an error quickly so the operating system can initiate recovery. The entire node may appear to be down, causing the cluster to initiate recovery actions for all disks in the node - not the failed disk if the disk in a node requires a long delay before returning an error.

• Short command timeouts mean less impact. The short command timeout value inherented in most enterprise-class disks allows recovery efforts to occur while other system disks continue to support system disk access requests by Swarm.

JBOD, not RAID

Controller-compatible

• Evaluate controller compatibility before each purchasing decision.

• Buy controller-compatible hardware. The more types of controllers in a cluster, the more restrictions exist on how volumes can be moved. Study these restrictions, and keep this information with the hardware inventory.

• Avoid RAID controllers. Always choose JBOD over RAID when specifying hardware for use in Swarm. RAID controllers are problematic, for these reasons:

  • Incompatibilities in RAID volume formatting

  • Inability of many to hot plug, so the ability to move volumes between machines is lost

  • Problems with volume identification (disk lights)

• HP Proliant systems with P840 RAID controllers - Swarm performance degrades when this card runs in HBA mode. For best performance, use only single raid 0s per disk, but volumes in this system cannot hotplug.

Track kernel mappings

Keep track of controller driver to controller firmware mappings in the kernel shipped with Swarm Storage. This is particularly important when working with LSI-based controllers (which are the majority), because a mismatch between driver and firmware in LSI's Fusion MPT architecture can introduce indeterminate volume behavior (such as good disks reporting errors erroneously due to a driver mismatch).

Monitor the hardware inventory with special attention to the disk controllers when administering the cluster. Some RAID controllers reserve part of the disk for controller-specific information (DDF). Once Swarm formats a volume for use, it must be used with a chassis having that specific controller and controller configuration.

Many maintenance tasks involve physically relocating volumes between chassis to save time and data movement. Use the inventory of the disk controller types to spot when movement of formatted volumes is prohibited due to disk controller incompatibility.

To determine controller compatibility safely, test outside of the production cluster.

1. Set up two spare chassis, each with the controller being compared.

2. Format a new volume in Swarm in the first chassis.

3. Move the volume to the second chassis and watch the log for error messages during mount or for any attempt to reformat the volume.

4. Retire the volume in the second chassis and move it back to the first.

5. Watch for errors or attempts to reformat the volume.

6. Erase the disk using dd and repeat the procedure in reverse, where the volume is formatted on the second chassis if all goes well.

Swap volumes between these chassis within a cluster if no problems occur during this test. Do not swap volumes between these controllers if this test runs into trouble.