Swarm has an Overlay Index, which provides increased scalability in a storage cluster. When enabled, the Overlay Index tracks object locations in a shared cluster-wide index, minimizing multicast traffic in the cluster network.

The Overlay Index creates a dynamic master index in RAM from the local node indexes of all existing objects in the cluster. The Overlay Index locates the nodes containing the targeted object and directs the request to an appropriate node without multicasting to locate it when an SCSP request for an existing object is sent to a Swarm storage cluster. This process minimizes the multicast traffic and associated processing in a storage cluster.

The knowledge of existing nodes in the cluster is refreshed regularly and the Overlay Index evaluates whether changes to its model of the cluster are required every 5 minutes, verifying the Overlay Index is optimally spread across all available nodes in the cluster as new nodes are added or existing nodes are removed from the cluster. Adjustments to the Overlay Index's distribution due to changes in the nodes participating in the Overlay Index (particularly immediately after a cluster reboot) happen quickly but are not instantaneous. Distribution adjustments should not affect client activity.

In the Swarm UI, the Overlay settings appear under the Index section of the Cluster Settings page:

By default, the Overlay Index is enabled for clusters with 3 or more nodes. the Overlay Index can be configured without rebooting the cluster by setting it as above in Cluster Settings or through its SNMP OIDs:

Requirements

The Overlay Index provides these features:

• Enables the Health Processor to quickly discover and remove over-replication generated by failed volume recovery (FVR) and erasure coding recovery (ECR), which maximizes the disk space on cluster nodes.

• Reduces the multicast traffic in a storage cluster network.

• Provides faster 404: Not Found response time for nonexistent immutable and aliased objects when Optimize 404 is enabled.

To support these features, the Overlay Index requires additional index slots in cluster node RAM to store the additional index information. The Overlay Index is not populated and the cluster continues to use multicast to locate objects if there are not enough nodes or RAM resources to hold the additional index information. Additional RAM must be added and the nodes rebooted to take advantage of the Overlay Index if the current node indexes are full.

See the https://perifery.atlassian.net/wiki/spaces/public/pages/2443808741 for RAM requirements.

Determining Status

Check the value of the indexOverlayStatus SNMP OID to determine the current state of the Overlay Index at a given time. The status of the Overlay Index is one of the following:

• Disabled: When the overlay is turned off.

• Uninitialized: When the overlay is first activated.

• Operational: When population of the Overlay Index is in process.

• Authoritative: When the node servicing the SNMP query has determined the Overlay Index is fully populated for the entire cluster. The Optimize 404 feature only works when the Overlay Index is authoritative.

During normal operation, the state of the Overlay Index toggles between "authoritative" and "operational" as the structure of the cluster changes with new nodes being added or existing nodes being removed. The status may also indicate a variety of reasons why the Overlay Index is not operational, such as insufficient nodes or inadequate Overlay Index space.