The hardware platform for the Elasticsearch servers can scale from one virtual machine to multiple physical machines. Scaling Elasticsearch involves both the number of nodes and the number of shards for each index.

Number of Nodes and Shards

Elasticsearch data is organized into indices, which are made up of one or more shards. Each shard is an instance of a Lucene index, which is like a self-contained search engine that indexes and handles queries for one specific portion of Elasticsearch data. Elasticsearch distributes these shards across the cluster of nodes. Refer to the Elasticsearch guidelines.

Scaling is critical: Having too few nodes slows indexing and searching, and too few shards can make them expand excessively large (over 50 GB), which limits the cluster's ability to recover from failures.

* Some workloads like Veeam Backup produce a high listing load which requires more elasticsearch nodes, up to one node per shard. Veeam also produces a high object delete and retention period update load, resulting in deleted documents being almost half of the elasticsearch index. Deleted documents are gradually “merged” out of the index, but they have the same impact on listing performance and sizing requirements as much as live documents.

* Very large object sizes may have the effect of many more objects. An application creates an equal number of Swarm objects if an application backs up large VM snapshots to Swarm. A large object count in Swarm is created, needing a larger ES cluster, if the backup application is chunking and storing every 5MB of each snapshot as a separate Swarm object (which has separate metadata).

Increasing the Shards

The Swarm setting search.numberOfShards allows adjusting the number of shards on new search indices as the implementation is scaled. The setting has no effect on existing indices, it must be set before creating a new Search Feed. (v12.0)

To adjust the shard number, update the configuration and start a new feed based on the new shard count.

1. Set search.numberOfShards to the new value in the cluster.cfg.

2. Reboot the cluster, or apply it to the running cluster using the command in the Support Tools bundle, as directed by Support: 

   swarmctl -a -C search.numberOfShards -V 20

3. Create a new search feed using the new shard count:

   • No listing downtime: Create a new feed, make it Primary once it completes, and then delete the old one.

   • With listing downtime: Delete the existing ES index and Refresh the feed.

Single-Node ES Clusters

The default shard protection (1 replica) causes a permanent yellow status when an Elasticsearch cluster has one node. This is only suitable for demo or proof of concept deployments. The yellow status occurs because the primary and replica shards are recommended to be hosted on separate nodes. To deploy a single-node ES cluster, explicitly change the shard replicas to zero. 

Configure the indices to have zero replicas using the script provided if configuring an ES cluster with one ES data node:

Use the help command to view the complete options for changing the number of replicas on existing indices.