Reducing Redundancy for Lesser Content

To make the most cost-effective use of your storage footprint, you can reduce the redundancy of Swarm's content protection. These are types of content that may be candidates for reduced redundancy:

  • Nightly backups, which you can restore from using any save set within a few days

  • Intermediate files in a multi-step operation

  • Files that can be recreated on demand, as needed

  • Content of little business impact if lost

Balancing the downside to losing the file versus the likelihood that a particular one is needed, protection can be lowered as described below.

How to Enable Reduced Redundancy?

Follow the below steps to enable reduced redundancy:

  1. First, lower the cluster-wide minimum for replication, which sets the absolute lower bound for the cluster: Change policy.replicas: min  to 1, if it is a 2 or higher.

    policy.replicas min:1 max:<int> default:<int>

Note

The minimum is not the default. This setting can be changed dynamically, using the UI or SNMP. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443812082.

Caution

Parity of 1 for erasure coding lowers protection and so should not be used as your cluster's default (policy.ecEncoding=N:1); as N increases, N:1 approaches the protection of having only a single copy of the object in the cluster.

  1. Best practice: If a category of lesser content regularly needs to be stored, provide its own container (context) and set reduced redundancy on that container only.

    1. Create or designate a specific domain or bucket to be dedicated to this content.

    2. Update the domain or bucket properties to enforce single replicas, no erasure coding, and no versioning, using the Content UI or a manual command:

      curl -iL -XPOST --post301 --data-binary "" -H "Policy-Replicas: default:1" -H "Policy-ECEncoding: disabled" -H "Policy-Versioning: disabled" "http://{cluster}/myBucket?domain=myDomain"

      See https://perifery.atlassian.net/wiki/spaces/public/pages/2443812150.

Why Disable Erasure Coding?

When storing a category of lesser content with reduced redundancy, erasure coding should not be used. If an erasure-coded object degrades (loses more than p segments in a k:p encoding, such as 2:1), then the remaining segments still take up cluster space, and the health processor continues to look for k segments to attempt to reconstruct missing ones. With whole replica encoding, losing lesser data results in the entire object being removed from the cluster, reclaiming that space and avoiding extra processing.

  1. Apart from your special domain or bucket, when you have an individual file (such as a backup) that is a candidate for reduced redundancy, write it with lifepoint headers that lower redundancy. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443821972.
    In this example, a chunked upload (which must be EC encoded) has two lifepoints: the first lifepoint specifies an EC encoding that expires in one day, and the second specifies that the cluster needs to keep only one replica after that.

    Transfer-Encoding: chunked Lifepoint: [Wed, 4 Apr 2019 15:59:02 GMT] reps=2:1 Lifepoint: [] reps=1

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.