Configuring Encryption at Rest

Owned by Rumena Zlatkova (Deactivated)
Last updated: Feb 21, 2024 by Bala Harish A(AC)
7 min read

About Encryption at Rest

Swarm provides an option to encrypt all user data on drive volumes. Swarm encrypts the data as it writes it to the drive and decrypts it on access. Because this occurs down at the kernel level, the effect is invisible: there is no difference in accessing encrypted versus unencrypted objects. Encryption is controlled entirely through [disk] settings in the configuration, but these cannot be changed dynamically (using the Swarm UI or SNMP).

Warning

Due to the high CPU and IO requirements of encryption at rest, using this feature is not recommended in storage virtual machine deployments.

What it Protects

Swarm volumes generally contain sensitive and proprietary client information. Implementing encryption at rest provides two types of protection:

  • Security for the data on all removed and failed physical drives.

  • The ability to render all data in a cluster inaccessible by purging the encryption key. 

Important

Swarm data is only encrypted when “at rest” (stored physically on the Swarm drives). The following is not encrypted:

  • objects within Swarm memory

  • objects in network transit between Swarm volumes or clusters

  • object metadata sent to Elasticsearch for indexing

What it Does

Encryption exists at the drive level, not during transmission. This is how encryption at rest works:

  • Data on each new drive is encrypted with an administrator-supplied key Swarm accesses from the volume.cfg/cluster.cfg file (Swarm does not persist any copies of the keys within the cluster).

  • The data on an encrypted drive removed from a chassis cannot be accessed anywhere, without the administrator-supplied key.

  • Swarm can access the data on an encrypted drive moved to another cluster by using the administrator-supplied key. 

  • All data on encrypted Swarm volumes are permanently and safely inaccessible if the administrator-supplied key is destroyed. 

  • There is no method for recovering data if the administrator-supplied key is lost.

Caution

DataCore cannot restore lost keys, and there is no master key. Swarm cluster administrators are solely responsible for managing these keys, and they must protect the key by storing a copy in a safe and secure location.

Key Concepts

Volumes encrypting data make use of encryption keys, and Swarm does not reformat existing Swarm volumes to enable encryption. Here are some implications:

Example: Suppose there are three nodes in an existing source cluster and it is decided to enable encryption at rest. Update the configuration with the needed settings. Add a fourth node (Node 4) to the cluster and boot it individually. Nodes 1 through 3 do not have the encryption settings because they have not been rebooted, but Node 4 does have the settings to apply to new volumes. Since this node has booted with fresh, non-Swarm drives, Node 4's volumes are all encrypted at rest. For each request Node 4 receives, it uses encryption keys to write and read from the encrypted volumes. That is, if Node 3 needs to write an object to Node 4, Node 4 uses keys to store the object with encryption automatically. Given a remote replication cluster, the principle is the same. The source cluster (with Node 4) tells the target cluster to fetch specific objects for replication. Node 1 at the target cluster asks Node 4 at the source cluster for some objects. Since Node 4 has the keys, it can read the object from a drive and return the object to Node 1 in the target cluster. How Node 1 in the target cluster stores the object is up to that node's configuration. Because encryption is completely local, the target cluster volumes are encrypted if set up.

Performance Impact

Encryption while reading and writing is a CPU-intensive activity and can typically expect to see a 10-30% performance overhead depending upon workload and hardware. The 2010 Intel Core processor family and later include special AES-NI instructions that implement the more complex and performance intensive steps of AES encryption. These instructions are implemented by AMD in processors starting late in 2011. Swarm's kernel takes advantage of the AES-NI instruction set if available in the CPU.

For more information, see https://software.intel.com/en-us/articles/intel-advanced-encryption-standard-instructions-aes-ni and https://wikipedia.org/wiki/AES_instruction_set .

Best Practices

  • Approach encryption as a cluster-wide setting. Although it is possible to enable encryption separately on each chassis (each has a separate volume.cfg), use the same encryption settings cluster-wide.

    • Exception: Use chassis-specific encryption keys, if needed.

  • Provide physical security measures to verify encryption keys are protected (on the physical chassis with USB sticks, or on the CSN) from both theft and loss.

  • Update the configuration settings and then systematically retire and reformat existing drives to verify an entire cluster is encrypted.

  • Systematically retire and reformat existing drives so they use the new key to change encryption keys.

  • Elasticsearch recommends implementing the encryption method dm-crypt to guarantee Elasticsearch metadata content is also encrypted. Swarm does not automate or manage the encryption key process for Elasticsearch volumes.

Encryption Settings

Volume encryption settings and keys are configured through [disk] settings in the volume.cfg/cluster.cfg file.

[disk] Settings

Default

Description

[disk] Settings

Default

Description

encryptNewVolumes

0 (False)

Indicates new Swarm volumes should be encrypted with the specified primary key.

Set disk.encryptionKeyPrimaryif set to 1 (True).

encryptionKeys

{}

Swarm uses them to read volumes encrypted with these keys.

Example:

{'cluster_key_5_15_2016': 'a24f8ec391ab3341',
'cluster_key_5_12_2015': 'de3498245ce8bf89'}

Use the disk.encryptionPrimaryKey setting to select which key in disk.encryptionKeys is “primary”. The other keys are considered to be “secondary”, and they are only used to read/write volumes already encrypted.

encryptionKeyPrimary

empty string

String. Contains the name of the encryption key to be used to format new encrypted Swarm volumes.

encryptionCipher

aes-xts-plain64

String. Indicates the preferred cipher algorithm to use when formatting new volumes. Keep the default unless directed otherwise.

Valid values: aes-xts-plain64, aes-cbc-essiv

encryptionKeySize

512

Integer (in bits). Indicates the preferred key size to use when formatting new volumes. Keep the default unless directed otherwise.

Valid values: 128, 256, 512

encryptionHash

sha512

Indicates the preferred hash algorithm to use when formatting new volumes. Keep the default unless directed otherwise.

Valid values: sha256, sha512

encryptionIterationTime

5000

Integer (in milliseconds). Indicates the maximum iteration time LUKS uses to derive a key to use when formatting new volumes. Keep the default unless directed otherwise.

Valid values: positive integers ≥ 1000

Generating Encryption Keys

Create encryption keys using tools available from the standard Linux and Windows OS package repositories.

Here is an example script for generating a key:

Generate Encryption Key
#!/usr/bin/python import random random.seed() print "%064x" % random.getrandbits(512)

Enabling Encryption on a Cluster

Swarm begins encrypting every new (not formatted by Swarm) volume it detects and formats when enabling encryption for a cluster and setting up the required encryption keys.

To implement encryption across a cluster, do the following:

  1. Open the cluster configuration (node.cfg/cluster.cfg file) for editing.

  2. Add appropriate values for these settings:
    [disk]
    encryptNewVolumes = true
    encryptionKeys = {'key_2018-03-19': 'a24f8ec391ab3341', 'key_2016-09-27': 'de3498245ce8bf89'}
    encryptionKeyPrimary = key_2018-03-19

  3. Secure copies of the encryption keys.

  1. Reboot the cluster to activate the settings change.

  2. Add any new hardware, which Swarm formats for encryption.

Encrypting Existing Swarm Volumes

Reformat and remount the volumes if retiring volumes to implement encryption at rest. Contact DataCore Support for a utility to streamline this process. (v10.1)

Alternatively, see Encrypting Existing Swarm Volumes Manually.

Troubleshooting Encryption

Unmountable Volumes

This is how Swarm handles volumes it cannot mount:

Encrypted Swarm volume Swarm cannot open

This occurs if the key is missing. Swarm ignores the volume and logs an error.

Encrypted non-Swarm volume

Swarm ignores the volume and does not format it.

Unencrypted non-Swarm volume

A 60-second countdown timer is displayed on the physical console to provide time to prevent erroneous formatting if non-blank storage volume that appears to have data from other operating systems is detected when a Swarm Storage node boots. The most likely scenario for this happening is if a server or laptop is inadvertently PXE booted from the storage network. The countdown timer provides the opportunity to power off the system before the drive is formatted.

After the 60-second countdown timer expires, Swarm formats the volume.

Encryption Status and Logging

To see the encryption status of the volumes, view the SNMP volumes table for the cluster. Alternatively, view the status on the Chassis Details page () of the Storage UI or the Node Status page () of the legacy Admin Console: Swarm puts "(encrypted)" after the volume ID of volumes that are encrypted. Swarm writes a console message if it cannot open a volume because of a problem with the encryption key.

On startup or hotplug events, this is how Swarm logs encryption:

  • Each non-encrypted drive that is mounted is logged to the console, “Mounted non-encrypted volume /dev/sda” if encryption is enabled (encryption.primaryKey is set). An error is logged to syslog.

  • An error is logged to the console, “Unable to mount encrypted volume /dev/sda” if an encrypted volume cannot be mounted (such as for a missing key). An error is logged to syslog.

  • The log entry of that volume includes the encryption status of the volume when a volume is mounted.

  • During hotplug events, a volume with a non-Swarm encrypted partition is mounted and formatted as a Swarm volume immediately.

Disabling Encryption on a Cluster

The change affects how Swarm formats any new volumes it detects when disabling encryption on a cluster. Existing encrypted volumes, even if moved (hot plugged), remain encrypted and accessible only with encryption keys.

To remove encryption entirely from a cluster, do the following:

  1. Edit the cluster configuration (.cfg) file.

    1. Disable new volume encryption.

    2. Remove the primary encryption key designation. (This makes it a secondary encryption key.)

  2. Reboot all nodes requiring unencrypted volumes (to activate the settings change).

  3. Systematically retire all encrypted volumes. Swarm relocates the data to other volumes.

  4. Add back each volume.

    1. Swarm formats each new volume as unencrypted and mounts it when detected.

    2. As new data fills up each volume, it is unencrypted and requires no key.

Decommissioning an Encrypted Cluster

Perform the following to decommission an encrypted cluster, guaranteeing none of the encrypted data is ever retrievable: 

  1. Delete the encryption keys from the cluster configuration (.cfg) file. 

  2. Destroy all copies of the encryption keys.

  1. Reboot the cluster.
    Without the keys, Swarm cannot mount the volumes, so all are out of service.

  2. Remove, reformat, and repurpose the volumes.

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.