The S3 Backup Restore Tool is the standalone utility for performing DR from the S3 backup bucket, either to the original cluster or to an empty cluster that is meant to replace the original. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443815149.

Once the data is backed up in S3, the restore tool allows both examining a backup and controlling how, what, and where it is restored:

• List all domains and buckets, or within a domain, with the logical space used for each.

• List all objects within a bucket or unnamed objects in a domain, optionally with sizes and paging.

• Restore either the complete cluster contents or a list of domains, buckets, or individual objects to restore. 

• Rerun the restore, should any part of it fail to complete.

• Partition the restoration tasks across multiple instances of the command line tool, to run them in parallel.

Installing the Restore Tool

The S3 Backup Restore tool has a separate install package available by Support request. Install it on one or more (for parallel restores) systems where the restore processes run.

Preparation (One-Time)

The swarmrestore package is delivered as a Python pip3 source distribution. Each machine needs to be prepared to be able to install this and future versions of swarmrestore.

1. As root, run the following command:

   yum install python3

2. Verify version 3.6 is installed:

   python3 --version

3. Upgrade pip

Installation

Uninstall Python 2 generation of the tool (caringo-swarmrestore-1.0.x.tar.gz) if installed:

Rerun this installation when a new version of swarmrestore is obtained:

1. Copy the latest version of the swarmrestore package to the server.

2. Run the following as root:

3. swarmrestore is likely in /usr/local/bin and is already in the path.

4. Repeat for any additional servers if planning to perform partitioning for parallel restores.

Restore Tool Settings

The tool uses a configuration file, .swarmrestore.cfg, because the file contains sensitive passwords. The tool warns if the configuration file is not access-protected (chmod mode 600 or 400).

The configuration file follows the format of Swarm Storage settings files, using sections listing name = value pairs. These setting names map to the S3 Backup feed definition, where the values have the same meaning.

1. Locate the sample configuration file where it is installed:

2. Copy the file into the home directory and rename it, and open it for editing:

sample-.swarmrestore.cfg

Additional Restore Configuration

• Gateway: Add the IP of the machine where the Restore tool runs to the https://perifery.atlassian.net/wiki/spaces/public/pages/2443810201 scsp.allowSwarmAdminIP if communicating with a Swarm cluster via Gateway.

Using the Restore Tool

The restoration tool runs using batch-style operation with commands given on the command line. The tool logs the actions to the log file or server in the log configuration section. The restoration tool uses the following command format:

Options:

• --help - Displays a summary of the current configuration. 

• --version - Reports the version of the tool.

• ls --help - Displays help on the ls command, for listing and enumerating. 

• restore --help - Displays help on the restore command, for selective restore and disaster recovery. 

ls Subcommand

Enumeration and selection are handled by the ls command, which is modeled after the Linux command ls and whose results are captured with standard Linux stdout. Use the command to visualize what domains and buckets are backed up in S3 and are available to be restored. The output is sorted by name and interactively paginated to help manage large result sets by default.

The ls subcommand has this format:

Command options, which can be combined (for example, -Rvl): 

• -R or --recursive - Recursively lists the given domain or bucket, or else the entire cluster. Without this option, the command lists the top-level contents of the object.

• -v or --versions - List previous versions of versioned objects. Versions are not listed by default.

• -l or --long - Lists details for each item returned in the output:

  • Creation date

  • Content length of the body

  • ETag

  • Archive status:

    • AN - Archived; not available for restoration

    • AR - Archived with an archive restore in progress; not available for restoration

    • AA - Archived with a copy available for restoration

    • OK - Not archived and fully available

  • Objectspec

  • Alias UUID, if the object is a domain or bucket

• <objectspec> - If none, the command runs across the entire contents of the S3 backup. If present, filters the command to a specific domain or bucket (context object) in Swarm. Use this format:

When running the command without any options, it returns the list of domains that are included in this S3 bucket for the Swarm cluster:

Run a command like this if wanting a complete accounting of every object backed up for a specific domain, redirecting to an output file:

restore Subcommand

Object restoration and verification is handled by the restore subcommand, which has the following format:

<objectspec> - If none, applies the command to the entire cluster backup. If present, filters the command to a specific domain, bucket, object, or object version.
To target a command to a specific context (domain/bucket) or content object in Swarm, format the type of object as follows:

Any number of command options can be used, and the short forms may be combined with a single dash (-Rv). The <objectspecs>, -R, and -v options iterate over objects the same way as the ls command.

Options:

• -R or --recursive - Recursively restore domains, buckets, or the entire cluster with an empty object spec. See above for what is iterated over when -R is not used.

• -v or --versions - Include previous versions of versioned objects. They are not included by default.

• -f <file> or --file <file> - Use objectspecs from a file instead of the command line.

• -p <count>/<total> or --partition <count>/<total> - Partition work for a large restore job (but every instance restores buckets and domains before objects).

  • Example: To run 4 instances in parallel, configure each option to be one of the series: -p 1/4, -p -2/4, -p -3/4, -p -4/4

• -n or ---noop - Perform the checking of a restore, but do not restore any objects.

  • Does not change the cluster state. The option can be used before and after a restore, as both a pre-check and a verification.

• <objectspecs> - Any number; newlines separate objects. If none, the top level of the cluster’s backup contents is the scope.

  • Using no object specification with the command options -Rv causes Swarm to restore all backed up objects in the entire cluster, including any historical versions of versioned objects.

What is Restored: Restore copies an object from S3 to the cluster if the cluster object is missing or else older than the S3 object. Note: context objects restore before the content they contain: restore first restores any domains or buckets needed before restoring objects within them.

Output of Restore: At the end of the restoration, the tool reports the number of objects restored and the number of objects skipped, for being either identical to or newer than the backed up copy. The command output lists each object spec with its status:

• current - The object was not restored because the target cluster already has the same version of the object.

• older - The object was not restored because it is older than the one in the target cluster.

• obsolete - The object was not restored because the cluster does not allow the object to be written. Usually it means the object is deleted.

• needed - The object needs restoration, but the -n option was used.

• restored - The object was successfully restored.

• nocontext - The object cannot be restored because its parent domain or bucket cannot be restored.

• failure - The object cannot be restored. Consult the logs for details.

• archived - The object is archived and the restore tool is not configured for archive restoration. This is a failure condition.

• initiated - The object is archived and the tool has issued an object restoration request. See the Amazon S3 API RestoreObject Request Syntax. This is also a failure condition, but the object is counted in the archive retrieval initiated stats. It is these operations that incur expense to the bucket owner by the restore tool.

• ongoing - The object is in archive and a restoration request has already been initiated. Restoration from archive is in progress. This is also a failure condition.

Rate of Restore: Restoration may take a long time run, especially if recursion (-R) is used on domains or buckets. To boost the rate of restore, install the S3 Backup Restore tool on multiple servers and then run the restore command with partitioning parameters (-p) across all instances of the tool, which allows restoring faster in parallel, with minimal overlap.

Headers for Audit: When the S3 Backup feed writes an object to the S3 bucket, it adds to the S3 copy a header (Castor-System-Tiered) that captures when and from where the object was tiered. When the S3 Backup Restore tool writes the S3 object back to Swarm, it includes that S3 header and then adds another one of the same, to capture when and from where the object was restored. These paired headers (both named Castor-System-Tiered) provide the audit trail of the object's movement to and from S3. Swarm persists these headers but does not include them in Entity-MD5 or Header-MD5 calculations. The dates are of the same format as Last-Modified (RFC 7232, section 2.2). See https://perifery.atlassian.net/wiki/spaces/public/pages/2443820996.

Audit Headers