Configuring the "duplicity" command-line tool to backup to Gateway S3

Created 4/5/2016 jamshid.afshar · Updated 2/21/2017 aaron.enfield

BACKUP WITH THE DUPLICITY COMMAND-LINE TOOL

"duplicity" is a tried and true linux utility that's great for backing up a folder or drive for archive or transfer. It does incremental backups too, so you can run it on a periodic basis to quickly and reliably backup a large folder as changes are made.

Here is how easy it is to use it with your Swarm storage.

1. To install just:

Red Linux: sudo yum install -y duplicity

Orange Linux: sudo apt-get install -y duplicity

macOS: brew install duplicity

Ubuntu on Windows (Freezing Demon): apt-get install -y duplicity

Btw if anyone gets that last one to work, please blow our minds and drop us a note.

2. Create a domain (e.g. mydomain.cloud.caringo.com), a bucket in the domain (e.g. backups-duplicity), and an S3 token via the REST api Creating Tokens or the CloudScaler Content Portal Setting Tokens.

3. Configure the generated token and secret as the access key and secret key for "duplicity":

export AWS_ACCESS_KEY_ID=50a60f34811739e938a2dea56c73671b export AWS_SECRET_ACCESS_KEY=2tyKk6zFp77fbY8UJMfXLimGK6CQsb5yf6PRHtCB

You can create a boto config file instead of using environment variables, but the location seems to vary by OS. This works on CentOS:

cat 

> ~/.boto <<EOF [Boto] is_secure = True [Credentials] aws_access_key_id = {S3 TOKEN} aws_secret_access_key = {S3 SECRET} EOF

4. Now you're ready to backup a directory, e.g. ~/Pictures/SFW, to your domain and bucket under the prefix "mypics" (duplicity refers to this as a collection). First be sure the machine where you're running "duplicity" can lookup your domain via DNS or /etc/hosts. A request like this must connect to your CloudScaler S3: "curl -i https://mydomain.cloud.caringo.com".

$ duplicity --no-encryption ~/Pictures/SFW s3://mydomain.cloud.caringo.com/backups-duplicity/mypics

5. Then list the contents to verify it has what you expect:

$ duplicity --no-encryption list-current-files s3://mydomain.cloud.caringo.com/backups-duplicity/mypics

Tip: "No signature chains found" means it doesn't find any backups in the "backups-duplicity" bucket under the prefix "mypics"

6. Finally, you can restore the backup to a different location to verify it is correct. Remember, untested backups are digitized heartbreak.

$ duplicity --no-encryption s3://mydomain.cloud.caringo.com/backups-duplicity/mypics /tmp/Pictures/ ...

$ diff ~/Pictures/SFW /tmp/Pictures

(no output means the directories are identical)

CAVEATS AND COMMAND-LINE OPTIONS

The "--no-encryption" flag is needed because "duplicity" expects you to use client-side (i.e., warrant-incompatible) encryption. A future article will show how to configure GPG, but remember: IF YOU LOSE YOUR KEYS, YOU'VE LOST YOUR DATA FOREVER.

Speaking of sensible secure-by-default options, "duplicity" also expects "https" access to the S3-compatible service. If you haven't set up an "https" proxy in front of your CloudScaler (may Schneier forgive you), you can use "http" but you must always use the option "--s3-unencrypted-connection" AND is_secure must be set to False in the boto config file.

Warning: "duplicity" appears to ONLY WORK WITH DEFAULT PORTS (443 for https or 80 for http)! Configure your CloudScaler S3 port by setting "bindPort=80" under the "[s3]" section in /etc/caringo/cloudgateway/gateway.cfg.

Complete details, like using "-v 9" for debug info, are at http://duplicity.nongnu.org/docs.html.

Thanks for reading, if you try it out please drop us a comment.