consulbackinator

Command line Consul KV backup and restore utility


Keywords
backup, consul, consul-acl-backup, consul-backup, consul-kv-backup, consul-query-backup, docker, encrypted-backup, nomad
License
MPL-2.0
Install
brew install consulbackinator

Documentation

Go Report Card Mozilla Public License GoDoc

consul-backinator

Summary

Flexible Consul KV pair backup and restore tool with a few unique features. This was written for and tested in a production environment.

Note

There was a potentially breaking change in the operation of this tool on May 09, 2016 when the different operations were broken out into sub commands to simplify the flag listings. The functionality remains the same and previous backup files are not affected. However the command options and structure have changed and any scripts which embedded this tool will need to be updated.

Key Features

  • Written in Golang using the official Consul API
  • No limits on the number of keys that can be backed up or restored
  • Backup files are written as gzip compressed and AES256 encrypted JSON data
  • Data integrity validation via HMAC-SHA256 signature of the raw data
  • Optional path transformation (path replacement) on backup and/or restore
  • Clean well documented code that's simple to follow

Installing

With a proper Go environment simply run:

go get -u github.com/myENA/consul-backinator

If you have glide (https://github.com/Masterminds/glide) ...

git clone https://github.com/myENA/consul-backinator.git
cd consul-backinator
./build.sh -i

Usage

Summary

ahurt$ ./consul-backinator --help
usage: consul-backinator [--version] [--help] <command> [<args>]

Available commands are:
    backup     Perform a backup operation
    dump       Dump a backup file
    restore    Perform a backup operation

Backup Options

Option Description
file The backup file target. The signature will be the same with a .sig extension appended. The default names are consul.bak and consul.bak.sig
key The passphrase used for data encryption and signature generation. The default string password will be used if none specified. This should be a secure pseudo random string.
transform Optional argument that affects the key paths written to the backup file. See the transformation notes below for more information.
prefix Optional argument that specifies the starting point for the backup tree. The default prefix is the root / prefix. To perform a partial tree backup specify a prefix.

Restore Options

Option Description
file The source file. The default is consul.bak
key The passphrase used for data decryption and signature validation. This must match the key used when the backup was created.
delete Optionally delete all keys under the specified prefix prior to restoring the backup file. The default is false.
prefix The prefix with the delete option. The default is / root. THIS WILL DELETE ALL DATA IN YOUR KEYSTORE if not changed when using -delete.

Shared Consul Options (backup/restore)

Option Description
addr Optional consul agent address and port. The default is read from the CONSUL_HTTP_ADDR environment variable if specified or set to 127.0.0.1:8500.
scheme Optional scheme http or https used when connecting to the consul agent. The default is set to https if the CONSUL_HTTP_SSL environment variable is set to true otherwise the default is http.
dc Optional datacenter specification. The default value is the datacenter of the agent to which you are connecting.
token Optional consul access token. The default value is read from the CONSUL_HTTP_TOKEN environment variable if specified.

Dump Options

Option Description
file The source file. The default is consul.bak
key The passphrase for the backup file to be dumped. The default is password if not passed.
plain Optionally decode the backed up key values and only display the key name and decoded values without showing any metadata.

Transformations

Transformations are simple string operations and will affect the path anywhere there is a match. For example, passing -transform="foo,bar" would rewrite /apple/foo/key => /apple/bar/key as well as /orange/thing/foo/key => /orange/thing/bar/key. To avoid potential errors in transformations you should always use the most exact path possible. Using the previous example if you only wanted to affect keys under apple you should pass -transform="apple/foo,apple/bar" to prevent other paths from being modified inadvertently.

Example

ahurt$ ./consul-backinator backup -key=superSecretStuff
2016/05/09 17:14:11 [Success] Backed up 289 keys from / to consul.bak
Keep your backup (consul.bak) and signature (consul.bak.sig) files in a safe place.
You will need both to restore your data.
ahurt$ ls -la *.sig *.bak
-rw-------  1 ahurt  staff  11167 May  9 17:14 consul.bak
-rw-------  1 ahurt  staff     44 May  9 17:14 consul.bak.sig