Adding timestamps on S3 metadata

CONTRIBUTORS.txt

@@ -7,4 +7,5 @@ Bart Swedrowski
 Chris Love
 Tony Li Xu
 Mathew Kamkar
-Alessandro Pieri
+Alessandro Pieri
+Romain Hardouin

README.md

@@ -33,6 +33,8 @@ Usage
 
 You can see the list of parameters available via `cassandra-snapshotter --help`
 
+If you want to make `cassandra-snapshotter` more chatty just add `--verbose`.
+
 #### Create a new backup for *mycluster*:
 
 
@@ -43,6 +45,7 @@ cassandra-snapshotter --s3-bucket-name=Z \
                       --aws-access-key-id=X \ # optional
                       --aws-secret-access-key=Y \ # optional
                       --s3-ssenc \ # optional
+                      --verbose \ # optional
                       backup \
                       --hosts=h1,h2,h3,h4 \
                       --user=cassandra # optional
@@ -122,12 +125,124 @@ If you dont want to use incremental backups, or if for some reason you want to c
 Its not in the scope of this project to clean up your S3 buckets.
 S3 Lifecycle rules allows you do drop or archive to Glacier object stored based on their age.
 
-
 ### Restore your data
 
-cassandra_snaphotter tries to store data and metadata in a way to make restores less painful; There is not (yet) a feature complete restore command; every patch / pull request about this is more than welcome (hint hint).
+There are two types of restore:
+ * using `sstableloader` (manual or automatic)
+ * local restore: download data on the local server
+
+#### sstableloader (automatic)
+
+Mandatory parameters:
+
+ * `--target-hosts TARGET_HOSTS`: the comma separated list of hosts to restore into. These hosts will received data from sstableloader.
+ * `--keyspace KEYSPACE`: the keyspace to restore.
+
+Optional parameters:
+
+ * `--hosts`: comma separated list of hosts to restore from; leave empty for all
+ * `--table TABLE`: the table (column family) to restore; leave blank for all
+ * `--restore-dir`: where to download files from S3 before to stream them. Default: `/tmp/restore_cassandra/`
+ * `--snapshot-name SNAPSHOT_NAME`: the name of the snapshot to restore
+
+Example to restore `Keyspace1` on `10.0.100.77` with data fetched from a two nodes backup:
+
+``` bash
+cassandra-snapshotter --s3-bucket-name=Z \
+                      --s3-bucket-region=eu-west-1 \
+                      --s3-base-path=mycluster \
+                      --aws-access-key-id=X \ # optional
+                      --aws-secret-access-key=Y \ # optional
+                      restore \
+                      --hosts=10.0.2.68,10.0.2.67 \ # optional
+                      --target-hosts=10.0.100.77 \ # where to restore data
+                      --keyspace Keyspace1 \
+                      --snapshot-name 20160614202644 \ # optional
+                      --restore-dir /tmp/cassandra_restore/ # optional
+```
+
+Make sure there is enough free disk space on the `--restore-dir` filesystem.
+
+#### sstableloader (manual)
+
+If you want to restore files without loading them via `sstableloader` automatically, you can use the `--no-sstableloader` option.
+Data will just be downloaded. Use it if you want to do some checks and then run sstableloader manually.
+
+The files are saved following this format:
+
+``` bash
+<--restore-dir>/<--keyspace>/<--table>/<HOST>_<filename>
+```
+
+Example:
+
+``` bash
+/tmp/cassandra_restore/Keyspace1/Standard1/10.0.53.68_Keyspace1-Standard1-jb-1-Data.db
+```
+
+Again, make sure there is enough free disk space on the `--restore-dir` filesystem.
 
-In case you need, cassandra_snapshotter stores the ring token description every time a backup is done ( you can find it the ring file in the snapshot base path )
+#### Local restore
 
-The way data is stored on S3 should makes it really easy to use the Node Restart Method (https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html#task_ds_vf4_z1r_gk)
+If you have lots of data you probably don't want to download data on a server and then stream these data with `sstableloader` on another one server.
+The `--local` option allows you to restore data directly on the local server where the command is run. Note this procedure allows to restore only one host,
+so if you want to restore several nodes you have a to run this command on each server.
+
+On production, do not restore directly on the C* data directory (e.g. --restore-dir /var/lib/cassandra/data/ ) it's safer to restore on a different location.
+Then make some checks (Number of files seems good? The total size seems correct? Files are uncompressed? What about file permissions?) and if all is OK just `mv` the data.
+To make the `mv` command fast, just download data on the same filesystem.
+Also prior to run the command ensure that the filesystem can host the data (enough free disk space).
+
+Example:
+
+Let's say we have a filesystem mounted on `/cassandra`.
+We want to download files on `/cassandra/restore/` (no need to create the `restore` directory, `cassandra_snapshotter` will create it).
+
+The following command will restore from `10.0.2.68` on the local server into `/cassandra/restore/`:
+
+``` bash
+cassandra-snapshotter --s3-bucket-name=Z \
+                      --s3-bucket-region=eu-west-1 \
+                      --s3-base-path=mycluster \
+                      --aws-access-key-id=X \ # optional
+                      --aws-secret-access-key=Y \ # optional
+                      --verbose \
+                      restore \
+                      --hosts=10.0.2.68 \ # select the source host. Only one node is allowed in local restore.
+                      --snapshot-name 20160614202644 \ # optional
+                      --restore-dir /cassandra/restore/ \ #
+                      --keyspace Keyspace1 \
+                      --local
+```
+
+The files are saved following this format:
+
+``` bash
+<--restore-dir>/<--keyspace>/<--table>/<filename>
+```
+
+Note the filenames are *not* prefixed by `<HOST>_` because we restore from only one node in this mode, so it would be useless.
+
+Example:
+
+``` bash
+/tmp/cassandra_restore/Keyspace1/Standard1/Keyspace1-Standard1-jb-1-Data.db
+```
+
+Here are the DataStax procedures to follow when using a local restore:
+ * Cassandra v2.0: https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_backup_snapshot_restore_t.html
+ * Cassandra v2.1: https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_backup_snapshot_restore_t.html
+ * Cassandra v2.2: http://docs.datastax.com/en/cassandra/2.2/cassandra/operations/opsBackupSnapshotRestore.html
+ * Cassandra v3.x: http://docs.datastax.com/en/cassandra/3.x/cassandra/operations/opsBackupSnapshotRestore.html
+
+### Ring information
+
+In case you need, cassandra_snapshotter stores the ring token description every time a backup is done.
+
+You can find it in the `ring` file in the snapshot base path, for instance:
+
+```
+/test-cassandra-snapshots/cassandrasnapshots/20160614202644/ring
+```
 
+[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/tbarbugli/cassandra_snapshotter/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

cassandra_snapshotter/__init__.py

@@ -1,3 +1,3 @@
-__version__ = '1.0.0'
+__version__ = '1.1.0'
 __maintainer__ = 'Tommaso Barbugli'
 __email__ = '[email protected]'

cassandra_snapshotter/agent.py

@@ -69,7 +69,11 @@ def upload_file(bucket, source, destination, s3_ssenc, bufsize, reduced_redundan
             if mp is None:
                 # Initiate the multi-part upload.
                 try:
-                    mp = bucket.initiate_multipart_upload(destination, encrypt_key=s3_ssenc, reduced_redundancy=reduced_redundancy)
+                    mtime_epoch = os.path.getmtime(source)
+                    atime_epoch = os.path.getatime(source)
+                    ctime_epoch = os.path.getctime(source)
+                    file_mtime = time.strftime('%Y-%m-%d:%H:%M:%S:%Z', time.localtime(mtime_epoch))
+                    mp = bucket.initiate_multipart_upload(destination, encrypt_key=s3_ssenc, reduced_redundancy=reduced_redundancy, metadata={'modified': file_mtime, 'mtime': mtime_epoch, 'atime': atime_epoch, 'ctime': ctime_epoch})
                     logger.info("Initialized multipart upload for file {!s} to {!s}".format(source, destination))
                 except Exception as exc:
                     logger.error("Error while initializing multipart upload for file {!s} to {!s}".format(source, destination))

cassandra_snapshotter/main.py

@@ -3,7 +3,9 @@
 # From system
 from collections import defaultdict
 from fabric.api import env
+import os.path
 import logging
+import sys
 
 # From package
 from .snapshotting import (BackupWorker, RestoreWorker,
@@ -94,8 +96,8 @@ def list_backups(args):
     path_snapshots = defaultdict(list)
 
     for snapshot in snapshots:
-        base_path = '/'.join(snapshot.base_path.split('/')[:-1])
-        path_snapshots[base_path].append(snapshot)
+        dir_path = os.path.dirname(snapshot.base_path)
+        path_snapshots[dir_path].append(snapshot)
 
     for path, snapshots in path_snapshots.iteritems():
         print("-----------[{!s}]-----------".format(path))
@@ -111,9 +113,8 @@ def restore_backup(args):
         args.aws_secret_access_key,
         args.s3_base_path,
         args.s3_bucket_name,
-        get_s3_connection_host(args.s3_bucket_region)
+        get_s3_connection_host(args.s3_bucket_region),
     )
-
     if args.snapshot_name == 'LATEST':
         snapshot = snapshots.get_latest()
     else:
@@ -123,14 +124,32 @@ def restore_backup(args):
                            aws_secret_access_key=args.aws_secret_access_key,
                            snapshot=snapshot,
                            cassandra_bin_dir=args.cassandra_bin_dir,
-                           cassandra_data_dir=args.cassandra_data_dir)
+                           restore_dir=args.restore_dir,
+                           no_sstableloader=args.no_sstableloader,
+                           local_restore=args.local_restore,
+                           s3_connection_host=get_s3_connection_host(args.s3_bucket_region))
 
     if args.hosts:
         hosts = args.hosts.split(',')
+
+        if args.local_restore and len(hosts) > 1:
+            logging.error(
+                "You must provide only one source host when using --local.")
+            sys.exit(1)
     else:
         hosts = snapshot.hosts
 
-    target_hosts = args.target_hosts.split(',')
+        if args.local_restore:
+            logging.error(
+                "You must provide one source host when using --local.")
+            sys.exit(1)
+
+    # --target_hosts is mutually exclusive with --local and --nosstableloader
+    if args.target_hosts:
+        target_hosts = args.target_hosts.split(',')
+    else:
+        # --local or --nosstableloader: no streaming will occur
+        target_hosts = None
 
     worker.restore(args.keyspace, args.table, hosts, target_hosts)
 
@@ -272,29 +291,47 @@ def main():
     restore_parser.add_argument(
         '--hosts',
         default='',
-        help="Comma separated list of \
-            hosts to restore from; leave empty for all")
-
-    restore_parser.add_argument(
-        '--target-hosts',
-        required=True,
-        help="The comma separated list of hosts to restore into")
+        help="Comma separated list of hosts to restore from; "
+             "leave empty for all. Only one host allowed when using --local.")
 
     restore_parser.add_argument(
         '--cassandra-bin-dir',
         default='/usr/bin',
         help="cassandra binaries directory")
 
     restore_parser.add_argument(
-        '--cassandra-data-dir',
-        default='/usr/local/cassandra/data',
-        help="cassandra data directory")
+        '--restore-dir',
+        default='/tmp/restore_cassandra/',
+        help="Directory where data will be downloaded. "
+             "Existing data in this directory will be *ERASED*. "
+             "If --target-hosts is passed, sstableloader will stream data "
+             "from this directory.")
+
+    restore_type = restore_parser.add_mutually_exclusive_group(required=True)
+
+    restore_type.add_argument(
+        '--target-hosts',
+        help='The comma separated list of hosts to restore into')
+
+    restore_type.add_argument(
+        '--local',
+        action='store_true',
+        dest='local_restore',
+        help='Do not run sstableloader when restoring. If set, files will '
+             'just be downloaded and decompressed in --restore-dir.')
+
+    restore_type.add_argument(
+        '--no-sstableloader',
+        action='store_true',
+        help="Do not run sstableloader when restoring. "
+             "If set, files will just be downloaded. Use it if you want to do "
+             "some checks and then run sstableloader manually.")
 
     args = base_parser.parse_args()
     subcommand = args.subcommand
 
     if args.verbose:
-        logging.basicConfig(level=logging.INFO)
+        logging.basicConfig(level=logging.INFO, format='%(message)s')
 
     if subcommand == 'backup':
         run_backup(args)