Cleaning mirror scripts

README.md

@@ -66,8 +66,8 @@ be super user). For example, the `borealis.daemon` is run with the following
 Description=Borealis data flow inotify daemon
 
 [Service]
-User=radar
-ExecStart=/home/radar/data_flow/inotify_daemons/borealis.daemon
+User=<user>
+ExecStart=/home/{USER}/data_flow/inotify_daemons/borealis.daemon
 Restart=always
 
 [Install]
@@ -106,12 +106,12 @@ HOST [address of the campus computer, either hostname or IP]
    where username and address are the same as those set in the config file. The output will be something like:
 ```
 Success:
-transfer@pgrdist205:~> ssh -N -f dataman@sdc-serv.usask.ca; ssh -O check dataman@sdc-serv.usask.ca
+site_user@site:~> ssh -N -f [username]@[address]; ssh -O check [username]@[address]
 Master running (pid=15739)
 
 Failure:
-transfer@pgrdist205:~> ssh -N -f dataman@sdc-serv.usask.ca; ssh -O check dataman@sdc-serv.usask.ca
-Control socket connect(/home/transfer/.ssh/controlmasters/3354587955ba492d0d5f595f8619d902ac0192a7): No such file or directory
+site_user@site:~> ssh -N -f [username]@[address]; ssh -O check [username]@[address]
+Control socket connect(/home/[site_user]/.ssh/controlmasters/3354587955ba492d0d5f595f8619d902ac0192a7): No such file or directory
 ```
 
 ### Installing data flow
@@ -127,8 +127,8 @@ is required for the sending of inotify flags between computers.
     `ssh-keygen -t ecdsa -b 521`
     - Copy the public key to the destination computer: `ssh-copy-id user@host`
     - Computers that must be linked: Borealis -> Site-Linux, Site-Linux -> sdc-serv
-    - For telemetry purposes, each data flow computer must also be linked to the logman user on
-      sdc-serv, so copy the ssh keys to logman@sdc-serv as well
+    - For telemetry purposes, each data flow computer must also be linked to a logging user on campus, so copy the 
+      ssh keys to that user on sdc-serv as well
 4. Install the inotify daemon for the respective computer (for example, install borealis.daemon 
 with borealis_dataflow.service on the Borealis computer). As super user, do the following:
     - Copy the correct `.service` file from `inotify_daemons/services/` to 
@@ -152,8 +152,8 @@ with borealis_dataflow.service on the Borealis computer). As super user, do the
 7. For telemetry purposes, summary logs are availabe for each script in the 
 `~/logs/[script name]/summary/` directory. These logs contain the status of all operations on each
 file and easily parseable to monitor data flow operation. Each script rsyncs the summary files to
-logman@sdc-serv for uploading to the Engineering dashboard. SSH password-free connection must be
-setup between each computer and logman@sdc-serv for this to work correctly. 
+the logging user on sdc-serv for uploading to the Engineering dashboard. SSH password-free connection must be
+setup between each computer and logging user on sdc-serv for this to work correctly. 
 8. To modify the data flow easily, a `config.sh` file is provided. This file specifies:
     - If the data flow can use the NAS at a site
     - What Borealis filetypes are to be converted and restructured

mirror/README.md

@@ -0,0 +1,55 @@
+## Mirror
+This section of the Data Flow repo contains scripts related to the campus to mirror rawacf data flow. These scripts
+are run via crontab on sdc-serv. The scripts that are scheduled via crontab are located in the top level directory.
+The remaining scripts are located in the `tools/` subdirectory and are utility scripts for the data flow operations.
+
+### Main Scripts
+- **download_vt_data** - This script logs onto the Virginia Tech (VT) server and downloads all rawacfs for a given
+radar from the holding directory on the VT server to our holding directory on campus. Run the script like:
+    - `download_vt_data holding_dir radar`
+- **sync_mirror_data.sh** - This script logs onto one of the SuperDARN mirrors and downloads all rawacfs that our
+mirror is missing. Any rawacfs that are in the failed list or blocklist will not be downloaded from the external mirror.
+This script is designed to run for a given mirror and yyyymm. The yyyymm argument is optional and if no yyyymm is given,
+the script will only sync with the given mirror for the current yyyymm. Run the script like:
+  - `flock nssc_filelock -c 'sync_mirror_data.sh NSSC_holding NSSC yyyymm'` to sync with the NSSC server and
+  - `flock bas_filelock -c 'sync_mirror_data.sh BAS_holding BAS yyyymm'` to sync with the BAS server
+- **gatekeeper_globus.py** - This script acts on one of the three holding directories to transfer the rawacfs to the
+SuperDARN Canada mirror via the globus_sdk python library. Below is a brief overview of this process, see the
+SuperDARN Canada Wiki for details:
+  - Remove any rawacfs in holding_dir if they are in the failed list or blocklist.
+  - Hash the holding_dir and compare to the hash files on the mirror. Remove matching files from holding_dir and move
+  nonmatching files to `nomatch/` subdirectory in holding_dir.
+  - Check that all remaining rawacfs can be successfully unzipped and are not empty. Files who fail any of these tests
+  are moved to `holding_dir/failed/`.
+  - Update the failed files list `all_failed.txt` on the mirror and transfer these failed files to `local_data/failed/`
+  on the mirror.
+  - Transfer the remaining files in holding_dir to the mirror at `raw/yyyy/mm/` as they have passed all checks
+  - Update yyyymm.hashes and master.hashes
+
+  Run the script like:
+
+  - python `-u gatekeeper_globus.py -d holding_dir -m mirror_dir`
+
+- **batch_sync_mirror** - This script is run on both a weekly and monthly schedule for each of the NSSC and BAS servers.
+On the weekly run, the script syncs the previous 12 months between the USASK and NSSC (or BAS) mirrors. On
+the monthly run, the script syncs all data since 2006 between the USASK and NSSC (or BAS) mirrors. Run the script like:
+  - `flock nssc_filelock -c 'batch_sync_mirror NSSC weekly'` or `flock nssc_filelock -c 'batch_sync_mirror NSSC 
+monthly'` for syncing with NSSC and
+  - `flock bas_filelock -c 'batch_sync_mirror BAS weekly'` or `flock bas_filelock -c 'batch_sync_mirror BAS monthly'`
+for syncing with BAS
+
+### Tools
+- **delete_files_globus.py** - This script is designed to log on to the USask SuperDARN mirror via globus in order to 
+check for and remove files given a list of files. Run the script like:
+  - python `delete_files_globus.py -t 'raw' -r 'mirror_root_dir/' -d 'deletions_dir/'
+        -l '~/log_dir/' files_to_delete.txt`
+  - For usage instructions run python `delete_files_globus.py -h`
+- **flag_experiment_files.py** - Script to check for and move local special experiment files to a subdirectory. Main 
+usage is to move files out of the holding directory when special experiment files are not flagged earlier in the data 
+flow chain. Note that although this script will normally be run on the holding directory, it is capable of running
+on any directory with RAWACFs and will move all special experiment files to a subdirectory called `special_experiments/`.
+- **gatekeeper_class.py** - This script contains utility functions for `gatekeeper_globus.py` as well as the
+'Gatekeeper' class that is instantiated at the beginning of `gatekeeper_globus.py` and whose methods are called
+throughout the script. This script should not be executed directly in the command line. Instead, simply import the
+'Gatekeeper' class and other functions into the running script. For example, both `gatekeeper_globus.py` and
+`delete_files_globus.py` import items from `gatekeeper_class.py` in this way.

mirror/batch_sync_mirror

@@ -10,6 +10,9 @@
 #   Compare our mirror to the BAS/NSSC mirror 1 month at a time going back to January 2006
 #   Download any rawacf files that we are missing. Runs monthly.
 
+readonly USER=$(whoami)
+readonly SCRIPTDIR=/home/${USER}/data_flow/mirror/
+
 # Valid MIRROR and MODE values
 readonly VALID_MIRRORS=("BAS" "NSSC")
 readonly VALID_MODES=("weekly" "monthly")
@@ -54,7 +57,7 @@ then
   while [ "$d" != "${enddate}" ]
   do
     yearmonth=$(date -d "$d" +%Y%m)
-    /home/dataman/data_flow/mirror/sync_mirror_data.sh ${HOLDING_DIR} ${MIRROR} ${yearmonth}
+    ${SCRIPTDIR}/sync_mirror_data.sh ${HOLDING_DIR} ${MIRROR} ${yearmonth}
     d=$(date -d "$d + 1 month" +%Y%m%d)
   done
 
@@ -72,13 +75,13 @@ then
   while [ "$d" != "${enddate}" ]
   do
     yearmonth=$(date -d "$d" +%Y%m)
-    /home/dataman/data_flow/mirror/sync_mirror_data.sh ${HOLDING_DIR} ${MIRROR} ${yearmonth}
+    ${SCRIPTDIR}/sync_mirror_data.sh ${HOLDING_DIR} ${MIRROR} ${yearmonth}
     d=$(date -d "$d + 1 month" +%Y%m%d)
   done
 
   # Update mirror comparison
-  HASHES_DIRS="/home/dataman/tmp_hashes_usask_${mirror}_cmp/${today}.*"
-  CMP_DIR=/home/dataman/Documents/mirror_comparison_${today}/
+  HASHES_DIRS="/home/${USER}/tmp_hashes_usask_${mirror}_cmp/${today}.*"
+  CMP_DIR=/home/${USER}/Documents/mirror_comparison_${today}/
   mkdir -p ${CMP_DIR}/sorted
 
   find ${HASHES_DIRS} -name '*data.different' -exec cat '{}' \; > ${CMP_DIR}/${mirror}.different

mirror/download_vt_data

@@ -6,7 +6,7 @@
 # Modified from Kevin Sterne's script
 #
 # Directory structure of the VT server's holding area is:
-# /home/usask/outgoing/sas
+# /home/${REMOTEUSER}/outgoing/sas
 #		      /rkn
 #		      /hal
 #		      /bks
@@ -19,9 +19,9 @@
 # 
 # In order for public keys to work properly, permissions and owners must be
 # set up correctly.
-# For /home/usask/ The permissions must be 755
-# For /home/usask/.ssh the permissions must be 700
-# For /home/usask/authorized_keys the permissions must be 644
+# For /home/${REMOTEUSER}/ The permissions must be 755
+# For /home/${REMOTEUSER}/.ssh the permissions must be 700
+# For /home/${REMOTEUSER}/authorized_keys the permissions must be 644
 #
 # This script downloads any available radar rawacf data from Virginia
 # Tech's server's holding area.
@@ -45,6 +45,8 @@
 ##############################################################################
 # Initialize Some Variables
 ##############################################################################
+USER=$(whoami)
+
 # Valid 3-letter RADAR values
 readonly VALID_RADARS=("ade" "adw" "bks" "bpk" "cly" "cve" "cvw" "dce" "dcn" "fhe" "fhw" "fir" "gbr" "hal"
                        "han" "hkw" "hok" "ice" "icw" "inv" "jme" "kap" "ker" "kod" "ksr" "lyr" "mcm" "pgr"
@@ -79,7 +81,7 @@ CURYEAR=$(date +%Y)
 CURMONTH=$(date +%m)
 
 # Setup logfile
-LOGGINGDIR=/home/dataman/logs/vt/${CURYEAR}/${CURMONTH} # Add _test for testing purposes
+LOGGINGDIR=/home/${USER}/logs/vt/${CURYEAR}/${CURMONTH} # Add _test for testing purposes
 LOGFILE=${LOGGINGDIR}/${DATE}.${RADAR}.log
 # Make the log file directory if it doesn't exist
 mkdir -p ${LOGGINGDIR}
@@ -161,9 +163,9 @@ fi
 # Create shortcut for sftp command
 SFTP=/usr/bin/sftp
 # User, hostname and directory to connect to on VT server
-USER=usask
-REMOTEHOST=sd-data1.ece.vt.edu
-REMOTEDIRBASE=/home/usask/outgoing
+REMOTEUSER=${VT_USER}
+REMOTEHOST=${VT_HOST}
+REMOTEDIRBASE=/home/${REMOTEUSER}/outgoing
 REMOTEDIR=${REMOTEDIRBASE}/${RADAR}
 # Path to hashfile for files on VT server (e.g., /data/holding/sas/hashes.remote)
 HASHESFILE=hashes.remote
@@ -178,7 +180,7 @@ REMOTEHASHESSTART=$(date +%s)
 cd ${LOCALRADARDIR} || exit
 # This will write message to stderr if sha1sum fails
 # ssh into VT and hash the rawacf files for the given radar, output to hashes.remote
-ssh ${USER}@${REMOTEHOST} "cd ${REMOTEDIR}; ${HASHPROG} *${RADAR}*rawacf.bz2" > ${HASHESFILE} 2> ${HASHESERRORS}
+ssh ${REMOTEUSER}@${REMOTEHOST} "cd ${REMOTEDIR}; ${HASHPROG} *${RADAR}*rawacf.bz2" > ${HASHESFILE} 2> ${HASHESERRORS}
 # Capture return value from the above sha1sum
 RETURNVALUE=$?
 REMOTEHASHESEND=$(date +%s)
@@ -298,7 +300,7 @@ echo "Downloading files..."
 # Only execute sftp batch file if there is at least one file to download from VT
 if [[ ${totalFiles} -gt 0 ]]
 then
-  ${SFTP} -p -b ${SFTPBATCH} ${USER}@${REMOTEHOST} 2> ${SFTPERRORS}
+  ${SFTP} -p -b ${SFTPBATCH} ${REMOTEUSER}@${REMOTEHOST} 2> ${SFTPERRORS}
 fi
 DOWNLOADTIMEEND=$(date +%s)
 
@@ -381,7 +383,7 @@ echo "Deleting files..."
 # Only execute sftp batch file if there is at least one file to remove from VT 
 if [[ ${totalFilesDelete} -gt 0 ]]
 then
-  ${SFTP} -p -b ${SFTPBATCHDELETE} ${USER}@${REMOTEHOST} 2> ${SFTPERRORSDELETE}
+  ${SFTP} -p -b ${SFTPBATCHDELETE} ${REMOTEUSER}@${REMOTEHOST} 2> ${SFTPERRORSDELETE}
 fi
 DELETETIMEEND=$(date +%s)
 

mirror/gatekeeper_globus.py

@@ -44,7 +44,7 @@
 import logging
 import argparse
 
-from gatekeeper_class import Gatekeeper, parse_data_filename
+from tools.gatekeeper_class import Gatekeeper, parse_data_filename
 
 # Make sure there is only one instance running of this script
 from tendo import singleton
@@ -54,14 +54,19 @@
 HOME = expanduser("~")
 TRANSFER_RT_FILENAME = f"{HOME}/.globus_transfer_rt"
 PERSONAL_UUID_FILENAME = f"{HOME}/.globusonline/lta/client-id.txt"
+GATEKEEPER_APP_FILENAME = f"{HOME}/mirror_id_files/gatekeeper_app_id.txt"
 
 if isfile(PERSONAL_UUID_FILENAME):
     with open(PERSONAL_UUID_FILENAME) as f:
         PERSONAL_UUID = f.readline().strip()
 
 # Client ID retrieved from https://auth.globus.org/v2/web/developers
-# gatekeeper_app_CLIENT_ID = 'bc9d5b7a-6592-4156-bfb8-aeb0fc4fb07e'  # Saif's app
-gatekeeper_app_CLIENT_ID = 'e70228d0-56a2-4d85-bf63-7fbccc92dcd3'  # Rem's app
+if isfile(GATEKEEPER_APP_FILENAME):
+    with open(GATEKEEPER_APP_FILENAME) as f:
+        file = f.readlines()
+    for line in file:
+        if "CM app" in line:
+            gatekeeper_app_CLIENT_ID = line.split("=")[1].split()[0]
 
 
 def main():
@@ -107,7 +112,7 @@ def main():
     # Check script arguments as well as existence of various directories
     logger = gk.logger
 
-    # Clear out working directory /home/dataman/tmp/* before use
+    # Clear out working directory ~/tmp/* before use
     if isdir(gk.get_working_dir()):
         shutil.rmtree(gk.get_working_dir())
         mkdir(gk.get_working_dir())
@@ -118,6 +123,7 @@ def main():
 
     logger.info(f"Args: {args.holding}  {args.mirror}  {args.pattern}")
 
+    # Set holding directory, mirror directory, yearmonth, radar, and sync pattern from parsed arguments
     # Set holding directory, mirror directory, yearmonth, radar, and sync pattern from parsed arguments
     gk.set_holding_dir(args.holding)
     gk.set_mirror_root_dir(args.mirror)
@@ -322,38 +328,35 @@ def main():
                 yearmonth_dict.pop(ym)
             else:
                 logger.info(f"{ym} hash file retrieved from mirror.")
-                # sha1sum files in holding_dir and compare to yyyymm.hashes now in working dir (-c == compare)
-                command_string = f"cd {gk.get_holding_dir()}; sha1sum -c {gk.get_working_dir()}/{ym}.hashes"
-                sha1sum_process = subprocess.Popen(command_string, shell=True,
-                                                   stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                out, err = sha1sum_process.communicate()
-                sha1sum_decoded_output = out.decode().split("\n")
-                sha1sum_decoded_error = err.decode().split("\n")
-                # Loop through result of sha1sum comparison for each file
-                # Only remove from files_to_upload if file exists both in holding_dir and hashfile in working_dir
-                # Need further investigation into "Failed open or read" and "" results
-                for sha1sum_result in sha1sum_decoded_output:
-                    hashed_file = sha1sum_result.split(":")[0]
-                    if sha1sum_result.find("FAILED open or read") != -1:
-                        pass
-                    # If hashes do not match, add file to nonmatching files list, remove from files_to_upload
-                    elif sha1sum_result.find("FAILED") != -1:
-                        logger.warning(f"{hashed_file} hash doesn't match. Adding to no match list, and removing from list of files to upload.")
-                        non_matching_files.append(hashed_file)
-                        files_to_upload_dict.pop(hashed_file)
-                    # If hashes match, remove from files_to_upload as it is already on mirror
-                    elif sha1sum_result.find("OK") != -1:
-                        logger.info(f"{hashed_file} already exists on mirror and hash matches. Removing from files to upload.")
-                        files_to_upload_dict.pop(hashed_file)
-                        # Comment out removal of matching files from holding dir for testing purposes
-                        try:
-                            remove(f"{gk.get_holding_dir()}/{hashed_file}")
-                        except OSError as error:
-                            logger.error(f"Error trying to remove file: {error}.")
-                    elif sha1sum_result == "":
-                        pass
-                    else:
-                        logger.warning(f"Error, I don't know how to deal with: {sha1sum_result}.")
+
+                # Create dictionary to contain filenames as keys and hashes as values for ym.hashes of current iteration
+                ym_hashes = {}
+                with open(f"{gk.get_working_dir()}/{ym}.hashes", 'r') as hash_file:
+                    for line in hash_file:
+                        (val, key) = line.split()
+                        ym_hashes[key] = val
+
+                # loop over files in holding dir for ym of current iteration and compare hashes to ym.hashes
+                for holding_file in list(yearmonth_dict[ym].keys()):
+                    # Remove file from files to upload if this filename is already on the mirror
+                    # Compare hashes to see if the file should go to nomatch/ directory or just be removed from holding
+                    if holding_file in ym_hashes.keys():
+                        files_to_upload_dict.pop(holding_file)
+                        # If hashes do not match, add file to nonmatching files list (to be moved to nomatch/ directory)
+                        if files_to_upload_dict[holding_file]['hash'] != ym_hashes[holding_file]:
+                            logger.warning(f"{holding_file} hash doesn't match. Adding to no match list, and removing "
+                                           f"from list of files to upload.")
+                            non_matching_files.append(holding_file)
+                        # If hashes match, remove from holding directory
+                        else:
+                            logger.info(f"{holding_file} already exists on mirror and hash matches. Removing from files "
+                                        f"to upload.")
+                            # Comment out removal of matching files from holding dir for testing purposes
+                            try:
+                                remove(f"{gk.get_holding_dir()}/{holding_file}")
+                            except OSError as error:
+                                logger.error(f"Error trying to remove file: {error}.")
+
         # If yyyymm.hashes DNE, create it ONLY IF yyyymm is the current year and month
         else:
             # Need to check if this is the current month, otherwise error out