Skip to content

Commit 49c9ced

Browse files
grantfitzsimmonsgrantfitzsimmons
committed
Add Exoscale backup instructions
1 parent 2eb7807 commit 49c9ced

File tree

3 files changed

+129
-0
lines changed

3 files changed

+129
-0
lines changed

‎sphinx/index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Welcome to Specify Developer documentation!
2828
:caption: Server Management:
2929

3030
server_management/managed_servers
31+
server_management/exoscale_backups
3132

3233
.. toctree::
3334
:maxdepth: 1

‎sphinx/server_management/exoscale_backups.md

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# Backups on Exoscale
2+
3+
To configure the backups, you first need to create an [Exoscale bucket](https://portal.exoscale.com/u/specify-collections-consortium/storage).
4+
5+
For our backups, we have one named 'swiss-backups'.
6+
7+
In this instance, I chose the `CH-GVA-2` zone so that the data remains in Geneva.
8+
9+
> Exoscale has [great instructions](https://community.exoscale.com/documentation/storage/quick-start/) on how to connect to a bucket.
10+
11+
First, you must install `s3cmd` to connect to the Exoscale Bucket (as `root`):
12+
```sh
13+
apt-get update && apt-get install s3cmd
14+
```
15+
16+
`s3cmd` enables us to create and use Exoscale buckets. Our configuration file (located at `/home/ubuntu/.s3cfg`) looks like this:
17+
18+
```
19+
[default]
20+
host_base = sos-ch-gva-2.exo.io
21+
host_bucket = %(bucket)s.sos-ch-gva-2.exo.io
22+
access_key = $EXO_SOS_KEY
23+
secret_key = $EXO_SOS_SECRET
24+
use_https = True
25+
```
26+
(**Note:** $EXO_SOS_KEY and $EXO_SOS_SECRET are hidden for this guide)
27+
28+
I created an IAM Role named 'Create Backups' that will be used to connect the compute instances to the bucket. Once this was done, I generated an API key pair for that role.
29+
30+
Now that this is configured, you can place files into the bucket by simply running `s3cmd put ${file_name} s3://swiss-backups/${file_name}`:
31+
32+
```sh
33+
ubuntu@sp7cloud-swiss-1:~$ touch hello-world.txt
34+
ubuntu@sp7cloud-swiss-1:~$ s3cmd put hello-world.txt s3://swiss-backups/hello-world.txt
35+
upload: 'hello-world.txt' -> 's3://swiss-backups/hello-world.txt' [1 of 1]
36+
12 of 12 100% in 0s 110.91 B/s done
37+
```
38+
39+
Now that this is configured, we can use a script to backup the databases nightly:
40+
41+
```sh
42+
#!/bin/bash
43+
44+
# Get the current date in YYYY_mm_dd format
45+
current_date=$(date +"%Y_%m_%d")
46+
47+
# Create the date directory in the backup bucket
48+
bucket_dir="s3://swiss-backups/${current_date}"
49+
50+
# Fetch the list of databases dynamically
51+
databases=$(mysql -h 0.0.0.0 -P 3306 -uroot -p"$MYSQL_ROOT_PASSWORD" -e 'SHOW DATABASES;' | grep -Ev '^(Database|information_schema|performance_schema|mysql|sys)$')
52+
53+
# Loop through each database and perform the backup
54+
for db in $databases; do
55+
# Create the backup file name for the upload
56+
backup_file_name="${db}_${current_date}.sql.gz"
57+
58+
echo "Backing up database: $db"
59+
60+
# Perform the mysqldump and upload directly to the Exoscale bucket
61+
mysqldump --max-allowed-packet=2G --single-transaction -h 0.0.0.0 -P 3306 -uroot -p"$MYSQL_ROOT_PASSWORD" "$db" | gzip | s3cmd put - "${bucket_dir}/${backup_file_name}"
62+
63+
# Check if the upload was successful
64+
if [ $? -eq 0 ]; then
65+
echo "Backup of $db completed and uploaded successfully to $bucket_dir."
66+
else
67+
echo "Backup of $db failed or upload failed." >&2
68+
fi
69+
done
70+
```
71+
72+
See the process after running the script at `/home/ubuntu/.backup/backup_script.sh`:
73+
```sh
74+
ubuntu@sp7cloud-swiss-1:~/.backup$ sh backup_script.sh
75+
Backing up database: geo_swiss
76+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/geo_swiss_2024_08_14.sql.gz' [part 1 of -, 6MB] [1 of 1]
77+
6365386 of 6365386 100% in 0s 26.16 MB/s done
78+
Backup of geo_swiss completed and uploaded successfully to s3://swiss-backups/2024_08_14.
79+
Backing up database: mcsn
80+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/mcsn_2024_08_14.sql.gz' [part 1 of -, 4MB] [1 of 1]
81+
4602206 of 4602206 100% in 0s 20.33 MB/s done
82+
Backup of mcsn completed and uploaded successfully to s3://swiss-backups/2024_08_14.
83+
Backing up database: mhnc
84+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/mhnc_2024_08_14.sql.gz' [part 1 of -, 4MB] [1 of 1]
85+
4588230 of 4588230 100% in 0s 16.26 MB/s done
86+
Backup of mhnc completed and uploaded successfully to s3://swiss-backups/2024_08_14.
87+
Backing up database: mhnf
88+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/mhnf_2024_08_14.sql.gz' [part 1 of -, 5MB] [1 of 1]
89+
5861026 of 5861026 100% in 0s 25.46 MB/s done
90+
Backup of mhnf completed and uploaded successfully to s3://swiss-backups/2024_08_14.
91+
Backing up database: naag
92+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/naag_2024_08_14.sql.gz' [part 1 of -, 10MB] [1 of 1]
93+
11213948 of 11213948 100% in 0s 24.78 MB/s done
94+
Backup of naag completed and uploaded successfully to s3://swiss-backups/2024_08_14.
95+
Backing up database: nmb_rinvert
96+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/nmb_rinvert_2024_08_14.sql.gz' [part 1 of -, 6MB] [1 of 1]
97+
6450282 of 6450282 100% in 0s 30.53 MB/s done
98+
Backup of nmb_rinvert completed and uploaded successfully to s3://swiss-backups/2024_08_14.
99+
Backing up database: sp7demofish_swiss
100+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/sp7demofish_swiss_2024_08_14.sql.gz' [part 1 of -, 15MB] [1 of 1]
101+
15728640 of 15728640 100% in 0s 30.45 MB/s done
102+
upload: '<stdin>' -> 's3://swiss-backups/2024_08_14/sp7demofish_swiss_2024_08_14.sql.gz' [part 2 of -, 6MB] [1 of 1]
103+
6776807 of 6776807 100% in 0s 23.11 MB/s done
104+
Backup of sp7demofish_swiss completed and uploaded successfully to s3://swiss-backups/2024_08_14.
105+
ubuntu@sp7cloud-swiss-1:~/.backup$
106+
```
107+
108+
I configured a cron job to run this at 2 AM CEST (12 AM UTC) every day:
109+
```sh
110+
# m h dom mon dow command
111+
0 0 * * * /home/ubuntu/.backup/backup_script.sh
112+
```
113+
114+
Now this backup will run and be available within the bucket for future retrieval as needed!
115+
116+
```sh
117+
ubuntu@sp7cloud-swiss-1:~/.backup$ s3cmd ls s3://swiss-backups/
118+
DIR s3://swiss-backups/2024_08_14/
119+
ubuntu@sp7cloud-swiss-1:~/.backup$ s3cmd ls s3://swiss-backups/2024_08_14/
120+
2024-08-14 00:44 6365385 s3://swiss-backups/2024_08_14/geo_swiss_2024_08_14.sql.gz
121+
2024-08-14 00:44 4602206 s3://swiss-backups/2024_08_14/mcsn_2024_08_14.sql.gz
122+
2024-08-14 00:44 4588230 s3://swiss-backups/2024_08_14/mhnc_2024_08_14.sql.gz
123+
2024-08-14 00:45 5861026 s3://swiss-backups/2024_08_14/mhnf_2024_08_14.sql.gz
124+
2024-08-14 00:45 11213948 s3://swiss-backups/2024_08_14/naag_2024_08_14.sql.gz
125+
2024-08-14 00:45 6450282 s3://swiss-backups/2024_08_14/nmb_rinvert_2024_08_14.sql.gz
126+
2024-08-14 00:45 22505447 s3://swiss-backups/2024_08_14/sp7demofish_swiss_2024_08_14.sql.gz
127+
ubuntu@sp7cloud-swiss-1:~/.backup$
128+
```

‎sphinx/server_management/swiss-backups.png

479 KB
Loading

0 commit comments

Comments
 (0)