Skip to content

Commit 7094b81

Browse files
josh-heyerjosh-heyer
authored
Merge pull request #1588 from EnterpriseDB/content/bart/bart_migration_journey
2 parents 7ba2bde + 2609dd0 commit 7094b81

File tree

7 files changed

+384
-0
lines changed

7 files changed

+384
-0
lines changed

‎product_docs/docs/bart/2.6/bart_migration/01_choosing_your_tool.mdx

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
title: "Choosing your Backup Tool"
3+
4+
---
5+
6+
This section provides information on the two alternative back up tools.
7+
8+
## Barman
9+
10+
EDB recommends BART users move to barman and plans to add the following capabilities soon:
11+
- Integration with PEM
12+
- Integration with other backup solutions
13+
- An API for implementing other integrations
14+
15+
Besides the benefits of choosing the tool that EDB is investing in, Barman is the right choice if you are doing remote backups using the PostgreSQL port and you are not using passwordless ssh. See the [Barman documentation](https://www.enterprisedb.com/docs/supported-open-source/barman/) for more information.
16+
17+
## pgBackRest
18+
19+
pgBackRest’s advantages include:
20+
- Performance benefit for backing up large (multi-terabyte) databases. The internal algorithm for detecting file changes in delta backups adds less overhead, and partial restore allows to restore only one database from a large multi database cluster.
21+
- Supplies backup compression, which is a planned feature for Barman.
22+
23+
See the [pgBackRest documentation](https://www.enterprisedb.com/docs/supported-open-source/pgbackrest/) for more information.
24+
25+
## Tool Comparison
26+
27+
The following selection matrix differentiates between the
28+
tools on a generic level:
29+
30+
| **Capability** | **Barman** | **pgBackRest** |
31+
| ------------------------------------------ | -------------------------------------------------------------------------------- | ---------------- |
32+
| PostgreSQL protocol | Yes | - |
33+
| RPO=0 | Yes | - |
34+
| Rate limiting | Yes | -|
35+
| Custom WAL sizes | Yes | Yes ( v11+) |
36+
| Backup compression | Coming soon | Yes |
37+
| Partial restore (only selected databases) | - | Yes |
38+
| PEM integration | Coming soon | - |
39+
| License | [GPLv3](https://github.com/EnterpriseDB/barman/blob/master/LICENSE) | [MIT](https://github.com/pgbackrest/pgbackrest/blob/master/LICENSE) |
40+
41+
42+
43+

‎product_docs/docs/bart/2.6/bart_migration/02_configuring.mdx

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
---
2+
title: "Configuring the New Tool"
3+
4+
---
5+
6+
This section draws a comparison between the three tools and shows how
7+
they differ in configuration (config file), and in scheduling. During a
8+
migration it is important to be aware of all of these differences, and
9+
apply changes as necessary.
10+
11+
### How Features are Configured
12+
13+
The following sections describes how to enable features in the different tools so you can easily compare the implementation details .
14+
15+
- [Postgres connection details](#postgres_conn_details)
16+
- [Use of direct file copy mode (ssh or rsync) instead of pg_basebackup](#direct_file_copy_mode)
17+
- [Incremental backups](#incremental_backup)
18+
- [WAL archive compression](#wal_archive_compression)
19+
- [Backup compression](#backup_compression)
20+
- [Parallel backup and restore](#parallel_backup_restore)
21+
- [Retention Policy](#retention_policy)
22+
- [Configure repository location](#config_repo_location)
23+
24+
<div id='postgres_conn_details' class='registered_link'></div>
25+
26+
#### Postgres connection details
27+
28+
29+
| Tool | Configuration |
30+
|---------------|---------------------------------------------------------|
31+
| BART | In the [SERVER] section of the config file: <br/>[example]<br/>...<br/>`host = localhost`<br/>`user = postgres` <br/>`port = 5432`|
32+
| Barman | In the config file: <br/> [example]<br/>...<br/>`conninfo = host=localhost user=postgres dbname=postgres`|
33+
| pgBackRest[^*]| In the config file: <br/> [example]<br/>...<br/>`pg1-host = localhost`<br/>`pg1-path = /var/lib/postgres/data`<br/>`pg1-user = postgres`<br/>`port = 5432`|
34+
35+
[^*] Specify pg1-path when pgBackRest is running locally, and specify pg1-host when pgBackRest is running remote
36+
<div id='direct_file_copy_mode' class='registered_link'></div>
37+
38+
#### Use of direct file copy mode (ssh or rsync) instead of pg_basebackup
39+
40+
| Tool | Configuration |
41+
|------------|---------------------------------------------------------|
42+
| BART | Increase threads >1 in the [SERVER] or [BART] section of the config file:<br/>[example]<br/>...<br/>`thread_count = 2`<br/>Or, as a commandline option<br/> `--no-pg_basebackup`|
43+
| Barman | In the config file: <br/>[example]<br/>...<br/>`backup_method = rsync`|
44+
| pgBackRest | pgBackRest does not support pg_basebackup option|
45+
46+
<div id='incremental_backup' class='registered_link'></div>
47+
48+
#### Incremental backups
49+
50+
| Tool | Configuration |
51+
|------------|---------------------------------------------------------|
52+
| BART | Retrieve the `backup_id` or `backup_name` for the parent:<br/>`bart SHOW-BACKUPS example`<br/>Now supply he `backup_id` or `backup_name` as parent:<br/>`bart BACKUP –s example --parent { backup_id \| backup_name }`|
53+
| Barman | In the config file:<br/>[example]<br/>...<br/>`reuse_backup = link`<br/>Or, as a command line option:<br/>`--reuse-backup=link`|
54+
| pgBackRest | `pgbackrest --stanza=example --type=incr backup`|
55+
56+
<div id='wal_archive_compression' class='registered_link'></div>
57+
58+
#### WAL archive compression**
59+
60+
| Tool | Configuration |
61+
|------------|---------------------------------------------------------|
62+
| BART | In the [SERVER] or [BART] section of the config file: <br/> [example]<br/>...<br/>`wal_compression = enabled`|
63+
| Barman | In the config file:<br/>[example]<br/>...<br/>`compression = gzip`|
64+
| pgBackRest | Global compress settings. Can be overloaded for [global:archive-push]|
65+
66+
<div id='backup_compression' class='registered_link'></div>
67+
68+
#### Backup compression
69+
70+
| Tool | Configuration |
71+
|------------|---------------------------------------------------------|
72+
| BART | Command line option: <br/>`Enable: --gzip`<br/>`Level: --compress-level`|
73+
| Barman | N/A|
74+
| pgBackRest | `compress=y`<br/>`compress-level=9`<br/>`compress-type=gz`<br/>`compress-level-network=3`|
75+
76+
77+
<div id='parallel_backup_restore' class='registered_link'></div>
78+
79+
#### Parallel backup and restore
80+
81+
| Tool | Configuration |
82+
|------------|---------------------------------------------------------|
83+
| BART | Set `thread_cound` in the [SERVER] or [BART] section of the config file:<br/> [example]<br/>...<br/>`thread_count = 4`<br/>Or, as a Command line option<br/> `--thread-count=4`|
84+
| Barman | Set `parallel_jobs` in the configfile:<br/>[example]<br/>...<br/>`parallel_jobs = 4`<br/>Or, as a command line option<br/> `--jobs 4`|
85+
| pgBackRest | Set `process-max` in the configfile:<br/>[example]<br/>...<br/>`process-max = 4`<br/>Or, as a command line option<br/> `--process-max=4`|
86+
87+
88+
<div id='retention_policy' class='registered_link'></div>
89+
90+
#### Retention Policy
91+
92+
| Tool | Configuration |
93+
|------------|---------------------------------------------------------|
94+
| BART | Set `retention_policy` in the [SERVER] or [BART] section of the config file:<br/>[example]<br/>…<br/>`retention_policy = 2 BACKUPS` # Or DAYS, WEEKS, or MONTHS|
95+
| Barman | Set `retention_policy`, and/or `wal_retention_policy` options in the config file:<br/>[example]<br/>…<br/>`retention_policy =REDUNDANCY 2`<br/>or<br/>`retention_policy = RECOVERY WINDOW OF 2 DAYS` # WEEKS/MONTHS.|
96+
| pgBackRest | Can be set in the config file:<br/>[example]…<br/>`repo-retention-full-type = count #can also be time`<br/>`repo-retention-full = 2`<br/>`repo-retention-diff = 6`<br/>`# Retain WAL archives for only 1 full backup:`<br/>`repo-retention-archive-type = full #can also be diff or incr`<br/>`repo-retention-archive = 1`<br/>See Retention Policy in [edb documentation](https://www.enterprisedb.com/docs/supported-open-source/pgbackrest/05-retention_policy/) for more information.|
97+
98+
<div id='config_repo_location' class='registered_link'></div>
99+
100+
#### Configure repository location
101+
102+
| Tool | Configuration |
103+
|------------|---------------------------------------------------------|
104+
| BART | In the config file in the [BART] section:<br/>[BART]<br/>…<br/>`backup_path = /tmp/bart`|
105+
| Barman | In the config file in the main chapter:<br/>`backup_directory = /tmp/barman`|
106+
| pgBackRest | [global]<br/>`repo1-path=/var/lib/pgbackrest`|
107+
108+
109+
110+

‎product_docs/docs/bart/2.6/bart_migration/03_planning.mdx

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
title: "Planning the Migration"
3+
---
4+
5+
Review this section for items EDB recommends you include in your migration plan.
6+
7+
## Testing and Verification
8+
9+
Every organization will have different policies and capabilities which
10+
define what should be done in which environment. Some organizations
11+
restore a production database after migration to a temporary system in
12+
the test environment, where others rather test the process in a test
13+
environment, and expect their production environment to behave the same
14+
as their test environment. Whatever best suits your organization, make
15+
sure all of the following are updated and verified:
16+
17+
- Deployment and management automation
18+
19+
- Processes for restoring a database
20+
21+
- Integrations such as integrations with the backup infrastructure
22+
23+
EDB advises to set up a temporary test system, specifically for
24+
onboarding the new tool, so that all of these actions can take place
25+
without impacting the existing environments. Make sure your test plan
26+
includes standing up the test system, planning for all actions that
27+
should take place on this test system, and cleaning up the system after
28+
migration.
29+
30+
Some organizations also require verifying a successful restore for every
31+
database that has been migrated to the new tool.
32+
33+
## Sizing Considerations
34+
35+
Depending on the actual migration, different sizing considerations
36+
apply. Two distinct migration paths exist:
37+
38+
1. New storage will be attached for the new backups.
39+
40+
In this case, a good starting point would be to size the storage equally to the current sizing requirements. Note this might be a good opportunity to scale down oversized storage locations. Optionally, consider running both backup tools simultaneously. If the impact is acceptable, it can be an option for rollback and extra security for successful backups. EDB recommends this approach as it is the most straightforward and least error prone.
41+
42+
2. The mount point for the existing backups is reused for the backups.
43+
44+
An important note to make is that the repositories for the different tools have different layouts, and are not interchangeable. That means that the exact location needs to be distinct. But they can exist on the same mount point. The upside is that the extra required storage can be expected to be less than when attaching new storage. That being said, during migration extra storage is still required, and downscaling might not be an option. Furthermore, this option leaves no room to run both backup tools simultaneously. Extra sizing requirements would depend on your exact backup scheme, maintenance schema, and the size difference for differential backups. A good starting point would be to prepare for an extra set of backups (one full and all differentials), and an extra full backup. Make sure that the monitoring thresholds are also properly adjusted as required.
45+
46+
Note that it is crucial that you make sure that compression options are configured similarly between the old and new tool. Alternatively, extra storage space is required to compensate for the backup and/or WAL size.
47+
48+
During the planning phase the following needs to be taken care of:
49+
50+
- Select the approach.
51+
52+
- Size the expected extra storage and check for availability.
53+
54+
- Plan to attach or increase the extra storage as part of the migration plan.
55+
56+
- Next to the expected extra storage, also make sure that during the migration extra storage space is directly available should it be required due to unforeseen circumstances.
57+
58+
- When using the first approach (extra mount point), plan to clean out the old storage after migration. Make sure that the old repositories are preserved long enough to meet your organization's rollback policies.
59+
60+
## Resource Availability and Timeline Communication
61+
62+
Depending on the size of your environment, the number of available
63+
DBAs to run the migrations, and the amount of automation,
64+
the migration can take a considerable amount of time. During the
65+
migration your organization may be stressed with some parts of the
66+
environment already running with the new tool, extra storage
67+
requirements being identified, and DBA resources focused on the
68+
migration and less available for other tasks.
69+
70+
It is therefore crucial to plan the migration per environment and derive
71+
the duration of the migration from that plan. Communicate these
72+
timelines to the rest of the organization so that they are aware that
73+
the DBA team is extra occupied during this period of time. Furthermore,
74+
make sure that the storage team is available to prepare for and attach
75+
the extra storage as required.
76+

‎product_docs/docs/bart/2.6/bart_migration/04_executing.mdx

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
title: Executing the Migration
3+
4+
---
5+
6+
Execution of the migration can be very straight forward. The steps include:
7+
8+
1. Adding extra storage capacity to the backup server:
9+
10+
- As a new mount point when migrating to a new mount point (or Cloud Storage endpoints).
11+
12+
- As extra storage on the existing mount point when reusing the same mount point.
13+
14+
2. Installing the new tool on the backup server.
15+
16+
3. Configuring the configuration file for the new tool. See [How features are configured](#how-features-are-configured) for more information on changing a configuration file for one tool to another tool.
17+
18+
4. Reconfiguring the scheduling. See [Scheduling](#scheduling) for more information on changing the scheduling from one tool to another tool.
19+
20+
5. Checking the configuration:
21+
22+
- `barman check <example>`
23+
24+
- `pgbackrest --stanza=<example> --log-level-console=info check`
25+
26+
Fix any issues.
27+
28+
6. Running an initial backup manually, before relying on the scheduling
29+
(recommended). The check commands will recover the most common
30+
issues, but there could always still be unforeseen issues, like
31+
storage space issues, IO or network latency issues, etc. It would
32+
be unfortunate if the scheduled backups fail since that would
33+
increase risk for unsuccessful restores. As your team runs more
34+
migrations, they will gain more confidence to skip this step as
35+
required.
36+
37+
7. Some organizations might prefer to actually restore and recover the
38+
backup (on a separate system), and check for the expected data.
39+
This is only an optional extra step, which greatly depends on your
40+
organization's policies.

‎product_docs/docs/bart/2.6/bart_migration/05_warming_up.mdx

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
title: "Maintaining and Cleaning Up"
3+
4+
---
5+
6+
A transition period is required after the migration. This is because:
7+
8+
- With an empty repository there will be more situations where a full
9+
backup is created, where in a normal situation a differential
10+
backup would be taken.
11+
- The maintenance process of cleaning old backups kicks in only after
12+
exceeding the retention period.
13+
14+
During this transition period, extra care needs to be taken on the backups to
15+
make sure that any backup issue is identified and fixed. Options
16+
include:
17+
18+
- Increasing monitoring notification levels and decreasing alert thresholds for free space.
19+
- Planning manual checks of the backups during the migration.
20+
21+
EDB recommends keeping the original backup solution around during this
22+
transition period. You may want to keep the systems around for a longer
23+
period of time as an extra safety measure or extra rollback option. When
24+
the new backup system is fully operational, and the extra rollback time has exceeded,
25+
the old systems can be deprovisioned by:
26+
27+
- Removing the original tool from the system.
28+
- Deprovisioning the old mount point and storage after the new mount point is attached for the new storage repository.
29+
- Deprovisioning the old servers If the new tool runs on new servers.

‎product_docs/docs/bart/2.6/bart_migration/06_scheduling.mdx

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
---
2+
title: Scheduling
3+
4+
---
5+
6+
Since BART, Barman, and pgBackRest all schedule the backups using cron,
7+
changing the scheduling simply requires changing the scheduled commands.
8+
Example commands for BART, Barman, and pgBackRest:
9+
10+
1. Run a full backup for cluster \'example\':
11+
12+
| Tool | Command |
13+
|------------|---------------------------------------------------------|
14+
| BART | `bart BACKUP –s example`|
15+
| Barman | `barman backup example`|
16+
| pgBackRest | `pgbackrest backup --type=full --stanza=example`|
17+
18+
19+
20+
2. Run a full backup for all configured servers:
21+
22+
| Tool | Command |
23+
|------------|---------------------------------------------------------|
24+
| BART | `bart BACKUP –s all`|
25+
| Barman | `barman backup all`|
26+
| pgBackRest | pgbackrest does not have an option to run for all stanzas with one command|
27+
28+
29+
30+
3. Run an incremental backup:
31+
32+
| Tool | Command |
33+
|------------|---------------------------------------------------------|
34+
| BART | Retrieve the `backup_id`/`backup_name` for the parent (select a full for an incremental, select another inc/diff for a differential backup plan):<br/>`bart SHOW-BACKUPS example`<br/>Now supply the `backup_id`/`backup_name` as parent:<br/>`bart BACKUP –s example --parent { backup_id \| backup_name }`<br/>|
35+
| Barman | `barman backup example --reuse-backup=link`<br/>This can also be set in the configuration file at the global/server level with `reuse_backup = link`|
36+
| pgBackRest | Incremental:`pgbackrest backup --type=incr --stanza=example`<br/>Differential: `pgbackrest backup --type=diff --stanza=example`|
37+
38+
39+
### Retention management
40+
41+
- With BART usually a cron job is set up to run maintenance on the
42+
BART repository (bart MANAGE). By supplying the -d option, the
43+
obsoleted backups are automatically cleaned (together with their
44+
WAL archives).
45+
46+
- Similarly, Barman uses a cron command to run maintenance on the
47+
Barman repository \`barman cron\`. The \`barman cron\` command
48+
takes care of more things (like copying streamed WAL files to the
49+
WAL archive directory), and needs to be scheduled to run every
50+
minute. The Barman rpm or debian package automatically creates a
51+
cron entry running every minute as the barman user.
52+
53+
- pgBackRest runs maintenance with the expire command (pgbackrest
54+
expire), but the expire command is run automatically after each
55+
successful backup, and is not required to be separately scheduled.
56+
57+
While migrating to a new tool, make sure that cron is reconfigured to
58+
run the proper retention management commands.

0 commit comments

Comments
 (0)