Skip to content

Commit 9edb28e

Browse files
drothery-edbdrothery-edb
authored
Merge pull request #1900 from EnterpriseDB/docs/livecompare/initial-import
LiveCompare initial import
2 parents c22377d + 170194b commit 9edb28e

15 files changed

+3029
-3
lines changed

‎.gitignore

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,7 @@ dev-sources.json
8080
product_docs/content/
8181
product_docs/content_build/
8282
static/nginx_redirects.generated
83-
temp_kubernetes/
84-
temp_bdr/
85-
temp_pglogical3/
83+
temp_*/
8684

8785
# Track base direnv file
8886
!.envrc

‎build-sources.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
"eprs": true,
99
"hadoop_data_adapter": true,
1010
"jdbc_connector": true,
11+
"livecompare": true,
1112
"migration_portal": true,
1213
"migration_toolkit": true,
1314
"mysql_data_adapter": true,

‎gatsby-config.js

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@ const sourceToPluginConfig = {
3232
name: "jdbc_connector",
3333
path: "product_docs/docs/jdbc_connector",
3434
},
35+
livecompare: {
36+
name: "livecompare",
37+
path: "product_docs/docs/livecompare",
38+
},
3539
migration_portal: {
3640
name: "migration_portal",
3741
path: "product_docs/docs/migration_portal",

‎product_docs/docs/livecompare/1.17/advanced_usage.mdx

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
---
2+
navTitle: Advanced usage
3+
title: Advanced Usage
4+
originalFilePath: advanced_usage.md
5+
6+
---
7+
8+
After the end of execution of LiveCompare, you will notice it created a folder
9+
called `lc_session_<session_id>` in the working directory. This folder contains
10+
the following files:
11+
12+
- `lc_<execution_mode>_<current_date>.log`: log file for the session;
13+
14+
- `summary_<current_date>.out`: shows a list of all tables that were processed,
15+
and for each table it shows the time LiveCompare took to process the table, the
16+
total number of rows and how many rows were processed, how many
17+
differences were found in the table, and also the maximum number of ignored columns,
18+
if any.
19+
20+
To get the complete summary, you can also execute the following query against
21+
the output database:
22+
23+
```postgresql
24+
select *
25+
from <output_schema>.vw_table_summary
26+
where session_id = <session_id>;
27+
```
28+
29+
- `differences_<current_date>.out`: if there are any differences, this file
30+
shows useful information about each difference. This file is not generated if
31+
there are no differences.
32+
33+
For example, the difference list could be like this:
34+
35+
```text
36+
+-------------------+-------------------------+-----------------+---------------------+
37+
| table_name | table_pk_column_names | difference_pk | difference_status |
38+
|-------------------+-------------------------+-----------------+---------------------|
39+
| public.categories | category | (7) | P |
40+
| public.categories | category | (10) | P |
41+
| public.categories | category | (17) | P |
42+
| public.categories | category | (18) | P |
43+
+-------------------+-------------------------+-----------------+---------------------+
44+
```
45+
46+
To get the full list of differences with all details, you can also execute the
47+
following query against the output database:
48+
49+
```postgresql
50+
select *
51+
from <output_schema.vw_differences
52+
where session_id = <session_id>;
53+
```
54+
55+
To understand how LiveCompare consensus worked to decide which databases are
56+
divergent, then the view `vw_consensus` can provide details on the consensus
57+
algorithm:
58+
59+
```postgresql
60+
select *
61+
from <output_schema.vw_consensus
62+
where session_id = <session_id>;
63+
```
64+
65+
- `apply_on_the_first_<current_date>.sql`: if there are any differences, this
66+
file will show a DML command to be applied on the **first** database, to make
67+
the **first** database consistent all other databases. For example, for the
68+
differences above, this script could be:
69+
70+
```postgresql
71+
BEGIN;
72+
73+
DELETE FROM public.categories WHERE (category) = 7;
74+
UPDATE public.categories SET categoryname = $lc1$Games Changed$lc1$ WHERE (category) = 10;
75+
INSERT INTO public.categories (category,categoryname) VALUES (17, $lc1$Test 1$lc1$);
76+
INSERT INTO public.categories (category,categoryname) VALUES (18, $lc1$Test 2$lc1$);
77+
78+
COMMIT;
79+
```
80+
81+
LiveCompare generates this script automatically. In order to fix the
82+
inconsistencies in the **first** database, you can simply execute the script in
83+
the **first** database.
84+
85+
LiveCompare generates a similar `apply_on_*.sql` script for each database that
86+
has inconsistent data.
87+
88+
## Which differences to fix
89+
90+
LiveCompare is able to identify and provide fixes for the following differences:
91+
92+
- A row exists in the majority of the data connections. The fix will be an
93+
`INSERT` on the divergent databases;
94+
- A row does not exist in the majority of the data connections. The fix will be
95+
a `DELETE` on the divergent databases;
96+
- A row exists in all databases, but some column values mismatch. The fix will
97+
be an `UPDATE` on the divergent databases.
98+
99+
By default `difference_statements = all`, which means that LiveCompare will try
100+
to apply all 3 DML types (`INSERT`, `UPDATE` and `DELETE`) for each difference
101+
it finds. But it is possible to specify which type of DML LiveCompare should
102+
consider when providing difference fixes, by changing the value of
103+
the setting `difference_statements`, which can be:
104+
105+
- `all` (default): Fixes `INSERT`s, `UPDATE`s and `DELETE`s;
106+
- `inserts`: Fixes only `INSERT`s;
107+
- `updates`: Fixes only `UPDATE`s;
108+
- `deletes`: Fixes only `DELETE`s;
109+
- `inserts_updates`: Fixes only `INSERT`s and `UPDATE`s;
110+
- `inserts_deletes`: Fixes only `INSERT`s and `DELETE`s;
111+
- `updates_deletes`: Fixes only `UPDATE`s and `DELETE`s.
112+
113+
When `difference_statements` has the values `all`, `updates`, `inserts_updates`
114+
or `updates_deletes`, then it is possible to tell LiveCompare to ignore any
115+
`UPDATE`s that would set `NULL` to a column.
116+
117+
## Difference log
118+
119+
Table `difference_log` stores all information about differences every time
120+
LiveCompare checked them. Users can run LiveCompare in re-check mode multiple
121+
times, so this table shows how the difference has evolved over the time window
122+
where LiveCompare was re-checking it.
123+
124+
- **Detected (D)**: The difference was just detected. In re-check and fix modes,
125+
LiveCompare will mark all Permanent and Tie differences as Detected in order to
126+
re-check them.
127+
128+
- **Permanent (P)**: After having re-checked the difference, if data is still
129+
divergent, LiveCompare marks the difference as **Permanent**.
130+
131+
- **Tie (T)**: Same as Permanent, but there is not enough consensus to determine
132+
which connections are the majority.
133+
134+
- **Absent (A)**: If upon a re-check LiveCompare finds that the difference does
135+
not exist anymore (the row is now consistent between both databases), then
136+
LiveCompare marks the difference as **Absent**.
137+
138+
- **Volatile (V)**: If upon a re-check `xmin` has changed on an inconsistent
139+
row, then LiveCompare marks the difference as **Volatile**.
140+
141+
- **Ignored (I)**: Users can stop difference re-check of certain differences by
142+
manually calling the function
143+
`<livecompare_schema_name>.accept_divergence(session_id, table_name, difference_pk)`
144+
in the Output PostgreSQL connection. For example:
145+
146+
```postgresql
147+
SELECT livecompare.accept_divergence(
148+
2 -- session_id
149+
, 'public.categories' -- table_name
150+
, $$(10)$$ -- difference_pk
151+
);
152+
```

0 commit comments

Comments
 (0)