Skip to content

Commit 4c2ba07

Browse files
committed
[doc] Add doc for precommit-compact
1 parent 74408a5 commit 4c2ba07

File tree

2 files changed

+15
-2
lines changed

2 files changed

+15
-2
lines changed

docs/content/append-table/streaming.md

+14-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,20 @@ You can streaming write to the Append table in a very flexible way through Flink
3030
Flink, using it like a queue. The only difference is that its latency is in minutes. Its advantages are very low cost
3131
and the ability to push down filters and projection.
3232

33-
## Automatic small file merging
33+
## Pre small files merging
34+
35+
Pre means that this compact occurs before committing files to the snapshot.
36+
37+
If Flink's checkpoint interval is short (for example, 30 seconds), each snapshot may produce lots of small changelog
38+
files. Too many files may put a burden on the distributed storage cluster.
39+
40+
In order to compact small changelog files into large ones, you can set the table option `precommit-compact = true`.
41+
Default value of this option is false, if true, it will add a compact coordinator and worker operator after the writer
42+
operator, which copies changelog files into large ones.
43+
44+
## Post small files merging
45+
46+
Post means that this compact occurs after committing files to the snapshot.
3447

3548
In streaming writing job, without bucket definition, there is no compaction in writer, instead, will use
3649
`Compact Coordinator` to scan the small files and pass compaction task to `Compact Worker`. In streaming mode, if you

docs/content/primary-key-table/changelog-producer.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,6 @@ For `input`, `lookup`, `full-compaction` 'changelog-producer'.
138138
If Flink's checkpoint interval is short (for example, 30 seconds) and the number of buckets is large, each snapshot may
139139
produce lots of small changelog files. Too many files may put a burden on the distributed storage cluster.
140140

141-
In order to compact small changelog files into large ones, you can set the table option `changelog.precommit-compact = true`.
141+
In order to compact small changelog files into large ones, you can set the table option `precommit-compact = true`.
142142
Default value of this option is false, if true, it will add a compact coordinator and worker operator after the writer
143143
operator, which copies changelog files into large ones.

0 commit comments

Comments
 (0)