Skip to content

Add LZ4 compression support#7

Merged
vbp1 merged 5 commits intomasterfrom
feature/lz4-compression-upd
Dec 25, 2025
Merged

Add LZ4 compression support#7
vbp1 merged 5 commits intomasterfrom
feature/lz4-compression-upd

Conversation

@vbp1
Copy link
Copy Markdown
Owner

@vbp1 vbp1 commented Dec 14, 2025

Summary

  • Add LZ4 compression algorithm support for backup data compression
  • Support both fast LZ4 compression (level 0) and LZ4_HC high compression (levels 1-12)
  • Update documentation with LZ4 usage examples and build requirements

Changes

  • Implement LZ4 compression/decompression in src/data.c
  • Add LZ4 algorithm to compression options in src/pg_probackup.h
  • Update Makefile to link with liblz4
  • Add comprehensive tests for LZ4 compression including:
    • Basic stream/archive backups with LZ4
    • Different compression levels (0, 6, 12)
    • Mixed compression chains (zlib + lz4)
    • Merge operations with LZ4 backups
    • Edge cases (incompressible data, empty/sparse tables)

Build requirements

  • liblz4-dev package required for LZ4 support

Usage

# Fast LZ4 compression
pg_probackup backup -B /backup --instance main --compress-algorithm=lz4

# LZ4 high compression (better ratio, slower)
pg_probackup backup -B /backup --instance main --compress-algorithm=lz4 --compress-level=6

vbp1 added 5 commits December 15, 2025 15:31
- Add LZ4_COMPRESS to CompressAlg enum
- Implement lz4_compress/lz4_decompress functions using LZ4 and LZ4_HC
- Support compression levels 0 (fast) and 1-12 (HC high compression)
- Add liblz4 detection in Makefile via pkg-config
- Add liblz4-dev to Travis CI dependencies
- Add runtime validation for LZ4 compression level
- Update help strings to include 'lz4' option
- Add compression tests for LZ4 (stream, archive, levels)
Add new tests for LZ4 compression feature:
- test_merge_lz4_backups: MERGE with LZ4 compressed backups chain
- test_validate_lz4_backup: VALIDATE for LZ4 compressed backups
- test_mixed_compression_chain_zlib_lz4: mixed compression chain restore
- test_merge_mixed_compression_zlib_lz4: MERGE with mixed algorithms
- test_lz4_incompressible_data: random/incompressible data handling
- test_lz4_empty_and_sparse_tables: edge cases with empty/sparse tables

Also fix linter warnings:
- Remove unused imports (idx_ptrack, datetime, timedelta, subprocess)
- Remove unused variables in test_lz4_compression_levels
- Apply ruff formatting to entire file
Replace gen_random_bytes() from pgcrypto with md5(random()) concatenation
to generate pseudo-random data. This avoids dependency on pgcrypto
extension which may not be available in CI environment.
@vbp1 vbp1 force-pushed the feature/lz4-compression-upd branch from 5b8fd7b to 5eb7681 Compare December 15, 2025 12:32
@vbp1 vbp1 merged commit 0c8cb1e into master Dec 25, 2025
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant