Related to #3195

Overview

Today, rdbcompression only affects individual string payloads inside an otherwise normal RDB stream. This PR adds Valkey Compressed Stream (VKCS) support for RDB persistence, with lz4 as the first supported whole-stream codec.

The RDB can now be wrapped in a VKCS envelope and compressed as a single stream at the rio layer. The default behavior remains unchanged with lzf. On load, the RDB path probes for the envelope first. If present and valid, the input is transparently decompressed before normal RDB parsing continues. If absent, loading falls back to the existing plain RDB path. If the envelope is malformed or incompatible, the load fails early.

Architecture

+------------------------+       +------------------------------+       +----------------------+
|    RDB save/load       |       |         rio wrappers         |       |      transport       |
|                        |       |                              |       |                      |
| rdbSaveRio             |------>| compress_rio_t               |------>| dump.rdb / sync RDB  |
| rdbLoadRio             |<------| decompress_rio_t             |<------| file rio             |
| rdbSaveRawString       |       | stream_writer / reader       |       |                      |
| valkey-check-rdb       |       | VKCS envelope + codec state  |       |                      |
+------------------------+       +------------------------------+       +----------------------+

Key Design Decisions

1. Whole-stream compression is implemented as a rio decorator rather than as part of object serialization.
2. VKCS adds a small envelope ahead of the compressed payload so loaders can classify the stream before normal RDB parsing begins.
3. lzf remains the default and plain RDB compatibility is preserved.
4. Per-string LZF is skipped when whole-stream compression is active, so the two compression modes are not stacked.
5. stream_kind validation rejects incompatible VKCS streams early.
6. rdbLoad and valkey-check-rdb both handle plain and VKCS-wrapped inputs transparently.
7. Diskless replication stays on the existing non-VKCS path in this PR.

Data Flow: Save Path

rdbSave()
   |
   | if rdbcompression=yes and algo=lz4
   v
rioInitWithCompress(...)
   |
   | write VKCS envelope
   | feed normal RDB bytes into LZ4 frame encoder
   v
dump.rdb

Important notes:

• The VKCS header stores magic, version, codec, flags, and stream_kind.
• rdbchecksum yes enables codec-native integrity signaling for the compressed stream.
• When whole-stream compression is active, the existing per-string LZF path is bypassed.

Data Flow: Load Path

rdbLoad() / valkey-check-rdb
   |
   | rdbInputStreamPrepare()
   | probe first bytes
   v
[plain RDB] ------------------------> passthrough rio -> normal RDB parser
[VKCS + kind=RDB + valid header] ---> decompress_rio -> normal RDB parser
[malformed/incompatible VKCS] ------> fail early

This PR is intentionally scoped to RDB streaming compression and the persistence/load paths that need to understand it. It does not add diskless sync and replication compression APIs.

Benchmarks

Benchmarked on r7g.2xlarge (Graviton, 8 vCPUs, 61GB RAM, NVMe). All results averaged over 3 repeats.

Datasets

• Improved Realistic JSON: Synthetic JSON with natural language text and varied field types. This data is more compressible due to natural language. Tested at 100B–10KB value sizes.

• BlockMesh Tweets — 1M/5M unique real tweets from BlockMesh/tweets. Multilingual, avg ~270B, zero cross-key repetition.

Notes:

1. LZ4 streaming beats LZF on every metric at every size. 30-77% faster saves, 24-73% faster loads, 45-73% smaller RDBs.
2. LZ4 library is currently vendored from https://github.com/lz4/lz4. The decision was taken here [NEW] RDB Compression via LZ4, Batching, and Batch-Level Dictionary #1962 (comment)