feat(puffin): add format constants, utilities, and JSON serialization#603
feat(puffin): add format constants, utilities, and JSON serialization#603zhaoxuan1994 wants to merge 1 commit intoapache:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds core Puffin file-format helpers and JSON (de)serialization support for Puffin metadata, plus unit tests and build-system wiring to compile/run the new components.
Changes:
- Introduce
puffin_formatconstants/utilities (footer flags, little-endian I/O, compression dispatch stubs). - Add
puffin_json_internalJSON serialization/deserialization forBlobMetadataandFileMetadata. - Add
puffin_testand register it in both CMake and Meson test targets.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/iceberg/puffin/puffin_format.h |
Declares Puffin format constants and helper APIs (flags, endian I/O, compress/decompress). |
src/iceberg/puffin/puffin_format.cc |
Implements flags/endian helpers and compression dispatch stubs. |
src/iceberg/puffin/puffin_json_internal.h |
Declares JSON (de)serialization APIs for Puffin metadata. |
src/iceberg/puffin/puffin_json_internal.cc |
Implements JSON (de)serialization and string parsing for FileMetadata. |
src/iceberg/test/puffin_test.cc |
Adds unit tests for endian/flags/compression stubs and JSON round-trips. |
src/iceberg/test/CMakeLists.txt |
Registers puffin_test in CMake-based test build. |
src/iceberg/test/meson.build |
Registers puffin_test in Meson-based test build. |
src/iceberg/meson.build |
Adds new Puffin .cc sources to the main Meson library build. |
src/iceberg/CMakeLists.txt |
Adds new Puffin .cc sources to the main CMake library build. |
src/iceberg/puffin/meson.build |
Updates Puffin header installation list. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| /// Default compression codec for footer payload. | ||
| static constexpr PuffinCompressionCodec kFooterCompressionCodec = | ||
| PuffinCompressionCodec::kLz4; |
There was a problem hiding this comment.
PuffinFormat::kFooterCompressionCodec is set to kLz4, but Compress/Decompress currently return NotSupported for LZ4. This constant is likely to be used as the default when writing/reading footer payloads, and will cause immediate failures once that code lands. Consider defaulting to kNone until LZ4 support is implemented, or clearly documenting that the default codec is not yet supported and must not be used for writing.
| PuffinCompressionCodec::kLz4; | |
| PuffinCompressionCodec::kNone; |
zhaoxuan1994
force-pushed
the
feat/puffin-format-utils
branch
from
a9a1d22 to
18a4dcd
Compare
zhaoxuan1994
force-pushed
the
feat/puffin-format-utils
branch
from
18a4dcd to
39d4a91
Compare
| ICEBERG_EXPORT void WriteInt32LittleEndian(int32_t value, std::span<uint8_t, 4> output); | ||
|
|
||
| /// \brief Read a 32-bit integer from a fixed-size span in little-endian format. | ||
| ICEBERG_EXPORT int32_t ReadInt32LittleEndian(std::span<const uint8_t, 4> input); |
There was a problem hiding this comment.
There is very code for this in the roaring bitmap code it should probably standardized in a common place?
|
|
||
| /// \brief Decompress data using the specified codec. | ||
| ICEBERG_EXPORT Result<std::vector<uint8_t>> Decompress(PuffinCompressionCodec codec, | ||
| std::span<const uint8_t> input); |
There was a problem hiding this comment.
Roaring bitmap is using string_view as collection of bytes, we should probably standardize on one of them.
|
|
||
| /// \brief Deserialize a FileMetadata from a JSON string. | ||
| ICEBERG_EXPORT Result<FileMetadata> FileMetadataFromJsonString( | ||
| const std::string& json_string); |
There was a problem hiding this comment.
nit: std::string_view?
There was a problem hiding this comment.
seems like tests should be seperated per .cc?
Second PR in the Puffin series (following #588).
puffin_format.h/.cc: format constants, footer flag operations, little-endian I/O, compress/decompress dispatch (LZ4/Zstd stubbed as NotSupported)
puffin_json_internal.h/.cc: JSON serialization for BlobMetadata and FileMetadata