Skip to content

feat(core): add checkpoint-backed materialization catch-up#121

Closed
marcus-pousette wants to merge 11 commits intocybersemics:codex/materialization-frontierfrom
marcus-pousette:codex/materialization-checkpoints
Closed

feat(core): add checkpoint-backed materialization catch-up#121
marcus-pousette wants to merge 11 commits intocybersemics:codex/materialization-frontierfrom
marcus-pousette:codex/materialization-checkpoints

Conversation

@marcus-pousette
Copy link
Copy Markdown
Collaborator

Summary

  • add persisted materialization checkpoints on top of the replay-frontier model from PR feat(core): replace dirty materialization with replay frontier #120
  • restore the latest checkpoint whose head is strictly before the replay frontier, then replay only the suffix after that checkpoint head
  • add key-range op-log scans (scan_after) so Postgres and sqlite can replay from a concrete op-key boundary instead of rescanning from the beginning
  • keep external behavior unchanged while reducing the normal catch-up cost for out-of-order persisted remote ops

Why This Is A Separate PR

Checkpoint Model

  • the op log remains canonical
  • materialization metadata still tracks:
    • the materialized head
    • an optional replay frontier
  • adapters now also persist historical materialization checkpoints at committed heads:
    • first committed head (seq = 1)
    • then every 64 committed materialization seqs
  • catch-up now restores the latest checkpoint whose head is strictly before the replay frontier, then replays only the op-log suffix after that checkpoint head
  • if no earlier checkpoint exists, catch-up falls back to replay-from-start

Core Changes

  • Storage gained scan_after(...) so adapters can stream ops strictly after a concrete (lamport, replica, counter) boundary
  • core catch-up no longer treats replay as “always rebuild from the beginning” in the normal path
  • core now asks the backend for:
    • the latest checkpoint before the frontier
    • checkpoint restore into live materialized stores
    • suffix replay after the restored checkpoint head

Postgres

  • new checkpoint tables:
    • treecrdt_checkpoints
    • treecrdt_checkpoint_nodes
    • treecrdt_checkpoint_payload
    • treecrdt_checkpoint_oprefs_children
  • ensure_materialized_in_tx now restores the latest checkpoint before the frontier and replays only the suffix after that checkpoint head
  • successful head updates during append/local/catch-up also persist checkpoints when the seq crosses the checkpoint policy
  • Postgres overrides scan_after(...) with a SQL key-range scan over (lamport, replica, counter)

SQLite

  • new checkpoint tables:
    • materialization_checkpoints
    • checkpoint_nodes
    • checkpoint_payload
    • checkpoint_oprefs_children
  • treecrdt_ensure_materialized() now restores the latest checkpoint before the frontier and replays only the suffix after that checkpoint head
  • successful head updates during append/local/catch-up also persist checkpoints when the seq crosses the checkpoint policy
  • sqlite overrides scan_after(...) with the equivalent SQL key-range scan

Cases Improved By This PR

  • normal out-of-order persisted remote ops no longer imply replay-from-start when a prior checkpoint exists
  • catch-up after restart can restore the latest checkpoint before the frontier instead of reconstructing the entire materialized prefix from the beginning of the log
  • Postgres and sqlite now share the same checkpointed catch-up behavior

Remaining Fallback

  • if a replay frontier predates the earliest retained checkpoint, catch-up still falls back to replay-from-start
  • this PR does not implement delta checkpoints or in-place suffix mutation of existing materialized rows; it uses persisted snapshots plus suffix replay

Tests

  • cargo test -p treecrdt-core
  • cargo test -p treecrdt-postgres --test postgres_test
  • cargo test -p treecrdt-sqlite-ext --features ext-sqlite --test extension_roundtrip
  • pnpm run clippy

@marcus-pousette
Copy link
Copy Markdown
Collaborator Author

Superseded by #123, which is the preferred follow-up to #120.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant