Skip to content

feat: add direct rewind materialization fast path#123

Merged
Antonov548 merged 13 commits intomainfrom
codex/materialization-direct-rewind
Apr 21, 2026
Merged

feat: add direct rewind materialization fast path#123
Antonov548 merged 13 commits intomainfrom
codex/materialization-direct-rewind

Conversation

@marcus-pousette
Copy link
Copy Markdown
Collaborator

@marcus-pousette marcus-pousette commented Apr 17, 2026

Summary

  • build on #120's frontier model with a direct rewind/replay fast path for common out-of-order insert / move / payload appends
  • keep delete / tombstone on the conservative replay-from-frontier path for now
  • add shared Postgres/sqlite conformance coverage for both the fast path and the conservative fallback path

Why

After #120, materialization state is explicit (head + replay_from) and out-of-order remote ops no longer use a coarse dirty flag.

This PR improves the normal catch-up path for common out-of-order sync cases.

Instead of deferring those cases to a later read and replaying broadly from the op log, append-time catch-up now:

  1. records the earliest invalidation frontier
  2. scans the invalidated suffix in canonical op-key order
  3. rewinds the already-materialized suffix directly on backend stores
  4. truncates suffix oprefs_children
  5. replays the invalidated suffix forward directly on backend stores
  6. clears the frontier when complete

What Changes

For common out-of-order insert / move / payload appends, the normal path no longer relies on:

  • read-time catch-up as the primary repair path
  • full materialized-table reset
  • full op-log replay
  • full in-memory clone of the materialized doc

A narrow payload shortcut also remains in place for dominated out-of-order payload ops that do not change materialized payload state.

Fallback Behavior

This PR is intentionally conservative for the annoying cases:

  • if the invalidated suffix contains delete or tombstone, append falls back to the existing replay-from-frontier catch-up path
  • deferred ensure_materialized recovery still uses the conservative replay path

So the fast path is targeted at the common structural/payload out-of-order cases, while delete/revival semantics stay on the safer route.

Backend Changes

Postgres

  • add ordered frontier-range scans and per-node predecessor lookups in PgOpStorage
  • use the direct rewind/replay fast path during append-time immediate catch-up

sqlite

  • add the same frontier-range scans and per-node predecessor lookups in SqliteOpStorage
  • use the direct rewind/replay fast path during append-time immediate catch-up

Tests

Verified locally with:

  • cargo test -p treecrdt-core --test materialization_helpers
  • cargo test -p treecrdt-postgres --test postgres_test
  • cargo test -p treecrdt-sqlite-ext --features ext-sqlite --test extension_roundtrip
  • pnpm run clippy
  • pnpm -C examples/playground typecheck
  • pnpm -C examples/playground build

The shared backend conformance suite covers:

  • common out-of-order append immediate catch-up
  • dominated out-of-order payload shortcut behavior
  • replay-from-start frontier catch-up
  • deferred ensure_materialized recovery
  • explicit delete/tombstone fallback behavior
  • rollback behavior when immediate catch-up fails

@marcus-pousette marcus-pousette force-pushed the codex/materialization-direct-rewind branch from df92cf5 to 0f59afb Compare April 17, 2026 14:45
@marcus-pousette marcus-pousette changed the base branch from codex/materialization-frontier to main April 17, 2026 14:45
@marcus-pousette marcus-pousette marked this pull request as ready for review April 20, 2026 13:07
@Antonov548 Antonov548 merged commit feabe5b into main Apr 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants