Skip to content

Experiment: immediate frontier catch-up for out-of-order materialization#122

Closed
marcus-pousette wants to merge 1 commit intocodex/materialization-frontierfrom
codex/materialization-reverse-replay
Closed

Experiment: immediate frontier catch-up for out-of-order materialization#122
marcus-pousette wants to merge 1 commit intocodex/materialization-frontierfrom
codex/materialization-reverse-replay

Conversation

@marcus-pousette
Copy link
Copy Markdown
Collaborator

Summary

This is an experimental follow-up on top of codex/materialization-frontier.

Instead of leaving a recorded replay frontier to be handled later on read, this branch tries to make common out-of-order remote append catch up immediately during append.

What Changed

  • keeps the frontier-based invalidation model from feat(core): replace dirty materialization with replay frontier #120
  • adds a payload no-op shortcut for out-of-order payload ops that are already dominated by a later payload winner
  • replaces broad persisted reset/rebuild with in-place materialized patching of affected nodes/payloads plus suffix oprefs
  • performs immediate append-time catch-up in Postgres and sqlite when catch-up is needed
  • reduces core catch-up from two full op-log scans to one full in-memory scan
  • adds a shared backend conformance test-support crate so Postgres and sqlite run the same scenario coverage
  • adds deferred ensure_materialized recovery coverage and rollback coverage for failed immediate catch-up in both backends

Current Behavior

Normal in-order append still uses incremental materialization.

For common out-of-order append cases, the flow is now:

  1. persist inserted ops
  2. compute the earliest replay frontier if needed
  3. replay from that frontier in memory
  4. patch affected materialized rows and suffix oprefs in place
  5. clear the frontier in the same append flow

So this branch no longer treats common out-of-order append as "persist now, rebuild later on read".

Important Limitation

This branch still reconstructs prefix context by replaying the op log in memory from the start. The cost model is better than the old broad persisted rebuild, but it is not yet true suffix-only replay or undo/rewind.

Why Draft

This is intended as an alternative follow-up direction to the checkpoint-based branch. It improves the normal steady-state path and storage footprint, but it still needs review against the remaining prefix replay cost tradeoff.

@marcus-pousette
Copy link
Copy Markdown
Collaborator Author

Superseded by #123, which replaces this earlier experiment on top of #120.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant