Skip to content

Add synthetic-transaction regression bench#3024

Open
Al-Kindi-0 wants to merge 2 commits intonextfrom
al/synthetic-tx-row-targets
Open

Add synthetic-transaction regression bench#3024
Al-Kindi-0 wants to merge 2 commits intonextfrom
al/synthetic-tx-row-targets

Conversation

@Al-Kindi-0
Copy link
Copy Markdown
Contributor

Purpose:

Adds a new benchmark crate under benches/synthetic-tx-kernel/ that reproduces the proving-cost brackets of a real Miden transaction from a small JSON snapshot, without pulling in any protocol-level code. The result is a VM-level regression detector that isolates prover changes from workload changes: when you change the VM, the bench moves; when protocol changes its transaction kernel, miden-vm's bench stays put until a new snapshot is explicitly refreshed.

This is a simplified variant of the approach originally prototyped in 0xMiden/protocol#2391. The key simplification: instead of maintaining a fused cross-repo benchmark that depends on protocol's transaction machinery at runtime, we row counts and emit an equivalent synthetic MASM program on the miden-vm side. The two repos stay loosely coupled via a hand-maintained schema, and miden-vm's bench has no runtime dependency on protocol.
The snapshot producer lives in a companion protocol PR (bin/bench-transaction/src/bin/tx-trace-snapshot.rs).

How it works

Per bench run:

  1. Calibrate each MASM snippet's per-iteration cost against the current VM (once).
  2. For each snapshot JSON: solve for per-snippet iteration counts via fixed-point refinement, emit a begin ... end MASM program, verify it lands in the target padded-trace brackets (padded_core_side and padded_chiplets), then run execute + execute_and_prove Criterion groups.

A bracket miss fails the bench; sub-bracket drift is reported but tolerated because proving cost is driven by padded trace length, not raw row count.

Validations

Wall-clock prove times against protocol's real bench-transaction prove benches:

scenario protocol (real) synthetic (miden-vm) delta % diff
consume-single-p2id 970 ms 998 ms +28 ms +2.9%
consume-two-p2id 1886 ms 1888 ms +2 ms +0.1%

Copy link
Copy Markdown
Collaborator

@huitseeker huitseeker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, this looks great! I added a commit that improves how the snapshot test catches wrong, missing, or extra committed JSON files.

@huitseeker huitseeker marked this pull request as ready for review April 23, 2026 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants