[benchmark/filtered-search prep] Make benchmarks stateful

[hildebrandmw]

A recurring problem with our current benchmark infrastructure is the SearchPhase enum (selecting what kind of search is conducted) does its job a little too well: every time a new variant is added, we need to either update all users of SearchPhase (bloating compile times) or explicitly opt-out of a particular search phase, which is brittle especially with respect to Benchmark::try_match consistency.

For example see:

• diversity search
• filtered search

This PR takes the first step towards systematically solving this problem by allowing benchmarks registered with diskann_benchmark_runner::Benchmarks to have state rather
than being purely type-level constructs. Stateful benchmarks can have "search plugins" dynamically registered at construction time. These plugins participate in
Benchmark::try_match, Benchmark::description, and Benchmark::run, allowing individual benchmarks to opt into new search-phase variants without requiring changes across all
benchmarks. See #996 as a follow-up implementing this idea

Suggested Reviewing Order

In diskann-benchmark-runner:

• benchmark.rs: This is the main change. It simply changes the Benchmark and Regression traits to receive by &self.
• registry.rs: Change the signatures of Benchmarks::register and Benchmarks::register_regression to receive the benchmark type by-value.
• The rest of the changes are updates to the test infrastructure.

In diskann-benchmark: The main changes involve cleaning up the 'static hack and removing the BuildAndSearch/BuildAndDynamicRun indirection traits that are no longer necessary.

In diskann-benchmark-simd: Feel free to skip.

[Copilot]

Pull request overview

This PR updates the benchmark framework to support stateful benchmarks by switching Benchmark/Regression APIs from type-level (static) methods to instance methods (&self) and updating the registry to register benchmark values (enabling future dynamic “search plugin” registration per benchmark instance).

Changes:

• Convert Benchmark and Regression trait methods (try_match, description, run, check) to take &self.
• Update Benchmarks::register / register_regression to accept a benchmark instance by value and store it behind a type-erased wrapper.
• Refactor benchmark implementations across diskann-benchmark and diskann-benchmark-simd to remove the prior 'static/dispatcher indirection patterns.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
diskann-benchmark/src/utils/mod.rs	Updates stub benchmark registration and trait method signatures to the new &self API.
diskann-benchmark/src/backend/index/spherical.rs	Refactors spherical index benchmarks to unit-like/stateful benchmarks and inlines prior dispatch indirection.
diskann-benchmark/src/backend/index/scalar.rs	Refactors scalar quantized index benchmarks to constructable/stateful benchmark instances.
diskann-benchmark/src/backend/index/product.rs	Refactors PQ index benchmark to a constructable/stateful benchmark instance.
diskann-benchmark/src/backend/index/benchmarks.rs	Removes BuildAndSearch/BuildAndDynamicRun indirection and moves logic directly into Benchmark::run(&self, ...).
diskann-benchmark/src/backend/filters/benchmark.rs	Converts metadata index benchmark into a stateful/unit-like benchmark and extracts run logic into a free function.
diskann-benchmark/src/backend/exhaustive/spherical.rs	Refactors exhaustive spherical benchmarks to unit-like/stateful benchmarks.
diskann-benchmark/src/backend/exhaustive/product.rs	Refactors exhaustive product benchmarks to unit-like/stateful benchmarks.
diskann-benchmark/src/backend/exhaustive/minmax.rs	Refactors exhaustive minmax benchmarks to unit-like/stateful benchmarks.
diskann-benchmark/src/backend/disk_index/benchmarks.rs	Refactors disk index benchmark/regression to stateful benchmarks and updates registration accordingly.
diskann-benchmark-simd/src/lib.rs	Updates SIMD regression benchmarks to be instance-based and adjusts kernel execution to pass arch at call time.
diskann-benchmark-runner/src/test/typed.rs	Updates typed test benchmarks/regressions to instance-based implementations (adds constructors).
diskann-benchmark-runner/src/test/mod.rs	Updates benchmark registration in test harness to pass benchmark instances.
diskann-benchmark-runner/src/test/dim.rs	Updates dim test benchmarks/regression to the new &self trait signatures.
diskann-benchmark-runner/src/registry.rs	Changes registry APIs to accept benchmark instances and stores them in the wrapper.
diskann-benchmark-runner/src/benchmark.rs	Changes core traits to &self methods and updates internal type-erased wrapper and regression plumbing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.55801% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.50%. Comparing base (767879f) to head (74995cc).

Files with missing lines	Patch %	Lines
diskann-benchmark/src/backend/index/benchmarks.rs	45.21%	63 Missing ⚠️
diskann-benchmark-simd/src/lib.rs	95.65%	5 Missing ⚠️
diskann-benchmark/src/backend/filters/benchmark.rs	91.93%	5 Missing ⚠️
diskann-benchmark/src/utils/mod.rs	75.00%	1 Missing ⚠️

❌ Your patch status has failed because the patch coverage (79.55%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #995      +/-   ##
==========================================
+ Coverage   89.48%   89.50%   +0.02%     
==========================================
  Files         448      448              
  Lines       84095    84167      +72     
==========================================
+ Hits        75250    75333      +83     
+ Misses       8845     8834      -11     

Flag	Coverage Δ
miri	89.50% <79.55%> (+0.02%)	⬆️
unittests	89.34% <79.55%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-benchmark-runner/src/benchmark.rs	89.21% <100.00%> (-0.60%)	⬇️
diskann-benchmark-runner/src/registry.rs	88.26% <100.00%> (+0.26%)	⬆️
diskann-benchmark-runner/src/test/dim.rs	89.21% <100.00%> (+1.30%)	⬆️
diskann-benchmark-runner/src/test/mod.rs	100.00% <100.00%> (ø)
diskann-benchmark-runner/src/test/typed.rs	96.47% <100.00%> (+0.51%)	⬆️
diskann-benchmark/src/backend/exhaustive/minmax.rs	100.00% <ø> (ø)
...iskann-benchmark/src/backend/exhaustive/product.rs	100.00% <ø> (ø)
...kann-benchmark/src/backend/exhaustive/spherical.rs	100.00% <ø> (ø)
diskann-benchmark/src/backend/index/product.rs	100.00% <ø> (ø)
diskann-benchmark/src/backend/index/scalar.rs	100.00% <ø> (ø)
... and 5 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[arkrishn94]

Thanks Mark, looks good to me.

[JordanMaples]

lgtm