Skip to content

iobuf/tests: scale as_scattered batch by input size#30427

Open
travisdowns wants to merge 1 commit intoredpanda-data:devfrom
travisdowns:td-iobuf-rpbench-investigation
Open

iobuf/tests: scale as_scattered batch by input size#30427
travisdowns wants to merge 1 commit intoredpanda-data:devfrom
travisdowns:td-iobuf-rpbench-investigation

Conversation

@travisdowns
Copy link
Copy Markdown
Member

The as_scattered_* PERF_TESTs in iobuf_bench.cc build n deep copies of
their source iobuf upfront before the timed region. With n=inner_iters
(1000) and the large-case source of 965 KiB, the upfront allocation peaked
at ~942 MiB — within the 1 GiB benchmark budget when the test ran in
isolation, but not after the cumulative residual from earlier PERF_TESTs in
the same binary. This caused iobuf_rpbench_test to crash intermittently
in the smoke test with a seastar bad_alloc.

Compute n inside as_scattered_bench from source.size_bytes() so that
upfront allocation is capped at ~100 MiB (10% of the 1 GiB budget) for any
input size. Per-iteration cost is unchanged (instructions per averaged
iteration are within measurement noise of before).

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v25.3.x
  • v25.2.x
  • v25.1.x

Release Notes

  • none

The as_scattered benchmark builds n deep copies of the source iobuf
upfront before timing.  With n=inner_iters (1000) and a 965 KiB source,
the as_scattered_large case peaked at ~942 MiB which fit the 1 GiB
benchmark budget alone but overflowed it once cumulative residual from
earlier PERF_TESTs in the binary was accounted for, causing the
iobuf_rpbench_test smoke test to crash with a seastar bad_alloc.

Compute n inside as_scattered_bench so the upfront allocation is capped
at ~100 MiB (10% of the 1 GiB budget) for any input size.  Per-iteration
cost is unchanged (inst column is per averaged iteration):

  test                          inst before    inst after
  iobuf.as_scattered_small         5288.61       5288.61
  iobuf.as_scattered_large        23247.02      23086.50
  iobuf.as_scattered_many_frags 2214615.3     2214693.0
Copilot AI review requested due to automatic review settings May 8, 2026 21:06
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts the as_scattered_* iobuf microbenchmarks to avoid large upfront allocations that can intermittently trigger Seastar bad_alloc in CI when multiple PERF_TESTs run in the same binary.

Changes:

  • Compute the as_scattered_bench batch size (n) from source.size_bytes() to cap pre-timed deep-copy allocation to ~100 MiB.
  • Update as_scattered_small/large/many_frags PERF_TESTs to use the new as_scattered_bench(const iobuf&) signature.
  • Document the intent of the new sizing logic next to the benchmark helper.

Comment thread src/v/bytes/tests/iobuf_bench.cc
@vbotbuildovich
Copy link
Copy Markdown
Collaborator

CI test results

test results on build#84242
test_status test_class test_method test_arguments test_kind job_url passed reason test_history
FLAKY(PASS) ShadowLinkBasicTests test_link_creation_checks {"source_cluster_spec": {"cluster_type": "redpanda"}} integration https://buildkite.com/redpanda/redpanda/builds/84242#019e0976-744a-4883-8cb0-7a546330c986 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0022, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkBasicTests&test_method=test_link_creation_checks
FLAKY(PASS) ShadowLinkingReplicationTests test_with_restart {"storage_mode": "cloud"} integration https://buildkite.com/redpanda/redpanda/builds/84242#019e0977-fddf-4470-864b-719a6fc7344f 28/31 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0306, p0=0.2331, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.4114, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ShadowLinkingReplicationTests&test_method=test_with_restart

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants