Normalize the number of workers when performing parallel operations

[Edouard-chin]

Note

Benchmarks

Git gems

Benchmark matrix
Ruby: ruby 3.4.8 (2025-12-17 revision 995b59f666) +PRISM [arm64-darwin25]
Source: https://rubygems.org
Iterations: 3
Runs:
  more-downloads: bundler (/Users/edouard/src/opensource/rubygems)
  master: bundler (/Users/edouard/src/opensource/rubygems-clone)

=== Scenario: git-gems (40 gems, more-downloads) ===
  Running cold benchmark (3 runs)...
Benchmark 1: more-downloads (cold)
  Time (mean ± σ):      8.062 s ±  0.166 s    [User: 6.937 s, System: 13.693 s]
  Range (min … max):    7.925 s …  8.246 s    3 runs

  Running warm benchmark (3 runs)...
Benchmark 1: more-downloads (warm)
  Time (mean ± σ):      4.947 s ±  0.292 s    [User: 5.391 s, System: 13.048 s]
  Range (min … max):    4.732 s …  5.280 s    3 runs

  Cold median: 8.02s  Warm median: 4.83s

=== Scenario: git-gems (40 gems, master) ===
  Running cold benchmark (3 runs)...
Benchmark 1: master (cold)
  Time (mean ± σ):     12.101 s ±  0.197 s    [User: 6.874 s, System: 12.217 s]
  Range (min … max):   11.884 s … 12.268 s    3 runs

  Running warm benchmark (3 runs)...
Benchmark 1: master (warm)
  Time (mean ± σ):      5.750 s ±  0.064 s    [User: 5.252 s, System: 10.975 s]
  Range (min … max):    5.709 s …  5.824 s    3 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

  Cold median: 12.15s  Warm median: 5.72s

Results written to /Users/edouard/src/opensource/bundler-bench/results/more-downloads_20260316_194118.json
Results written to /Users/edouard/src/opensource/bundler-bench/results/master_20260316_194118.json

=== Comparison Summary ===

Scenario: git-gems (40 gems)
                             Cold     +/-                        Warm     +/-
  ------------------------------------------------------------------------------
  more-downloads            8.02s   0.17s  baseline             4.83s   0.29s  baseline
  master                   12.15s   0.20s  51.6% slower         5.72s   0.06s  18.4% slower

Gemserver (Fake server with a 300ms latency)

Benchmark matrix
Ruby: ruby 3.4.8 (2025-12-17 revision 995b59f666) +PRISM [arm64-darwin25]
Source: http://localhost:9292
Iterations: 3
Runs:
  more-downloads: bundler (/Users/edouard/src/opensource/rubygems)
  master: bundler (/Users/edouard/src/opensource/rubygems-clone)

=== Scenario: no-deps (249 gems, more-downloads) ===
  Running cold benchmark (3 runs)...
Benchmark 1: more-downloads (cold)
  Time (mean ± σ):     11.219 s ±  0.045 s    [User: 0.772 s, System: 0.735 s]
  Range (min … max):   11.178 s … 11.267 s    3 runs

  Running warm benchmark (3 runs)...
Benchmark 1: more-downloads (warm)
  Time (mean ± σ):     987.0 ms ±  60.0 ms    [User: 494.8 ms, System: 494.6 ms]
  Range (min … max):   928.9 ms … 1048.7 ms    3 runs

  Cold median: 11.21s  Warm median: 0.98s

=== Scenario: no-deps (249 gems, master) ===
  Running cold benchmark (3 runs)...
Benchmark 1: master (cold)
  Time (mean ± σ):     16.346 s ±  0.106 s    [User: 0.799 s, System: 0.750 s]
  Range (min … max):   16.282 s … 16.469 s    3 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (16.469 s). This could be caused by (filesystem) caches that were not filled until after the first run. You are already using the '--prepare' option which can be used to clear caches. If you did not use a cache-clearing command with '--prepare', you can either try that or consider using the '--warmup' option to fill those caches before the actual benchmark.

  Running warm benchmark (3 runs)...
Benchmark 1: master (warm)
  Time (mean ± σ):     872.1 ms ±  79.5 ms    [User: 418.8 ms, System: 452.1 ms]
  Range (min … max):   809.0 ms … 961.4 ms    3 runs

  Cold median: 16.29s  Warm median: 0.85s

Results written to /Users/edouard/src/opensource/bundler-bench/results/more-downloads_20260316_195046.json
Results written to /Users/edouard/src/opensource/bundler-bench/results/master_20260316_195046.json

=== Comparison Summary ===

Scenario: no-deps (249 gems)
                             Cold     +/-                        Warm     +/-
  ------------------------------------------------------------------------------
  more-downloads           11.21s   0.04s  baseline             0.98s   0.06s  baseline
  master                   16.29s   0.11s  45.3% slower         0.85s   0.08s  14.0% faster

What was the end-user or developer problem that led to this PR?

I'd like to normalize the number of workers when downloading gems and use the BUNDLE_JOBS configuration (or default to Etc.nprocessors). Right now the number of workers when doing parallel work seems a bit random.

We can make Bundler download gems even faster.

What is your fix for the problem, implemented in this PR?

Context

I originally added those workers count in:

1. https://github.com/ruby/rubygems/pull/9087/changes#diff-524173391e40a96577540013a1ad749433454155f79aa05c5d0832235b0bdad1R11
2. https://github.com/ruby/rubygems/pull/9100/changes#diff-04ae823e98259f697c78d2d0b4eab0ced6a83a84a986578703eb2837d6db1a32R1105

For 1. (downloading gems from Rubygems.org)I opted to go with a hardcoded worker count of 5 and not anything higher as I was concerned that we could hammer RubyGems.org. I think this concern is not valid, because requests to download gems don't even hit RubyGems.org server as there is the fastly CDN in front of the s3 bucket.

For 2. I went with a worker count of 5 to match, without giving this a second thought.

Solution

Use BUNDLE_JOBS (or default to Etc.nprocessors) as the number of workers.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

cc/ @eileencodes

[kou]

+1

[colby-swandale]

Is there any documentation that we need to update to include this information?

[Edouard-chin]

Thanks good call. I updated the man page to reflect that BUNDLE_JOBS affects the number of parallel downloads, lmk what you think.