Reduce event loop saturation for scan throughput

[liquidsec]

Summary

• Offload CPU-bound work (DeepDiff, xmltodict, ML prediction) from event loop to run_in_executor_cpu
• Reduce module thread counts (paramminer 12→4, iis_shortnames 8→4, excavate 8→6, badsecrets capped at 4)
• Replace request_batch/request_custom_batch task-per-URL pattern with fixed worker pool
• Remove unnecessary async lock acquisitions from module worker pre/postchecks
• Binary search for paramminer count_test instead of linear probe
• Single mid-scan canary check per extension in web_brute instead of per-hit

Results (same scan, same targets)

Metric	Before	After
paramminer loop share	43.0%	16.5%
Peak concurrent tasks	579	317
URLs produced	237 (89 min)	1,059 (63 min)
cloudcheck backlog	15,015	gone

[github-actions]

📊 Performance Benchmark Report

Comparing blasthttp-integration-clean (baseline) vs eventloop-optimization (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.17ms	4.24ms	+1.7% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.21ms	17.41ms	+1.2% ⚪	✅
Large Closest Match Lookup	350.37ms	356.67ms	+1.8% ⚪	✅
Realistic Closest Match Workload	188.33ms	187.92ms	-0.2% ⚪	✅
Event Memory Medium Scan	1772 B/event	1773 B/event	+0.1% ⚪	✅
Event Memory Large Scan	1760 B/event	1760 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	414.61ms	410.41ms	-1.0% ⚪	✅
Event Validation Full Scan Startup Large Batch	572.78ms	581.74ms	+1.6% ⚪	✅
Make Event Autodetection Small	30.64ms	31.05ms	+1.3% ⚪	✅
Make Event Autodetection Large	310.79ms	315.23ms	+1.4% ⚪	✅
Make Event Explicit Types	13.78ms	13.96ms	+1.3% ⚪	✅
Excavate Single Thread Small	3.919s	3.983s	+1.6% ⚪	✅
Excavate Single Thread Large	9.915s	9.861s	-0.5% ⚪	✅
Excavate Parallel Tasks Small	4.153s	4.180s	+0.6% ⚪	✅
Excavate Parallel Tasks Large	6.593s	6.680s	+1.3% ⚪	✅
Is Ip Performance	3.14ms	3.23ms	+2.9% ⚪	✅
Make Ip Type Performance	11.45ms	11.64ms	+1.7% ⚪	✅
Mixed Ip Operations	4.48ms	4.57ms	+2.1% ⚪	✅
Memory Use Web Crawl	246.6 MB	254.4 MB	+3.2% ⚪	✅
Memory Use Subdomain Enum	19.3 MB	19.4 MB	+0.3% ⚪	✅
Scan Throughput 100	4.550s	4.499s	-1.1% ⚪	✅
Scan Throughput 1000	36.533s	37.023s	+1.3% ⚪	✅
Typical Queue Shuffle	64.61µs	63.87µs	-1.1% ⚪	✅
Priority Queue Shuffle	725.66µs	730.39µs	+0.7% ⚪	✅

🎯 Performance Summary

✅ No significant performance changes detected (all changes <10%)

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 92.59259% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (64c1afe) to head (c6431ab).

Files with missing lines	Patch %	Lines
bbot/core/helpers/web/web.py	94%	3 Missing ⚠️
bbot/modules/web_brute.py	75%	3 Missing ⚠️
bbot/modules/paramminer_headers.py	88%	2 Missing ⚠️

Additional details and impacted files

@@                     Coverage Diff                     @@
##           blasthttp-integration-clean   #3019   +/-   ##
===========================================================
+ Coverage                           91%     91%   +1%     
===========================================================
  Files                              443     443           
  Lines                            37741   37754   +13     
===========================================================
+ Hits                             34160   34175   +15     
+ Misses                            3581    3579    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.