Replace httpx (python)/httpx (project discovery)/curl/ffuf, etc with blasthttp

[liquidsec]

Summary

Replaces BBOT's entire HTTP infrastructure with blasthttp, a Rust-based HTTP library with Python bindings. This eliminates the httpx Go binary subprocess, the curl subprocess helper, and the HTTPEngine ZMQ subprocess — all HTTP now runs in-process through a shared blasthttp client.

What changed

New HTTP engine:

• All HTTP requests go through helpers.request() → WebHelper → shared blasthttp.BlastHTTP() client
• Rate limiting via web.http_rate_limit config, enforced at the client level across all callers
• resolve_ip parameter for DNS pinning (like curl --resolve) — connects to a specific IP while preserving hostname for Host header and TLS SNI
• request_target parameter for request-line override (SSRF/smuggling testing)
• TLS certificate info (CN, SANs, issuer) available on every HTTPS response via response.cert_info

Shared event loop (blasthttp 0.2.0):

• blasthttp upgraded to 0.2.0 which uses pyo3-async-runtimes to return native Python coroutines via future_into_py()
• All run_in_executor_io() wrappers around blasthttp calls replaced with direct await — HTTP requests no longer consume OS threads
• IO thread pool shrunk (only remaining caller is wafw00f)
• Thread pool backlog status line removed (no longer relevant)

Removed:

• httpx Go binary module and its test (bbot/modules/httpx.py)
• ffuf Go binary module and its test (bbot/modules/ffuf.py, ffuf_shortnames.py)
• HTTPEngine ZMQ subprocess (bbot/core/helpers/web/engine.py)
• AsyncClient / httpx Python library (bbot/core/helpers/web/client.py)
• helpers.web.curl() subprocess helper
• DEP_FFUF and DEP_CURL shared dependency definitions
• httpx Python library from dependencies

Added:

• bbot/modules/http.py — native HTTP module using blasthttp batch API (replaces httpx Go binary)
• bbot/modules/web_brute.py — native web fuzzer using blasthttp batch API (replaces ffuf)
• bbot/modules/web_brute_shortnames.py — IIS shortname resolver using ML prediction
• bbot/modules/generic_ssrf.py — SSRF detection module
• bbot/modules/output/webhook.py — renamed from output/http.py to avoid collision with scan module
• bbot/core/helpers/web/blast_response.py — response wrapper for blasthttp PyO3 objects
• bbot/test/mock_blasthttp.py — mock infrastructure for test HTTP interception
• Rate limit test (test_web_rate_limit.py)
• Download timeout default (5 minutes) for large wordlist files

Updated:

• sslcert module rewritten to use blasthttp cert_info instead of independent pyOpenSSL connections
• host_header and generic_ssrf modules converted from curl() to request() with resolve_ip/request_target
• elastic output module fixed to import from webhook instead of deleted http
• All module tests updated for blasthttp mock API (blasthttp_mock.add_response())
• Presets updated: dirbust-light, dirbust-heavy, dotnet-audit reference web_brute instead of ffuf
• Test mock conftest passes through resolve_ip=127.0.0.1 requests to real blasthttp

Bug fixes

• Blacklist.get() NoneType crash — _make_event_seed() could return None when host validation fails, causing AttributeError on .host access. Events hitting this path were silently dropped from the scan pipeline. Added None guard with defensive tests.
• DNS CNAME escaped quotes — clean_dns_record() didn't strip quote characters that dnspython's to_text() can produce on certain record types, causing ValidationError and silently skipping DNS children. Added .strip("'\"") before rstrip(".").
• Certspotter rate-limit crash — API returns a JSON dict (not list) when rate-limited, causing AttributeError: 'str' object has no attribute 'get' when iterating dict keys. Added isinstance(json, list) guard.
• Excavate YARA blocking event loop — yara_rules.match() was called synchronously on the event loop, serializing all 8 excavate workers despite _module_threads = 8. Offloaded to run_in_executor_cpu() for real parallelism (YARA releases the GIL). ~2.5-3x throughput improvement.
• Stale ip-* and http-title-* tags — The new http module was creating ip-{hostname} and http-title-{title} tags instead of using _resolved_hosts and http_title like the old httpx module did. Fixed to match post-naming-standardization conventions from PR Preset naming standardization / tag cleanup #2986.
• _drain_queues() infinite loop on Ctrl+C — When a module's queue was None/False, the drain loop never raised QueueEmpty and spun forever, causing the scan to hang after "Aborting scan" until a second Ctrl+C. Moved the None/False check outside the loop.
• web_brute sequential bottleneck — _module_threads defaulted to 1, meaning queued URLs were fuzzed one at a time. Bumped to 4. Added configurable concurrency option. Wired the previously unused rate option through to blasthttp request_batch.
• blasthttp rate limit override — When both a global rate limit and a per-call rate limit were set, the global always won unconditionally. Fixed in blasthttp 0.1.4 (blacklanternsecurity/blasthttp#12) to use min(global, per_call) so modules can enforce tighter limits.

Dependency changes

• Added: blasthttp>=0.2.0
• Removed: httpx>=0.28.1

[github-actions]

📊 Performance Benchmark Report

Comparing 3.0 (baseline) vs blasthttp-integration-clean (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	4.24ms	4.22ms	-0.6% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	17.27ms	17.19ms	-0.5% ⚪	✅
Large Closest Match Lookup	354.14ms	355.18ms	+0.3% ⚪	✅
Realistic Closest Match Workload	189.18ms	187.51ms	-0.9% ⚪	✅
Event Memory Medium Scan	1784 B/event	1770 B/event	-0.8% ⚪	✅
Event Memory Large Scan	1768 B/event	1760 B/event	-0.5% ⚪	✅
Event Validation Full Scan Startup Small Batch	404.46ms	406.22ms	+0.4% ⚪	✅
Event Validation Full Scan Startup Large Batch	583.11ms	583.71ms	+0.1% ⚪	✅
Make Event Autodetection Small	30.83ms	30.36ms	-1.5% ⚪	✅
Make Event Autodetection Large	314.20ms	311.62ms	-0.8% ⚪	✅
Make Event Explicit Types	13.73ms	13.61ms	-0.9% ⚪	✅
Excavate Single Thread Small	3.981s	3.959s	-0.6% ⚪	✅
Excavate Single Thread Large	9.619s	9.554s	-0.7% ⚪	✅
Excavate Parallel Tasks Small	4.146s	4.140s	-0.1% ⚪	✅
Excavate Parallel Tasks Large	7.254s	6.567s	-9.5% ⚪	✅
Is Ip Performance	3.17ms	3.16ms	-0.3% ⚪	✅
Make Ip Type Performance	11.45ms	11.47ms	+0.2% ⚪	✅
Mixed Ip Operations	4.48ms	4.51ms	+0.6% ⚪	✅
Memory Use Web Crawl	50.7 MB	259.1 MB	+411.3% 🔴🔴🔴	⚠️
Memory Use Subdomain Enum	19.3 MB	19.3 MB	-0.3% ⚪	✅
Scan Throughput 100	7.705s	4.578s	-40.6% 🟢🟢🟢	🚀
Scan Throughput 1000	41.573s	36.668s	-11.8% 🟢🟢	🚀
Typical Queue Shuffle	63.68µs	66.39µs	+4.3% ⚪	✅
Priority Queue Shuffle	727.07µs	740.63µs	+1.9% ⚪	✅

🎯 Performance Summary

+ 2 improvements 🚀
! 1 regression ⚠️
  21 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Web Crawl: 411.3% 🐌 more memory
• Scan Throughput 100: 40.6% 🚀 faster
• Scan Throughput 1000: 11.8% 🚀 faster

---

🐍 Python Version 3.11.15

[codecov]

Codecov Report

❌ Patch coverage is 91.15646% with 182 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (6b359ac) to head (57b6b38).
⚠️ Report is 12 commits behind head on 3.0.

Files with missing lines	Patch %	Lines
bbot/modules/web_brute.py	77%	54 Missing ⚠️
bbot/core/helpers/web/web.py	88%	26 Missing ⚠️
bbot/test/mock_blasthttp.py	90%	21 Missing ⚠️
bbot/core/helpers/web/blast_response.py	83%	16 Missing ⚠️
bbot/modules/generic_ssrf.py	91%	14 Missing ⚠️
bbot/scanner/scanner.py	34%	10 Missing ⚠️
bbot/modules/http.py	95%	9 Missing ⚠️
bbot/modules/sslcert.py	80%	8 Missing ⚠️
bbot/test/test_step_1/test_python_api.py	67%	5 Missing ⚠️
bbot/core/helpers/command.py	67%	4 Missing ⚠️
... and 9 more

Additional details and impacted files

@@          Coverage Diff           @@
##             3.0   #2992    +/-   ##
======================================
- Coverage     91%     91%    -0%     
======================================
  Files        440     443     +3     
  Lines      37230   37838   +608     
======================================
+ Hits       33711   34261   +550     
- Misses      3519    3577    +58     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.