fix(cron): cancel orphan coroutine on delivery timeout before standalone fallback#13495
fix(cron): cancel orphan coroutine on delivery timeout before standalone fallback#13495VTRiot wants to merge 2 commits intoNousResearch:mainfrom
Conversation
…one fallback When the live adapter delivery path (_deliver_result) or media send path (_send_media_via_adapter) times out at future.result(timeout=N), the underlying coroutine scheduled via asyncio.run_coroutine_threadsafe can still complete on the event loop, causing a duplicate send after the standalone fallback runs. Cancel the future on TimeoutError before re-raising, so the standalone fallback is the sole delivery path. Adds TestDeliverResultTimeoutCancelsFuture and TestSendMediaTimeoutCancelsFuture.
|
Merged via PR #13517 — your commits ( The only change from your PR: the two timeout tests were rewritten to invoke Thanks @VTRiot! |
|
Thanks for the quick merge and the test refactor — calling the actual |
What does this PR do?
In
cron/scheduler.py, two places schedule a coroutine on the gateway event loop viaasyncio.run_coroutine_threadsafe()and then block withfuture.result(timeout=N):_deliver_result()— the main cron delivery path (timeout=60), which falls back to a standalone delivery path on timeout._send_media_via_adapter()— the media send path (timeout=30).When
future.result()raisesTimeoutError, the original coroutine is not cancelled — it may still complete on the event loop afterward.For
_deliver_result(), this causes a concrete duplicate-delivery bug: the live-adapter send times out, the standalone fallback runs and succeeds, and the original coroutine eventually also succeeds → the user receives the same cron output twice.This PR wraps both
future.result(timeout=N)calls with atry/except TimeoutError: future.cancel(); raiseblock. The existing error handling that calls the standalone fallback is untouched —raiselets theTimeoutErrorpropagate to the surroundingexcept Exceptionclause exactly as before, but with the orphan coroutine now properly cancelled.Why this matters more now
After #13021 (
fix(cron): run due jobs in parallel to prevent serial tick starvation), multiple cron jobs run concurrently throughThreadPoolExecutor. The chance of multiple in-flight delivery coroutines sharing the same event loop has increased, which also increases the exposure to this orphan-coroutine bug. Cancelling the future on timeout is a small but proactive fix that complements the parallelisation.Changes Made
cron/scheduler.py_deliver_result()(timeout=60): wrapfuture.result()intry/except TimeoutError: future.cancel(); raise._send_media_via_adapter()(timeout=30): same pattern for orphan-coroutine prevention on the media send path.tests/cron/test_scheduler.pyTestDeliverResultTimeoutCancelsFutureandTestSendMediaTimeoutCancelsFuturethat exercise thetry/except/cancel/raisepattern and assertfuture.cancel()is invoked.scripts/release.pyAUTHOR_MAP(separate commitchore: register VTRiot in AUTHOR_MAP).How to Test
Reproduction scenario (conceptual)
future.result(timeout=60)raisesTimeoutError.After this fix
TimeoutError,future.cancel()aborts the in-flight coroutine before the standalone fallback runs.Automated tests
pytest tests/cron/test_scheduler.py::TestDeliverResultTimeoutCancelsFuture \ tests/cron/test_scheduler.py::TestSendMediaTimeoutCancelsFuture -vBoth tests pass locally. Full
tests/cron/test_scheduler.pysuite: 82 passed / 3 pre-existing failures unrelated to this change (environment-dependentrun_jobintegration tests requiring inference provider credentials).Implementation detail
The pattern mirrors the existing precedent in
tools/mcp_tool.py(the only other in-tree use offuture.cancel()on arun_coroutine_threadsafefuture):concurrent.futures.Future.cancel()on a future returned byasyncio.run_coroutine_threadsafe()callsasyncio.Task.cancel()on the underlying task.cancel()is idempotent: if the coroutine is already completed, it is a no-op. The re-raise preserves the outer control flow unchanged.Checklist
Code
pytest tests/cron/test_scheduler.py -vand the suite shows no regressionDocumentation & Housekeeping