[JS & Python] Add cancellation and structured error handling to live audio streaming

[rui-ren]

Summary

Adds cancellation and structured error handling to the JavaScript and Python SDKs' live audio transcription clients, closing two cross-language API parity gaps from the realtime-audio analysis.

What's new

Structured errors

Language	Surface
JS	New CoreError class code + isTransient from CoreErrorResponse exposed on every error thrown by start / pushLoop / stop. Re-exported from foundry-local-sdk.
Python	FoundryLocalException messages still parse via existing CoreErrorResponse.try_parse; sample now demonstrates how to use it for transient-error recovery.

Cancellation

Language	Surface
JS	audioClient.createLiveTranscriptionSession(signal?: AbortSignal) set once on the session, applies to every subsequent start / append / stop / getTranscriptionStream call. Mirrors C# CancellationToken shape.
Python	audio_client.create_live_transcription_session(cancel_event: Optional[threading.Event] = None) same pattern. Backpressured append() and idle get_transcription_stream() poll a documented _CANCEL_POLL_INTERVAL (100 ms) when cancellation is configured. Zero overhead on the fast path.

Both APIs are intentionally symmetric: cancellation is configured once on the constructor, not threaded through every method. Aborted append() does NOT enqueue the chunk (no risk of late delivery to native core). Aborted session also best-effort releases the native handle so it doesn't leak.

Files changed

File	Change
sdk/js/src/openai/liveAudioTranscriptionTypes.ts	New CoreError class + wrapCoreError() helper
sdk/js/src/openai/liveAudioTranscriptionClient.ts	AbortSignal plumbing, abort-aware AsyncQueue.write(), listener-leak fixes, native-handle release on abort
sdk/js/src/openai/audioClient.ts	createLiveTranscriptionSession(signal?)
sdk/js/src/index.ts	Re-exports CoreError
sdk/js/test/openai/liveAudioTranscription.test.ts	+CoreError tests, +AbortSignal listener-cleanup test (Proxy-wrapped to actually count listeners), +non-Error signal.reason test
samples/js/live-audio-transcription/app.js	Demonstrates CoreError + AbortSignal graceful shutdown
sdk/python/src/openai/audio_client.py	create_live_transcription_session(cancel_event=...)
sdk/python/src/openai/live_audio_transcription_client.py	cancel_event plumbing, _CANCEL_POLL_INTERVAL constant
sdk/python/test/openai/test_live_audio_transcription.py	+cancellation tests covering pre-set cancel, backpressure unblocking, generator clean exit
samples/python/live-audio-transcription/src/app.py	Demonstrates cancel_event + CoreErrorResponse.try_parse

Tests

• JS: 19/19 live-audio tests pass
• Python: 25/25 live-audio tests pass
• E2E synthetic-audio test still skips gracefully when the Nemotron model is not cached
• Both samples syntax-verified

Backwards compatibility

100% every new parameter is optional. Existing callers see no behavior change.

Out of scope (parity report items not in this PR)

• Missing id field in C++/Rust LiveAudioTranscriptionResponse addressed in [Rust & Python] Add id field to LiveAudioTranscriptionResponse #687 / [C++] Fix Nemotron-ASR of LiveAudioTranscriptionResponse #689.
• C++ polling vs. async streams different SDK; separate refactor.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	May 2, 2026 8:39pm

[Copilot]

Pull request overview

This PR improves the JS live audio transcription streaming API by adding (1) structured native-core error surfacing via a new CoreError type, and (2) optional cancellation via AbortSignal across streaming operations to close cross-language parity gaps.

Changes:

• Introduces CoreError + wrapCoreError() to parse and surface code / isTransient from native-core error payloads.
• Adds optional AbortSignal support to start(), append(), stop(), and getTranscriptionStream().
• Re-exports CoreError from the JS SDK entrypoint and adds unit tests for structured/unstructured wrapping behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
sdk/js/src/openai/liveAudioTranscriptionTypes.ts	Adds CoreError and wrapCoreError() to provide structured error handling for live audio streaming.
sdk/js/src/openai/liveAudioTranscriptionClient.ts	Adds AbortSignal support and switches several error paths to throw CoreError instead of plain Error.
sdk/js/src/index.ts	Re-exports CoreError from the public package entrypoint.
sdk/js/test/openai/liveAudioTranscription.test.ts	Adds unit tests covering CoreError wrapping behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[kunal-vaishnavi]

Can you show how these changes would work in the JS example and how they are different than the below error handling?

Foundry-Local/samples/js/live-audio-transcription/app.js

Lines 69 to 72 in 58bba93

	} catch (err) {
	if (err.name !== 'AbortError') {
	console.error('Stream error:', err.message);
	}

[rui-ren]

Great question the existing catch block is fine for the happy path but throws away every signal we now expose. Pushed 83e8dc4 to the sample (samples/js/live-audio-transcription/app.js) so the diff is concrete:

Before (the lines you linked):

} catch (err) {
    if (err.name !== 'AbortError') {
        console.error('Stream error:', err.message);
    }
}

This treats every non-abort error the same and loses code + isTransient.

After:

} catch (err) {
    if (err.name === 'AbortError') return;
    if (err instanceof CoreError) {
        if (err.isTransient) {
            console.warn(` Transient ASR error (\): \. Continuing...`);
            return;  // recoverable  don't kill the app on a momentary native stall
        }
        console.error(` Stream error [\]: \`);
        return;
    }
    console.error(' Stream error:', err.message);
}

So the user-visible difference is:

• Transient blips (e.g., native-side recoverable hiccup) no longer silently log-and-die the loop continues, which is what a customer building a long-running transcription UI actually wants.
• Hard errors include the machine-readable code, so customers can route on it (telemetry, retry policy, UI message) instead of regex-matching err.message.

For AbortSignal, I also wired it through to show what it buys:

const shutdown = new AbortController();
await session.start(shutdown.signal);
for await (const r of session.getTranscriptionStream(shutdown.signal)) { ... }
await session.append(pcm, shutdown.signal);

process.on('SIGINT', async () => {
    shutdown.abort();              // unblocks any in-flight append/iterator
    if (audioInput) audioInput.quit();
    await session.stop();           // now drain-and-stop without racing
    ...
});

Previously, if session.append() was backpressured (waiting for native-core to drain), Ctrl+C had to wait for stop() to finish draining which could take seconds. With shutdown.abort(), the in-flight append resolves immediately with AbortError, the pump exits, and we shut down cleanly.

Let me know if you'd like the sample changes split out into a separate PR.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rui-ren]

Both points addressed in df4260c.

JS session-level signal (no more passing it everywhere)

const shutdown = new AbortController();
const session = audioClient.createLiveTranscriptionSession({ signal: shutdown.signal });

await session.start();           // signal applies automatically
await session.append(pcm);       // signal applies automatically
for await (const r of session.getTranscriptionStream()) { ... }
await session.stop();

process.on('SIGINT', () => shutdown.abort());

Per-call signals still work as overrides if both session-level and per-call are set, they compose via AbortSignal.any so EITHER aborting cancels. Sample (samples/js/live-audio-transcription/app.js) now uses this one-line pattern.

Python cancellation parity (closes the gap from the analysis doc)

cancel = threading.Event()
signal.signal(signal.SIGINT, lambda *_: cancel.set())

session = audio_client.create_live_transcription_session(cancel_event=cancel)
session.start()
session.append(pcm)
for result in session.get_transcription_stream():
    ...
session.stop()

Same composition rule: each method also accepts an optional per-call cancel_event; either being set cancels. Backpressured append() and idle get_transcription_stream() poll on a 100 ms timeout when cancellation is configured; zero overhead on the fast path with no cancel sources. An aborted append() does NOT enqueue the chunk (no late delivery to native core).

Tests: JS 20/20, Python 26/26 (+4 new Python cancellation tests covering pre-set cancel, backpressure unblocking, and clean generator exit).

[kunal-vaishnavi]

Can we revert the session API changes to pass in cancel_event since we already have a cancel_event in the session constructor? We want JS and Python to have similar APIs for start, stop, append, etc.

[rui-ren]

Done in eee2cbe. Reverted the per-call signal / cancel_event parameters on both JS and Python so cancellation is configured once on the constructor and the method signatures stay symmetric.

JS start(), append(), stop(), getTranscriptionStream() now take no abort parameter; they read this.sessionSignal directly. Removed the resolveSignal / AbortSignal.any composition machinery.

Python same: start(), append(), stop(), get_transcription_stream() take no cancel_event. _is_cancelled() simplified to check only the session-level event.

Final API now matches across both languages:

const session = audioClient.createLiveTranscriptionSession({ signal: shutdown.signal });
await session.start();
await session.append(pcm);
for await (const r of session.getTranscriptionStream()) { ... }
await session.stop();

session = audio_client.create_live_transcription_session(cancel_event=cancel)
session.start()
session.append(pcm)
for r in session.get_transcription_stream(): ...
session.stop()

Tests: JS 19/19, Python 25/25 (-2 tests for the dropped per-call paths; backpressure-unblock test still validates the session-level cancel works).

[kunal-vaishnavi]

Instead of creating a new class to represent the parameters passed into the session, can we instead make an optional parameter for signal in createLiveTranscriptionSession? Then, we can remove the need for the LiveAudioTranscriptionSessionOptions class. The new optional parameter for signal can behave similar to a CancellationToken parameter.

[rui-ren]

sure, I will update the cancellation pattern. Thanks

[rui-ren]

Done in 6ab38c4. Dropped LiveAudioTranscriptionSessionOptions and made signal a plain optional parameter on createLiveTranscriptionSession, matching C#'s CancellationToken shape:

const shutdown = new AbortController();
const session = audioClient.createLiveTranscriptionSession(shutdown.signal);

(was createLiveTranscriptionSession({ signal: shutdown.signal }))

JS 19/19 tests pass; sample updated.

[kunal-vaishnavi]

What is the timeout for?

[rui-ren]

The 100 ms is a cancellation polling interval. Pushed d5683c3 to extract it as _CANCEL_POLL_INTERVAL with a docstring at the top of the file so this isn't a magic number anymore.

Why we poll at all: queue.Queue.get / put in standard Python can't be interrupted by a threading.Event there's no equivalent of select() over them. So when the user has configured a cancel_event, we fall back to a poll-with-timeout pattern: wait up to 100 ms for queue I/O, then check _is_cancelled() and either return / raise or retry.

Why 100 ms specifically: balances responsiveness vs. overhead.

• Cancellation latency: a SIGINT takes effect within ~100 ms (one poll period).
• Idle CPU overhead: ~10 wakeups/sec per blocked call negligible.

Important: this is a no-op on the fast path. If the user never passes a cancel_event, the original blocking put() / get() is used unchanged with no polling overhead at all (see the if self._cancel_event is None branch in each method).

[kunal-vaishnavi]

Same question as here

[rui-ren]

The 100 ms is a cancellation polling interval. Pushed d5683c3 to extract it as _CANCEL_POLL_INTERVAL with a docstring at the top of the file so this isn't a magic number anymore.

Why we poll at all: queue.Queue.get / put in standard Python can't be interrupted by a threading.Event there's no equivalent of select() over them. So when the user has configured a cancel_event, we fall back to a poll-with-timeout pattern: wait up to 100 ms for queue I/O, then check _is_cancelled() and either return / raise or retry.

Why 100 ms specifically: balances responsiveness vs. overhead.

• Cancellation latency: a SIGINT takes effect within ~100 ms (one poll period).
• Idle CPU overhead: ~10 wakeups/sec per blocked call negligible.

Important: this is a no-op on the fast path. If the user never passes a cancel_event, the original blocking put() / get() is used unchanged with no polling overhead at all (see the if self._cancel_event is None branch in each method).

[kunal-vaishnavi]

Same question as here but for the Python example

[rui-ren]

Done in 143a8b6. Updated samples/python/live-audio-transcription/src/app.py with the same pattern as the JS sample concrete diff so the API value is visible.

Before: the sample created the session with no cancellation token and used a parallel local stop_event for the mic loop, which couldn't reach inside the SDK:

session = audio_client.create_live_transcription_session()
...
stop_event = threading.Event()
def shutdown(*_):
    stop_event.set()  # only the mic loop sees this
    session.stop()    # has to drain everything

After: one event drives the whole shutdown path SIGINT just sets it:

shutdown_event = threading.Event()
session = audio_client.create_live_transcription_session(cancel_event=shutdown_event)
...
def shutdown(*_):
    shutdown_event.set()
    #  this single call:
    #   - aborts any in-flight session.append() blocked on backpressure
    #     (FoundryLocalException, no late delivery to native core)
    #   - ends session.get_transcription_stream() iteration cleanly
    #   - short-circuits session.stop()'s drain wait
    #   - exits the mic capture loop
    session.stop()

I also added CoreErrorResponse.try_parse in the read loop to inspect structured error metadata (code + is_transient) transient blips no longer kill long-running sessions, mirroring the JS CoreError demo.

Tests: 25/25 pass; sample syntax-verified.