feat: prompt prefix caching with TTL eviction and TurboQuant support#995

Open

eloe wants to merge 7 commits intoBlaizzy:mainfrom

eloe:upstream/prompt-cache

Commits on Apr 9, 2026

feat: add prompt prefix caching to server endpoints

eloe
and
claude
committed
fix: handle TurboQuant KV cache in prompt cache trimming

eloe
and
claude
committed
feat: TurboQuant-compatible prefix reuse + cache_key routing + cached_tokens

eloe
and
claude
committed
fix: prevent cache corruption from short probes and partial prefix matches

eloe
and
claude
committed
fix: handle stale KV cache gracefully + sanitize error messages

eloe
and
claude
committed
feat: add prompt cache TTL with automatic eviction

eloe
and
claude
committed
test: add real-world scenario tests for prompt cache TTL

eloe
and
claude
committed