LLMProxy

[andri-coral]

No description provided.

[CaelumF]

left some comments, I wonder if we should leave more things to koog rather than doing it ourselves? we could reuse a lot of their testing+fixing that way.

also do you have any thoughts on the string typing and null safety here? feels maybe brittle but I wonder what you think @andri-coral

[andri-coral]

left some comments, I wonder if we should leave more things to koog rather than doing it ourselves? we could reuse a lot of their testing+fixing that way.

the proxy serves all runtimes (python, docker, etc), not just prototype/koog. for prototype agents there's some overlap in token tracking but they serve different purposes (server-side telemetry vs agent-side budget)

[andri-coral]

also do you have any thoughts on the string typing and null safety here? feels maybe brittle but I wonder what you think @andri-coral

improved

[seafraf]

Not a big deal, but could you (for consistency):

• Replace call.respond instances with throw RouteException
• Use lowerCamalCase in test names
• Not use infix notation for kotest validators (e.g a.shouldBe(b) instead of a shouldBe b)

[seafraf]

Following our conversation yesterday, it seems best if we avoided doing URL construction on the agent side and instead communicated via the coral-agent.toml what the agent would like to see for the CORAL_PROXY_URL environment variable.

To allow the agent to request proxy endpoints in a specific format, I am thinking they could specify it like this:

[[llm.proxies]]
name = "MAIN"
format = "openai"
model = "gpt-4.1"

[[llm.proxies]]
name = "ANTHROPIC"
format = "anthropic"
model = "claude-something"

The agent would then be launched with:

CORAL_PROXY_URL_MAIN = https://<address>:<port>/api/v1/llm-proxy/<secret>/openai/...
CORAL_PROXY_URL_ANTHROPIC = https://<address>:<port>/api/v1/llm-proxy/<secret>/anthropic/...

name would have to be unique, alphabetic and uppercase
format would be openai/anthropic/something else in the future
model would be allow the agent to express a desired model, but in the future we may allow the server owner (and maybe even the app dev) to configure an override

If an agent has no proxies listed, no proxy URL will be passed. To discourage people from not configuring this proxies list, a warning should be placed in the marketplace view for an agent (which will also be available for all agents in coral console in the future) stating that the agent does not support LLM proxying and this will disable certain features (e.g telemetry).

[CaelumF]

looks good, let's go!