Guardrails and Safety

[jackgerrits]

Tracking issue for work related to agent guardrails and safety.

[darfaz]

Glad to see this being tracked. For teams building with AutoGen, one practical approach to guardrails is adding a scanning layer that inspects messages between agents before they're processed.

ClawMoat is an open-source prompt injection detection library that could be integrated as an AutoGen middleware/hook. In a multi-agent setup, it can scan:

• Inbound messages to each agent for prompt injection attempts
• Tool call arguments for data exfiltration patterns (e.g., URLs with encoded sensitive data)
• Agent outputs for PII or sensitive data leakage

In AutoGen's architecture, this could plug in as a message transform or a custom agent wrapper that scans messages before passing them to the underlying model.

Would be happy to contribute an example integration if there's interest.

[darfaz]

Great thread. Multi-agent systems like AutoGen have a unique security challenge: an injection in one agent's context can propagate through the entire conversation, effectively compromising every agent in the group chat.

A few things that have worked well for us in practice:

1. Scan at agent boundaries — every time a message passes from one agent to another, run it through a security scanner. This catches indirect injection that enters via tool results or retrieved content.

2. Tool call auditing — log and validate every tool invocation. An agent executing subprocess.run("curl ...") with exfiltrated data is the endgame of most prompt injection attacks.

3. Session audit trails — being able to replay and analyze what happened in a multi-agent conversation is critical for post-incident analysis.

We built ClawMoat specifically for this — it's a zero-dependency security layer that can wrap agent inputs/outputs and detect prompt injection, data exfiltration, and tool misuse at runtime.

For AutoGen specifically, hooking into the message pipeline before generate_reply would be the natural integration point. Happy to explore what a proper middleware API for this would look like if there's interest.

Relevant reading: Simon Willison's prompt injection "lethal trifecta" — access to tools + untrusted input + no security boundary = guaranteed exploitation.

[chudharyowesome-eng]

Agent Approval Gateway looks like a strong fit here as a pre-action approval layer.

Add an approval checkpoint before agents send, escalate, purchase, or mutate external state.

Instead of trusting a raw model output at the moment of send/publish/purchase/escalation/state change, the framework can call /approve once and branch on allow|deny|review plus confidence, reason, sources, and risk_flags.

Mainnet: https://claim-verification-agent-mvp.muhammandahmad158609.workers.dev
Agent card: https://claim-verification-agent-mvp.muhammandahmad158609.workers.dev/.well-known/agent.json