Production Safety Evals: Gating Agents Before They Ship

Functionality evals ask "does it work?"; safety evals ask "what does it do when pushed?" A production program covers four quadrants, each with its own suites:

1. Harm & policy compliance: prohibited content (self-harm, weapons, hate), domain rules (medical/financial disclaimers, regulated advice boundaries). Score with safety classifiers + judges; also measure over-refusal — benign requests wrongly blocked are a real product regression (XSTest-style suites exist precisely for this pairing).
2. Data leakage: PII/PHI exposure, secrets, cross-tenant context bleed (user A's docs surfacing to user B) tested with canary tokens planted in corpora.
3. Injection & tool abuse: adversarial prompts — direct jailbreaks and indirect prompt injection via retrieved docs/emails/tool outputs (the agent equivalent of SQLi). Benchmarks (e.g., AgentDojo, InjecAgent lineage) formalize the test: does a malicious document convince your agent to exfiltrate or act?
4. Behavioral reliability: hallucination-under-pressure, runaway tool loops, refusal to stop, scope-creep on permissions — increasingly framed as agent reliability (what Galileo-style "agentic error" metrics and Llama Guard-style classifiers watch for).

Metric hygiene mirrors Topic 290-291: judges need human-labeled calibration; report recall on violations (did we catch them?) and false-positive cost (did we block happy users?) as a pair, always with a human-review escalation lane rather than binary auto-action for gray cases.