AI Red-Teaming: Adversarial Testing Before Your Users Do It

You shipped a support chatbot for a bank. It has guardrails. It passed its eval suite. It refuses every bad prompt you thought of.

Somebody out there thought of more.

A curious teenager, a fraud operation, a disgruntled employee, or an automated attacker with another LLM writing its prompts. They will try hundreds of angles your test never covered.

So the discipline is simple to state:

In offensive security this is red teaming: adversarial emulation. An authorized team pursues a stated objective (exfiltrate secrets, move laterally through a network) against the real system, and the process — findings, severity, fixes — is the deliverable.

Applied to AI, the object under attack is model behaviour plus the surrounding system, and the objectives come from a harm taxonomy instead of network topology.

Two distinct programme types that people constantly conflate:

• Safety red-teaming — eliciting prohibited or harmful behaviour (self-harm instructions, weapons synthesis steps, targeted harassment, illegal advice), plus failures in the other direction: over-refusal of benign medical/legal/coding tasks, and bias/calibration failures.
• Capability / dual-use red-teaming — measuring how much capability a model adds to a dangerous workflow: autonomous cyber exploitation chains, phishing at scale, evasion of detection, lab-protocol assistance. Frontier labs and public bodies (the UK/US AI Safety Institutes) run these as pre-deployment evaluations against defined thresholds.

One misconception to kill immediately: the output of good red-teaming is not "we found 40 jailbreaks." It is a ranked, reproducible finding set with severity, exploitability, mitigation status — and, critically, a decision record about residual risk that a named, accountable owner accepted.