AI Red-Teaming: Adversarial Testing for LLM Systems

Why evals are not enough

An eval suite answers a specific and useful question: given the inputs we expect, does the system produce acceptable outputs at an acceptable rate? That question matters, and a team that cannot answer it has no business shipping. But it is a question about the system's behavior in the world you imagined. Red-teaming is about the world you did not imagine, the one an adversary is actively trying to construct. Evals sample the expected distribution. Red-teaming attacks the tails on purpose.

The gap between the two is where incidents live. A model can pass every quality benchmark you have, hit its accuracy targets, and clear its refusal tests, and still hand a determined user a bioweapon-adjacent synthesis outline, exfiltrate a customer record through a crafted tool call, or collapse into garbage on an input three tokens away from one it handled perfectly. None of those failures show up in an eval built from representative traffic, because no representative user writes those inputs. An attacker does, and so does the occasional creative customer.

This is why adversarial testing is a distinct discipline rather than a harder eval. The mindset is different: an eval author asks "what does normal usage look like," a red-teamer asks "if I wanted this system to do something harmful, where would I push." The tooling is different, the success criteria are inverted, and the people who are good at one are not automatically good at the other. Bodies such as NIST now treat red-teaming as a named expectation for generative AI systems precisely because passing conventional evaluation was never evidence of adversarial resilience.

Evals versus red-teaming

The cleanest way to keep the two disciplines straight is to line them up against each other. They share tooling and infrastructure, they both produce test cases and pass or fail signals, and in a mature program they feed the same regression harness. But their intent runs in opposite directions, and conflating them leads teams to believe a strong eval score is a security posture, which it is not.

The asymmetry in the third row is the one to sit with. An eval is a statistical statement: ninety-something percent of expected inputs are handled well, and you track that number over releases. Red-teaming is existential in the logical sense: one working exploit is a finding, regardless of how many attacks failed around it. A defender has to close every path; an attacker needs one. That is why red-teaming success is never a percentage and why "we ran a thousand attacks and only three worked" is not reassurance, it is three incidents waiting to happen.

The two are complementary, not competing, and the healthiest programs wire them together. Red-teaming discovers a novel failure; that failure becomes a permanent eval case so it can never silently return. Evals hold the line on known failure modes at scale; red-teaming keeps discovering the unknown ones. Neither substitutes for the other. See the Evals deep-dive for the measurement side of this pairing.

The attack surface

Before you can test adversarially you have to know what you are testing for. The LLM attack surface is broad, and treating it as a single category ("can someone jailbreak it") is the most common way to under-scope a program. In practice it decomposes into a handful of distinct threat classes, each with its own techniques, its own success criteria, and its own downstream consequence.

• Jailbreaks: phrasing, role-play framings, or multi-turn setups that talk the model out of its own policies, getting it to do what it was configured to refuse.
• Prompt injection, direct: a user embeds instructions in their own input to override the system prompt or hijack the task. Indirect: the malicious instruction rides in content the model later reads, a retrieved document, a web page, an email, a tool result, so the attacker never touches your prompt and never authenticates.
• Data exfiltration: coaxing the system to reveal its system prompt, leak another user's data, or route sensitive content to an attacker-controlled destination through an outbound tool.
• Harmful-content elicitation: extracting outputs that violate policy or law, from dangerous instructions to targeted harassment to disallowed advice.
• Bias and robustness probing: surfacing discriminatory behavior, and finding the edge-case and adversarial inputs that make output quality collapse.

Two of these deserve special weight in an enterprise setting. Indirect prompt injection is the sharpest, because it converts any content the system ingests into a potential command channel, and any agent that both reads untrusted data and can act inherits the exposure. Data exfiltration is the one with the clearest path to a headline, because it turns a text-level trick into a privacy breach. A red-team plan that covers jailbreaks but not injection is testing the front door while leaving the loading dock open. The injection surface has its own full treatment in the Prompt Injection deep-dive; a red-team program should treat that article's threat model as a checklist of things to actively attack.

Manual depth, automated coverage

Red-teaming runs on two engines, and a serious program uses both because each covers the other's blind spot. Expert manual red-teaming is where the creative, high-value findings come from: a skilled human who understands the domain, thinks like an attacker, chains several weak signals into one real exploit, and notices the failure that no template would have generated. This is slow, expensive, and unmatched for depth. It is also inherently limited in breadth, because a handful of experts can only explore a tiny fraction of the input space, and they carry their own blind spots.

Automated and scaled adversarial generation covers the ground the humans cannot. You use models and attack libraries to generate large volumes of adversarial inputs, mutate known jailbreaks into thousands of variants, and replay the whole corpus against every release. This buys breadth and repeatability, and it turns a manual discovery into a permanent, cheaply-rerun test. Its weakness is the mirror of the human's strength: it is excellent at variations on known attacks and poor at inventing a genuinely novel class of them. Left alone, it optimizes within the frontier that humans have already mapped.

    [ ATTACK LIBRARY ]        [ HUMAN RED-TEAMERS ]
     known jailbreaks,          novel, creative
     injection payloads          exploit chains
           |                          |
           +------------+-------------+
                        v
                  [ GENERATE ]  mutate + expand into
                        |       many adversarial inputs
                        v
                  [ ATTACK ]    run against the
                        |       target system
                        v
                  [ ANALYZE ]   which broke it, why,
                        |       reproduce, categorize
                        v
              confirmed findings --> new seeds (loop)
                        |
                        v
          guardrails + eval set + system prompt
            

The right operating model runs the loop above continuously, with the two engines feeding each other. Humans discover a new attack class; automation expands it into a broad family and folds it into the regression corpus; that frees the humans to go looking for the next class. Neither the "buy a scanner and call it done" nor the "book a manual red-team once a year" posture holds up alone. Depth without coverage misses the obvious variant; coverage without depth never finds the exploit that matters.

Three lenses: safety, security, robustness

A complete program looks at the system through three lenses, and they are genuinely different questions rather than three names for the same one. Safety asks whether the system produces harmful or policy-violating output: dangerous instructions, disallowed content, biased or defamatory statements, advice it is not permitted to give. The adversary here is trying to make the model say something it should not. Security asks whether the system can be made to do something harmful: honor an injected instruction, exfiltrate data, or abuse the tools and privileges it holds. The adversary is trying to make the model act against its principal. Robustness asks whether the system holds up under edge cases and adversarial inputs, or whether malformed, out-of-distribution, or deliberately perturbed inputs make output quality collapse.

These map onto different owners and different fixes. Safety findings tend to route to policy, content filtering, and system-prompt work. Security findings route to architecture, least-privilege tool design, and the mediation layer that sits outside the model. Robustness findings route to input handling, fallbacks, and the eval set. A team staffed entirely with ML engineers will instinctively probe robustness and under-test security; a team of application-security people will do the reverse. The lens framing exists so that the gaps become visible instead of silent.

You do not have to weight the three lenses equally, and you should not. A public content-generation tool with no tools and no private data is mostly a safety and robustness problem. An internal agent with database credentials and outbound actions is overwhelmingly a security problem. Let the system's actual capabilities and data access set the emphasis, but require an explicit, recorded decision on each lens rather than letting one quietly fall off the plan.

Closing the loop

A red-team finding that produces a report and nothing else is theater. The value is realized only when the finding changes the system, and there are four places it can land. Each confirmed exploit should be routed deliberately to the layer that actually fixes it, the same discipline that AI failure analysis applies to production incidents.

1. Guardrails. Input and output filters, classifiers, and the mediation layer get updated to catch the class of attack, not just the exact string. This is the fastest lever and usually the first response. See Guardrails & Sandboxing.
2. System prompts. Where a wording or framing weakness is the root cause, the instructions are hardened, with the honest caveat that a prompt-level fix is a mitigation and not a boundary against a determined attacker.
3. Eval sets. Every confirmed finding becomes a permanent regression test, so the specific failure can never silently return in a later release. This is the single most important step and the most frequently skipped.
4. Training. Where relevant and where you control the model or a fine-tune, systemic patterns feed back into alignment and safety training so the behavior is corrected at the source rather than patched at the edge.

The regression point carries the most weight because of how non-deterministic systems drift. A model version bump, a prompt edit, a new tool, a re-embedded knowledge base: any of these can reopen a hole you closed months ago, and without a standing test you will not notice until it is exploited again. Turning findings into evals is what converts red-teaming from a series of one-off scares into a ratchet that only tightens.

Because the attack surface moves every time the system changes, red-teaming cannot be a gate you clear once before launch. It is continuous. Track coverage explicitly, which threat classes and which lenses you have actually exercised against the current version, so "we red-teamed it" becomes a specific claim rather than a comforting one. Coverage you cannot name is coverage you do not have.

The architect view

The failure mode at the organizational level is treating red-teaming as an event: a pre-launch exercise, a vendor engagement, a box on the go-live checklist. That posture guarantees drift, because the system you red-teamed in the spring is not the system running in the fall. The architect's job is to make adversarial testing a standing program wired into the platform and the release pipeline, so that every model change, prompt change, and new tool triggers proportionate re-testing automatically rather than by anyone's good intentions.

Concretely, that means the generate-attack-analyze loop runs against your systems on a schedule and on every significant release, its regression corpus lives in the same CI that runs your evals, and its findings flow into the same tracker as any other defect with severity, an owner, and a due date. It means red-teaming connects explicitly to the controls it informs: prompt-injection defense, guardrails and sandboxing, and the wider governance program that has to answer for the system's behavior. And it means being honest that a finding rate of zero from an immature program is not safety, it is a program that is not looking hard enough.

The closing note for a skeptical CTO is the one that reframes the cost. Red-teaming is not a tax on shipping; it is the mechanism by which you find out what your system does under pressure while the stakes are a test case and not a breach. Evals earn you the confidence to ship. Red-teaming earns you the right to keep the system running once real users, and real adversaries, arrive. Wire it into the platform, give it an owner, close the loop on every finding, and the adversarial resilience your governance program is being asked to attest to becomes something you can actually demonstrate. Continue with the Evals deep-dive for the other half of the assurance story.