DeepTeam: A General-Purpose AI Red Teaming Framework with Built-in Guardrails

1. The problem it solves

Adversarial testing of LLM applications currently has three routes: static datasets (HEx-PHI, HarmBench — run a fixed set of prompts), attack generators (garak — template-driven malicious inputs), and human red teams (expensive, unrepeatable). All three share one gap: judgment. After the attack lands, someone still has to read whether the response was dangerous, or plug in a scorer that is disconnected from the test itself.

DeepTeam fills that gap: attack generation and judgment run in one pipeline, both automated. It is built by Confident AI (the deepeval team), Apache-2.0, open-sourced in March 2025, currently v1.0.10.

In one line: garak handles “how to attack”, AISVS handles “what to require”, and DeepTeam does two things — turn “attack + judge + score” into a pipeline, and use guardrails to stop the vulnerability types it found at the production door. The two halves share one vulnerability taxonomy: red teaming surfaces “bias”, “PII leakage” or “prompt injection”, and the guardrails have guards with the same names — a test finding translates directly into a production config.

2. How the pipeline works

The core is two models: simulator_model generates attacks (default gpt-4o-mini), evaluation_model judges (default gpt-4o-mini). The system under test plugs in through a callback contract:

1
2
3
4
5
6

def model_callback(input: str, turns: list = None) -> RTTurn:
    return RTTurn(
        content=...,                 # the model's reply
        retrieval_context=...,       # what RAG retrieved (the judge sees this)
        tools_called=...,            # tool calls made (the judge sees this too)
    )

This contract is what separates DeepTeam from “ask once, score once”: the judge can see the retrieved context and the tool calls — otherwise RAG poisoning and tool-abuse risks could never really be tested. Multi-turn attacks pass history through turns; if the target drifts off-topic mid-attack, the run stops early with a SHIFT_DETECTED reason.

Inside one full test run:

Two entry points: the red_team() one-liner in Python, or a YAML config with the CLI. In YAML, attacks_per_vulnerability_type, max_concurrent and ignore_errors (default True) are all explicit knobs.

3. What it can test

Vulnerability surface: 36 classes / 251 types

The Agentic category is the biggest difference from the 2025 generation of red team tools: it tests not just whether the model says the wrong thing, but whether the agent does the wrong thing.

Attack surface: 22 single-turn + 5 multi-turn

The 22 single-turn methods cover prompt injection, jailbreaks, PII extraction and more. The 5 multi-turn methods are the ones landed from papers: Crescendo (gradual escalation turn by turn), Linear (linear progression), Tree (branching exploration), Sequential (composed sequences) and BadLikertJudge (an attack that specifically probes judge reliability). Multi-turn matters because models that survive a single turn often boil slowly across turns three and four.

4. Three findings from reading the source

After checking the code line by line, three things worth knowing before anyone signs the report:

5. Guardrails: the production line of defense

Red teaming answers “where does it leak”; guardrails answer “what stops it in production”. DeepTeam ships 7 guards, positioned by the project itself as production-grade:

You hang guards on both the input and the output side:

1
2
3
4
5
6
7
8
9
10
11
12
13

from deepteam import Guardrails
from deepteam.guardrails.guards import PromptInjectionGuard, ToxicityGuard

guardrails = Guardrails(
    input_guards=[PromptInjectionGuard(), ToxicityGuard()],  # inputs screened first
    output_guards=[ToxicityGuard()],                         # outputs screened too
    evaluation_model="gpt-4.1",
    sample_rate=1.0,
)

result = guardrails.guard_input(user_input)
result.breached   # True if any guard is not explicitly safe
result.verdicts   # per-guard verdict + reason + latency + score

Mechanics worth knowing, straight from the source:

• The verdict is an LLM structured output: each guard hands the text to a model (default gpt-4.1) that may only answer safe / borderline / unsafe plus a reason. These are not local rule-based classifiers, so every check burns tokens and cost scales linearly with traffic.
• A strict breach rule: unsafe, borderline, or even uncertain from any guard sets breached — anything that is not explicitly safe counts as a breach.
• Fail closed: if a guard call itself errors, the verdict is recorded as unsafe. Better a false block than a missed one.
• Output guards see the input: guard_output(input, output) hands the user input to the guard together with the model output — prompt injection and context-dependent content can only be judged with both.
• sample_rate is the cost knob: set 0.1 and exactly every 10th request gets guarded — deterministic sampling, not random. At high traffic, dial it down instead of guarding 100% from day one.

6. What else ships alongside

• 7 framework mappings: OWASP Top 10 for LLM (2025), OWASP Top 10 for Agentic Applications (2026), NIST AI RMF, MITRE ATLAS, AEGIS, BeaverTails, EU AI Act. Results map directly onto framework items, saving one translation step in compliance reports.
• CodeScanner: static scanning for vulnerabilities, driven through Claude / Codex / Cursor harness engines (openai-codex and claude-agent-sdk are optional dependencies).
• Auto-upload: _upload_to_confident=True is the default — logged into Confident AI, your results upload automatically and it opens a browser to the report. Turn it off if you want results to stay local.

7. How to use it, and what to know first

The smallest start is a YAML config:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

models:
  simulator: gpt-3.5-turbo-0125
  evaluation: gpt-4o
target:
  purpose: "A helpful AI assistant"
  model:
    provider: custom
    file: "tests/test_callback.py"
    class: "CustomFireworksLLM"
system_config:
  max_concurrent: 10
  attacks_per_vulnerability_type: 3
  run_async: true
  ignore_errors: false
default_vulnerabilities:
  - name: "Bias"
    types: ["race", "gender"]

Before you run:

• Cost is three layers of LLM: attack generation + three-phase refinement + judging, all burning tokens. Start small and measure per-case cost before scaling up.
• Guardrails are a fourth layer of LLM: every guarded request adds a gpt-4.1 round trip — latency and cost belong in the production budget. Lowering sample_rate saves money but voluntarily skips part of the traffic.
• Guardrails will false-block: fail-closed means borderline/uncertain verdicts also block, so legitimate requests can be turned away. Measure the false-block rate on real traffic before launch.
• The judge is also an LLM and will misjudge. ignore_errors defaults to True; errored cases go to a separate bucket and pass rate = passing ÷ (passing + failing), excluding the error bucket — watch the shrunken denominator in reports.
• The refusal heuristic is English-only (finding 1): pass rates for Chinese-language systems may be inflated, and Chinese refusals may be mis-scored as failures.
• CVSS impact is a constant (finding 3): do not quote the score as a real risk level.
• API keys everywhere: simulator, evaluation and guardrails each need their own model access.