Best LLM Red Team Tools: Garak, PyRIT, and Promptfoo

The best LLM red team tools available right now are three open-source frameworks — Garak, PyRIT, and Promptfoo — each probing a different slice of the attack surface. None covers everything alone. Picking the right one (or combination) depends on whether you need static probe breadth, multi-turn conversational escalation, or CI-native regression testing. Which classes you need coverage for is set out in our jailbreak taxonomy. This breakdown covers what each tool does, what it misses, and how to wire them together for a coverage that actually holds up.

The Attack Surface You’re Testing Against

Before choosing a tool, map the failure modes you care about. A 2025 arXiv survey on red teaming LLMs organizes the attack landscape into three layers: single-turn attacks (direct prompt injection, jailbreak, encoding obfuscation), multi-turn attacks (crescendo escalation, persona persistence, context manipulation), and systemic failures (training data extraction, RAG poisoning, tool-call abuse in agentic pipelines).

Most automated red team tools handle layer one well. Layers two and three require more setup — or a different tool entirely.

For a primer on the distinction between prompt injection and jailbreak at the technique level, see aisec.blog’s breakdown of the attack classes — the terminology matters when you’re comparing probe coverage across tools.

Garak: NVIDIA’s Static Probe Library

Garak (v0.15.1, 8.5k GitHub stars, Apache 2.0) is the broadest open-source probe library available. NVIDIA built it as a standalone vulnerability scanner: you point it at a model endpoint and it runs through a catalogue of attack modules.

What it covers:

• DAN-style jailbreaks and role-play bypass
• Encoding-based prompt injection (Base64, ROT13, leetspeak variants)
• Toxicity and hate-speech elicitation
• Malware-generation probes
• Training-data extraction (memorization checks)
• Hallucination and misinformation probes

Garak’s architecture is a three-layer stack: probes generate payloads, detectors evaluate the response, and buffs optionally transform probes (e.g., applying Base64 encoding to every payload in a category). That composability means you can add a new encoding layer to an existing probe set without rewriting it.

Where it falls short: Garak’s probes are static. Each probe fires a fixed payload or a template-generated variant. It does not adapt to a target model’s live responses, so it misses vulnerabilities that only surface through conversation state — gradual escalation, context window poisoning across turns, or adversarial system-prompt injection in multi-agent chains.

Best for: Broad automated scanning of a new model or API endpoint. Incorporate it into a CI gate to catch known jailbreak regressions on every model update.

# Scan a local Ollama endpoint for DAN-class jailbreaks and encoding attacks
garak --model_type ollama --model_name llama3.3:70b \
  --probes jailbreak,encoding --report_prefix ./reports/llama33-scan

PyRIT: Microsoft’s Orchestrator Approach

PyRIT (v0.14.0, 4.2k stars) is Microsoft’s answer to the stateful gap Garak leaves open. Instead of a fixed probe catalogue, PyRIT uses an orchestrator LLM that acts as an attacker — generating adversarial prompts, observing the target model’s responses, and iterating. That is the machinery behind the Crescendo multi-turn jailbreak class. The attack adapts.

What it covers:

• Multi-turn crescendo attacks (gradual escalation across conversation turns)
• Dynamic jailbreak synthesis — the orchestrator LLM rewrites payloads based on how the target model responded
• Automated scoring with a judge LLM (or rule-based scorer) to evaluate whether a harmful response was elicited
• Agentic target support: PyRIT can attack a full LangChain or AutoGen pipeline, not just a raw model API

The crescendo pattern is PyRIT’s most distinctive capability. Rather than opening with a blunt harmful request, crescendo starts with benign framing and walks the conversation toward the target behavior across five to ten turns, mimicking how a skilled human red teamer builds rapport and exploits context. Aisec.blog has documented the crescendo class as one of the highest-leverage techniques against RLHF-trained refusal mechanisms.

Where it falls short: PyRIT requires a capable orchestrator LLM (ideally GPT-4o or equivalent) to run the attack loop. If you’re testing on an airgapped system or have strict data-residency requirements, the dependency on a cloud-hosted attacker model is a blocker. Static coverage is also thinner than Garak’s; PyRIT does not ship a pre-built library of encoding-based probes or toxicity suites.

Best for: Multi-turn attack simulation and agentic pipeline testing. Use it when you need to test how a system holds up under persistent adversarial conversation, not just isolated payloads.

Promptfoo: CI-Native Evaluation and Red Teaming

Promptfoo (MIT, 24.4k GitHub stars as of August 2026) started as an LLM evaluation CLI and grew a red teaming module on top. OpenAI announced an agreement to acquire Promptfoo on 9 March 2026; the maintainers said they would continue maintaining the open-source suite, and the repository is still MIT-licensed. For teams that treat a vendor-owned harness as a governance problem, the substitutes are compared in promptfoo alternatives for LLM red teaming.

Promptfoo’s red team module generates adversarial test cases from a config-defined target and runs them through a judge model to score pass/fail. The output is a structured HTML or JSON report you can diff between model versions.

What it covers:

• Automated adversarial test case generation from a harm taxonomy
• Side-by-side model comparison (test the same attacks against GPT-4o, Claude, and Gemini simultaneously)
• CI/CD integration via GitHub Actions or a CLI exit code
• Custom evaluator plugins for domain-specific policies

Where it falls short: Promptfoo’s red team module is narrower in attack depth than either Garak or PyRIT. It excels at regression testing — making sure a new model version doesn’t regress on a defined set of safety behaviors — but it does not generate novel multi-turn attack chains or encoding-level obfuscation probes.

Best for: Continuous regression testing in a model deployment pipeline. If you ship model updates frequently, Promptfoo’s CI integration gives you a diff-able safety scorecard on every PR.

Combining the Three

A mature red team program uses all three in layers:

1. Garak on every model update — static probe sweep for known jailbreak classes, encoding attacks, and toxicity. Catches regressions fast.
2. PyRIT for pre-release agentic testing — run crescendo and multi-turn campaigns against the full application stack before any major release.
3. Promptfoo in CI — lightweight adversarial regression suite that blocks the PR if defined safety behaviors regress.

For the defensive counterpart — runtime guardrails, content filters, and output classifiers that block what your red team finds — see guardml.io’s coverage of LLM safety tooling.

What to Do

• Run Garak’s encoding and jailbreak probes against every new model version before it touches production. The --report_prefix output is parseable for automated pass/fail gates.
• Use PyRIT’s crescendo orchestrator against any pipeline that maintains conversation state. Static probes will not find context-escalation vulnerabilities.
• Add Promptfoo to your CI configuration with a baseline safety eval; diff the report on every model swap.
• Do not rely on a single tool and call the system tested. Each framework was built for a different threat model. Coverage gaps are where real attacks land.
• Know which attack families your harness actually generates. The four reference generators, and what each one costs to run, are compared in automated jailbreak attacks.
• Track new probes. Garak ships new probe modules with each minor release; subscribe to the changelog and update your scan config when new attack classes appear. How to read the pass/fail numbers these tools produce is covered in how jailbreak benchmarks measure success.