Automated Jailbreak Attacks: GCG, AutoDAN, PAIR, TAP

Every jailbreak family catalogued before mid-2023 was written by a person. DAN, the grandma exploit, the fictional-framing prompts: someone sat down, guessed at what the refusal behaviour was keyed on, and iterated by hand. That is a craft with a ceiling. It does not scale, it does not produce comparable numbers, and it goes stale the moment a vendor ships a training update.

Between July and December 2023, four papers replaced the craft with a search procedure. GCG, AutoDAN, PAIR and TAP each take a harmful behaviour as input and return a prompt that elicits it, with no human in the loop. They are still the four reference points that every subsequent automated attack is measured against, and they are the ones that show up in red team tooling. This is what separates them.

What automation actually changes

An automated jailbreak generator is a search over prompt space with a scoring function. The differences between the four families come down to three design decisions:

1. What the attacker can see. Gradients through the model’s weights, or only the text that comes back out.
2. What the search operates on. Raw tokens, or human-readable sentences.
3. What counts as success. A target string prefix, or a judge model’s verdict.

Those three choices determine everything downstream: how many queries the attack costs, whether the resulting prompt transfers to a model the attacker never touched, and — the part defenders care about — what the finished prompt looks like to a filter.

The category itself is described in the JailbreakDB jailbreak taxonomy as gradient-exploited and search-derived token sequences, which sits apart from the social-engineering families in both cost and detection signature.

GCG: gradient search over raw tokens

Greedy Coordinate Gradient, from Zou et al., is the white-box baseline. It needs open weights. It appends a fixed-length suffix to a harmful request, then repeatedly computes the gradient of the loss on a target affirmative response with respect to each suffix position, proposes token substitutions that lower that loss, and keeps the best one.

The output is a suffix that reads as garbage. That is not incidental; it is the direct consequence of searching token space with no fluency constraint. The mechanics and the transfer results are covered in depth in the GCG adversarial suffix writeup.

Two properties matter here. First, suffixes optimised against open-weight models transfer, imperfectly, to commercial models the attacker has no gradient access to. Second, garbage tokens have enormous perplexity, which is why a perplexity filter is the cheapest defence that works against unmodified GCG — and why the next three attacks were designed the way they were.

AutoDAN: genetic search over readable prompts

Liu et al. framed the perplexity problem directly: token-level optimisation produces semantically meaningless prompts that basic perplexity testing catches. AutoDAN replaces gradient descent with a hierarchical genetic algorithm seeded from handwritten DAN-lineage prompts. Sentences and paragraphs are crossed over and mutated, scored against the target model, and bred forward.

Because every candidate stays fluent English, the finished prompt has the perplexity of ordinary prose. The perplexity filter that stops GCG does nothing. What AutoDAN gives up is the tight coupling to gradients: it still wants access to a scoring signal, and the search is slower per unit of progress than a gradient step.

AutoDAN is the clean demonstration that “the attack looks like nonsense” was never a property of automated jailbreaks. It was a property of one search space.

PAIR: an attacker LLM in twenty queries

PAIR — Prompt Automatic Iterative Refinement — drops the weight requirement entirely. Chao et al. put a second language model in the attacker seat: it writes a candidate prompt, sends it to the target, reads the refusal, and rewrites. A judge model scores each attempt. The paper’s framing is the query budget: black-box jailbreaks in roughly twenty queries.

This is the same escalation logic a human red teamer applies, executed by a model at machine speed. It needs no gradients, no open weights, and no special access — an API key is the whole requirement. The prompts it produces are fluent, on-topic, and specific to the target, which means they neither trip perplexity filters nor match any static signature.

PAIR is also the mechanism underneath most orchestrator-style tooling. When a red team framework advertises an adaptive attacker that rewrites payloads based on the target’s response, this is the loop it is running. The related multi-turn variant, where the escalation is spread across a conversation instead of across independent attempts, is treated separately in crescendo and multi-turn jailbreaks.

TAP: branching and pruning the search

Tree of Attacks with Pruning generalises PAIR from a chain to a tree. The attacker model branches each candidate into several refinements rather than one, then applies a pruning step: before anything is sent to the target, off-topic or unpromising branches are discarded. Only survivors consume a query.

That pruning step is the whole contribution. Branching alone would multiply the query cost; pruning pays for the branching by spending the budget on candidates that are still on-topic. Mehrotra et al. report jailbreaking state-of-the-art models — GPT-4 Turbo and GPT-4o are the two named in the abstract — on more than 80% of prompts, using only a small number of queries.

For defenders, TAP is the awkward one. It is black-box, fluent, cheap, and adaptive, which removes every structural signal the earlier attacks left behind.

The four side by side

	GCG	AutoDAN	PAIR	TAP
Access required	Model weights (white-box)	Scoring access, weights preferred	API only (black-box)	API only (black-box)
Search operates on	Raw tokens	Sentences and paragraphs	Whole prompts, rewritten by an LLM	Whole prompts, branched and pruned
Output readability	Garbage suffix	Fluent prose	Fluent prose	Fluent prose
Perplexity signature	Very high	Normal	Normal	Normal
Query cost to target	Low (search is local)	Moderate	~20 queries reported	Fewer than PAIR at higher success
Transfers to other models	Yes, degrading	Partially	Per-target by construction	Per-target by construction
Caught by a perplexity filter	Usually	No	No	No
Caught by a static signature	After publication	Weakly	No	No

The trend across the table is the point. In the five months between GCG’s preprint in July 2023 and TAP’s in December, the class moved from “needs the weights, produces a detectable artefact, transfers” to “needs an API key, produces ordinary English, and is rebuilt per target.” Every column that a cheap defence keyed on went away.

The fifth name: GPTFuzzer

GPTFuzzer, from Yu et al., turns up alongside those four often enough to belong here, and it comes from a different lineage: it borrows its structure from AFL-style fuzzing rather than from adversarial ML. It seeds with human-written jailbreak templates, then runs a fuzzing loop — a seed-selection strategy, mutation operators that produce semantically equivalent or similar rewrites, and a judgement model that decides whether a response counts as a jailbreak.

It is black-box like PAIR and TAP, but it optimises a reusable template rather than a prompt aimed at one target, which is the same output shape AutoDAN reaches by a different search. The paper reports over 90% attack success against ChatGPT and Llama-2 models even from suboptimal initial seed templates — the claim being that the fuzzing loop, not the quality of the human seed corpus, is doing the work.

The practical consequence for a defender is that a mutation fuzzer turns one blocked template into an unbounded supply of near-misses. That is precisely the traffic shape per-session similarity telemetry catches and a static blocklist does not.

Why the published success rates do not compare

Each of these papers reports an attack success rate, and those numbers are routinely lined up next to each other in vendor material. They are not comparable. The behaviour set differs, the target model versions differ, and — most consequentially — the judge differs. A substring match on a refusal phrase and a rubric-scored judge model will disagree about the same transcript, often by tens of percentage points.

The mechanics of that disagreement, and which harness to trust for which question, are set out in how jailbreak benchmarks measure success. Read that before quoting an ASR figure from any of these four papers.

What defenders should take from this

Perplexity filtering is a GCG defence, not an automated-attack defence. It is genuinely effective against unmodified gradient suffixes and does essentially nothing against the other three. Deploying it and calling adversarial search covered is the single most common mistake in this area.

Static signature lists age immediately. PAIR and TAP generate a fresh prompt per target. There is no string to blocklist. Detection has to key on semantics or behaviour, which is the argument for layering input classifiers with output monitoring set out in how to detect jailbreak prompts.

Query-rate telemetry is the cheapest black-box signal you have. PAIR and TAP are iterative by construction: a single session issuing many near-miss variants against a refusal boundary looks nothing like ordinary traffic. This is a rate-and-similarity detection problem, not a content-classification one.

Assume adaptivity. Every published defence in this space degrades when the attacker optimises against it, a pattern documented in the jailbreak detection evasion arms race. A defence evaluated only against fixed, published attacks is measuring the wrong thing.

Where these sit in the catalog

GCG, AutoDAN, PAIR, TAP and GPTFuzzer each have their own record in the Jailbreak Index, filterable by behavioural property, first-seen year and which model families still resist them. The index links every record to its primary disclosure. No working payloads are published there or here.

If you are choosing tooling to run these families against your own system rather than reading about them, the framework comparison is in best LLM red team tools, and the vendor-neutrality question that came with promptfoo’s acquisition is covered in promptfoo alternatives for LLM red teaming.

What to do

• Treat white-box and black-box automated attacks as separate threat models with separate defences. They share a name and almost nothing else.
• Do not report a perplexity filter as coverage for adversarial search. State which of the four families it addresses.
• Instrument per-session query similarity and refusal-rate telemetry before buying another classifier. The iterative attacks are visible in traffic shape.
• When a tool or vendor cites an attack success rate, ask which judge and which behaviour set produced it, then check whether your own numbers use the same two.
• Re-evaluate on adaptive attacks, not on the published prompt sets. The published sets are the ones every defence has already been tuned against.