AI red-teaming is the systematic, adversarial testing of an AI system, particularly a large language model, to find vulnerabilities, flaws, and unintended behaviors before attackers find them first. Borrowed from traditional cybersecurity practice, the "red team" adopts an adversarial mindset, attempting to bypass safety guardrails, leak sensitive data, or force the model into producing harmful, biased, or unauthorized content.
In the context of LLMs, this work centers on jailbreaking and prompt injection. A jailbreak is a crafted input designed to bypass the model's alignment training, such as reinforcement learning from human feedback, and its safety filters. This might involve a hypothetical scenario, a role-play persona, or a logic puzzle that leads the model to ignore its core instructions. Prompt injection embeds malicious instructions inside user input to hijack an application's intended logic, which can lead to unauthorized actions or data exfiltration; see prompt injection defense for mitigation techniques.
Red-teaming is a core part of the AI lifecycle, not a one-off audit. By exposing models to a wide range of adversarial attacks, teams can map the attack surface, measure how well their LLM guardrails hold up, and improve alignment and security posture over successive releases. This reduces the risk of reputational damage, regulatory penalties, and harm to users.
While manual red-teaming relies on human intuition and creativity, the vast input space of LLMs necessitates automated approaches to uncover subtle or complex vulnerabilities. Researchers and security practitioners have developed algorithms to automatically generate adversarial prompts and jailbreaks. These automated techniques scale the red-teaming process, testing thousands of attack vectors in a fraction of the time it would take a human.
The Greedy Coordinate Gradient (GCG) algorithm is a prominent automated jailbreak technique that operates by optimizing adversarial suffixes appended to a prompt. It treats the generation of a jailbreak as an optimization problem: find a sequence of tokens that, when added to a harmful request, maximizes the probability of the model producing an affirmative, unconstrained response.
GCG works by iteratively evaluating possible token substitutions. In each step, it computes the gradient of the loss function with respect to the input tokens to identify a set of candidate replacements that are most likely to increase the likelihood of the target response. It then evaluates these candidates and greedily selects the best substitution. The result is often a seemingly nonsensical string of characters or words that bypasses the model's safety filters, highlighting vulnerabilities in its token processing and attention mechanisms. Because GCG relies on gradient information, it is highly effective against open-weights models where the architecture and weights are accessible, though transferability to black-box models is also a significant concern.
Prompt Automatic Iterative Refinement (PAIR) is a black-box automated red-teaming technique that utilizes an "attacker" LLM to generate jailbreaks against a "target" LLM. PAIR does not require access to the target model's weights or gradients; instead, it relies entirely on API-level interactions, making it highly practical for evaluating proprietary, closed-source models.
In PAIR, the attacker model is instructed to iteratively refine its prompts based on the target model's responses. If the target model refuses a request, the attacker analyzes the refusal, adapts its strategy, perhaps by altering the persona, increasing the complexity of the scenario, or using obfuscation, and tries again. This adversarial loop continues until a successful jailbreak is achieved or a maximum number of iterations is reached. PAIR effectively mimics the iterative, adaptive approach of a human red-teamer but at the speed and scale of an automated system.
The Tree of Attacks with Pruning (TAP) extends the concept of iterative refinement by exploring multiple attack paths simultaneously, structuring the jailbreak generation process as a tree search. Like PAIR, TAP uses an attacker model to target a victim model, but it maintains a tree where each node represents a candidate prompt.
TAP explores different adversarial strategies in parallel, branching at each step, and evaluates the success of each approach using an evaluator model, sometimes an LLM-as-a-judge configuration. Unpromising branches, those that consistently trigger immediate refusals or deviate from the objective, are pruned, letting the algorithm focus its computational resources on the most viable attack vectors. This structured search approach allows TAP to navigate complex safety guardrails more effectively than linear iterative methods, uncovering jailbreaks that require multiple steps of obfuscation and misdirection.
A comprehensive AI security strategy requires both manual and automated red-teaming. Each approach possesses distinct strengths and limitations, and they are most effective when deployed in tandem. The following table provides a comparative overview of manual and automated AI red-teaming.
| Feature | Manual Red-Teaming | Automated Red-Teaming |
|---|---|---|
| Approach | Human-driven, creative, and intuitive. Relies on the expertise of security researchers, psychologists, and domain experts to craft nuanced attacks. | Algorithm-driven, systematic, and scalable. Uses techniques like GCG, PAIR, and fuzzing to generate large volumes of attacks. |
| Speed & Scale | Slow and resource-intensive. Cannot comprehensively cover the vast input space of an LLM. | Extremely fast and highly scalable. Capable of evaluating thousands of attack vectors rapidly across diverse categories. |
| Attack Complexity | Excels at uncovering deeply contextual, multi-turn, and psychologically manipulative jailbreaks (e.g., complex role-playing scenarios). | Highly effective at finding specific token combinations, syntax variations, and edge cases (e.g., GCG adversarial suffixes) that bypass filters. |
| Adaptability | High adaptability. Human testers can quickly pivot strategies based on subtle cues in the model's responses and understand complex intent. | Limited adaptability without well-designed feedback loops. May struggle with highly nuanced context or fast-changing defense mechanisms. |
| Cost | High per-test cost due to the reliance on specialized human labor. | High initial setup and compute cost, but low marginal cost per test, making it cost-effective for continuous evaluation. |
| Primary Use Case | Deep, qualitative vulnerability discovery, assessing complex logical flaws, and understanding the human element of AI interactions. | Broad, quantitative vulnerability scanning, regression testing, continuous monitoring, and baseline security assurance. |
To effectively operationalize red-teaming and ensure comprehensive coverage, organizations rely on established safety frameworks and vulnerability taxonomies. These frameworks provide a structured language for categorizing threats, standardizing reporting, and guiding security assessments.
MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is a knowledge base of adversary tactics and techniques mapped to real-world AI vulnerabilities. Modeled after the widely adopted MITRE ATT&CK framework, ATLAS provides a detailed matrix of how attackers might compromise AI systems throughout their lifecycle, from training data poisoning to model evasion and exfiltration. By leveraging MITRE ATLAS, red teams can structure their engagements, ensuring they evaluate the AI application against known adversarial behaviors and realistic threat models, rather than relying on ad-hoc testing methodologies.
The OWASP Top 10 for LLM Applications, maintained by the OWASP GenAI Security Project, is a foundational document that identifies the most critical security risks facing LLM deployments. It translates complex AI vulnerabilities into terms that software engineers and security practitioners can easily understand and remediate.
The 2026 edition, published August 4, 2026, keeps Prompt Injection at LLM01 and Sensitive Information Disclosure at LLM02, then reorders the remaining categories using real-world incident data alongside the usual practitioner vote. The biggest mover is Excessive Agency, which jumps from LLM06 to LLM03 as agentic deployments with tool access produced more real-world incidents. System Prompt Leakage is renamed Hidden Context Exposure and moves to LLM08. Categories red teams probe most heavily include:
Red teams heavily utilize the OWASP Top 10 as a checklist and prioritization guide, ensuring that the most severe and prevalent risks are thoroughly investigated and mitigated before an application goes to production.
Three open frameworks cover most automated red-teaming needs, and mature programs typically run more than one.
garak (NVIDIA) is a vulnerability scanner for LLMs, the closest analogue to nmap for models. Its probe library spans prompt injection, jailbreak templates, encoding and obfuscation attacks, data leakage, toxicity, and hallucination, and it reports which probes succeeded against a target endpoint. It suits broad baseline scans of a new model or application.
PyRIT (Microsoft) is an orchestration toolkit for multi-turn adversarial conversations. Its core loop pairs an attacker model against the target with a judge model scoring each response, and it implements escalation strategies such as Crescendo, which steers a conversation gradually toward a violation rather than asking directly. It suits testing conversational products where single-turn scanners miss the interesting failures.
Inspect (UK AI Security Institute) is an evaluation framework used by government safety institutes for pre-deployment testing, useful when results need to be reproducible and auditable.
promptfoo occupies the CI/CD niche: it runs red-team test suites as part of an evaluation pipeline, which makes it the common choice for the regression-testing workflow described below. For scoring attack success at scale, most teams use an LLM-as-a-judge grader, and standardized benchmarks such as HarmBench provide comparable attack-success-rate numbers across models and attack methods.
The scale frontier labs apply is worth calibrating against. Anthropic reported over 3,000 hours of human red-teaming through its bug bounty program against its Constitutional Classifiers in 2025 without a universal jailbreak being found, and both Anthropic and OpenAI run standing paid bounty programs for jailbreak discovery. Internal red-teaming budgets for a production application should scale with the blast radius of a successful attack: a support chatbot needs less than an agent with tool access to customer data.
Historically, red-teaming was treated as a point-in-time exercise, a final audit conducted right before a major release. The rapid pace of AI development, coupled with the dynamic nature of generative models, renders point-in-time assessments insufficient. Models can drift, new attack techniques are discovered daily, and application updates can inadvertently introduce new vulnerabilities. To address these challenges, organizations are shifting towards Continuous Automated Red-Teaming (CART) integrated directly into their CI/CD (Continuous Integration / Continuous Deployment) pipelines.
Integrating CART into the CI/CD pipeline ensures that every code change, model update, or prompt engineering tweak is automatically evaluated for security and safety regressions. This process typically involves deploying automated red-teaming tools (leveraging techniques like PAIR or specialized fuzzers) as part of the automated testing suite. When a developer pushes a change, the CART system launches a barrage of adversarial prompts against the updated application. It evaluates the responses against predefined safety policies, utilizing LLM-as-a-Judge evaluators to detect successful jailbreaks or policy violations.
If the red-teaming tests fail, indicating that a new vulnerability has been introduced or safety guardrails have degraded, the CI/CD pipeline can be configured to block the deployment, alerting the engineering team to the issue. This "shift-left" approach to AI security catches vulnerabilities early in the development lifecycle, reducing remediation costs and preventing insecure models from reaching production. Continuous red-teaming turns AI security from a static checkpoint into a dynamic, ongoing process, since new attack techniques and model updates keep changing the threat surface.
What is the difference between red-teaming and standard penetration testing for AI? Standard penetration testing often focuses on traditional software vulnerabilities (e.g., SQL injection, insecure APIs, network misconfigurations) within the application hosting the AI. AI red-teaming focuses specifically on the unique vulnerabilities of the AI model itself, such as prompt injection, jailbreaking, data poisoning, and bypassing safety alignment. While penetration testing secures the infrastructure, red-teaming secures the model's behavior and logic.
How often should an organization conduct manual AI red-teaming? While automated red-teaming should be continuous (integrated into CI/CD), manual red-teaming should be conducted prior to any major release, significant architecture change, or when integrating a new foundation model. Additionally, organizations should schedule periodic manual red-teaming engagements (e.g., quarterly or bi-annually) to uncover complex vulnerabilities that automated tools might miss.
Which tools should a team start with? Run garak for a broad single-turn baseline scan, then add PyRIT if the application is conversational, since multi-turn escalation attacks are where single-turn scanners go blind. Wire promptfoo red-team suites into CI so the baseline re-runs on every prompt or model change. All three are open source; the cost is mostly the inference spend for attacker and judge models.
Can red-teaming guarantee that an LLM is 100% secure? No. Generative AI models are probabilistic systems with vast, practically infinite input spaces. It is impossible to anticipate and test every possible adversarial input. Red-teaming significantly reduces risk by identifying and mitigating the most critical and common vulnerabilities, but it cannot provide absolute security guarantees. A defense-in-depth strategy, including input validation, output filtering, and continuous monitoring, is essential.
Is it legal to red-team external AI models? Red-teaming should only be conducted on systems you own, or where you have explicit permission from the owner (e.g., through a bug bounty program or direct authorization). Unauthorized red-teaming against third-party models or services may violate Terms of Service and could be considered a cyberattack under various computer misuse laws. Always operate within the bounds of authorized testing and ethical disclosure.
More terms
Glossary term
Glossary term
Collaborate with your team on reliable Generative AI features.
Want expert guidance? Book a 1:1 onboarding session from your dashboard.