Automated LLM Red Teaming in Production: Comparing Garak, PyRIT, and Promptfoo

Static penetration testing and manual prompt probing cannot secure non-deterministic language models or agentic systems. Manual testing provides anecdotal security at best: the attack surface of large language models spans thousands of adversarial permutations, multi-turn conversational steering, payload encoding, and indirect prompt injections introduced through external retrieval.

To systematically identify failure modes before deployment, engineering teams rely on automated red teaming frameworks. Three open-source tools have emerged as standard implementations across the industry: NVIDIA Garak, Microsoft PyRIT, and Promptfoo. While each probes for model vulnerabilities, they approach adversarial testing with fundamentally different architectural models, execution strategies, and integration points.

The LLM Attack Surface in Production

Securing generative AI pipelines requires testing across three distinct structural layers:

1. Model-Level Safety and Alignment: Direct jailbreaks, toxic content generation, unauthorized system prompt extraction, and unsafe instruction following.
2. Multi-Turn Conversational Escalation: Gradual steering attacks such as Crescendo, where an adversary incrementally shifts context over multiple turns to bypass single-turn alignment filters without triggering safety guardrails.
3. Application and Agentic Integration: Indirect prompt injection via retrieval-augmented generation (RAG) documents, tool manipulation, server-side request forgery (SSRF) via agents, data exfiltration through Markdown rendering, and unauthorized database writes.

Because LLMs do not inherently separate control instructions from untrusted data, automated security tooling must generate diverse attack vectors, transform payloads through encoding layers, and evaluate non-deterministic responses using automated judges.

NVIDIA Garak: Exhaustive Static Vulnerability Scanning

Developed by NVIDIA's AI Red Team, Garak (Generative AI Red-teaming & Assessment Kit) operates as a vulnerability scanner for language model endpoints. Structured similarly to traditional network vulnerability scanners like Nessus, Garak emphasizes broad, standardized probing against known failure modes.

Architecture

Garak's modular design separates probing into five discrete components:

• Generators: Abstraction layers that interface with target models. Garak supports more than 20 backends, including OpenAI, Anthropic, Hugging Face local pipelines, NVIDIA NIMs, vLLM, and generic REST endpoints.
• Probes: Attack modules that formulate and emit adversarial prompts. Garak contains over 120 probe modules covering prompt injection, jailbreaks (such as DAN and Grandma exploits), hallucination benchmarks, encoding bypasses, and data leakage.
• Detectors: Evaluation modules that inspect target outputs. Using regex pattern matching, toxicity classifiers, or judge models, 28 detector types determine whether a probe succeeded in inducing a failure.
• Buffs: Dynamic prompt mutators applied before dispatch. Buffs transform payloads using techniques like Base64 encoding, character substitution, translation, or paraphrasing to test model resilience against obfuscation.
• Harnesses: Orchestration controllers that select probes, map them to corresponding detectors, manage batch dispatch, and produce structured JSONL/HTML vulnerability reports.

Operational Characteristics

Garak excels at rapid, standardized baseline sweeps of raw model checkpoints. It provides deterministic coverage across established academic benchmarks and known CVEs. However, Garak primarily targets single-turn request-response cycles, making it less suitable for auditing multi-step autonomous agents or complex conversational workflows.

Microsoft PyRIT: Stateful Multi-Turn Exploit Orchestration

The Python Risk Identification Tool (PyRIT), developed by Microsoft's AI Red Team, is designed for dynamic, stateful red teaming operations. Rather than firing static prompt lists, PyRIT automates adversarial campaigns where an attacker model actively converses with and manipulates the target system.

Architecture

PyRIT structures red teaming campaigns around a four-pillar framework:

• Targets: Endpoints under assessment, encompassing REST APIs, text and multimodal models, and browser-driven interactive agents automated via Playwright.
• Converters: Payload transformation layers that alter attack prompts on the fly. Converters handle encoding (Base64, ROT13), linguistic translation, noise injection, and social engineering formatting. Converters can be stacked sequentially.
• Scorers: Evaluation engines that classify target responses. PyRIT includes binary judges (SelfAskGptClassifier), Likert-scale scorers (SelfAskGptLikertScorer), and substring heuristics to quantify risk levels.
• Orchestrators: The core execution engine. Orchestrators manage multi-turn attack strategies, tracking conversation state and driving adaptive exploitation.
• Memory Subsystem: Centralized SQLite or DuckDB databases that record every conversation turn, score, converter transformation, and attack trajectory for auditability.

Dynamic Attack Strategies

PyRIT's primary differentiator is its support for complex, algorithmic attack strategies:

• Crescendo Attacks: The orchestrator begins with benign questions, gradually introducing sensitive subtopics across 5 to 10 conversational turns to lead the target into generating prohibited content.
• Tree of Attacks with Pruning (TAP): PyRIT explores branching conversation trees in parallel, scoring intermediate responses and pruning unsuccessful exploitation paths to identify effective jailbreak routes.
• Multimodal Probing: PyRIT natively supports cross-modality attacks, evaluating text-to-image models, visual document parsers, and speech interfaces.

PyRIT provides deep operational insight into adaptive vulnerabilities, though its reliance on attacker LLMs increases inference costs and execution runtimes.

Promptfoo: Declarative CI/CD Red Teaming and Agentic Audits

Promptfoo is a declarative LLM testing and red teaming framework designed to integrate directly into continuous integration pipelines. Unlike research-oriented scanners, Promptfoo bridges developer evaluation workflows and enterprise security compliance.

Architecture

Promptfoo utilizes declarative configuration files (YAML/JSON) to define targets, vulnerability plugins, and mutation strategies:

# promptfooconfig.yaml
description: Customer Support RAG Security Audit
targets:
  - id: https://api.example.com/v1/chat
    label: Production Agent Gateway

redteam:
  purpose: "Customer support agent with database query and order refund tools"
  plugins:
    - owasp:llm
    - indirect-prompt-injection
    - pii:direct
    - excessive-agency
    - contracts
  strategies:
    - jailbreak
    - crescendo
    - base64
    - multilingual

Key architectural components include:

• Purpose-Driven Probe Synthesis: Promptfoo uses an adversarial generator to synthesize context-specific attack prompts tailored to the application's stated purpose and system prompt.
• OWASP LLM & Agentic Presets: Built-in plugin suites map directly to the OWASP Top 10 for LLM Applications and OWASP Agentic Security Risks, covering excessive agency, insecure output handling, prompt injection, and sensitive data leakage.
• Strategy Mutators: Automated attack transforms apply jailbreak wrappers, Crescendo multi-turn sequences, and encoding variations to synthesized probes.
• Headless CI/CD Integration: Promptfoo runs natively in GitHub Actions, GitLab CI, and command-line environments, allowing teams to set pass/fail thresholds and block pull requests that introduce safety regressions.

Architectural Trade-Offs

Choosing among Garak, PyRIT, and Promptfoo depends on the target layer, execution frequency, and pipeline maturity:

Target Scope

• Garak: Model-centric. Best for foundation models, open-weight checkpoints, and raw inference endpoints.
• PyRIT: System-centric and multimodal. Best for deep penetration tests, multi-turn reasoning agents, and complex attack paths.
• Promptfoo: Application-centric. Best for RAG pipelines, enterprise agents with tool access, and full conversational applications.

Attack Generation

• Garak: Curated static datasets and academic benchmarks mutated via Buffs.
• PyRIT: Dynamic attacker LLMs driving iterative, stateful conversation trees.
• Promptfoo: Synthetic generation tailored to application intent, combined with heuristic vulnerability plugins.

Integration Workflow

• Garak: Standalone Python CLI and library, integrated into model validation suites (such as NVIDIA NeMo Guardrails).
• PyRIT: Programmatic Python framework for custom offensive security scripts and red team campaigns.
• Promptfoo: Declarative YAML CLI, web dashboard, and native CI/CD test harness.

Designing a Multi-Tier Red Teaming Pipeline

In production engineering environments, these tools serve complementary roles across the deployment lifecycle:

Tier 1: Continuous Integration (Promptfoo)

Run targeted Promptfoo scans on every pull request that modifies system prompts, RAG retrieval logic, or tool definitions. Test specifically for regression against OWASP LLM top risks and core brand policies within a 5-to-10 minute execution budget.

Tier 2: Nightly Baseline Audits (Garak)

Execute comprehensive Garak sweeps across all hosted model endpoints and fine-tuned checkpoints. This verifies that underlying model upgrades or quantization passes have not compromised baseline safety boundaries.

Tier 3: Periodic Deep Multi-Turn Campaigns (PyRIT)

Schedule weekly or pre-release PyRIT campaigns utilizing TAP and Crescendo orchestrators. These tests systematically probe for complex logic flaws, privilege escalation, and multi-turn jailbreaks across autonomous agent workflows.

Tier 4: Runtime Defense Calibration

Feed successful red team payloads back into runtime defense layers, such as NeMo Guardrails or Llama Guard. Use empirical bypass rates to calibrate input classifiers, response filters, and system-prompt constraints.

Sources

• Garak: A Framework for Security Probing Large Language Models (arXiv)
• NVIDIA Garak Repository
• Microsoft PyRIT Framework
• Cloud Security Alliance: Evaluating PyRIT for Agentic AI Red Teaming
• Promptfoo Red Teaming Documentation
• OWASP Top 10 for Large Language Model Applications
• NVIDIA NeMo Guardrails Documentation
• Meta AI: Llama Guard Publication