promptfoo LLM red teaming and evaluation framework with CI/CD integration
Garak LLM vulnerability scanner
AI-Infra-Guard LLM vulnerability scanner with Web UI, REST APIs, and Dockerized
LLM Guard Security toolkit for LLM interactions
DeepTeam LLM red teaming framework (prompt injection, hallucination, data leaks, jailbreaks)
Agentic Security Security toolkit for AI agents
API Relay Audit: Local CLI for auditing third-party LLM relays and proxies for prompt injection and response anomalies
AI-Scanner AI model safety scanner built on NVIDIA garak
LLMmap Tool for mapping LLM vulnerabilities
LLaMator Framework for testing vulnerabilities of LLMs
Plexiglass Security toolbox for testing and safeguarding LLMs
AgentBench: Benchmark to evaluate LLMs as agents
MCP Scanner Scan MCP servers for potential threats & security findings
Agentic Radar Open-source CLI security scanner for agentic workflows
Awesome MCP Security Curated list of MCP security resources
MCP Shield Security scanner for MCP servers
Agent-Wiz: CLI for threat modeling agentic workflows from LangChain, LangGraph, CrewAI, and AutoGen
Humanbound: Runs LLM-generated adversarial attacks against agents over HTTP, scored against OWASP LLM and Agentic Top 10
MCP Safety Scanner Automated MCP safety auditing and remediation using Agents
Repo Forensics: Offline scanner for agent repos, skills, plugins, and MCP servers detecting supply-chain and injection risk
Agent Security Scanner MCP MCP server for scanning code for web vulnerabilities, prompt injection, and AI-hallucinated package detection
Awesome LLM Agent Security LLM agent security resources, attacks, vulnerabilities
- Skill Safe: Free hosted security checker for AI agent skills and MCP servers
PoisonedRAG Poisoned RAG systems
- RAG Attacks and Mitigations RAG attacks, mitigations, and defense strategies
- Awesome Jailbreak on LLMs - RAG Attacks RAG-based LLM attack techniques
Jailbreak LLMs: Real-world prompt jailbreak dataset (15k+ examples)
Awesome Jailbreak LLMs: Collection of jailbreak techniques, datasets, and defenses
Jailbreaking LLMs (PAIR): Black-box jailbreak generation via automatic prompt refinement
Prompt Fuzzer: Harden your GenAI applications
Open Prompt Injection: Evaluate prompt injection attacks and defenses on benchmark datasets
LLMFuzzer: Fuzzing framework for LLM prompt generation
Spikee: Prompt injection toolkit
Strix
Shannon
PentAGI
PentestGPT
HexStrike AI: MCP server that lets agents run offensive-security tools for automated pentesting
deepsec: Security harness for codebase vulnerability scanning with coding agents
Claude-Red: Offensive-security skill library that primes Claude for specific attack surfaces
DeepAudit: Multi-agent AI red-team platform with Docker sandbox exploit validation
Agentic Bug Hunter: AI-powered bug bounty hunting toolkit
Raptor
PentestAgent
Pentest-Swarm-AI Go-native agents to autonomously perform full-cycle pentests
Pentest-Copilot
AutoCVE: Multi-agent platform for vulnerability detection, verification, and reporting
HackingBuddyGPT
Darkmoon: Autonomous AI pentest platform with per-tech sub-agents and evidence trail per finding
OpenAnt: LLM-powered vulnerability discovery with a second adversarial verification stage
BreachSeek - PENA
Bifrost: AI gateway with guardrails, rate limits, and access controls for governed model and MCP traffic
Guardrails: Add structured validation and policy enforcement for LLMs
NeMo Guardrails: Protects against jailbreak and hallucinations with customizable rulesets
PurpleLlama: Tools to assess and improve LLM security from META
PyRIT: Python Risk Identification Tool for generative AI
LangKit: Functions for jailbreak detection, prompt injection, and sensitive information detection
Prompt Injection Defenses: Practical and proposed defenses against prompt injection
Adrian: Runtime security for AI agents detecting and blocking prompt injection, malicious tool use, and out-of-remit actions
HOL Guard: Local-first agent security layer intercepting tool calls and scanning MCP, skills, and plugins
Vigil: Prompt injection detection toolkit and REST API for LLM security risk scoring
Invariant Trace analysis tool for AI agents
Agent Threat Rules Detection rule standard for AI threats (e.g., prompt injection and MCP attacks)
TraceAI: Tracing for LLM and agent apps
Plexiglass: Security tool for LLM applications
Last Layer: Low-latency pre-filter for prompt injection prevention
ShellWard: AI Agent security middleware
LocalMod: Self-hosted content moderation API with prompt injection detection, toxicity filtering, PII detection, and NSFW filtering
AI Evaluation: Guardrail scanners (jailbreak, PII, prompt-injection); AutoEval pipelines
Veritensor: AI model scanner to detect Pickle/PyTorch malware, check licenses, and verify HF hashes
Aigis: Firewall for AI agents. OWASP LLM Top 10, RAG context filter, MCP 3-stage scanning
- AIDEFEND: Practical knowledge base for AI security defenses
- OWASP Agent Memory Guard: Reference implementation for ASI06 (Memory Poisoning). Runtime defense for LLM agent memory.
OpenShell: Policy-enforced sandbox runtime for autonomous agents, isolating files, syscalls, network, and credentials
Gram: MCP control plane that enforces agent access policies and logs tool activity
Tenuo: Capability-based authorization for AI agents
Lelu: Authorization engine gating agent tool calls on policy and prompt injection
Nika: Rust agent workflow runtime with default-deny permits, secret-flow analysis, and hash-chained traces
- APort: Runtime policy and verification layer for AI agents and MCP-connected tools
- Tuning Engines: AI control and evidence layer for governed model, MCP, skill, and agent traffic with guardrails, policy decisions, approvals, traces, cost analytics
- Penholder: Human-approval write-gate holding agent database writes pending until a human commits
- Speakeasy: Enterprise AI control plane governing agent and MCP access with policy enforcement and auditability
- Alovia Watchdog: Runtime allow/flag/block check of agent actions against a per-agent mission
- Praesidia: Tool permissions, approvals and signed decision records for agent and MCP calls routed through Praesidia
- JailbreakBench: Evaluating and analyzing jailbreak methods for LLMs
L1B3RT45: AI jailbreaking tools
Easy Jailbreak: Python framework to generate adversarial jailbreak prompts
LLM Hacking Database: Attacks against LLMs
- Helium Model Worldview Benchmark: 304 paired cue-swap prompts measuring refusal and bias flip consistency across models
- ThreatModels: Repository for LLM threat models
- Pangea Attack Taxonomy: Comprehensive taxonomy of AI/LLM attacks and vulnerabilities
- AI Risk Taxonomy
- AIR-Bench 2024
- Gandalf: Prompt injection wargame
Damn Vulnerable LLM Agent
LLMVault: Hands-on OWASP LLM Top 10 training lab and CTF
PromptMe
PwnzzAI OWASP LLM ToP 10 vulnerabilities
- PromptTrace: Prompt injection and AI security 10 labs + 15-level CTF with real LLMs.
- AI RiskAtlas: Interactive LLM and agent security lab with incident cases and attack simulations
- RansomLeak AI Security: Browser exercise on indirect prompt injection against an AI assistant
CipherChat: Secure communication tool for LLMs
LLMs Finetuning Safety: Safety for fine-tuning LLMs
Visual Adversarial Examples: Jailbreaking LLMs with visual adversarial examples
FigStep: Jailbreaking vision-language models via typographic visual prompts
OWASP Agentic AI: OWASP Top 10 for Agentic AI
WhistleBlower: Infer the system prompt of an AI agent based on its generated text outputs.
BrokenHill: Automated attack tool for GCG attack
Weak-to-Strong Generalization: Eliciting strong capabilities with weak supervision
AnyDoor: Arbitrary backdoor instances in LLMs
Image Hijacks: Image-based hijacks of LLMs
- ATLAS: Forensic study of prompt-injection cascades across a four-agent enterprise workflow
- RELAY: Study of authority framing and laundered code in multi-agent CI/CD pipelines
- AI Security Explained: Short essential theoretical knowledge
- AI Agents for Pentest: Using agents for penetration testing
- Prompt Injection and Jailbreaking: Practical short lab studies
- hego.red: Practical AI/LLM red teaming notes on prompt injection, jailbreaks, RAG poisoning, and agent attacks
- LLM Security startups
- LLM Security Problems at DEFCON31 Quals: The world's top security competition
- 0din GenAI Bug Bounty from Mozilla: GenAI models threats (prompt injection, training data poisoning, DoS)
- Adversarial Prompting: Documentation
- OWASP Top 10 for LLMs: Official list of key LLM risks including prompt injection
- 🐦 X: @llm_sec
- 🐦 X: @SanderSchullhoff
- 📝 Blog: LLM Security (by @llm_sec)
- 📝 Blog: Embrace The Red
- 📝 Blog: Simon Willison
- 📰 Newsletter: AI safety takes
- 📰 Newsletter & Blog: Hackstery
This repository is actively maintained as a fork of the original project. It includes pending contributions, removes broken links, and separates academic papers from other resources for better organization.
- Main list: projects with broader adoption (typically ≥50★)
- Emerging Projects: early-stage projects under 50★
- Unavailable Projects: commercialized, disappeared, or inactive entries
- Research Papers
Contributions are always welcome. Please read the Contribution Guidelines before contributing.
Alternative: Awesome LLMSecOps