Community
Star history
Track ARGUS's growth on GitHub.
Closed-loop multi-agent framework for LLM vulnerability discovery. Adaptive attack strategy. CVSSv4.0 scoring. 8 regulatory frameworks. SARIF output for CI/CD.
Full coverage across direct endpoints and complex agentic topologies: the surfaces static scanners miss entirely.
Classic LLM attacks against completion APIs: alignment bypass, system-prompt extraction, role confusion, and jailbreaking.
Indirect cross-prompt injection via retrieval documents. Attacks the model through the data layer it trusts.
Model Context Protocol attack surface: tool-call interception, poisoned context injection, cross-server propagation.
Agent-to-agent injection that hops trust boundaries and propagates exploitation through downstream agents.
Exploiting function-calling interfaces: schema confusion, argument injection, and output manipulation.
CVSSv4.0 vectors for every finding, automatically mapped to applicable regulatory frameworks. SARIF v2.1 output for CI/CD gates.
Four specialized agents in a continuous revision loop. The Planner reads live behavioral signals and rewrites its strategy: not a fixed probe list.
Target LLM / Pipeline
|
v
┌───────────────────────────────────────────────┐
│ Orchestrator │
│ │
│ Planner ────────> Attacker ────> Evaluator │
│ (Claude Opus) (synthesizer) (3-layer) │
│ ^ | | │
│ | v v │
│ └────── Revision <──── Findings + CVSS │
│ | │
│ Reporter │
└───────────────────────────────────────────────┘
|
v
SARIF v2.1 + JSON ──> CI/CD gates + GRC tooling
Cosine distance against a confirmed-attack embedding space. Flags responses that land in known attack-success neighborhoods, independent of surface form: catches paraphrases and evasive reformulations.
Multi-model verdict with configurable affirmative quorum and confidence threshold. Cross-model agreement eliminates single-model blind spots and dramatically reduces false positives.
Pipeline telemetry anomaly detection. Catches exploitation that produces no suspicious output text but alters tool-call patterns, token budgets, or downstream API behavior.
Static probe-and-detect tools catch known patterns. ARGUS reasons, adapts, and discovers novel vulnerabilities autonomously.
| Feature | Garak (NVIDIA) | PyRIT (Microsoft) | LLM-Fuzzer | ARGUS |
|---|---|---|---|---|
| Multi-agent architecture | ✗ | ✗ | ✗ | ✓ |
| Adaptive attack strategy | ✗ | ✗ | ✗ | ✓ real-time |
| Cross-session memory | ✗ | ✗ | ✗ | ✓ ChromaDB |
| OWASP LLM Top 10 (2025) | partial | partial | ✗ | ✓ all 10 |
| RAG / pipeline attacks | ✗ | partial | ✗ | ✓ |
| MCP server mesh attacks | ✗ | ✗ | ✗ | ✓ |
| Multi-agent propagation | ✗ | ✗ | ✗ | ✓ |
| CVSSv4.0 scoring | ✗ | ✗ | ✗ | ✓ |
| SARIF v2.1 output | ✗ | ✗ | ✗ | ✓ |
| 8-framework compliance mapping | ✗ | ✗ | ✗ | ✓ |
| LLM-as-judge panel detection | ✗ | partial | ✗ | ✓ quorum |
| Behavioral trace analysis | ✗ | ✗ | ✗ | ✓ |
| Academic backing | ✓ | ✓ | ✓ | ✓ IEEE |
Every design claim is backed by a measured outcome. All results are reproducible from the public repository.
The scoring engine implements the official FIRST MacroVector/lookup-table/interpolation algorithm (ported from FIRSTdotorg/cvss-v4-calculator), not an approximation. 48 hand-crafted test vectors validated against that reference implementation: 48/48 exact matches, 0.0 mean deviation. Five representative cases:
| Attack scenario | CVSSv4.0 vector | ARGUS | Reference | Δ |
|---|---|---|---|---|
| Direct prompt injection — system-prompt extraction | AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:N/VA:N/SC:H/SI:N/SA:N | 9.2 Critical | 9.2 | 0.0 |
| Cross-agent privilege escalation via MCP poisoning | AV:N/AC:H/AT:P/PR:L/UI:N/VC:H/VI:H/VA:H/SC:H/SI:H/SA:H | 9.0 Critical | 9.0 | 0.0 |
| RAG indirect injection, low downstream impact | AV:N/AC:H/AT:N/PR:N/UI:N/VC:L/VI:N/VA:N/SC:L/SI:N/SA:N | 6.3 Medium | 6.3 | 0.0 |
| Tool-call schema confusion, no downstream reach | AV:N/AC:L/AT:N/PR:L/UI:N/VC:L/VI:L/VA:N/SC:N/SI:N/SA:N | 5.3 Medium | 5.3 | 0.0 |
| Zero-impact probe (all Vulnerable & Subsequent metrics: None) | AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:N/SC:N/SI:N/SA:N | 0.0 None | 0.0 | 0.0 |
20-candidate pools generated for 3 OWASP categories × 2 open-weight local targets (qwen2.5:7b, llama3 via Ollama), embedded with a real dense model (nomic-embed-text). At the shipped default threshold δ = 0.35, constrained synthesis reduces mean intra-batch cosine similarity by 14.5% versus unconstrained generation (0.53 vs. 0.62), averaged across all six configurations — yield and similarity decline smoothly across δ with no sharp optimum, so 0.35 is a moderate-yield/moderate-diversity default, not a calibrated best point. Only one configuration (LLM01, qwen2.5:7b) retained enough admitted candidates across the full δ sweep to trace a real curve; llama3 produced highly repetitive completions on the other five configurations (unconstrained similarity 0.83–0.95), collapsing them to 1–2 admitted candidates by δ=0.25–0.30 — a genuine model-dependent limitation, reported as such rather than smoothed into the aggregate. Full δ sweep and methodology in the paper, Section VII.C.
"Full" below means a dedicated payload module, scoring path, and compliance mapping exist per category (verified by source inspection against garak v0.15.x main branch) — it is an architectural comparison, not a measured detection rate for the other eight categories. A live-target empirical pilot with real per-category precision/recall/F1 has now been run for LLM01 and LLM06 (see below); the remaining eight categories are architectural coverage only.
| ID | Category | garak v0.15 | ARGUS v1.0 | Coverage gain |
|---|---|---|---|---|
| LLM01 | Prompt Injection | Full | Full + XPIA / MCP | Extended |
| LLM02 | Insecure Output Handling | Partial | Full | + |
| LLM03 | Training Data Poisoning | Partial | Full | + |
| LLM04 | Model Denial of Service | Partial | Full | + |
| LLM05 | Supply Chain Vulnerabilities | None | Full | New |
| LLM06 | Sensitive Information Disclosure | Full | Full | Equal |
| LLM07 | Insecure Plugin Design | Partial | Full | + |
| LLM08 | Excessive Agency | None | Full | New |
| LLM09 | Overreliance | Partial | Full | + |
| LLM10 | Model Theft | None | Full | New |
100 real (prompt, response) pairs across qwen2.5:7b and llama3 (via Ollama) for LLM01 (prompt injection) and LLM06 (secret disclosure), scored by a faithful reimplementation of garak's StringDetector (verified against garak v0.13.3 source), the real SemanticProximityDetector, a local cross-model LLM-judge, and the actual shipped EvaluatorAgent pipeline. Ground truth: a disclosed deterministic rule (canary/secret present with no refusal marker), not manual labeling — 22 ground-truth leaks, 78 negatives.
| Detector | Precision | Recall | F1 | FPR |
|---|---|---|---|---|
| garak-style substring match | 0.59 | 1.00 | 0.75 | 0.19 |
| Semantic detector alone | 0.48 | 0.68 | 0.57 | 0.21 |
| LLM-judge alone | 0.80 | 0.55 | 0.65 | 0.04 |
| Shipped pipeline (Layer 1 gates Layer 2) | 0.80 | 0.36 | 0.50 | 0.03 |
The garak-style detector's false positives come entirely from refusals that quote the trigger string while declining it — a concretely observed instance of keyword-matching's failure mode. The judge alone is the strongest single detector; gating it behind the semantic layer (the actual shipped design) reduces F1 below the judge running alone, a genuine calibration finding rather than a smoothed-over one. Scope: 2 of 10 categories, 2 models, single-rule-labeled, 100 pairs — a first real measurement, not the full controlled benchmark. Methodology and raw data: benchmarks/live_target_pilot.py in the repo.
ARGUS emits a standards-compliant SARIF v2.1 report at the path specified in the scan config. No post-processing or schema conversion is needed.
A single upload-sarif step with the output path is the only pipeline addition. No format-version pinning, no custom mapping. garak requires a hand-written JSONL-to-SARIF converter before this step.
Findings appear with correct rule IDs, severity labels, and descriptions. The 45-second window was measured on a representative repository; GitHub's ingestion pipeline, not ARGUS, is the bottleneck.
Install once, scan any supported target with a single command. API key resolved from env or --api-key flag.
git clone https://github.com/\ sunilgentyala/argus cd argus pip install -e .
argus scan \ --target anthropic \ --model claude-sonnet-4-6 \ --profile quick
argus scan \ --target openai \ --model gpt-4o \ --system-prompt "..." \ --profile compliance
argus show \ ./argus-reports/\ <session-id>.report.json
Every confirmed finding is automatically tagged to applicable regulatory articles: ready for your audit trail or GRC team.
Govern, Map, Measure, Manage: full control framework mapping.
High-risk system requirements, transparency and conformity obligations.
Federal AI safety, security reporting, and red-team requirements.
AI Safety Institute evaluation and testing standards.
Cybersecurity incident reporting and AI governance guidelines.
International AI management system standard controls.
API protection controls for cloud-native LLM deployments.
Track ARGUS's growth on GitHub.
Latest release ships HTML reports, direct SARIF output, and threat intelligence.
Self-contained HTML output with severity-colour-coded findings table. Attach to GRC tickets or email to security stakeholders: no external assets required.
New --sarif argus.sarif flag lets you specify the SARIF output path directly: perfect for CI pipelines that pick up a specific filename.
New argus/intelligence/ module ships a curated LLM attack signal database: helping the Planner bias toward highest-yield vectors per model family.
Planner now defaults to claude-opus-4-8: Anthropic's latest reasoning model. Attack strategy quality improves measurably on complex agentic surfaces.
A star on GitHub helps security researchers find ARGUS and signals that open LLM security tooling matters.