AgentShield: turning "AI agent configuration audit" from a Claude Code hackathon project into a CI-ready compliance tool

1. TL;DR

★★★★☆ (4/5). An AI agent configuration auditor built by affaan-m at the Claude Code Hackathon (Cerebral Valley × Anthropic, Feb 2026), MIT, 1,229 stars / 268 forks (as of 2026-09-27), part of his Everything Claude Code ecosystem (42K+ stars). It does not scan your business code for vulnerabilities — it scans your own .claude/ directory for hardcoded secrets, overly permissive tool rules, command injection in hooks, MCP server exposure, and agent prompt-injection triggers. 268 rules (per the README) graded A–F with a 0–100 score, drop it into CI with fail-on-findings: "true" and you have a gate. One star off for: three different rule counts in the README (268 / 102 / 17+40+49+41, see §7.1), only 6 months of history, and --opus/--injection send your .claude/ contents to the Anthropic (or OrcaRouter) API — redact before you point them at anything sensitive. Claude Code power users: install it now. Other harnesses: treat coverage as a bonus. Public internet / multi-tenant: skip the LLM-backed modes.

Who is it for (out of 5):

Customisation tiers (MIT):

2. What it is, and what it isn’t

Five things AgentShield is not:

Not a general SAST/DAST. It does not scan src/, backend/, business code — that’s Semgrep / CodeQL / Snyk territory. It scans the agent configuration you authored: the “Always run this without asking” in your CLAUDE.md, the Bash(*) in .claude/settings.json, the curl ${user_input} in a hook script, the npx -y in an MCP server command, the zero-width character in a sub-agent prompt. Different layers, different scanners.

Not a runtime sandbox. Static + reporting scanner. It does not execute or intercept your hooks. The --sandbox flag executes hooks inside a sandbox and observes behavior — that’s active validation, not enforcement. --taint is data-flow tracking (experimental). Real prevention of malicious hook execution still depends on Claude Code’s own PreToolUse hooks — AgentShield just tells you where you need one.

Not an MCP server vulnerability scanner. It does not audit MCP server code; it audits whether the MCP server configuration in .claude/mcp.json is dangerous (high-risk server type, sensitive files in args, shell metacharacters, missing version pin, environment inheritance, autoApprove, remote URL transport, etc.). To audit server code itself you need other tooling.

Not Claude Code only. The README documents local harness adapter evidence (marker-based, no external calls): Claude Code / OpenCode / Codex / Gemini / Zed / VS Code / dmux / terminal-agent wrappers / project-local templates all get recognized. But this is “the directory has a marker file” — not deep multi-harness support. Claude Code is the deepest; treat the rest as a bonus.

Not safe to point at production without thinking. --opus / --injection / --provider orcarouter send your .claude/ contents (potentially including secrets, URLs, tokens) to an LLM API. The README itself warns: do not use it on configs containing secrets you have not redacted. In CI, either rely on default redaction or strip secrets before running.

What is it, then? The official tagline:

AI agent security scanner. Detect vulnerabilities in agent configurations, MCP servers, and tool permissions. Available as CLI, GitHub Action, ECC plugin, and GitHub App integration.

Four keywords: agent configuration (not business code) / graded score (0–100, not boolean) / CI-native (GitHub Action is first-class) / recognize-and-attribute (recognized defenses get listed but neither penalized nor credited, blocking score-gaming).

3. Five scanning surfaces (268 rules)

AgentShield breaks down by file type into five modules, matching the five kinds of configuration in ~/.claude/.

A few design choices worth calling out:

Secrets catches env-var leakage even when your config writes ${SECRET} instead of a literal — if a hook command contains echo $SECRET or any env-var passthrough, it’s flagged critical. Smarter than grep.

Permissions grades by source: a fully-specified allowlist in settings.local.json downgrades permissions-no-deny-list from high to medium; wildcards keep high. Same rule, different severity by context.

Hooks targets non-shell implementations specifically. A manifest-referenced hook (e.g. hooks/hooks.json pointing at scripts/hooks/session-start.js) gets traced to its actual implementation, where the scanner looks for explicit output(...) context injection, transcript input access, and child-process exfiltration. Closer to the real risk than parsing shell strings.

MCP runtimeConfidence bucketing is the tool’s strongest design — see §4. The same npx -y in an active .mcp.json is critical; the same string in a docs/ example is template-example and weights at 0.25× with a 10-point per-file cap.

Agents prompt-injection rules ignore defensive content. Mentioning fetch(userProvidedUrl) in a teaching prompt does not trip agents-injection-surface; actually instructing the agent to fetch/process external content does. Defensive prompts don’t trigger their own detectors.

4. Scoring and false-positive governance (where the value actually is)

The moat isn’t rule count — it’s how it scores and how it governs false positives.

4.1 Scoring

Each category starts at 100; deductions are findings-only:

• critical −25 / high −15 / medium −5 / low −2 / info 0

Final score → letter grade (A 90+ / B 80+ / C 70+ / D 60+ / F <60).

Recognized Defenses are listed separately in the report:

• Not penalized (a deny rule hit still counts as allow)
• Not credited (no extra points for decorative deny rules)

The score can only be lowered by real findings. Important for GRC reports: a high score means “no real findings detected”, not “wrote a lot of denies”.

4.2 runtimeConfidence (7 levels)

This is the most sophisticated false-positive mechanism:

The seven tiers answer “is this config actually running right now?”, and the weight multiplies the deduction directly:

active-runtime (weight 1.0×) — Actually-running config: .mcp.json, mcp.json, .claude/mcp.json, .claude.json, active settings.json

project-local-optional (0.75×) — Local-only overrides like settings.local.json, scoped to the current project

plugin-cache (0.5×) — Installed plugin cache, e.g. .claude/plugins/cache/...

plugin-manifest (0.5×) — Declarative hook manifests, e.g. hooks/hooks.json

template-example (0.25×, 10-point per-file cap) — Template/catalog definitions, e.g. mcp-configs/ or config/mcp/

docs-example (0.25×) — Docs and tutorials, e.g. docs/guide/settings.json or commands/*.md

hook-code (1.0×, narrow rule set) — The non-shell implementation a manifest resolves to, e.g. scripts/hooks/session-start.js

The practical effect: a risky MCP server template in docs/ does not score like an enabled one. It gets tagged template-example, weighted at 0.25×, capped at 10 points per file. That’s the difference between “many false positives” and “tiered triage”.

4.3 --fix with tamper-evident rollback

agentshield scan --fix is not a sledgehammer:

1. Apply only auto: true transforms (replace hardcoded secrets with ${ENV_VAR}, tighten Bash(*) to Bash(git *) etc.)
2. Re-scan after applying
3. If the score regressed or new high/critical findings appeared (the kind of churn a naive permission tighten can cause), roll every modified file back
4. If OK, emit a tamper-evident attestation digest binding the before/after scores and finding deltas

That turns --fix from “auto-breaks-stuff-and-doesn’t-tell-you” into “auto-breaks-stuff-and-undoes-itself-with-evidence”. Crucial for any production-config automation.

4.4 The FP audit workflow

The repo ships false-positive-audit.md with a complete triage workflow:

1. Start with --format json to inspect file paths, runtimeConfidence, and score impact directly
2. Separate active-runtime findings from lower-confidence source kinds before changing any rules
3. Validate suspected false positives against at least one real repo and one minimal synthetic fixture
4. Prefer source-aware reclassification and wording changes over blanket suppression
5. Keep real secrets and explicit execution paths visible even inside examples, manifests, or templates
6. Re-run targeted tests, then the full gate, before changing release behavior

Use this workflow verbatim as a template for any static scanner’s FP governance.

5. Advanced: Opus adversarial / compliance / supply chain

5.1 Three-agent adversarial analysis (--opus)

Pure rule matching misses cross-rule composite attacks. --opus spins up a three-agent pipeline:

1. Attacker (red team, Claude Opus) — finds exploitable vectors and multi-step chains
2. Defender (blue team, Claude Opus) — evaluates existing defenses and recommends hardening
3. Auditor — synthesizes both into a prioritized risk assessment

The Attacker finds “curl hook + ${file} interpolation + Bash(*) = command-injection pivot”; the Defender notes no PreToolUse hook to stop it; the Auditor chains them into a prioritized action list.

Requires ANTHROPIC_API_KEY (or ORCAROUTER_API_KEY for an OpenAI/Anthropic-compatible gateway). Calls Anthropic’s API and transmits your .claude/ content — see §7.2.

5.2 Compliance mapping (--compliance)

1
2
3

agentshield scan --compliance soc2          # SOC 2 Trust Services Criteria
agentshield scan --compliance soc2,pci,iso  # multiple
agentshield scan --compliance all

Emits a control coverage table (control id / title / highest severity / finding count / examples), sorted by severity. Gives GRC teams an auditor-readable coverage artifact instead of raw findings.

It honestly labels itself as finding-category-level guidance (SOC 2, PCI DSS v4.0, ISO/IEC 27001:2022 Annex A), not a certified crosswalk. Confirm applicability with your auditor. The disclaimer is written well.

5.3 Supply chain verification (--supply-chain)

The npm npx -y flow is a common typosquatting vector. --supply-chain extracts MCP package references + root package.json / package-lock.json dependency evidence and reports:

• npm vs git source counts
• pinned vs unpinned
• known-good packages
• npm-registry metadata

Add --supply-chain-online to query npm for downloads, maintainers, postinstall scripts, deprecation, and package age.

Also checks .npmrc / .yarnrc.yml / pnpm-workspace.yaml for plaintext registry credentials, minimumReleaseAge / npmMinimalAgeGate cooldown config. GitHub Action defaults supply-chain: true, failure mode is controllable.

5.4 MiniClaw: a bundled secure-by-default HTTP agent

npx ecc-agentshield miniclaw start boots an isolated agent runtime with a single HTTP endpoint. Compared to typical agent platforms that expose Telegram / Discord / email / community plugins (each a separate attack surface), MiniClaw exposes localhost:3847 by default + isolated sandbox + safe tool set only. Lightweight option for “I want to run an agent locally too”. Also importable as a library: import { startMiniClaw } from 'ecc-agentshield/miniclaw'.

6. Deployment and integration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

# Fastest path: zero install
npx ecc-agentshield scan

# Global install
npm install -g ecc-agentshield
agentshield scan

# Scan a specific directory
agentshield scan --path /repo/.claude

# Auto-fix safe items (rolls back if it regresses, emits attestation)
agentshield scan --fix

# CI-friendly outputs
agentshield scan --format json | jq .
agentshield scan --format sarif --output agentshield.sarif

# Self-contained HTML report
agentshield scan --format html > report.html

# Verifiable audit bundle (SHA-256 manifest)
agentshield scan --evidence-pack ./agentshield-evidence
agentshield evidence-pack verify ./agentshield-evidence

# Stable-fingerprint fix queue (no raw secret values)
agentshield scan --remediation-plan ./remediation.json

# Three-agent adversarial analysis
agentshield scan --opus --stream

# Active prompt-injection testing
agentshield scan --injection

# Compliance mapping
agentshield scan --compliance soc2,pci,iso

# Supply chain
agentshield scan --supply-chain --supply-chain-online

# Organization policy
agentshield policy init --pack enterprise
agentshield policy export --output-dir .github/agentshield-policies
agentshield policy promote --manifest .github/agentshield-policies/manifest.json --pack enterprise

# Initialize a hardened baseline config
agentshield init

The GitHub Action is first-class:

1
2
3
4
5
6
7
8
9
10

- name: AgentShield Security Scan
  uses: affaan-m/agentshield@v1
  with:
    path: "."
    min-severity: "medium"
    fail-on-findings: "true"
    format: "sarif"               # → SARIF → github/codeql-action/upload-sarif
    baseline: ".github/agentshield-baseline.json"   # → drift gate
    supply-chain: "true"
    evidence-pack: "./agentshield-evidence"

25+ outputs including score / grade / critical-count / new-findings / resolved-findings / score-delta / policy-status / supply-chain-critical-count / evidence-pack-digest — all routable to other workflow steps.

Exit codes: 0 no critical / 1 CLI error / 2 has critical. CI-friendly.

7. Limits and risks (honest weaknesses)

7.1 Three rule-counts, one README, no reconciliation

This is what tripped me up most when writing the article:

Three totals that don’t reconcile. Likely explanation: “rules” and “patterns” are mixed; the architecture summary lists only “core” rules; some are hard/soft tiered. The README never spells this out. Hidden trap for any team that wants to reason about rule coverage in CI — you don’t actually know if 100 or 300 rules are guarding your code.

Workaround: run --format json and jq '.findings | length' to see the actual finding-rule-id set, then make CI decisions on real numbers rather than README figures.

7.2 --opus / --injection exfiltrate .claude/ content

The README says:

With --provider orcarouter the scanned configuration contents are sent to OrcaRouter’s API instead of Anthropic’s, so do not use it on configs containing secrets you have not redacted.

This applies to both --opus and --injection. Implications:

1. In CI, everything in .claude/ on the runner (secrets, URLs, usernames, internal paths) goes to Anthropic or OrcaRouter
2. The evidence-pack redaction targets local paths / usernames / emails / token-shaped strings — it does not necessarily cover your business-specific sensitive data
3. Cross-border data-flow compliance may apply — certain regulated environments (finance, government) cannot put runners that call foreign LLMs on production configs

Mitigations:

• agentshield scan --no-evidence-redact is the inverse (turns off redaction) — don’t enable it casually
• If you must run --opus, manually redact PII / secrets in .claude/ first
• Or run --opus only against an isolated, scrubbed copy

7.3 Young project, hackathon origins

• First commit 2026-02-11
• Latest push 2026-09-10
• 6 months of history (very young by AI-security-tool standards)
• Hackathon-origin projects iterate fast — the rule counts changed during this article’s preparation

Implications:

• APIs, field names, and rule IDs can break pre-v2 (pin @v1 for the GitHub Action; pin CLI version locally)
• Report schemas will evolve (runtimeConfidence is itself recent)
• Watch for stability before long-term org-level adoption

7.4 FP governance still requires human triage

The README admits:

The current scan profile is not dominated by broken matchers. It is mostly dominated by lower-confidence source kinds that need different interpretation.

Specifically:

• Template MCP inventory is the biggest noise source (large ecosystems like everything-claude-code ship mcp-configs/ templates)
• Example/tutorial config needs example-aware interpretation, not blanket suppression
• Broad agents-* clusters across files are usually policy review, not false-positive suppression

runtimeConfidence buckets the noise, but a human still triages it. Dumping the JSON report into a GRC pipeline is not enough.

7.5 Harness adapters are marker evidence only

README:

harnessAdapters is local marker evidence only. It does not call external services or imply a hosted/team entitlement.
Adapter confidence is strong when a primary harness marker exists, and partial when only supporting directories or secondary markers are present.

OpenCode / Codex / Gemini / Zed are recognized because their directories contain certain marker files — not because the scanner does deep harness-specific analysis. Claude Code is the deepest. If you’re a Codex-only shop, AgentShield is at 60% value.

7.6 Doesn’t validate runtime behavior

--sandbox (execute hook in sandbox, observe) and --taint (data-flow tracking) are active validation, not passive monitoring. Claude Code’s own PreToolUse hooks are the real execution gate. AgentShield is just a scanner. Don’t treat it as runtime protection.

7.7 Commercial sponsor embedded

The README opens with a “Preferred compute sponsor: Itô Markets” line and an ECC affiliate link. Legal, but worth flagging for neutral technical evaluation: the ECC ecosystem is commercializing. AgentShield itself is MIT with no paid tier; ECC Tools Pro / GitHub App may be paid products.

8. Getting started

Step 1: run it once, see how bad your score is.

1

npx ecc-agentshield scan

This will surface a pile of findings in your .claude/: “Always run” in CLAUDE.md → agents-auto-run, Bash(*) in .claude/settings.json → permissions-permissive-*, ${file} in hooks/*.sh → hook-command-injection. First run can easily produce 50+ findings. Don’t panic.

Step 2: understand the tiers before you start fixing.

1
2
3
4
5
6
7

npx ecc-agentshield scan --format json > report.json

# Bucket by runtimeConfidence
jq '.findings | group_by(.runtimeConfidence // "none") | map({
  runtimeConfidence: (.[0].runtimeConfidence // "none"),
  count: length
})' report.json

active-runtime + project-local-optional are the genuinely high-signal buckets — read these first. Triage the others as needed.

Step 3: use --fix for the auto-fixable items.

1

npx ecc-agentshield scan --fix

This will auto-replace hardcoded secrets + tighten some wildcards. If the score drops or new high/critical findings appear, it auto-rolls back and gives you an attestation digest. One of the tool’s best features.

Step 4: install the GitHub Action.

1
2
3
4
5
6
7
8
9
10
11
12
13

# .github/workflows/agentshield.yml
- uses: affaan-m/agentshield@v1
  with:
    path: "."
    fail-on-findings: "true"
    min-severity: "medium"
    format: "sarif"
    sarif-output: agentshield.sarif
    baseline: ".github/agentshield-baseline.json"   # progressive rollout
- uses: github/codeql-action/upload-sarif@v3
  if: always()
  with:
    sarif_file: agentshield.sarif

Progressive rollout tip: first run with baseline: "", save the output as agentshield-baseline.json and commit it. Subsequent runs with --gate will only fail on new critical/high findings.

Step 5: roll out an org policy.

1
2
3

agentshield policy init --pack enterprise   # Start with the enterprise template
agentshield policy export --output-dir .github/agentshield-policies
agentshield policy promote --manifest .github/agentshield-policies/manifest.json --pack enterprise --dry-run --json  # dry-run first to see review items

Promote policy.json into .agentshield/policy.json and the Action will fail-on-policy automatically.

First move for new users: npx scan to see your score. Don’t install the Action with strict gating first — you’ll get buried in findings. Learn the tiers, then progressive-roll with baseline + --fix.

9. One-line conclusion

AgentShield is the only MIT tool that takes “AI agent configuration audit” all the way to a 0–100 score, CI gating, FP tiering, compliance mapping, and active adversarial analysis — but its rule-count inconsistency, noise-management overhead, .claude/ exfiltration to LLMs, and “Claude Code-first” depth mean you have to use it deliberately: take a baseline, read the runtimeConfidence tiers, redact before --opus/--injection, treat non-Claude-Code adapter coverage as a bonus. For Claude Code power users: just install it, no other tool is this close to your daily surface. For multi-harness teams: use it as the Claude Code coverage layer and supplement the other harnesses with separate tools.