Pointing a frontier model at a codebase and asking it to “find the bugs” is tempting. It is also a reliable way to generate a pile of confident text that nobody should trust yet.
That is the useful part of Cisco’s Foundry Security Spec. It does not pretend the model is magic. It treats the model as one component inside a larger security evaluation system: indexed code, scoped goals, isolated execution, durable work queues, evidence gates, validation, reporting, coverage tracking, and budget controls.
That framing matters. In real security work, the harness matters more than the model.
A model can notice things. A harness decides what counts.
What Foundry Actually Is
Foundry is an open specification for agentic AI security evaluation. The important word is specification. The Cisco repository is explicit that Foundry is not Cisco’s internal scanner, not a managed service, and not a tool you can pip install and point at a repo this afternoon.
The repository intentionally contains no implementation code. The spec is the deliverable.
That makes it more interesting, not less. Cisco is publishing the shape of a system that worked internally: the roles, invariants, finding lifecycle, coordination substrate, governance rules, and integration decisions that turn open-ended AI analysis into something bounded enough for security teams to use.
There is one caveat worth keeping up front: spec.md is a seed, not a finished implementation spec. It is meant to be clarified against your own environment: your target repositories, LLM provider, datastore, issue tracker, testbed, sandbox, budget limits, and reporting workflow. If you skip that clarification step, you are not building Foundry. You are copying headings.
The other important assumption is authorization. Foundry is for evaluating software you are allowed to test, with source access and optionally a running testbed. It is not a black-box bug bounty robot and it is not a license to aim autonomous agents at random targets.
Why “Find Bugs” Is Not a Workflow
The failure mode is familiar by now: ask a model for vulnerabilities and it will produce something that looks like a security report. Some findings may be sharp. Some will be plausible nonsense. The output is unbounded, hard to reproduce, and usually missing a credible answer to a basic question: are we done?
The usual problems show up quickly:
- The model cites code that does not exist.
- Every candidate sounds urgent.
- Parallel agents duplicate each other’s work.
- Long-running sessions wander into expensive dead ends.
- Reviewers get flooded and stop reading.
- Safety exists only in a prompt, which means it does not really exist.
Foundry’s answer is not “write a better prompt.” It is system design.
The most important principle in the Foundry constitution is evidence over assertion: a finding’s verdict is based on checkable evidence, not model confidence. A model can propose a bug. It cannot simply declare one true because the prose sounds convincing.
That is the difference between AI-assisted security work and AI-shaped noise.
The Eight Core Roles
Foundry breaks the harness into eight core responsibilities. You do not have to implement them with eight products or eight model providers, but the boundaries matter because each role catches a different failure mode.
Orchestrator
The operator’s control surface. It validates configuration, starts and stops the fleet, tracks status, enforces budget, handles operator questions, and queues steering requests. It should not quietly become a detector, triager, or reporter.
Indexer
The code-understanding layer. It builds symbols, references, call graphs, imports, embeddings if you use them, and whatever other structure later roles need. A detector that cannot ask “who calls this?” is mostly guessing.
Cartographer
The security-context layer. It creates the architecture summary, attack surface, trust boundaries, data flows, and threat model. This gives the rest of the system a map of where bugs would matter.
Detector
The high-volume candidate generator. It has two lanes: systematic rule sweeps and exploratory hunting. It writes candidates to the internal finding store. It does not write tickets.
Triager
The noise filter. It investigates candidates and assigns verdicts. A true positive has to pass the evidence gate. Anything that does not gets rejected, recorded, or marked for review.
Validator
The proof filter. If a safe testbed exists, the Validator independently tries to reproduce the headline impact. This is the only role that should set an “exploited” flag.
Coverage-Guide
The “are we done?” role. It turns the operator’s goals into a checklist and tracks whether the requested ground has been credibly attempted.
Reporter
The human-facing output layer. It writes confirmed findings, severity, reproduction steps, and rollups. The Reporter should publish only what survived triage.
That decomposition is not bureaucracy. Detection without triage creates noise. Triage without validation creates plausible fiction. Validation without coverage gives you a pile of bugs with no claim to completeness. Reporting without deduplication creates the same issue again every time a line number changes.
A Practical MVP Harness
A small team should not start by implementing every Foundry requirement. Start with the smallest version that preserves the important invariants.
A reasonable first build could be:
- One operator-controlled machine.
- A read-only checkout of the target repository.
- A pinned target revision and explicit scope file.
- SQLite in WAL mode, or another atomic datastore, for the substrate.
- Docker, Podman, a VM, or another real isolation boundary with locked-down network egress.
- Markdown reports before issue tracker integration.
- Three main agent loops: Detector, Triager, Reporter.
- Validator disabled unless a disposable testbed exists.
- Coverage tracked from a small goals document.
- CodeGuard-compatible rules, or a simple custom YAML rule format.
That MVP is not the whole Foundry system, but it keeps the part that matters: candidates are internal, true positives require evidence, and reports only surface what survived review.
The Substrate Is the Product
The substrate is the boring machinery that makes a parallel agent fleet behave like one system instead of several chat sessions fighting each other.
For an MVP, SQLite can do more than people think. A first schema might include:
tasks(id, role, priority, state, payload, created_at, updated_at)
claims(id, task_id, agent_id, heartbeat_at, claimed_at)
findings(id, fingerprint, state, verdict, severity, title, summary)
evidence(id, finding_id, kind, path, symbol, line, note)
coverage(id, goal, area, technique, state, last_attempt_at)
agent_sessions(id, role, started_at, ended_at, token_count, cost_estimate)The exact schema is not sacred. The contracts are.
The work queue needs atomic claims. Two agents cannot claim the same task. Claims need to expire only when their holder stops heartbeating, not because an arbitrary timeout was reached. A task that repeatedly fails should become blocked instead of cycling forever.
The finding store needs stable fingerprints. Foundry recommends identity based on normalized path, symbol, and vulnerability class, not line number. That matters because line numbers drift constantly. If the fingerprint includes line numbers, every re-run after a refactor becomes a fake wave of new findings.
The store also needs atomic persistence. No delete-then-write. No half-written state. No shared text file where concurrent agents politely hope nothing races.
If this sounds less exciting than “autonomous AI security,” that is because it is the part that decides whether the system works.
The Evidence Gate
The evidence gate is the core of the whole thing.
In Foundry’s model, a true positive needs evidence for:
- Reachability: how execution gets to the vulnerable code.
- Trust boundary: where untrusted or lower-trust input crosses into the relevant path.
- Impact: what security property can actually fail.
Those claims need citations to code locations that mechanically resolve. If the model cites a file, symbol, or line that does not exist in the target revision, the finding is demoted. Not “less confident.” Demoted.
That is a clean rule, and it is worth keeping even in a toy harness.
One narrow exception makes sense: some vulnerability classes are “presence is the vulnerability.” Hard-coded credentials, committed secrets, deprecated crypto primitives, and similar findings do not always need the same trust-boundary path. But even there, the finding should cite the build-relevant file and the concrete impact.
The goal is not to make the model timid. The goal is to make the system honest.
Candidate, Verdict, Confirmed, Published
A good harness needs a finding lifecycle with sharp words.
The Detector creates a candidate. The Triager assigns a verdict: true-positive, false-positive, needs-review, not-applicable, or code-quality. A true-positive is not just a candidate with confident language; it is a candidate that passed the evidence gate.
If a safe testbed exists, the Validator can independently reproduce the claimed impact and mark the confirmed finding as exploited. If there is no testbed, the finding may still be a real vulnerability, but it should not be labeled exploited.
Only after those gates should the Reporter produce the human-facing artifact: a published report, issue, ticket, or rollup.
That discipline protects reviewer trust. Once “exploited” means “the model seemed very sure,” the label is dead.
The Sandbox Cannot Be a Prompt
Foundry’s sandbox principle is blunt: network and filesystem limits must be enforced by infrastructure, not instructions.
That is not paranoia. These agents read untrusted source code, documentation, test responses, comments, and logs. Any of those can contain instructions. If the boundary lives only in a system prompt, the boundary is negotiable. A safety boundary that can be negotiated by text is not a safety boundary.
For a small implementation, that means:
- Mount target source read-only.
- Mount prompts and configuration read-only.
- Provide one writable artifact directory.
- Allow network egress only to the model provider, issue tracker if used, and testbed if configured.
- Block everything else at the container, VM, firewall, network policy, or sidecar layer.
- Test the boundary from inside the sandbox before trusting it.
If an agent can curl a random external host, write to the target source, or reach production through the testbed, the sandbox is not done.
The CodeGuard Flywheel
Foundry pairs naturally with Project CodeGuard, an AI model-agnostic security skills and rules framework maintained by the Coalition for Secure AI, which is an OASIS Open Project.
The important part is not just that the Detector can run rules. The important part is the loop:
- Rules sweep the codebase and find known classes.
- Exploratory agents hunt for target-specific issues.
- Triager confirms an exploratory finding.
- The system records that no existing rule caught it.
- A human generalizes the gap into a reusable rule.
- The next evaluation catches that class earlier.
- The same rule can also guide coding agents during development.
That is the detection-to-prevention flywheel.
This is where Foundry gets more durable than a one-off AI scan. A confirmed exploratory finding should not just become a ticket. It should become institutional memory. The bug class moves from “the model found it once” to “our system now looks for this every time.”
Human judgment still matters. Generalizing from one confirmed bug to a reusable rule means deciding what made the finding representative, what would create false positives, and where the rule belongs in the corpus. That is engineering work, not automatic wisdom.
What to Build First
If I were building a Foundry-style harness for a small technical team, I would start here.
1. Repo target loader
Accept a Git URL or local path, pin the commit, and store a scope file. Do not scan “whatever is currently on disk” without recording the revision.
2. Goals document
Require a short file that says what matters: auth bypass, secret exposure, unsafe deserialization, SSRF, privilege escalation, supply-chain risk, or whatever is relevant to the target. Empty goals should block the run.
3. Basic indexer
Start simple. File tree, language detection, imports, symbols, function boundaries, and call references where available. Tree-sitter, language servers, Semgrep metadata, or language-specific parsers are enough for a useful first pass. LLM extraction can enrich the index, but it should not be the only source of truth.
4. Cartographer template
Generate an architecture map, trust-boundary list, sensitive assets, entry points, and data-flow notes. Store it as Markdown and let later roles cite it as context, not as truth.
5. Candidate-only detector
Make the Detector write only to the internal finding store. No GitHub issues. No Jira tickets. No Slack alerts. It should be cheap to create candidates because most candidates should die quietly.
6. Evidence-gated triager
Force every promoted finding through the same checklist: reachable path, trust boundary, impact, resolving citations, and whether a testbed is available.
7. Reporter that refuses weak output
The Reporter should not launder uncertainty into polished prose. If a finding is needs-review, say that internally. Do not publish it as a vulnerability just because it would make the report look busy.
What Not to Do
Do not let Detectors publish issues. That is the fastest path to making developers ignore your tool.
Do not use line numbers as finding identity. You will create duplicates on every commit.
Do not call something exploited without a live, authorized reproduction.
Do not treat the sandbox as a prompt convention. Build the boundary below the agent process.
Do not start with auto-remediation. It is tempting, but patch generation before reliable detection and triage is theater with a pull request button.
And do not confuse “the model found something” with “the organization learned something.” The learning happens when the system turns confirmed findings into better rules, better coverage, and better development-time guardrails.
Bottom Line
Foundry matters because it focuses on the machinery around the model: roles, evidence, substrate contracts, sandboxing, coverage, budget, and reporting discipline.
That is less glamorous than a demo where an agent yells “critical vulnerability” after three prompts. It is also much closer to how security work survives contact with real repositories, real reviewers, and real risk.
If you build the harness correctly, the model becomes a useful member of the security pipeline.
Without the harness, it is just a noise generator with a large token budget.
Sources
- CiscoDevNet, Foundry Security Spec repository
- CiscoDevNet, Foundry
spec.md - CiscoDevNet, Foundry
constitution.md - Cisco Blogs, Announcing Foundry Security Spec
- Project CodeGuard, Project CodeGuard documentation
- CoSAI/OASIS, Project CodeGuard repository