Module 6: Capstone: Implement Foundry-lite#

The capstone: build the machine that does the auditing. Using spec-kit and pi, you implement Foundry-lite — a sanctioned subset of Foundry, Cisco’s open specification for an agentic security-evaluation system — and run your implementation against the Lab 5 vulnbox to find real vulnerabilities. The cut is not a compromise the class made behind the spec’s back: the spec’s §4.2 explicitly asks which roles you will merge, split, or omit, and answering that question through the clarify step is the spec-kit lesson. This module ends the class with a protected 30-minute block: the room walk comparing what each implementation caught, the class debrief, and a short closing lecture: fine-tuning vs. context engineering, taking agentic workflows to production, and what to build first back at your desk.

Questions this module answers#

• What is the Foundry security spec, and what problem does it solve?
• What does spec-driven development change about building with agents?
• Why does a trustworthy evaluation system need roles and quality gates instead of one “find bugs” prompt?
• How do I cut a seed spec down to my own situation — and where does the spec itself tell me I may?
• Why does a mechanical citation check beat asking the model to be careful?
• When should I fine-tune versus prompt and context-engineer?
• What does it take to run agentic workflows in production — cost, logging, evaluation?
• What should I build first when I’m back at my desk?

Slides#

Lab 6.1 (Capstone): Implement Foundry-lite#

Goal#

Implement Foundry-lite — the full Foundry value chain, run sequentially, with every quality gate intact — using spec-kit and pi, then point it at the Lab 5 vulnbox and make it produce a finding it can prove. This is the post-lunch block and the rest of the day belongs to it; it’s also the Day 2 sponge, so the time box flexes with how the day has gone.

Success criterion: the spec’s own SC-001, verbatim: “On a target with at least one known seeded vulnerability, an end-to-end run produces a published true-positive finding for it with evidence satisfying §7.3, with no operator intervention between up and publication.” Your seeded target is the vulnbox — you hold this morning’s flags as ground truth.

The spec, in one paragraph#

Foundry distills Cisco ASIG’s internal agentic security-evaluation systems into an organization-neutral specification: eight core agent roles (indexer, cartographer, detector, triager, validator, reporter, and friends — each catching the previous role’s failure mode), a finding lifecycle with an evidence gate, a coordination substrate, and ~130 functional requirements, plus a constitution of eleven inviolable principles, each encoding a production failure. There is deliberately no code in the repo: the spec is the deliverable, and it’s written to be consumed by GitHub’s spec-kit — you clarify its open questions against your environment, then plan, task, and implement with your agent. Your environment is this classroom, and it demands a much smaller system than Cisco’s.

The cut: Foundry-lite#

Five roles, run in order, one process:

flowchart LR
    M["Cartographer-lite<br/>security map"] --> D["Detector<br/>rule sweep + exploratory"]
    D -->|"candidates<br/>(never surfaced)"| T["Triager<br/>three-leg evidence gate"]
    T -->|"true-positive"| V["Validator-lite<br/>live PoC vs. vulnbox"]
    T -.->|"citation fails to resolve:<br/>auto-demote"| N["needs-review<br/>(stays internal)"]
    V -->|"exploited?"| R["Reporter-lite<br/>vuln-report skill → markdown"]

• Cartographer-lite — one security-map doc: endpoints, auth model, trust boundaries (FR-031/FR-032). Kept because the spec’s §7.3 rationale says the trust-boundary map is what makes the evidence gate tractable.
• Detector — rule sweep (FR-037) over the class rule corpus (FR-041, provided) plus exploratory hunting (FR-040). Candidates carry location, class, and reasoning (FR-043) and go to the finding store only — never to a human (FR-044).
• Triager — implemented in full: verdicts (FR-050), real investigation (FR-051), the three-leg evidence gate — reachability, trust-boundary crossing, concrete impact (FR-052/FR-087) — with mechanical citation resolution and auto-demotion to needs-review (FR-088), the honesty rule (FR-053) — the production version of Lab 5’s “no flag, no finding”: that was a CTF discipline, while real-world reporting keeps credible-but-unproven findings alive as needs-review with the confidence stated — and reasoning recorded (FR-054).
• Validator-lite — reproduce each true-positive’s headline impact with a runnable curl PoC against your live instance; only this role sets exploited (FR-089). For the vulnbox run this is required, not optional: at least one published finding must carry exploited from your Validator’s own live PoC. (On the second target — your own ShareBox — exploited is a stretch.)
• Reporter-lite — render true-positives, and only true-positives (FR-057), to markdown through your own Lab 2.3 vuln-report skill.

Dropped, with the spec’s blessing (each is an answer to a [NEEDS CLARIFICATION] it asks): the Orchestrator (a ~20-line sequential runner — the throttled proxy forbids fleets anyway), the Indexer (the target fits in context), the Coverage-Guide (the timebox is the done signal), dependency/secret scanning, the entire §8 substrate, all five extension roles, observability, integrations. The finding store is a directory of JSON files with the lifecycle candidate → triaged → validated → published, keeping the FR-090 fingerprint (hash of path + symbol + class) so a second run files nothing twice.

What no cut may touch: the gates. Merging and omitting roles is sanctioned; dropping a responsibility is not.

Provided#

• The Foundry spec repo, mirrored on your instance at ~/foundry-security-spec (/home/ubuntu/foundry-security-spec): README.md, spec.md, constitution.md, and docs/ — read the README’s “Getting started” section carefully; it is the lab manual
• spec-kit, preinstalled on your instance
• The starter detection-rule corpus: labs/06-capstone/rules/rules.md — eight rules derived from the Lab 5 checklist, versioned per FR-041
• The target: the Lab 5 vulnbox source (labs/05-audit/vulnbox/), plus your hosted instance for the Validator (address on your student portal page, or aws s3 cp s3://$CLASS_BUCKET/portal/$CLASS_STUDENT_ID.md -)
• The classroom clarification key: labs/06-capstone/clarifications.md — handed out after your own clarify pass
• Two helpers in labs/06-capstone/provided/ so your 65-minute build goes into the pipeline, not plumbing: llm.py (a chat-completions client that survives 429s and connection resets) and citation_gate.py (the FR-088 mechanical citation resolver). Read them — the gate especially is the point of the module.
• Throttle headroom: the instructor raises the class proxy’s per-student rate limit for this block if it was lowered earlier. Implementing a system like this is token-hungry and that’s expected; your spend cap still applies.

One constraint to design around from the start: the throttle counts estimated prompt tokens + your max_tokens, per request. Whole-file prompts will never fit — chunk your sweeps by function — and a max_tokens set too low silently truncates replies mid-JSON, which is the most confusing failure you can build for yourself.

Ground rules#

Build lab: attack nothing, with one exception — your Validator may reproduce findings against your own hosted vulnbox instance. No parallel agent fleets: the proxy is throttled per key, so run your stages sequentially and back off when it pushes back. Findings stay on your instance and are never published anywhere; “published” in the lifecycle means “written to your local findings directory”. Full text: labs/06-capstone/rules-of-engagement.md.

Steps#

1. Read the constitution end to end (it’s short), then the README’s getting-started flow. Set up spec-kit in a fresh project — the exact command is specify init foundry-lite --integration pi — then install the Foundry constitution and seed spec.md per the README (the mirror is at ~/foundry-security-spec).

2. The clarify pass: run /speckit.clarify and answer the markers yourself, cutting toward the Foundry-lite shape above. This is the exercise: the spec’s §4.2 asks you which roles to merge or omit, and each role’s section prices the omission. Then /speckit.specify, and clarify again if new markers appear — do not carry a [NEEDS CLARIFICATION] into planning. Compare against the classroom key when it’s handed out.

   Hint 1: how to answer ~43 markers in fifteen minutes (step 2)

   Answer only the markers the classroom key covers — roles, pipeline shape, finding store, testbed — and bulk-accept the suggested defaults for everything else. Most markers are integration questions with one-line classroom answers anyway: one VM, sequential runner, JSON files on disk, local git, no issue tracker, class proxy as the only provider, no extensions. Spend your thinking on the two that matter: the §4.2 role cut (keep the value chain, drop the scale) and the §5.6/§11.12 testbed question — you have a live testbed, your own vulnbox instance, which is what makes the Validator and the exploited flag worth building.

3. Build: /speckit.plan, /speckit.tasks, /speckit.implement. Supervise the way Module 2 taught you: watch the loop, interrupt early, keep tasks small. If the build eats the run window, cut breadth (fewer rules swept, a thinner map) — never a gate.

   While the agent works — the implement step is long; don’t watch it scroll:

   • Draft your security map by hand. You’ll know instantly whether the Cartographer’s is shallow.
   • Pre-write PoC skeletons (curl one-liners) for the vulns you remember from Lab 5 — your Validator will want them.
   • Read provided/citation_gate.py; the gate is the point of the module.
   • Plan your custom feature (see “Stretch: make it yours” below).
   • Open a second pi session and build yourself a small personal tool or extension — the Module 2 muscle — or take on the Module 2 bonus lab (the edit-context extension) if you never got to it.

   Hint 2: the mechanical citation check is small (step 3)

   FR-088 sounds heavyweight and isn’t: for each cited location (file, function, line-or-symbol), check the file exists, the symbol is defined in it, and the quoted line actually occurs. Thirty lines of Python, no LLM involved. Wire it to run after the Triager writes its verdict and before the verdict is accepted into the store — demote to needs-review on any failure. The point of the lab is watching that demotion fire on a confident, fluent, fabricated finding.

4. Run: the vulnbox first — known ground truth, and the target SC-001 is scored on. Point it at the vulnbox source; give the Validator your hosted instance’s address. One command, no hand-feeding: candidates accumulate, verdicts land, the gate demotes, the Validator fires curls, reports render through your vuln-report skill.

5. Run it a second time on the unchanged target: zero duplicate findings, or your FR-090 fingerprint has a bug.

6. Then your own ShareBox. Point the same pipeline at your frozen Day-1 app — the one you shipped yesterday. This is a core step, not a stretch: Day 1’s output is Day 2’s target, and the question is what your own build got wrong. The contract: the source is at ~/work/sharebox (where Module 3 told you to build it); for the Validator, run the app locally on your VM on port 8080 and point PoCs at http://127.0.0.1:8080 (same own-instance rule as the vulnbox). Getting exploited set on this target is a stretch, not a requirement.

7. Score it against your Lab 5 ground truth and bring the numbers to the room walk: caught, missed, hallucinated (split into “the gate demoted it” and “it got through and died against the live instance”), any rule-gap — a confirmed exploratory finding no rule in the corpus would have produced (FR-042) — what it found in your own ShareBox, and how your three Lab 3.1 threat-model predictions held up against both runs.

Done when#

An end-to-end run publishes at least one true-positive on the vulnbox with gate-satisfying evidence and no operator intervention — SC-001, on a target you personally proved is vulnerable this morning — and at least one published vulnbox finding carries the exploited flag, set by a live PoC your Validator ran itself against your hosted instance.

Stretch rungs, in order: (1) the exploited flag on a finding in your own ShareBox run (your app running locally on your VM, port 8080); (2) a peer’s ShareBox — trade source with another student and audit each other’s (source only; attack nothing); (3) one custom feature, below.

Stretch: make it yours#

After SC-001, extend your implementation with one feature of your choosing. Suggestions, spanning difficulty:

• A new detection rule written from something you found in Lab 5 that the corpus misses — closes the FR-042 rule-gap loop by hand.
• Variant-Hunter-lite: take one confirmed finding and hunt the same pattern everywhere else in the target (spec §6.2).
• Severity scoring: wire your Lab 2.3 cvss.py into the Triager or Reporter so findings carry a score.
• Re-run diff mode: use the FR-090 fingerprints to report what’s new since the last run.
• Reporter upgrade: a one-page rollup across all findings, not one file per finding.
• A second exploratory pass with a different persona (hostile insider vs. external attacker); compare the candidate sets.

Whatever you build, show it in the room walk.

Then the class debrief — the room-wide caught/missed/hallucinated table, plus what Foundry-lite found in the room’s own apps — and the closing lecture: fine-tuning vs. context engineering (almost always the latter; LoRA in one slide), what it takes to run agentic workflows in production — cost, logging, evaluation at scale — and what to build first when you’re back at your desk.