Spec Pipeline Roadmap

How the roadmap is cut

Riskiest judgment firstRequirements analysis is the step most likely to be wrong and the cheapest to measure, since seeded recall needs no human. So it goes first.
Every milestone ends in an evalExit criteria are suite results from evals/thresholds.yaml, not "feature complete".
Humans are budgetedBA and architect time goes to labelling and approvals, and it is named in each milestone so it can be planned.
Decide when first neededOpen decisions carry the milestone that forces them. Until then they stay open.

Sequence

The critical path runs M1 → M3 → M4 → M5 → M6. M2, the judge, runs alongside M1 because the requirements phase can't gate on a judge nobody has calibrated.

Scroll sideways on narrow screens.

Layers by milestone

The platform is built in layers (decision D23, proposed; see architecture and Discussion #21). Each milestone adds a slice of each layer it needs, never a whole layer at once.

Layer	Now (M1)	Next (M2–M4)	Later (M5 on)
UI thin client, talks only to the control plane	None. The CLI and PRs stand in	M3: PR comments plus a chat CLI	M5: UI v1, where buttons map to PR actions
Control governance, registries, API	`pipeline.lock.yaml` as the registry; litellm gateway	M3: command API (`/answer`, `/revise`…), CODEOWNERS gates	Policy decision point, identity per agent and human
Orchestration reconciler, runner, gates	`step-runner` stands in: clean workspace, verify, repair	M2: calibrated eval gate. M3: reconciler v0 with loop limits	Containers, then microVMs; concurrency and kill switch
Execution · graph engine actions as nodes, transitions and routes as edges	Step order and `route:` values on paper (`workflow.md`)	M3: v0 for the requirements phase. M4: design phase and back-edges	Execution-loop graphs (plan → code → QA)
Execution · loop engine + harness the pi loop inside one action	pi sessions with guards, lint repair, pinned AGENTS.md. Done	M3: pi RPC for the intent chat (`ask_user`)	Shell-command guard, per-run credentials
Execution · skills, tools, MCP	`validate_artifact`, `trace_link`, `term_lookup`, `kb_search`/`kb_get`. Done	M3: `cml_check`. M4: design skills	MCP servers with tool allowlists
Cross-cutting · evidence	Artifact headers, `trace.yaml`, run folders	M3: commit trailers, one PR per phase, tags	Signed commits per agent
Cross-cutting · observability	`run_id` and cost on every call	M3: OpenTelemetry traces (D21)	M5: metrics dashboard, alerts
Knowledge + memory placement open in #21	File-based org KB (D18)	No change	Knowledge graph behind the same `kb_search` interface

Milestones

Effort figures are rough, in focused weeks for two engineers. They're there to size the milestones against each other, not to commit to dates.

M0

Contracts and eval harness

done

Proves the pipeline can be specified precisely enough to verify mechanically and evaluate consistently.

Delivered

Header schema, block grammar, 60+ validation rules, reconciler workflow
Templates, AGENTS.md, sample constitution, SPEC-0001 worked example
Scenario walkthrough (page, Excalidraw, Mermaid)
Eval design, 5 rubrics, judge contract, thresholds, 7 cases
Prototype linter and scorer (Python)

Carried forward

D6, D7 and D8–D13 are still "proposed"
Prototypes are throwaway. The real tools are TypeScript pi extensions
Nothing had run against a real model yet; M1 changed that

M1

One real agent

in progress6 of 7 steps merged

Proves a pi agent can do real requirements analysis inside the contracts, and that we can measure how well it does. No reconciler yet: the agent is driven straight from the pi SDK.

Build

spec-lint in TypeScript: the full rule set, git-aware pins (H3/H4), shipped both as the validate_artifact pi extension and as a CLI. Done
pi extensions: trace_link, term_lookup, and kb_search/kb_get over a file-based KB. Done
req-analysis prompt and skill, AGENTS.md, and the first pipeline.lock.yaml. Done
litellm gateway in front of every call, with traces and cost per run. Done
Cost tiers: a local Ollama model ($0), free OpenRouter models, Anthropic models. In review
Scorer ported to TypeScript and a suite runner for k = 3. Done

Exit criteria

SD-RA-0001 at release thresholds: blocker recall 1.0 on the worst of 3 runs, recall.all ≥ 0.85 on the mean. Open: step 7 runs the suite
One more seeded case written from a real past spec. Open: waiting on the BA
A baseline report committed, with cost and tokens per run. Open: step 7
Lint prototype retired. Open: TypeScript parity is proven on every fixture; retirement waits for the parity review

BA ≈ ½ day: write the second seeded case from a spec they know went wrongDecide: generator model, by the step 7 suite results. KB location is decided (D18)

M2

A judge you can trust

parallel with M1≈ 1–2 weeks + labelling

Proves the eval gate's scores agree with your BA and architect well enough to block on, starting with the requirements rubrics.

Build

Judge pi session: rubric loader, constitution articles turned into criteria, output validated against the schema
Labelling sheet: artifacts stripped of provenance, scored blind with the same anchors
Calibration report: κ per criterion, bias checks, advisory or calibrated status

Exit criteria

JF-REQ-0001 sent back and JF-REQ-0002 passed, 3 runs out of 3
RQ-1, RQ-3 and CONST-ART-2 calibrated (κ ≥ 0.6 and ≥ 0.8 × human ceiling)
Every other criterion recorded as advisory, with its κ

BA + architect ≈ 2 days each: label 30 requirements and 30 requirements-analysis artifactsDecide: judge model (must differ from the generator)

M3

Requirements phase, end to end

planned≈ 4–6 weeks

Proves the reconciler can carry a spec from the user's words to approved requirements with real PRs, real people and the gates in place.

Build

Reconciler v0: state derivation, next-step rules 0–11, branches and PRs, commit trailers, loop limits
Graph engine v0: the requirements phase as a graph, with actions as nodes and transitions and routes as edges. Its split from the reconciler is settled in Discussion #21 first
Comment commands (/answer, /accept-risk, /revise…) and CODEOWNERS gates
Intent agent over pi RPC with ask_user. The first UI is PR comments plus a chat CLI
DDD agent (strategic) with cml_check running ContextMapper in CI
Eval mode with the simulated reviewer

Exit criteria

G-0001 through requirements: ref_coverage ≥ 0.85, must_miss 0
CS-0001 req_stability ≥ 0.75. ADV-0001 passes every assertion
One new real spec, not the refund case, taken to approved requirements with your BA

BA ≈ 1 day: drive a real spec through PRsDecide: git host, one PR per phase vs per spec (D7), stale pins block or warn, graph engine vs reconciler (#21)

M4

Design phase

planned≈ 3–4 weeks

Proves the pipeline produces designs an architect approves, keeps the boundaries intact, and can send problems back to requirements.

Build

Design-analysis, DDD (tactical) and design-generation agents
Back-edges as graph edges: /confirm, blocked-by labels, change-request mode, re-run on stale pins
Boundary check (D9) and CML generators (PlantUML) for review
Design and design-analysis rubrics calibrated with the architect

Exit criteria

SD-DA-0001 at release thresholds, including the rate-limit trap
SPEC-0001 fully regenerated by the pipeline and approved by the architect
One back-edge exercised end to end on a case

Architect ≈ 2–3 days: labelling plus approving real designsDecide: domain/ ownership model (D6)

M5

Pilot with one team

planned≈ 4–6 weeks, mostly calendar

Proves the pipeline helps on real work, with the runtime gate live, a thin UI, and costs within budget.

Build

UI v1: a thin client with the artifact, findings and metrics panes, where buttons map to PR actions
Metrics dashboard over metrics.json and the litellm traces
Monthly drift re-labelling for the judge
Suites grown from pilot misses

Exit criteria

3–5 real specs approved through the pipeline
BA and architect time-to-approve measured and accepted
Cost and wall time within budgets
Pilot retro, with D6–D13 settled

One team: BA, architect, 2–3 requestersDecide: promote the proposed decisions, or change them

M6

Execution handoff and the lagging loop

planned≈ 6–8 weeks

Proves good-scoring specs actually lead to less rework, which closes the measurement loop you asked for at the start.

Build

Plan → code → QA consumes only tagged specs and adds implements/tests links
Lagging signals attributed to IDs: clarification requests, spec-rooted rework, escaped defects
Correlation report between leading metrics and lagging-signal density
Flywheel: every spec-rooted miss proposes a seeded case

Exit criteria

Pilot specs implemented through the loop
First evidence of which leading metrics predict rework, and which ones to drop
At least 5 curated cases added from real misses

QA lead: root-cause tagging (spec vs code)Decide: execution harness shape (same reconciler or a separate loop)

Decisions and when they're due

Each one has a current lean and the milestone that forces it.

Decision	Due	Options	Current lean
Generator model(s) per step	M1	One model everywhere, or per step (a stronger model for analysis)	Per step, all through litellm. A provisional pick is recorded (D19); step 7 chooses by suite results
KB location	M1	Files in a KB repo · Typegraph · both	Decided (D18): plain Markdown files; Typegraph can sit behind the same `kb_search` interface later
Judge model	M2	Any model outside the generator's family	Pick by calibration κ, not by leaderboard
Git host	M3	GitHub · GitLab · self-hosted	Whatever your teams already review in
PR granularity (D7)	M3	One PR per phase · one per spec with labels	Per phase, for clean approvals and history
Stale pins (H4)	M3	Block review · warn only	Warn during M3 and block from the pilot on
Graph engine vs reconciler (D23, Discussion #21)	M3	The graph is a pinned definition the reconciler walks · the graph engine drives and orchestration only schedules and runs	Open
LLM gateway placement (D23, #21)	M3	Control-plane shared service · execution infrastructure	Control plane: it holds budgets and routing policy
Knowledge and memory placement (D23, #21)	Before the KG	Cross-cutting · execution · split (store cross-cutting, access through execution tools)	Open; the split is on the table

Risks to watch

Risk	Early signal	Response
Labelling time for BA and architect runs out	M2 labelling slips past a week	Calibrate only hard-fail criteria first. Everything else stays advisory
Contracts feel heavy to the BA	Time-to-approve rises and `/revise` gets used for format issues	Tune rules to warnings, and let the agent absorb the format while humans judge content
ContextMapper toolchain friction in CI	Java or CLI issues block `cml_check`	Containerise it once in M3, and fall back to a parse-only check
Model updates shift behaviour	The baseline report moves with no lock change	Pin model versions in the lock and re-run the suites on any provider update
`domain/` merge conflicts across specs	Escalations from concurrent glossary edits	Serialise domain changes through the architect, and consider domain-only PRs

Later: the platform horizon

These come from the wider platform vision and follow once the spec pipeline is proven. Each gets its own contracts-first pass.

The full control layerPolicy decision point, permissions, identity and the agent registry, above the orchestration layer.
Knowledge graph and memoryTypegraph as the trace store: trace.yaml stays the source, synced into the graph for querying intent-to-code links. Where it sits in the layers is open in #21.
Product loopThe long-running agent that watches requirements, design and code for drift and flags it to a human.
Reverse engineeringCode → spec, using the same contracts, so existing systems can join the pipeline.
Live architecture viewsC4 and deployment diagrams generated from CML and design artifacts.
Enterprise levelDeferred by choice until system and architecture levels are proven.

Effort figures assume two engineers who already know pi and TypeScript. Treat them as relative sizes. Real dates come after M1, once there is a baseline.