Engineering Case Study
AgentForge
Governed Autonomous Software Engineering Platform
A LangGraph-orchestrated multi-agent system (Planner, Architect, Researcher, Developer, Reviewer, QA, Security) where no agent's output is trusted by default. Every change a real LLM agent proposes has to pass a deterministic Verification Gate (real type-checking, linting, test execution, static security scanning) and a risk-based Policy Engine before it can merge, and neither of those two gates contains a single LLM call.
Core Principle
LLMs propose and reason. Deterministic systems verify.
AgentForge is a multi-agent system where Planner, Architect, Researcher, Developer, and Reviewer are real LLM calls, but none of them, including the Reviewer, gets the final word on whether a change ships. QA and Security run deterministic checks with no LLM involved at all, and a separate Verification Gate plus a risk-based Policy Engine make the actual pass/fail and merge/hold-for-approval decisions, enforced server-side so a frontend can never quietly bypass either one.
All 12 planned phases complete and verified against a real running stack: real LLM calls through a provider-agnostic Anthropic/OpenAI abstraction, real git operations, real GitHub PR creation, a real Next.js dashboard, and four failure-injection demos that prove the deterministic gate overrides an incorrect simulated approval. Not asserted; run and observed.
Architecture
How a change moves through the pipeline
- 1Planner, Architect, Researcher: real LLM calls that turn a requirement into a plan, a design, and grounded research
- 2Developer: a real LLM call that writes real file changes and commits them to a real git branch
- 3Reviewer: a real LLM call producing structured findings, fed back to the Developer on a high-severity finding
- 4QA: no LLM. Runs the project's real test suite via subprocess; a failed test sends control back to the Developer regardless of the Reviewer's approval
- 5Security: no LLM. A real regex scan for secret-shaped strings and risky constructs
- 6Verification Gate, then Policy Engine: no LLM in either. Real type-check/lint/test/security results feed a fixed rule set, then a real risk classification decides auto-merge, block, or human approval
Why it's built this way
A single LLM judgment, however well-prompted, can miss a bug, hallucinate a false "looks good," or be inconsistent between two runs of the same input. Putting the Verification Gate and Policy Engine in a separate module with zero LLM calls means the same inputs always produce the same decision, independent of any model, and makes that decision independently testable without ever needing a real API key.
Verified, Not Assumed
Engineering Highlights
- Core principle enforced in code, not just claimed: the Reviewer's approval is one more artifact in shared state, never the final decision. A separate Verification Gate module contains zero LLM calls (ADR-003).
- A real LangGraph StateGraph with conditional retry edges: a failed real test or a high-severity Reviewer finding sends control back to the Developer, capped at MAX_AGENT_ITERATIONS so nothing loops forever.
- Postgres-backed checkpointing (AsyncPostgresSaver) keyed by run_id, real workflow timeouts via asyncio.wait_for, and real per-run cost-budget enforcement computed from actual provider token usage, not estimated.
- Risk-based policy engine (ADR-004) classifies real changed file paths and real diff content: a database migration is high risk, a docs-only change is low risk. Enforced server-side so a frontend can never bypass it by simply not rendering an approval prompt.
- QA and Security agents are deterministic, not LLM calls: QA runs the real project's test suite via subprocess, Security runs a real regex scan for secret-shaped strings and risky constructs.
- Failure-injection demos construct real disposable git repos with a genuinely broken function, a genuinely hardcoded AWS-key-shaped secret, and a genuine Alembic migration file, then run the real deterministic pipeline against each and check the actual outcome.
- Real observability: structured JSON logs, OpenTelemetry spans per agent node, and Prometheus-format metrics computed live from the database at request time, never a stale counter.
- An evaluation harness and an n8n/Slack webhook integration both reuse the exact same orchestration entrypoint a live API-triggered run uses. No separate 'demo mode' code path anywhere in the system.
Real Engineering Bug
The logging call that hid the real crash
A logging call silently swallowed the real crash, hanging every run forever
2026-09 (Phase 8)Test failed
Every orchestration node was wrapped with a new instrumentation decorator to record a real span, a structured log line, and a database ToolCall row per execution. Immediately after, timeout-sensitive orchestration tests started failing. Runs that used to finish were now hanging at status="running" until the test's own timeout gave up.
Initial assumption
The first suspicion was a deadlock in the new database write inside the decorator (an extra session/commit added to every node), since that was the most recently added I/O.
Investigation
Instrumenting further and re-running showed the wrapped node's own real exception was never reaching the orchestrator at all. Something inside the decorator itself was raising a second, different exception in its `finally` block, and that second exception was the one actually propagating (and being silently absorbed by the graph's own error handling), not the original one.
Root cause
structlog's bound logger consumes its first positional argument as the log call's own `event` name. The decorator's log call passed the agent name positionally *and* an explicit `event=...` keyword for a different purpose, producing `TypeError: got multiple values for argument 'event'`, inside a `finally` block, which meant this new TypeError replaced whatever real exception the node itself had raised, and the caller only ever saw the logging bug, never the underlying failure.
Fix
Removed the conflicting `event=` keyword from every structlog call site that also relied on the positional event name; introduced `event_name=` as the keyword to use whenever a log line needs to name an application-level event distinct from the log call's own event string. The same class of bug recurred narrowly in a later phase's webhook notifier and was caught immediately by the test suite, before it ever reached a live run, using the exact same fix.
Verification
Re-ran the full orchestration test suite after the fix: every timeout-sensitive test that had started failing passed again, and a run's real terminal status (completed/blocked/awaiting_approval) was reachable again instead of hanging at "running" indefinitely.
Engineering lesson
A `finally` block that itself can throw is a place where the *original* exception can silently disappear. The second exception wins, and nothing about that is visible from the call site. Caught here not by code review but because a downstream test suite's behavior changed the moment this was wired in; the fix generalizes to any logging/tracing wrapper added around existing error-handling paths.
4 Architecture Decision Records
Architecture Decisions
ADR-001
LangGraph for Agent Orchestration
A real LangGraph StateGraph, not a hand-rolled loop, with Postgres checkpointing, conditional retry edges, an iteration cap, timeout, and cancellation, all real and tested from Phase 3 onward.
ADR-002
Postgres for Shared Project State
Postgres, not in-memory state or Kafka, as the system of record for everything that must survive a restart and be queried relationally. Every agent artifact (plans, diffs, findings, results, approvals) is a normalized row.
ADR-003
Verification Gate Is Deterministic, Separate From LLM Review
The Reviewer's output is one more artifact in shared state, never a decision. A separate Verification Gate module contains zero LLM calls and makes the actual pass/fail call, the project's central design principle.
ADR-004
Risk-Based Human-in-the-Loop, Enforced in the Backend
Every action is classified by real risk level and the policy is enforced as backend logic, not a UI convention. A frontend that doesn't render an approval prompt cannot bypass a high-risk classification.
Proof, Not Assertion
Failure-Injection Demos
The project's single most important claim, that the LLM is not the final source of truth, needed to be proven, not just documented. Four failure-injection scenarios construct real disposable git repos and run the real deterministic pipeline against genuinely broken/risky content, with the Reviewer's approval explicitly simulated and labeled as such (never hidden).
make demo-failure: real outcomes
MeasuredEach scenario commits real content to a real git repo (a subtraction bug where addition was required, a hardcoded AWS-key-shaped string, a real Alembic migration file) on a disposable branch, then runs the actual qa_node, security_node, verification_node, and policy_node in sequence, the same deterministic tail LangGraph itself would run, and checks the real resulting decision.
| Scenario | Real trigger | Actual outcome |
|---|---|---|
| QA overrides a simulated Reviewer approval | A genuinely broken add() | BLOCKED_BY_VERIFICATION |
| Security blocks a committed secret | A real AWS-key-shaped string | BLOCKED_BY_VERIFICATION |
| Migration requires human approval | A real Alembic migration file | PENDING_HUMAN_APPROVAL |
Validates: A real test failure and a real detected secret genuinely override an approval, simulated or not, before the change can merge, and a real migration-path change is genuinely classified high-risk and routed to a human, server-side, every time this was run in this environment.
Does not validate:
- — Scenario 1 (the Reviewer itself genuinely catching the bug via a real LLM call) is gated on real Anthropic/OpenAI credentials. It correctly reports "Integration unavailable" rather than fake a result when no key is configured.
- — The Developer step in scenarios 2-4 is a scripted commit standing in for "an agent just wrote this," not a real LLM-authored change. Only the downstream deterministic checks are being demonstrated here.
Numbers That Are Real
Measured Metrics
Backend test suite
Measured143 passed, 1 skipped (credential-gated), 0 failed
Static analysis
Measuredruff + mypy clean across 67 source files
Planned phases delivered
Measured12/12
Architecture Decision Records
Measured4 ADRs, all accepted
Security scan run against this repo's own source
Measured1 match found, the deliberate demo secret; 0 real secrets
Failure-injection scenarios, fully real outcome
Measured3 of 4 (the 4th needs real LLM credentials to run)
Technology