Technical Deep Dive

Weaver Harness: The System Runs Continuously; Humans Judge Asynchronously

§1 — Core Definition & Approach

Weaver is an orchestration layer wrapped around coding agents (claude -p / codex exec). The main chain has only three objects: Spec → Task Graph → Node Loop. Spec defines what counts as done. Task Graph persists work and dependencies. Node Loop takes each node through execution, reflection, and adversarial audit, followed by adversarial acceptance from an independent task on the graph. Sessions may end; the project does not lose its memory with them.

The problem it targets: a single session cannot carry a large project to completion. The issue is not just context length. A large project is too wide, in both experience and process. A commercial app typically spans demand discovery, market research, product design, implementation, launch, and operations. These stages have different inputs, different standards of judgment, and different failure modes, and they keep rewriting one another over days or even weeks. Compressing the whole chain into one conversation means betting the project's state on a short-lived process.

The system splits into two layers. The code control layer stays restrained: a thin wrapper over the coding agent, a DAG state machine, hard reliability constraints, and a continuously polling daemon. Its job is to keep work flowing, keep state from being lost, and keep obvious failures from being silently swallowed. The prompt asset layer carries orchestration practice, role postures, experience references, and infrastructure knowledge. The former handles mechanically decidable facts; the latter handles work that requires model judgment.

The expected input is a goal plus available resources; the expected output is a shippable product. Humans never enter the execution loop. They do two things, both asynchronously: input goals, and judge output. Feedback is rewritten by the Planner into new nodes, dependencies, or acceptance requirements, then enters the same graph. The machine keeps running; humans judge direction at their own pace.

§2 — Core Components & Architecture

Figure 1 draws the system boundary. The code layer does not try to become a second agent. It stores state, selects runnable nodes, assembles context, launches the underlying agent, and routes results back to the acceptance surface. The real work is still done by models. The division of labor is close to the orchestrator-workers pattern^[1], except that what gets scheduled is not workers within a single request but a persistent task graph spanning long stretches of time.

Task Graph: moving project state out of the session

Weaver picks Session = Task as its execution granularity. Each task maps to one agent session, whose identity is kept in task.agent_session_id. The first run opens a new session; subsequent reflection and revision resume the same one. Context belongs to the node, dependencies belong to the graph, and neither gets mixed into a single endlessly growing conversation.

The DAG recognizes only explicit state. An upstream node in depends_on must reach DONE before the downstream unlocks. Runnable nodes are ranked by how many successors they would unlock, with Planner meta-tasks first. A failed session therefore affects exactly one node. After the process exits, the graph, the dependencies, and the acceptance pointers are still on disk.

Daemon: running continuously is not one very long call

The daemon's key property is asynchrony. It does not wait for one agent to fully finish before the system resumes living; it keeps polling the task graph and dispatches runnable nodes to child processes. Children write back incrementally over stream-json, while verification and archiving advance along a separate path. Human feedback never needs to preempt the current session either: rewrite the graph, and the next poll sees the new frontier.

Long-running operation rests on a set of unclever but necessary recovery rules. When continueExpiredInProgressTasks finds an IN_PROGRESS node with no live child process and an expired deadline, it flips the node back to TODO and increments failure_count. The overall task timeout defaults to 1440 minutes. Ten minutes of output-stream silence routes into deferTaskForRetry or deferOrEscalate. Transient errors retry up to 10 times with backoff; five accumulated failures escalate to NEEDS_HUMAN.

The scheduling watchdog uses a conjunction of conditions to avoid mistaking slow for dead: only when a lock has been held for over 30 minutes, the log has been silent for over 5 minutes, and no live child process exists does it send a NEEDS_HUMAN inbox notification. It detects anomalies; it does not substitute for judgment.

Telling a genuine dead loop apart from silent-but-working is delegated to dead-loop arbitration, whose principle is arbitrate before kill. When a stream-silence timeout fires, the system does not kill the process first: an independent session reads the tail of the transcript and makes the call. A WORKING verdict extends the window in place — the process keeps running and no retry budget is consumed. Only DEAD_LOOP or HUNG leads to terminate-and-restart. A mechanical failure of the arbiter itself is recorded as ARBITER_ERROR and takes the traditional restart path, without polluting the statistics. Each task gets at most two arbitrations, after which it escalates stably to NEEDS_HUMAN. The determination is JUDGMENT, so it goes to a model reading context rather than being written as a regex. The mechanism merged to trunk after three rounds of cross-model review and is still in its production-validation period.

Asynchronous collaboration: the human is a dependency on the graph, not an approver in the loop

Humans enter the system through exactly two doors, neither of which interrupts execution. Inputting goals: new requirements and feedback are rewritten by the Planner into nodes, dependencies, or acceptance requirements, effective at the next poll. Judging output: verdicts on delivered results also land back on the graph — as repair tasks, rewired dependencies, or modified SPEC nodes. There is no synchronous approval step anywhere in the system; human opinions are consumed the same way as every other input, as changes to the graph.

The interface in the opposite direction is NEEDS_HUMAN. When a node confirms it is blocked by a genuinely external condition — missing credentials, missing accounts, a product decision only a human can make — it escalates to NEEDS_HUMAN, writes to the inbox, and sends an outbound notification; the gate carries its own reply protocol. The key property is localized blocking: only the downstream that depends on it stops, while the rest of the graph's frontier keeps advancing. The human replies at their own pace, the task flips back to TODO, and the unblocks chain re-opens what follows. Waiting freezes a single dependency edge, not the system.

The status is deliberately narrowed so it cannot become an exit for offloading responsibility. The constitution constrains it to a last resort: when resources fall short, the agent is expected to research alternatives on its own rather than “quietly deliver a suboptimal result, or silently park at NEEDS_HUMAN.” Blockers are handled by type: process-class failures (STALE_PROCESS_BLOCKER) are recovered automatically by the daemon without bothering anyone; only genuinely external blockers (REAL_EXTERNAL_BLOCKER) wait for a human. Every escalation path is mechanical: five accumulated failures, a watchdog conjunction alert, an exhausted arbitration budget. The constitution's title for this whole design is just eight characters in the original: AI self-driven, humans asynchronous.

Node Loop: local self-correction, accountability from independent context

A node starts when runDevScheduler opens its execution session. Then resumeDevScheduler reads continuation.md and runs multiple rounds of reflection inside the same session. Reusing the session here is not about saving tokens; it lets the reviser keep the local context in which the decisions were just made. This corresponds to reflection's classic position^[2].

After reflection, the Devil launches an adversarial audit through runImprovementAgent. Only ACCEPT lets the node complete. REJECT produces a REVISE and returns to the original execution session, for at most devil_review rounds. The final adversarial acceptance is not stuffed into this task. The Planner schedules a downstream audit task on the graph, which inspects the upstream artifact with fresh context. The first three stages correct “I believe I am done”; the fourth tests “does an independent observer agree.”

Prompt: a just-in-time composition of posture, knowledge, and the current acceptance surface

Role definitions stay thin. Across a dozen or so roles — Planner, Coder, Tester, Devil, and others — each role.md answers only three questions: who am I, what is my goal, when do I escalate for help. Standing capabilities come from bound-skills.yaml; task-relevant capabilities are retrieved on demand as searched skills. Process knowledge is not copied into every role.

The buildVars order is fixed: role_context → bound_skills → searched_skills → prd_index → acceptance → task fields. The last segment carries raw_request, title, and notes. The acceptance section live-renders the current spec and spec-state on every build, rather than copying old text into the task. The prompt is therefore not a giant manual — just a temporary composition of role posture, needed knowledge, input data, and the current acceptance surface.

These trade-offs are governed by a more general discipline: “when the root cause is insufficient posture, the cure is stronger posture, not another checklist.” Another sits closer to the implementation boundary: “treat the LLM as a reliable person: design role relationships that naturally check each other (Coder→Tester), instead of piling runtime guards into runner.ts.” The code layer defends only FACTs, state transitions, and unacceptable silent failures. JUDGMENT stays inside role relationships that have context.

The asset layer: a long accretion of orchestration practice, skills, and references

The code layer can be thin; the asset layer cannot. It decides whether the same coding agent can produce non-amateur work in a concrete domain, and it is the more labor-intensive half of the system. Beyond a dozen roles and fifty-plus skills, there is a declarative infrastructure.yaml: it gathers available providers, defaults and alternatives, and twenty-plus provision commands in one place, so the system provisions external services — Supabase, Stripe, OpenRouter, FAL — automatically, instead of scattering keys and setup procedures across prompts.

Below that sits a systematic reference library: design exemplars and brand references, living corpora for naming, hundreds of script and content references, provisioning recipes for each service, and a full body of orchestration practice. This is the raw material of “how to do it well.” Skills and references are not static documents written once; they are continuously added and removed as real projects expose failures — if a skill drags output below the load-nothing baseline, the constitution requires deleting it, not patching it further. The asset layer is thus a long line that keeps converging, not a one-time configuration.

SPEC: not requirements text, but acceptance state that can be invalidated

As the Planner decomposes tasks it writes specRefs pointing at SPEC node ids. The acceptance surface is split into three layers: spec.yaml holds stateless declarations, spec-state holds verification results and evidence snapshots, and tasks hold only pointers. Writing a verdict into spec.yaml fails loudly, because once declarations and conclusions are mixed, an old conclusion can masquerade as a current fact.

# 1) spec.yaml: declares "what must be true"; this file never stores verdicts
nodes:
  - id: core-loop-playable        # stable node ID; tasks and the ledger address by it
    text: agent completes a game   # current acceptance proposition; editing it must invalidate old verdicts
    hardness: HARD                 # HARD participates in the done-gate; SOFT only adds quality signal
    verify: e2e-full-game          # suggested verification; how evidence should be produced
    refs:                          # external ground truth the acceptance depends on; content changes at the same path also force re-verification
      - docs/rules/game.md

# 2) spec-state/<node-id>.yaml: records "how it was judged last time, against what"
verdict: PASS                      # verdict in an independent ledger; never written back into spec.yaml
verdict_by: audit-task-42          # verdict source; keeps responsibility and re-check paths traceable
verified_against:                  # node snapshot at verdict time; CAS prevents verify-while-edit
  text: agent completes a game
  hardness: HARD
  verify: e2e-full-game
refs_fingerprint: sha256:…         # content fingerprint of external references; catches silent same-path swaps

# 3) tasks.yaml: tasks hold pointers only; no copied acceptance text, no verdicts
- id: implement-core-loop
  specRefs:                        # the SPEC nodes this task answers for
    - core-loop-playable

Validity comes from three mechanisms. First, write-path invalidation: the moment a node is edited, its old verdict is cleared. The system does not compare two texts at the delivery gate, because that kind of gate can be bypassed by re-copying the old text. Second, compare-and-swap: a verdict write carries an --expected snapshot of the node; if the node changed during verification, the write fails. Third, refs fingerprints: a design file repainted at its original path still triggers re-verification. Together these guarantee that PASS describes the current proposition and the current references.

Polis R14→R15 exposed the danger of a static acceptance surface. The owner's mandate was “a local agent must win a full Civ match.” It existed only in task prose and one QUEUED spec-write; it never became a live non-PASS node. When reconcile advanced the frontier from spec-state, it saw all 50 HARD nodes on the old mock at PASS, so it cleared the in-flight tasks and declared completion. The acceptance surface was all green, and the core argument had never run once. The problem was not a weak verifier — the system verified the wrong world. A requirement must first become an invalidatable node before the machine can see it.

visual-pairs taught the opposite lesson: not every judgment should be mechanized. That gate matched screenshots pixel-wise against design files. The Coder's optimal response was to ship a static WebView: visually closest, and completely non-interactive. The harder the gate, the more complete the Goodharting. The mechanism was later reverted, and design taste was handed to independent whole-product E2E evaluation. FACTs suit hard gates; JUDGMENT needs full context and adversarial assessment. The constitution puts it more plainly: “silently stamping DONE on broken work and faking a UI violate the same principle.”

Spec Kit and OpenSpec offer adjacent reference points^[3]. They mostly use specs as pre-development human-alignment documents. Weaver cares more about state during unsupervised execution: which proposition is still valid, who issued the verdict, which version the verdict targeted, whether the references have changed. Similar format, different runtime semantics.

Governance: the optimizer must not rewrite the definition of success

lint scans the whole repo for consistency. constitution.md is maintained by humans; it defines what the system counts as better, and evolve has no permission to modify it. The evaluation function and the optimization process must stay separate — otherwise the easiest move for self-improvement is to rewrite success into whatever it already does. The constitution's editing principle is kept deliberately executable: “delete → replace → merge → split → extend; extend is the last resort.”

Ablations: which injections amplify, which suppress

An orchestration layer is neither an inherent gain nor an inherent drag. Adding things onto a coding agent cuts both ways: some injections raise the output, others depress it. The value of several isolated experiments (same brief, blind selection, n>1) is not answering “is the harness good” but separating the two kinds.

Methodology: each row is an isolated comparison on the same brief, blinded with randomized ordering, final judgment by human blind selection (logo n=4 pairwise, content N=5, one set each for naming and the design pipeline); raw artifacts and verdicts are archived. Samples are small — read as directional evidence.

The pattern is consistent. Amplification comes from content: real top-tier references, living cultural corpora, define-then-converge — they feed in the bit of domain specificity the model lacks. Suppression comes from apparatus: multi-stage routines, negation-first framing, coverage ratchets, post-hoc hardening — they trade the model's judgment for the “completeness” of process and pull output back to safe mediocrity. So harness engineering is not about adding less or adding more; it is about telling, for every layer, whether what is added is content or apparatus: content gets retained and thickened; apparatus gets withdrawn the moment the model does better alone. A self-improving system must be able both to accumulate experience and to retract its own complexity.

§3 — References

1. [1] Anthropic. Building Effective Agents. 2024. This essay adopts its workflows / agents distinction and orchestrator-workers as an adjacent architectural reference.
2. [2] Weng, Lilian. LLM Powered Autonomous Agents. Lil’Log, 2023. Planning, reflection, and memory map to the Planner, Node Loop reflection, and file-based state.
3. [3] GitHub. Spec Kit. Fission-AI. OpenSpec. Both serve as comparisons for spec-driven development.

§4 — Outlook & Future Work

The first boundary is evaluation — the meta-bottleneck that almost every “which is better” eventually hits. Without a trustworthy comparative evaluator, self-improvement can only optimize proxy metrics, and most architectural claims cannot be cleanly ablated. The taste layer I envision is not a trained scoring model but a comparative evaluating eye: absolute scores are untrustworthy (model self-evaluation systematically inflates), while pairwise blind selection is easy and stable — and its value lies not only in the ranking but in the transferable reasons it is forced to produce. It naturally rides base-model upgrades, instead of freezing today's judgment into a small self-trained engine.

For this eye to become part of the evolution loop, the experimental design has to be strict enough. Change exactly one thing on the same brief; treat position bias with blinding and randomized ordering; and add a PLACEHOLDER arm — swap the skill for a minimal stub: if it performs no differently from the real skill, what is doing the work is the slot's structure, not its content. The measurement system must also prove itself first: three physically isolated datasets (calibration / per-version benchmark / a never-tuned holdout), with the drift of human judgment itself as the error floor — an automated scorer cannot be required to be steadier than that. My guess for the taste layer's endgame is two forms: the standardized part gets built into general-purpose harnesses, while personalized taste stays in each user's own data layer.

Craft → Harden still needs ablation in the full pipeline. Existing experiments hint that front-loading every constraint pushes generation toward safe solutions. Craft first to define the product, then Harden to fill in state, edge cases, and SPEC — that may be the saner order. The risk is equally clear: Harden may sand off what made Craft distinctive. That some isolated experiments hold does not mean it holds with Weaver in the loop.

Whether roles are necessary is not a constant either. The current system divides responsibility across a dozen roles. Lope, Weaver's minimalist experimental branch, shrinks the loop core to roughly 1,300 lines (loop / decider / facts / adapter / store / assembler / gates, excluding infrastructure hookup) to test a role-free, large-phase, single-writer loop. It has already produced one result: on GDPval's single-artifact document tasks, Lope showed no improvement over bare codex, at higher cost. This negative result is useful — it says the harness's value is not in polishing a single generation prettier, but in the state accumulation and rework of a multi-step project loop; and that deleting structure down to the minimum in one stroke throws away, along with the apparatus, the deposits of experience that were actually carrying load. Contraction needs discipline; it is not a teardown. Which end wins depends, in the end, on whether models can hold goals, judgment, and self-correction over longer spans on their own — role boundaries useful today may become tomorrow's context tax as models grow stronger.

Parallel scheduling remains an engineering to-do. The current single-slot serial default has caused critical-path starvation, including one case of $627 spent for zero output. Parallel slots and critical-path priority are worth building, but they do not raise quality by themselves. Parallelism only shortens the time of a correct graph — and amplifies the cost of a wrong one faster.

The longer-horizon question is whether innovation paths can be abstracted, and therefore orchestrated. The vast majority of commercial apps do not depend on a single inexplicable flash of genius. Innovation usually comes from permutation, combination, and traversal: a new scenario need meets a new technical approach, and a new product space appears. With “scenario need × new technical approach” as the combination axes, the system can already traverse a large candidate space, then converge step by step with resources, constraints, and evaluation. Human history does contain a few leaps that resist reduction to combination — but they are not the normal case a commercial innovation system must depend on.

This is the basis of the long-term vision: seed the system with strategy and resources and let it keep searching the combination space; humans asynchronously supply goals and taste, writing the valuable directions back into the graph. What can be orchestrated is not inspiration itself, but the path that generates, tests, and eliminates candidates.

The final limit is still the model. Evaluation, taste, and long-range coherence do not fully belong to the harness. Standing outside the model, an orchestration layer can only keep adapting to a capability distribution someone else trained, and keeps deleting old mechanisms as models upgrade. The real leverage is not making the harness ever thicker, but letting harness and model co-evolve: the model exposes new capability boundaries, the system turns those boundaries into training signal through real projects and ablations, and the next generation of models simplifies the system in return.

← Back to home