How We Built Charlie, Part 1: Why Dependable Agents Need a Coordination Layer

This post is part of How We Built Charlie, a technical series on the runtime and product decisions behind dependable autonomous engineering work. This installment covers the coordination layer: the durable ownership that keeps work moving across follow-ups, delegation, failures, and verification.

Agents can look capable in one run. The more challenging problem is keeping one engineering task moving when that run ends, new information arrives, or an external system fails halfway through a request.

Consider a flaky integration test. An agent reproduces it and starts a patch. While the work is underway, an engineer adds a missing detail: the failure only appears behind a new feature flag. The agent delegates a focused log investigation and uses the result to revise the fixture. It then asks GitHub to open a pull request. GitHub may have accepted the request, but the response disappears before the PR number is recorded. CI later finds another failing case.

The task is still open. The system must keep the engineer’s follow-up attached to the active work, collect the delegated result, determine whether the pull request already exists, respond to CI, and report evidence for the final state. None of those responsibilities belongs to a model process after it stops.

A coordination layer keeps the task owned across bounded execution, follow-ups, delegation, external effects, and evidence. It gives each boundary a narrower job: the scheduler owns lifecycle state, the executor persists selected progress, communication policy keeps people informed, and operation-specific recovery decides what to do after an uncertain write.

Coordination does not eliminate failure. It also does not provide global exactly-once execution across the scheduler, GitHub, and CI. It gives the agent enough durable state and authority to continue carefully, preserve uncertainty when needed, and avoid declaring success without proof.

---

Coordination is continuity of responsibility

Many agent diagrams show a loop: receive input, call a model, invoke tools, return an answer. That diagram describes one period of execution. It does not say who owns the objective before execution starts, after a process exits, while a follow-up waits, or when an external write has an uncertain outcome.

Engineering objectives routinely cross those boundaries. Tests finish after the model turn that started them. Review comments arrive after a patch exists. Providers sometimes accept a write and lose the response. Delegated investigations complete on a different schedule. A worker can restart with persisted state but without the exact in-memory conditions of the previous process.

A useful conceptual model separates three durable concerns. The task record identifies the owned objective, lifecycle state, lineage, and later input. Persisted execution state records selected progress from one bounded execution, such as its phase, turn, transcript, or pending-tool state. External artifacts and observations provide evidence about effects outside that execution, such as a commit, pull request, provider read-back, test result, or CI run. This is a reasoning model, not a claim that every system implements one universal EffectRecord schema.

Each concern answers a different question. Execution state describes progress, while the task record says whether the objective is still active. A terminal task records a lifecycle outcome but does not establish that a provider-side effect persisted. A pull request URL identifies an artifact without proving that its patch is correct.

The task remains the durable reference point; execution state, messages, child tasks, and evidence each have narrower roles.

Execution advances the task. Child tasks handle scoped work. Evidence supports claims about the outcome. None of them replaces the task record.

Useful components with narrower jobs

Several parts of an agent system hold important state. Problems start when one of them is asked to stand in for durable ownership.

Component	Strength	Does not own
Agent loop	Maintains local context and chooses the next action	The job after the loop stops
Queue	Delivers an event, perhaps more than once	Whether an event starts new work
Sandbox	Retains a checkout, files, and processes	Current intent, Task state, or effect safety
Thread	Preserves human discussion	Task lineage, cancel state, or result proof

Making a loop longer only delays its boundary. A queue can tell a consumer that a message is ready. The scheduler still has to decide whether a matching pending delivery permits a claim.

A persistent sandbox can retain a checkout and useful processes while becoming stale or unauthorized. A GitHub, Linear, or Slack thread preserves collaboration, but its chronology is not a lifecycle protocol. The coordination layer anchors each component to a task and gives it a smaller, testable responsibility.

Three identities that should stay separate

Reliability improves when every boundary names what is being identified.

Identity	Names	Boundary
Task	The durable objective	Owns lifecycle, lineage, mailbox, and result; it is not one delivery or process
Delivery	One ready delivery or dispatch	Validates a claim; it does not permit repeating every effect
Effect	One external action or artifact	Supports resume and recovery; it does not prove the objective is correct

A ready notification can repeat while the task identity stays stable. The useful distinction is among the durable task record, one bounded execution’s persisted state, the pending-delivery identity used at claim time, and external effects or observations used as evidence. Current Charlie can route follow-ups into an active task mailbox. If the target is terminal, missing, or cannot safely accept input, the follow-up may produce a new task instead.

One request ID cannot solve every replay problem. A transport identity can make claim handling safe at one seam while leaving a GitHub comment or pull request creation ambiguous at another. A task ID groups the work but is too coarse to identify which comment, branch operation, or CI wait is being retried. The identities need explicit links rather than overload.

Durable task authority

The task record is the durable statement that an objective exists. In Charlie, it has a stable ID, root ID, optional parent ID, timestamps, lifecycle status, and, when available, a terminal result. A common lifecycle path is:

scheduled -> queued -> running -> succeeded | failed | canceled | timed_out

That line is a useful summary, not a complete transition graph. Recovery races can skip an intermediate view. A task may move from scheduled to running when an executor claims the matching pending delivery before the scheduler has persisted queued. Result reporting can also terminate a non-terminal task without every intermediate status appearing.

The scheduler remains the lifecycle authority. Executors report outcomes to it; they do not independently redefine terminal state. Other components should read scheduler state rather than infer authority from a local process, queue message, or conversation thread.

A ready delivery has a narrower meaning: work may be claimable. Ready notifications may be redelivered. The scheduler accepts a claim only when it matches the pending-delivery token; later claims without a match are rejected. This local rule does not establish a global “one owner” invariant across every system.

A repeated notification can be harmless when one claim consumes the matching pending-delivery token and a duplicate claim finds no match.

The scope of this protection matters. Charlie deduplicates at defined scheduler boundaries; that is not a claim of global exactly-once execution. A downstream provider write still needs its own identity and recovery policy.

Lineage adds another boundary. A root task identifies the overall objective. A child task identifies a delegated sub-objective, and its parent link records who requested the work. Each child has scoped input, a separate lifecycle, and a persisted terminal result. The root does not automatically become successful because one child succeeded, or failed because one child failed.

{
  "id": "task_fix_flake",
  "rootId": "task_fix_flake",
  "status": "running",
  "message": "Fix the flaky test"
}

This shortened example omits unrelated fields. The important property is durable addressability: a later process can ask what this task is and whether it still authorizes work.

Bounded runs recover at checkpoints

One bounded execution answers a narrower question: how is this attempt progressing?

Conceptually, an executor claims work, fetches context, checks scheduler state, calls the model, dispatches tools, waits for results, and reports an outcome. It persists selected state at useful checkpoints rather than trying to serialize every detail of a live process:

• The current phase and turn can survive a runtime restart.
• Transcript and pending-tool state can be persisted where appropriate.
• Recovery can distinguish work that never started from work that started without a recorded result.
• Scheduler checks provide defined places to observe cancellation and later input.
• Limits on turns, tools, time, and delegation create escalation points instead of an endless process.

The hardest case is an external side effect. If a non-resumable operation was dispatched but no result was recorded, the executor should surface that interruption instead of blindly creating another comment, branch, or pull request. Interrupted after dispatch; effect unknown preserves information that a generic retry would erase.

Some operations can resume around stable identities. Waiting for an existing child task differs from creating another child. Polling a known CI run differs from starting a new one. Recovery should preserve those distinctions.

Cancellation is cooperative. An executor can check whether the task or an ancestor still authorizes work before the next phase or tool dispatch. It cannot promise to interrupt every in-flight API call instantly. Descendant tasks also need to observe ancestor cancellation at defined boundaries so delegated work does not continue indefinitely after the root has stopped.

Follow-ups join at safe boundaries

Humans change requirements while work is underway. They add reproduction details, correct an assumption, narrow scope, or ask the agent to stop. Durable follow-up handling prevents a run from continuing with stale instructions.

Charlie attaches later input to a per-task mailbox. At scheduler-check boundaries, the executor reads unseen messages, deduplicates defined mailbox operations by message identity, adds new input to the model-visible record, and incorporates it in a subsequent turn. The message survives even if it cannot interrupt the provider or tool operation currently in flight.

On GitHub, a second mention or comment in an active issue or pull request can join the in-progress task when the system can identify an active task that can safely accept input. Linear comments and Slack thread replies follow the same general shape. If the intended task is terminal, unavailable, or cannot safely accept the message, the system can fall back to a new task rather than dropping the follow-up or pretending it joined old work.

Durable delivery does not mean immediate interruption. Transcript inclusion also proves only that the model received a message, not that it interpreted the correction correctly.

The mailbox is a continuation channel, not general-purpose memory. Its job is to keep later input attached to the responsibility it changes.

Delegation creates scoped child objectives

Charlie models specialist delegation as a task tree. The parent creates a focused child assignment with an explicit contract. The child gets the context supplied for that assignment, has its own lifecycle and bounds, and returns a persisted terminal result. By policy, the parent remains responsible for synthesis and the final response.

Lineage makes the delegated work attributable and inspectable: the system can identify who requested the investigation, which instructions it received, and whether it finished. Ancestor stop checks apply at defined boundaries, but task-tree structure alone does not guarantee that every external action or worker process is instantly stopped.

A useful delegation contract includes the goal, relevant evidence, stable identifiers, constraints, and what the child should return. Child tasks do not automatically inherit the parent’s entire transcript or in-process state. The narrow contract limits accidental authority and makes recovery easier to inspect.

Deterministic scheduling can deduplicate creation of the same child task at a defined boundary. It does not make every action inside the child exactly once. Provider effects still follow their own rules.

A fictional issue-to-patch trace

This is an illustrative, fictional trace rather than a record from a customer or Charlie’s production systems.

An engineer mentions Charlie on GitHub issue BUG-42: “The checkout integration test fails intermittently. Find the cause and prepare a patch.”

The objective remains unresolved across new input, delegation, an uncertain provider response, and failed CI until the active task produces stronger evidence.

1. Task T1 records the objective and source issue. Delivery D1 makes it claimable.
2. A bounded execution claims D1, reproduces the failure, and starts a patch.
3. The engineer adds a second comment: “It only happens when the new tax flag is enabled.” The comment joins T1 through its mailbox and appears at the next safe turn boundary.
4. T1 delegates a scoped log investigation to child task T2. By policy, the parent remains responsible for using the finding.
5. The transport redelivers D1. The scheduler finds no matching pending-delivery token and rejects the duplicate claim.
6. T2 reports that two setup operations race. The active execution updates the fixture and creates commit C1.
7. GitHub accepts a request to open a pull request, but the response is lost before the PR number is recorded. The effect is now ambiguous.
8. The example’s recovery policy searches by the known branch, finds PR-88, reads it back, and records the artifact. Without a stable lookup, it would preserve the uncertainty and request review instead of issuing another create.
9. While T1 remains active and waits for CI, run CI-9 fails a different test. A later phase incorporates that evidence, revises the patch, and pushes commit C2. Run CI-10 passes the required checks.
10. Charlie’s communication policy makes a best-effort attempt to post completion in the same GitHub thread with durable references to the pull request, commit, test results, and CI run. If that reply is unavailable, policy can use a fallback surface and report where the update landed.

The task can now reach succeeded, but the status is only one fact. The attached evidence supports stronger claims about the branch, pull request, revision, and checks. This reduces the risk of silent internal completion, where the runtime finishes but the engineer never receives an actionable result.

Effects need operation-specific recovery

An external error can mean the provider rejected a request, accepted it but lost the response, or never received it. Recovery therefore needs an operation-specific policy; this is engineering guidance, not one universal Charlie effect engine.

Operation shape	Example recovery policy
Read	Retry within bounds, as with reading issue metadata
Keyed	Retry with the same supported key under the provider’s contract
Discover	Read back by a stable correlation, such as finding a pull request by branch
Wait	Poll the same durable identity for an existing CI run or child task
Checkpoint	Inspect the persisted step and result before continuing a local transformation
Ambiguous	Stop or request review when a lost response has no stable lookup

When an operation has a stable lookup, such as branch-to-PR, a recovery policy can read provider state before deciding whether another create is safe. Without a supported idempotency key or reliable lookup, it should preserve uncertainty and escalate. Otherwise an ambiguous response can turn into duplicate comments or pull requests, while a false success can hide work that never became visible outside the runtime.

Status needs durable evidence

Completion is layered. Each observation supports a different strength of claim.

Observation	What it supports	What it does not prove
Task is running	The lifecycle authority accepted an active claim	A process is healthy or making useful progress
Task is succeeded	The scheduler accepted a success result for that task	External writes persisted or the solution is correct
Tool result says created	The tool saw a success response	The artifact still exists with expected contents
Stable artifact ID or URL	A specific artifact can be cited	The artifact is correct, merged, or complete
Provider read-back matches	The provider currently exposes those fields	The change satisfies the user’s intent
Local tests pass	Those tests passed locally	Remote CI, hidden cases, or live behavior will pass
Required CI checks pass	Required checks succeeded for a revision	The patch is correct or risk-free
Human or domain review accepts it	A reviewer judged it against stated criteria	No proof beyond those criteria

Status is useful because it records lifecycle authority. Durable evidence is useful because it lets another person or process inspect the claim. A dependable completion report usually needs both: the current state plus stable references such as a commit SHA, pull request URL, provider read-back, test output, or CI result.

The evidence required should match the consequence. A low-risk draft comment may need a stable reference and read-back. Shipping code usually calls for a commit, diff review, tests, and CI. Higher-impact actions may still require explicit human approval after automated checks pass.

System boundaries

Coordination is easier to trust when its limits are stated plainly.

• End-to-end execution is not exactly once. Deduplication applies only to defined scheduler, mailbox, and delegation keys. External effects need their own supported idempotency or reconciliation strategy.
• Cross-system writes are not atomic. Task state, executor checkpoints, GitHub, Linear, Slack, and CI do not commit together.
• Cancellation is cooperative and non-immediate. It cannot retract an API call that a provider already accepted.
• Follow-ups can join active work when the target can be identified and still accepts input. Some cases require a new task or another fallback.
• Child tasks receive scoped instructions rather than an automatic copy of all parent context.
• Root status does not automatically aggregate descendant outcomes. By policy, parent synthesis stays explicit.
• Status and provider read-back record facts about lifecycle and observed state. They do not establish semantic correctness by themselves.

These boundaries shape the recovery strategy: bounded retries where repetition is safe, reconciliation where effects are uncertain, escalation where proof is insufficient, and explicit communication when the work reaches a conclusion.

The durable owner is the foundation

Dependability starts with task ownership. The task record carries lifecycle and lineage. Persisted execution state records selected progress. Scheduler and mailbox keys protect specific replay boundaries. Provider artifacts and read-backs supply evidence, while operation-specific policy handles uncertain effects.

Those mechanisms have limits. They do not make cross-system writes atomic, turn cancellation into an instant interrupt, or prove that a patch is correct. They keep authority, state, and evidence separate enough that the agent can continue carefully and explain what remains unresolved.

The model supplies judgment, and tools supply agency. Coordination keeps the engineering task owned when any one bounded execution reaches its limit.

Continue the series

Next: How We Built Charlie, Part 2: The Task, Not the Chat. Browse the full How We Built Charlie series.