How We Built Charlie, Part 6: The Executor Loop

How We Built Charlie continues from Chapter 5: after claim, the executor must advance the Task through recoverable phases.

A release engineer asks Charlie to fix a regression introduced by a configuration change. Charlie loads the issue, repository guidance, and the failing test. He calls the model, which proposes inspecting the parser and running a focused test. The test reproduces the bug. Charlie delegates a narrow compatibility check to a worker, waits for the child result, revises the patch, runs the relevant suite, and prepares the final report.

That sequence crosses several systems and lasts longer than one request. The model call can finish while a tool is still running. A command can complete after the executor loses its response. A child Task can remain active while the parent is waiting. The process handling one step can disappear and later resume elsewhere.

The runtime therefore needs to preserve more than a prompt and a completion. It needs to preserve which phase owns the next decision.

Charlie’s executor is a durable state machine. Each tick reads the persisted run, performs one bounded phase, records the transition, and yields. A later tick continues from that recorded phase. The design keeps model calls, tool dispatch, waiting, recovery, and terminal reporting explicit instead of hiding them inside one long function.

One persisted phase owns each transition. The loop can stop between phases without forgetting what should happen next.

The phase is the recovery address

Long-running agent work is full of ordinary interruptions: a process restarts, a network response is lost, a tool takes longer than expected, or the scheduler asks a run to stop. Recovery becomes dangerous when the runtime cannot tell what was merely planned from what may already have happened.

A durable phase gives the executor a precise recovery address. “The run is active” is too broad. “The model produced tool calls and dispatch has not started” is useful. “These calls were dispatched, two results are recorded, and one child is still pending” is better.

The persisted run holds the current phase together with the state required by that phase. Transcript items preserve the model-visible chronology. Pending-tool records preserve the lifecycle of each tool request. Context and error state remain available across ticks. The executor does not need the previous process’s memory to decide what comes next.

Conceptually, one tick looks like this:

const execution = await loadExecution();

const transition = await runOnePhase(execution);

await persistTransition(transition);
await scheduleNextTickIfNeeded(transition);

The actual runtime has more validation and failure handling, but the contract is intentionally narrow. A phase receives durable state and returns the next durable state. It does not run the entire Task to completion.

This has a useful operational consequence: a phase can be retried as a state transition only when its effects have a defined recovery rule. The executor does not assume that replaying arbitrary code is harmless.

Phase	Durable fact it needs	Safe next decision
Load context	Which Task and run are active	Record the model input boundary
Call the model	Transcript and current context	Persist the returned items
Dispatch tools	Tool-call identity and dispatch state	Start eligible calls without losing identity
Await tools	Pending state, results, and deadlines	Poll, save state, or convert a known failure into a result
Advance the turn	Ordered tool outcomes	Add results to the transcript before another model call
Report terminal state	Final result or failure evidence	Notify the scheduler and finish executor cleanup

The phase name is not decoration. It is the instruction the next process follows.

A turn is a durable conversation boundary

The model does not receive a continuously changing stream while tools run. It receives an ordered input, returns a set of output items, and then waits for the runtime to complete the tool lifecycle before the next turn.

A simplified turn contains four boundaries:

1. The executor assembles the model-visible input from context and the transcript.
2. The model returns messages, reasoning items, and zero or more tool calls.
3. The executor dispatches the calls and records one authoritative outcome for each.
4. The outcomes enter the transcript in stable call order before the model is called again.

Tool execution happens outside the model call, but its outcomes rejoin the transcript at a defined turn boundary.

This separation prevents a partial batch from becoming an accidental prompt. Suppose the model asks Charlie to inspect two files, run one test, and wait for a delegated compatibility check. The file reads may finish quickly. The test may fail. The worker may take longer. Feeding whichever result arrives first back to the model would make the next turn depend on timing rather than on the requested tool set.

Charlie records individual outcomes as they arrive, but advances the model conversation only when the sibling set has reached a terminal outcome. The results are then materialized in tool-call order. Completion order remains useful operational evidence; call order remains the stable model-visible contract.

This is also why tool calls and tool results are separate durable objects before they become one semantic row in a debugging UI. The call records intent. The result records outcome. They may be written at different times by different processes.

Dispatch has to assume side effects

Some tools only read data. Others create branches, post comments, start devboxes, write files, or schedule child Tasks. The executor cannot treat all of them as pure functions.

Before invoking a tool, Charlie records that dispatch has started. If the process disappears after the external side effect but before recording the response, recovery sees an interrupted dispatch rather than a pristine call. That distinction prevents the default recovery path from blindly issuing the same action again.

This is an at-most-once safeguard, not an exactly-once guarantee. There is an ambiguity window between an external system accepting an action and Charlie recording the result. Marking dispatch first chooses the safer failure mode for side-effectful work: an explicit uncertain outcome that can be investigated or reconciled instead of an automatic duplicate.

Consider a tool that creates a pull request:

record dispatch started
        |
        v
request PR creation ------> GitHub accepts
        |
        x response is lost

On recovery, “no result recorded” does not mean “nothing happened.” Reissuing the request may create another PR unless the provider or tool has its own idempotency and read-back strategy. The executor keeps that uncertainty visible.

Waiting tools are different. A delegation wait or command-status poll is designed to resume observation of existing work. A restricted set of these operations can be redispatched with the same identity because they are continuing a wait, not repeating the original external action. That eligibility is explicit. It is not inferred from a tool name sounding harmless.

The rule is conservative:

Recovery state	Default treatment
Call was never dispatched	It can be considered for normal dispatch
Result is already recorded	Keep the first authoritative result
Dispatched read or write has no result after interruption	Do not blindly repeat it; record an interrupted outcome when recovery cannot reconcile it
Known resumable wait operation	Resume observation with the existing call identity
Late result arrives after settlement	Accept the report idempotently without replacing the recorded outcome

Tools can add stronger guarantees at their own boundary. A provider write may accept an idempotency key. A GitHub operation can read back the created artifact. A command runner can expose a stable execution ID that supports polling. Those mechanisms improve recovery, but the executor does not pretend they exist for every tool.

Bounded fan-out keeps one tick understandable

The model can request several independent tools in one turn. Starting them concurrently reduces avoidable waiting, especially when the calls read different systems or launch independent work. Unbounded fan-out would create a different problem: one model turn could monopolize executor capacity, multiply side effects, and produce a recovery set too large to reason about.

Charlie therefore dispatches work in bounded batches. The exact tuning is operational, not part of the public contract. The important behavior is that one tick handles a limited amount of new dispatch, checkpoints what it started, and leaves the rest for another tick.

This keeps the state machine responsive to stop signals and late context. It also limits the number of calls that can enter the ambiguous “started but result not yet recorded” state during one interruption.

Bounded dispatch does not require serial execution. Eligible calls within a batch can run concurrently and settle independently. The executor records each completed, failed, or deadline-expired outcome without waiting to write one giant batch result.

Pending tools are durable work, not promises in memory

A JavaScript promise is useful while one process is alive. It is not a recovery record.

For each tool call, the executor stores durable pending state: its stable call identity, dispatch state, timing information, and eventual result. The awaiting phase reads those records, polls work that exposes a status handle, and schedules another tick while unfinished calls remain.

Per-tool deadlines prevent one missing response from holding a turn forever. A deadline produces a known tool outcome for the model, but it does not imply the underlying external work was forcibly canceled. A command or provider request may complete later. If a late report reaches the executor after a result has already been settled, the first result remains authoritative and the late report becomes an idempotent no-op.

That policy avoids two common errors:

• treating a deadline as proof that no side effect occurred;
• allowing a late response to rewrite history after the model has already acted on an earlier outcome.

Tool failures are generally returned to the model as tool results. A failed test, unavailable API, or interrupted command is information the next turn can use. The whole Task does not need to fail because one action failed. The model may revise the patch, choose a narrower command, ask for input, or conclude that the request is blocked.

Checkpoint tool results before advancing

The executor’s most important tool-ordering rule is simple: record each tool result before advancing the model conversation.

For one tool call, settlement means the pending record has an authoritative outcome. For a sibling set, settlement means every call has reached a terminal outcome. These checkpoints are tool-scoped: they keep recovery and the next model turn consistent.

Recovery preserves uncertainty instead of erasing it. Tool results are checkpointed before the executor advances the conversation.

The ordering closes a subtle gap. If the executor advanced to the next model turn before inserting tool results, the model could see calls without their outcomes and continue from an incomplete record.

The loop therefore treats durable writes as transition boundaries:

tool handlers settle
    -> checkpoint each authoritative outcome
    -> materialize ordered results in the transcript
    -> move to the next model phase

model returns a final answer
    -> final assistant item is durable in the transcript
    -> enter terminal-reporting phase
    -> derive the Task result from the transcript
    -> write the result to the scheduler, retrying transient failures
    -> finish executor cleanup

There is no separate executor-owned run-result checkpoint. The final assistant item is already durable in the transcript. Terminal reporting derives the Task result from that item, records it with the scheduler, and retries transient scheduler-write failures. An already-terminal or missing Task is a benign recovery outcome, not a newly accepted result, so the model is not rerun. The executor marks itself done after handling either outcome.

The scheduler and executor still own different facts. The transcript preserves how the run ended; the scheduler owns the canonical Task result and lifecycle, including whether work is runnable, canceled, timed out, or terminal.

Failure reporting is also a phase

An unexpected exception inside a phase should not strand the run in an apparently active state. Charlie catches unhandled phase failures, persists structured error state, moves the execution toward failure reporting, and schedules another tick.

That last step matters. Reporting failure may itself require a scheduler call or another durable write. Doing it in the same collapsing stack frame would make the failure path less reliable than the success path. Treating failure reporting as a phase gives it the same recovery properties as ordinary execution.

This does not mean every tool error becomes a failed run. Expected tool outcomes stay in the transcript. The failure-reporting phase is for errors in the executor’s ability to continue the run or maintain its contracts.

The distinction appears in the product experience:

Event	What the engineer should see
Test command exits nonzero	Charlie can explain the failure and revise the approach
Delegated check is still running	The parent remains waiting with a visible child Task
Provider write has an interrupted, uncertain outcome	Charlie should report the ambiguity rather than claim success or repeat the write
Executor cannot continue its state transition	The run reports failure with durable diagnostic context
Scheduler has canceled or timed out the Task	The executor stops at a scheduler boundary and reports the lifecycle outcome accurately

The devbox is where work runs, not where truth lives

Charlie often executes code inside a devbox. The devbox provides repository files, processes, Git state, and command execution. It can outlive one executor tick, suspend, resume, or be reused when policy allows.

The durable executor remains the source of truth for the run. Process memory inside the devbox does not own Task status, mailbox handling, transcript history, retry policy, or the authoritative record of tool outcomes. When Charlie polls an asynchronous command, the command’s execution ID is evidence held by the tool lifecycle. When he inspects a reused repository, the runtime can reconstruct repository context from durable metadata and the environment rather than relying on a cache left by the process that started it.

This boundary keeps compute replaceable. Losing a process should be inconvenient, not equivalent to losing the engineering objective.

A dependable loop preserves what it cannot know

The executor loop does not remove partial failure. It turns partial failure into explicit state.

The runtime knows which phase should run next. It knows which tool calls were requested, which began dispatch, which produced results, and which remain pending. It knows when a deadline was reached without assuming the external work was canceled. It knows when an interrupted side effect cannot be safely repeated. Once the final assistant item is durable, terminal reporting can retry transient scheduler-write failures without asking the model to produce another answer.

That is enough structure for Charlie to load context, call the model, run tests and tools, wait on a child result, revise a patch, and continue after the process in the middle disappears. The loop remains bounded. The evidence remains ordered. Uncertainty remains visible where the system cannot honestly erase it.

Continue the series

Previous: How We Built Charlie, Part 5: Scheduling Durable Work. Next: How We Built Charlie, Part 7: Task Graph vs Transcript Ledger. Browse the full How We Built Charlie series.