How We Built Charlie, Part 5: Scheduling Durable Work

How We Built Charlie continues from Chapter 4: once routing assigns responsibility, the scheduler must keep that Task durable.

A production investigation begins just as part of the execution infrastructure restarts. The engineer who asked for help should not have to know which process was alive, whether a queue notification was delivered twice, or whether a worker had started before the restart.

The durable system has to answer more useful questions. Was the request accepted? Is it waiting to become runnable, available for claim, actively running, canceled, timed out, or complete? If delivery repeats, can another executor safely attempt the claim? If the executor disappears, who records that the work did not finish? If an ancestor Task is canceled, when do delegated Tasks stop?

Charlie puts those answers in the Task lifecycle and scheduler protocol rather than inferring them from process health. The scheduler owns durable state, readiness, claim authority, mailbox coordination, cancellation, timeout, and acceptance of terminal results. Executors perform bounded runs and report outcomes. Queues carry notifications, but they do not define truth.

The examples below are simplified architectural contracts. They omit private topology and exact operational settings. Where implementation details matter to a guarantee, the text states the boundary and caveat rather than turning a local mechanism into an end-to-end promise.

The scheduler owns the canonical lifecycle. Queue delivery and executor state are inputs to that lifecycle, not substitutes for it.

The Task status is a durable claim

Charlie uses a small canonical status set:

Status	Meaning
scheduled	The Task exists durably and is waiting for release or ready delivery
queued	Ready delivery has been published and the Task remains available to claim
running	The scheduler accepted a claim for the matching delivery token
succeeded	The scheduler accepted a successful terminal result
failed	The scheduler accepted an error terminal result
canceled	The scheduler recorded that the Task should no longer continue
timed_out	A running Task exceeded its scheduler-owned execution window

The terminal statuses are succeeded, failed, canceled, and timed_out. Once a Task is terminal, later result writes are rejected.

A readable common path is:

scheduled -> queued -> running -> succeeded
                               -> failed

The complete transition model is broader. Cancellation can terminalize any non-terminal Task. A running Task can time out. A valid claim can move a Task from scheduled directly to running if the ready notification was accepted before the queued view was persisted. Success or failure can also be recorded from an earlier non-terminal state when recovery or service behavior requires it.

This nuance prevents status names from being overinterpreted. queued does not mean execution completed, or even that an executor is currently healthy. It means the scheduler published readiness and still considers the Task claimable. running means the scheduler accepted a claim, not that every tool call is making progress. succeeded means the scheduler accepted the executor’s result, not that every external claim has been independently verified.

Scheduling persists before it notifies

When routing chooses a new Task, the scheduler first constructs and stores the durable Task record. It then places the Task into its managed schedule and arranges an alarm for the next eligible release.

Conceptually:

scheduleTask(request):
  validate request and admission policy
  task = buildTask(status: "scheduled")
  persist task
  emit task-created evidence
  add scheduled release entry
  persist managed queues
  arrange next alarm
  return task

The order matters. Publishing a ready notification for a Task that has not been durably stored would create work with no lifecycle authority behind it. Persisting first lets later reconciliation recover even if the service restarts before delivery.

Scheduled time can represent immediate work or delayed work. Both use the same durable record. The alarm processor releases entries when they become eligible and creates a pending-delivery token before publishing readiness.

The scheduler maintains several durable queue concerns:

• Scheduled releases track Tasks that are not yet eligible for delivery.
• Pending deliveries track ready events that may need publication or retry.
• Running timeouts track claimed Tasks whose allowed execution window can expire.

The earliest eligible item determines the next alarm. Queue mutations mark that calculation for recomputation under the scheduler’s coordination lock, so multiple code paths do not independently compete to set timing authority.

Ready delivery is at least once

The ready event tells execution infrastructure that a Task may be claimed. It includes the Task ID, customer and lineage context, source correlation, and a delivery event ID. That event ID is the claim token for this readiness attempt.

The pending-delivery record is durable before publication. If publishing fails, the same delivery can be retried. In the normal path, a successful publish is followed by reconciliation; recovery may skip a pending entry whose event was already published rather than replay it. Retries reuse the same event ID rather than manufacturing a fresh claim opportunity for the same pending delivery.

This makes task.ready an at-least-once notification. The same event can appear more than once. Consumers must treat it as permission to attempt a claim, not as proof that they uniquely own execution.

In the normal path, after a publish succeeds, the scheduler reconciles current Task state. If the Task is still scheduled, it records queued. If the Task is already running, terminal, or missing, it removes the stale pending entry instead of forcing the lifecycle backward.

That yields an intentionally careful sequence:

durable Task
  -> durable pending-delivery token D1
  -> publish ready(D1), possibly more than once
  -> reconcile current Task state
  -> record queued when still eligible

The notification and status write are not one atomic transaction across the queue transport and task store. Normal-path reconciliation closes the common gap without claiming that the systems commit together.

Ready can repeat. Ownership begins only when the scheduler accepts a claim against the matching pending-delivery token.

Claim consumes one scheduler token

An executor receiving ready(D1) asks the scheduler to claim the Task with D1. The scheduler verifies that the Task and pending-delivery token still match.

claim(taskId, deliveryEventId):
  load task and pending delivery
  require pending.eventId === deliveryEventId
  require task.status is scheduled or queued
  consume pending token
  persist task(status: "running", startedAt: now)
  add running-timeout entry
  return claimed task

If another executor receives the repeated ready event and presents D1 after the token was consumed, its claim is rejected. This is the scheduler’s local ownership boundary: one accepted claim for one pending-delivery token.

The implementation also handles failure while crossing that boundary. It removes the old token before writing running; if the durable Task write fails, it restores the old token when no newer token has replaced it. That recovery preserves claimability instead of silently losing the Task.

The claim protocol has a narrow job: repeated ready deliveries compete through one scheduler token, so ordinary duplicate claims converge. Tool replay and external-effect recovery belong to the executor and provider boundaries covered in Chapter 6.

The scheduler and executor own different facts

Durability becomes easier to reason about when authority is explicit.

Component	Authoritative for
Durable Task store	Canonical Task record and current lifecycle status
Task scheduler	Release timing, pending delivery, claim acceptance, mailbox, stop state, timeout
Executor	Bounded run progress, model turns, tool dispatch, selected checkpoints
Provider integrations	Current external artifact state and accepted effects

The executor cannot declare a Task succeeded by ending its process with a cheerful message. It reports a result to the scheduler. The scheduler checks that the Task is still non-terminal and records succeeded or failed while removing it from managed queues.

Likewise, the scheduler does not know that a patch is correct merely because it accepted success. Terminal status records lifecycle authority. A useful completion report still needs durable evidence such as a commit SHA, pull request URL, provider read-back, test command, or CI result.

The separation also supports restart recovery. After infrastructure restarts, the durable Task and queue records can say whether the investigation is scheduled, pending delivery, claimable, running, or terminal. Executor-local state can resume or explain its own checkpoint, but it does not rewrite the scheduler’s history based on process memory.

Durable delivery needs reconciliation

Reliable scheduling is less about never failing than about making failure states inspectable and repairable.

Consider several interruption points:

Interruption point	Durable evidence and recovery direction
After Task persistence, before schedule queue write	Stored Task exists; schedule reconciliation can restore release metadata
After pending token persistence, before ready publish	Pending delivery remains eligible for retry
After ready publish, before queued write	Claim can still accept scheduled; later reconciliation avoids backward state
After claim token removal, before running write	Claim path restores the old token when safe
After running, before executor progress checkpoint	Scheduler still owns active status and timeout; executor recovery handles run
After running, while an external effect is uncertain	Scheduler preserves lifecycle authority; executor/provider recovery supplies evidence

Mailbox operations participate in the same durable model. Appending a message to a scheduled or queued Task can trigger best-effort self-healing if expected schedule or pending-delivery metadata is missing. The mailbox does not replace readiness, but a relevant follow-up should not remain attached to work that can no longer be delivered because of a repairable queue inconsistency.

Reconciliation repairs scheduler-owned invariants and leaves external truth to the system that owns it. A missing scheduler lookup alone cannot prove that a prior owner or provider effect is gone.

Retries need bounded, durable policy

Ready publication can fail because a queue or downstream service is temporarily unavailable. The scheduler retries pending delivery with backoff and jitter, reusing the same event ID. Work is processed in bounded batches so one customer’s backlog does not turn an alarm pass into unbounded execution.

The exact retry constants are operational settings rather than product guarantees. The architectural requirements are more durable:

• Attempt count and next eligibility are persisted.
• Retries reuse stable delivery identity.
• Delay increases rather than hammering a failing dependency.
• Work per pass and publish concurrency are bounded.
• In normal processing, pending delivery is cleared for terminal, running, or missing Tasks.
• Exhaustion becomes an observable failure condition rather than an infinite loop.

Backoff does not make an unsafe operation safe to repeat. It is appropriate for publishing a stable ready event because the claim token absorbs duplicate delivery. External writes require their own replay contract.

Cancellation is cooperative and inherited at checks

Cancellation changes durable authority. The scheduler can mark a non-terminal Task canceled, and executors ask whether they should stop before later phases or tool dispatches.

For a Task with delegated children, the stop check follows ancestor lineage. If the Task itself or a fetched ancestor is terminal, the scheduler returns shouldStop: true. A root cancellation can therefore stop descendants cooperatively without eagerly rewriting every child record in one cross-object transaction.

This is a checkpoint mechanism, not a kill signal. An API call already accepted by a provider cannot be retracted by changing Task status. A long-running local command may finish before the executor reaches its next stop check. The executor should check at meaningful boundaries and avoid beginning new effects once durable authority has ended.

The stop route has an availability-first caveat. If the scheduler or store fails unexpectedly while evaluating the stop condition, the route returns shouldStop: false and records the failure for diagnosis. This fail-open behavior avoids terminating useful work merely because cancellation could not be verified. It can delay cancellation during an outage, so it must not be described as immediate or absolute.

The contract is:

verified missing Task                 -> stop
verified terminal Task or ancestor    -> stop
verified active Task and ancestors    -> continue
unexpected stop-check failure         -> continue for now, report the fault

That tradeoff is deliberate. A false positive stop can discard useful work. A false negative can let work continue until the next successful check. Teams may choose different priorities for higher-impact actions; those actions should also use explicit approval and narrower tool policy rather than relying only on stop polling.

Timeout turns disappearance into durable state

Once a claim is accepted, the scheduler adds a running-timeout entry. If the Task remains running beyond its allowed window, the alarm processor records timed_out and removes its timeout tracking.

Timeout is not a judgment about why execution failed. The executor may have crashed, lost connectivity, stalled on a tool, or exceeded its bounded run. The durable fact is that the scheduler no longer authorizes the Task to remain indefinitely active.

A timeout also does not prove that no external effects occurred. The executor may have pushed a branch before disappearing. Recovery should inspect reported effects, provider state, and Task context before deciding whether to retry, continue in a new Task, or ask for human review.

This distinction matters in the opening scenario. After the infrastructure restart, an engineer should see a durable status rather than an eternally “working” process indicator. If the Task is re-delivered and claimed, the scheduler records that path. If it exceeded the run window, timed_out states what the scheduler knows, while external evidence tells us what may already exist.

Admission controls protect shared durability

Scheduling durable work also means deciding whether the system can responsibly accept it. New root Tasks pass through usage and quota admission before they become active work. Child Tasks and idempotent replays have different treatment because they are continuations of already-owned objectives rather than unrelated new roots.

Admission is part of the lifecycle contract. A rejected request should produce a stable, inspectable outcome rather than partially scheduling work. The implementation can persist a canceled candidate for some rejected paths so retries receive consistent state.

Availability policy still matters. When an external usage checker is missing, malformed, or unavailable, the current scheduler can fail open rather than treating dependency failure as customer overuse. Durable local quotas remain a separate guard.

The precise thresholds are internal operational choices and can change. The design principles are:

• Apply admission before creating unbounded new ownership.
• Make idempotent replay return the prior result instead of charging again.
• Distinguish a new root from a child or mailbox continuation.
• Preserve enough durable evidence to explain rejection.
• Keep dependency failure separate from a verified policy denial.

Terminal reporting closes the lifecycle, not the proof

An executor reports success or error through the scheduler’s result endpoint. The scheduler rejects results for already-terminal Tasks, writes the appropriate terminal status for eligible Tasks, records completion time and result, and removes schedule, pending-delivery, and timeout entries.

recordResult(taskId, result):
  task = load task
  reject if terminal(task)
  next = result.success ? "succeeded" : "failed"
  persist task(status: next, completedAt: now, result)
  remove task from managed queues
  emit status-change evidence

The result can include a bounded summary of important durable effects. That helps a later process or reviewer find a branch, commit, pull request, document, or posted update. It is a recovery aid, not an exhaustive effect ledger and not independent verification.

Each mechanism has a narrow guarantee. Dependability comes from composing them without upgrading local properties into global promises.

Useful completion claims remain layered:

Claim	Evidence that supports it
The scheduler accepted the request	Durable Task ID and initial status
The Task became available for execution	Ready delivery evidence and current scheduler state
One claim token was accepted	Running status tied to the matching delivery event
The executor reported success	Terminal Task result
A repository revision exists	Full commit SHA and provider read-back
A pull request contains the expected patch	Pull request URL, head revision, diff review, and provider read-back
Tests passed	Exact command, revision, environment, and observed result
The production issue is resolved	Domain-specific validation beyond scheduler status

The scheduler can truthfully say what happened to the Task lifecycle. It cannot infer semantic correctness from status alone.

Reliability is a set of bounded semantics

The production investigation survives a restart because its important state does not live only in the process that happened to be running. The Task was persisted before readiness. Pending delivery can be retried. Repeated ready events compete through one claim token. The scheduler records running and terminal state. Cancellation is checked against the Task and its ancestors. Timeout prevents abandoned work from appearing active forever.

Each mechanism is intentionally narrow:

• Ready delivery is at least once.
• Claim protection applies to one scheduler delivery token.
• Queue and task-store writes are reconciled, not globally atomic.
• Retries are bounded and appropriate only where repetition is safe.
• Cancellation is cooperative and may be delayed by in-flight work or stop-check failure.
• Timeout records scheduler authority, not the absence of external effects.
• Terminal status records accepted lifecycle outcome, not proof that the engineering result is correct.

These limits are what make the system useful under failure. They tell recovery code what it may repeat, what it must read back, what it should stop, and what remains uncertain.

Durable scheduling does not promise that infrastructure never restarts. It makes a restart a condition the system can reason about instead of an event that erases ownership.

Continue the series

Previous: How We Built Charlie, Part 4: Routing Is Ownership. Next: How We Built Charlie, Part 6: The Executor Loop. Browse the full How We Built Charlie series.