How We Built Charlie, Part 11: Context Without Magic Memory

How We Built Charlie continues from Chapter 10: the execution environment needs context whose source and freshness are clear.

Ask Charlie to review a service and he needs more than the diff.

He needs the current repository and its tracked rules. He may need a skill that defines the team’s preferred review procedure. He needs to know which tools are available, what the Task asks for, and which customer or system facts are relevant to that request. He may need recent task history or repository knowledge, but only when its provenance and freshness are clear enough to support the decision.

An indiscriminate transcript of everything Charlie has ever seen would be a poor substitute. It would mix old and new state, unrelated customers, superseded instructions, partial investigations, and outputs written for different purposes. More context would make the prompt larger while making authority harder to reason about.

Charlie’s context system is an assembly pipeline. Typed providers gather bounded entries from named sources. The runtime validates and gates those providers, runs eligible work concurrently, orders the results deterministically, and records what was included or excluded. The first turn receives a snapshot. Later turns can opt into fresh context, but most providers do not rerun by default.

This architecture gives context a provenance. It also gives it an expiration problem. A sourceable snapshot can still become stale after it is fetched. Dependability comes from knowing where a fact came from, when it was assembled, and which sources must be read again before a consequential action.

The examples below simplify the public context contracts and omit private operational settings. The implementation can evolve while the durable idea remains: context should be sourceable and inspectable, with each entry kept as a bounded record. Metadata and outcomes that apply more broadly can live at the provider or run level.

Providers can fetch concurrently while the rendered prompt remains ordered, bounded, and inspectable.

Context is selected input, not a second task owner

The Task carries the current objective and durable lifecycle. The transcript records model-visible chronology. Context is material assembled for a run or turn to help the model reason about that objective.

Those objects overlap in content, but they have different jobs. A Task can reference a repository without containing its rules. A transcript can record a prior tool result without proving that result is still current. A context provider can summarize available repositories without granting permission to edit all of them.

Object	Primary role	Freshness model
Task	Durable objective, lineage, lifecycle, and current request	Changes through scheduler and mailbox protocols
Run log	Ordered model-visible messages, calls, and results	Append-oriented within one execution
Context	Selected source material for a run or turn	Snapshot determined by provider source and run policy
Artifact	External or repository state used as evidence	Must be read back when current state matters

Calling all four “memory” hides the boundaries that make them useful. The model can remember a tool result because it appears in the current transcript. The runtime can reconstruct provider choices because it persisted context metadata. A later run can read a repository rule again. None of those mechanisms implies universal recall across every Task.

Providers turn source access into a typed contract

A context provider declares its identity, category, configuration, capability requirements, and run function. The output is a list of entries rather than an arbitrary prompt string. That lets the runner apply common validation, ordering, timeout, budgeting, and reporting rules.

ContextProvider
  name
  category
    skills | memory
    system | rules
  required capabilities
  turn policy
  timeout and token budget
  run(request) -> entries

The contract does not force every provider to use the same source. A rules provider reads tracked files. A skills provider discovers SKILL.md files. A system provider may report runtime versions or currently reusable devboxes. Another provider can query customer-scoped knowledge or recent Task facts.

What they share is the surrounding behavior. The runtime can say that a provider was included, empty, timed out, capability-gated, turn-gated, invalid, or failed. Prompt construction does not have to infer those outcomes from missing text.

This also keeps source-specific complexity out of the model prompt. Git reads, CLI calls, repository discovery, and API parsing happen before rendering. The prompt receives labeled entries and warnings rather than raw integration machinery.

Categories describe purpose, not truth level

Charlie’s provider categories are skills, memory, system, and rules. The names support grouping and metadata order.

• skills describes discoverable procedures the agent can apply.
• memory covers selected persisted knowledge products when an agent configuration includes them.
• system describes runtime, tool, repository-inventory, or execution-environment facts.
• rules carries repository-owned instructions and constraints.

A category does not make an entry authoritative by itself. A rules entry is useful because its provider can identify the tracked source file and revision. A system entry may be a best-effort command snapshot. A memory entry can be well sourced and still need a freshness check. A skill can define a procedure without supplying current facts about the repository.

Current Charlie agent configurations select a particular provider set for a role. An entry agent and a focused worker do not need identical context. The selected agent configuration is therefore part of provenance: it tells us which providers were even eligible to contribute.

Concurrent fetching and deterministic rendering solve different problems

Provider calls can be independent. Reading repository rules should not wait for a version command when both can run safely at the same time. Charlie runs eligible providers concurrently to keep startup latency bounded.

Concurrency would be dangerous if completion timing determined prompt order. A slow provider would move around from run to run, making prompts and debugging harder to compare. The context runner separates execution order from rendering order.

Eligible providers run in parallel. Their results are then rendered according to stable policy, including cache-stability or declared position and entry order. Provider-run metadata also has a canonical category order. The exact ordering mode can evolve with an agent version, but it is selected intentionally rather than inherited from network timing.

eligible = validate + gate(providers)
results = run concurrently(
  eligible,
  with timeout,
)
prompt = render(
  results,
  stable ordering policy,
)

This gives retries a stable target. It also makes prompt snapshots meaningful in tests. A provider can become slower without silently changing instruction precedence.

Turn gates keep snapshots from becoming background noise

Most context providers default to the first turn. That is where Charlie needs the initial repository rules, skills index, environment snapshot, and customer or system facts selected for the run.

On turn zero, the context is merged into the initial user prompt with the Task content. The transcript does not add a second standalone local:context:0 message, because that would duplicate the same material.

On later turns, providers configured for all turns may run again. If they produce non-empty context, the kernel appends one deterministic user message for that turn, identified conceptually as local:context:<turnIndex>. First-turn-only providers are reported as excluded by the turn gate.

turn 0
  task + snapshot -> first prompt

turn 1
  model and tools by default

later opt-in
  fresh context -> one user item

The default avoids repeatedly paying for and re-injecting mostly static material. The opt-in supports sources where later freshness is worth the cost. It also prevents a provider from silently becoming a hidden polling loop merely because the executor has multiple turns.

“First turn” still means snapshot, not permanent truth. If the working branch changes after the prompt was assembled, a Git status entry from turn zero may be stale. Tools that need current state should read the repository again at the point of action.

Capability gates make unavailable context explicit

Some providers require a devbox or a particular integration capability. Charlie retains those requirements through execution resolution and evaluates them in the context runner.

If a provider needs repository read access and the current agent lacks it, the provider is excluded with a missing-capability reason. It does not receive a downgraded request that might return misleading partial data. This matters for worker roles: a read-only exploration worker can receive repo context without inheriting a parent’s write permissions, while a communication worker does not need the same repository providers at all.

Gating at the runner produces consistent metadata across providers. It also lets the agent configuration remain declarative. A provider can state what it needs, and the runtime decides whether the selected role satisfies that contract.

Capabilities are necessary but not sufficient for relevance. A provider with permission to read all installed repositories should still return a bounded inventory or select the repository associated with the Task. Access does not justify injecting every available fact.

Partial failure is a normal assembly outcome

A context pipeline that fails the entire Task whenever one provider is slow would make optional context a hard dependency. Charlie isolates provider outcomes so useful context can still be assembled.

Each provider runs under a timeout and receives an abort signal for cooperative cancellation. If it times out, fails, produces no entries, exceeds its deterministic budget, or is excluded by a gate, the runner records that outcome. Other providers continue.

Outcome	Meaning
Included	Bounded entries render in deterministic order
Empty	Provider ran but had nothing relevant
Turn-gated	Provider policy excludes this turn
Capability-gated	Current role lacks a required capability
Timed out	Provider is excluded; other results continue
Failed	Source could not be read or parsed; metadata records why
Budget-excluded	Policy drops lower-priority material in a deterministic order

Timeout cancellation is cooperative. The runner can stop waiting and signal the provider, but it cannot guarantee immediate termination of an underlying system that ignores cancellation. The public claim should remain narrow: one slow provider does not indefinitely block prompt construction.

Partial failure also affects how Charlie speaks. If repository rules were unavailable, he should not claim he followed them. If a customer-knowledge provider timed out, he may proceed using repository evidence for a reversible investigation while avoiding a preference-sensitive write. Provenance metadata supports that distinction even when the model never sees an internal error dump.

Skills use an index before full content

Skills are versioned procedures stored in the repository. Injecting every skill in full would waste context and make unrelated procedures compete with the Task.

The skills provider first discovers supported SKILL.md files and emits a compact index with each skill’s name, description, and location. When the Task explicitly selects a skill, such as $copywriting, the provider also includes that skill’s full content under a separate selected-skills section, subject to bounded reads and validation warnings.

<available_skills>
  <skill>
    <name>copywriting</name>
    <description>
      Draft and review public copy.
    </description>
    <location>
      .agents/skills/copywriting/
      SKILL.md
    </location>
  </skill>
</available_skills>

<selected_skills>
  <!-- selected full content -->
</selected_skills>

This supports two useful behaviors. The model can see what procedures are available without paying for all of them. An explicit selection becomes auditable input rather than a vague claim that the agent “knew” a house style.

Discovery can still produce warnings. Duplicate names, invalid metadata, missing selections, or read limits should be visible as provider entries rather than guessed away. A selected skill is also guidance, not current repository state. It may tell Charlie how to review a migration without telling him which migration is deployed.

Repository rules come from tracked source

Repository instructions have high authority, so their source matters. Charlie reads them from a versioned, inspectable repository source in deterministic order and reports omissions or truncation.

That revision boundary prevents mutable files from silently changing the rules halfway through a run. It also gives reviewers a source they can inspect.

New guidance becomes authoritative through the repository’s normal versioned review path. A tool action cannot quietly rewrite the rules governing its own in-flight execution.

Versioned source still needs precedence and freshness handling. Rules can conflict, age, or exceed budgets, so the context keeps paths and deterministic ordering instead of presenting one anonymous block of text.

Repository knowledge needs provenance and a review date

Repository knowledge can combine authored playbooks, repository facts, prior Task summaries, and generated observations. The stable contract is provenance: a useful entry says what source supports it, which repository or customer it applies to, and when it should be reviewed. A generated summary without those fields can outlive the code it describes.

Architecture facts are especially sensitive to age. Services, dependencies, and ownership change. The system must be able to retire or refresh a knowledge object instead of treating its presence as permanent truth.

For consequential work, Charlie should reopen the current source. A persisted statement such as “this service uses queue X” can help discovery, but the deployment or code path is the authority when deciding what to change.

Provenance identifies where context came from; freshness policy determines when that source must be read again.

Persisted digests support recovery, not universal recall

The executor persists context metadata for each turn, including a digest, character count, and provider-run outcomes. This lets recovery determine whether the context phase already completed and avoid rerunning providers unnecessarily.

The write order matters because context metadata and transcript items are separate writes. On a later turn, a crash could occur after the digest is stored but before the corresponding local:context:<turnIndex> message is appended. Charlie checks for that expected transcript item before trusting the fast path. If the invariant is not satisfied, it repairs the context/transcript state instead of assuming the model saw material that never reached the ledger.

consistent = transcript matches(
  stored context,
  turn index,
)

if stored context and consistent
  skip provider rerun
else
  fetched = fetch context
  append context if needed
  persist digest

The digest stabilizes one execution through retries. It is not a cross-customer memory index, a semantic recollection service, or proof that the source remains current. A new Task can run providers again. A current-state tool call can supersede an earlier prompt snapshot. Recovery consistency and real-world freshness are separate properties.

Prompt anatomy should make authority visible

The assembled first turn has several layers. The exact rendering varies by agent version, but the conceptual anatomy is stable:

1. System instructions define the agent role and safety boundary.
2. The Task supplies the current request, origin, and structured content.
3. Selected context providers add labeled source material.
4. Tool definitions describe actions available to the role.
5. The transcript accumulates later model messages, calls, results, mailbox input, and optional context updates.

The prompt is assembled from sources with different authority and freshness, rather than flattened into one anonymous memory block.

The order matters because instructions and facts are not interchangeable. A repository rule can constrain how a tool is used. A system snapshot can report that a devbox exists without authorizing its reuse. A customer fact can shape the desired outcome without overriding safety policy. A tool result can update the current state while leaving the original request intact.

Labeled wrappers also help debugging. When a model follows stale guidance, an engineer can ask which provider supplied it. Without source boundaries, prompt review becomes archaeology through one large text blob.

Freshness is source-specific

No single refresh interval fits all context.

The Task message is durable input, but a mailbox follow-up can amend it. Tracked rules are stable for a revision, but the branch can advance. A skill procedure changes when its file changes. Runtime versions are snapshots. A pull request status may change seconds after it is read. A customer preference may remain valid for months or be replaced by a new decision.

A practical freshness policy asks three questions:

• Could this fact have changed since it was fetched?
• What source is authoritative now?
• How costly or risky is acting on stale data?

For a read-only code search, a first-turn repository summary may be sufficient. Before pushing, Charlie should inspect the current branch and remote. Before reporting CI success, he should read the check attached to the intended revision. Before applying a customer-specific preference, he may need to verify the current issue or documented decision.

This is why context providers do not eliminate tools. Providers help the model start from a useful, bounded view. Tools obtain current evidence at the decision point.

Common failures this design prevents

Everything previously seen becomes prompt input

The prompt grows, provenance disappears, and stale unrelated material gains accidental authority. Provider selection and budgets keep context task-shaped.

One slow source blocks the run

Per-provider timeout and failure isolation allow the rest of the context to render, while metadata records the missing source.

Concurrent providers reorder instructions

Fetches can finish in any order. Deterministic rendering prevents timing from becoming precedence.

Every available skill is loaded in full

The skills index supports discovery. Explicit selection controls full-content injection.

Uncommitted changes rewrite governing rules

Tracked-HEAD reads give repository rules a durable revision boundary.

A persisted digest is mistaken for current truth

Recovery uses the digest to preserve execution invariants. Current actions still read authoritative sources again when freshness matters.

A context failure is hidden

Included and excluded provider-run metadata distinguishes an empty source, missing capability, timeout, gate, and execution error.

Sourceable context is more useful than magical memory

Engineers do not need Charlie to claim perfect recall. They need him to use the right sources, preserve instruction authority, notice when evidence may be stale, and explain what was unavailable.

Typed providers make that behavior composable. Agent configuration selects the sources. Capability and turn gates decide eligibility. Concurrent execution keeps startup practical. Deterministic rendering keeps precedence stable. Skills use an index before full inclusion. Repository rules come from tracked source. Per-turn digests let the executor recover without duplicating model-visible context.

Every one of those mechanisms is bounded. Providers can fail. Snapshots age. Budgets omit material. A source can be authoritative for one repository and irrelevant to another. That is a stronger foundation than an unlimited memory claim because the system can expose those limits and read the source again.

Charlie starts a Task with enough context to act responsibly, then uses tools and provider read-back to update facts as the work progresses. The goal is not to remember everything. It is to know what supports the next decision.

Continue the series

Previous: How We Built Charlie, Part 10: Devboxes as the Agent’s Body. Next: How We Built Charlie, Part 12: Daemons: Persistent Roles, Bounded Runs. Browse the full How We Built Charlie series.