How We Built Charlie, Part 3: Turning Webhooks into Work

This post is part of How We Built Charlie, a technical series on the runtime and product decisions behind dependable autonomous engineering work. Chapter 2 explained why the Task, rather than a chat session, owns the objective. This chapter covers how an incoming notification becomes work that can be attached to that Task.

A release starts failing after deployment. Sentry posts an alert into a Slack channel. An engineer mentions Charlie in the thread and asks him to investigate. Ten minutes later, another engineer adds that the failure only affects requests behind a new feature flag.

Those three inputs look related to a person reading the thread. At the boundary of an engineering system, they are separate provider deliveries with different shapes, identities, timing, and reasons for existing. One may be an automated alert, another an explicit request, and the third follow-up context for work already underway.

The visible behavior is straightforward: the first explicit request can create a Task, while later thread context can be appended to that Task’s mailbox when routing identifies it as relevant, rather than starting duplicate work.

Before a model can act, a heterogeneous notification must become a trustworthy description of work. That requires more than parsing JSON. The system has to verify the sender, preserve delivery facts, recover context, discard noise, stay within transport limits, and route the result without creating duplicate effects when providers redeliver. As processing continues, enrichment can turn a qualifying delivery into a Signal: a compact, durable routing record with work-aware context.

This chapter follows that path from webhook to Task. The examples are simplified and omit private identifiers and infrastructure details, but the boundaries and failure modes reflect Charlie’s current architecture.

GitHub, Linear, and Slack are direct inbound providers. Sentry is read-side diagnostic context; a Sentry alert can enter the work pipeline through Slack, where Slack remains the Signal’s provider.

Treat ingress as a security and evidence boundary

Webhook endpoints are public. Request routing may inspect untrusted data to choose how to handle an incoming request, but provider fields do not become authenticated facts at that point.

At the provider webhook boundary, Charlie verifies the raw request material before treating parsed fields as authenticated or constructing a trusted envelope. Slack and Linear also apply freshness protection so a valid old request cannot be replayed indefinitely. Failed verification stops that delivery from entering the work pipeline. Control-plane traffic, such as Slack URL verification, can receive the provider-specific response it needs without becoming work.

That boundary prevents two common mistakes. Re-encoding or otherwise changing the request before signature verification can discard the exact bytes needed for the check. Treating a provider-shaped body as authenticated lets an attacker manufacture a convincing mention, issue update, or deployment event.

For a normal provider-webhook dispatch, 202 Accepted means verification and asynchronous handoff succeeded. It does not prove that enrichment produced a Signal, routing created or updated a Task, or Charlie completed any work.

Normal provider-webhook dispatch outcomes

Outcome	Meaning
Verification fails or request is stale	The delivery is not trusted and does not enter the work pipeline.
202 Accepted	Verification and asynchronous handoff succeeded. Later Signal creation, Task effects, and completion are not guaranteed.
Accepted, then filtered downstream	A trusted delivery can still prove unsupported, duplicate, irrelevant, or outside activation policy, producing no scheduler work.
Retryable dispatch failure	Asynchronous handoff did not succeed, so the provider should retry according to its delivery contract.

Ingress then creates a provider-neutral envelope. The envelope preserves what was delivered rather than trying to decide the final work outcome at the edge.

// Production field names and nesting; values are illustrative.
const envelope = {
  id: 'event_example',
  provider: 'slack',
  charlieEnv: 'production',
  externalId: 'provider-delivery-id',
  eventType: 'app_mention',
  eventAction: undefined,
  requestId: 'request_example',
  receivedAt: '2026-07-09T16:20:00Z',
  path: '/webhooks/slack',
  payload: {
    channel: 'incident-channel',
    thread: 'release-alert-thread',
  },
};

Provider-specific source details stay inside the opaque, parsed payload. The envelope’s event ID and optional external delivery ID are distinct from the eventual Signal or Task identity. Together with the request ID, they let later stages recognize redelivery, correlate telemetry, and preserve the distinction between provider callbacks that happen to discuss the same resource.

Preserve first, interpret second

Provider payloads are optimized for provider APIs, not for deciding who asked for work or where a follow-up belongs. GitHub describes repositories, pull requests, comments, checks, and status events. Linear describes issues, comments, assignments, and app notifications. Slack describes workspaces, channels, messages, threads, subtypes, and bot authorship.

Enrichment converts those shapes into the compact, durable routing record introduced above while retaining access to the source material. The most useful additions are a canonical resource or thread identity, the Charlie customer, the sender, activation facts, optional event metadata, and a deterministic way to recover the stored source payload.

The envelope records what arrived. The Signal adds the facts needed to decide whether and where Charlie should act.

// Simplified: this shows the shape of the decision input, not a public API.
const signal = {
  provider: 'slack',
  eventType: 'app_mention',
  canonicalResource: 'slack-thread-identity',
  customer: 'resolved-customer',
  sender: {
    kind: 'human',
    displayName: 'release engineer',
  },
  activationFacts: {
    mentionsCharlie: true,
    authoredByCharlie: false,
    directMessage: false,
  },
  eventMeta: {
    r2Key: 'signals/v1/production/signals/signal_example/event-meta.json',
  },
};

The original webhook payload is stored separately at a deterministic artifact path derived from the environment and Signal ID. It is not a field on the Signal. The separation is deliberate: a routing decision should not need to reinterpret every provider’s raw payload, and an envelope should not pretend to know facts that require provider lookup or customer resolution.

Layer	Ownership boundary
Raw delivery	Owns request bytes, provider headers, and receipt time. Avoids assigning authenticity before verification.
Normalized envelope	Owns Charlie’s event ID, provider and environment, optional external delivery ID, event facts, request and receipt details, optional path, and opaque parsed payload. Avoids inventing the customer, sender, or canonical work target.
Enriched Signal	Owns resource or thread identity, customer, sender, activation facts, and optional eventMeta.r2Key. Source-payload recovery follows the deterministic storage convention. Avoids claiming a final routing choice or that work has begun.
Routing plan	Owns the drop, mailbox append, new Task, or daemon decision. Avoids claiming completion, semantic correctness, or global ordering.

Event metadata and the source payload are persisted before the Signal is published downstream. The Signal may carry eventMeta.r2Key; the source payload remains recoverable through its deterministic storage path. That ordering gives routing and execution a durable source to read if the inline message is intentionally small or if a later process needs details that do not belong in the routing contract.

This also establishes an important boundary for Sentry. Charlie’s current direct inbound providers for this path are GitHub, Linear, and Slack. Sentry is authenticated read-side diagnostic context, not a direct inbound webhook or Signal source. When a Sentry alert is posted to Slack, the normalized provider is Slack. A daemon or Task can then read Sentry to inspect the current issue, events, tags, and release context. Keeping that distinction explicit avoids giving read-side evidence the identity or trust semantics of an inbound provider delivery.

Filter before spending attention

Valid provider events can still be useless or dangerous as work triggers. Enrichment and routing apply filters before scheduling an agent.

Successful GitHub check and status events are common examples. They may be useful as evidence for an active Task, but a success callback rarely justifies starting new work by itself. Unsupported actions, incomplete payloads, duplicate Slack messages, and events that do not meet activation policy can also exit without producing a Signal or scheduler write.

Loop prevention needs a narrower rule than “ignore bots.” Slack integrations often post useful alerts, including Sentry notifications. The light Slack filter drops Charlie-authored message and app_mention events to prevent self-loops. It deliberately does not reject bot-authored messages; later routing and activation policy determine whether a bot or integration event can produce work.

Filtering is also where source-specific safety facts become useful. An explicit human mention can activate normal routing. A channel message without a mention may be irrelevant to the general agent but still match a narrowly scoped daemon. A follow-up in a known thread can matter even when it repeats no command, because its resource identity may match an active Task.

Keep payload size separate from work meaning

Webhook payloads can include long message blocks, review bodies, attachments, and provider metadata. Queue and model contexts have finite budgets, and those budgets should not decide whether a delivery is meaningful.

Charlie keeps the routing contract bounded. When a provider payload is too large for the transport path, the original can be moved temporarily out of line and the downstream message carries enough information to rehydrate it. During enrichment, the source payload and event metadata are persisted before publishing the Signal.

The order matters:

1. Preserve the trusted source material or a recoverable path.
2. Build the compact facts needed for routing.
3. Persist event metadata and the source payload.
4. Publish the Signal for orchestration.

A compact Signal carries the routing facts required for the next decision, optional event metadata, and enough identity to recover the source payload by convention.

At-least-once delivery requires idempotent effects

Providers and queues can redeliver. A process can also persist an effect and fail before acknowledging the message that caused it. Retries are necessary for availability, so duplicate delivery must be treated as normal operation.

Charlie has at-least-once components. It does not claim global exactly-once processing or strict global ordering across providers, queues, routing, the scheduler, and external systems. Under backlog, even work for one customer may not retain strict serial order.

The practical goal is convergence. Stable business identities and operation-specific idempotency keys let repeated attempts resolve to one intended scheduler effect. A redelivered mention should not create a second root Task when it represents the same routing action. A repeated follow-up should not append the same mailbox item twice.

Duplicate attempts converge at defined effect boundaries. Other branches may retry, drop, or no-op; the system does not claim end-to-end exactly-once execution.

There are several identities in play, and combining them carelessly causes bugs:

Identity	Purpose
Provider delivery	Recognize and trace a provider callback or redelivery
Canonical resource/thread	Determine which issue, pull request, Linear item, or Slack thread is meant
Signal	Identify one enriched routing input
Task	Own the engineering objective over time
Effect idempotency key	Make one scheduler operation converge across repeated attempts

These identities do not need to be equal. Two Slack messages in the same thread are distinct deliveries and may become distinct Signals, yet both can route to one active Task. Conversely, one provider redelivery should not become a second logical mailbox append merely because it was received again.

Routing decides what kind of work this is

Once a Signal has work-aware facts, optional event metadata, and a recoverable source payload, the orchestrator can compare it with active work and policy. The result is one of four broad outcomes:

• Drop the Signal when it is unsupported, unsafe, duplicate, or not actionable.
• Append a mailbox message to an existing Task when the Signal is relevant follow-up context.
• Create a new Task when the Signal describes a new owned objective.
• Route to a daemon when a specialized, pre-scoped workflow owns the event.

Return to the release incident. The first Slack mention asks Charlie to investigate and can create a new Task. A later mention or follow-up about the feature flag can resolve to the same thread. If routing selects the active Task, the new message is appended to its mailbox and becomes input at a safe execution boundary instead of opening a parallel investigation.

When a Sentry alert arrives through Slack, a narrowly scoped triage daemon may be eligible only if its Slack activation policy matches. The Signal remains Slack-originated, and the daemon reads Sentry as diagnostic context. For a general human mention, routing may instead create or update a normal engineering Task. The same provider surface can support both paths because activation facts, resource identity, and routing policy remain separate concerns.

Task creation and mailbox append also fail differently. Creating a Task establishes a new durable owner. Appending to a mailbox modifies an existing owner that may have become terminal between candidate lookup and the write. Routing needs to preserve that race and apply a bounded fallback only when it has enough evidence. Blindly converting every failed append into a new Task would turn ordinary timing into duplicate work.

Scheduler writes carry idempotency keys derived from the source identity, operation, and routing action. That protects the defined scheduler boundary when queue delivery repeats. It does not make later tool calls or provider writes exactly once; those effects still need their own keys, read-back, or reconciliation policy.

Work starts with a trustworthy handoff

The model enters after the system has authenticated the delivery, preserved its source facts, resolved the work-aware context needed for routing, and filtered noise and self-trigger loops. Oversized evidence remains available through durable references, repeated attempts converge at defined effect boundaries, and routing has selected an existing Task, a new Task, a daemon, or no work at all.

That handoff is what lets the Task model from Chapter 2 work across GitHub, Linear, and Slack. A Task can own a long-running objective only when new input arrives with enough identity to find it, enough evidence to trust it, and sufficiently narrow guarantees to recover when delivery repeats.

Continue the series

Previous: How We Built Charlie, Part 2: The Task, Not the Chat. Next: How We Built Charlie, Part 4: Routing Is Ownership. Browse the full How We Built Charlie series.