OpenSwitchboard

Appearance

Run history ​

Every run becomes a durable record — identity, timing, terminal status, the redacted event stream, and the friction diagnosis — kept for a retention window instead of evicted 60 s after finish; the record is tombstone-first (item 27), so even a run cut down before finish leaves an interrupted trace. The record is built the moment a run finishes and written after the reply, so persistence never delays the user; /runs/:id serves a finished run from the store through the same run page the live view uses (the web shell + a history seed, live-view.md), and the runs.* commands read every run — live or persisted — through one RunsService. This file covers the node-free record contract both the bot and the state Worker import (shape, validator, the one retention function both sides run, the byte budget), the RunStore seam with its three implementations (in-memory, host-disk file directory, and the RunHistoryDO on the state Worker), the runHistory config section, the dispatcher's write path, the RunsService read merge, and the friction ledger served from the store. The commands over this service are command-registry.md; the page is live-view.md.

Behavior ​

The record ​

  1. Node-free contract. src/core/runRecord.ts has no Node built-in imports and no I/O or clock: bytes are measured with TextEncoder, and callers pass nowMs. The Cloudflare state Worker imports it by relative path, the same way it imports frictionProposals.ts, so the bot and the Worker validate, trim and page with ONE implementation.
  2. Record shape. RunRecord = { id, label?, agent?, model?, channelId, userId, threadKey, channelVisibility, repo?, startedAt, finishedAt, receivedAt?, sealedAt?, replyOk?, stepCount?, schema?, status, eventCount, storedEventCount, truncated, events, diagnosis } (the five optional stamps per tracing.md: stepCount is the registry's content-event count, span records excluded; schema the stream schema, 2 once spans are emitted) with status ∈ completed | stopped_soft | stopped_hard | failed | interrupted and channelVisibility ∈ public | private | dm | machine | unknown — the channel's visibility as the ChannelDirectory reported it at dispatch (authorization.md item 7), what member-of reads; a record written before the stamp existed is accepted and reads as unknown (normalizeStored), never public — interrupted marks a run cut down before finish (container replaced or crashed; the provisional tombstone and the drain-deadline write of item 27 are its only writers). eventCount is what the run published; storedEventCount is events.length; truncated says the two differ (the registry backlog or the byte budget dropped events). Every stored event keeps the registry's seq (StoredRunEvent; storedEventSeqs falls back to positions only for a record whose events arrived without strictly increasing stamps). RunListItem is the record minus events plus an optional bytes (the stored JSON size) — diagnosis stays on the list item so the friction ledger can be served from a listing without loading events.
  3. Structural validator. isRunRecord(x) accepts only an object whose id matches RUN_ID_PATTERN (^[A-Za-z0-9_-]{1,64}$), whose identity/timing/count fields have the right types, whose status is one of the five, whose events is an array of objects each with a string type (the union grows over time — an older reader must still accept a newer record, and the run-page fields callId/exitCode/output/source ride through verbatim), and whose diagnosis is structurally a diagnosis (isStoredDiagnosis: a byCategory map of { count, durationMs } totals — NOT the current category list). Readers zero-fill categories a stored diagnosis lacks and drop unknown ones (normalizeDiagnosis), so adding a friction category never invalidates a stored record. Anything else — null, a bad id, an unknown status, an event without a type — is rejected. isRunListItem is the same check minus events, plus an optional numeric bytes.
  4. Retention policy. RetentionPolicy = { retentionDays, maxRuns, maxBytes }, default { 30, 5000, 2 GiB }. clampRetentionPolicy(partial) fills missing fields from the defaults, replaces non-finite values with the defaults, floors fractions, and clamps into RETENTION_BOUNDS: retentionDays [1, 365], maxRuns [1, 20000], maxBytes [16 MiB, 8 GiB].
  5. One retention function. applyRetention(items, policy, nowMs) drops items with finishedAt < nowMs − retentionDays, keeps the newest maxRuns by finishedAt desc (tie-break id desc — a total order, so bot and Worker cut identical rows), then drops the oldest while the cumulative bytes of the kept set exceeds maxBytes (missing bytes counts as 0). Returns the kept items newest-first and never mutates its input.
  6. Byte budget. fitRecordToBudget(record, maxBytes = MAX_RECORD_BYTES = 1.5 MiB) first caps each event to MAX_EVENT_BYTES (64 KiB) by truncating its text (input/context/assistant/answer events) or summary (tool/note events) with a trailing …, measured in UTF-8 bytes. If the record is still over budget, span records go first (tracing.md — spans displace no content): pair by pair (both records of one spanId) from the middle of the stream outward, never from the protected head (the leading run of head material), until the record fits or no span outside the head remains; a dropped tool.*/mcp.* twin is re-synthesized by normalizeSpans from its content pair. Only if that was not enough are content events dropped from the middle: a head and a tail are grown alternately from the two ends (head first) until the next event would not fit, so the request/context/first tool steps and the terminal notes survive. The result has truncated: true, eventCount unchanged, storedEventCount = events.length; an under-budget record comes back with its events intact and truncated untouched. The source record is never mutated.

The store ​

  1. RunStore seam. RunStore = { put, get, getSummary, list, events, delete } with put → { ok, retained, stored, rewritten } (stored: false when the record fell outside policy in its own write; rewritten: true when an existing record with that id was replaced by a different one). Every implementation rejects an id failing RUN_ID_PATTERN before touching storage — a bad id is not-found (get → null, getSummary → null, events → null, delete no-op), never a path or a request. getSummary(id) is the record minus its events — the listing row, bytes included — for callers that need identity, status, or the diagnosis but not the event set; it hides exactly what list hides. list is newest-first (finishedAt desc, id desc), default 50 rows (RUN_LIST_DEFAULT_LIMIT), capped at 200 (RUN_LIST_MAX_LIMIT), with the compound cursor { before, beforeId } (finished_at < ? OR (finished_at = ? AND run_id < ?) — same-millisecond siblings across a page boundary are never skipped) plus sinceMs/agent/channel filters and the caller's visibleTo — the authorization predicate in its wire form (RunVisibilityFilter: all | none | channels-in | user-is | repos-in | visibility-in | or | and, authorization.md item 6), ANDed with the other filters and answered by every store with the one truth table matchesVisibility (none matches nothing, a row without the stamp is unknown) — and never carries events; a full page returns nextBefore. events(id, { afterSeq, limit }) pages a record's events by their stored seq (strictly seq > afterSeq), default 1000 per page, capped at 5000, returning nextAfterSeq while more follow; an unknown or expired id is null, a run with nothing past the cursor is { events: [] }.
  2. In-memory and file stores. InMemoryRunStore (tests/dev). FileRunStore is a directory store and an explicit opt-in (store: "file"): data/runs/<id>.json per run, written temp-then-rename with mode 0600 in a 0700 directory, plus data/runs/index.jsonl of RunListItems with bytes — list and getSummary read only the index and get opens exactly one file. Retention is applied on every read (expired rows hidden) and on every write (files unlinked, index compacted to the kept rows); sweep() does the write-side work without a new record and buildRunStore runs it on start and every 6 h from an unref'd timer. A torn record file (size ≠ indexed bytes, or unparsable) is skipped by get, absent from list and null from getSummary (the two index readers apply the size check; get opens the one file instead of stat'ing them all — an unparsable file is the same not-found); an index line whose file is gone is hidden and dropped at the next put. Shrinking then growing retentionDays never resurrects a deleted record — the file is gone.
  3. Worker client. WorkerRunStore speaks POST /runs/put|get|summary|list|events|delete to the state Worker with { storeKey: "runs:default", … }, a STRING JSON body (the runtime derives the numeric Content-Length the Worker requires; the client never sets it by hand — the one header the memory/friction/schedule clients omit, and hand-setting it makes every request from the container end in fetch failed), authorization: Bearer <MEMORY_TOKEN>, and a 10 s AbortSignal.timeout. Only put carries policy + policyUpdatedAt (the bot's config load time); get/summary/list/events/delete never do. Responses are re-validated (isRunRecord / isRunListItem, diagnosis normalized); a malformed body is a PermanentStoreError. HTTP 404 → RouteMissingError (the route does not exist — the deployed Worker predates the run-history routes; never retried); 5xx/408/429/network → TransientStoreError (bounded retries); any other 4xx → PermanentStoreError.
  4. Config and selection. runHistory: { retentionDays?, maxRuns?, maxBytes?, includeContext?, store?: "worker" | "file", worker?: { baseUrl, tokenEnv? } } (includeContext, default true, read as !== false: publish the thread-context turns fed to the model into the run stream — live page and persisted record alike; run-visibility.md item 6). validateConfig rejects retentionDays/maxRuns < 1, a non-https worker.baseUrl, and an unknown store. buildRunStore(cfg, env, { dataDir, warn, now?, setInterval? }): no section → null (history OFF: runs stay live-only); store: "file" → FileRunStore under <dataDir>/runs; worker with its bearer (tokenEnv, default MEMORY_TOKEN) → WorkerRunStore; worker without the bearer → null plus a warning naming the env var (history off rather than a silent host-disk fallback). retentionPolicyOf(cfg) is the clamped policy the bot proposes and the live view's retention sentence reads.

The state Worker ​

  1. RunHistoryDO (state Worker). One SQLite Durable Object per store key: tables runs (listing columns including channel_visibility TEXT NOT NULL DEFAULT 'unknown', bytes, stored_at, diagnosis_json, summary_json = the record minus events), run_events(run_id, seq, json, PRIMARY KEY(run_id, seq)), meta(key, value); indexes on finished_at and, for the visibility predicate's leaves, (channel_id, finished_at DESC, run_id DESC), (channel_visibility, finished_at DESC, run_id DESC), (user_id, finished_at DESC, run_id DESC). The constructor runs the DO's one migration: a runs table created before the stamp gains channel_visibility with unknown for every existing row. put is ONE transactionSync: clamp finishedAt to now + 24 h and record stored_at; upsert runs; skip the event rewrite when the stored version is unchanged (event_count, finished_at, bytes equal → rewritten: false), else DELETE run_events WHERE run_id then batched INSERT (33 rows per statement); trim by the PERSISTED policy from both tables, oldest first, at most 500 rows per put (deletion fence, logged) — the just-written record is deleted regardless when it is outside policy (stored: false); orphan sweep; retained = rows the policy keeps. A failure anywhere inside rolls the whole put back (no runs row, no events). alarm() every 6 h (armed on the first put) deletes everything outside policy with no fence. Reads (get, summary, list, events) apply the persisted policy first, so rows still on disk behind the fence are invisible; a corrupt run_events row is skipped. summary reads the runs row alone (no run_events). list decides its path from ONE aggregate over the in-cutoff rows (COUNT(*), SUM(bytes) where finished_at >= cutoff): within maxRuns/maxBytes every in-cutoff row is kept, so the page is ONE indexed query — age cutoff, agent/channel filters, the visibleTo predicate compiled to SQL (visibilitySql: channel_id IN (…), channel_visibility IN (…), user_id = ?, repo IN (…), or/and as parenthesised groups; none is an empty page without a query, all adds nothing; a malformed filter or one binding more ids than the DO's 100 bound parameters allow is 400), the cursor predicate, ORDER BY finished_at DESC, run_id DESC, LIMIT — and no table scan; only when a bound is exceeded is the kept set computed (applyRetention over every (run_id, finished_at, bytes)) and the ordered rows walked until the page fills, stopping at the first row outside it (the kept set is the newest prefix). delete removes the run and all its events.
  2. Policy ownership. meta holds { retentionDays, maxRuns, maxBytes, policyUpdatedAt }, clamped with clampRetentionPolicy. Only /runs/put may carry a proposal, accepted when min(policyUpdatedAt, DO now) is strictly newer than the stored stamp — so a proposal dated a year ahead is stored with the DO clock and a later, correctly dated one still wins. A proposal with retentionDays/maxRuns/maxBytes < 1 is 400. get/list ignore any policy in the body. Cutoffs use the DO clock.
  3. Routes and fences. /runs/put|get|summary|list|events|delete require the bearer; unknown route 404, non-POST 405, missing/non-numeric Content-Length 411 (unchanged global rule). The body cap is per route, decided after routing and before request.json(): 2 MiB for /runs/put, 512 KB for everything else, measured in bytes (a 1.9 MB multibyte body is accepted; cap + 1 byte is 413). Unknown or expired id → { record: null } / { summary: null } / { events: null } with 200 (a 404 always means the route is missing). list accepts any limit but returns ≤ 200 rows plus nextBefore; a malformed beforeId is 400. GET /healthz → { ok: true, features: ["memory", "friction", "schedules", "runs"] }. wrangler: migration v4 (new_sqlite_classes: ["RunHistoryDO"]), binding RUNS. The Worker must be deployed before the bot version that writes to it — the bot's boot probe (item 18) and RouteMissingError (item 17) make the wrong order loud and non-destructive.
  4. Cloudflare DO SQLite limits (pinned in worker.ts from the platform limits page). 100 bound parameters per query → 33 event rows per INSERT (3 parameters each) and 100 ids per DELETE … IN (…); 100 KB per SQL statement; 2 MB per string/BLOB/row (an event is capped to 64 KiB upstream); 100 columns per table; 10 GB storage per object (Workers Paid) — maxBytes is clamped to 8 GiB, under that ceiling.

The write path ​

  1. Record built at finish, inside the run's try/catch. The dispatcher computes the terminal status first and hands it to registry.finish(id, status) — the registry stores it and every RunSummary projects status + finishedAt, so no consumer re-derives them (the one status the registry cannot know is a reply that throws after the loop: the record says failed, the registry row keeps completed for its TTL). Every record of a finished run is written by the dispatch's drain AFTER the run's seal (src/core/runEnding.ts; live-view.md item 4): assembleRunRecord appends the events the seal returned (the span records published between finish and seal), takes the larger published total, stamps sealedAt and replyOk, then fits the budget — so the record's duration and delivery caption equal the live page's, and a command run's record carries the stamps too. A reply that throws seals replyOk: false and flips a completed record to failed; a fenced run (another generation's) writes no record here but is still sealed by the backstop; a command run's status is its command's ok and never flips. Right after finish, the dispatcher reads registry.snapshot(id, token) ONCE — the same read feeds the friction diagnosis — and builds the RunRecord synchronously: identity (channelId, userId, threadKey, agent, model = the resolved <provider>/<model>, the registry's redacted label, repo when resolved), startedAt (the registry's create-time clock) and finishedAt, terminal status (failed when the runner or the reply threw; else stopped_hard / stopped_soft from the run's stop control; else completed), eventCount = the registry's monotonic published total, storedEventCount = the snapshot length, truncated true when the bounded backlog dropped events, then fitRecordToBudget. A failed run has a record too (the outer catch writes it after the error reply). A reply slower than the registry TTL cannot lose the record: it was built before the reply. snapshot returns startedAt and eventCount alongside the events (still token-gated).
  2. Write after the reply, fire-and-forget, drain-counted. The record is handed to CoreDeps.runHistoryWriter.write() only after sendAnswer (success) or the error reply (failure), so persistence never delays the user; write never throws. pending() increments before the outer finally's activeRuns-- and is part of the shutdown drain in src/index.ts (with the existing 15 min deadline), which also logs failures() at exit. Without a writer (history off, most tests) nothing is written and the run is unchanged. src/cli.ts builds the same store/writer and awaits settled() before exiting.
  3. Retry policy. createRunHistoryWriter({ store, warn, onPersisted?, sleep?, random? }) retries twice with jittered backoff (nominal 1 s then 4 s, each ±50%) on a TransientStoreError (network, timeout, 408/429/5xx) or any unclassified error; never on a PermanentStoreError (4xx such as 413, malformed response — one warn, failures() + 1); a RouteMissingError (404 — the deployed Worker predates the run-history routes) logs [run-history] state Worker has no /runs/put — deploy the state Worker with run-history routes before this bot version ONCE per process, sets degraded(), never retries, and counts every loss. Retries exhausted → one warn naming the run id and attempt count, failures() + 1. On success onPersisted(id) runs (exception-isolated) — the dispatcher wires it to registry.markPersisted, which sets persisted: true on the run's RunSummary (the key is absent until then) and emits one index upsert; a no-op for an unknown or already-evicted run. A 404/413 on /runs/put loses that run for the friction ledger too — it reads run history and has no other source.
  4. Startup. src/index.ts builds the store with buildRunStore(cfg.runHistory, env, { dataDir: "./data", warn }), logs which store was selected (host-disk file (data/runs), durable Worker (<baseUrl>), or off … live-only), serves the friction ledger through selectFrictionLedger(store) (none without a store), and — for the Worker store — probes GET <baseUrl>/healthz best-effort at boot: a features list without runs logs an ORDERING ERROR line; the probe never blocks or fails startup.

Reading runs: RunsService ​

  1. One service, token-free views. createRunsService({ registry, store, analyze? }) is the one async service behind every runs.* command and the live view; store: null means history is off (live-only). Every result is a Result<T> = { ok: true, value } | { ok: false, error: "not_found" | "conflict" } and NO output carries a run's capability token: live registry rows are projected field-by-field into RunView (id, label?, agent?, model?, channelId?, userId?, threadKey?, repo? — the RunMeta the dispatcher passes to registry.create(label, meta) — plus startedAt, finished, eventCount, stop?, persisted?, and for a finished registry run the finishedAt/status given to finish()) and a persisted RunListItem becomes the same RunView shape with finished: true, persisted: true, so a run in both sources projects identically except for the finish-only fields (finishedAt, status, storedEventCount/truncated/bytes, diagnosis). Callers are authorized one layer up (command scopes/chat gates + Access); the service only assumes it.
  2. Read merge. listRuns({ status, visibleTo, agent?, channel?, sinceMs?, limit?, before?, beforeId? }): limit default 50, cap 200 (the shared RUN_LIST_* limits). visibleTo is REQUIRED — the caller's authorization predicate (predicateFor(actor, "runs:read", "run"), authorization.md item 6; { kind: "all" } is an explicit choice, never a default): live rows are filtered by the reference evaluator matchesPredicate, the store receives its wire form as visibleTo (omitted for all), and none returns an empty page without touching either. active = registry runs not finished — never touches the store. finished = finished registry runs ∪ store.list; all = every registry run ∪ store.list. Persisted rows are fetched with the same agent/channel/sinceMs filters and a bound of min(200, limit + liveCount). Every view carries the run's channelVisibility (from the RunMeta live, the record persisted; absent on a hand-built live row) so the runs.* commands authorize a point read on the view itself. A run in both sources is ONE row. An UNFINISHED live row wins whole, and its store row is suppressed from every listing (finished included): that row is the run's provisional interrupted tombstone (item 27) — the truth only once the run is dead — so a live run always lists as live, never as interrupted. A FINISHED live row is merged { ...persisted, ...live }: the live row wins every field it carries (the final stop state, the registry's persisted flag) except the finish fields — the record's finishedAt/status win (it is the source of truth: a reply that threw after the loop is failed there while the registry row keeps completed) — and the store row contributes what only the record knows (diagnosis, bytes). The record's finishedAt IS the registry's finish clock (RunSnapshot.finishedAt), so the two agree whenever both are present. Rows sort by the store's own key — unfinished rows first (newest started first), then finishedAt desc, then id desc — so the page is a true top-N of the union and a finished registry run sits exactly where its record will; limit applies after the merge. A full page ending on a persisted row returns nextBefore: { finishedAt, id }; passing it back as before/beforeId pages the store with the compound cursor and omits live rows (they all sort ahead of any cursor and were on the first page). Live rows carry their RunMeta, so agent/channel filters apply to them exactly as to persisted rows (a live row created without meta is excluded by those filters); sinceMs compares finishedAt ?? startedAt. A store that throws degrades to { runs: <live rows>, storeUnavailable: true } — never a whole-command failure.
  3. Single-run reads. getRun(id, { include? }) reads the registry first (token-free getById/snapshotById), then the store — store.getSummary without include (metadata only, the event set is never loaded), store.get with include: "messages" (events present). getRunEvents(id, { afterSeq?, limit? }) returns events with seq > afterSeq (strict), at most 500 per page and ≤ 256 KiB of event JSON (always ≥ 1 event), with nextAfterSeq = the last returned seq while more follow; live from the registry backlog, persisted via store.events (never a full-record fetch). seq is the registry's stamp on BOTH sides, so an afterSeq cursor taken from the live stream addresses the same events after eviction. store.events → null is not_found; an existing run with nothing past afterSeq — a zero-event record included — is ok with { events: [] }. getRunFriction(id) → { id, finished, diagnosis }: live runs are analyzed with analyze(events, { finished, truncated }) (truncated = the snapshot's flag, item 5 of run-visibility.md), persisted runs return the stored diagnosis from store.getSummary — never the events. stopRun on a run outside the registry decides 409-vs-404 from store.getSummary too. A store failure inside listRuns degrades to live rows with storeUnavailable: true and ONE warn line carrying the error message (never a token) per failing call (createRunsService({ …, warn? }), default console.warn). Unknown, malformed, expired (retention-hidden) ids are not_found everywhere.
  4. Stop with actor. stopRun(id, mode, actor) calls the token-free registry.requestStopById(id, mode, actor); a live run → { id, mode, state: "stopping" } and a stop_requested run_note carrying actor: { kind: access | mcp | cli | chat, id } with id stripped to ^[A-Za-z0-9:@._-]{1,128}$ (unknown when nothing survives); a finished-but-unevicted run and a persisted run → conflict; anything else → not_found. The token-gated requestStop (the HTML button) publishes no actor.
  5. Registry capability model. RunRegistry enforces capability OR operator: the token-gated has/subscribe/snapshot/requestStop stay for the live HTML/SSE routes (defense in depth), and the token-free getById/snapshotById/requestStopById serve the service. authorizeLive(id, token) is synchronous and returns a LiveRunAccess (subscribe, snapshot, requestStop) bound to the registry when the token is right for a non-evicted run, else null — the live SSE path stays byte-identical through it.

Consumers ​

  1. The friction ledger is the store. RunStoreFrictionLedger(store) implements FrictionLedger: recent() pages store.list (never get) down to limit ?? 500 rows, projects only { runId, label, agent, finishedAt, diagnosis }, and orders oldest-first with the same runId tie-break as the in-memory test double; the caller's predicate (LedgerReadOptions.visibleTo) reaches store.list({ visibleTo }) in its wire form and none reads nothing. It never writes: the dispatcher's history write path is the one place a diagnosis is stored. See self-improvement.md.
  2. Live view on the service. src/channels/liveView.ts takes createLiveViewHandler({ shell, service, index: { listActive, subscribeIndex }, retention, audit? }) — shell is the bound web-shell renderer and the handler emits page seeds (see live-view.md's Rendering paragraph): every per-run read and the tokenless stop go through RunsService (a valid token → the synchronous authorizeLive live path; the page's Stop/Kill stay token-gated through LiveRunAccess.requestStop), the default /runs view is served from the registry's index face alone, and ?all=1 merges one index page of listRuns({ status: "all", limit: INDEX_PAGE_SIZE, before?, beforeId? }), attaching the registry token only to unfinished rows in the seed (an Older runs → link carries the cursor on). History-mode page, stored replay + end (from the same single getRun read), 409 stop, the omission marker, the retention tooltip, feed semantics and the audit line are live-view.md items 15–16. src/index.ts passes retentionPolicyOf(cfg.runHistory).retentionDays (null with no store), the viewer's Actor (accessActor: the browser session access:<sub> — a service token, access:svc:<common_name>, can only ever reach /api/*, see command-registry.md item 12) that the index's predicate and every tokenless run read decide on (authorization.md items 5–7), and a dev-bypass gate that is loopback-only on a localhost PUBLIC_BASE_URL (isLocalhostBase).

Deploy race ​

  1. Deploy race (design gap, partially mitigated). A run that starts in the gap between the bot deploy preflight's "no runs in flight" check and the container's SIGTERM is unprotected: it gets only the drain window (15 min; a coding run's budget is 45 min), and while the old container drains it holds the slot with its Slack socket closed, so followers wait up to the drain deadline (mentions are recovered late by the reconnect catch-up). Preflight cannot lock that window; only a resumable run can close it. The durable run record is the vehicle: it must carry enough for the NEXT container to either resume the run or fail it explicitly with a card edit — never a frozen card. Phase 1 delivers: the record shape (identity, thread, agent/model, full event stream with seq, status) sufficient to reconstruct the conversation, and a record written for every run that reaches finish (including a drained one that finishes or is killed inside the window, as failed). Phase 2 delivers (item 27): the provisional record at run START — written as terminal interrupted, not running, so a run cut down before finish leaves a durable trace with NO store-side fixup by the next container. The resume step (the next container picking a cut-down run back up) is items 36–38 (0019); the store's half of it is the live-run ledger, items 28–34 below — the earlier reading that the store needed nothing for it was wrong: a resume needs a lease, the transcript the model actually saw, and a record of what was in flight. Interim mitigations: the deploy preflight, the visible drain and the orphaned-card sweep.

  2. Tombstone-first records. A run's record is written twice. (1) At START — when the run loop takes the run, after the workspace attach; the registry row, the stream and the initial input/run_meta/context events date from the reservation (item 42), and a dispatch that ends before the run loop leaves no record at all — the dispatcher writes a provisional record from the run's current snapshot with status: "interrupted" and finishedAt = startedAt (provisional on purpose: after a crash nobody knows the real death time, so the tombstone keeps the start time). Because this record is already TERMINAL, a crash or a drain-abandonment needs no next-container reconciliation — the tombstone is already the truth, and runs list --status finished shows the lost run once the registry no longer holds it. The write goes through the same runHistoryWriter fire-and-forget (retries + drain accounting), never delays the first model call, and is provisional: it must NOT trigger onPersisted/markPersisted — the index's persisted flag means "finished and durably stored". (2) At FINISH, the item-15/16 write replaces the tombstone whole (same-id upsert; sameStoredVersion differs, so the Worker rewrites the event set). Final beats provisional: the writer stands a provisional write down the moment a final (non-provisional) write for the same id is enqueued — before every attempt, retry backoff included — and drops outright a provisional write enqueued after the final one; a stood-down write is not counted a loss. Without this, a start tombstone sitting in retry backoff when a fast run finishes, or a drain write racing a run that finishes inside the drain's write budget, could land after the finish record and overwrite it with interrupted. Between the two writes, the drain in src/index.ts upgrades what it abandons: when the drain deadline passes with runs still in flight, writeAbandonedRunRecords gives each still-active registry run an interrupted record built from its FULL snapshot (interruptedRunRecord: every event published so far, finishedAt = now), logged as [drain] wrote interrupted record for <id> (<n> events) and given a bounded ~10 s budget before process.exit — so the common abandonment carries a full transcript, not just the start-of-run tombstone. The drain writes are provisional like the start tombstone (the persisted flag would be a lie on a run that never finished, and the flag lives in a registry the process is about to drop) and therefore also stand down for a racing finish. While the run is alive its tombstone never surfaces: the RunsService merge suppresses a store row whose run is unfinished in the registry (item 20) — and, with the run ledger on, a store row whose run the ledger holds live (item 41), so a run live under ANOTHER generation is never shown as interrupted here. The tombstone itself stays with the ledger on: it covers the window before the claim (registry.create to the claim after the workspace attach is seconds) and the no-ledger mode, and the reclaim's finished record (item 36) replaces it whole like any finish. Inline command runs skip the tombstone: they are sub-second and their loss window is negligible. NOTE deploy ordering: the state Worker validates status through the shared runRecord.ts, so it must be redeployed (to accept interrupted) before a bot that writes tombstones — the usual Worker-before-bot rule (item 13).

  3. The live-run ledger — one seam, two implementations (0001, 0019). A run that must outlive the bot container leaves its live state on the state Worker: RunLedger (src/core/runLedger/ledger.ts) is what the bot calls — claim, seed, step, heartbeat, append, setState, pushInbox, requestStop, handoff, finishing, finish, reclaim, listLive, readTranscript — with WorkerRunLedger (HTTPS to the /runs/* ledger routes, the same bearer and error classes as WorkerRunStore, a 409 as a RESULT never a retry) and InMemoryRunLedger (the reference the bot's tests run against). Every decision is a pure function in src/core/runLedger/decisions.ts, imported by the Durable Object and the in-memory ledger alike, so the two agree by construction. Every owner write carries the bot generation as a fencing token (gen: process start plus a random suffix): checkFence refuses a write from a generation that does not hold the lease with 409 {ok:false, reason:"fenced"} (an unknown run answers unknown-run), so a zombie old container can never overwrite a run the next one resumed. Off-state: no runHistory config → no ledger (this item is inert; the bot behaves as before).

  4. One live run per thread, enforced by the store. POST /runs/claim {storeKey, run} inserts a live_runs row (run_id, thread_key UNIQUE, owner_gen, lease_until, started_at, phase, stop, meta_json, card_json, system_text, tools_json, state_json) on the RunHistoryDO; a second live run on the thread is 409 {ok:false, reason:"thread-live", live:{runId, agent, startedAt}} — what the steer message needs — and the owner's re-claim of the same run is idempotent (decideClaim). Live runs never enter the runs table: that table's finished_at drives retention and listing, so a live row there would list as finished and be trimmed first; POST /runs/live {storeKey} is how they are read, and the finished-runs routes (get, summary, events, list) answer for a live run as for an unknown id. The row carries what a resume needs and the dispatcher otherwise keeps in closures: the card {channel, ts}, the composed system prompt and the serialized tool definitions verbatim (a resume is a valid continuation on what the model saw, not a byte-identical request across a code deploy), the executor selection and worktree path, ref and head sha, and state_json for the dispatcher-local callbacks (verdict, PR description, checklist, pushed branch, review head, infra counters; POST /runs/state).

  5. The event stream appends as it happens (POST /runs/append {storeKey, runId, gen, events}): rows into the existing run_events(run_id, seq) under the registry's own seq (INSERT OR REPLACE, so an identical retry is a no-op), each event under MAX_EVENT_BYTES or the batch is 400. The bot batches through createAppendFlusher (src/core/runLedger/flusher.ts): every APPEND_FLUSH_MS (500 ms) or APPEND_FLUSH_EVENTS (32), whichever first; a push never blocks or throws; a failed batch is reported and dropped, because the transcript (item 32), not the event stream, is what a resume is rebuilt from, and the finish record still carries every event. The events appended since the last flush are the only loss a hard kill can cause.

  6. The step record is written before a step's tools run (POST /runs/step {storeKey, runId, gen, record}; run_steps(run_id, step) keyed by step number, replaced on retry): {step, seq, turnIndex, inFlight:[{callId, tool}], inboxConsumedSeq, remainingMs, turn, iteration}. The client's order is part of the contract — the step's transcript turns (item 32) land FIRST, the record second — so transcriptCompleteness decides what a resume does from the turn count alone: the count equals the last record's turnIndex (or the seed's turn count with no record yet) → resume there and settle its inFlight calls; two more → the next step's turns landed but its record did not, so nothing of it was dispatched and its tools simply run; anything else is a partial write and the run closes interrupted with every event held. heartbeat (POST /runs/heartbeat, every HEARTBEAT_MS = 10 s) extends lease_until by leaseMs iff the caller owns the run and answers the stop any generation requested (POST /runs/stop {runId, mode} sets it from anywhere and says whether the owner's lease is live) and the row's phase. Reclaim (POST /runs/reclaim {storeKey, gen, now, leaseMs}, called once per boot BEFORE the Slack socket opens) atomically takes every row whose lease is past (lease_until ≤ now) or whose phase is handoff, sets the new owner and lease and phase: live, and answers each with its last step record, the inbox items past inboxConsumedSeq, its state and its jobs; a live lease is left alone.

  7. The transcript is stored whole, one row per content part, and never edited. The model's thinking blocks are verified by signature and a conversation whose earlier turns changed is rejected, and the persisted events cap tool outputs and omit tool inputs, so a resume needs the raw ChatMessage[] the runner had. It lives in RunTranscriptDO, ONE object per live run named by the run id (RUN_TRANSCRIPTS binding, migration v6), kept apart from the history object so transcript bytes never queue behind an admission decision: run_messages(idx, part, json) (one {role, part} row per content part; turnRows refuses a part over TRANSCRIPT_PART_BYTES = 1.5 MB by name rather than truncating), attachments(ref, media_type, data) for base64 image/document data over ATTACHMENT_REF_BYTES = 1 MB (stored once, the row carries dataRef), and an owner(gen) row — the object's own fence, set at claim and replaced by reclaim, because this object and the history object commit independently and a zombie whose history write is about to be refused must not land transcript rows either (POST /runs/transcript/owner|write|read|clear; write before an owner is set is unknown-run). The client writes the seed once at start (seed) and, with each step, the previous step's results turn and this step's assistant turn (chunkRows keeps each request under the 2 MiB fence; attachments travel one per request). assembleTranscript rebuilds the exact array in (idx, part) order, re-inflating references, and names the first gap (turn N is missing, turn N is missing part M, attachment R is missing) while returning the turns before it, so an interrupted record can still carry what was stored.

  8. Phases and finish. phase is a compare-and-swap table (phaseTransition): live → handoff (SIGTERM: POST /runs/handoff {gen, runIds} marks this generation's live runs for the next one, never another generation's and never a finishing run), live → finishing (POST /runs/finishing, taken BEFORE the reply so a fenced old generation never answers a thread twice; a second finishing on the same run is 409), and back to live from either by a reclaim. POST /runs/finish {storeKey, runId, gen, record} writes the finished record exactly as put does (the same upsert, retention and policy proposal; record.id must equal runId) and deletes the run's live_runs, run_steps, run_inbox and run_jobs rows in ONE transactionSync — the reason the live tables live in RunHistoryDO and not a second object — then the client clears the transcript object best-effort (an orphaned transcript is harmless). After finish the thread is free to claim again and the record lists as finished.

  9. The durable inbox and jobs. POST /runs/inbox {storeKey, runId, message} appends a steered follow-up from any generation with an increasing seq (refused as {ok:false} for a run that is not live); the row is the IncomingMessage minus attachment bytes plus what rebuilds a channel handle, so the fresh-turn settle of unconsumed follow-ups works after a resume, and the step record's inboxConsumedSeq says which ones the run has read. run_jobs(run_id, kind) holds post-run work a handoff leaves for the next generation (one reflection per run, hence the key); both are deleted with the run at finish. Bodies: the ledger routes are validated before any object call (400 names the field), share /runs/put's 2 MiB fence where they carry a record, a transcript chunk or an event batch (/runs/finish, /runs/transcript/write, /runs/append) and the 512 KiB fence otherwise; observability lines carry ids and counts, never text.

  10. The bot's write-through (createLedgerWriteThrough, src/core/runLedger/writeThrough.ts; wired in src/index.ts beside the run store as CoreDeps.runLedger, Worker-backed history only — a store: file history has no ledger). The process mints ONE generation at boot (mintGeneration: an ISO stamp then eight random hex, e.g. 20260907T231512Z-3fa9c1d2, reported as generation on /healthz) and writes everything under it. Per run: claim once the prompt exists — after the workspace attach and the system composition, not at the in-process admission, because a row without its prompt could not be resumed (the registry row itself dates from the reservation, item 42) — with the composed system prompt and the tool definitions (mergeTools of the agent's toolset and the run's MCP tools, definitions only) verbatim, the card's message handle (StatusHandle.handle, which the Slack adapter sets to {channel, ts} and the coalescer passes through), the run meta (agent, model, effort, repo/ref/PR/head, readonly, selection, workspace), then the seed: the exact messages the first model call carries, followed by the seed record — step 0, nothing in flight, turnIndex = the seed's length, remainingMs = the agent's whole budget (budgetMs) — so a reclaim always has a record to judge the transcript against and a row with no record at all is known to have died before its conversation was stored. The claim is awaited (one round trip) so no step record precedes it; the whole claim is retried on a transient failure (200 ms, 800 ms) and a run is untracked — it runs and replies exactly as before, one warning — when the thread's row belongs to another run (a stale row; reclaim is item 31's), the routes are missing (warned once per process), or the claim kept failing. Each step: the runner's onStep hook (StepReport: the turns appended since the previous report — the previous step's results turn and this step's assistant turn — their first index, the calls in flight, turn, iteration, remaining wall clock) is awaited BEFORE any of the step's tools run, and the write-through turns it into item 31's step write (turns first, then {step, seq: the registry seq last seen, turnIndex, inFlight, inboxConsumedSeq: 0 until the durable inbox, …}); a transient failure is retried once after a 200 ms backoff. Events: the registry subscription feeds item 30's flusher with the registry's own seq. State: the checklist, the submit_verdict/submit_pr_description payloads and the pushed branch are merged into state_json as they happen (coalesced, newest wins; a transient failure is retried once after the same backoff, and one that fails twice leaves the state dirty so the next patch carries it — the last state of a run is never lost to one blip). Heartbeat every HEARTBEAT_MS, and a stop another generation set is relayed to the run's control once per mode. Finishing is taken before the card close and the reply (item 39: its answer gates the reply); finish goes through the writer as write(record, { via: run.sink }) — the ledger's one-transaction finish, with the writer's retries and drain accounting, falling back to the plain store when the ledger refuses (fenced, unknown-run) or lacks the route, so a record is never dropped and never written twice. The rule of this phase: a refused or failed write detaches the run (one warning … detached: <reason> — this run is not resumable; every later write a no-op, the heartbeat stopped) and never changes what the run does — except a fence, which item 39 turns into a hard stop. A ship pipeline is claimed for the live index and the finish only (system: "", no seed, no steps: each child round is its own runAgent), so it closes interrupted at a reclaim; inline command runs are not claimed. Finishing also records the status the record will carry (state.finalStatus) before the CAS, and the finish write goes out right after the reply, BEFORE the workspace release — the record does not depend on it, and on the ledger the finish is what frees the thread, which must not wait on a sandbox teardown (measured live at ~90 s).

  11. Reclaim at boot (reclaimRuns, src/core/boot.ts; awaited by src/index.ts before app.start() and repeated every lease interval — item 38). The new generation takes every row whose lease is past or whose phase is handoff (item 31's reclaim) — never a row it owns itself, whatever its lease or phase: the reclaim also runs on the sweep inside the live process, and a lapsed lease on our own row is a heartbeat that could not land (a state Worker blip), not a dead owner; taking it would launch the run a second time in the same process, and the fence, which compares generations, would never stop the first — and, unless item 38 can resume it, closes each with a record built from the ledger's own copy of the run — reclaimedRunRecord: the row's identity and meta, the events read back with POST /runs/live-events {storeKey, runId} (RunHistoryDO.liveEvents; the finished-runs reads never see a live run), eventCount = the last seq, finishedAt = the reclaim's clock — then finish (item 33), so the row, its steps and its transcript go and the record lists as finished. The status: a row taken from finishing had replied (its reply is in the thread) and closes with the finalStatus it recorded, completed when none; every other row closes interrupted — when item 38 cannot resume it — and its reason names what the completeness rule found (transcriptCompleteness against the last step record: resumable: the transcript's N turns match the last step record, resumable (next step's turns landed, record did not), a partial write, an incomplete transcript, or no step record for a row killed before its seed). One run's failure is reported and does not stop the others; its row stays this generation's and the next boot takes it again. A ledger without the routes or out of reach is one warning and the bot boots as before. Rows another generation still holds a current lease on (a rollout overlap) are left alone and handed back as liveElsewhere: the Slack adapter marks their cards (markForeignLiveCards) so the reconnect catch-up's orphan sweep (slack-channel.md item 8) asks isLiveCard — driven here, or live on the ledger elsewhere — and never closes a running run's card; the set is refreshed from the ledger before every catch-up scan (refreshForeignLiveCards, rows under a CURRENT lease held by another generation), so a generation that dies later loses its hold and its cards are swept like any orphan, and a failed refresh keeps the previous set. The cards of the runs closed here: a run that had replied (a finishing row) gets its card closed with how it ended — ✅ <agent> · completed, ⏹/⛔ for a stop, ❌ for failed — before the socket opens (closeReclaimedCards), so record and card agree; an interrupted run's card is closed there too — ❌ <agent> · interrupted with a note that says what to do next (closureNote): a ship pipeline's names the PR it had opened, if any, and the re-issue that continues it (agent:ship with the PR URL, or with the task when no PR existed), every other run's says to re-send the request — and a ship pipeline's note is also posted as a reply in its thread, because its work stands on GitHub with nobody driving it. The orphan sweep is the backstop for a card the reclaim could not close, no longer the only closer (it only looks two hours back, less than a pipeline's ceiling). POST /admin/crash (src/channels/adminCrash.ts) is the kill injection for testing a crash live: a SWITCHBOARD_INGRESS_TOKENS bearer with deploy:write — the same authorization as deploy restart — gets 202 {ok, generation, pid} and the process exits hard 50 ms later (process.exit(137): no drain, no handoff, no finish writes; the platform restarts the container). Not a self-SIGKILL: the bot is PID 1 in its container and the kernel drops a SIGKILL that init sends itself, so a self-SIGKILL would answer 202 and leave the run finishing on the same generation. No bearer 401, no scope 403, no token map 503, only POST. The Worker shim forwards it like any other path.

  12. Resuming a run — the pieces (src/core/runLedger/resume.ts, src/runner.ts, src/core/runRegistry.ts, src/core/runLedger/writeThrough.ts; the boot step that puts them together is the next item). The plan (planResume({transcript, lastStep, tools})) is pure: no step record → interrupted (killed before the seed); an incomplete transcript → interrupted naming the gap; then item 31's transcriptCompleteness decides. resume — the last step's record landed: its inFlight calls were dispatched and their results died with the process, so every tool_use of the transcript's last assistant turn (which must be exactly the recorded calls, else interrupted) is settled (settlementFor): a tool that no longer exists → a synthetic result saying so; a call runs again iff its tool is sideEffectFree (reads, GitHub reads, web, skills) or on the closed rerun-safe list (RERUN_SAFE_TOOLS: write_file, idempotent; update_status; the submit_* recorders); every other known tool — bash, the GitHub writes, a bridged MCP tool that mutates — gets the synthetic result "restarted while this call was in flight; its effects are unknown — re-check them before re-running it" (isError: true in the conversation, ok: false on the stream). The default is the synthetic result: re-running is earned, never assumed. The turn's calls and the recorded calls must be the same set — a call on either side the other does not carry is interrupted. With nothing in flight the transcript must end on a user turn (the seed, or a results turn) and the loop simply continues; the record's turn, iteration and remainingMs carry over and the step is stepRecorded. run-step-fresh — the next step's turns landed but its record did not: nothing of it was dispatched, so all of its calls run fresh, the step is unrecorded (step + 1, iteration + 1, turn + 1 unless the calls are all update_status). The runner's entry (RunOptions.resume: ResumeEntry): messages is the transcript; the deadline is now + remainingMs instead of the agent's budget; a resumed run note says how many calls were in flight and how each was settled; an unrecorded step is reported first through onStep with no new turns (its turns are on the ledger) and the calls in flight; each settlement runs through the same tool dispatch as a live step — without a new tool_call event, since the original is on the stream already under its replayed seq — or emits a tool_result with the synthetic text, and the results — one per call, in the calls' order — are appended as the user turn the next model call needs; the loop then continues at iteration + 1, and the next step's report carries that results turn. The registry (create(label, meta, {id, replay})) creates the resumed run under the ledger's run id with the events published before the restart replayed under their original seqs (bounded like any backlog) and the counter continuing past the highest, so the finish record and the ledger's event stream stay one contiguous record. The write-through (adopt({runId, threadKey, state, lastStep, lastSeq})) takes up a reclaimed row with no claim and no seed: the heartbeat starts at once, steps number on from the last record, appends continue past the last seq, patches merge into the state the previous generation recorded.

  13. Resuming a run — the launch (src/core/boot.ts, src/core/resumeLaunch.ts, src/core/dispatcher.ts, src/channels/slack.ts, src/channels/slack/statusCard.ts, src/core/nullChannelIo.ts; wired in src/index.ts). The reclaim hands resumable rows back instead of closing them: a row from live or handoff whose transcript is whole and whose last step record item 31's rule accepts (resume or run-step-fresh) comes out of reclaimRuns as resumable — the row (ours now), the record, the transcript and the events — while everything else closes as item 36 says. The reclaim repeats (startReclaimSweep, every LEASE_MS): a kill lands at an arbitrary point in the lease, and the next boot arrives seconds later, so at boot the dead generation's lease is usually still current and the rule rightly leaves the row alone — without the sweep a killed run would sit orphaned until the following restart; the sweep takes it within two intervals of the lease expiring, refreshes the cards other generations hold (item 36) and launches the resumes. One pass at a time; a failing pass is a warning. The launcher (launchResumes, after app.start() for the boot's rows and from every sweep pass) plans each run (item 37's planResume, with the agent's static tools as the known tools — a bridged MCP tool is unknown and gets the not-available result) and dispatches it through the ordinary dispatch() with a ResumeContext (the row, the last step record, the plan, the events published before the restart, their highest seq, the repo context from the row's meta); a run whose plan is interrupted, whose agent this build does not know, or whose channel cannot be resumed on is closed with that reason (closeReclaimed). The resumed dispatch runs through the same admission, workspace attach, tools, post-steps and finish as a fresh run, with these differences: the ledger row is adopted at the admission point — before the card, the repo resolution and the attach, so the heartbeat keeps the lease through a slow resident attach — never claimed or seeded; a resume is never a follow-up: a thread that already has a run in flight (the user re-mentioned the bot after the kill) closes the reclaimed row interrupted with no reply and no steer, and a resumed dispatch that ends before its run starts (an unknown provider, a refusal, a gate) closes its adopted row the same way, so the sweep never relaunches a run that cannot start; the repo context is the row's, never re-resolved from the message; the attach-head guard does not run (the stored head is the run's; the settle after the loop still reconciles a moved head); the conversation is the transcript and the system prompt is the row's, verbatim; the run is created in the registry under its own id, with its original start (the row's startedAt, so the record, the card's elapsed time and the admission slot's — the steer ack's "N in" — span the whole run) and the earlier events replayed under their seqs, so the run page URL and the record are the same run and the record is one contiguous stream; no second input/run_meta/context event and no tombstone; only this generation's events are appended to the ledger (the subscription starts after the highest replayed seq); the dispatcher's state — checklist, verdict, PR description, pushed branch — comes back from the row, the verdict and description re-validated through the tools' own parsers; the runner re-enters from the plan (item 37). The message the dispatch runs under pins agent:/model:/effort: from the row's meta with the original request text from its input event as the label — the model sees the transcript, not the message. The channel: a Slack run resumes on its thread (resumeSlackIO: the thread from the row's threadKey, the requester from its meta) and keeps the card the previous generation posted — status() edits it in place instead of posting a second one, so the thread shows one card whose frames carry on (a card deleted since falls back to a fresh one); an HTTP or MCP run, whose caller is gone, resumes on a null channel (nullChannelIO: the reply is logged, never sent — its deliverable is the record and, for a coding run, the PR); any other channel closes interrupted. The reconnect catch-up never re-dispatches the original mention: it wore the previous generation's 👀 and is handled. Ship pipelines are never resumable (item 35).

  14. The drain is a handoff, and a fence is a stop (src/index.ts, src/core/runLedger/writeThrough.ts, src/core/drain.ts). SIGTERM: after the Slack socket closes, the drain marks every run a resume can continue — LedgerRun.resumable: its seed and seed record landed, or it was adopted from a resume; never a ship pipeline, a detached run, or a run whose seed failed — handoff on the ledger in one call (LedgerWriteThrough.handoff, POST /runs/handoff), so the next generation takes them at once whatever their lease (item 31's reclaim, item 38's launcher). Those runs are not waited for: they keep running here until the exit, and their writes are fenced the moment the next generation reclaims them. Only the runs a resume cannot continue hold the drain, up to DRAIN_DEADLINE_MS as before; with none, the drain waits HANDOFF_BUDGET_MS (6 s) for pending history writes and reflections and exits — a deploy no longer waits on runs. A handoff the ledger refuses (unreachable, no routes) is a warning and the old wait. The drain-deadline abandonment writer skips the handed-off runs: their record is the ledger's. The finishing gate: finishing() answers ok (reply), fenced (another generation owns the run — the live → finishing CAS was refused because the row was reclaimed, or the row is gone because the other generation already finished it) or unavailable (the ledger could not be asked, or the run is untracked or was detached for a reason other than a fence: the run is this process's, reply as before) — a run detached BY a fence (a heartbeat, step, append or state write refused as fenced) answers fenced from then on, so the reply and the record stay the other generation's. A handed-off run that completes before the next generation reclaims it replies itself: handoff → finishing is allowed for the owner (the reclaim then finds no row), so the handoff never discards a finished answer. On fenced nothing more reaches the thread from this process — no reply, no card close, and no record: the run is the other generation's now and its record is theirs to write; a partial record from here could race the real finish. A fenced write stops the run: a heartbeat, step, append or state write refused as fenced detaches the run (item 35) and, once, tells the dispatcher (onFenced), which hard-stops it — its next tool call would act on a run someone else is driving. LedgerWriteThrough.liveRuns() names the runs this generation drives (opened or adopted, not finished, not detached); /healthz is unchanged.

  15. The durable inbox (thread-admission item 5 has the admission side). run_inbox holds the follow-ups steered into a live run, one row per push with a per-run seq (the message with its attachments when it fits the route's 512 KiB body — thread-admission item 5 has the cap and the drop note); pushInbox is never fenced (a steer arrives on whichever container is up) and refused only for a run with no live row; readInbox(runId, afterSeq) (POST /runs/inbox/read, any generation) is the same slice on demand — the resume's re-read at adopt time, so a push that landed after the reclaim's snapshot is found; the write-through answers it empty with a warning when the ledger cannot be asked. Each step record carries inboxConsumedSeq, the highest inbox seq among the follow-ups the run had folded in when the record was written (the runner keeps the counter; a resume starts it at the last record's), so the reclaim hands the resume exactly the items past it — ReclaimedRun.inbox, carried through ResumableRun and ResumeContext — and the finish deletes the run's inbox with its other live rows. The write-through's pushInbox(runId, message) answers the seq, or undefined with a warning when the ledger refuses or fails. /runs/live rows are what the admission map is built from (ThreadsElsewhere), replaced whole on every reclaim pass; LiveElsewhere names the thread, start and agent for it.

  16. One registry on every read surface (live-view item 17). With the ledger on, RunsService takes it as a read source: the ledger's live rows whose run is NOT in this process's registry — live under another generation (a rollout overlap), or reclaimed here and not yet launched; never a run this process is attaching, whose registry row dates from its reservation (item 42) — are live rows on every surface, built from the row's meta and start and the events the ledger holds (readEvents: count and activity), with ownerGen naming the generation driving them and never a capability token (the page token is the other generation's). listRuns lists them under all and active (never finished, never on a cursor page — live rows sort ahead of any cursor) and suppresses their store rows whatever was asked (item 27); liveElsewhere(visibleTo) is that slice alone, for the default index. getRun (with the events on a messages read), getRunEvents (a seq page) and getRunFriction (a live diagnosis) answer for such a row after the registry and before the store; stopRun asks the ledger (requestStop), which the owning generation reads on its next heartbeat, and answers stopping. A run in both the registry and the ledger is the registry's (once). The viewer's predicate applies to ledger rows as to any row. A ledger that cannot be read is one warning per call and the registry (and the store) alone — never a failed read. The web surface admits such a run tokenless (live-view item 16): its page token is the other generation's, so the viewer's attribute decision is its only gate; the page renders in history mode with the ledger's events (no stream to follow, no stop controls), the events route replays them and ends, the friction route answers the live diagnosis, and the tokenless stop route stops it through the ledger for a viewer holding runs:write (the command surface's runs.stop scope; a read-only viewer gets the 404). One ledger listing serves every service read within LEDGER_LIST_TTL_MS (2 s) — a page view's three reads, or a history read of a finished run, cost one ledger call — and the rows' events are read in parallel; a failed listing is not kept. Not covered: following such a run live — a row live elsewhere is a rollover's few seconds, and the sweep takes the run over on the next pass.

  17. A run is on the ledger from before its workspace attach, and a kill there restarts it (thread-admission item 5). Until this item the claim (item 35) came after the attach, so a run was carded and in flight for the whole attach — a minute on a resident's mirror mutex, minutes on a cold clone — with nothing on the ledger: a kill there lost it (no resume, no record, the card left for the orphan sweep; seen live on a review whose resident attach waited out the mirror mutex). Now the dispatcher reserves the row right before the attach (LedgerWriteThrough.reserve, POST /runs/claim with phase: attaching): the run's id (minted from the registry at admission, RunRegistry.mintId, and handed back to create), thread, start, card, the run meta, and the request in meta.request — the message in the durable inbox row's shape (durableInboxMessage: text with its directives, sender, link, thread, arrival time, the attachments when they fit DURABLE_INBOX_MAX_BYTES, else attachmentsDropped) — with an empty prompt; the heartbeat runs from here, so a long attach keeps the lease. The registry row is created at the same moment, right before the reserve (registry.create with the minted id, the card's start and the composed label), so the run is one row on every surface from admission on: the runs index lists it with its label and capability link, the run page serves it, the card carries the live link through the attach, the HTTP ingress answers with its id, and a stop relayed during the attach latches in its control for the run loop's first step. (Before this the row came after the attach, and the index showed the reservation meanwhile as a labelless item-41 ledger row with a tokenless link that 404'd once the run was promoted.) Its stream is live from that moment too: bindRun at the reservation (the spans before it — the root, the ack card, the repo resolution — backfilled; the reservation, the attach and the resident's grafted steps streamed as they happen), then input, run_meta and context published right after, before the attach — so the run page shows the request and the setup steps while the workspace is still being attached, and the index's event count moves. The request stays the record's first content event; the setup spans ahead of it are head material (isHeadMaterial: request, slack.receive, dispatch.*), so the protected head runs unbroken from the first event through the request. The tombstone (item 27) still waits for the run loop, so a dispatch that ends before it leaves no record. The claim once the prompt exists is the same request as before and promotes the row in place (decideClaimWrite: insert for a fresh thread, promote for the owner's attaching row — prompt, tools, card and state land, phase live, identity and start untouched — refresh for an owner's re-reserve, keep for any re-claim on a row past attaching), through the same tracked run (OpenRunRequest.reservation), so the seed and every later write are exactly item 35's. phaseTransition allows attaching → live and attaching → finishing (a dispatch that fails before its prompt); never attaching → handoff — an attaching run has nothing to resume from, so the drain waits for it as it always did. A ship run is never reserved: it claims its own row without a seed (item 35) and closes interrupted at a reclaim as before. Reclaim (item 36) leaves an expired attaching row in that phase (reclaimPhase) and hands it to the launcher as a RestartRun — row, unconsumed inbox, no transcript. The launcher (launchResumes) reads the request back (messageFromInbox) and dispatches it again as the message it was — directives included, so the same agent and model resolve — with a RestartContext {row, inbox}: the dispatcher claims the thread in-process with the row's original start, reserves the row again (the owner's idempotent re-claim, for the heartbeat), folds the carried inbox plus a re-read of anything later into the run's inbox (item 40's rule, no second ack), keeps the row's card (resumeSlackIO with the row's card.ts) and id (registry.create with id and startedAt), and runs the request from scratch — attach, prompt, promotion, steps, finish — so the record carries the original start and both the request and the follow-ups as input events. A restart onto a thread that has a newer run in flight (the user re-mentioned after the kill) closes the reserved row interrupted with no reply. A fenced reservation — the lease lapsed during the attach and another generation reclaimed the row — is told through onFenced; the dispatch stops at the attach's end, releases the workspace and says nothing: the run is theirs to restart, and a promotion that finds its own run under another generation answers the same way (claim → fenced, nothing seeded). A reservation never promoted — the dispatch ended before its prompt existed: a refusal after the reserve (the attach's ask-once branch, a moved head), a failed attach, a throw — is abandoned in the outer finally (LedgerRun.abandon, POST /runs/abandon, fenced like finish), and the registry row created with it is discarded the same way (RunRegistry.discard: no finished frame, the end frame to a live subscriber, the index feed's removed, the id unknown afterwards; a no-op for a finished run, whose record exists): the live rows go with NO record, because the run never started and a record of it would be noise; a fenced or untracked run's abandon is a no-op. The in-process steer's durable copy (item 40) lands from the reserve on; only a steer in the moment between admission and the reservation (the ack card, the repo resolution) rides in memory alone.

  18. A narrow lookup: the submitted PR description for a head (reading-diff.md item 7). findSubmittedPrDescription(store, { repo, pr, headSha }, limit = SUBMITTED_LOOKUP_LIMIT = 10) (src/core/reviewDescription.ts) is the one read of the store the dispatcher makes for a running run: store.list({ limit, visibleTo: { kind: "repos-in", repos: [repo] } }) — the repo the review is authorized for, newest first — then store.get on each row, scanning its events newest first for a review_artifact pr_description with origin: "submitted" whose repo, pr and headSha all equal the key; the first match wins, its fromRunId kept (a review run's copy names the coding run). At most limit records are read; no new store method, filter or column — the record's repo is the only index used, and a store that has no such record answers "none" after limit reads. RunDeps.runStore is the seam (NullRunStore in a process without run history: the lookup finds nothing and the review parses the body). A store failure surfaces to the caller, which degrades to the parsed body.

  19. A finished record gains events by exactly one path — a whole-record rewrite through put that only a review_artifact uses (reading-diff.md item 8). Records are otherwise immutable once finished (the finish write is the last one, item 16). The abridged reading diff is produced AFTER the review, so appendReviewArtifact (src/core/reviewAbridge.ts) re-reads the record, drops a previous meat artifact (a --force replaces, never accumulates), stamps the new event with the next seq after every event the record ever held (the replaced one included, so an afterSeq cursor that saw the old one sees the new), bumps eventCount for a new artifact (not for a replacement), refits the byte budget with fitRecordToBudget — whose per-event cap leaves a review_artifact alone: its diff is capped by its producer above MAX_EVENT_BYTES by design and its summary is one line — and puts the whole record: the same-id upsert every store already implements (rewritten: true, the DO rewrites the event rows in one transaction). It refuses by name when the record is gone (the run's record is gone — nothing to append to), when the artifact would not survive the budget, or when the store answers stored: false (the record fell outside retention in its own write); nothing is ever created by the rewrite. No other writer touches a finished record.

Validation criteria ​

CriterionEvidence
43: the lookup reads the repo's newest records first and each record's events newest first, matches only submitted + same repo, PR and head, keeps a copy's fromRunId, and stops after limit records[unit] src/core/reviewDescription.test.ts::findSubmittedPrDescription::*
44: the appended artifact takes the next seq after every existing event, a replacement keeps eventCount, the event cap leaves a review_artifact's summary alone, and a missing record / a stored: false put are named failures with nothing created[unit] src/core/reviewAbridge.test.ts::ReviewAbridger.abridge — the one path::absent → running…, ::is idempotent…, src/core/reviewAbridge.test.ts::ReviewAbridger — failures and the append::a record deleted…, ::a rewrite the store answers…, src/core/runRecord.test.ts::capEvent on a review_artifact::does not shrink the summary…
A 4 MB event list fits under 1.5 MiB; first and last events kept; head/tail balanced; truncated: true; eventCount preserved; storedEventCount = events.length; source untouched[unit] src/core/runRecord.test.ts::fitRecordToBudget::fits a 4 MB event list under 1.5 MB, keeping the first and last events, marking truncated, preserving eventCount
An under-budget record is returned with its events intact and truncated untouched[unit] src/core/runRecord.test.ts::fitRecordToBudget::returns an under-budget record unchanged (same events, truncated stays false)
A single 200 KB event is capped to ≤ 64 KiB with an ellipsis, on summary and on text alike[unit] src/core/runRecord.test.ts::fitRecordToBudget::caps a single 200 KB event to at most 64 KiB, ending its summary with an ellipsis, ::caps a \text` field the same way when an event carries text instead of summary`
Per-event cap measures UTF-8 bytes, not chars[unit] src/core/runRecord.test.ts::fitRecordToBudget::measures bytes, not chars: multi-byte text is capped by its UTF-8 size
A 31-day-old record is hidden at retentionDays: 30; a 29-day-old one kept[unit] src/core/runRecord.test.ts::applyRetention::hides a record finished 31 days ago under retentionDays 30 and keeps a 29-day-old one
Newest 3 of 4 kept at maxRuns: 3, newest first; ties broken by id desc[unit] src/core/runRecord.test.ts::applyRetention::keeps the newest 3 of 4 under maxRuns 3, newest first, ::breaks a finishedAt tie by id descending
maxBytes trims the oldest large runs even when maxRuns would keep them[unit] src/core/runRecord.test.ts::applyRetention::maxBytes trims the oldest large runs even when maxRuns would keep them
Missing bytes counts as 0; input never mutated[unit] src/core/runRecord.test.ts::applyRetention::treats a missing bytes field as 0 and never mutates the input
clampRetentionPolicy fills defaults, clamps out-of-range values to the bounds, replaces non-finite values, floors fractions[unit] src/core/runRecord.test.ts::clampRetentionPolicy::*
isRunRecord accepts the tracing stamps (receivedAt, sealedAt, replyOk, stepCount, schema; tracing.md) when typed and refuses them otherwise; the registry projects receivedAt from the meta onto the summary and the snapshot; the Worker clamps receivedAt/sealedAt like finishedAt; the history seed's duration is runDurationMs[unit] src/core/runRecord.test.ts::isRunRecord::accepts the tracing stamps when typed (docs/reference/specs/tracing.md) and refuses them otherwise, src/core/runRegistry/projections.test.ts::*::carries receivedAt from the RunMeta onto the summary and the snapshot, and omits it when absent (docs/reference/specs/tracing.md), deploy/cloudflare-memory/runs.test.ts::*::receivedAt and sealedAt a year ahead are clamped like finishedAt (docs/reference/specs/tracing.md), src/channels/liveView.test.ts::*::a record carrying receivedAt seeds a duration that opens there — the one definition (docs/reference/specs/tracing.md)
isRunRecord accepts a well-formed record (also after a JSON round-trip) and keeps the run-page event fields verbatim[unit] src/core/runRecord.test.ts::isRunRecord::accepts a well-formed record, ::round-trips the run-page fields on tool, input and model.turn span events verbatim …
A stored diagnosis missing a current category (or carrying an unknown one) is accepted; normalizeDiagnosis zero-fills and drops; the store and Worker reads normalize the same way[unit] src/core/runRecord.test.ts::isRunRecord::accepts a record whose diagnosis lacks a current category or carries an unknown one …, src/core/runStore.test.ts::* — RunStore contract::a stored record missing a diagnosis category still loads and lists …, src/core/runStoreWorker.test.ts::WorkerRunStore::get and list normalize a stored diagnosis missing a current category …, deploy/cloudflare-memory/runs.test.ts::run history routes::a stored record missing a diagnosis category still reads via get and list …
isRunRecord rejects a bad id (space, empty, > 64 chars)[unit] src/core/runRecord.test.ts::isRunRecord::rejects a bad id
isRunRecord rejects an unknown status, non-array events, an event without a string type, non-object input, wrong-typed identity/timing fields[unit] src/core/runRecord.test.ts::isRunRecord::rejects an unknown status, a non-array events, an event without a string type, and non-object input, ::rejects missing identity/timing fields and a non-boolean truncated
isRunListItem accepts a record minus events (with/without numeric bytes), rejects a bad bytes or a bad row[unit] src/core/runRecord.test.ts::isRunRecord::isRunListItem accepts a record minus events (with or without numeric bytes) and rejects a bad bytes or a bad row
storedEventSeqs keeps registry stamps when strictly increasing, else positions for every event[unit] src/core/runRecord.test.ts::isRunRecord::storedEventSeqs keeps the registry stamps when strictly increasing, else positions for every event
The module has no Node built-in imports (Worker-importable)[agent] grep -n "node:" src/core/runRecord.ts prints nothing, and cd deploy/cloudflare-memory && npm run typecheck (which compiles the import) is clean.
A 2 MB record round-trips in the in-memory and file stores; list carries no events and reports bytes[unit] src/core/runStore.test.ts::* — RunStore contract::round-trips a 2 MB record and lists without events
A malformed id (../../etc/x, spaces) is not-found on every method, never thrown, never touches the filesystem or the network[unit] src/core/runStore.test.ts::* — RunStore contract::rejects ids failing RUN_ID_PATTERN without throwing, ::FileRunStore::get("../../etc/x") is not-found and touches nothing outside data/runs, src/core/runStoreWorker.test.ts::WorkerRunStore::rejects a bad id locally without a request
list: newest-first, default 50, cap 200, sinceMs/agent/channel filters; the { before, beforeId } cursor shows every same-finishedAt sibling across a page boundary[unit] src/core/runStore.test.ts::* — RunStore contract::list is newest-first, capped at 200 (default 50) …, ::list cursor {before, beforeId}: two runs with identical finishedAt straddling a page boundary both appear
events(id, {afterSeq: 10, limit: 5}) returns five events with seq > 10 and nextAfterSeq; the last page has no cursor; unknown id → null; a run with no events → { events: [] }; events stamped 2001..7000 page from afterSeq: 6500 by their stored seq[unit] src/core/runStore.test.ts::* — RunStore contract::events(id, {afterSeq: 10, limit: 5}) returns five events with seq … and a cursor …, ::events keep the registry seq they were published with …
Expired record absent from get/list before any write; put of an already-expired record is stored: false; rewritten only on a changed record; delete removes run + events[unit] src/core/runStore.test.ts::* — RunStore contract::an expired record is absent from get/list before any write, ::put reports stored:false for a record already outside policy, ::rewritten is true only when an existing record changed, ::delete removes the run and its events
File mode 0600, dir 0700, no temp files left after rename[unit] src/core/runStore.test.ts::FileRunStore::writes <id>.json with mode 0600 inside a 0700 directory, temp-then-rename
Torn <id>.json skipped by get, absent from list, others still read; stale index line hidden then healed on the next put; expired record unlinked and the index compacted on the next put; shrink-then-grow does not resurrect; persists across instances; sweep() unlinks expired files with no writes[unit] src/core/runStore.test.ts::FileRunStore::*
buildRunStore: unconfigured → null (no timer); store: file → FileRunStore with a 6 h unref'd sweep timer that unlinks expired files; worker without bearer → null + warning naming the env var; worker with bearer → WorkerRunStore[unit] src/core/runStore.test.ts::buildRunStore::*
WorkerRunStore.put of a 1.9 MB record sends a string body with NO hand-set Content-Length (runtime-derived), bearer, policy + policyUpdatedAt; get/list/events/delete send no policy; get re-validates and returns null for {record: null}; 404 → RouteMissingError, 503/408/429/network → TransientStoreError, 400 → PermanentStoreError with the Worker's error text[unit] src/core/runStoreWorker.test.ts::WorkerRunStore::*
WorkerRunStore.put under a span is an http.client child with route /runs/put and the trace context for the configured host; the history writer hands its span option through to the sink (tracing.md item 24)[unit] src/core/runStoreWorker.test.ts::WorkerRunStore trace context::*, src/core/runHistoryWriter.test.ts::createRunHistoryWriter::write hands its span to the store's put…
Config: accepts a well-formed runHistory; rejects retentionDays: 0, maxRuns: 0, http:// worker.baseUrl, unknown store[unit] src/config.test.ts::runHistory config::*
DO: put→get round-trip with events in seq order; unknown id {record: null} 200; 5000 events in one transaction, paged by seq; registry seq preserved (2001..7000 pages from 6500); two puts one id → one coherent set; identical repeat rewritten: false; failed mid-put leaves no rows[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::put → get round-trips the record with events in seq order …, ::inserts 5000 events inside one transaction and pages them by seq, ::events keep the registry seq …, ::two puts for one id leave exactly one coherent event set …, ::a put that fails mid-transaction leaves no runs row and no events
DO policy: newer policyUpdatedAt wins, older ignored, retentionDays: 0 → 400, future stamp clamped to the DO clock so a later correct one wins; get/list cannot resurrect with a body policy[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::policy: the newer policyUpdatedAt wins …, ::get/list never accept a policy …
DO trim: a shrink dropping >25% deletes ≤500 per put while list/get/events hide the rest (age, then newest maxRuns with id tie-break, then maxBytes); the next put finishes; evicted run's events gone; re-put with 3 events leaves 3 rows; outside-policy put stored: false[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::a shrink dropping >25% deletes at most 500 rows per put while list already hides them, ::get/events hide exactly what list hides for rows still on disk …, ::after a maxRuns trim the evicted run's events are gone … (the get hiding assertion goes red when the read-side retention check is removed)
DO clock: finishedAt a year ahead clamped to now + 24 h, stored_at recorded; alarm with no writes deletes expired rows and re-arms[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::a finishedAt one year ahead is clamped to now + 24 h and stored_at is recorded, ::the alarm deletes expired rows with no writes, and get is then not-found
The retention sweep is a state.alarm root in the state Worker's log, ending with swept (tracing.md item 25)[agent] live: wrangler tail <state-worker> across one sweep interval.
Fences: 2 MiB at cap ok, +1 byte 413, 1.9 MB multibyte measured in bytes, other routes 512 KB, missing Content-Length 411; 400 on bad key/record/id/limit/beforeId; unknown route 404; no bearer 401; GET 405[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::body fence: 2 MiB cap on /runs/put …, ::validates: bad store key, malformed record, bad id, bad limit → 400 …, ::list cursor {before, beforeId}: … a bad beforeId is 400
Corrupt event row skipped; /healthz lists runs; list limit 1000 → 200 + cursor with filters and no events on the wire; delete removes all events[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::a corrupt event row is skipped …, ::/healthz lists runs, ::list: newest-first, limit 1000 → at most 200 rows plus a cursor …, ::delete removes the run and all its events …
State Worker (migration v4, run-history routes) deployed before the bot that writes to it; /healthz lists runs[agent] Deploy deploy/cloudflare-memory (npm run deploy) first; curl https://<state-worker>/healthz → features includes runs; the bot's startup log then has no ORDERING ERROR line.
Completed run → one put with status: completed, eventCount = the registry's published count = events.length, truncated: false, events include the input and answer events, identity/model/label fields set, diagnosis = analyzeRunFriction(events), persisted flagged on the index row, pending() back to 0[unit] src/core/dispatcher.test.ts::run history write path …::a completed run ends as one stored record: status completed, eventCount = published count, events include the user and assistant messages, identity fields set
Soft-stopped run → stopped_soft; a reply that throws after the loop completed → failed while a stop keeps its stopped_* status[unit] src/core/dispatcher.test.ts::run history write path …::a soft-stopped run is stored as stopped_soft…, ::a reply that throws after the run loop completed yields status failed (not completed) and seals the run once…
Provider throw → status: failed, the record (with the input event) is still written, the failure reply still sent, activeRunCount() back to 0[unit] src/core/dispatcher.test.ts::run history write path …::a provider throw yields status failed, the record is still written, and the failure reply is still sent
The record and the friction row carry the registry's redacted label — a pasted token in the request reaches neither[unit] src/core/dispatcher.test.ts::run history write path …::the record and the friction row carry the registry's redacted label …
A reply slower than the registry TTL (ttlMs: 10, 40 ms reply) still yields a full record; the record's events equal the finish-time snapshot even though it is assembled after the reply[unit] src/core/dispatcher.test.ts::run history write path …::a reply slower than the registry TTL: the finished run stays unsealed…, ::the record's events equal the registry snapshot taken at finish …
More published events than the backlog holds (backlogLimit: 16, 12 context turns) → eventCount = published total, storedEventCount: 16, truncated: true, and what survives is the protected head (the input, the context, the meta) plus the newest events[unit] src/core/dispatcher.test.ts::run history write path …::more published events than the backlog holds: eventCount is the published total, storedEventCount the backlog length, truncated true — and the protected head… (scaled stand-in; the 8000-count bound itself is covered in runRegistry.test.ts)
Byte budget: over budget, span records go first, pair by pair from the middle outward, never from the head; every content event survives when that suffices; a budget below the content alone still drops content from both ends[unit] src/core/runRecord.test.ts::fitRecordToBudget::over budget, span records go first…
The FINISH put is not called until after io.reply ran — when the reply runs, the store holds only the start-of-run tombstone (goes red when the finish write is moved before sendAnswer)[unit] src/core/dispatcher.test.ts::run history write path …::the finish write happens after the reply: only the provisional tombstone has been put when io.reply runs
Without a writer nothing is written and the run behaves as before[unit] src/core/dispatcher.test.ts::run history write path …::without a writer nothing is written and the run behaves as before
503 twice then 200 → exactly one stored record, three attempts, backoff 1 s then 4 s (jittered ±50%), pending() 0, failures() 0[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::503 twice then 200: three puts of the same record, backoff 1 s then 4 s (jittered), one success, pending 0, no failure counted, ::the jitter is bounded: a delay is within ±50% of its base value …, src/core/dispatcher.test.ts::run history write path …::put 503 twice then 200: exactly one record, pending back to 0, no failure counted
pending() counts a write sitting in retry backoff; concurrent writes tracked independently; settled() waits for all[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::pending() counts a write that is sitting in retry backoff, ::concurrent writes are tracked independently: pending() is the number in flight, settled() waits for all
Retries exhausted → counted once, warned once with run id + attempt count[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::three transient failures: the write is lost, counted once, warned once with the run id and attempt count
413 (PermanentStoreError) → one attempt per record (tombstone and finish write alike), no sleep, one warn each, failures() counted per loss; the reply unaffected and never mentions the store[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::a PermanentStoreError (413) is never retried: one put, one warn, failures +1, no sleep, src/core/dispatcher.test.ts::run history write path …::a 413 (PermanentStoreError) is not retried: one attempt, one warn, failures +1; the reply is unaffected
404 (RouteMissingError) → the deploy-ordering line exactly once per process across two runs, no retry, degraded() true, every loss counted; a lost record is also a run friction report never sees[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::a RouteMissingError (404) logs the deploy-ordering message ONCE per writer, sets degraded, never retries, counts every loss, src/core/dispatcher.test.ts::run history write path …::a 404 (RouteMissingError) logs once, is not retried, sets degraded; the runs are still counted
An unclassified error is retried; a throwing onPersisted hook is isolated[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::an unclassified error (a file store's fs failure) is treated as transient: retried, then counted, ::a throwing onPersisted hook is isolated: still counted as a success and pending drains
markPersisted sets the flag and emits one upsert; no-op for unknown/evicted runs; the key is absent until then; snapshot carries startedAt + monotonic eventCount[unit] src/core/runRegistry.test.ts::RunRegistry.markPersisted::*, src/core/runRegistry/projections.test.ts::RunRegistry.snapshot — record inputs …::*
Live run via getRun → finished: false, no token property, no events unless include: "messages"; persisted run → same shape with finished: true, persisted: true (goes red when liveView becomes a spread of RunSummary)[unit] src/core/runsService.test.ts::RunsService.getRun::returns a live run as finished:false with no token and no events unless asked, ::returns a persisted run in the same shape (finished:true, persisted:true), events only on include
Unknown, malformed (../etc), and expired (31-day-old) ids → not_found; store null → live reads work, persisted are not_found[unit] src/core/runsService.test.ts::RunsService.getRun::is not_found for an unknown id, a malformed id, and an expired record, ::works with history off (store null): live runs read, persisted ones not_found
Run in both registry and store listed once with the live stop state and the record's finish-only fields; same eventCount before and after TTL eviction, persisted: true both times; the live and persisted projections differ only in the finish-only fields[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::lists a run in both the registry and the store once, live stop state winning, same eventCount before and after eviction, ::a run in both sources projects identically from the live row and the persisted row, except for the finish-only fields
active = unfinished registry runs and never calls store.list (spy); finished = finished registry ∪ store; all = union[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::active = unfinished registry runs only and never calls the store; finished = finished registry ∪ store; all = union
Sort by the store's key: live rows first, then finishedAt desc, id desc — a true top-N even when an old run started late; nextBefore pages the store without skipping same-ms siblings and omits live rows past the first page[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::orders by the store's key — live rows first, then finishedAt desc, id desc …, ::pages with nextBefore = the last persisted row's {finishedAt, id} …
agent/channel/sinceMs filters apply to live rows through their RunMeta as to persisted rows[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::filters by agent, channel and sinceMs on live rows too — a live run carries the RunMeta given at create()
limit after the merge (10 → 10 rows, live rows newest), default 50, cap 200; store.list asked for limit + liveCount, never above 200; a 5000-run store fetches one bounded page[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::applies limit after the merge (default 50, cap 200) and fetches at most limit + activeCount persisted rows, ::listRuns({limit:10}) against a 5000-run store fetches a bounded page
Store throwing → all/finished return live rows + storeUnavailable: true; active unaffected and never calls the store; history off never reports storeUnavailable[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::degrades to live rows + storeUnavailable when the store throws…, ::with history off (store null) lists live rows and never reports storeUnavailable
getRunEvents strict seq > afterSeq live and persisted; a 5000-event persisted run → 500 events + nextAfterSeq: 500 read via store.events with limit: 500; limit: 100000 still 500; a live-taken cursor addresses the same events once persisted[unit] src/core/runsService.test.ts::RunsService.getRunEvents::live: seq … afterSeq strictly, honoring limit, with nextAfterSeq while more follow, ::persisted: seq … afterSeq strictly; a 5000-event run returns a bounded page (≤ 500 events) with nextAfterSeq, read via store.events, ::RunsService.listRuns — read merge::a live→persisted afterSeq cursor addresses the same events …
A page stops under 256 KiB of event JSON (4 × 60 KiB events) and resumes from nextAfterSeq; unknown/expired → not_found; a zero-event persisted run → ok with an empty page[unit] src/core/runsService.test.ts::RunsService.getRunEvents::caps a page at 256 KiB of event JSON and resumes from the cut, ::is not_found for an unknown or expired run; a persisted run with zero events is ok with an empty page (getRun ok too)
getRunFriction: live → analyze(events, { finished }) (spy sees the stamped events + flag), persisted → the stored diagnosis, unknown → not_found[unit] src/core/runsService.test.ts::RunsService.getRunFriction::*
stopRun live → { mode, state: "stopping" }, control driven, stop_requested note with actor; actor id stripped to the charset and capped at 128 (unknown when empty); finished/persisted → conflict; unknown → not_found[unit] src/core/runsService.test.ts::RunsService.stopRun::*, src/core/runRegistry.test.ts::RunRegistry — token-free operator reads …::*
authorizeLive → null for a wrong token or unknown run; a working subscribe/snapshot for the right token (events flow, onFinish fires)[unit] src/core/runsService.test.ts::RunsService.authorizeLive::*
JSON.stringify of every method's output lacks the fixture token[unit] src/core/runsService.test.ts::RunsService — no output carries the capability token::* (plus expectNoToken inside most other cases)
The friction ledger is the store: differential ordering vs the in-memory double, field-explicit projection, the predicate in wire form, list-only reads bounded to 500, compound-cursor paging, an unreadable store rejects with its error, no store → no ledger[unit] src/core/frictionLedger.test.ts::RunStoreFrictionLedger::*
LiveRunAccess.requestStop drives the registry's token-gated stop; a store row merged into its live twin keeps the live stop/persisted and gains finishedAt/status[unit] src/channels/liveView.test.ts::live view on RunsService: history pages + index toggle …::persisted run events, friction and stop::a valid token still stops a live run (200) and 409s a finished one …, ::index: active by default, everything with ?all=1::a persisted row's flag rides the seed; the store spy sees exactly one list call for ?all=1
History page, stored replay, 409 stop, 404 shapes, index toggle, row mirror[unit] the live view on RunsService rows in live-view.md
SIGTERM during retry backoff: the drain waits on pending() and exits within the deadline[agent] With runHistory.store: file and data/runs made read-only, run npx tsx src/cli.ts ask "hi" in one shell and kill -TERM the bot mid-backoff: the [drain] line reports 1 history write(s) in flight, then exits ≤ 15 min later reporting the write as lost. (No unit test: the drain loop lives in main().)
Locally with runHistory.store: file, a CLI run creates data/runs/<id>.json and an index.jsonl row[agent] npx tsx src/cli.ts ask "what is 2+2" with runHistory: { store: file } in the config, then ls data/runs/ shows one <uuid>.json whose status is completed.
getSummary (every store): the listing row with bytes, no events; null for unknown/expired/malformed ids; the file store hides a torn file from it like list does[unit] src/core/runStore.test.ts::* — RunStore contract::getSummary returns the listing row …, ::FileRunStore::a torn <id>.json is skipped by get and absent from list …
WorkerRunStore.getSummary posts /runs/summary, re-validates (isRunListItem), {summary: null} → null, malformed → PermanentStoreError, bad id → no request[unit] src/core/runStoreWorker.test.ts::WorkerRunStore::getSummary posts /runs/summary …
/runs/summary returns the row without events and reads no run_events (statement spy); {summary: null} for unknown; bad id 400[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::/runs/summary returns the listing row …
RunsService.getRun (no include), getRunFriction and stopRun read store.getSummary and never store.get; include: "messages" is the one full read[unit] src/core/runsService.test.ts::RunsService — summary-only persisted reads::*
RunHistoryDO.list fast path: within maxRuns/maxBytes the page is ONE SELECT … LIMIT ? with the agent/channel filters in SQL and no retention scan (statement spy); over a bound the kept set is computed and rows, order, cursor and filters are identical; get hides the same rows[unit] deploy/cloudflare-memory/runs.test.ts::run history routes::list fast path …, ::get/events hide exactly what list hides …
channelVisibility on the record: required on write, unknown when a stored record or listing row lacks it (normalizeStored), an unknown value rejected by isRunRecord; RunVisibilityFilter is the predicate's wire form (toVisibilityFilter sorts the sets), isRunVisibilityFilter bounds depth and width and rejects unknown kinds/visibilities, matchesVisibility is the one truth table (an empty and/or and a missing stamp match nothing but unknown)[unit] src/core/runRecord.test.ts::isRunRecord::channelVisibility defaults to unknown …, src/core/runRecord.test.ts::run visibility filter — the wire form of an authz Predicate…::*
list({ visibleTo }) on every store: the in-memory/file stores apply matchesVisibility ANDed with the other filters; the DO compiles it into the one page query (channel_id IN, channel_visibility IN, user_id =, or/and), returns an empty page for none without a query, answers a malformed or too-wide filter 400 (never all); a pre-stamp table gains the channel_visibility column with unknown and a put without the stamp stores unknown[unit] src/core/runStore.test.ts::* — RunStore contract::list applies \visibleTo` …, deploy/cloudflare-memory/runs.test.ts::run history routes::list with `visibleTo` (authorization.md item 6) …, ::a table created before the visibility stamp gains the column …`
RunsService.listRuns requires visibleTo: live rows filtered by matchesPredicate, the store asked with the wire form (none for all), none touches neither; every view carries channelVisibility; RunStoreFrictionLedger.recent({ visibleTo }) pushes it to the store in wire form[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::\visibleTo` is pushed down …, src/core/frictionLedger.test.ts::RunStoreFrictionLedger::recent() takes the actor's predicate …`
Dispatch stamps every run's RunMeta and record with the channel directory's visibility (static default: http:/mcp: machine, slack:D… dm, slack:G… private, else unknown); an injected directory is asked once per run; a failing directory stamps unknown[unit] src/core/dispatcher.test.ts::run history write path …::channel visibility stamp …
registry.finish(id, status) stores the status; the summary and the index upsert carry status + finishedAt, and the seal's upsert adds sealedAt/replyOk; a completed / soft-stopped dispatch leaves the registry row saying so[unit] src/core/runRegistry/projections.test.ts::RunRegistry — terminal status + finishedAt …::*, src/core/dispatcher.test.ts::run history write path …::a soft-stopped run is stored as stopped_soft, and the registry summary carries the same status …, ::a completed run's registry summary says \completed` …`
Seal after the reply: a delivered run is sealed after its reply (never before) with replyOk: true and its record and row carry one sealedAt; a reply that throws seals once with replyOk: false and the record says failed; a reply slower than the TTL keeps the run unsealed through it and the TTL runs from the seal; a fall-through's command run seals with no replyOk and the agent run after its reply; a fenced run is sealed by the backstop with no record; a ship pipeline's final reply that throws seals replyOk: false; the ending itself: order, flips, drop, idempotent drain, logged writer failures[unit] src/core/dispatcher.test.ts::run history write path …::a delivered run is sealed after its reply…, src/core/dispatcher.test.ts::run history write path …::a reply that throws after the run loop completed…, src/core/dispatcher.test.ts::run history write path …::a reply slower than the registry TTL…, src/core/dispatcher.test.ts::deterministic ops fast-path …::a non-onboarded repo natural-language ask falls through…, src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a fenced finishing means another generation owns the run…, src/core/dispatcher.test.ts::agent:ship (pipeline)::a final reply that throws writes the run record as \failed`…, src/core/runEnding.test.ts::createRunEnding — seal after the reply, records after the seal::*`
RunsService projects a finished registry row's status/finishedAt from the summary; unfinished rows sort first, finished rows by finishedAt desc[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::a finished registry row carries the status and finishedAt …, ::orders by the store's key …
Store failure: listRuns warns once per failing call with the error message, never a token; storeUnavailable: true[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::degrades to live rows + storeUnavailable when the store throws, warning once …
A head-truncated backlog (RunSnapshot.truncated) stamps the diagnosis truncatedInput: true; formatFrictionReport and the friction/proposals report say so; a full backlog leaves the field absent; normalizeDiagnosis keeps it[unit] src/core/runFriction.test.ts::analyzeRunFriction — truncated input …::*, src/core/selfImprovement.test.ts::formatSelfImprovementReport::counts runs diagnosed on a truncated event stream …, src/core/dispatcher.test.ts::run history write path …::a completed run's registry summary says \completed`; a truncated backlog stamps the diagnosis `truncatedInput` …`
An inline command run (`friction reportpropose, cron firings) is created with RunMeta(agentcommand, the caller's channel/user/thread) and persisted through the same writer: status from ok, events [input, answer]`, so it outlives the 60 s TTL
Run in flight across a container replacement → resumed, or failed visibly (its card edited, never frozen), within 60 s of the new container's Slack socket connecting; the run's record exists afterwards with a terminal status[agent] Start a long coding run, then trigger a bot deploy so the run is in flight when the old container receives SIGTERM; after the new container logs its Slack connected, watch the run's status card and runs get id=<id>: within 60 s the card must show a terminal state (resumed-and-finished, or failed with a reason) and the record must be retrievable. Until the resume step ships, the expected outcome is the failed-visibly branch via the orphaned-card sweep.
A finished run is readable long after eviction in production[agent] Behind Access, open a run's page more than 60 s after it finished: request, context, tool steps, reply and the grey finished · completed header render; /runs?all=1 lists it tokenless with duration and finished-at; the default /runs omits it; runs list --json from the CLI and GET /api/runs.list?status=finished (service token) both include its id.
Tombstone written at start: the run's first put is a provisional interrupted record — finishedAt = startedAt, the request/context events, isRunRecord-valid — and the finish write replaces it (one stored row, final status/events win)[unit] src/core/dispatcher.test.ts::run history write path …::tombstone-first provisional records …::a provisional interrupted record is written at run start — finishedAt = startedAt, the request/context events — and the finish write replaces it (goes red when the provisional write is removed)
The provisional write never marks the run persisted: a run whose finish write is permanently lost keeps persisted unset; the writer's provisional writes skip onPersisted but keep retries and drain accounting[unit] src/core/dispatcher.test.ts::…::the provisional write never marks the run persisted: a run whose finish write is lost keeps \persisted` unset, src/core/runHistoryWriter.test.ts::createRunHistoryWriter::a provisional write … skips onPersisted but is stored, retried and drain-counted like any write`
A live run never lists as interrupted: its tombstone is suppressed under all AND finished while the run is unfinished in the registry, and getRun serves the live row[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::a live run's provisional interrupted tombstone … never surfaces … (goes red when the merge fix is reverted)
Crash case (no drain): with the registry empty, the terminal tombstone is the record — visible in runs list --status finished as interrupted[unit] src/core/runsService.test.ts::RunsService.listRuns — read merge::a crash leaves the tombstone as the record …
The drain-deadline write carries the FULL snapshot: interruptedRunRecord(summary, snap, now) yields every event published so far with status: interrupted, finishedAt = the drain clock, the run's identity/label[unit] src/core/dispatch/record.test.ts::…::interruptedRunRecord (the drain deadline's seam) builds a full-snapshot interrupted record from a live run's summary + snapshot
The drain's abandonment pass writes one PROVISIONAL record per unfinished registry run (never a finished one), from the full snapshot, and reports the count so main() knows whether to await the write budget[unit] src/core/dispatch/record.test.ts::…::writeAbandonedRunRecords (the drain deadline's pass) writes one PROVISIONAL full-snapshot interrupted record per unfinished run, skips finished ones, returns the count (only the Promise.race budget still lives in main() — see the [agent] row below)
Final beats provisional: a provisional write in retry backoff stands down when the run's final write is enqueued (two puts, the final record last, no loss counted); the stand-down is per id; a provisional write enqueued after the final one never attempts[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::a final write for the same id stands down a provisional write sitting in retry backoff …, …::the stand-down is per id …, …::a provisional write enqueued after the run's final write is dropped before its first attempt (backoff test goes red when the stand-down check is removed)
The DO accepts interrupted through the SHARED validator (no worker code change; redeploy required) and the finish put replaces the provisional record whole (rewritten: true, final events)[unit] src/core/runRecord.test.ts::isRunRecord::accepts every terminal status — \interrupted` … included …, deploy/cloudflare-memory/runs.test.ts::run history routes::tombstone-first …: the DO accepts status `interrupted` (shared validator), and the finish put replaces the provisional record whole`
interrupted renders everywhere a status shows: red dot/chip on the index and run page, the word passes through statusLabel unchanged; runs list filters stay finished-ness (`activefinished
Deploy-race re-run: a run SIGTERM'd mid-flight and abandoned at the drain deadline has a runs get-retrievable record with a terminal status within 60 s of the next container's connect, and the [drain] wrote interrupted record for <id> (<n> events) line appears[agent] Re-run the deploy-race exercise (the container-replacement row above) after deploying the state Worker THEN the bot; check runs.list?status=finished for the probe's thread and the drain log line.
28: the fence — the owner passes, another generation is fenced, an unknown run is named; the claim decision (no live run, a different live run refused with its identity, the owner's re-claim idempotent)[unit] src/core/runLedger/decisions.test.ts::decideClaim — one live run per thread::*, ::checkFence — the owner generation on every write::*
28: the Worker client — routes and bodies, the bearer, the step write's transcript-first order and its stop on a fenced transcript write, fenced answers as results, RouteMissingError/TransientStoreError/PermanentStoreError, a malformed id never leaves the process[unit] src/core/runLedgerWorker.test.ts::WorkerRunLedger::*
28: the reference ledger runs the whole protocol — claim → seed → steps → finishing → finish, thread refusal, every owner write fenced, heartbeat/stop, reclaim with the last step and unconsumed inbox, handoff[unit] src/core/runLedger/inMemory.test.ts::InMemoryRunLedger::*
29: claim, thread-live 409 with the live run, idempotent re-claim, /runs/live lists the row, a live run is absent from the finished listing; validation 400s and 401[unit] deploy/cloudflare-memory/runLedger.test.ts::run ledger — claim and admission (item 29)::*
28: the object fences heartbeat, append, step, state, finishing and finish from another generation; heartbeat extends the lease and reports a stop; stop says whether the owner is live[unit] deploy/cloudflare-memory/runLedger.test.ts::run ledger — the fence (item 28)::*
30: the flusher — one timer per window, maxEvents sends at once, flush()/pending(), a failed send reported and dropped, close() refuses later pushes[unit] src/core/runLedger/flusher.test.ts::createAppendFlusher::*
30–31: append lands run_events rows by seq (the finished-runs routes do not see a live run), an over-cap event is 400, step records replace by step; the inbox increments seq from any generation; state replaces[unit] deploy/cloudflare-memory/runLedger.test.ts::run ledger — steps, events, inbox, state (items 30–31)::*
31: reclaim selection (expired and handoff, never a live lease, equal-now is expired); the phase CAS table; the transcript-completeness rule (resume, run-step-fresh, interrupted with the reason)[unit] src/core/runLedger/decisions.test.ts::selectReclaim — expired leases and handed-off runs::*, ::phaseTransition — the CAS table::*, ::transcriptCompleteness — what a resume does with what it finds::*
31, 33: finishing is a CAS taken once; finish writes the record and removes every live row in one step, the record lists as finished, the thread is free again, a mismatched record id is 400; handoff marks this generation's live runs; reclaim takes expired and handed-off rows with the last step and unconsumed inbox, re-owns them, leaves a live lease alone[unit] deploy/cloudflare-memory/runLedger.test.ts::run ledger — finishing, finish, handoff, reclaim (items 31, 33)::*
32: part rows carry idx/part and the part verbatim, attachments over the threshold by reference, small ones inline, an over-budget part refused by name; request chunking under the fence; assembly in (idx, part) order re-inflating references, naming a missing turn, part or attachment[unit] src/core/runLedger/transcript.test.ts::*
32: the transcript object — unknown-run before an owner, the owner writes and another generation is fenced, read in (idx, part) order with attachments, owner replaced by reclaim, clear empties everything, a 1.4 MB part row under the fence, malformed rows/attachments/ids 400[unit] deploy/cloudflare-memory/runTranscript.test.ts::run transcript object::*
35: the runner reports each step before its tools run — the turns since the last report (seed + reports = the model's conversation), their first index, the calls in flight; a report that throws fails the step before any tool runs[unit] src/runner.test.ts::step reports (docs/reference/specs/run-history.md item 35)::*
35: the write-through — the generation id; claim with prompt/tools/card/meta and the seed; a thread with another run's row, missing routes (warned once), a claim that kept failing → untracked; a refused seed detaches but the finish still clears the row; a ship claim without a seed[unit] src/core/runLedger/writeThrough.test.ts::mintGeneration::*, ::open — claim and seed::*
35: steps — turns first then a record numbered from 1 with the registry seq, the turn index after the write and the calls in flight; a fenced step detaches without throwing; one retry after a backoff on a transient failure, none on a permanent one[unit] src/core/runLedger/writeThrough.test.ts::step — turns first, then the record::*
35: events batched with the registry seq and nothing after the finish; state merged and coalesced, a transient state failure retried after a backoff and left dirty for the next patch when the retry fails too; the heartbeat extends the lease and relays a stop once per mode; a refused heartbeat detaches and stops the timer[unit] src/core/runLedger/writeThrough.test.ts::events, state, heartbeat::*
35: finishing is taken once; the sink's finish replaces the live rows with the record (events flushed first) and never touches the fallback; a refused finish or missing route goes to the fallback store; a transient finish failure propagates for the writer's retry, which is idempotent[unit] src/core/runLedger/writeThrough.test.ts::finishing and finish::*
35: the writer routes one record through via with the same retries and accounting, the store untouched for it and used again for the next write[unit] src/core/runHistoryWriter.test.ts::createRunHistoryWriter::\via` routes one record to another sink with the same retries and accounting; the store never sees it, and the next write uses the store again`
35: end to end in the dispatcher — the claim carries the system prompt, tool definitions only, the card handle and the meta; the seed equals the first model call's conversation; the step record precedes the tools and names both calls; the checklist lands in state; finishing is set when the reply goes out; the finish empties the live rows, lands the record in the ledger with every event in seq order, and the fallback is never used[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::claims the run once its prompt exists (system, tools, card, meta, seed), records each step before its tools, appends events, takes finishing before the reply and finishes through the ledger
35: an untracked run (another run's row on the thread) runs and replies as before, its record reaches the store, one warning; a review's verdict lands in the ledger state[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a thread whose ledger row belongs to another run leaves this run untracked: it runs and replies as before, its record goes to the store, one warning, ::a review's verdict lands in the run's ledger state as it is submitted
35: a ship pipeline is claimed without a seed, takes finishing before its final reply and finishes through the ledger[unit] src/core/dispatcher.test.ts::agent:ship (pipeline)::a pipeline is claimed on the run ledger without a seed (item 35) and finishes through it: the live row goes, the record lands in the ledger, the plain store is never the fallback
36: reclaim at boot — a run the completeness rule refuses closes interrupted with a record from the ledger's events, meta and identity (row, steps, transcript gone; the verdict is the reason); a finishing row closes with its recorded finalStatus or completed; no step record and a handoff are named; a partial step write is named; rows another generation holds are left alone and returned as liveElsewhere; a row this generation owns is never taken however stale its lease; a closure carries the note its card and (for a pipeline) its thread get, naming the PR the events recorded; missing routes or an unreachable ledger are a warning and an empty outcome; one run's failure is reported and does not stop the others[unit] src/core/boot.test.ts::reclaimRuns::*, src/core/runLedger/decisions.test.ts::selectReclaim — expired leases and handed-off runs::*, src/core/runLedger/inMemory.test.ts::InMemoryRunLedger::reclaim takes the expired and handed-off runs, gives them to the new generation with the last step, the unconsumed inbox and the jobs, and re-fences the transcript, deploy/cloudflare-memory/runLedger.test.ts::run ledger — finishing, finish, handoff, reclaim (items 31, 33)::handoff marks this generation's live runs; reclaim takes expired and handed-off rows with the last step, the unconsumed inbox and jobs, and re-owns them; a live lease is left alone
36: the record a reclaim closes with carries the row's identity, meta and every appended event; the live-events read answers a live run's rows in seq order, an unknown run empty, a bad id 400; a reclaim says which phase each row came from[unit] src/core/boot.test.ts::reclaimRuns::closes a run the rule refuses (a partial step write) \interrupted` with a record built from the ledger's events, meta and identity; the row, steps and transcript go; the reason is the verdict, deploy/cloudflare-memory/runLedger.test.ts::run ledger — steps, events, inbox, state (items 30–31)::, src/core/runLedger/inMemory.test.ts::InMemoryRunLedger::`
36: the finish reaches the ledger before the workspace release completes, and finalStatus is on the row when finishing is taken[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::the finish reaches the ledger BEFORE the workspace release completes (item 36), and the final status rides on the row before finishing: a slow sandbox teardown never keeps the thread's row live after the reply
36: POST /admin/crash — a deploy:write bearer gets 202 with the generation and the hard exit is deferred past the response; 401/403/503 exit nothing; only POST[unit] src/channels/adminCrash.test.ts::POST /admin/crash::*
36: the sweep's isLive predicate — a card driven here or marked live on the ledger is never an orphan; the foreign set is refreshed from its source on every scan (a dead generation loses its hold, a failed refresh keeps the set); a replied run's card is closed with how it ended, an interrupted run's card is closed with its closure note, a failed edit is isolated[unit] src/channels/slackCatchUp.test.ts::findOrphanedCards (pure selection of the bot's own frozen live cards)::*, src/channels/slack/statusCard.test.ts::live cards::*
37: the settlement rule — bash and GitHub writes get the restart result, reads/write_file/update_status/submit_* run again, an unknown tool gets the not-available result; the plan for resume (settlements from the last turn, counters carried, nothing in flight → continue from a user turn), run-step-fresh (all calls fresh, counters advanced, update_status-only keeps the turn), and every interrupted reason[unit] src/core/runLedger/resume.test.ts::settlementFor — D4::*, ::planResume::*
37: the runner's entry — the settlement runs before the first model call (a read re-runs, bash gets the note), the model sees transcript + results in the calls' order, the resumed note and the events name what happened, the recorded step is not reported again and the next report carries the results turn; an unrecorded step is reported first with no turns and then its calls run; the plan's remaining budget is the deadline[unit] src/runner.test.ts::resume (docs/reference/specs/run-history.md item 37)::*
37: the registry creates a run under a given id with its original start and earlier events replayed under their seqs and the counter continuing past the highest; the write-through adopts a reclaimed row without claim or seed — heartbeat at once, steps and seqs continuing, state merging[unit] src/core/runRegistry.test.ts::RunRegistry.create::a resume creates the run under a given id, with its original start and its earlier events replayed under their seqs; new events continue past the highest, src/core/runLedger/writeThrough.test.ts::adopt — a reclaimed run continues under this generation (item 37)::*
39: finishing is a tri-state gate — ok once, a refusal is fenced (and detaches), a run detached by a fence keeps answering fenced, one detached otherwise or an unreachable ledger is unavailable; a fenced write tells the run once (onFenced); liveRuns names the runs driven here; handoff marks the resumable ones (not a ship claim, not a detached run), remembers it, and reports a ledger failure instead of throwing[unit] src/core/runLedger/writeThrough.test.ts::finishing and finish::*
39: a fenced finishing means no reply, no card close and no record from this generation[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a fenced finishing means another generation owns the run: nothing more reaches the thread and no record is written from here — the run is theirs (D9)
39: live — deploy restart (or a rollout) with a resumable run in flight: the old container logs [drain] handed N run(s) to the next generation and exits within seconds; the new one resumes the run (item 38's receipt) — nobody waits on the run[agent] With a slow run in flight, switchboard deploy restart (since durable-runs Phase 5 the preflight warns for runs in flight and proceeds — no --force); time from SIGTERM to the new generation on /healthz, then the item 38 checks.
38: the reclaim hands a resumable row back untouched (row ours, record, transcript, events kept) and closes the rest; a handed-off row with a whole transcript is resumable; the sweep repeats every lease interval, reports only non-empty outcomes, runs one pass at a time, and a failing pass is a warning[unit] src/core/boot.test.ts::reclaimRuns::*
38: the launcher — the agent's static tools as known tools, the request text from the input event, the message pinning agent/model/effort under the row's identity, the repo context from the meta; a resumable run is dispatched with the full ResumeContext; interrupted plans, unknown agents and unresumable channels are closed with their reason; a throwing dispatch is logged and the others still launch[unit] src/core/resumeLaunch.test.ts::the pure pieces::*, ::launchResumes::*
38: end to end — a reclaimed run resumes under its own id: the row adopted, the transcript the conversation, the calls in flight settled before the first model call (a known tool re-runs and refreshes the row's state, an unknown one gets the not-available result), the earlier events replayed under their seqs with no second input event, only this generation's events appended, the finish closing the same row[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a reclaimed run resumes under its own id (item 38): the row is adopted at admission, the transcript is the conversation, the calls in flight are settled before the first model call, the earlier events are replayed under their seqs, and the finish closes the same row
38: a resume onto a thread with a newer run closes the reclaimed row interrupted with no reply; a resumed dispatch that ends before its run starts closes its adopted row[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::a resume onto a thread that has a newer run in flight is never steered or refused as a follow-up: the reclaimed row is closed interrupted with no reply and no run, ::a resumed dispatch that ends before its run starts (a repo refusal) closes the adopted row interrupted instead of leaving it for the sweep to relaunch forever
38: the Slack IO of a resumed run edits the existing card instead of posting one (a deleted card falls back to a fresh one), and resumeSlackIO builds the thread IO from the row's parts[unit] src/channels/slack.test.ts::SlackIO.status on a resumed run (existing card)::*
40: the step record carries the inbox seq the run has consumed; pushInbox answers the ledger's seq for a live run, undefined with a warning when refused or failed[unit] src/core/runLedger/writeThrough.test.ts::finishing and finish::the step record carries the inbox seq the run has consumed (run-history item 40); pushInbox hands back the ledger's seq for a live run — undefined, with a warning, when the ledger refuses or fails
40: readInbox is the inbox past a seq — the same slice the reclaim hands over, on demand; empty for an unknown run; the Worker route validates afterSeq; the client posts it[unit] src/core/runLedger/inMemory.test.ts::* (readInbox), src/core/runLedgerWorker.test.ts::WorkerRunLedger::readInbox posts the run id and the seq to read past, and returns the Worker's items (item 40); [workerd] deploy/cloudflare-memory/runLedger.test.ts::* (/runs/inbox/read)
40: the resume plan carries the last record's inbox seq[unit] src/core/runLedger/resume.test.ts::* (the inboxConsumedSeq: 7 assertions)
40: the reclaim hands a resumable run its inbox past the last record; a row live elsewhere names its thread[unit] src/core/boot.test.ts::* (r.inbox, liveElsewhere shape)
40: the launcher passes the inbox through to the ResumeContext[unit] src/core/resumeLaunch.test.ts::launchResumes::plans and dispatches a resumable run with the full ResumeContext — the plan, the row, the last step, the events, the highest seq, the repo context — under the row's identity
41: a ledger row not in the registry lists as a live row under all/active, never finished, with its meta, events, activity and ownerGen; its tombstone never surfaces; a row also in the registry lists once (registry); the predicate applies; getRun/getRunEvents/getRunFriction/stopRun answer for it; a ledger failure is one warning and the registry alone; liveElsewhere is the slice for the index[unit] src/core/runsService.test.ts::RunsService with the run ledger — one registry across generations (run-history item 41)::*
41: one ledger listing serves every read within the TTL; the rows' events are read in parallel; a failed listing is not kept[unit] src/core/runsService.test.ts::RunsService with the run ledger — one registry across generations (run-history item 41)::one ledger listing serves every read within the TTL — a page view's run, events and friction reads cost one listLive; the events of several rows are read in parallel; a failed listing is not kept
41: a run live elsewhere opens tokenless — the page in history mode with the ledger's events and no token, the events route a replay that ends, friction the live diagnosis, the tokenless stop through the ledger[unit] src/channels/liveView.test.ts::live view on RunsService: history pages + index toggle …::index: active by default, everything with ?all=1::a run live on the ledger under another generation opens tokenless (item 41): the page in history mode with the ledger's events and no token, the events route a replay that ends, friction a live diagnosis, and the tokenless stop goes to the ledger
41: the default /runs seeds the rows live elsewhere after this process's rows, tokenless, with the live count, and still never reads the store[unit] src/channels/liveView.test.ts::live view on RunsService: history pages + index toggle …::index: active by default, everything with ?all=1::the default view also seeds the runs live on the ledger under another generation (run-history item 41): tokenless, live, after this process's rows; still never a store call
41: live — during a rollout (or after a kill, before the resume), /runs on the new container shows the run the old one is still driving (or the row awaiting launch) as live with its generation; the run page opens; runs stop on it stops it[agent] Start a slow run, deploy restart --force; while both containers exist, open /runs on the new one and the run's page; then switchboard runs stop <id>.
38: live — kill the bot mid-run with POST /admin/crash while a step is in flight; within two lease intervals of the boot the run continues on the new generation: the card's frames carry on, the run page shows the resumed note after the pre-kill events, the reply lands in the thread, and runs get <id> is completed with one contiguous event stream[agent] Same procedure as item 36 on a slower run (agent:research model:anthropic/claude-fable-5); check /healthz for the new generation, then the thread and the record.
36: live — kill the bot mid-run with POST /admin/crash; the next container's boot log shows [reclaim] <id> <thread> closed interrupted (from live; …), runs get <id> returns an interrupted record whose events are the ones published before the kill, the card is closed ❌ by the sweep, and POST /runs/live is empty[agent] With a deploy:write token: start <@bot> agent:general … on a quiet thread, then curl -s -X POST -H "authorization: Bearer $TOKEN" https://<bot>/admin/crash; watch /healthz for a new generation; check the run and the ledger.
35: live — a run's rows are visible on the ledger while it runs: POST /runs/live on the state Worker lists the row with ownerGen equal to the bot's /healthz generation, and the row is gone once the run finishes[agent] With the state Worker and then the bot deployed: start any run; curl -s -X POST -H "authorization: Bearer $MEMORY_TOKEN" -H 'content-type: application/json' -d '{"storeKey":"runs:default"}' https://<state-worker>/runs/live shows the run; compare ownerGen with curl -s https://<bot>/healthz | jq .generation; re-run after the reply → the run is absent and runs get <id> returns its record.
42: the CAS table admits attaching → live and attaching → finishing, never attaching → handoff or anything back to attaching; decideClaimWrite is insert / promote / refresh / keep; reclaimPhase keeps attaching and makes every other phase live[unit] src/core/runLedger/decisions.test.ts::phaseTransition — the CAS table::an \attaching` row (reserved at admission, item 42) goes live when its prompt lands or finishing when the dispatch fails before that; nothing goes back to attaching and an attaching row is never handed off (it is not resumable — a restart re-dispatches it), ::decideClaimWrite — what a claim does to the thread's row (item 42)::, ::reclaimPhase — the phase a reclaimed row lands in::`
42: the reference ledger and the Durable Object agree — a claim with phase: attaching reserves the thread with the request and no prompt (another run refused meanwhile); the owner's claim with the prompt promotes in place (identity and start kept); a re-claim on a live row changes nothing; a bad phase is 400; reclaim keeps an expired attaching row's phase and hands the request and inbox back; abandon drops the live rows with no record, fenced, unknown afterwards[unit] src/core/runLedger/inMemory.test.ts::InMemoryRunLedger::a run reserved \attaching` at admission (item 42) holds the thread with its request and no prompt; the owner's claim with the prompt promotes it to live in place (card, prompt, tools, state land; identity and start stay); another run on the thread is refused meanwhile; a reclaim of an expired attaching row keeps the phase and hands the request back, deploy/cloudflare-memory/runLedger.test.ts::run ledger — claim and admission (item 29)::a claim with `phase: attaching` reserves the thread before the prompt exists (item 42): the row lists as attaching with its request and an empty prompt; the owner's later claim with the prompt promotes it to live in place; reclaim keeps an expired attaching row's phase`
42: the write-through — reserve claims an attaching row with the request and card, starts the heartbeat, is tracked and live but not resumable and never handed off; open with the reservation promotes the same tracked run (one heartbeat, seed written, resumable from there); a reservation another generation took is fenced through the heartbeat or the promotion (onFenced once, undefined, nothing seeded); a refused reservation is undefined with one warning; abandon drops the row with no record and is a no-op on a fenced run[unit] src/core/runLedger/writeThrough.test.ts::reserve — the row before the prompt (item 42)::*
42: the reclaim hands an expired attaching row to the launcher as a restart with its request and inbox, and closes one without its request interrupted naming why[unit] src/core/boot.test.ts::reclaimRuns::a row reserved at admission (item 42) whose owner died is handed to the launcher as a restart — the row still attaching, its request and unconsumed inbox in hand, nothing closed; one reserved without its request is closed interrupted naming why
42: the launcher dispatches a restart's own request — directives, sender, link, attachments — under the row's identity with a RestartContext, and closes a row whose request it cannot read[unit] src/core/resumeLaunch.test.ts::launchResumes::a restart (item 42) dispatches the row's own request — text with its directives, sender, link, attachments — under the row's identity with a RestartContext carrying the row and the inbox; a row whose request cannot be read is closed with the reason
42: RunRegistry.discard drops a live run with no finished frame — the index feed's removed, the end frame to a live subscriber, the id unknown afterwards — and is a no-op for an unknown or a finished run[unit] src/core/runRegistry.test.ts::RunRegistry.discard::*
42: end to end in the dispatcher — the row is reserved BEFORE the attach with the request, card and no prompt under the run's id, the registry row (label, token, the same start) exists from that moment and is the index's only row for the run, and the claim promotes the ledger row in place; a dispatch that ends before its prompt exists abandons the row and discards the registry row, with no record and an index feed that saw the row come and go; a restart runs the request again under the row's id and card, folds the carried follow-ups in with no second ack, promotes the row and finishes it with the original start; a restart onto a thread with a newer run closes the row interrupted with no reply[unit] src/core/dispatcher.test.ts::run ledger write-through (docs/reference/specs/run-history.md item 35)::the run is reserved on the ledger BEFORE the workspace attach (item 42): an attaching row with the request (text, sender, link, attachments), the card and no prompt, under the id the run will have, and the registry row — label, token, the same start — exists from that moment too; the claim once the prompt exists promotes that row in place — one row, one id — and the finish clears it, ::a dispatch that ends before its prompt exists — here the attach's ask-once branch refusal — abandons its reservation and discards its registry row (item 42): both go with no record and no warning, the index feed sees the row come and go, so nothing restarts or lists a run that never started, ::a run reserved at admission whose owner died is restarted under its own id (item 42): the request is dispatched again from the row, the card is the row's, the follow-ups steered in meanwhile are folded in with no second ack, this generation promotes the row and the finish closes it — the record keeps the original start, ::a restart onto a thread that has a newer run in flight closes the reserved row interrupted with no reply and no run (item 42)
42: live — kill the bot (POST /admin/crash) while a run is still attaching its workspace (the card reads attaching the workspace…, POST /runs/live shows its row with phase: attaching and no system); the next generation's boot log shows [reclaim] <id> <thread> restartable (from attaching …) and [resume] <id> <thread>: restarting from its request, the same card carries on, and the run finishes under the same id with startedAt the original[agent] On a quiet thread, mention a review or research run against a repo whose resident is busy (or a cold sandbox clone), crash within the attach, watch /healthz for the new generation, then the thread, runs get <id> and POST /runs/live. Pending live receipts.