Resident repo environments

One always-warm service per onboarded repo: a dedicated resident Worker (switchboard-resident) fronts per-resource Durable Objects on Cloudflare Sandbox 1.0 (@cloudflare/sandbox@next, exact-pinned). Residency is a generic resource-typed primitive — every route carries a resource id of the form <type>:<id> (repo:<owner>/<name> is the first and only type; the GitHub clone URL derives from the id), and the Durable Object name IS the resource id. This file covers the service surface (auth scopes, the atomic cap, onboard/offboard/reconfigure/status/residents, x-env hardening), the lifecycle engine (alarm-driven provisioning, wake-path rehydration, refresh alarms, the watchdog, stamped snapshots, repo-scoped GitHub token minting), the operator data plane (/attach /exec /read /write: per-thread worktrees off the bare mirror, ref binding, dep materialization, OS-user isolation, the mirror mutex, inactivity eviction), the bot-side wiring (ResidentExecutor, warm-gated selection, the per-repo gate, named fallback with a probe circuit breaker), the dispatch integration (pre-model repo/ref resolution, the needs-ref ask-once flow, resident prompt variants), the management surface (repo onboard/offboard/reconfigure/rebuild/list chat commands behind the fail-closed canManageRepos gate, --dry-run itemized plans, the /rebuild route, the onboard-time installation-membership check, and the watchdog's auto-rebuild), and the deterministic-ops surface (the /op route running readonly command-table entries in disposable per-op checkouts, per-entry effects profiles, the bot-side Operations seam with resident-backed and local implementations, and the dispatcher's modelless fast-path with repo test/build commands and conservative natural-language recognition), and residency garbage collection (items 45–46: finished-ref reclamation polled from the refresh alarm, and opt-in resident-level LRU eviction past the cap).

Code: deploy/cloudflare-resident/ (worker.ts, wrangler.jsonc, Dockerfile); bot side: src/execution/resident.ts (client + ResidentOperations), src/execution/executor.ts (LocalOperations), src/execution/factory.ts (selection + circuit breaker), src/config.ts (canUseRepo, canManageRepos), src/core/repoContext.ts (repo/ref resolver), src/core/residentAdmin.ts (the resident admin client + parseSlug/validRef), src/core/commands/repo.ts (the repo.* registry commands: management verbs + the repo test/build op verbs), src/channels/residentsView.ts (residents dash: Access-gated GET /residents + /residents/<owner>/<name>), src/core/operations.ts (the Operations seam + op recognizer), src/core/dispatch/resolve.ts (resolveTarget: the repo/ref resolution started before the ack), src/core/dispatch/authorize.ts (authorizeRepo: the not-onboarded, unverified and access gates, item 29), src/core/dispatch/fastPath.ts (the ops fast-path), src/core/dispatcher.ts (ask-once + prompt selection + note surfacing), src/agents/registry.ts (CODING_SYSTEM_RESIDENT, REVIEW_SYSTEM_RESIDENT), src/cli.ts (--thread); the resident engine's pure decisions, imported by the Worker: src/execution/residentIncarnation.ts (the incarnation id and the leases, item 22), src/execution/residentStepPlan.ts (the engine steps' read-then-act plans, item 22)
Docs: AGENTS.md; the thread-sandbox counterpart is execution.md
Tests: src/execution/resident.test.ts (incl. ::ResidentOperations.run), src/execution/residentHead.test.ts (item 51), src/execution/executor.test.ts::LocalOperations, src/execution/factory.test.ts::makeExecutor resident selection, src/config.test.ts::per-repo access (canUseRepo) + ::repo management gate (canManageRepos), src/core/repoContext.test.ts, src/core/commands/repo.test.ts (incl. ::repo test / repo build …), src/core/operations.test.ts, src/core/dispatch/authorize.test.ts (the repository gates, one outcome each), src/core/dispatcher.test.ts::resident repo dispatch + ::repo/ref resolution + resident prompt selection … + ::repo management commands … + ::deterministic ops fast-path …, src/agents/registry.test.ts::resident prompt variants, src/cli.test.ts; the resident Worker's request/engine code has none (deploy/* convention: verified by npm run typecheck in deploy/cloudflare-resident plus the live checks below), but its PURE decision logic is unit-tested under plain Node: deploy/cloudflare-resident/preflight.test.mjs (deploy preflight, item 44) and deploy/cloudflare-resident/gc.test.ts (residency GC, items 45–46, incl. parseRefListing); the refresh-cycle perf helpers are unit-tested from src/: src/execution/residentCleanliness.test.ts (the one-spawn clean check, item 16b), src/execution/residentDepCache.test.ts (depCacheScript/parseDepCacheScriptOutput/mutableCacheSwapScript, item 18), src/execution/residentDepsStore.test.ts (the deps store: layout, hit/join/install plan, semaphore size, scratch clone, commit script, listing, eviction — item 57), src/execution/residentRefresh.test.ts::withTimeout (the R2 transfer budget, items 5/7), src/execution/residentIncarnation.test.ts (the incarnation id, the lease predicate, take/release, the in-flight row — item 22), src/execution/residentStepPlan.test.ts (each step plan called twice, the snapshot's compare-and-swap — item 22); the disk budget (item 55) is src/execution/residentDiskBudget.test.ts (measurement, reserve, projection, admission, eviction order, the refusal text) plus deploy/cloudflare-resident/instanceSizing.test.ts (the instance arithmetic over the same reserve constants); the purge decision (item 60) is src/execution/bindingPurge.test.ts

Base URL for all [agent] checks: the resident Worker's own hostname, written https://<resident hostname> (or ...) below. $ADMIN / $OPERATOR are the RESIDENT_ADMIN_TOKEN / RESIDENT_OPERATOR_TOKEN secrets (npm run secrets in deploy/cloudflare-resident). A convenient fixture for the [agent] checks is a tiny public Node repo such as repo:jshttp/vary (default branch master, no committed lockfile); Go repos (e.g. boldsoftware/meat) are unsuitable — the container image has no Go toolchain. With the GitHub App configured, onboarding is limited to repos in the installation (item 35).

Behavior

Three bearer scopes, fail closed: the admin token guards POST /onboard, POST /offboard, POST /reconfigure, POST /rebuild, all of POST /debug, GET /residents; the optional read token (RESIDENT_READ_TOKEN) opens ONLY GET /residents and the pure-read /debug ops info, schedules, threads — any other op with it is 403 (authenticated but under-scoped; 401 stays "no valid bearer") even though the route is reachable (checked after auth, so an unauthenticated caller still learns nothing); it never reaches an operator or admin route, and is for dashboards and humans who need to look without being able to change anything; the operator token guards POST /attach /exec /read /write /op and GET /status. Admin is a strict superset (valid on operator routes); the operator token never opens an admin route. Comparison is constant-time (byte-XOR accumulate), never a plain !==. Unset/empty secrets grant nothing; unknown paths answer 401 before revealing 404. The only unauthenticated route is GET /healthz (deploy wake ping; touches no DO, returns {ok:true, build:{commit, builtAt?}} — the commit the deploy injected, which makes a deploy's edge propagation provable from outside without auth; execution.md item 13 replaced the hand-bumped u marker with it).
Atomic cap: the singleton ResidentRegistryDO holds the onboarded set + command table and enforces RESIDENT_CAP (6 — the team's concurrent-repo count) in one input-gated read-count-insert — an over-cap onboard is refused with 429 by the registry, never by the platform (max_instances is 10, deliberately above the cap).
Onboard returns immediately; provisioning is alarm-driven: POST /onboard {resource, commands:{test,build,...}, defaultRef, diskBudgetMb?, provisioningTimeoutMs?} validates, inserts into the registry, persists state onboarding, arms two schedules on the resident DO (the provisioning run at +1s and the fail-closed deadline at +provisioningTimeoutMs — via the Container schedule() API, never the raw DO alarm slot, which the Sandbox base class owns), and answers 202 without starting a container. If arming the resident fails, the registry slot is freed (no half-onboarded residents). repo ids must be lowercase GitHub <owner>/<name> slugs (the whole resource ≤ 63 chars — it doubles as the sandbox id).
Provisioning pipeline: git clone --mirror https://github.com/<slug>.git into /workspace/mirror (root-owned) → resolve the default branch (the configured defaultRef when it exists in the mirror, else the mirror's HEAD) → local clone into /workspace/checkout, chowned to the unprivileged build user worker1 → full install (when the command table has one) + build via su -s /bin/bash worker1 -c 'cd /workspace/checkout && <cmd>', token-free — then two stamped snapshots (mirror + checkout, createBackup({localBucket:true}) into the BACKUP_BUCKET R2 binding, 1-year TTL as a leak backstop), facts {defaultRef, sha, lockfileHash} recorded in DO storage, state warm, refresh chain armed. Every step runs under the per-resource provisioningTimeoutMs as its exec budget; a failed step → down with reason provision-failed at <step>: … and the registry slot is KEPT (the admin sees the named reason on /status and offboards/re-onboards); a STUCK onboarding (deadline schedule or watchdog finds onboarding past its budget) → down(provision-timeout…) and the cap slot IS released.
Warm means rehydration, not persistent disk: DO SQLite is the source of truth (config, lifecycle state+reason, last-fetched sha, lockfile cache key, snapshot handles); container disk is a cache. On every wake (refresh alarm or attach, finding the runtime dead or the disk not matching the marker), the resident persists restoring (reason rehydrating) BEFORE any restore work, restores mirror + checkout from the stamped snapshots — the two restores run SEQUENTIALLY (mirror, then checkout — the SDK serializes backup operations on one queue anyway, so a concurrent pair only let the second one's clock run while it waited) and each is judged by its BYTES, not by a clock (item 61, restoreWithProgress → the pure judgeRestoreProgress in src/execution/residentRefresh.ts; the SDK's restore call takes no timeout, progress callback or AbortSignal): while du of the staging archive plus the target keeps growing the wait continues, a stall (no growth for RESTORE_STALL_MS) or the RESTORE_MAX_MS cap fails visibly as down(r2-restore-failed: <mirror|checkout> restore stalled|capped: …) with the bytes and timing named, instead of stranding restoring until the 30-min watchdog — and a previous attempt's restore still running is awaited before the pre-restore clean, never raced by it — re-verifies the stamp against the restored disk (mirror rev-parse must equal the stamped sha; the ls-tree lockfile key at that sha must equal the stamped key), then flips warm and records lastRestore {at, ms}. A restore failure → down(r2-restore-failed: …); a stamp that does not verify → down(snapshot-stamp-mismatch: …). Restore is provably not a re-clone: lastRestore.ms is a fraction of provisioning time, and snapshot ids/provisionedAt stay untouched.
Snapshot stamps: snapshots are written ONLY by onboarding provisioning and default-branch refresh — never by attach-time or op executions — and each is stamped {ref, sha, lockfileHash}. The lockfile cache key is the sha256 of git ls-tree <sha> -- <lockfile candidates> output from the MIRROR — a pure function of the commit. It is deliberately NOT a hash of the working directory: installs generate uncommitted lockfiles (npm writes package-lock.json), which would poison a disk-derived key and fail the post-restore stamp check — which is why the key derives from the committed tree.
Refresh alarm owns freshness: each resident self-reschedules onRefreshAlarm every 600s (< the 20m sleepAfter, so a healthy resident is re-warmed before the platform can sleep it; the alarm doubles as the keep-warm heartbeat). One cycle: rehydrate if needed → mint a repo-scoped token (skipped when the GitHub App is unconfigured; public repos fetch anonymously) → refreshing → git fetch --prune into the mirror → if the default-branch sha moved: plan the rebuild against the disk checkpoints (item 48) — update the checkout (fetch from the local mirror + reset --hard + clean -fdx, as worker1 — -x drops the gitignored build caches so the rebuild allocates FRESH inodes instead of writing through any hardlinked into already-attached, sha-pinned worktrees, review 1b; node_modules is excluded from the clean and install skipped when the committed lockfile key is unchanged), run build, write a new stamped snapshot pair (the mirror and checkout uploads run concurrently, each bounded by the 5-min R2_TRANSFER_TIMEOUT_MS — a hung upload degrades the cycle as snapshot-failed: … timed out instead of stranding refreshing for the watchdog), delete the replaced snapshot's R2 objects (both prefixes swept concurrently) → facts + snapshot updated in one storage write → warm. Failures degrade with a named reason (github-unreachable: …, <step>-failed: …, and disk-full: … for a full container disk — item 54, the one failure the resident repairs itself) while the last snapshot keeps serving; the chain re-arms itself in a finally for every outcome except down.
Token discipline: the resident Worker holds the GitHub App credentials (GITHUB_APP_ID/GITHUB_APP_INSTALLATION_ID/GITHUB_APP_PRIVATE_KEY) in its own wrangler secrets; the App JWT is RS256 over WebCrypto (crypto.subtle, PKCS#8 import with an ASN.1 wrapper for GitHub's PKCS#1 PEMs; base64url via btoa) — node:crypto is never touched. Every mint POSTs repositories: [<own repo name>] so a token never grants more than the resident's one repo; mints are cached per slug, but a cached token is only served while it has more than CREDENTIAL_EXPIRY_MARGIN_MS (~25 min) of life left — the same margin the per-exec refresh keys on, so the cache never hands a fresh attach a token it would have to immediately re-mint (this was 5 minutes, too little for a 20-minute exec to run under). The private key NEVER enters the container: git commands receive the minted token through a one-shot credential file under the root-only /workspace/.resident/ dir (credential.helper=store --file=… passed as argv; the file is deleted in a finally), never process-wide env and never argv-embedded token material. Install/build executions carry no token at all. A token-mint failure is a command-level error: recorded in lastRefreshError (token-mint-failed (command-level, fetching anonymously): …), the resident keeps serving, lifecycle is NEVER flipped by it — and the cycle continues with an anonymous fetch, exactly as when the App is unconfigured, so a public repo the App is not installed on stays fresh (the cycle ends warm with this cycle's mint error kept in lastRefreshError — only prior cycles' errors are cleared) and a private one fails at the fetch into a visible degraded(github-unreachable: token-mint-failed …; then <fetch error>) — the mint is named in the reason, so a GitHub-reachable-but-unmintable repo is never misread as a network outage. The cycle never returns early on a mint failure — an early return would freeze whatever state the resident was in, with a stale mirror.
Watchdog cron: every 10 minutes (strictly < SLEEP_AFTER 20m) the Worker iterates the registry and calls each resident's watchdogCheck() — storage/schedule reads only, containers are started by the re-armed alarms, not the watchdog. A dead refresh chain (no pending onRefreshAlarm schedule) is re-armed at +5s AND the resident is marked degraded(alarm-missed: …) until the re-armed refresh succeeds (an in-flight restoring is re-armed without the degrade flip). A refreshing/restoring marker older than STALE_MIDFLIGHT_MS (30 min) with no cycle or restore actually running (refreshesInFlight === 0, no hydration promise) is an orphan from an interrupted cycle (DO evicted by a deploy): the watchdog re-reads state right before flipping it (a cycle that started during its reads owns the state) and then marks it degraded(stale-mid-flight: …) and pulls the next cycle to +5s. Separately, and before that check so it is never skipped by its early return, the watchdog re-arms a dead SWEEP chain (live bindings exist, no onWorktreeSweep pending, and no sweep executing (sweepInFlight) — same eviction failure mode; previously only an attach re-armed it) at +5s with no state flip of any kind; /debug schedules reports sweep. A pending sweep row due further out than the current SWEEP_INTERVAL_S (+5 min slack) is treated the same way and replaced — a row armed by older code with a longer cadence (a daily sweep) would otherwise be honored until it fires; a deploy that shortens the cadence now takes effect within one watchdog pass. An onboarding resident past its recorded deadline (+30s grace) → down(provision-timeout…), all schedules cancelled, cap slot released (the DO releases it; the Worker removes the registry row as backstop).
Command table is admin-writable only: commands exist solely in the registry and change solely through /onboard and /reconfigure (admin scope). commands must include test and build; extra named commands are allowed (install is honored by the engine); reconfigure.commands replaces the whole table. Command strings execute inside the resident as worker1 — never on the Worker, never as root. The record also carries per-entry effects profiles (item 39), equally admin-writable-only.
Offboard is a full teardown: registry removal first (slot frees atomically), then resident teardown (cancel all three schedule names, delete the R2 objects behind the stored snapshot handles — they live under backups/<uuid>/, OUTSIDE resident/<resource>/, and the handles die with the storage wipe, so this runs first — best-effort container destroy, DO storage wipe), then deletion of every R2 object under resident/<resource>/. The response itemizes registryRemoved, schedulesCancelled, containerStopped, storageCleared, backupObjectsDeleted, r2ObjectsDeleted, errors.
Lifecycle states are persisted and causal: onboarding|warm|refreshing|restoring|degraded|down, with a reason string required on every degraded/down. GET /status (operator) returns exactly {state, reason} — operators see lifecycle, not config. GET /residents (admin) returns each registry record plus the live engine view: state/reason, defaultRef, sha, lockfileHash, provisionedAt, lastRefreshAt, lastRefreshError, lastRestore, the snapshot stamp with backup ids, pending-schedule counts, and threads — every thread binding (threadKey, ref, sha last attached at, user, deps, boundAt, lastAttachAt, evicted/evictedAt; newest attach first; worktreePath omitted as internal layout).
Admin debug surface (POST /debug, admin scope; built for these live checks and kept for operability): ops info (engine view without the registry), schedules, kill-refresh (simulate a dead alarm chain), refresh-now (pull the next refresh to +1s), stop-container (simulate a platform sleep), force-onboarding (fault injection for the watchdog timeout path), mint-token (attempt a mint; returns the command-level error shape, never token material), run-watchdog (the cron's exact pass, on demand), force-down (fault injection: persist down with a rehydration-flavored reason and stop the refresh chain, so the watchdog auto-rebuild path is exercisable without corrupting real R2 objects), plus the thread ops threads (enumerate bindings; worktreePath omitted, like the engine view), backdate-thread (age a binding's lastAttachAt), sweep-now (run the eviction pass now), purge-bindings (item 60: delete a load run's evicted synthetic bindings), the GC ops reclaim-now (item 45) and the registry-wide set-test-overrides (item 49: lower the effective cap / LRU floor for live over-cap checks; deploy-scoped). Side effects are explicit and admin-gated; nothing here returns secrets.
x-env hardening: caller-supplied x-env-* headers are ignored on every route (nothing reads request headers besides authorization; no request header is forwarded into a resident — a deliberate deviation from the thread-sandbox Worker). The engine's ONLY injected env var is GIT_TERMINAL_PROMPT=0 (so anonymous git fails fast instead of prompting), and it passes through validateEnvNames (^[A-Z_][A-Z0-9_]*$) like all future injections must.
Version pinning is exact: @cloudflare/sandbox is pinned to 0.13.0-next.751.1 in package.json and the Dockerfile FROM docker.io/cloudflare/sandbox: tag must match it exactly — bump both together. Cadence pairing: refresh cadence (600s) = watchdog cron (10 min) < SLEEP_AFTER (20m). The image provisions unprivileged users worker1..worker17 (uid 2001–2017); sudo is absent and root's password is locked, so su demotion works one way only. worker1 is the engine's build user; worker2..worker17 are the thread-user pool (16 users, item 19).
Attach: per-thread worktrees + sticky ref binding: POST /attach {resource, threadKey, refHint?, readonly?, sha?} (operator scope; readonly → item 50, sha → item 51; resource picks the DO exactly like GET /status — the service hosts many residents). Inputs are validated BEFORE anything derives a path or a git argument (P1): threadKey against ^[a-z]{1,32}:[A-Za-z0-9._:-]{1,128}$ (platform-namespaced, conservative charset — no /, no whitespace, no shell metacharacters), refHint against the strict branch pattern (^[A-Za-z0-9][A-Za-z0-9._/-]{0,199}$, no ../@{/.lock). The thread's ref binding is resolved from DO storage keyed by threadKey: absent + refHint → verify the ref resolves in the mirror (fetching under the mutex if needed) and bind; absent + no refHint → 409 {error:"needs-ref: …", needs:"ref", defaultRef:"<the resident's default branch>"} (the defaultRef lets the caller bind by default — item 30 — but the binding is still only ever made by an attach that names a ref); present → the bound ref wins for the thread's whole life (a differing refHint is ignored — the response's ref field is authoritative). A nonexistent ref → 400 {error:"unknown-ref: …"}, nothing created. The worktree lives at /workspace/threads/<threadKey-slug>-<hash8>/<ref-slug> (the hash suffix prevents slug collisions); the response is {workspace, ref, sha, user, reconciled, recreated, deps, credentials, credentialsError?, mutexWaitMs, attachMs}. The answer also carries the attach's step trace (item 63). 16a. Detach: a run gives its pool user back when it ends (POST /detach {resource, threadKey, force?}, operator scope): the pool is sized for SIMULTANEOUS runs, not for every thread ever seen — one-shot review threads would otherwise exhaust it and later reviews fall back cold. The bot calls executor.release(mode) when the run ends — AFTER the answer has been sent to the thread (or the failure card closed), never between "answer ready" and the reply; best-effort (a failure is a log line, never a failed run): "always" (force:true) for read-only agents (toolset: "readonly" — nothing to preserve), "if-clean" otherwise. Without force the resident keeps a worktree that has uncommitted changes (git status --porcelain) or unpushed commits (git rev-list --count HEAD --not --remotes) and answers {released:false, reason:"dirty: …"}; both checks run AS THE THREAD USER via threadRun (never root git in a thread-writable tree — safe.directory would refuse it, and repo-local config is the root-execution vector it guards against); a failed clean check also keeps (never destroy work on a guess). A thread with an exec/read/write in flight (DO-memory counter around those routes) is kept with reason busy, so a release can never remove a worktree under a concurrent command — unless force:true: a hard stop drops the bot's /exec fetch but the command keeps running in the container, so a busy guard alone would hold the pool user until the hourly sweep. Force-detach with an op in flight first kills every process owned by the thread's pool user — su -s /bin/bash <user> -c 'kill -9 -1' as root (reaches exactly that uid: the thread's su … bash -c shell, the command, anything it backgrounded; no procps; the shell kills itself so any exit code is fine), then waits for the thread's in-flight counter to drain, polling every 250 ms up to 6 s (FORCE_DETACH_DRAIN_MS; the kill is bounded at 2 s but is a millisecond syscall, so kill + drain leaves ~2 s of the bot's 10 s client bound for the rm — a worst-case overshoot is tolerated: the bot logs [release] … failed while the DO method completes the eviction anyway). The counter drops only after run() collected the killed process's exit, so a drained counter means the op's processes are dead before the eviction's rm -rf of its cwd. Still busy after the bound → {released:false, reason:"busy after kill: N op(s) still in flight — kept"}. The decision is the pure planForceDetach in src/execution/residentDetach.ts (imported by the Worker): non-force keeps the plain busy guard; force refuses (refusing to kill: "<user>" is not a pool user) unless the binding's user is a THREAD_USERS member — never root, empty, or the build user. The killed op itself completes normally (exit 137 through streamThreadExec to a client that has already hung up) and decrements the counter; one resident log line per kill: detach: force — killed <user>'s processes for <threadKey> (N op(s) were in flight). The bot bounds /detach at 10 s (control-plane POST, not the 5-minute exec ceiling) and only calls it once the answer is out, so a sick resident never delays the reply. Release = the sweep's eviction (evictBinding): tree removed under the mirror mutex when the runtime is up, user returned, binding KEPT marked evicted so the ref stays sticky and the next attach recreates the tree. No binding → 404 no-binding; already evicted → {released:false, reason:"already-evicted"}. Lifecycle state is never touched. 16b. Clean-idle release + image reconcile + idle sleep (follow-up to 16a): (a) the hourly sweep also releases a live binding that has been idle ≥ 1h (CLEAN_IDLE_RELEASE_S), has no op in flight (re-checked right before removal, like detach), and whose tree is provably clean (worktreeCleanliness, run as the thread user; a worktree that no longer exists — disk recycled by a sleep — counts as releasable, nothing to preserve) — the run that used it is over; dirty or unverifiable trees keep to the TTL; a slept container has no tree left (sleep destroys the disk), so an idle binding on an inactive runtime is released outright — only the pool user is at stake. This is what drains bindings left by runs that predate /detach. (b) Pool users live in the image (Dockerfile) while THREAD_USERS lives in the Worker; wrangler deploy swaps the app image but a RUNNING container keeps the old one (row "deploy while warm" — the container survives). reconcileImage probes id -u <last pool user> on every refresh cycle and on attach; if it is missing and nothing is in flight (inFlightCount() = thread exec/read/write + disposable /op users + attaches past their own image check, mid clone/install) it stops the container so the next start uses the current image (disk is a cache); attach answers 503 {error:"image-stale…", state:"restoring"} and the bot falls back cold once. (Without it, an attach after a pool-growing deploy fails attach-failed at thread-dir: install: invalid user '<new pool user>'.) (c) Idle sleep: on a refresh cycle that finds the resident SETTLED (warm at entry, or degraded with the SAME refresh-produced reason (github-unreachable, <step>-failed) for DEGRADED_PARK_AFTER_CYCLES = 3 consecutive cycles — a chronically broken default branch must not bill the container 24/7 for retries that cannot succeed; any other state resets the streak — a refreshing/restoring marker at entry is an orphan from an interrupted cycle and runs the full cycle instead, and a degraded stamped by the WATCHDOG (alarm-missed, stale-mid-flight) never counts toward the streak and never settles: the watchdog pulled the cycle to +5 s so a refresh RUNS — counting those was self-fulfilling and parked residents degraded for hours at a time; accepted cost: a resident oscillating between a refresh-produced failure and watchdog stamps never parks and bills full cadence — a watchdog stamp makes the prior "same reason" observation untrustworthy, and preserving the streak across it would re-open the hole), if no live binding attached within IDLE_AFTER_S (1h), no op is in flight, and (runtime up) no live tree is dirty (the per-tree clean checks run concurrently — disjoint trees — and each is ONE spawn: worktreeCleanlinessScript/parseWorktreeCleanliness in src/execution/residentCleanliness.ts fold the presence test and both git probes into a single tagged-output script, the git probes still privilege-dropped to the thread user), the fetch is skipped, facts.idleSince is set and the alarm re-arms at IDLE_REFRESH_INTERVAL_S (6h) instead of 10 min — so SLEEP_AFTER (20m) can elapse and the container sleeps. A dirty live tree pins the container awake (sleep destroys the disk; worktrees are not snapshotted). Wake path is BOUNDED: attach → refreshIfStale (when idleSince is set or the last refresh is older than the active cadence) fetches the mirror under the lock (seconds) so the bound ref is current, clears idleSince, and arms the full refresh cycle at +1s in the background — an attach never waits on a checkout rebuild/install, so a wake cannot become a cold-fallback generator. isIdle() uses the same inFlightCount() predicate. The watchdog is unchanged: a pending refresh schedule always exists (just farther out). Engine view carries idleSince; the dash shows "idle since …" / "awake". Cost effect: an unused resident bills ~20 min per 6 h instead of 24/7. 16c. Race fixes on the wake, idle and sweep paths: refreshIfStale bounds its wake-fetch mutex wait by ATTACH_MUTEX_WAIT_MS (a full checkout rebuild holding the lock can no longer stall a wake attach for minutes — on expiry the attach proceeds on the last mirror and the +1 s background cycle repays it) and RE-READS facts before clearing idleSince (never writes a stale snapshot over a concurrent cycle's sha/lastRefreshAt; the idle branch of the refresh cycle re-reads likewise). inFlightCount() now also counts a refresh cycle past its idle/reconcile gates (refreshesInFlight), so an attach-path reconcileImage cannot stop the container under an in-flight rebuild. The sweep's clean-idle release and detachThread both re-read the binding right before evictBinding and skip when it is gone, evicted, or its lastAttachAt changed (a re-attach completed inside the clean-check await) — a fresh tree is never removed from a stale snapshot; evictBinding itself re-reads once more AFTER its awaited rm (mirror lock) and gives way if the binding moved meanwhile, so user:"" is never written over a re-attach that started inside the eviction (the sweep re-reads active per binding for the same reason — a container that woke mid-sweep must get the rm). The refresh cycle's final facts put also drops idleSince (it spreads facts read at alarm entry; without this a wake cycle resurrected the stale idle marker, showing a stale "idle since" and forcing the wake-fetch path on every attach).
Worktree mechanism + dirty/stale discipline: per-thread trees are local git clone --no-hardlinks --branch <ref> clones of the mirror, NOT git worktree checkouts and NOT hardlinked object stores — a git worktree would keep the thread's index/objects inside the root-owned mirror (commits would need write access there), and hardlinked objects chowned to the thread user would let that user chmod inodes shared with every other tree. The clone is chowned to the thread's user; origin is repointed at https://github.com/<slug>.git so fetch/push use the per-attach credential file (the mirror itself is deliberately unreadable to thread users). On every attach an existing tree is examined AS THE THREAD USER: dirty tracked files (git status --porcelain -uno — untracked scratch files are the thread's own state and survive) or a stale HEAD (neither the mirror's current ref tip nor a local descendant of it — thread commits on top of the tip are kept) → the tree is wiped and recreated (recreated:true); otherwise it is reused byte-for-byte (recreated:false). A slept container loses /workspace/threads entirely (disk is cache); the binding survives in DO storage and the next attach recreates.
Dep/build materialization: after (re)creating a tree, the committed-lockfile key at the thread's sha (git ls-tree from the mirror — same pure-function key as the refresh engine) is compared to the warm checkout's recorded key. Equal → no install runs (reconciled:false) and each cached dir is materialized from /workspace/checkout by the mechanism the pure depCacheMaterialization (src/execution/residentDepCache.ts, imported by the Worker) names for it: node_modules is hardlink-copied (cp -al; deps:"hardlink") with only DIRECTORIES chowned to the thread user — file inodes stay worker1-owned and stripped of group/world write, so a thread can delete/replace entries in its own tree but can never mutate the inodes shared with the warm checkout or another thread; cp -al failure (e.g. cross-device) falls back to a plain copy (deps:"copy", fresh inodes, fully chowned). Build output dirs (dist build out .next) are ALWAYS plain-copied (fresh, fully thread-owned inodes): the review agent is instructed to run the project's build and /op build runs it too, and compilers rewrite dist/** in place (open+truncate through the existing inode) — with a hardlinked read-only dist/ the build died with EACCES and a review run reported the worktree as read-only. A copy shares no inode with the warm checkout, so the isolation invariant holds without the write-protection. The same class exists one level down, INSIDE the hardlinked node_modules: tool-managed paths that builds and test runs rewrite in place — every top-level dot entry (.cache babel-loader/eslint/prettier/webpack; .vite + .vitest vite's optimizer and vitest's results; .prisma the client prisma generate regenerates; .bin shims; .package-lock.json npm's hidden lockfile) plus any .cache directory at any depth (a package's own cache, node_modules/<loader>/.cache) — except .pnpm, pnpm's virtual store: a top-level dot entry that IS the package content, immutable after install, so it stays hardlinked (a .cache nested inside it is still copied). copying it made every thread tree of a pnpm workspace a full 2.6 GB copy of its deps and a fresh attach ~190 s, and one thread filled an 8 GB disk — so after the cp -al the Worker enumerates them with the exact find from mutableCacheFindArgv (a -path "<root>/.*" glob, never -maxdepth, which GNU find applies globally even inside parentheses), mutableCachePaths dedupes descendants under a listed ancestor, and each is swapped for a real copy (rm -rf + cp -R + chown -R; steps deps-mutable-list/rm/copy/chown). The whole materialization runs as TWO container forks, not a spawn per probe/walk/path: depCacheScript (src/execution/residentDepCache.ts) handles all five dirs in one sh -c — per-dir src/dst gating, the cp -al with its directory chown AND file-write hardening as ONE combined find walk (( -type d -exec chown … + ) -o ( -type f ( -perm -g+w -o -perm -o+w ) -exec chmod go-w … + ); step deps-perms), the plain-copy fallback, the copy dirs, and the mutable-cache listing — emitting one tagged dir:<name>=<mechanism> line per materialized dir plus the raw mutable= listing, which the DO parses (parseDepCacheScriptOutput) and feeds through the unchanged mutableCachePaths into mutableCacheSwapScript (the second fork, all swaps). The resulting ownership/permission state is identical to the per-spawn version; failures still surface as named steps via err= tags. Packages themselves stay hardlinked and tamper-proof; nested dot dirs other than .cache (a package's vendored .bin/.github) are package content and stay shared. Other locations a tool might write (.tsbuildinfo, .turbo, coverage, .eslintcache, packages/*/node_modules, packages/*/dist) are not materialized at all — a thread builds them from scratch into its own tree — so they cannot hit this. The reported deps is decided by node_modules (the dependency cache the field describes; build-dir copies only fill in when the warm checkout has no node_modules — foldDepsMechanism). Differing key → the tree is SEEDED from the warm checkout exactly like the equal case (the same hardlinked node_modules, mutable paths swapped) and THEN the repo's install command runs inside the worktree, token-free, as the thread user, reconciling only the packages whose version differs (deps:"reconcile", reconciled:true; the decision is the pure planThreadDeps, item 58) — never an install from an empty tree; a differing key with no install command in the table seeds nothing (deps:"none"): a seed nothing can reconcile would be the wrong deps presented as ready.
OS-layer thread isolation: each attach allocates a user from the worker2..worker17 pool (16 users — sized for simultaneous runs, since a run gives its user back via /detach when it ends; memory, not the pool, is the real concurrency ceiling) in a storage-only critical section (atomic under the DO input gate — two concurrent attaches cannot claim the same user); an exhausted pool → 429 {error:"user-pool-exhausted: …"}. Every /exec /read /write runs privilege-dropped via su -s /bin/bash <user> -c … with the worktree as cwd. Thread dirs are mode 700 owned by their user (peer threads cannot traverse in); the mirror top dir is root:worker1 750 (worker1 still fetches from it for refresh; thread users are denied at traversal); /workspace/.resident/ stays 700 root. The allocation is persisted in the binding; re-attach reuses it; eviction releases it.
Per-attach credential files: when the GitHub App is configured, attach mints a repo-scoped token and lands it in <worktree>/.git/github-credentials (0600, owned by the thread user) via the SDK file API into a 700 per-user staging dir followed by a privilege-dropped cat — the token never appears in argv or process-wide env, so another thread's ps sees file paths at most. The worktree's git credential.helper points at the file (store --file=…). A refresh whose reason is empty (git's store erased a token GitHub 401'd) mints FRESH — mintRepoScopedToken(env, slug, { fresh: true }) drops the per-slug cache first — so a repudiated token is never re-served from the cache back into the file. Mint failure (or unconfigured App) is command-level: the attach still succeeds with credentials:"unavailable" + credentialsError; nothing lifecycle-flips. The file is refreshed during the run, not only at attach: an installation token lives one hour, a coding run can push later than that (45-min budget plus the attach-to-first-push gap, or a follow-up run re-using an earlier attach), and git's store helper ERASES a credential the remote rejects with 401. The refresh is driven by the token's OWN expiry, not the file's age: attach records the minted token's tokenExpiresAtMs on the binding beside credentialsWrittenAt, and before every writable /exec the Worker sizes the file (one stat) and asks the pure shouldRefreshThreadCredentials (src/execution/residentCredentials.ts, imported like residentDetach): refresh when the file is missing, empty (git erased it), or the token is within CREDENTIAL_EXPIRY_MARGIN_MS (~25 min — BASH_TIMEOUT_MAX_MS plus slack, the same margin the thread sandbox keeps) of its expiry (expiring); never for a read-only binding (item 50). Invariant: a writable exec never starts on a token with less than CREDENTIAL_EXPIRY_MARGIN_MS of life left. This closes the bug where a token minted for an earlier thread on the same repo — cached, with as little as five minutes left — was written for a new attach and read "fresh" by file age until CREDENTIAL_REFRESH_AFTER_MS (45 min) later, so every writable exec in between ran on a dead token and the first git write 401'd. A binding that predates tokenExpiresAtMs (null) falls back to the file-age rule (written more than CREDENTIAL_REFRESH_AFTER_MS ago, or an unknown write time → stale) so an old binding still refreshes. A refresh is mintRepoScopedToken (cached per slug, only serving a token with more than the same margin left) → the same staged 0600 write → credentialsWrittenAt AND tokenExpiresAtMs updated on the binding. A mint failure at exec is command-level: one log line (credentials: refresh failed (<reason>) — command runs without a fresh token), the command still runs, lifecycle never flips. /read and /write never refresh — only exec reaches the network.
Thread exec/read/write: POST /exec {resource, threadKey, command, timeoutMs?} mirrors the thread-sandbox Worker's convention — the command body is bounded at MAX_EXEC_COMMAND_LENGTH (64 000 chars; a sanity guard far above any legitimate one-liner or heredoc, raised from 8 000 which refused real agent commands the sandbox/local executors accept — real file content belongs in /write at 512 KB; the bot-side exec wrapper's ( cd <worktree> && … framing counts against the bound) → over it, a pre-validation plain 400 naming the limit (agent-fixable: the client raises a normal Error, never ExecInfraError) — then immediate headers, a whitespace heartbeat every 15s, then one JSON document over HTTP 200: {stdout, stderr, exitCode, truncated} (5-minute default; the body's timeoutMs raises it per call up to the shared 20-min ceiling, clamped server-side by clampBashTimeout — execution.md item 11; timeout → exit 124 with a stderr note naming the limit and the timeoutMs knob; output capped at 100k chars/stream with a truncation note; post-validation failures arrive in-body as {error, needs?, stdout:"", stderr:error, exitCode:127}). POST /read {resource, threadKey, path} → {content, truncated}; POST /write {resource, threadKey, path, content} → {ok:true, bytes}. Paths are confined to the thread's worktree (conservative charset, no .. segments, resolved-prefix check → 400 path-escape), and the file ops themselves run AS THE THREAD USER — the OS layer, not just the prefix check, is what bounds a planted symlink; written content travels through the per-user staging dir, never argv. An unattached/evicted/disk-recycled thread gets a named not-attached/evicted/worktree-missing error with needs:"attach" (409 on read/write; in-body on exec). Exec and attach both bump lastAttachAt. Hot-path cost: only /attach consults the singleton registry (its 404 names not-onboarded, and the record rides on to the DO so attach never re-reads it) — /exec /read /write /detach skip the registry hop because they fail closed anyway: no binding → not-attached/no-binding, and a never-onboarded resource's DO has no state → 503 not-serviceable (that wording, not a 404, is the deliberate trade for removing a fleet-wide serialization point from every tool call). Output is bounded at the source: /exec, /op runs, and /read wrap the command via capWrappedCommand (src/execution/residentExecWrap.ts) — the full streams go to container-disk temp files (disk is a cache) and only the first capBytesFor(cap) = 4×cap+4 bytes of each cross the RPC, so a verbose test/build run can no longer materialize tens of MB inside the 128 MB DO isolate before the char-cap slice (the OOM → runtime-replaced-mid-run path). Exit code, stream separation, and the DO-side truncated logic are byte-for-byte preserved (4 bytes ≥ any UTF-8 char, so the char-count flag stays exact); the engine's own probes (cleanliness, credentials, markers) stay unwrapped. A timeout keeps its debugging clue: the kill skips the wrapper's own head/cleanup lines, so /exec and /op write to FIXED unguessable file paths (execCapFiles) and, on timedOut, run one follow-up command (recoverCapturedOutput) that salvages the same capped heads the wrapper would have emitted and removes the files — a hung test run answers exit 124 WITH what it printed before dying, as it did before source-capping. Recovery is best-effort (its failure returns the bare timeout result) and a still-writing orphan sees its file unlinked, reclaimed when it dies. Named edges: mktemp failing (disk full) exits 125 before the command runs; a recovery that itself fails leaves the two files until the container recycles. The per-request preflight's hydration probe is memoized per container incarnation (60 s TTL; cleared at the run() choke point whenever a RuntimeReplacedError is minted, on every lifecycle transition, and at every deliberate stop — a sleep cannot race the TTL because SLEEP_AFTER is 20 min of idleness while the memo lives 60 s past activity; the refresh alarm's 10-min cadence always re-probes for real). The same incarnation memos cover the one-fork git-setup script (was three forks per attach/op, /etc/gitconfig rewritten each time) and the per-user staging dir (write/credential paths fold their chown+chmod into one fork).
Mirror mutex — a durable lease judged against the incarnation, and idempotent engine steps (pure src/execution/residentIncarnation.ts and src/execution/residentStepPlan.ts, applied by the Worker's withMirrorLock and its step methods fetchMirror / installDeps / runBuild / snapshot / restoreCheckout): a DO yields at every await, so concurrent requests interleave mid-handler — every mirror mutation (provisioning clone, refresh fetch, attach's verify-fetch + worktree create/remove, sweep removals, the checkout rebuild and the snapshot) runs under one mutex with two halves. In process, a promise chain queues takers in arrival order — the fast path within one incarnation. Durably, the row resident:mirrorMutex {holder, incarnation, expiresAt, step} is the truth across incarnations: the resident mints an incarnation id when its isolate starts and again whenever its container runtime is replaced under it — and only then: a lifecycle transition clears the object's memos and never the incarnation, so the cycle that flips the state to refreshing still holds its own lease afterwards — and a taker past the chain reads the row and takes it when free, when the holder's incarnation is not the current one (an isolate swap dropped the chain while the holder's process may still be writing), or when expiresAt — the step's own budget — has passed (the backstop for a holder that hung without a swap); only a live holder of the current incarnation is waited for, and a release deletes only the holder's own row. Attach waits under a named 60s timeout (ATTACH_MUTEX_WAIT_MS) → 503 {error, state, reason:"mirror-busy"} on expiry; a timed-out waiter releases its queue slot so later waiters are never stuck behind a ghost. The same lease shape carries the two in-flight facts the watchdog's stale-mid-flight branch reads — the refresh cycle and the hydration, in resident:inFlight, each alive only for the current incarnation inside its budget, and the branch clears a dead lease only against the holder it read, so a cycle that recorded a fresh lease meanwhile keeps it — and the per-key dependency install (resident:depsLease:<key>: the install's exclusive resource is the key's store entry, never the mirror, item 59; a live holder of the current incarnation is joined, a dead holder's scratch tree and staging dir are swept of build-user processes and removed before the next attempt starts, item 56). Every engine step is a public method that reads the facts it is about to change (a stored record, the disk markers, the store), asks the pure plan whether the work is done — done issues no command, so a second call with the same inputs has no effect — takes its lease, runs its commands under the step's own budget, writes its result and releases; snapshot commits with compare-and-swap on the record it read at the start and answers superseded (its fresh objects dropped) when the stamp moved meanwhile, never a throw; the recorded facts move to the stamp in the same write as the record. The alarm chain drives the steps in the order it always did; /debug info shows incarnation, mirrorMutex and leases.
Inactivity eviction keeps the binding: a self-rescheduling hourly onWorktreeSweep schedule (TTL eviction plus the clean-idle release of item 16b) (armed by attach whenever none pends; dies when no live binding remains) removes worktrees whose lastAttachAt is older than the TTL (default 7 days; per-resident worktreeTtlDays 1–365 settable at onboard/reconfigure), releases the user to the pool, and KEEPS the binding record marked evicted with its ref — the next attach recreates the tree on the same ref with a fresh pool user. A slept container is never woken just to delete files the sleep already destroyed (the sweep marks bindings evicted without exec when the runtime is down). Debug ops threads / backdate-thread / sweep-now exist to enumerate bindings and exercise the TTL path without waiting days.
Bot-side selection is warm-gated and probe-per-dispatch: makeExecutor(opts, {threadKey, agent, repo?, ref?}) returns {executor, note?}. Resident selection requires a repo-declaring agent, a configured execution.resident {baseUrl, tokenEnv?, probeTimeoutMs?} (operator bearer read from tokenEnv, default RESIDENT_OPERATOR_TOKEN), AND a resolved ctx.repo — populated by the repo resolver (item 29); undefined means the per-thread path with NO probe (total input contract). Membership is never cached: each dispatch makes one operator GET /status probe. A serviceable state → ResidentExecutor (isServiceable in src/execution/residentState.ts, the lifecycle union shared with the resident Worker so a renamed state is a compile error on both sides): warm with a POSITIVE note = resident · <owner/name> · <ref>@<sha7> taken from the attach answer (the resident path is named on the card, never inferable only from the absence of a fallback note); every OTHER note — a failed attach, an unreachable or unserviceable resident, a repo not onboarded — is also published to the run's stream as a cold_sandbox run note with the same text, after the attach and before the first turn, head material like the rest of the setup (tracing.md item 16), so the run page explains a sandbox run whose attach span carries the resident's grafted steps — the card is not the only witness; refreshing (the resident keeps serving the last snapshot — item 7 — and /attach refuses nothing; the mirror lock serializes the attach against a refresh's fetch/rebuild, bounded by ATTACH_MUTEX_WAIT_MS → a mirror-busy 503 falls back like any attach failure; so does a disk-pressure 503 — item 55 — whose error carries the budget math onto the card); and degraded only for reason classes that prove the checkout intact — github-unreachable: … (fetch failed before the checkout was touched) and alarm-missed: … (a dead chain re-armed; nothing ran); not disk-full: … (item 54 — the checkout may be intact, but no attach can succeed on a full disk, and recording that failure as github-unreachable sent every run to attach and die at git-setup); not stale-mid-flight: …, whose orphaned cycle may have died inside the rebuild (the +5s recovery cycle rebuilds within seconds anyway) — because a failure inside the rebuild lock section (checkout-update-failed / install-failed / build-failed / snapshot-failed / refresh-failed) leaves the checkout at the new sha with absent or partial deps after git clean -fdx, and a fresh thread whose lockfile key matches would hardlink that broken cache; those, and unknown reasons, stay cold until the next cycle rebuilds. A non-warm attach carries an informational note = resident <state> (<reason>) · <ref>@<sha7> — attached to the last snapshot on the card. Gating on warm alone (the original rule) sends every run cold for the whole of every refresh window — with an active default branch (each merge a rebuild on the 10-min cycle) plus each resident deploy's restore, that is most of a working day of resident refreshing — using fresh sandbox. 404/not-onboarded → the ordinary per-thread path (AE4 — no resident exists, nothing degraded) but with a visible note = repo not onboarded as a resident — running in a cold per-thread sandbox; onboard it (`repo onboard <owner/name>`) for a warm, deps-ready environment so the cold fall-through is never a silent surprise; the engine-owned states onboarding (nothing to attach), restoring (disk being rehydrated), down (only a rebuild escapes) → per-thread fallback with note = resident <state> (<reason>) — using fresh sandbox. Every fallback note rides on the dispatcher's status-frame title (a degraded or cold-fallback resident is loud, never a silent stall).
Probe circuit breaker: probe TRANSPORT failures only (fetch error / probeTimeoutMs timeout, default 2000ms) fall back as resident unreachable (…) and arm a 30s in-process negative cache, so a resident-service outage costs one timeout, not one per concurrent dispatch. Definite lifecycle answers — including degraded/down — are NEVER cached (the next dispatch must see a recovery immediately). In-process only; the restart-survival invariant is untouched.
Per-repo gate: restrict.repos (with the invoking user's repos grant) is checked by the dispatcher BEFORE executor selection. An unlisted repo = open to every allowed coding-agent user; a listed repo refuses ungranted users with a named 🚫 reply naming the repo (admins always pass) — a refusal is a reply the user sees, never a silent per-thread fallback, and no executor is created.
ResidentExecutor client (attach-on-open): open() attaches before the first tool call so needs-ref / not-onboarded / mirror-busy surface legibly at selection time — the 409 needs:"ref" answer throws a TYPED ResidentNeedsRefError the dispatcher catches for the ask-once flow (item 30). Every route POSTs {resource, threadKey, ...} with the operator bearer and no x-env headers; /exec responses are read as full text (heartbeat whitespace + one JSON document) and parsed from the body, never the HTTP status; a non-zero exitCode is a result (exit N: …), not an error; an in-body {error} without needs is thrown verbatim and never blind-retried. A needs:"attach" error on any op (evicted / disk-recycled worktree — the binding survives) triggers exactly one re-attach + retry of the same call; a second needs:"attach" is a legible error. A reason:"runtime-replaced" answer is handled per item 43 (re-attach once; retry only the idempotent routes).
Plumbing: RESIDENT_OPERATOR_TOKEN reaches the bot container through the Worker shim's explicit envVars allowlist (deploy/cloudflare/worker.ts + deploy/secrets.manifest.json, the same path as SANDBOX_TOKEN); the production config (config/config.production.yaml) points execution.resident at the resident Worker's URL. src/cli.ts accepts --thread <key> / --thread=<key> or SWITCHBOARD_THREAD (default stays ephemeral cli:<timestamp>) so repeated CLI invocations act as ONE thread — the vehicle for re-attach/binding-persistence verification.
Repo/ref resolved BEFORE the model turn: src/core/repoContext.ts is the production default behind the dispatcher's CoreDeps.resolveRepoContext seam (still injectable for tests). Extraction sources in priority order: explicit signals in the CURRENT message — an owner/name slug token, a github.com repo URL (Slack markup <url|label> unwrapped; /tree/<ref> yields the ref too), a PR URL or owner/name#N shorthand — then the thread's previously-established repo derived from history like lastThreadDirectives (user turns, last wins; restart-safe, never stored), then none → {} → the per-thread path with no probe (AE4). Addressed repos (without this, a thread bound to one repo by an issue link keeps every later in acme/api run on that repo's resident — a bare slug is weak and could not rebind — while an opening in api carries no signal at all and goes cold): the first token after the word in (outside code) names the request's TARGET — in owner/name anywhere in the message, or a bare in name in the DIRECTIVE position only (nothing but agent:/model: directives or a mention before it — agent:coding in api, …; "the crash is in api, see the logs" is prose even when a repo is called api). Vetted against the resident registry it is STRONG like a URL: it binds a fresh thread and rebinds a bound one, and "last strong wins" holds across URLs and addresses alike (the resolver walks the thread's user turns backwards, vetting each address). A bare NAME resolves only through the registry listing (GET /residents, the admin bearer — the route is read-scoped) and only when exactly one onboarded repo carries it; an unknown (in production) or ambiguous name is prose and binds nothing, a failed listing is no answer. The strength and code-span guards below stand: an address the probe refuses (in try/catch, in docs/reference/specs/memory.md) changes nothing, a merely-mentioned onboarded slug ("also check acme/web") stays weak, and without a probe (local/dev) nothing can be vetted so an addressed slug stays the weak token it always was. One probe per candidate per resolution. Silence is not a refusal: the probe answers "unreachable" (transport failure, timeout, the outage window) apart from false (not onboarded), and when the current message addresses a slug the registry could not be asked about, the resolver returns unverifiedRepo — never a silent fall back to the thread's old repo — and the dispatcher replies "couldn't verify … the resident registry didn't answer" without starting a run (an unanswered address in history is simply no answer; a fresh thread whose only candidate went unanswered reports unverifiedRepo rather than "not onboarded"). The run card names the bound repo (resident · <owner/name> · <ref>@<sha7>) so a wrong binding is readable from Slack, not only from a sha. A PR yields BOTH repo and head ref via ONE GET /repos/{owner}/{repo}/pulls/{n} REST call authenticated with resolveGithubToken() — NEVER a gh shell-out from the dispatcher (AGENTS.md invariant 5); unauthenticated works for public repos, any fetch failure degrades to repo-only, and cross-fork head refs are never bound (they don't resolve in the mirror). The fetch runs whenever the current message names a PR of the resolved repo — regardless of ref phrasing beside it (agent-review.md §11): the PR is the explicit target, its head branch is the ref for that message, and a prose on main / on branch X in the same message is only the fallback ref when the fetch fails. (Before, a bound ref skipped the fetch — a re-review saying "rebuilt on main" lost its head SHA and the review was refused.) Ref phrasing is conservative (explicit-or-ask-once, never a silent guess): on branch X / branch:X, bare on X only for well-known default branches (main/master/develop/trunk) or slash-shaped refs; every candidate is validated against the resident's strict ref pattern (no ../@{/.lock); slugs are validated owner/name shapes, lowercased (resident resource ids are lowercase) — invalid tokens are ignored whole, never partial garbage. Code never establishes a repo: a token inside an inline code span (`unset/unset`, `src/core`) or a fenced code block (pasted logs/diffs) is code or a path being talked about, not a repo switch, so it is excluded from the bare-slug branch ONLY — URL/PR forms and refs are unaffected, so a backticked github.com link still counts and on `main` still binds the ref (src/core/repoContext.test.ts::a token inside a code span never establishes a repo…, ::a slug inside a fenced … block…, ::a backticked ref still binds…, ::repoFromThread ignores code-spanned tokens in history too). An on <owner/name-shaped> token is ambiguous (slug or slashy branch): it binds as a ref when a repo is independently established (current message or thread), else it is the repo mention. Signals have two strengths, and the thread binding is sticky against weak ones: a github.com URL (repo/PR//tree) or owner/name#N is STRONG — unambiguously a repository; a bare owner/name-shaped token is WEAK — it is also the shape of every relative file path and of ordinary prose, and code spans only catch the backticked ones. A thread bound by a strong signal is rebound ONLY by another strong signal (in the current message or a later user turn); weak tokens are consulted only while nothing strong has bound the thread. Otherwise a prose token shaped like a path (e.g. docs/reference/specs/memory.md) would select a cold sandbox for a non-existent repo AND unbind the thread's PR. (src/core/repoContext.test.ts::a bare slug-shaped token never rebinds a thread bound by URL…, ::a bare slug in an EARLIER follow-up…, ::redirecting the thread to another repo BY URL…, ::repoFromThread: the last STRONG user-turn signal wins….) Bare tokens are guarded two more ways. (1) A bare token never overrides a repo the thread already established, whatever its strength — the first binding stays until a STRONG signal replaces it, so a thread opened with a bare in owner/name is not hijacked by later prose (reflection/review-post, try/catch, comment/spec are all slug-shaped). (2) In a thread with no repo yet, a bare token binds only when the resolver's injectable resident probe (resolveRepoContext(msg, history, isResident?) / repoFromThread(history, isResident?)) confirms it names an onboarded resource; the production probe (residentOnboardedProbe in src/execution/factory.ts) is one operator GET /status through the same negative cache as executor selection — any lifecycle state of an onboarded resource is a yes, not-onboarded and every unreachable/error answer is a no (fail-closed). No resident configured, or its bearer unset → no probe → bare tokens bind unvetted as before (local/dev). A refusal is reported when it is the whole story: when nothing binds and the first bare candidate the probe refused exists, the resolver answers { rejectedRepo } and the dispatcher — for a repo-needing agent only — closes the ack card not started (repo not onboarded) and replies 📦 with the slug, repo onboard <slug>, and the github.com URL form as the way to run cold anyway; no attach, no model turn (before: an empty workspace and fatal: not a git repository). A thread that already has a repo never carries rejectedRepo (its prose slugs are never probed), so the silence there is unchanged. (src/core/repoContext.test.ts::bare prose slugs never hijack a thread …::*, ::a STRONG repo signal in the current message beats the thread's; a bare slug does not, ::repoFromThread: a bare slug binds a thread that has no strong signal — first bind wins….)
Needs-ref binds to the repo default loudly; asks only when it can't: when attach answers 409 needs:"ref" (typed ResidentNeedsRefError) the body names the resident's defaultRef (item 16), and the factory re-attaches ONCE with that ref — the message named no branch, and the cold path already works on the default branch without asking, so the warm path no longer costs an extra human turn on every fresh thread. It is never silent: the note becomes resident · <owner/name> · <ref>@<sha7> (repo default — no branch named), and the binding is made by that explicit second attach, not guessed resident-side. A 409 that carries no defaultRef (a Worker predating the field) keeps the ask-once flow: the dispatcher replies ONE clarifying question naming the repo — 🌿 "Which branch of <repo> …" — before any run card exists and without any provider call (mirrors the named-refusal reply shape); the user's answer in the thread ("on main") is an ordinary next message: the resolver reads the ref from it and the repo from thread history, and re-attach binds it. A second needs-ref after binding by default is a resident bug and surfaces as a plain named error, never a loop. Either way the binding persists resident-side thereafter — a later differing hint is ignored, the binding wins; to work on another branch, start a new thread naming it.
Resident prompt variant, selected after executor resolution: the dispatcher composes the effective system prompt AFTER makeExecutor returns — executor is a ResidentExecutor → the agent's residentSystem (exported as CODING_SYSTEM_RESIDENT / REVIEW_SYSTEM_RESIDENT in src/agents/registry.ts) plus a line naming the resolved repo; any other executor → the agent's own prompt untouched. The variant flows through RunOptions.system; the shared AgentDef is never mutated (concurrent dispatches share it). Variant content: the workspace is a READY git worktree on the thread's bound branch, deps installed, build warm — no cloning, no installs, no repo discovery, no scope-first survey; gh is NOT in the resident image (git + node only), so coding pushes with git using the token in the worktree's .git/github-credentials, calls diff_digest to shape the description, and submits it through submit_pr_description — OpenSwitchboard renders the body at the observed pushed head and opens (or edits) the PR from the bot process, so the prompt carries no PR-creation API call at all (pr-description.md item 5). It is told honestly that credentials may not be provisioned yet — an auth-refused push is said plainly, never retried blind — and a run that pushed without producing a PR gets the branch compare URL reported honestly by the dispatcher, never a claimed PR.
Repo-management commands, fail-closed: repo onboard <owner/name> [--ref <branch>] [--test "<cmd>"] [--build "<cmd>"] [--install "<cmd>"] [--evict-coldest] / repo reconfigure <owner/name> [--ref …] [--test …] … / repo offboard <owner/name> [--dry-run] / repo rebuild <owner/name> [--dry-run] are registry commands (repo.onboard|reconfigure|offboard|rebuild in src/core/commands/repo.ts, command-registry.md item 20; the admin client and the slug/ref validators live in src/core/residentAdmin.ts): answered inline through the registry's chat adapter, never a model turn, reachable as /api/repo.*, MCP repo_* (repo:write) and the CLI, all gated in chat by repoManager = canManageRepos — fail-closed in src/config.ts: repo:write is never a baseline — ADMINS ONLY until granted — because these commands provision billable always-on compute and bind GitHub credentials. A refusal is a 🚫 reply naming the admins. repo list is the command registry's repo.list (src/core/commands/repo.ts; command-registry.md item 17): repo:read scope, chat gate open, the JSON is the resident Worker's /residents body and the text is the historical repo list reply (renderResidentList) — so the same read-only live registry view is now also GET /api/repo.list, MCP repo_list, and CLI repo list; parseRepoCommand still recognizes the verb and handleRepoCommand yields it (null) to the registry stage, and the resident admin client is resolved per call by residentAdminFromConfig (shared with the mutating verbs). Only the known verbs match — prose like "repo onboarding is done how?" flows to the model untouched. Parsing is key="value" tokens (curly quotes from Slack autoformat normalized), slugs are validated owner/name shapes lowercased, refs pass the resident's strict pattern, and omitted onboard commands are DETECTED from the repo root (item 52; the npm table npm install --no-audit --no-fund / npm run build --if-present / npm test — the table the first residents were onboarded with — is only the fallback when the root cannot be inspected, and the reply says so). The commands reach the resident Worker's admin routes via a fetch client using RESIDENT_ADMIN_TOKEN (env name settable as execution.resident.adminTokenEnv); the bearer reaches the bot container through the Worker shim's envVars allowlist (deploy/cloudflare/worker.ts + deploy/secrets.manifest.json), same path as RESIDENT_OPERATOR_TOKEN. Because the resident's /reconfigure replaces the whole command table, a partial chat repo reconfigure merges its patch onto the current table (fetched live from /residents) before posting.
--dry-run is the stateful-op checkpoint: POST /offboard {resource, dryRun:true} answers 200 with {resource, dryRun:true, wouldRemove:{registryRecord, schedules, snapshotBackupIds, backupObjects, r2Objects, threadBindings, container}} — the same itemization the real teardown reports, computed READ-ONLY (R2 objects counted, never deleted; schedules/state untouched). POST /rebuild {resource, dryRun:true} answers 200 with the rebuild plan (item 34) and executes nothing. Both chat commands render the plan and say "Nothing was changed", naming the non-dry command to run.
Rebuild is the down→onboarding escape hatch: POST /rebuild {resource} (admin) discards the recorded snapshots — R2 objects deleted FIRST, before initResident wipes the stored handles (same ordering as teardown) — and reprovisions from scratch through the ordinary alarm-driven pipeline, reusing the registry record's command table/ref/budget. Answers 202 with {…plan, backupObjectsDeleted, state:"onboarding"}. The registry record (cap slot) and thread bindings are KEPT (worktrees on stale shas are recreated by the next attach); refused with 409 while the engine is mid-flight (onboarding/refreshing/restoring) so two engine chains never race one disk. A rebuild whose provisioning then fails follows the ordinary onboard failure paths (named down reason; timeout releases the slot).
Onboard checks installation membership, honestly: when the GitHub App is configured, /onboard requires the repo to already be in the installation's repository list — proven by a repo-scoped mint (the token API refuses repos outside the installation), so the check IS the scoping mechanism it protects; failure → 403 {error:"not-in-installation: …"} BEFORE any registry insert (no cap slot consumed, nothing provisioned). When the App is NOT configured the check is SKIPPED — never faked — and the 202 carries warning:"github-app-not-configured: installation membership was NOT verified …", which the chat reply surfaces.
Watchdog auto-rebuild: a resident down with a rehydration-flavored reason (r2-restore-failed / snapshot-stamp-mismatch / no-snapshot — the states only a rebuild can escape, since down chains never retry hydration) accumulates one strike per watchdog pass (resident:rebuildStrikes in DO storage); at 3 strikes (~30 min at the 10-minute cron) the watchdog triggers the same rebuild as item 34 and reports action:"auto-rebuilt". Provision-failure downs never auto-rebuild (they would loop against the same broken build); any serving-state pass clears the counter, as do manual rebuilds and offboard.
POST /op is the deterministic modelless path (operator scope): {resource, op, ref?} where op resolves ONLY through the fixed enum {test, build, status} into the onboard-time command table — request text is never interpolated into a shell command (the command STRING is admin-written; the ref rides as a git argv element). ref passes the strict branch pattern at the Worker (400 on failure) and must additionally resolve in the mirror (fetching once under the mutex if unknown; still unknown → in-body unknown-ref); absent ref = the resident's default branch. Not-onboarded → 404 (like /status). test/build stream with /exec's convention (immediate headers, 15s whitespace heartbeats, ONE JSON document over HTTP 200; post-validation errors in-body) and answer {ok, op, resource, ref, sha, summary, stdout, stderr, exitCode, truncated, deps, reconciled, durationMs} — ok is the command's verdict: a failing run is a RESULT (ok:false with a named summary), never an error path. status touches no checkout and no exec at all — DO storage reads only — answering lifecycle + sha + lastRefreshAt (ok = not down; degraded still serves).
Ops run in disposable per-op checkouts, never a thread's worktree: each test/build op transiently allocates a pool user (in-DO set, excluded from AND excluding thread allocations — an op never shares an OS user with a thread), clones --no-hardlinks from the mirror under the mirror mutex into a 700 user-owned /workspace/ops/<uuid>/checkout, materializes deps through the EXACT thread mechanism (materializeThreadDeps: same committed-lockfile key → cp -al from the warm checkout; differing key → scoped token-free install, reported as reconciled:true), runs the table command privilege-dropped with a 5-minute budget, and DELETES the checkout in a finally (success or failure) before releasing the user. No snapshot is ever written by an op; pool exhaustion is a named in-body user-pool-exhausted error.
Per-entry effects profiles gate the modelless path: the registry record carries effects?: {<command>: "readonly"|"mutating"} (admin-writable only — /reconfigure accepts it and REPLACES the whole map, {} restores the default; absent entry = readonly, so test/build/status are readonly by construction). /op refuses a non-readonly entry with 409 {error:"op-refused: …"} naming the entry and its profile — the guard rail for future mutating entries; the refusal reaches the user verbatim as a 🚫 reply.
Bot-side Operations seam, two implementations: src/core/operations.ts defines named op → structured {kind:"result", ok, summary, output?} with distinct non-run kinds (refused, not-onboarded, error) — the 3-method Executor cannot express modelless ops. ResidentOperations (src/execution/resident.ts) posts /op with the operator bearer and parses the streamed body (404 → not-onboarded; op-refused… → refused; other in-body error → error; a failing run stays a result). LocalOperations (src/execution/executor.ts) is the dev/CLI implementation against the thread's local workspace dir — honest about its limits: fixed Node conventions (test → npm test, build → npm run build --if-present, status → workspace existence), a requested ref reported as ignored. The dispatcher builds the default per dispatch (resident configured → resident-backed; local execution → local; other per-thread backends → none) and CoreDeps.operations injects for tests.
Deterministic asks answer with zero model turns: (a) the explicit commands repo test <owner/name> [<ref>] / repo build <owner/name> [<ref>] are the registry's repo.test|build (src/core/commands/repo.ts; command-registry.md item 24): bound in stage A by the shared grammar — a hostile ref or slug fails the schema with a named refusal (ref: expected a plausible git branch ref (e.g. main)) before any backend, the chat gate is agentRun (= the coding allowlist, NOT repoManager), canUseRepo is checked inside, machine callers need repo:exec; (b) after the history fetch, recognizeOperation matches conservative whole-message natural forms — run [the] test[s] on <ref> [in <owner/name>] / build <ref> [in <owner/name>] — with the repo from the phrase or the thread's established repo (repoFromThread), the ref validated against the strict pattern, an owner/name-shaped "on" token treated as ambiguous, and natural recognition disabled entirely when the message carries explicit agent:/model: directives. ONLY the model call is skipped: the implicit target agent is coding, so canRunAgent(user, "coding") answers the exact refusal a normal coding request gets, and canUseRepo (item 26) refuses by repo name — both before anything executes. A result posts as ✅/❌ + summary + tail-clipped output; a refusal posts 🚫 verbatim; on the natural-language path not-onboarded repos and backend errors FALL THROUGH silently to the agent (which can still serve the ask), while explicit commands always get a reply (config-family commands never silently become a model turn).
Residents dash — the browser twin of repo list (src/channels/residentsView.ts, wired in src/index.ts): GET /residents lists every onboarded repo — state dot (warm green; onboarding/refreshing/restoring amber; degraded/down red; unreachable grey), reason, default ref, short sha, last refresh — each row a link to GET /residents/<owner>/<name>, which shows the interesting engine view for one resident: lifecycle (state/reason/last refresh error/last restore), pinned facts (ref, full sha linked to the GitHub commit, lockfile hash, provisioned/refreshed timestamps), the snapshot stamp (ref/sha/lockfile hash/backup ids), pending schedule counts, thread worktrees (each thread's ref + last-attached sha linked to its commit, OS user, deps mechanism, bound/last-attach times, evicted marker — the "which PRs are live on this resident" view, since only the default branch is pre-built), the command table with effects, and registry settings, plus the chat commands to manage it. Both pages sit behind the SAME Cloudflare Access gate as /runs (fail-closed in src/index.ts; they list every repo and its build commands, so never exposed without SSO) and are cross-linked with the runs dash. The registry is read from the admin GET /residents route on EVERY request with the admin bearer — never cached, and the bearer never reaches the page. Off-path: non-GET → 405; resident admin not configured → 503 naming execution.resident; resident Worker non-200 or unreachable → 502 carrying the upstream status/reason (never a 500 with a stack); unknown slug → 404 "not onboarded"; non-slug paths fall through. Since the Vue port the handler serves the shared web shell with the listing/record as the seed (ResidentsIndexSeed/ResidentDetailSeed; slug/live/tone helpers shared by handler and page in src/channels/residentsModel.ts), and the pages render in web/src/pages/{ResidentsIndexPage,ResidentDetailPage}.vue (mobile layouts included): hostile strings are inert by construction (the seed island escapes every angle bracket, the components render text — no v-html), only well-formed slugs become detail links, under the same WEB_HTML_HEADERS (strict CSP + X-Frame-Options: DENY + no-store) as /runs. Tab favicon: the index tab wears the dot favicon (live-view.md item 21) painted the fleet's worst tone — red if any resident is degraded/down, else amber if any is onboarding/refreshing/restoring, else green only when every resident is warm, else grey (no residents, or an unknown/unreachable one with nothing worse to show; a fleet is "all up" only when all of it is) — so a glance at the tab bar says whether the fleet needs attention. residentsFleetTone (residentsModel.ts) is the one worst-of rule, and the shell renders the dot straight from the seed (webShell.ts's pageFavicon): the page is a snapshot with no live feed, so the server render is the truth and the client does not repaint it. The detail page wears the neutral mark — one resident's state is the page's own dot, not a fleet claim.
Exec survives a deploy — a runtime replacement is named, never mistaken for a dead sandbox (a wrangler deploy of switchboard-resident mid-run replaces the ResidentDO isolate and the Sandbox SDK rejects the running command's handle with Process handle refers to a previous runtime incarnation; unclassified, two such exec failures trip the runner's fail-fast and the run aborts as a fake "OOM"). Resident side (run() in worker.ts): the spawn and output-collection phases are separated. A runtime replacement during collect means the process started and its output is lost — never re-run. isRuntimeReplacement classifies one from the pinned @cloudflare/sandbox@0.13.0-next.751.1, typed checks first, wording last:
- StaleProcessHandleError — the SDK's own verdict on a process handle; its control-plane wrapper also collapses any interruption it sees into this class.
- RuntimeIdentityInactiveError — an exported class the SDK throws raw at ~10 process/exec sites before its adapter would translate it.
- OperationInterruptedError with reason runtime_replaced, sandbox_lifetime_changed, or container_stopped. NOT transport_disposed / sandbox_destroyed / recovery_exhausted (real failures) and NOT unknown (no evidence — conservative).
- RPCTransportError with kind peer_closed, connection_failed, upgrade_failed, or session_disposed — how a container stop/restart surfaces while output is being collected: the SDK's log-subscription reader deliberately does NOT translate transport loss into an interruption, so the raw transport error reaches run(). NOT invalid_frame / protocol_error (wire-format bugs) or unknown. A mere network blip lands here too, and that is fine: the outcome is the same safe one (re-attach once, never re-run, the model re-checks effects).
- the platform's superseded-isolate reset (isDurableObjectCodeUpdateReset).
- fallback: the SDK's replacement wordings (RUNTIME_REPLACEMENT_WORDING, shared with the refresh classifier in src/execution/residentRefresh.ts — the six SDK messages, Process supervisor is closed (the spawn refusal a stopped container answers until it restarts), and the raw workerd binding refusal The container is not running, consider calling start()), matched on the error and its cause chain (bounded to 8 like the SDK's own selfAndCauses). That last wording is the container ROLL case, distinct from the container_stopped interruption above: a deploy that rolls the container out (not just swaps the DO isolate) drops it onto an empty ephemeral disk, and a /exec racing that window gets the spawn refused outright — the SDK's own auto-start (runWaking → startAndWaitForPorts) re-throws it untyped when the roll outlasts its port-ready bound, so there is no typed class and the wording is the only signal. It can never match a genuine container crash (container exited with unexpected exit code) or the readiness probe (the container is not listening), which stay ordinary failures. During spawn, the resident retries ONCE on a fresh process sandbox only when the SDK marks the interruption retryable (its own vouch that the process never started, e.g. container still starting); the SDK sets retryable:false for runtime_replaced, so a deploy-time spawn failure is also never re-run. Both unsafe cases answer the NAMED error {error:"runtime-replaced: …", status:409, reason:"runtime-replaced"} on /exec (in-body over the 200 stream, with reason now always forwarded), /read, and /write. Client side (ResidentExecutor.opWithReattach): on reason:"runtime-replaced" it re-attaches once (proving the new isolate serves the thread), then retries only the idempotent routes (/read, /write); /exec is handed back to the model as ordinary output ("the command may have started; re-check its effects before re-running") — not an ExecInfraError, so one deploy never counts toward the dead-sandbox breaker. A second runtime-replaced with no successful op between IS an ExecInfraError (a deploy storm or a flapping resident), so the breaker keeps its teeth; any successful op resets the streak. What a container roll costs the run: it is recovered, not free. The single re-attach drives the resident's worktree-missing recovery — the container is auto-started and the tree rebuilt/restored from R2 (~1–2 min observed) — during which the /exec that hit the roll BLOCKS, so the model's next command lands on the warm, restored tree. The bound is the same streak: a container that cannot come back keeps answering runtime-replaced, and the second one with no success between aborts the run.
Deploy preflight — a Worker deploy never lands on an in-flight run (a deploy swaps every ResidentDO isolate and invalidates the Sandbox SDK process handles — Process handle refers to a previous runtime incarnation — which reconcileImage's container-restart deferral cannot see). In-flight activity is now observable: GET /status answers {state, reason, inFlight} and GET /residents carries live.inFlight per resident plus an aggregate inFlight — the sum over residents whose live view answered with a count, null (with inFlightUnknown = how many) as soon as any resident's view failed or carried no count, so a reader can never mistake "unknown" for 0 — (same inFlightCount() predicate as reconcileImage/isIdle: thread exec/read/write, disposable /op users, attaches past their image check, refresh cycles past their gates). npm run deploy in deploy/cloudflare-resident/ is node preflight.mjs && wrangler deploy && npm run wake: the preflight GETs /residents with the first of RESIDENT_ADMIN_TOKEN / RESIDENT_OPERATOR_TOKEN / RESIDENT_READ_TOKEN in the env and exits 1 naming every busy resident and its count. Fail closed: no bearer, unreachable/non-2xx/non-JSON Worker, a resident whose live view errored, a live view without inFlight (a Worker predating this item), or an impossible count (negative / non-integer — a counter bug) all refuse with the operator hint. RESIDENT_DEPLOY_FORCE=1 (or --force on a direct node preflight.mjs) deploys anyway with a warning that still names what it is killing. The decision (decide()) is pure and unit-tested; only main() touches the network. A provisioning resident refuses too; a refresh or restore mid-cycle only warns: a resident whose live state is onboarding has provisioning executing, which an isolate swap kills exactly like a thread run and which has no checkpoint to resume from — provision-failed, down, a rebuild to recover — so it refuses by name. refreshing and restoring have the engine executing too, but the swap only INTERRUPTS them: a refresh re-arms in 45 s and resumes from its disk checkpoints (below, and item 48), a restore is retried by the next hydrate, which first unmounts and removes whatever the interrupted one left (item 61); they used to refuse like onboarding, and one release deploy spent its whole budget behind a resident stuck in restoring — a state the swap would have cleared — and went red. They are now allowed with a WARNING that names the resident and the state (preflight ok: … — WARNING: mid-cycle: repo:… (restoring) — …), never REFUSED; runs in flight still refuse whatever the state. (Without the short re-arm the interruption itself was costly: degraded(build-failed: exit 143: Session terminated, killing shell…), self-healed only at the next 600 s alarm — a full cycle of cold runs.) A deploy that lands mid-cycle anyway (forced, or a path the preflight cannot see) recovers within a minute: the pure classifyRefreshFailure (src/execution/residentRefresh.ts) recognises a step killed from outside — exit 143, Session terminated, SIGTERM; never a step the cycle itself timed out — as degraded(refresh-interrupted: <step> …) instead of <step>-failed; that reason is not evidence about the repo, so it never counts toward the DEGRADED_PARK_AFTER_CYCLES streak (same non-evidence gate as the watchdog stamps), and the chain re-arms at INTERRUPTED_REARM_S (45 s — longer than a container restart + rehydration, an order of magnitude under the cadence) via nextRefreshDelayS instead of 600 s — for at most INTERRUPTED_REARM_MAX_CONSECUTIVE (3) consecutive interrupted cycles (resident:interruptedStreak, cleared by any other outcome), after which the cadence returns so a repo whose own build output happens to carry the signature every cycle cannot retry at 13× cost forever. An image-stale container stop (reconcileImage) re-arms short the same way, uncapped (it cannot repeat by the same cause). The gate is an allow-list of known states — warm/degraded/down with inFlight: 0 allow; refreshing/restoring allow with the warning above; onboarding refuses by name; any state the script does not recognize refuses as unknown (the script is plain JS outside the shared ResidentLifecycleState type, so vocabulary drift fails closed). Runs and cycles are reported independently — a resident that is both busy and mid-cycle names both. Since inFlight counts the refresh cycle itself (inFlightCount = runs + refreshesInFlight), a resident that is merely refreshing reads ≥ 1 and would refuse as busy whatever the state rule says; so the live view also carries runsInFlight — the runs alone (thread exec/read/write, disposable /op users, attaches mid-flight; runsInFlightCount) — and the preflight judges busy on that when present, on inFlight for a Worker without the field (over-refusing, never under). The first live sample of the revision (a refresh window on the switchboard resident, the real preflight.mjs every 0.2 s) showed exactly that gap: the WARNING suffix named refreshing on every sample, but each was still REFUSED … (1 in flight) for the cycle's own count.
Event-triggered reclamation — a finished ref gives its worktree back within one refresh cycle (gc.ts + reclaimFinishedRefs in worker.ts): idle signals (items 16b/23) collapse a worktree an hour to seven days after its last attach; this reclaims it when the WORK finishes. It is a POLL folded into the existing refresh alarm, not a webhook — decided from what the App is configured for: GET /app answers events: [] and GET /app/hook/config 404s (the App was created with webhooks off, per the README recipe), and turning one on would mean a public unauthenticated ingress with HMAC verification, a webhook secret in a second credential domain, and a UI-side App change, all to deliver a signal the cycle's git fetch --prune already carries for free. Each cycle, after the prune and the warm flip, every LIVE binding whose ref is not the default branch is classified in a CONCURRENT pre-pass over the distinct refs: branch existence comes from ONE git for-each-ref --format='%(refname:short)' refs/heads/ listing of the mirror (parseRefListing in gc.ts — one spawn for the whole pass, not a rev-parse per ref; an unreadable listing leaves membership unknown so the PR lookup decides, never "every branch gone") — a ref absent from it → gone; otherwise ONE GET /repos/<slug>/pulls?state=all&head=<owner>:<ref> per distinct ref, the lookups running concurrently under Promise.all (the cycle's minted repo-scoped token, or anonymous when the App is unconfigured; 10 s budget each) → an open PR wins (open), else merged beats closed (merged/closed), none → no-pr; any transport/non-200/unparsable answer → unknown. The eviction loop itself stays serial — its re-read guards depend on ordering. reclaimDecision evicts ONLY gone/merged/closed bindings with no op in flight and a tree that is provably clean as the thread user (worktreeCleanliness — a merged PR can still hold unpushed local commits; dirty or unverifiable → kept dirty, the TTL sweep remains the backstop). Squash merges do NOT defeat the clean check: git rev-list HEAD --not --remotes counts against the worktree's OWN remote-tracking refs, which are local snapshots advanced by the worktree's own git push — GitHub rewriting history on merge never touches them, so a pushed-then-squash-merged branch reads 0 unpushed. The one exposure is a thread that itself ran git fetch --prune after the branch was deleted (its origin/<ref> vanishes and the branch commits count as unpushed against origin/main) — kept dirty by design, falling to the TTL; a slept container has no tree left, so nothing needs preserving and the pool user is released outright. The default branch is never reclaimed; open/no-pr/unknown keep with the reason named. The eviction is the sweep's evictBinding with the same re-read guards (binding re-read + op counter right before removal; lastAttachAt unchanged), and it now stamps the binding with evictedWhy (merged #N, gone, clean-idle, ttl, detach) — surfaced in /residents live.threads, /debug threads, and the dash's evicted marker — so every eviction is auditable. A pass failure is a log line, never a lifecycle flip. When the runtime is DOWN the mirror is not consulted (a run would wake the container and read an empty disk as "every branch gone"); only the PR lookup can speak. POST /debug {"op":"reclaim-now"} (admin) runs a fetch --prune (bounded by ATTACH_MUTEX_WAIT_MS) then the exact pass, answering {reclaimed:[{threadKey, ref, why}], kept:[{threadKey, ref, why}], fetch}.
Resident-level LRU eviction — opt-in, default off (pickEvictionCandidate in gc.ts): POST /onboard {…, evictColdest:true} (admin route; a non-boolean value → 400) changes the over-cap answer from 429 to "make room": when the registry refuses with 429, the Worker collects every resident's registry record + live view (collectResidentViews), picks the coldest ELIGIBLE one, swaps the registry rows in ONE input-gated section (ResidentRegistryDO.replace: the victim's slot is freed and the newcomer inserted atomically — a concurrent onboard can never take the freed slot and leave a resident destroyed for nothing; 409 if the victim was offboarded meanwhile or the newcomer already exists), and only THEN tears the victim down through the item-11 teardown minus the registry step (teardownResident, shared with POST /offboard's offboardResident). Eligible = live state warm (idle-parked is still warm; degraded/down/mid-flight/unknown are rejected by name), inFlight === 0 (null — a live view that failed — is rejected, never read as idle), ZERO live worktrees (a live tree may hold uncommitted work; evicted bindings do not count), and last activity — the newest lastAttachAt across all bindings, or provisionedAt/onboardedAt when never attached — older than LRU_FLOOR_S (1 h, = IDLE_AFTER_S) so a repo used minutes ago never goes cold to make room. Coldest = oldest last activity, ties on resource name. The 202 carries evicted: {resource, lastActivityAt, registryRemoved, schedulesCancelled, containerStopped, storageCleared, backupObjectsDeleted, r2ObjectsDeleted, errors}; no eligible candidate → the ordinary 429 with rejected: [{resource, why}] itemizing every resident (e.g. active 12m ago (floor 60m), 2 live worktree(s), state down), so the admin can offboard by hand with the facts in front of them; a refused swap (409/429 from replace) destroys nothing and names the candidate as wouldHaveEvicted. Without the flag nothing changes: the registry's 429 now also names the opt-in. Chat: repo onboard <owner/name> [...] --evict-coldest (behind the same fail-closed canManageRepos gate — the flag provisions AND destroys billable compute) sends evictColdest:true; the reply adds a ♻️ Made room: evicted … line naming the resident and its last use, and an over-cap refusal relays the per-resident reasons as bullets. --evict-coldest on repo reconfigure is refused by name.
Not here — lower-contention refresh: the refresh cycle's checkout rebuild (git clean -fdx + install + build + snapshot) runs UNDER the mirror mutex because materializeThreadDeps hardlink-copies CHECKOUT_DIR under the same lock (a torn cache would produce a false ❌ from repo test), so a heavy install can push concurrent attaches past ATTACH_MUTEX_WAIT_MS into mirror-busy 503s and cold fallbacks. Moving install/build into a staging checkout and swapping under the lock touches the snapshot stamp, READY_MARKER, and the restore path — a separate change with its own live proof, kept out of GC.
Refresh install gate + disk checkpoints — refreshes stay under the attach wait (the complement of item 24's serviceable-state attach): the rebuild on a sha move is decided by the pure planRefresh (src/execution/residentRefresh.ts, imported by the Worker like residentDetach) from the committed-lockfile key at the new sha (item 6's pure-function key, computed from the mirror BEFORE any checkout work) and three disk facts read by readRefreshDisk: the checkout's real HEAD (as worker1) and two root-only markers under /workspace/.resident/ — deps-key (the lockfile key whose install fully completed into node_modules) and built (the sha whose build fully completed). Rules: (a) lockfile key unchanged → rebuild {install:false, clean:"keep-deps"}: git clean -fdx -e node_modules (a git exclude that survives -x, matching workspace packages' node_modules at any depth) removes every other gitignored path so the build still allocates fresh inodes, and the build-written caches INSIDE node_modules (.cache, .vite — NODE_MODULES_CACHE_DIRS, pruned at any depth) are swept too, since those open+truncate in place and the build runs as the inode owner (review 1b holds — nothing installs, and nothing the build writes lands on a kept inode; residual: a build that writes elsewhere inside node_modules is outside the sweep), then build only — seconds, not the 1–3 min unconditional reinstall that pushed refreshes past the bot's 60 s ATTACH_MUTEX_WAIT_MS; (b) key changed, or no deps-key marker (a container from before the markers, or a full clean that was interrupted) → the unchanged full path: git clean -fdx + install + build; (c) checkpoint hit — HEAD, built and deps-key all already match the target → reuse: skip checkout/install/build and go straight to the snapshot (facts + stamp must still move together, so the snapshot is never skipped) — this is what a DO reset mid-cycle (a resident-Worker deploy; the container disk survives it) leaves behind, and it previously cost a full rebuild; (d) HEAD ≠ sha or built ≠ sha with a matching key → keep-deps rebuild (never reuse on a marker alone). Marker discipline: the markers for the steps being redone are removed BEFORE the step starts (built always; deps-key only on a full install), written right after the step completes, written from the stamp on restore (ensureHydrated) and at the end of provisioning, and wiped with the workspace on every rm -rf of the disk. facts.sha is NOT checkpointed early: DO facts and the snapshot stamp are verified against each other on hydrate (item 5), so advancing facts before the snapshot would turn a sleep into snapshot-stamp-mismatch/stale-serving. Each planned cycle logs refresh: <old> → <new>: <action> (<why>) and refresh: <sha> <action> done in <ms>ms for wrangler tail receipts. A cycle a deploy kills mid-step leaves these checkpoints behind and re-arms in 45 s as refresh-interrupted (item 44), so the retry is a reuse or a keep-deps rebuild a minute later, not a full rebuild ten minutes later.
Test overrides — the over-cap path stays provable at any production cap (POST /debug {"op":"set-test-overrides", cap?, floorS?}, admin scope; gc.ts parseTestOverrides / effectiveLimits, registry limits() / setTestOverrides()): the 429 / rejected[] / eviction behaviors of item 46 are reachable only when the registry is FULL, and RESIDENT_CAP is sized for the team's real fleet — proving item 46 by changing the compiled cap costs a deploy each way plus the floor hour, and would be impossible once six real residents exist (lowering the compiled cap below the fleet size refuses the team's own onboards). This is the resident's fault-injection pattern (item 13: backdate-thread, force-down, force-onboarding, …) applied to the two limits. Mechanics: the op stores {cap?, floorS?, setAt, build} in the registry DO under testOverrides (outside the resident: prefix, so it never counts as a slot); limits() answers min(override, constant) for each field and is read INSIDE the same input-gated section as the count/insert (onboard, replace), so an override flip can never interleave with an onboard; handleOnboard's LRU pick uses the effective floorS. Guard rails by construction: (a) admin only — not in READ_DEBUG_OPS, so the read token gets 403; (b) an override can only LOWER: cap must be an integer in [1, RESIDENT_CAP], floorS in [0, LRU_FLOOR_S], else 400 naming the range — it can never become a back door past wrangler's max_instances, and a stored value above the constant (a later deploy lowered it) is clamped, never honored; (c) deploy-scoped — the record carries BUILD_MARKER (the /healthz u, bumped every deploy-worthy change) and a different build ignores it (ignored:"stale-build <marker>"), so a forgotten test cap cannot outlive the session that set it; (d) visible — GET /residents reports the ENFORCED cap plus capDefault and, when active, testOverrides {cap?, floorS, floorDefaultS, setAt, build}; repo list appends ⚠️ test overrides active (set <when>): cap N (default M), LRU floor Ns (default Ms) — clear with …; the watchdog summary's cap is the effective one too. An empty body (neither field) clears. The op answers {cap, capDefault, floorS, floorDefaultS, override}.
Read-only attach — a read-only run gets a worktree that cannot fetch or push, enforced by the tree, not requested of the model (the complement of the reviewed-head guard, agent-review.md item 8): a review run that is read-only only "by convention" holds a per-attach credential file and a GitHub origin, so it can fetch another PR's branch and land its verdict on the wrong PR; the guard makes the post-step refuse such a review, this removes the capability. POST /attach accepts readonly: true (validated boolean; absent = false, so an older bot's body means what it always did — parseReadonly in src/execution/residentReadonly.ts). The bot sends it from the agent's declared toolset (ResidentExecutor opts readonly, set by makeExecutor iff agent.toolset === "readonly" — the review agent), never from the prompt. On a read-only attach the pure planReadonlyAttach (imported by the Worker like residentDetach) yields: no token for the tree (nothing to leak, nothing to push with; credentials: "none" — a new honest member of the union, distinct from unavailable = wanted one and could not get it — while the mirror's own recovery fetch for a ref pushed since the last refresh cycle still mints a fetch-only token as root, outside the tree: a read-only attach loses no availability, only the tree's credentials), origin left at the local mirror path (git remote set-url origin /workspace/mirror instead of the GitHub URL) — thread users are denied traversal into the mirror (root:worker1 750, item 19), so git fetch/git push fail legibly while the clone-time remote-tracking refs (origin/<default> and every branch the mirror had) still serve git diff origin/<base>...HEAD, and the tree is scrubbed on every read-only attach, reused tree included (rm -f .git/github-credentials + git config --unset-all credential.helper), because the file was written per attach before this rule. The binding records readonly (absent on pre-field bindings = writable) and AttachOk.readonly echoes it. Mode switch recreates: a live tree built for the other mode (read-only attach on a writable tree, or a writable attach on a read-only one) is wiped before the ordinary dirty/stale checks — a credential-less mirror-origin tree must never serve a writable run, and a GitHub-origin tree must never serve a read-only one; recreating is the simple safe option (a tree carries only scratch state between attaches, and one thread is normally one agent for its whole life). An evicted prior's recorded mode never forces a switch (the tree is gone anyway). Deploy order: resident first (an older resident ignores the unknown field — handleAttach reads named fields only — so a new bot against an old resident degrades to today's writable attach, loudly visible as credentials: "ok" on a review attach), then the bot.
Attach fetches a STALE ref, not only a missing one — the caller names the commit it expects: the mirror is fetched on the refresh cycle (item 7) and, on attach, when the bound ref is missing; a ref present at an older tip (pushed to since the last cycle) would otherwise be cloned as-is, and a review of it is refused by the reviewed-head guard (agent-review.md item 8) — a head the bot had already resolved. POST /attach accepts sha (optional; a full 40-hex lowercase commit — a full sha names exactly one commit, no prefix ambiguity — validated before it can become a git argument: parseWantSha in src/execution/residentHead.ts; absent = the previous behavior, so an older bot's body means what it always did). The bot sends RepoContext.headSha whenever repo resolution found a PR head (ExecutorContext.headSha → ResidentExecutor opts sha), on both the review and coding paths. The sha applies to the ref it was resolved for: the pure wantShaForBinding drops it when a refHint was named that is not the thread's sticky bound ref (such a thread can never be at that commit, so it must not pay a fetch on every attach); no refHint means the caller intends the bound ref. Under the mirror mutex the pure mirrorNeedsFetch({refExists, mirrorSha, wantSha}) decides: ref missing → fetch (unchanged); sha named and the mirror's tip of the ref is not that commit → fetch, then re-read the tip; otherwise no fetch. A read-only attach still gets the fetch-only token for this recovery fetch (item 50's rule covers the stale case as well as the missing one). The attach then clones/reuses at the mirror's tip as before — the stale-tree rule in ensureThreadWorktree (HEAD neither the tip nor a descendant → recreate) wipes the old tree, so a sticky thread binding never pins the previous run's commit. If the tip is STILL not the expected commit after the fetch (a push racing this attach, a force-push), the attach proceeds on the fetched tip and reports it in sha; the reviewed-head guard remains the backstop for what a review of it may post. A ref that is gone after the fetch is not yet a refusal: the pure attachTarget({refExists, wantSha, commitInMirror}) decides — the ref's tip whenever the ref exists; when it does not but the caller named a commit the mirror holds (git cat-file -e <sha>^{commit}; a --mirror clone carries refs/pull/*, so a merged PR's head outlives its deleted branch), the thread tree is cloned --no-checkout and checked out at that commit detached (worktree-detach), reported in sha as any attach; only a ref that is gone with no expected commit, or one the mirror does not hold either, is unknown-ref (the message now says which). Before this a review of a merged PR — its branch deleted on merge — fell back to a cold sandbox that cloned the same commit itself, after the resident had already fetched. Deploy order: resident first (an older resident ignores the unknown field), then the bot.
The onboard command table comes from the repo root, and provisioning reports its outcome into the thread (a bare onboard that assumes npm leaves a pnpm workspace down on npm install with EOVERRIDE and a Terraform repo with no package.json down with ENOENT — for hours, when the acknowledgement says only "watch repo list", which nobody does). Two changes. (a) Detection: repo onboard reads the repo root over the GitHub REST API with the App's read-scoped installation token (githubRepoInspector in src/execution/githubRepoInspect.ts: the tree of the onboard ref, then package.json when the tree has one — two GETs, never a clone, never gh; invariant 5) and the pure detectCommands (src/core/repoToolchain.ts) decides the table: package manager from the packageManager field, else the root lockfile (pnpm-lock.yaml → pnpm, yarn.lock → yarn, --immutable under a .yarnrc.yml, bun.lock[b] → bun, package-lock.json/npm-shrinkwrap.json → npm), else npm; install is the manager's frozen-lockfile form (npm keeps the proven npm install --no-audit --no-fund; every manager detection can name — pnpm and yarn included — is baked into the resident image, execution.md item 10, so a detected install never fails on a missing binary); build/test are <pm> run build / <pm> test ONLY when package.json has that script, else the no-op true (the resident's parseCommands requires both keys; the reply renders it as "none"); a root with no package.json gets NO install and no-op build/test — never an npm command that can only fail. An explicit --install/--build/--test wins per key and is rendered verbatim (an explicit --build true is the operator's command, not a detected "none"). The reply names the toolchain and the reasons (Toolchain: pnpm (package manager from pnpm-lock.yaml; no build script — build is a no-op)); an uninspectable root (no credential, a 404 for a repo outside the installation or an unknown ref, a failed call, no inspector wired) falls back to the npm table WITH a ⚠️ line saying the root was not inspected and which keys took a default (using npm defaults for build and test; when all three were given explicitly the line says nothing was assumed) — the table is never silently assumed. The resident side matches: a table with no install skips the thread-install on a lockfile mismatch (materializeThreadDeps → deps: "none") instead of the old npm install fallback. (b) Outcome: repo.onboard and a non-dry repo.rebuild carry the registry's settle (command-registry.md item 26): after the 202 reply, the chat adapter polls the resident's GET /status (admin client status(resource)) every SETTLE_POLL_MS (10 s) while the state is onboarding and posts a SECOND reply in the thread — ✅ … is warm, or ❌ … failed to provision: <the resident's reason> followed by the hint that fits the reason: when the failure was at install/build/test (a step the table controls), the two commands that fix a bad table (repo reconfigure … --install/--build/--test, then repo rebuild …); for any other step or reason (clone, snapshot, timeout, none recorded) a plain retry (repo rebuild …) naming the resident/Cloudflare side as the likely fault — never the table hint (a failed snapshot put is not a table problem, and the table hint would mislead); a 404 (offboarded meanwhile), a non-200, a transport failure, another state, and the SETTLE_MAX_MS (12 min) bound each produce their own sentence, never silence. The acknowledgement itself now says a report will follow and that repo list shows the live state meanwhile. Best-effort by design: the poll lives in the bot process, so a restart mid-provision loses the follow-up — the resident's state is never in doubt (repo list, the residents dash, the watchdog's deadline), only the notification is.
A failed step names the failure, and the image's toolchain is pinned to a version, not to a build date (the shape that forced it: a resident down at provision-failed at install: exit 1: [WARN] The "pnpm" field in package.json is no longer read by pnpm … — a warning that cannot fail an install: the same command prints those exact bytes on stderr and exits 0, reproduced inside the resident's own base image. The real error was on stdout and was thrown away, so that failure's cause was unrecoverable). Two changes, one for the report and one for the cause it was hiding. (a) The report never chooses between the streams: describeStepFailure (src/execution/residentStepReport.ts, pure) replaces tail(r.stderr || r.stdout) everywhere a command result becomes an error — assertOk (every provisioning/refresh step), runDepScript (thread dep-cache), POST /read — and reports BOTH streams, labelled, tail-first (a tool's error is its last output), each bounded on its own (400 chars) so one noisy stream cannot crowd out the other; a step that failed silently says no output instead of an empty tail, a timeout still says (timed out), and describing a success throws rather than write a lie into the record. The || was not a style choice but a bug with a family attached: the pnpm/bun family reports through its own logger on stdout, so for those tools stderr-first is exactly backwards. On failure the DO also console.logs stepFailureLog — 4000 chars of each stream into the Worker log (observability is on) — because a 400-char reason is a pointer, not a diagnosis. (b) Pinned toolchains: the resident image installed pnpm@latest yarn@latest, which made the pnpm major a property of the last image build; an image rebuild moved pnpm 10 → 11, and pnpm 11 stopped reading package.json's pnpm field, so a repo keeping overrides / patchedDependencies / onlyBuiltDependencies there fails --frozen-lockfile with ERR_PNPM_LOCKFILE_CONFIG_MISMATCH (reproduced). Both images now name exact versions and assert them at build time (a grep on the installed version), src/deploy/imagePins.ts parses the Dockerfiles and imagePins.test.ts fails if a floating tag returns. pnpm is pinned on the 10 line deliberately: 10.x reads that field, and pnpm ≥10 self-manages packageManager, so a repo pinning 11/12 still gets its own version — the pin is a floor, not a ceiling. That self-management also corrects a claim the Dockerfile used to make: a repo with a packageManager field downloads that version at provision time (a repo pinning pnpm@10.10.0, say), so a pnpm install needs the registry, not just the image.
A full container disk is disk-full, never serviceable, and the resident recycles it (pure decisions in src/execution/residentDisk.ts, the classifier in residentRefresh.ts). The shape: every attach fails attach-failed at git-setup: exit 4: stderr: error: failed to write new configuration file /etc/gitconfig.lock and falls back cold, while the resident's own state reads degraded(github-unreachable: Failed to write file '/workspace/.resident/git-credentials': ENOSPC: no space left on device …). The obvious hypothesis — a stale /etc/gitconfig.lock left by a killed git — is not this failure: a pre-existing lock makes git config say could not lock config file /etc/gitconfig: File exists and exit 255 (reproduced), and /etc is image layer, recycled with the container. failed to write new configuration file <x>.lock is git's write_error() — a write() on the freshly created lock file failed — and it is reached with exit 4 exactly when the disk is full but an inode is still creatable (reproduced on a full Linux tmpfs: byte-identical stderr). git prints no errno there, which is how a disk full of thread trees read like a lock bug. The defect is twofold: (a) ENOSPC during the cycle's credential-file write was recorded as github-unreachable, a reason on the bot's serviceable allow-list (item 24), so the bot kept attaching to a resident that could not take a worktree, a credential file, or a config lock; (b) nothing freed the disk — the resident stayed that way until the container happened to recycle. Classification: classifyRefreshFailure gains a third class, disk-full: <step> <message> (/workspace: <n> KiB free). A message carrying the errno wording (ENOSPC, No space left on device — Node, git, cp, pnpm all pass it through) is disk-full outright; a kill signature without it stays refresh-interrupted whatever the disk holds; otherwise the Worker runs ONE probe (df -Pk /workspace, only after a step has already failed, never on the hot path) and a free space below DISK_FULL_FREE_KIB (128 MiB — less than one checkout of the largest onboarded tree plus git's scratch, so a fetch or worktree add cannot complete anyway) names the disk; no probe answer is never treated as full. Applied at every step failure the cycle records (the fetch/mint catch included — the github-unreachable site) and to a failed attach step, which flips the resident degraded(disk-full: …) and pulls the refresh cycle to +1 s; disk-full: is deliberately NOT on the serviceable allow-list, so the next dispatch goes cold with resident degraded (disk-full: …) — using fresh sandbox on the card and no attach. The cycle owns the verdict: a cycle entering on degraded(disk-full: …) re-probes before fetching — still below the floor → it decides recovery and returns (a fetch that happened to fit would flip warm, the bot would attach, git-setup would fail, attach would flip it back: a flap loop); space back (a detach or the hourly sweep freed trees) → the cycle runs and earns warm honestly. Recovery: the disk is a cache, so planDiskFullRecovery stops the container (this.stop(), the same swap reconcileImage does) and re-arms at INTERRUPTED_REARM_S (disk-full-restart), so the next alarm takes the wake path — restoring → restore from R2 → warm on an empty disk; live bindings keep their users and their trees are recreated on the next attach (worktree-missing → attach, item 21). A recycle is refused, with the why on lastRefreshError (… — container kept: <why>) and the degraded reason kept clean, when (checked in this order) the last recycle is inside DISK_FULL_RECYCLE_COOLDOWN_MS (1 h: a disk that refills within the hour is a working set the instance cannot hold — the why says to resize or offboard, and a second recycle would only throw away another restore), when any operation is in flight (the calling cycle excluded from the count), or when any live tree is dirty/unpushed or unreadable as its thread user (a recycle destroys the tree; an unreadable one is kept, never guessed clean). The recovery lives in the refresh cycle only: an attach is in flight by definition, so it never stops the container itself.
Disk is a measured, budgeted resource — never discovered by ENOSPC (pure module src/execution/residentDiskBudget.ts, imported by the Worker like residentDisk/residentRefresh). Item 54 names a disk that HAS filled; this keeps it from filling. Measure: at the end of every completed refresh cycle, and — deferred by one second so the du stays off the hot path — after every attach, every detach and every sweep eviction, the Worker runs ONE df -Pk /workspace (parseDfKiB: total/used/free) and ONE du -xsk over the parts in duArgv's fixed order — the mirror, then checkout/node_modules BEFORE the checkout (GNU du counts a hardlinked inode once per invocation and charges it to the first argument that reaches it, so the shared deps land on the deps term and the checkout term reads as history + tree + build output), then each live thread's dir (its UNIQUE bytes: own history clone, tree, the item-18 per-thread copies; an install thread's own node_modules), then every pool user's home (homes: a pnpm store or npm cache an install left outside the tree) — and persists DiskSample {at, totalKiB, usedKiB, freeKiB, parts:{mirror, deps, checkout, threads:{<threadKey>: KiB}, homes:{<user>: KiB}, other}} at resident:disk (other = used minus everything itemized: the image, /tmp; a part du could not read is null, never 0). Surfaced as live.disk on GET /residents (and /debug info), as · disk <used>/<total> (<pct>%) on every repo list line, as the gauge on the /residents index row and a Disk section on /residents/<slug> (used/total, free — under the diskBudgetMb cap when one is set — the reserve and its two terms, the headroom as "room for N more hardlinked / M deps-installing trees", the sample time, and every component with homes above 1 MiB), and as · disk max <pct>% (<owner/name>) on the watchdog's firing line (watchdogCheck carries each resident's gauge from storage — the watchdog never touches the container). POST /debug {"op":"measure-disk"} (admin) takes the sample now and answers it. A sleeping container is never woken to measure; before the first measurement of an incarnation the dash says "not measured yet". Measured on a freshly restored pnpm-workspace resident with one hardlinked thread tree: df 15 086 920 KiB total / 4 262 360 used; mirror ≈ 371 MB (the checkout's .git), deps 2 244 052 KiB, checkout rest 462 888 KiB, the thread tree 541 860 KiB unique, /home/worker1 4 KiB — there is NO pnpm store on a restored disk (the store is not in the snapshot) and where an install has run the store hardlinks node_modules on the same ext4 filesystem (link count 2 on every .pnpm file = checkout + the one thread; a store copy would read 3), so deps are never stored twice; du of the readable tree (3 880 312 KiB) + the root-only mirror accounts for df used to within 10 MB. Budget admission (admitThreadDisk, before the mirror lock — making room takes it): a thread whose tree is on disk is reuse (0); otherwise the cost is projected from the LAST sample's checkout parts — hardlink = parts.checkout when the committed lockfile at the ref's mirror tip equals the warm checkout's key (a ref not yet in the mirror is projected as hardlink, the common case), install = parts.checkout + parts.deps when it differs — and the tree is created only when free − reserve ≥ projected (checkDiskAdmission), where free is this attach's own fresh df minus the projections of attaches admitted but not yet on disk (diskCommittedKiB, released when the attach settles), capped at diskBudgetMb − used when the record sets diskBudgetMb below the disk (the first thing that ever read the field; effectiveFreeKiB), and reserve = staging + floor with staging 0.6 × (mirror + deps + checkout) (the SDK may stage the R2 snapshot on local disk; the same ratio instanceSizing.test.ts sizes the instance with — imported, so they cannot drift) and floor max(1 GiB, 5 % of total) (1 GiB ≈ two hardlinked trees of slack for what no projection sees — git pack scratch, a cp -al falling back to a plain copy, /tmp — and 8× item 54's 128 MiB, so admission always speaks before the failure classifier has to). With no sample yet the projection is null and the tree is admitted while free clears the reserve (a fresh resident is never refused on a guess; a blind admission past the floor is still refused); no df answer → admitted (unknown is never refused, item 54's rule). Making room: when it does not fit, live trees are ordered by orderEvictionCandidates — never the requesting thread, never a tree with an op in flight, never the default branch (the one most likely re-attached; reclaimDecision's rule), never one attached within DISK_EVICT_MIN_IDLE_MS (10 min: longer than any gap between a run's tool calls, shorter than the hourly clean-idle release) — coldest lastAttachAt first; each candidate's cleanliness is checked as its thread user (worktreeCleanliness; dirty or unreadable → kept, named), the sweep's re-read guards run (binding unchanged, op counter 0), then evictBinding removes the tree (why disk-pressure) and df is re-probed, until it fits. Refusal: still short → 503 {error: "disk-pressure: …", state, reason: "disk-pressure"} — the same shape as mirror-busy, so ResidentExecutor.attach throws and makeExecutor falls back to the cold sandbox with resident attach failed (… disk-pressure: …) — using fresh sandbox on the card (item 24) — and the error text IS the math: need <projected> for a new tree (<kind>), but <free> free[ under the <cap> diskBudgetMb cap] minus the <reserve> reserve (snapshot staging <s> + floor <f>) leaves <x> — short by <y>; evicted <n> idle tree(s) (<GiB> back): <keys>; kept <m>: <key> (<why>), … (diskPressureReason; an unmeasured projection says so). Never a lifecycle flip: the checkout is intact and every existing tree keeps serving; only new trees are refused. Eviction also clears the user's leftovers: every evictBinding (sweep, reclaim, detach, disk-pressure) removes ~/.local/share/pnpm, ~/.cache, ~/.npm, ~/.yarn, ~/.bun under the pool user's home (threadUserCacheCleanArgv; never the build user's) — an install thread's pnpm store is its tree's hardlink source (0 unique bytes while the tree lives, ALL of them once it is removed) and a pool user is an arbitrary slot, so a store left behind is almost never reused and 16 users × a 2.3 GB store is the whole disk. Sizing (instanceSizing.test.ts, re-derived with these inputs): image 552 MB + mirror 380 + checkout 2772 + reserve (staging 1891 + floor 1074) = 6669 MB base; a hardlinked tree 555 MB (measured whole); an installing tree 2853 MB; the 20 000 MB instance (4 vCPU / 12 GiB / 20 GB, the platform's custom-type ceiling: the CPU is the decision — 16 threads sharing one core was the fifty-concurrent-runs plan's D6 — memory follows the vCPUs at 3 GiB each, the disk follows the memory at 2 GB per GiB) mounts as ~18 858 MB usable → 16 hardlinked + 1 deps-installing trees, or 12 + 2, fit with the reserve; the pool's maximum (16 + 2 = 21 255 MB) does not fit even this ceiling, so admission is mandatory at every size (the previous 16 000 MB, 15 449 usable, fit 10 + 1 or 5 + 2). Known unmodeled transient: a lockfile change on the default branch while threads hold the old deps (their inodes survive the checkout's git clean) costs one deps term until they release; the reserve (2.9 GB) covers a reinstall (2.3 GB) but not a reinstall plus a concurrent snapshot staging.
No build-user step starts beside a process the last one left behind, and no wait abandons a live process (pure decisions in residentRefresh.ts / residentStepReport.ts; applied in the Worker's buildUserRun and run). The shape that forced it, on a busy resident: main moved six times in twenty minutes with a lockfile change, so every cycle was a full install sharing the 1 vCPU with two thread runs; the install crossed the 5-min budget, the supervisor's kill lagged past the SDK's 30 s output grace, proc.output() rejected Process output did not complete within 330000ms, the cycle recorded refresh-failed: … — and npm install kept extracting into the checkout. The next cycle (no deps marker on disk — full install) ran git clean -fdx over it: warning: failed to remove node_modules/dayjs: Directory not empty on exactly the packages being written (git unlinks while iterating, an entry added underneath it survives the rmdir), checkout-update-failed, 600 s later the same again with a second orphan sharing the CPU — degraded and every run cold for as long as main kept moving. Reproduced negative: git clean -fdx on 300/1000/3000-entry directories inside the resident's own filesystem (ext4, git 2.34.1) succeeds — the leftovers are a concurrent writer, not the filesystem. Sweep: buildUserRun first runs killStaleBuildProcessesCommand(BUILD_USER, CHECKOUT_DIR) as root — steps on the checkout are strictly sequential, so a live worker1 process whose cwd is inside the checkout at step start is by definition a leftover (an abandoned wait, or a step orphaned by a Worker-only deploy resetting the DO while the container kept running); the scope matters because the same user's deps-store installs (item 59) run in their own scratch trees, in parallel, and are live work. Survivors are named on stdout (the Worker log shows WHAT was still running: <step>: killing stale worker1 processes under /workspace/checkout: <pid> <cmd>), then kill -KILL (SIGTERM would let npm keep writing under the clean), then a bounded wait until every matched pid is gone; a process that survives SIGKILL for 5 s fails the step as <step>-stale-sweep-failed — nothing starts beside it. Kill on abandoned wait: run catches the SDK's ProcessWaitTimeoutError from output(), kills the process (SIGKILL), waits up to KILL_EXIT_WAIT_MS for its exit, and returns abandonedWaitStepResult — timedOut: true, the observed exit status or -1 with "no exit status observed", "killed" in the report — so it is the step's own timeout (install-failed: exit 137 (timed out) …), never refresh-failed and never an interruption. Budget: the refresh install runs under REFRESH_INSTALL_TIMEOUT_MS (10 min, twice the build's): a full switchboard install is ~4 min on an idle vCPU and thread runs share it; a timed-out install spends the whole budget and leaves no deps, so the next cycle repeats it — a slow install is strictly cheaper than a killed one, and the cycle is background work (runs keep attaching to the last snapshot). The clean itself is unchanged (git clean -fdx, item 48): it was never the defect.
A timed-out install resumes, and every cycle failure is logged (the second half of item 56's incident, after its sweep: with the checkout swept clean of the orphan, every later cycle still found no deps marker, planned the conservative full clean, wiped node_modules and began the same cold npm install from zero — four cycles in a row, none finishing inside the budget while thread attaches paid their own full install on the same vCPU). (a) The deps-installing marker (INSTALLING_MARKER, the lockfile key) is written before the install step and removed after the deps key lands; a cycle that ends in between leaves it behind, and planRefresh then plans rebuild WITH install on a keep-deps clean — npm reconciles the partial tree to the lockfile, so an install longer than one budget converges over cycles instead of restarting. A marker for another key is stale evidence (full clean, as before); a landed deps key always wins (depsMatch is checked first). Safe for the hardlink invariant (item 48): threads only link deps whose key the deps marker vouches for, and no marker vouched for the partial tree. (b) onRefreshAlarm logs refresh: cycle failed — <classified reason> for every failure, not only StepErrors: a failure between steps (an SDK error, the markers, the snapshot) used to reach only the state entry, which the next cycle's failure overwrites — the install timeout that started the incident had left no trace once the follow-up cycle failed. Proof: src/execution/residentRefresh.test.ts::planRefresh: a timed-out install resumes instead of starting over plus the live row below.
A lockfile-diverged thread reconciles on top of the shared cache, never from an empty tree (pure decision planThreadDeps + threadDepsMechanism in src/execution/residentDepCache.ts, the reconcile cost kind and RECONCILE_DEPS_RATIO in src/execution/residentDiskBudget.ts, applied in the Worker's materializeThreadDeps and admitThreadDisk). Before: a thread whose committed lockfile differed from the warm checkout's key ran the install from nothing — install 1.94 GiB projected per tree, 5+ min on the 1 vCPU it shares with the refresh cycle and every other thread, so a burst of seven reviews within 13 minutes exhausted the disk (three of them diverged) and a ship run's /attach died before its install finished. Now the diverged tree is seeded from the checkout by the item-18 mechanism (hardlinked node_modules, mutable paths swapped) and the install runs on top: npm/pnpm replace a changed package with fresh inodes and can never write through a shared one (worker1-owned, write bits stripped), so the delta is all the tree costs in bytes and time. Admission projects such a tree as checkout + RECONCILE_DEPS_RATIO × deps (0.25 — a bound above the typical few-package delta, to be re-measured against du of live reconciled trees; unmeasured deps stay null, never a guess), and the residents page's headroom line counts lockfile-diverged (reconciling) trees at that cost. A diverged tree whose command table has no install is not seeded (deps:"none"): nothing could reconcile it. The attach answer names the mechanism reconcile whatever the seed's own mechanism was. Proof: src/execution/residentDepCache.test.ts::planThreadDeps …, src/execution/residentDiskBudget.test.ts::projecting …, web/src/pages/residents.test.ts::ResidentDetailPage::item 55: the Disk section …, plus the live row.
Dependencies live in one content-addressed store per resident, behind one primitive (pure module src/execution/residentDepsStore.ts, the view via depCacheScript's nodeModulesSrc in residentDepCache.ts; applied in the Worker's materializeDeps / linkDepsView / adoptCheckoutDeps / sweepDepsStore). Before this, deps were installed IN PLACE and PER CONSUMER — provisioning into the checkout, the refresh cycle into the checkout again on every lockfile change, a thread whose branch lockfile differed from the checkout's into its own 1.9 GiB tree (item 18's install mode). Three install sites, three budgets, none aware of the others; item 56's refresh spiral and item 58's ship attach that died mid-install both lived in the seams, and one resident ran 32 installs in a day for ~14 distinct lockfile keys because "older than main's lockfile" and "different from main's lockfile" were the same comparison. Store: /workspace/deps/<lockfileKey>/node_modules — the key is the item-48 hash of the committed lockfile (a pure function of the commit; anything not 64 lowercase hex is refused as a path segment), the entry holds the tree's top-level node_modules exactly as the install produced it, plus .complete (written LAST, inside the entry so the rename carries it) and .used (touched on every hit — the LRU clock). An entry is immutable once complete: its files are owner-read-only (deps-harden, chmod u-w before it becomes visible) so even worker1's build fails EACCES on a write through a shared inode instead of mutating every consumer's tree — Flyweight (GoF): many consumers, one tree by identity. The primitive materializeDeps(key, sha, installCmd, budget): planDepsMaterialization — complete entry → hit (touch .used); an install for that key already running in this incarnation → join its promise (never a second install; a complete entry beats a stale memo); else install: git clone --shared --no-checkout of the mirror into .scratch-<attempt> (objects via alternates, seconds), checkout --detach <sha>, chown to worker1, the repo's install command as worker1 in that scratch tree under REFRESH_INSTALL_TIMEOUT_MS, harden, then depsStoreCommitScript as root: mv scratch/node_modules → .staging-<key>-<attempt>/, mv staging → entry (atomic; a racer that finds the entry complete drops its staging and keeps the winner), touch .complete .used, rm -rf scratch. An install that produced no node_modules is a failure when the commit has a lockfile (there was something to install) and an empty entry when it has none (depsHardenScript, emptyOk for NO_LOCKFILE_KEY — the sha256 of the empty ls-tree listing): a repo whose install is a no-op (true, a terraform tree) gets an empty node_modules owned by the build user, and every consumer's view links an empty directory — without it a rebuild of such a repo dies at deps-harden (find: '…/node_modules': No such file or directory), and only dropping its install command brings it back. The install runs outside the mirror lock — it touches no consumer's tree — so a full install no longer holds every attach behind the mutex (the staging step item 47 asks for). Parallelism: distinct keys install concurrently up to nproc (depsInstallSemaphoreSize, read once per incarnation, unreadable → 1) — never a constant; on today's 1 vCPU that is 1 by arithmetic. Consumers: provisioning materializes main's key and links the checkout as a view (linkDepsView = depCacheScript with the entry as nodeModulesSrc + the item-18 tool-cache swap, whose copies are made writable again — chmod -R u+w — because cp copies the hardened mode bits); the refresh cycle on a lockfile change materializes the new key before taking the mirror lock, then under it cleans (-x, item 48) and re-links the view; a thread attach materializes its branch's key and links — a thread on a stale branch, a resident lagging main, or a key that flipped mid-life all hit an entry main built, and only a branch that genuinely changes deps installs (once, for every thread on that key). The attach answer's deps is hardlink/copy for every key now; install is history. Adoption: a checkout that holds node_modules for a key with no entry — a disk from before the store, or a fresh R2 restore (the snapshot carries the checkout's tree, not the store) — is adopted under the mirror lock (adoptCheckoutDeps: the same commit script with keepScratch, then the view): a rename plus hardlinks, seconds, same inodes before and after, so threads that linked from the checkout earlier are untouched. Eviction (sweepDepsStore, after every disk measurement): depsStoreListScript lists entries with complete flag, size and .used; protected = the checkout's key, every live binding's depsKey, every install in flight; planDepsEviction removes debris (incomplete, nothing in flight) first, then the coldest complete spares beyond DEPS_STORE_MAX_UNREFERENCED (1 — the item-55 sizing leaves room for about one ~2 GiB entry beyond the checkout's), and every .scratch-*/.staging-* leftover once nothing is in flight. Disk accounting (item 55): parts.deps is now the store's total (du charges the shared inodes to it, measured before the checkout and the threads), so checkout reads as history + tree + build output as before; the install projection (checkout + deps) over-projects when the store holds several entries — conservative, unchanged. Snapshot: the checkout archive EXCLUDES its top-level node_modules and each entry has its own archive — item 61's content-addressed snapshots (the store no longer "rebuilds lazily" on a wake: the warm key's entry is restored, or installed when it has no archive). /attach streams like /exec: heartbeat whitespace then ONE JSON document over HTTP 200, a refusal's status in the body (ResidentExecutor.attach reads it there; pre-validation 400/404 keep real statuses), so an attach that waits on an install cannot lose the connection the way a plain response did live (fetch failed at 272 s). Not here: pre-warming keys the mirror fetch sees, the resident as an R2 producer, CI as the producer (the key then gains the image digest, and the miss path tries R2 before the local installer).
Purging synthetic bindings (load-harness.md item 8; pure decision src/execution/bindingPurge.ts, imported by the Worker like residentDiskBudget.ts). Item 23 keeps a binding after eviction on purpose, so a load run that attaches fifty load:<runId>:<i> threads would leave fifty evicted rows on /residents/<slug> and in /debug threads forever. POST /debug {"op":"purge-bindings", "resource", "prefix"} (admin) deletes the EVICTED bindings whose threadKey starts with prefix and answers {purged: [keys], keptLive: [keys]}. selectBindingsToPurge refuses (HTTP 400, the reason named) a prefix that is not a whole thread-key namespace or longer (load:, load:r1: — never empty, never a bare partial namespace) and refuses the production namespaces outright (slack:, http:, mcp:, cli:), so a purge can only ever touch synthetic keys; a live binding under the prefix is never deleted, only listed as keptLive. Nothing on disk is touched: an evicted binding has no worktree and no pool user.
A restore is judged by the bytes still arriving, never abandoned to a clock (pure judgeRestoreProgress + RESTORE_POLL_MS / RESTORE_STALL_MS / RESTORE_MAX_MS in src/execution/residentRefresh.ts, applied by the Worker's restoreWithProgress in doHydrate). The shape that forced it: a ~2 GiB checkout restore completes in 481 s — the same snapshot had transferred in 47 s and 104 s earlier the same day — but a fixed 300 s budget (R2_TRANSFER_TIMEOUT_MS, item 7) has already declared it failed, a second hydrate runs rm -rf over the tree the first one is still filling, and the resident goes down(r2-restore-failed) on a disk with 11.5 GiB free until the watchdog rebuilds it from GitHub. The SDK's restore cannot be cancelled or observed, so the wake path observes the disk instead: every RESTORE_POLL_MS it samples du -xsk of the SDK's staging archive (restoreArchivePath: /var/backups/<backupId>.sqsh, where the download lands first — the target stays empty until extraction, which is how a probe watching only the target calls a healthy download stalled with 0 bytes in the target) PLUS the target (where extraction lands); while the sum grows it waits, however long; it fails as stalled when nothing has been written for RESTORE_STALL_MS (the clock runs from the start until the first byte, and a sample du could not take neither counts as growth nor resets it) or as capped past RESTORE_MAX_MS — ONE deadline for the whole hydrate, shared by the wait for a previous attempt's restore and both restores, so the worst-case restoring span is the cap itself, under the watchdog's 30-min stale-mid-flight window, and the verdict stays the wake path's. The two restores run sequentially (the SDK serializes backup operations on one queue; a concurrent pair only let the second one's clock run while it waited). Restores this incarnation started are tracked in pendingRestores; a later hydrate awaits them (bounded by what remains of that same deadline) BEFORE clean-before-restore, and goes down with the disk untouched if they will not settle — a clean never races a writer. On success the size and rate are logged (<what>: <GiB> in <s> (<MiB/s>)) so the constants can be revisited from evidence. The snapshot upload keeps its fixed budget (there is no target to observe from the resident's side). Proof: src/execution/residentRefresh.test.ts::judgeRestoreProgress … plus the live row. A restore that goes down takes the container with it: pendingRestores keeps the next HYDRATE off the directories, but a down resident's only exit is a rebuild, and provisioning owns the same directories — a restore the wake path has given up on lands into the checkout a rebuild has just cloned and linked; tar overwrites in place through the deps store's hardlinks (item 59), resetting every hardened entry file from 444 to the archive's 644 (.package-lock.json, the tool-cache copy the view does not share, alone stays 444) and rewriting freshly cloned tracked files. Same lockfile key, so identical bytes; a different key would be corrupted silently. So both down exits of the wake path — a stalled/capped restore, and the bounded wait for an earlier attempt's stream expiring — first stop the container (clearIncarnationMemos + stop(), the disk-full recovery's own move; the disk is ephemeral and the stream dies with it) and say so in the reason (… — container stopped so the transfer cannot land on a rebuild); and provisioning waits for pendingRestores (bounded by RESTORE_MAX_MS) before its clean-workspace, failing the provision as await-restores rather than cloning over a writer. The bytes do not travel through the Durable Object (pure backupTransferMode in src/execution/residentBackupTransfer.ts, applied in takeSnapshot and reported on GET /healthz). The SDK's localBucket: true is its documented LOCAL-DEVELOPMENT mode, in which the DO reads the archive from the R2 binding and pumps it to the container over the control RPC (and the reverse on upload), so a 128 MB isolate sits in the data path of every transfer — and a 1.16 GB checkout restore exceeds the isolate's memory while the progress probe's du shares the connection: the isolate is reset, the transfer dies with it (backup.restore … interrupted because the runtime changed), the state stays restoring with no down, and only a manual force-down + rebuild brings the resident back. In presigned mode the DO signs GET/PUT URLs and the container moves the bytes itself (downloadBackupParallel, resumable from prepareRestore's existingSize); the SDK's requirePresignedURLSupport reads exactly CLOUDFLARE_ACCOUNT_ID, BACKUP_BUCKET_NAME (wrangler vars: {{account}}, {{script}}-cache — the bucket name MUST equal the BACKUP_BUCKET binding's, which the offboard sweep deletes through) and R2_ACCESS_KEY_ID / R2_SECRET_ACCESS_KEY (secrets, deploy/secrets.manifest.json, an R2 API token scoped Object Read & Write to the one bucket). backupTransferMode(env) is presigned (localBucket: false) only when all four are non-blank; anything missing is local with the missing names — fail-closed to local mode, never to a broken snapshot — logged once per snapshot and shown on /healthz as backupTransfer (+ backupTransferMissing). The mode rides on the SDK handle, so a snapshot restores the way it was taken; records from before the switch keep restoring through the DO until the next cycle re-snapshots. Once the DO is out of the data path, item 61's du probe during a download is harmless. Content-addressed snapshots (pure CHECKOUT_SNAPSHOT_EXCLUDES, depsBackupStorageKey, DEPS_BACKUP_TTL_S, depsBackupsToDrop and the restore branch of planDepsMaterialization in residentDepsStore.ts; applied in takeSnapshot, backupDepsEntry, restoreDepsEntry, doHydrate, sweepDepsStore, teardown). Without this every refresh cycle re-uploads the whole checkout — hundreds of MB to over a GB, a dozen times a day on a busy repo — and every wake restores it whole, although since item 59 the node_modules inside it is a hardlink view of an immutable store entry: the one part that never changes once written. Now the checkout archive is taken with excludes: ["node_modules"] (the top-level view only; mksquashfs anchors a bare pattern at the archive root), and each entry is archived ONCE, right after deps-commit or deps-adopt (backupDepsEntry: the entry's node_modules dir, off the caller's critical path, 180-day TTL, recorded on the DO as resident:depsBackup:<key>; presigned mode only — local-bucket mode would put the DO in the data path of a deps-sized upload; the key is protected from eviction while the upload runs). A wake restores mirror + tree as before, then materializeDeps(stamp key) with the new restore backing: the archive comes down into a private .scratch-<attempt> (the SDK extracts wherever the handle's dir says, so the handle is re-pointed), judged by restoreWithProgress like every restore, then the same commit script an install ends with — staging, atomic rename, .complete LAST — so a partial download never becomes an entry; a restore that fails for any reason (expired, missing, stalled) drops the record and falls through to the installer, which records a fresh archive. A snapshot from before this change still carries node_modules and is adopted on restore as before. The wake decides by what the CHECKOUT holds, not by what the store holds: a checkout with node_modules after the adopt (an old snapshot) is done; one without gets materializeDeps — a hit when the entry survived on disk, else the archive, else the installer — and the view linked; the deps checkpoint is written ONLY with the view in place, so a wake whose materialization fails (or whose registry lookup fails) goes warm with the tree whole and the checkpoint unwritten, and the next refresh cycle's plan repairs it (no deps marker on disk — full install) — a cache miss is never a down. The materialization lives INSIDE the hydrate's one deadline (planWakeDepsBudget): the restore is judged against that deadline, the installer's budget is the smaller of its own and what remains, and under WAKE_DEPS_MIN_MS nothing starts — so the worst-case restoring span is still RESTORE_MAX_MS. Every other caller of the restore backing (an attach) gets min(budget, RESTORE_MAX_MS) from now, never longer than it would wait for an install. A presigned restore is a MOUNT, and the resident extracts it (pure residentRestoreExtract.ts: restoreMountDir, extractRestoreScript, unmountRestoreScript, unmountAllRestoresScript; applied by the Worker's restoreExtracted for the mirror, the checkout and a deps entry, and by the unmount-restores step before every clean-before-restore / clean-workspace). In presigned mode the SDK's restore does not extract the archive, it mounts it — squashfuse on the .sqsh under /var/backups/mounts/<id>_<ts>_<rand>/lower, fuse-overlayfs at the handle's dir with a writable upper beside it — and leaves the .sqsh in /var/backups (the SDK's own comment: unsquashfs extraction is its LOCAL-DEV path). A mount point breaks code written for one ext4 filesystem: rm -rf /workspace/mirror → Device or resource busy (the resident loops degraded on its wake path), chown -R after a restore → a copy-up of every inode (chown-failed: exit 143 (timed out)), du -x → the mirror and checkout measured as ~1 MiB, and every hardlink or mv between the checkout and the deps store crossing devices. So the handle is re-pointed at a staging mount beside the target (<target>.restore-<attempt>, still under /workspace where the SDK allows it), judged by bytes arriving like every restore, then extractRestoreScript puts a real tree in place — unsquashfs -n -no-xattrs straight from the downloaded .sqsh when the image has squashfs-tools (the Dockerfile installs it; the log says extract: unsquashfs), cp -a out of the mount while an older image lacks it (extract: cp) — unmounts the staging mount and then every squashfuse lower under the SDK's <backupId>_* dirs (found by the backup id, not by the overlay's options — fuse-overlayfs exposes no lowerdir= in /proc/mounts, so an unmount that looked there would leave the lowers mounted, each pinning its unlinked .sqsh), removes those dirs and the .sqsh, and renames the extracted tree in LAST, so a failure leaves neither a half target nor a mount. After a wake the disk is exactly what it was before presigned mode: one filesystem, hardlinks and renames work, du -x is right, chown -R is cheap. The mount model could later be turned to our advantage (a per-consumer fuse-overlayfs over one squashfuse lower is copy-on-write isolation without hardlinks) — a separate design. Eviction drops the archives of the entries it evicts (depsBackupsToDrop); teardown deletes every archive; a rebuild keeps them, so provisioning after a rebuild restores the warm key instead of installing it. POST /debug {"op":"deps-backups"} (read scope) lists the records. The disk budget's staging term (item 55) is left as it was — it still counts mirror + deps + checkout — because an entry archive can stage concurrently with a snapshot; it is conservative, and re-derived from measurements when the store's archive sizes are known.
Resident text is made safe at the seams, never at the surface. A resident's reason, error and summary strings are built from remote output (git, npm, the container's shell) and once carried other threads' identifiers, and they reach card titles, Slack replies, repo list and stored records. One helper, residentText (src/execution/residentText.ts: strip terminal control sequences, redact credential shapes, cap at RESIDENT_TEXT_CAP), is applied at three seams and nowhere else: on the resident at the write (setResidentState, recordRefreshError) and at the exit (json() and the heartbeat-streamed document, through sanitizeResidentBody, which rewrites only error, reason, summary and a stderr that mirrors error); on the bot at the parse (parseResidentBody, the /status probe's own body read, the admin client), permanently, because a reason stored by an older resident survives that resident's deploy and its rollbacks. The probe's state is validated against the lifecycle union plus not-onboarded/unknown, never echoed. The disk-pressure refusal names sizes, counts and keep tokens (DiskKeepWhy, with other as the caller's fallback) and never another thread's key or its free text. Step-report tails are stripped and redacted before the cut. /op stdout and stderr are redacted before the clip. Every card note is one line (oneLine), and the setup-failure close and reply are redacted. The same rule closes the GitHub half of the class: every client that slices a response body into an error redacts first. stdout from /exec is tool output and keeps its publish-time redaction; resident-minted ids and refs are outside the class.
Every attach and op answer carries its step trace (tracing.md item 19). AttachOk.trace and OpRunOk.trace — and a ThreadErr's, on a refusal — list the commands the Worker ran for that request — runOk's named steps, the op's own command, each mutex_wait — as offsets from the request's start with their exit and timeout, bounded (64 steps, 8 KB) and sanitized (src/execution/residentStepTrace.ts, shared with the bot). The collector is per request (an AsyncLocalStorage around attachThread and runOp), so two concurrent attaches never share steps and a refresh cycle records none. The bot grafts them under the span that made the call; nothing else reads them, and an older bot ignores the field. The same steps are the children of the resident's own resident.attach / resident.op root on its span log, which joins the bot's trace when the request carried one (tracing.md item 22).

Validation criteria

Criterion	Proof
62: `residentText` strips ANSI, redacts and caps, and is the identity on `runtime-replaced`, the `op-refused` message and the `disk-pressure:` prefix; `sanitizeResidentBody` rewrites only `error`/`reason`/`summary` and a mirrored `stderr`, passes non-objects through and leaves the input untouched; `residentState` admits the closed table only	`[unit]` `src/execution/residentText.test.ts::residentText::`, `::sanitizeResidentBody::`, `::residentState::*`
62: the bot's parse — a poisoned `/status` reason and an off-table state, a poisoned non-2xx probe body, a poisoned `/op` refusal/summary with a credential in stdout, a poisoned `/exec` error — never reach a probe result, an op result or a thrown message raw	`[unit]` `src/execution/resident.test.ts::resident text is made safe at the parse (item 62)::`; `src/core/residentAdmin.test.ts::makeResidentAdminClient sanitizes resident text at the parse (item 62)::`
62: the note a poisoned probe reason produces is one redacted line; a setup failure carrying remote text closes the card with one redacted line and redacts the reply	`[unit]` `src/execution/factory.test.ts::makeExecutor resident selection::a poisoned probe reason reaches the note as one redacted line (item 62)`; `src/core/dispatcher.test.ts::*::a setup failure carrying remote text closes the card with one redacted line and redacts the reply (resident-repos item 62)`
62: the disk-pressure refusal names counts and keep tokens, never a thread key or a free-text detail; a step-report tail is stripped and redacted before the cut	`[unit]` `src/execution/residentDiskBudget.test.ts::diskPressureReason — the refusal names free, reserve, projected, what was evicted and what was kept::full shape`; `src/execution/residentStepReport.test.ts::describeStepFailure::strips terminal control sequences and redacts credentials before keeping the tail (item 62)`
62: `oneLine` keeps the first non-empty line with collapsed whitespace and composes with `redactAndCap` for a title	`[unit]` `src/core/redact.test.ts::oneLine::*`
52: `detectCommands` — pnpm workspace → `pnpm install --frozen-lockfile` / no-op build / `pnpm test`; no root `package.json` → no install + no-op build/test (a lockfile alone is still `none`); `packageManager` beats the lockfile, unknown field falls through; npm with/without lockfile; yarn classic vs berry; bun lockfiles; unparseable `package.json` = no scripts; every note text	`[unit]` `src/core/repoToolchain.test.ts::detectCommands::*`
52: `githubRepoInspector` — tree + `package.json` reads with the bearer and the raw accept, deciding fields only; no `package.json` → one call; 404 names both causes, other statuses carry the code; no credential → no request; throwing fetch named; unparseable → `null`; ref URL-encoded	`[unit]` `src/execution/githubRepoInspect.test.ts::githubRepoInspector::*`
52: `repo onboard` — inspects `<slug>@<ref>`, posts the detected table, replies with toolchain + reasons + `— none` cells; no-package.json table lacks `install`; explicit flags win per key; uninspectable → npm table + the ⚠️ not-inspected line (also with no inspector wired) naming only the defaulted keys, or "nothing was assumed" when all three are explicit; an explicit `--build true` renders verbatim, not as "none"	`[unit]` `src/core/commands/repo.test.ts::repo onboard::item 52*`
52: settle — onboard + real rebuild settle, dry-run/list/offboard/reconfigure/test/build do not; warm after N polls at `SETTLE_POLL_MS`; down at install/build/test → reason + the fix commands, down elsewhere (snapshot/clone/timeout/no reason) → reason + a retry hint and never `repo reconfigure`; 404 / non-200 / throw / other state / admin unavailable each a sentence; still onboarding at `SETTLE_MAX_MS` → bounded ⏳ report	`[unit]` `src/core/commands/repo.test.ts::provisioning follow-up (item 52)::`
52: chat — the accepted onboard's outcome lands as a second reply in the thread (warm; down with the resident's reason + fix line); a dry-run rebuild posts no follow-up	`[unit]` `src/core/dispatcher.test.ts::registry chat commands in the fast-path chain…::every repo verb is the registry's`, `::item 52: a provisioning that fails after the acknowledgement`
52: admin client `status(resource)` → `GET /status?resource=` with the bearer	`[unit]` `src/core/residentAdmin.test.ts::status`
52: Live: `repo onboard <owner>/<pnpm workspace>` → reply names `Toolchain: pnpm`, table `pnpm install --frozen-lockfile` / none / `pnpm test`; the thread gets `✅ … is warm` (or a ❌ with the resident's own install/build error) within `SETTLE_MAX_MS`; `repo onboard <owner>/<repo with no package.json>` → `Toolchain: none`, no install, and reaches `warm`	`[agent]` Run both onboards in a channel the bot is in after the bot deploy; compare the replies to the wording above and `repo list` to the follow-up.
Typecheck passes	`[agent]` `cd deploy/cloudflare-resident && npm install && npm run typecheck` — exit 0.
Every route 401s without a bearer	`[agent]` For each of `/onboard /offboard /reconfigure /rebuild /residents /debug /status /attach /exec /read /write /op`: `curl -s -X POST https://<resident hostname>/<route>` → `401 {"error":"unauthorized"}`; `/rebuild` is admin-scope, so it also 401s with the operator token.
Operator token refused on admin routes	`[agent]` `curl -s -X POST -H "Authorization: Bearer $OPERATOR" .../onboard` (and `/offboard`, `/reconfigure`, `/debug`, plus GET `/residents`) → 401.
Admin token valid on operator routes (superset)	`[agent]` `curl -s -H "Authorization: Bearer $ADMIN" ".../status?resource=repo:<owner>/<name>"` of an onboarded resource → 200.
Onboard → provisioning → warm with recorded sha	`[agent]` `curl -s -X POST -H "Authorization: Bearer $ADMIN" -H "content-type: application/json" -d '{"resource":"repo:jshttp/vary","commands":{"test":"node -e \"require('"'"'./index.js'"'"')\"","build":"npm pack --dry-run","install":"npm install --no-audit --no-fund"},"defaultRef":"master","provisioningTimeoutMs":600000}' .../onboard` → 202 `{"state":"onboarding"}`; poll `/status` → `warm`; `/debug {"op":"info"}` shows `sha` equal to GitHub's `master` HEAD, a stamped snapshot `{ref, sha, lockfileHash}` with two backup ids, and `schedules.refresh == 1`.
Refresh with stable sha does not reinstall or re-snapshot	`[agent]` `POST /debug {"op":"refresh-now","resource":…}` (admin), wait ~10s, `/debug {"op":"info"}` → `lastRefreshAt` bumped, state `warm`, `snapshot.createdAt` and both backup ids UNCHANGED.
Natural (unforced) refresh cadence	`[agent]` With no debug calls for >10 min, `/debug {"op":"info"}` shows `lastRefreshAt` advancing on its own every ~600s and state `warm`.
Wake path: restoring → warm by restore, not re-clone	`[agent]` `POST /debug {"op":"stop-container"}` then `{"op":"refresh-now"}`; poll `/status` every ~4s → a visible `{"state":"restoring","reason":"rehydrating"}` window, then `warm`; `/debug {"op":"info"}` → `lastRestore.ms` well under provisioning time, `provisionedAt` and snapshot ids unchanged.
Stamp mismatch is refused → down(snapshot-stamp-mismatch)	`[agent]` A restored disk that fails the stamp verification → `/status` `{"state":"down","reason":"snapshot-stamp-mismatch: restored disk {…} != stamp {…}"}` and the refresh chain stops. The check verifies the mirror's rev-parse sha and the ls-tree lockfile key against the stamp on every restore; to trigger deliberately, corrupt/replace one of the two `backups/<id>/` R2 objects and force a wake.
Watchdog re-arms a dead chain and marks degraded(alarm-missed)	`[agent]` `POST /debug {"op":"kill-refresh"}` → `{"remaining":0}`; `POST /debug {"op":"run-watchdog"}` (the identical function the `*/10` cron runs) → `action:"rearmed"`; `/status` → `degraded` with reason `alarm-missed: refresh chain was dead; re-armed by watchdog`; within ~10s the re-armed refresh flips `warm` and `schedules.refresh == 1` again.
Provisioning failure fails closed with a named step	`[agent]` Onboard a nonexistent repo (e.g. `repo:acme/no-such-repo`) → within ~60s `/status` → `down` with `provision-failed at clone: … terminal prompts disabled` (anonymous git fails fast; no hang). The registry slot is KEPT (status is 200, not 404) until offboard. A too-small `provisioningTimeoutMs` (10s) on a real repo → `down` `provision-failed at install: … (timed out)` — per-step budgets fail closed.
Stuck onboarding → down(provision-timeout) + slot released	`[agent]` Fault injection: on a resident whose recorded deadline has passed, `POST /debug {"op":"force-onboarding"}` then `{"op":"run-watchdog"}` → result `action:"provision-timed-out"`, and `/status` → 404 (cap slot released; `/residents` count drops). The organic path — kill a mid-provision DO and wait out the deadline — is equivalent but needs a mid-flight redeploy; the injected check exercises the same `provisionTimedOut()`.
Mint without secrets is command-level, never a lifecycle flip	`[agent]` With GITHUB_APP_* secrets unset: `POST /debug {"op":"mint-token","resource":…}` → `{"ok":false,"error":"github-app-not-configured: …"}`; `/status` immediately after → still `{"state":"warm","reason":""}` and `lastRefreshError` untouched by the debug mint.
A watchdog-stamped `degraded` always runs the next cycle — never parks on a streak	`[agent]` On an idle resident (no attach for >1 h) with a dead chain: `POST /debug {"op":"kill-refresh"}` → `{"op":"run-watchdog"}` → `degraded(alarm-missed…)`; `/debug schedules` must show the re-armed refresh at ≤ 600 s (never 21600), and the resident returns to `warm` (or a refresh-produced `degraded` reason) within ~60 s (the failure shape this rules out: re-armed at 21600 s, still `degraded(alarm-missed)`, `lastRefreshAt` untouched).
A mint FAILURE (App configured, repo outside the installation) still refreshes: the cycle records `token-mint-failed (… fetching anonymously)` and returns to `warm` with `lastRefreshAt` advanced	`[agent]` On `repo:jshttp/vary` (public, not in the installation): `POST /debug {"op":"kill-refresh"}` → `{"op":"run-watchdog"}` → `/status` `degraded(alarm-missed…)`; within ~60s of the re-armed cycle `/status` → `warm` and `/debug info` shows `lastRefreshAt` within the last minute and `lastRefreshError` naming the mint (`token-mint-failed (command-level, fetching anonymously): … HTTP 422 …`); the failure shape this rules out: state stays `degraded(alarm-missed)` after the cycle ran and `lastRefreshAt` does not advance.
Repo-scoped mint: cross-repo access is refused	`[agent]` Mint + private clone: `repo onboard <owner>/<private repo>` provisions to `warm` — impossible anonymously. `[gap]` The SCOPE proof: (1) `POST /debug {"op":"mint-token","resource":"repo:<owner>/<name>"}` for an onboarded repo → `{"ok":true}`; (2) prove the scope by minting by hand — create the App JWT, then `curl -s -X POST -H "Authorization: Bearer <jwt>" -H "Accept: application/vnd.github+json" https://api.github.com/app/installations/$GITHUB_APP_INSTALLATION_ID/access_tokens -d '{"repositories":["<name>"]}'` → 201 whose `repositories` array lists ONLY `<name>`; (3) `curl -s -H "Authorization: Bearer <minted token>" https://api.github.com/repos/<owner>/<other-repo-in-installation>/contents/README.md` → 403/404, while the same call against `<owner>/<name>` → 200. Also verify a private-repo refresh works end-to-end (fetch via the credential file).
Offboard removes everything and reports it	`[agent]` `POST /offboard {"resource":…}` → 200 with `registryRemoved:true, schedulesCancelled:true, containerStopped:true, storageCleared:true, backupObjectsDeleted:4, r2ObjectsDeleted:<n>, errors:[]` — `backupObjectsDeleted` is 4 for a provisioned resident (2 snapshots × archive+metadata under `backups/<uuid>/`, deleted via the stored handles). A second offboard and a `/status` probe both → 404.
Duplicate onboard refused / cap enforced / invalid input 400s / wrong method 405s	`[agent]` Duplicate onboard → 409; an over-cap onboard → 429 naming the cap; `"resource":"Repo:BAD"` → 400 naming the format; `"resource":"repo:probe"` (no owner/name slash) → 400 naming the slug form; GET on `/onboard` → 405.
Deploy wake ping — the runner GETs the resident's `/healthz` once after its deploy	`[unit]` `src/deploy/plan.test.ts::planDeploy::each step spawns…` (the resident step's `wakeUrl`); `[agent]` `npm run cli -- deploy all --only resident` logs `resident: awake — GET https://<resident hostname>/healthz → HTTP 200` after `Current Version ID`.
Deploy preflight: idle fleet → allow	`[unit]` `deploy/cloudflare-resident/preflight.test.mjs::…decide()…::idle everywhere → allow, not forced`, `::no residents onboarded → allow`, `::force on an idle fleet is a plain allow, not flagged`.
Deploy preflight: busy → refuse, naming residents and counts	`[unit]` `preflight.test.mjs::…decide()…::busy → refuse, naming every busy resident with its count` (idle residents are NOT named; hint names `RESIDENT_DEPLOY_FORCE=1`). `[agent]` Start a long `/exec` (e.g. `sleep 120`) on an attached thread of an onboarded resident, then `RESIDENT_ADMIN_TOKEN=… npm run preflight` → exit 1, message lists that resident `(1 in flight)`; after the exec ends → exit 0.
Deploy preflight: fail closed on no token / unreachable / unknown resident	`[unit]` `preflight.test.mjs::…decide()…::unreachable / no token → refuse with the operator hint`, `::a resident whose live view failed is unknown → refuse (fail closed)`, `::a payload without per-resident inFlight (old Worker) is unknown → refuse`, `::a negative or non-integer inFlight (counter bug) is unknown, never idle → refuse`, `::malformed payload → refuse`; `readToken()::prefers admin, then operator, then read; never returns a blank`. `[agent]` (with `RESIDENT_BASE_URL=https://<resident hostname>` in the env — `deploy all` sets it from the profile; the script alone needs it) `RESIDENT_ADMIN_TOKEN= RESIDENT_OPERATOR_TOKEN= RESIDENT_READ_TOKEN= npm run preflight` → exit 1 naming the three env vars; `RESIDENT_ADMIN_TOKEN=bogus npm run preflight` → exit 1 quoting `HTTP 401`; `RESIDENT_ADMIN_TOKEN=x RESIDENT_BASE_URL=https://127.0.0.1:9 npm run preflight` → exit 1 `fetch failed`; `RESIDENT_BASE_URL= npm run preflight` → exit 2 naming `RESIDENT_BASE_URL` (release-and-deploy.md item 17).
Deploy preflight: a resident provisioning (`onboarding`, 0 in flight) → refuse naming the state; `refreshing`/`restoring` with 0 in flight → allow with a WARNING naming the state; settled states allow; force overrides a provisioning with the state named	`[unit]` `preflight.test.mjs::…decide()…::a resident provisioning (onboarding) → refuse…`, `::a resident refreshing or restoring with 0 in flight → allow with a WARNING…`, `::warm / degraded / down with 0 in flight are not mid-cycle → allow`, `::busy AND mid-cycle are both reported…`, `::an unrecognized lifecycle state is unknown → refuse…`, `::force overrides a provisioning…`. `[agent]` `npm run preflight` during a refresh window (`/debug refresh-now` first) → exit 0 with `WARNING: mid-cycle: repo:<owner>/<name> (refreshing)`; during an onboard → exit 1 quoting `provisioning: repo:… (onboarding)`.
Deploy preflight: force bypasses with a warning	`[unit]` `preflight.test.mjs::…decide()…::force overrides busy — allowed, flagged, and the warning still names the busy residents`, `::force overrides unreachable — allowed and flagged`. `[agent]` `RESIDENT_ADMIN_TOKEN= RESIDENT_DEPLOY_FORCE=1 npm run preflight` → exit 0 with `preflight WARNING` on stderr.
`/status` and `/residents` expose in-flight counts	`[agent]` `curl -s -H "Authorization: Bearer $ADMIN" ".../status?resource=repo:<owner>/<name>"` → `{"state":"warm","reason":"","inFlight":0}` at rest; `GET /residents` → top-level `inFlight` (a number when every resident answered, `null` with `inFlightUnknown` ≥ 1 otherwise) and `residents[].live.inFlight`; during a running `/exec` both read ≥ 1.
Engine state survives a redeploy	`[agent]` `env -u CLOUDFLARE_API_TOKEN npm run deploy` while a resident is warm → immediately after, `/status` → `warm` and `/debug {"op":"info"}` shows the same facts/snapshot and a pending refresh schedule (DO SQLite + schedule rows persist; the container disk does not need to — the next wake restores).
x-env-* headers are inert	`[agent]` Black-box through `/exec`: `curl -s -X POST -H "Authorization: Bearer $OPERATOR" -H "content-type: application/json" -H "x-env-EVIL: pwned" -H "x-env-GH_TOKEN: sneaky" .../exec -d '{"resource":"repo:jshttp/vary","threadKey":"slack:u4a","command":"env
Timing-safe comparison is constant-time	`[agent]` Not provable over the network; held by code review — read `timingSafeEqual` in `deploy/cloudflare-resident/worker.ts` and confirm it is still a byte-XOR accumulate over equal-length buffers with no early return other than the length check (the length is what any comparison leaks).
Unprivileged worker pool + no root escalation in the image	`[agent]` Through the engine: provisioning/refresh run install/build via `su -s /bin/bash worker1 -c …` (a root-owned command table executing as uid 2001) — a failing `su` would fail provisioning. Direct `/exec` proof: `command:"id"` on an attached thread → `uid=2002(worker2) gid=2002(worker2) groups=2002(worker2)` — every thread exec is privilege-dropped to its own pool user.
attach materializes a warm worktree WITHOUT install	`[agent]` `curl -s -X POST -H "Authorization: Bearer $OPERATOR" -H "content-type: application/json" -d '{"resource":"repo:jshttp/vary","threadKey":"slack:u4a","refHint":"master"}' .../attach` → 200 `{"workspace":"/workspace/threads/slack-u4a-27aa257c/master","ref":"master","sha":"1220b9c4…","user":"worker2","reconciled":false,"recreated":true,"deps":"hardlink","credentials":"unavailable","credentialsError":"github-app-not-configured: …","mutexWaitMs":0,"attachMs":5497}` — deps present with NO install (`deps:"hardlink"`, `reconciled:false`; `attachMs` a fraction of provisioning-with-install), proven by an immediate `/exec` `ls node_modules` listing packages and `stat` showing hardlinked file inodes still `worker1 664` under thread-user-owned dirs.
distinct worktree + user per thread on the same ref	`[agent]` Same attach body with `"threadKey":"slack:u4b"` → `{"workspace":"/workspace/threads/slack-u4b-aa489653/master","user":"worker3",…}` — different path, different OS user, same sha.
OS-layer cross-thread isolation	`[agent]` threadA `/exec` `echo … > notes-a.txt && chmod 600 notes-a.txt` (file lands `worker2 600`); threadB `/exec` `cat /workspace/threads/slack-u4a-27aa257c/master/notes-a.txt` → `Permission denied` exit 1; `ls /workspace/threads/slack-u4a-27aa257c/` → `Permission denied` exit 2 (700 thread dir denies traversal, not just the file).
engine plumbing unreadable from threads	`[agent]` threadB `/exec` `cat /workspace/mirror/config` → `Permission denied` (mirror is `root:worker1 750`; worker1 keeps group read for the checkout's fetch-from-mirror); `cat /workspace/.resident/git-credentials; ls /workspace/.resident/` → both `Permission denied` (700 root).
shared dep inodes are tamper-proof	`[agent]` threadA `/exec` `echo poison >> node_modules/.package-lock.json` → `Permission denied` while `stat` shows the inode `worker1 worker1 664` — hardlink materialization shares file inodes read-only; a thread can replace entries in its own dirs but cannot mutate bytes the warm checkout (or another thread) sees.
no token content in the process list	`[agent]` Start a long command on thread A (`/exec` `sleep 12`); from thread B, mid-flight, `/exec` `ps auxe \| grep -c "x-acces[s]-token"` and `ps aux \| grep -c <token prefix>` → `0` and `0` — nothing token-shaped in argv or visible env (commands run via argv-only `su`, tokens travel only by 0600 file). Residual (info disclosure, accepted): `su` demotes UID/GID but does not namespace `/proc`, so thread A's non-secret command line/cwd/thread-name ARE visible to thread B via `ps` — only token material is protected, not process metadata; `/proc hidepid=2` (or a PID namespace) would close the visibility. `[gap]` The credential FILE (`<wt>/.git/github-credentials`, `credentials:"ok"`; without `GITHUB_APP_*` on the Worker every attach returns `credentials:"unavailable"` + `credentialsError`, the designed command-level shape): attach → `credentials:"ok"`, `/exec` `stat -c "%U %a" .git/github-credentials` → `<threaduser> 600`, peer-thread `cat` → denied, `git ls-remote origin` succeeds via the helper.
20: credential refresh decision — missing / empty → refresh; a token within `CREDENTIAL_EXPIRY_MARGIN_MS` of expiry → refresh (`expiring`), a long-lived token → no refresh, keyed on the token's own expiry not the file's age; read-only binding → never	`[unit]` `src/execution/residentCredentials.test.ts::shouldRefreshThreadCredentials…` (every branch, the exact expiry boundary, precedence, read-only short-circuit).
20: a near-expiry token (3 min left) refreshes before the first writable exec though the file was just written; the margin covers the longest single exec	`[unit]` `src/execution/residentCredentials.test.ts::expiry-driven refresh…` and `::the expiry margin covers the longest single exec…` (`CREDENTIAL_EXPIRY_MARGIN_MS` ≥ `BASH_TIMEOUT_MAX_MS`, < 60 min).
20: a binding predating `tokenExpiresAtMs` (null) still refreshes via the old file-age `stale` rule (>45 min or unknown write time); a fresh null-expiry file → no refresh	`[unit]` `src/execution/residentCredentials.test.ts::old bindings that predate tokenExpiresAtMs fall back to the file-age stale rule…`.
20: a coding run whose first push succeeded can push again >60 min after attach — the token is refreshed at exec	`[agent]` On an onboarded private repo, attach a writable thread (`credentials:"ok"`), `/exec` `git ls-remote origin` → succeeds; wait past the token's expiry margin with no exec, then `/exec` `git ls-remote origin` again → succeeds (no 401), and the Worker log shows `credentials: refreshed for <threadKey> (expiring)`; `/debug threads` shows the binding's `credentialsWrittenAt` advanced past the attach time.
20: a fresh attach that reuses a cached near-expiry token refreshes before the first git write, not 45 min later	`[human]` Attach thread A on a repo, let its token age to within `CREDENTIAL_EXPIRY_MARGIN_MS` of expiry, then attach thread B on the SAME repo: B's first writable `/exec` logs `credentials: refreshed for <B> (expiring)` and `git ls-remote origin` succeeds — B never runs on A's near-dead token.
20: an `empty` refresh (a 401'd, git-erased token) mints fresh — the per-slug cache is bypassed so the same rejected token is never rewritten into the file	`[agent]` A resident coding run whose token is rejected (e.g. after a scope change): the first git write 401s, `store` blanks the file, and the NEXT writable exec writes a genuinely NEW token (a second `access_tokens` mint in the resident logs), not the same one — the push then succeeds.
20: an emptied credential file (git's erase after a 401) is rewritten on the next exec	`[agent]` On a writable thread with `credentials:"ok"`: `/exec` `: > .git/github-credentials && stat -c %s .git/github-credentials` → `0`; then any `/exec` (e.g. `stat -c "%s %U %a" .git/github-credentials`) → a non-zero size, `<threaduser> 600`, Worker log `credentials: refreshed for <threadKey> (empty)`, and `git ls-remote origin` succeeds via the helper. A read-only thread (`agent:review`) doing the same → the file stays absent (`test -f` fails), no refresh log line.
re-attach is idempotent; untracked scratch survives	`[agent]` Re-attach `slack:u4a` (no refHint — the binding is sticky) → same `workspace`, `recreated:false`, `reconciled:false`, `deps:"none"`; `/exec` shows `notes-a.txt` intact and `git status --porcelain -uno` empty.
18: build outputs are rebuildable in a thread tree	`[agent]` On a fresh-thread attach of a repo with a build step, `/exec npm run build` exits 0 and `stat -c '%U %h' dist/index.js` shows the thread user with link count 1. The failure shape this rules out: `npm run build` → `EACCES … dist/…` with `dist/index.js` `worker1 644`, link count 3 (a hardlinked build output).
18: tool-managed paths inside node_modules are rebuildable	`[agent]` on a fresh attach, `stat -c '%U %h' node_modules/.vite` (or `.cache`) shows the thread user with link count 1 (a real copy) while `stat -c '%U %h' node_modules/react/package.json` shows `worker1` with link count > 1 (still hardlinked); `/exec npx vitest run` rewrites `node_modules/.vite/vitest/results.json` and exits 0; `echo poison >> node_modules/react/package.json` is still `Permission denied`.
dirty tracked state → wipe + recreate	`[agent]` `/exec` `echo "// dirt" >> index.js` (status `M index.js`), then re-attach → `recreated:true`, `deps:"hardlink"` (rematerialized); `/exec` `git status --porcelain` empty and `notes-a.txt` gone — a fresh clean tree.
hostile inputs are refused before anything derives	`[agent]` `refHint:"../evil"` → 400 naming the ref pattern; `threadKey:"foo;rm"` → 400 naming the threadKey pattern; `/debug {"op":"threads"}` immediately after shows no new binding — nothing created.
nonexistent ref → named error, nothing bound	`[agent]` `refHint:"no-such-branch-again"` on a fresh threadKey → `400 {"error":"unknown-ref: ref \"no-such-branch-again\" does not resolve in the mirror (even after a fetch)"}` and `/debug threads` shows the fresh allocation rolled back (no binding, no leaked pool user).
needs-ref contract names the default branch	`[agent]` Attach on a fresh threadKey with no refHint → `409 {"error":"needs-ref: this thread has no ref binding yet — supply refHint","needs":"ref","defaultRef":"<default branch>"}` (the bot-side bind-by-default flow is item 30).
mirror mutex serializes refresh vs attach	`[agent]` `POST /debug {"op":"refresh-now"}` then an immediate attach on a fresh thread → attach completes with `mutexWaitMs` > 0 (it queued behind the refresh's mirror fetch); `/debug info` right after shows the refresh also completed (`lastRefreshAt` bumped, state `warm`). Both finish; neither fails.
/read /write confined to the worktree, as the thread user	`[agent]` `/write {"path":"docs/u4-note.txt","content":"…"}` → `{"ok":true,"bytes":25}`; `/read` returns the exact content; `/exec stat` shows the file `worker2`-owned (written privilege-dropped, content staged via file — never argv). Escapes: `path:"../../mirror/config"`, `path:"/workspace/.resident/git-credentials"`, write to `"../../checkout/pwned.txt"` → all `400 path-escape`.
unattached/evicted/recycled threads get named errors	`[agent]` `/exec` on a never-attached threadKey → in-body `{"error":"not-attached: …","needs":"attach","exitCode":127}`; on an evicted thread → `{"error":"evicted: …","needs":"attach"}`; after a container stop+wake (disk recycled, worktrees gone) → `{"error":"worktree-missing: …","needs":"attach"}`, and the follow-up attach recreates the same workspace (`recreated:true`, `deps:"hardlink"`).
Hot path: repeated `/exec` on a warm attached thread is materially faster than before the memo round, and correctness held — a mid-run resident deploy still surfaces as `runtime-replaced` (item 43) and the next call re-probes	`[agent]` On a warm resident thread, time 20 × `POST /exec {command:"true"}` back-to-back before and after the deploy carrying this change; expect a clear median drop (the removed work: 1 registry hop + 3 storage gets + 2 container forks per call). Then `POST /debug {"op":"stop-container"}` (or a resident deploy) mid-thread → the next `/exec` answers `runtime-replaced`/`worktree-missing` per items 21/43, never a stale success.
Exec/op/read streams are capped at the source: exit code preserved (0 and non-zero), stdout/stderr separate, each stream cut at capBytes, cd failure surfaces as before, arbitrary command text (quotes/&&/subshell/trailing comment) unchanged, BOTH temp files cleaned on exit	`[unit]` `src/execution/residentExecWrap.test.ts::capWrappedCommand (run under real bash)::*` — the generated script runs under a real bash in the test
Timeout salvage: a SIGKILL mid-command leaves the fixed files; the recovery script returns the capped pre-kill output and removes them; recovery over missing files is a clean no-op	`[unit]` same suite, `::fixed-file mode + recovery…` (a real SIGKILL from the test harness) and `::recovery over missing files…`
`/exec` command bound: a command over 64 000 chars → pre-validation 400 naming the limit (a normal client Error bot-side, never infra); a long-but-legitimate command (e.g. a ~20 KB heredoc) runs	`[unit]` `src/execution/resident.test.ts::…command-too-long rejection…`, `::two consecutive command-too-long rejections don't increment the infra counter…` (fixtures at the 64 000 wording). `[agent]` on a warm thread: `/exec` a 65 001-char command → 400 `at most 64000 chars`; a 20 KB heredoc writing a file → exit 0 and the file exists.
Live: a thread `/exec` emitting ~10 MB (e.g. `yes … \| head -c 10000000`) answers with the usual 100k-char truncated streams and a clean exit — and the resident stays up (no `runtime-replaced` from DO memory pressure)	`[agent]` Run it on a warm resident thread; expect `truncated: true`, exit 0, and `/status` still `warm`.
Hot path: `/exec` on a never-onboarded resource answers a named refusal (503 `not-serviceable: resident is not provisioned yet…`), not a hang and not a 404 — the registry hop is gone from the data plane by design	`[agent]` `POST /exec {resource:"repo:acme/nonexistent", threadKey:"t", command:"true"}` with the operator bearer → in-body error naming not-provisioned; `/attach` on the same resource still answers `404 … is not onboarded`.
Sweep clean-idle release: a live binding idle ≥1h with a clean tree and no op in flight is released before its TTL; a dirty one is kept	`[agent]` `POST /debug {"op":"sweep-now"}` on a resident with idle review bindings → `evicted` lists them; a coding thread with an uncommitted edit stays live; `/residents` `live.threads` confirms.
Image reconcile: after a deploy that changed the pool, the running container is stopped within one refresh cycle (or on the next attach) and comes back on the new image	`[agent]` After a deploy that changed the pool, `wrangler tail switchboard-resident` shows `image-stale (refresh
Idle sleep: with no attach for 1h and clean trees, `/debug {"op":"schedules"}` shows the refresh ≈6h out, `idleSince` set in `/residents`; a new review attaches after a visible refresh (`lastRefreshAt` advances) and the schedule is back to 10 min	`[agent]` (needs a quiet hour) `/debug {"op":"info"}` before/after; dash "idle since" row.
Dash shows idle mode	`[unit]` `web/src/pages/residents.test.ts::ResidentDetailPage::shows idle mode when the resident parked its refresh, 'awake' otherwise`
Read token: `GET /residents` → 200; `POST /debug {"op":"info"}` / `schedules` / `threads` → 200; `POST /debug {"op":"stop-container"}` (and `sweep-now`, `refresh-now`, …) → 403; `POST /onboard`, `/rebuild`, `/attach`, `/status` → 401	`[agent]` `curl -s -H "Authorization: Bearer $READ" …` per route/op.
48: refresh plan — unchanged sha → `unchanged`; key unchanged → `rebuild` without install on a `keep-deps` clean; key changed or no deps marker → full clean + install, unless the `deps-installing` marker names the same key — then `rebuild` WITH install on a `keep-deps` clean (a resumed install, item 57), a marker for another key being stale evidence (full clean), and a landed deps key always winning over a leftover installing marker; HEAD + built + deps all matching → `reuse`; a matching built marker with a different HEAD, or a matching HEAD with an unfinished build, never reuses; keep-deps clean is exactly `git clean -fdx -e node_modules`, full is `git clean -fdx`	`[unit]` `src/execution/residentRefresh.test.ts::planRefresh …::`, `::checkoutUpdateCommand::` (red-verified: module absent)
48: Live: a default-branch merge that does not touch `package-lock.json` → `wrangler tail switchboard-resident` shows `refresh: … rebuild (lockfile unchanged — deps kept, build only)` and `… done in <ms>` in the seconds range (well under 60 s), `/residents` `lastRefreshAt` advances with state `warm`; a merge that changes the lockfile → `(lockfile changed — full install)`	`[agent]` `wrangler tail switchboard-resident` across a default-branch merge, then `/debug info` → `warm`, `lastRefreshAt` advanced, `inFlight 0`. `[gap]` The lockfile-changed half needs a default-branch merge that touches `package-lock.json`.
48: Live: the first refresh after a resident-Worker deploy holds/reaches `warm` fast — `/status` `warm` within one cycle, no full reinstall in the tail; a deploy landing mid-rebuild is followed by `reuse (checkout, deps and build already materialized …)` or a keep-deps rebuild, never a second full install	`[agent]` `wrangler tail switchboard-resident` across a resident-Worker deploy (`npm run deploy` in `deploy/cloudflare-resident`), then poll `/status` through the next alarm: `restoring(rehydrating)` → `refreshing` → `warm` within one cycle with no reinstall in the tail. `[gap]` The `reuse` plan needs a deploy to land between build completion and snapshot.
44 (the interruptible revision): the preflight decision — `onboarding` (even with 0 in flight) refuses by name with the force hint; `refreshing`/`restoring` with 0 in flight allow with `preflight ok: … WARNING: mid-cycle: <resource> (<state>)` and never `REFUSED`, listed under `interrupting`; a run in flight refuses whatever the state; force over a provisioning is allowed and flagged; busy is judged on `runsInFlight` when the live view carries it (a refreshing resident whose only in-flight work is its own cycle allows with the warning; an impossible `runsInFlight` is unknown) and on `inFlight` for a Worker without the field	`[unit]` `deploy/cloudflare-resident/preflight.test.mjs::resident deploy preflight — decide()::a resident provisioning (onboarding) → refuse, naming the state, even with 0 in flight: an isolate swap fails the provision and only a rebuild recovers it` + `::a resident refreshing or restoring with 0 in flight → allow with a WARNING naming the state: the cycle re-arms in 45 s after the swap (item 44), a restore is retried by the next hydrate (item 61)` + `::busy AND mid-cycle are both reported — neither shadows the other; runs in flight refuse even when the cycle alone would only warn` + `::force overrides a provisioning — allowed, flagged, and the warning names the state` + `::\`runsInFlight` decides busy when the live view carries it: a refreshing resident whose only in-flight work is its own cycle (inFlight 1, runsInFlight 0) → allow with the WARNING; a run in flight → refuse; a Worker without the field → inFlight decides as before`(red-verified:`interrupting`absent, refreshing refused; then`runsInFlight` ignored)
44: a step killed from outside is `refresh-interrupted: <step> …` (exit 143 / `Session terminated` / `SIGTERM`, any of install, build, checkout-update); an ordinary failure stays `<step>-failed: …` verbatim; a step the cycle itself timed out (`(timed out)`) is never an interruption; a bare "killed" in compiler output is not either	`[unit]` `src/execution/residentRefresh.test.ts::classifyRefreshFailure …`
44: re-arm delay — `interrupted` and `image-stale-restart` → `INTERRUPTED_REARM_S` (30–60 s band), `normal` → the cadence, `idle` → the idle interval; more than `INTERRUPTED_REARM_MAX_CONSECUTIVE` (3) consecutive interruptions → the cadence again (a step whose output chronically carries the kill signature cannot hot-loop at 45 s); the cap never applies to image-stale restarts	`[unit]` `src/execution/residentRefresh.test.ts::nextRefreshDelayS…` (7 tests)
44: `refresh-interrupted` never counts toward the park streak (the entry gate treats it like a watchdog stamp: streak reset, cycle runs)	`[unit]` the classifier pins the `refresh-interrupted:` prefix the Worker's `NON_EVIDENCE_REASON` gate keys on (`::classifyRefreshFailure::exit 143 during build → refresh-interrupted, reason prefixed for the streak gate`); the gate itself is Worker code without a harness — `[agent]` the live recipe below: after recovery the resident is `warm` and the next `/debug {"op":"info"}` shows no degraded streak
44: Live: a container replacement landing mid-refresh → visible `refresh-interrupted: …` and `warm` within ~90 s, no streak	`[agent]` What can interrupt: only a CONTAINER replacement — an image-changing deploy or `stop-container`; a Worker-only `wrangler deploy` swaps the DO isolates but the container and its processes survive, so it interrupts nothing (attempted live: the refresh completed untouched under one). The classifier lives in `onRefreshAlarm`, so the interrupted step must be a REFRESH cycle, not `POST /rebuild` (that is the provision path: `onboarding` → `warm`, never `refresh-interrupted`), and `planRefresh` only rebuilds when the default branch has MOVED since the resident's `sha` (`unchanged` otherwise). On a repo with no build script the build step is a no-op (~26 ms observed) and the kill lands in `install` (lockfile-change cycles) or `snapshot` (~30 s, every cycle) — a repo with a real build/install exercises the build path. Recipe: land a commit on the default branch (any merge), `GET /residents` → `live.sha` still the old sha, then `POST /debug {"op":"refresh-now","resource":"repo:<owner>/<name>"}` (admin) and, as soon as `/status` leaves `warm` (`refreshing`), `POST /debug {"op":"stop-container"}` (kills the running step; refuse if other execs are in flight — it kills them too); poll `/status` every 5 s: `refresh-interrupted: <step> …` visible as the `degraded` reason or — if a concurrent attach flips the state to `restoring · rehydrating` — in `/residents` `live.lastRefreshError`, then `warm` within ~90 s (45 s re-arm + one cycle), versus waiting for the 600 s alarm; no park streak, and the next cycle's `refresh: … reuse` / keep-deps line in `wrangler tail`.
44/#216: a step killed from outside is `refresh-interrupted: <step> …` (exit 143 / `Session terminated` / `SIGTERM`, any of install, build, checkout-update); an ordinary failure stays `<step>-failed: …` verbatim; a step the cycle itself timed out (`(timed out)`) is never an interruption; a bare "killed" in compiler output is not either	`[unit]` `src/execution/residentRefresh.test.ts::classifyRefreshFailure (a build SIGTERM…`
Race fixes (16c) hold: wake-fetch never waits past `ATTACH_MUTEX_WAIT_MS`; facts written after the wake-fetch carry the concurrent cycle's `lastRefreshAt`; a re-attach during a sweep clean check keeps its tree	`[code review]` typecheck-only Worker; behaviors are timing-dependent and reviewed by construction (single re-read before every write/evict; bounded lock waits).
Stuck mid-flight marker normalizes: `refreshing` older than 30 min with nothing running → watchdog `degraded(stale-mid-flight)` → next cycle → `warm`; the idle gate never parks from a non-warm state	`[agent]` `POST /debug {"op":"run-watchdog"}` on a resident showing stale `refreshing` → `action:"rearmed"`, reason `stale-mid-flight…`; `/status` within a minute → `warm`.
Sweep releases idle bindings on a SLEEPING resident (tree already gone)	`[agent]` Dash for a slept resident with idle bindings after the next hourly sweep → the bindings show `evicted`; `/residents` `live.threads` users `""`.
Detach: `release("always")` POSTs `/detach` `force:true`; `"if-clean"` `force:false` and a kept (dirty) tree reports `released:false` + reason; non-2xx / transport failure → `released:false` with a legible reason, never a throw	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.release …::*` (red-verified)
16a: force-detach decision — nothing in flight → proceed; non-force + in flight → the unchanged `busy` refusal; force + in flight → kill as the binding's pool user; force never kills as a non-pool user (root, empty, `worker1`, out-of-range) and names why	`[unit]` `src/execution/residentDetach.test.ts::planForceDetach …::*`
16a: Live: Kill (hard stop) on a coding run mid-`sleep 300` from `/runs` → within 10 s `/residents` shows the thread `user:""`, `evicted:true`; bot log `[release] … released`; `wrangler tail switchboard-resident` shows `detach: force — killed workerN's processes for <threadKey> (1 op(s) were in flight)` and the abandoned `/exec` completes with exit 137	`[agent]` Attach `wrangler tail switchboard-resident` first (bot container stdout is not ingested, so the bot's `[release]` line is only visible in a local run); start `agent:coding … run sleep 300` on an onboarded repo, confirm `/residents` shows the thread with `inFlight 1`, Kill it from `/runs`, poll `/residents` for 10 s → `user:""`, `evicted:true`, `inFlight 0`. The failure shape this rules out: `busy: 1 operation(s) in flight — kept`, the user held until the hourly sweep.
Dispatcher releases the workspace in the run `finally`: `if-clean` for coding, `always` for the read-only review agent; a throwing release never fails the run	`[unit]` `src/core/dispatcher.test.ts::releases the executor's workspace when the run ends …`, `::a release that throws never fails the run …` (red-verified)
Live: after a review run on an onboarded repo, that thread's binding is evicted and its user free; a coding run with uncommitted work keeps its tree (`kept (dirty: …)` in the bot log); `wrangler tail switchboard-resident` shows the `/detach`	`[agent]` Post `agent:review` on a PR of an onboarded repo; when the card completes, the resident detail shows that thread's row `evicted` within seconds of its last attach, user `""`. Then `agent:coding` that leaves an uncommitted edit → its row stays live and the bot log reads `kept (dirty: …)`.
inactivity eviction releases the user, keeps the binding	`[agent]` `POST /debug {"op":"backdate-thread","threadKey":"slack:u4d","days":10}` then `{"op":"sweep-now"}` → `{"evicted":["slack:u4c","slack:u4d"],"kept":2}`; `/debug threads` shows the evicted bindings RETAINED with `evicted:true`, `user:""`, ref intact; `/exec ls /workspace/threads/` from a live thread shows the evicted dirs gone. Re-attach of `slack:u4d` (no refHint) → recreated on the sticky ref with a fresh pool user. (The unforced path is the same `onWorktreeSweep` on its self-rescheduled hourly schedule.)
wake path unaffected by the perms hardening	`[agent]` `{"op":"stop-container"}` + `{"op":"refresh-now"}` → restore completes (`lastRestore {at, ms}`), state `warm`, and `/exec stat /workspace/mirror` from a re-attached thread → `root worker1 750` (the post-restore `ensureGitSetup` re-locks the restored mirror).
user-pool exhaustion is a named refusal	`[agent]` Attach 16 distinct threadKeys on one resident (no detach); a 17th run's status frame reads `resident attach failed … user-pool-exhausted: all 16 thread users are allocated` and it falls back cold. Refusal shape by code: `allocateThreadUser` returns `429 {"error":"user-pool-exhausted: all 16 thread users are allocated; …"}` from a storage-only section that is atomic under the DO input gate, so two racing attaches cannot double-allocate a user.
warm probe → ResidentExecutor; no ctx.repo → per-thread with ZERO probes	`[unit]` `src/execution/factory.test.ts::makeExecutor resident selection::warm probe → ResidentExecutor, attached on open…`, `::ctx.repo undefined → per-thread path with ZERO probe calls (total input contract)`
named fallback carries state+reason verbatim; not-warm never cached	`[unit]` `src/execution/factory.test.ts::…::not-warm probe → fallback carrying state and reason verbatim; no attach…`, `::not-warm states are NOT cached — the next dispatch probes again`
serviceable non-warm states attach — `refreshing`, and `degraded` for intact-checkout reasons → `ResidentExecutor` with the informational note; `degraded` by an in-rebuild failure stays cold; a mirror-busy attach during a refresh falls back with the named attach-failed note; `onboarding`/`restoring`/`down` fall back with the verbatim state/reason note and no attach	`[unit]` `src/execution/residentState.test.ts::isServiceable::*` (reason-class matrix), `src/execution/factory.test.ts::…::refreshing probe → ResidentExecutor WITH an informational note…`, `::degraded probe → ResidentExecutor, note carries the reason`, `::degraded by an in-rebuild failure (install-failed) → per-thread fallback, no attach`, `::refreshing probe then a mirror-busy attach (503) → per-thread fallback…`, `::%s probe → per-thread fallback with the verbatim state/reason note; no attach` (×3)
Live: a run dispatched while the resident is `refreshing` attaches to the last snapshot instead of going cold	`[agent]` With the residents dash showing `state refreshing` (a default-branch advance within the last minutes, or `POST /debug {"op":"refresh-now"}` as admin), post a review run in a channel the bot is in → the status card reads `resident refreshing — attached to the last snapshot` (not `— using fresh sandbox`) and the resident detail shows a new thread worktree for that thread.
outage circuit breaker — one probe per window, transport-only	`[unit]` `src/execution/factory.test.ts::…::probe transport failure → named fallback + negative cache (second dispatch makes no fetch)`; `src/execution/resident.test.ts::ResidentExecutor.probeStatus` (transport flag: network=true, HTTP-level=false)
not-onboarded repo runs on the per-thread path with a named cold-fallback note pointing at onboarding (the fall-through is visible)	`[unit]` `src/execution/factory.test.ts::…::404 not-onboarded → per-thread path with a named cold-fallback note pointing at onboarding`
exec parses streamed heartbeats + in-body errors; re-attach exactly once	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.exec` (streamed parse, non-zero-exit-as-result, needs:attach recovery, second-failure legibility, no blind retry)
read/write round-trip, 409 re-attach, path-escape legible	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.readFile / writeFile`
43: one `runtime-replaced` on `/exec` → re-attach once, the command is NOT re-run, the named outcome reaches the model as output	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.exec::runtime-replaced (a deploy mid-command) re-attaches once and returns the outcome as tool text — the command is NEVER re-run` (asserts the call sequence is exactly `/exec`, `/attach`)
43: two `runtime-replaced` in a row (no success between) → `ExecInfraError`; a success between resets the streak	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.exec::a second runtime-replaced in a row (no success between) is an ExecInfraError — a flapping resident, not one deploy`, `::a successful op between two runtime-replaced outcomes resets the streak (each is a one-off deploy)`
43: one deploy never counts toward the dead-sandbox breaker	`[unit]` `src/execution/resident.test.ts::ResidentExecutor infra classification through ExecHealthTracker …::a single runtime-replaced (one deploy) never counts toward fail-fast`
43: idempotent routes retry once after re-attach; a second `runtime-replaced` is infra	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.readFile / writeFile::a 409 runtime-replaced on read (idempotent) re-attaches once and retries; a second one is infra`, `::runtime-replaced then needs:"attach" on the retried read (deploy + evicted worktree) is the precise worktree-unavailable infra error`
43: the resident answers `reason:"runtime-replaced"` for a stale process handle / runtime_replaced interruption on exec, read, and write, and never re-issues a command the SDK does not vouch as unstarted	`[gap]` resident-side classification (`isRuntimeReplacement`, two-phase `run()`) is typechecked (`cd deploy/cloudflare-resident && npm run typecheck`) but the resident Worker has no test harness; live proof: `wrangler deploy` (Worker code only, same image) while a `/exec sleep 60` is in flight on `repo:jshttp/vary` → in-body `{error:"runtime-replaced: …", reason:"runtime-replaced", exitCode:127}`, the bot's tool result shows the re-check text, and the run continues (no ⚠️ abort)
43: a CONTAINER restart under a running command (not just an isolate swap) is also named `runtime-replaced` — the `RuntimeIdentityInactiveError` / `container_stopped` / `RPCTransportError` shapes	`[gap]` live proof, distinct from the row above because a Worker-only deploy never exercises these: with `/exec sleep 120` in flight on `repo:jshttp/vary`, deploy the resident with a CHANGED container image (touch the `Dockerfile`, e.g. bump a comment, then `wrangler deploy` — the platform replaces the container, not only the isolate). Expected: the in-body error's parenthetical names the SDK shape (`RPCTransportError … peer_closed`, `Runtime identity is no longer active`, or `container_stopped`) — NOT a bare 500 / `ExecInfraError`; the bot's tool result shows the re-check text; the run continues. Repeat once with the `/read` route mid-restart: the resident's re-attach + retry answers the file content on the second attempt.
43: the raw stopped-container spawn refusal (`The container is not running, consider calling start()`) — the container ROLL a deploy causes, distinct from the `container_stopped` interruption — classifies as `runtime-replaced` through the shared wording (on the message and its cause chain), and does NOT broaden to a genuine crash (`container exited with unexpected exit code`) or the readiness probe (`the container is not listening`)	`[unit]` `src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING …::classifies the stopped-container spawn refusal as a runtime replacement`, `::matches on the cause chain too (the resident walks selfAndCauses), and is case-insensitive`, `::does NOT broaden to genuinely-fatal container failures — a crash or the readiness probe stay ordinary failures`
43: at the fail-fast boundary the bug tripped — a single container roll rides through as recoverable (0 infra failures, exactly one re-attach, the named outcome reaches the model); the bound still holds — a second roll with no success between is an `ExecInfraError` so a container that cannot come back still aborts	`[unit]` `src/execution/resident.test.ts::ResidentExecutor infra classification through ExecHealthTracker …::a single container roll rides through as recoverable — 0 infra failures, re-attached once`, `::bound: a container that cannot come back still aborts — a second roll with no success between is infra`
attach-on-open surfaces needs-ref (name a branch) and not-onboarded legibly	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.open (attach-on-open)`
a successful attach records `binding {ref, sha}`; needs-ref carries the Worker's `defaultRef` (undefined from an older Worker)	`[unit]` `src/execution/resident.test.ts::records the attach result's ref@sha as the thread binding …`, `::409 needs:"ref" carries the resident's defaultRef …`
the warm path is named positively on the card — `resident · <owner/name> · <ref>@<sha7>`; bind-by-default re-attaches once on the resident's `defaultRef` and says so; a 409 without `defaultRef` still propagates `ResidentNeedsRefError`	`[unit]` `src/execution/factory.test.ts::warm probe → ResidentExecutor …` (note asserted), `::needs-ref WITH the resident's defaultRef → re-attach once on that ref …`, `::warm probe then a needs-ref attach failure → ResidentNeedsRefError propagates …`
per-repo gate (item 26) — open-when-absent; configured allowlist refuses BY NAME with no executor	`[unit]` `src/config.test.ts::per-repo access (canUseRepo)`; `src/core/dispatcher.test.ts::resident repo dispatch::a canUseRepo refusal is a named reply and no executor is created`
fallback note reaches the status card AND the run's stream as a `cold_sandbox` note — after the attach, before the loop, head material	`[unit]` `src/core/dispatcher.test.ts::resident repo dispatch::a resident fallback note appears in the status frames and on the run's stream as a cold_sandbox note (named, never silent)`
CLI stable thread key (`ask --thread <key>` > ephemeral)	`[unit]` `src/cli.test.ts::parseCliArgv — the \`ask` built-in…`;` [agent]npx tsx src/cli.ts ask --thread cli:demo "help"` answers inline (flag parsed, no crash).
live executor round-trip against the deployed resident	`[agent]` tsx script driving `ResidentExecutor` at `repo:jshttp/vary`: `probeStatus` → `{state:"warm"}`; `open` attaches; exec `node -e` → `hi from resident` as `worker2` on `master`; write/read round-trip byte-identical; `ls /no/such/dir` → `exit 2:` result (not an error); `/debug backdate-thread + sweep-now` eviction then exec → transparent auto-re-attach (recreated worktree, clean `git status`); fresh-thread `open` without refHint → legible "name the branch" error.
resolver extracts explicit signals, degrades conservatively	`[unit]` `src/core/repoContext.test.ts` — slug + `on branch X`, `branch:X`, Slack `<url\|label>` unwrap, `/tree/<ref>`, PR URL / `owner/name#N` → head ref via mocked REST (auth header from `resolveGithubToken` when configured; fetch/HTTP failure and cross-fork heads degrade to repo-only), lowercasing, no-signal → `{}` with zero fetches, metacharacter/path tokens ignored whole, hostile ref phrases pattern-refused, prose `on <word>` not a ref.
thread-established repo inherited; ask-once answer binds	`[unit]` `src/core/repoContext.test.ts::…thread history inheritance…` — follow-up inherits the repo, "on main"/"on fix/x" bind against it, current-message repo beats the thread's, assistant turns never establish a repo, PR URLs in history contribute repo with NO fetch.
a refused bare slug is REPORTED, not silently dropped — only when nothing bound	`[unit]` `src/core/repoContext.test.ts::bare prose slugs never hijack a thread…::rejectedRepo` — fresh thread + rejected slug → `{ rejectedRepo }` (no repo); thread weakly bound to a rejected repo → that repo on a signal-less follow-up; rejected `on <slug>` reported; an accepted candidate clears it; no probe → nothing rejected; the bound-thread payload tests still answer `{ repo }` with no `rejectedRepo`. `src/core/dispatcher.test.ts::resident repo dispatch::fresh thread + rejected bare slug + a repo-needing agent → one not-onboarded reply, no run…` — one 📦 reply naming the slug, `repo onboard <slug>` and the URL alternative; zero model calls; `makeExecutor` never called; the ack card closes `not started (repo not onboarded)`. `::the same rejected slug with a no-repo agent (general) runs unchanged…` (resolver never called). `::a bound thread with a prose slug in the follow-up stays silent…`. `[agent]` live: a fresh `agent:coding in <owner>/<not-onboarded>` thread gets the 📦 reply and no run/sandbox; a prose `try/catch` follow-up in a bound thread produces no message.
addressed repos — `in <owner/name>` vetted by the probe is STRONG (binds a fresh thread, rebinds a URL-bound one); a refused address changes nothing; a merely-mentioned onboarded slug and a code-spanned one stay weak; `in <name>` resolves through the registry listing only when exactly one onboarded repo carries the name (ambiguous/unknown = prose; failed or absent listing = no answer; listed at most once per resolution); an address in history binds the follow-up; last strong wins across URLs and addresses; without a probe an addressed slug stays weak; `repoFromThread` honors vetted addressed slugs and ignores names	`[unit]` `src/core/repoContext.test.ts::addressed repos: … bind and rebind once the registry vets them::*` (15)
bare names only in the directive position (`agent:coding in api: …` binds; "the crash is in api" is prose); a slug is addressed anywhere	`[unit]` `src/core/repoContext.test.ts::addressed repos…::a bare name counts only in the directive position…`
silence is not a refusal — an addressed slug the registry did not answer for → `unverifiedRepo` (bound or fresh thread; a throwing probe too; a URL beside it still wins); an unanswered address in history is no answer; a fresh thread's unanswered weak slug → `unverifiedRepo`, not `rejectedRepo`; the dispatcher replies "couldn't verify", no model turn, no executor	`[unit]` `src/core/repoContext.test.ts::addressed repos…::the registry did not ANSWER for an addressed slug…`, `::an unanswered address in HISTORY…`; `src/core/dispatcher.test.ts::…::unverified repo (registry unreachable) + a repo-needing agent → one could-not-verify reply, no run`
the onboarded probe — `true` for any lifecycle state, `false` for 404 / non-transport HTTP errors, `"unreachable"` for transport failures and the outage window they open; undefined without config/bearer	`[unit]` `src/execution/factory.test.ts::residentOnboardedProbe::*`
the registry listing for names — admin bearer, `GET /residents`, `repo:` prefixes stripped and lowercased; non-2xx, malformed body, transport failure (→ outage window), no bearer/config → undefined	`[unit]` `src/execution/factory.test.ts::residentSlugsLister::*`
the card names the bound repo — `resident · <owner/name> · <ref>@<sha7>` on the warm and non-warm attach notes	`[unit]` `src/execution/factory.test.ts::makeExecutor resident selection::warm probe → ResidentExecutor…`, `::refreshing probe…`, `::degraded probe…`; `src/core/dispatcher.test.ts::…::needs-ref WITH a defaultRef…`
production default resolver wired into dispatch (no injection)	`[unit]` `src/core/dispatcher.test.ts::repo/ref resolution + resident prompt selection …::the production default resolver (no injection) extracts repo/ref from the message text`
needs-ref WITH `defaultRef` → bound to the repo default, no question, the run proceeds, the ✅ title carries `resident · <owner/name> · <ref>@<sha7> (repo default — no branch named)`	`[unit]` `src/core/dispatcher.test.ts::…::needs-ref WITH a defaultRef → bound to the repo default with a loud note, no question, the run proceeds`. `[agent]` `agent:coding` on an onboarded repo in a fresh thread naming no branch → no 🌿 question; the card names the default branch.
needs-ref WITHOUT `defaultRef` → ONE question, zero model calls; the ack card closes as not started (no run card)	`[unit]` `src/core/dispatcher.test.ts::…::needs-ref from attach → ONE clarifying question; no model turn; the ack card closes as not started`; typed error: `src/execution/resident.test.ts::409 needs:"ref" is a TYPED error…`
the thread answer rebinds via re-attach and runs	`[unit]` `src/core/dispatcher.test.ts::…::the thread answer "on main" rebinds via re-attach and runs` (attach body carries `refHint:"main"` from the follow-up + the history-inherited repo)
resident prompt variant reaches the provider; fallback prompt unchanged	`[unit]` `src/core/dispatcher.test.ts::…::a resident run gets the agent's resident system variant naming the repo`, `::the per-thread fallback path keeps the agent's own system prompt (regression)`; content pinned by `src/agents/registry.test.ts::resident prompt variants` (ready worktree, no clone/install/gh instructions, push-then-submit — `::coding variant pushes the branch and submits the description; PR creation is not its job` — fallback prompts untouched)
canManageRepos is fail-closed	`[unit]` `src/config.test.ts::repo management gate (canManageRepos)` — no `repo:write` grant → non-admins refused, admins pass; a `repo:write` grant admits its holder + admins; no admin at all still closed.
non-admin repo management → 🚫 naming admins (the registry's `repoManager` gate, no resident call); `repo list` open to a non-admin with the historical text; no model turn; a mutating verb is an inline run with a receipt	`[unit]` `src/core/commands/repo.test.ts::gates … and scopes`; `src/core/dispatcher.test.ts::repo management commands …`, `::registry chat commands in the fast-path chain…::a mutating repo verb is an inline run…` (inline replies, zero provider calls, mocked resident client asserted); `src/core/commands/repo.test.ts::repo.list::*` (golden text, `unavailable` preconditions, `repo:read` on machine surfaces); `src/channels/commandContract.test.ts::adapter contract for migrated commands — $name::repo.list…` (HTTP/MCP/CLI hand back the `/residents` body).
onboard — Node defaults + `main`, `--ref` / quoted `--test`/`--build`/`--install` overrides (the shared grammar; the tokenizer normalizes smart quotes), slug lowercased, invalid slug / hostile ref / unknown option each refused BEFORE any client call without echoing the value, resident 4xx relayed as `conflict`/`not_found`/`invalid_input` with `HTTP <n>: <error>`, the onboard `warning` surfaced	`[unit]` `src/core/commands/repo.test.ts::repo onboard::*`
dry-run replies render the itemized plans	`[unit]` `src/core/commands/repo.test.ts::repo offboard / rebuild (--dry-run)::*` — `dryRun:true` reaches the client; replies carry counts/ids and "Nothing was changed"; unknown flags are usage errors; a 404 is `not_found`.
reconfigure merges onto the live command table	`[unit]` `src/core/commands/repo.test.ts::repo reconfigure::*` — partial patch merged onto `/residents`' current table (whole-table replacement semantics preserved); ref-only patch skips the fetch; nothing to change → `invalid_input`; not onboarded → `not_found`.
offboard dry-run — itemized plan, resident fully intact	`[agent]` `POST /offboard {"resource":"repo:jshttp/fresh","dryRun":true}` → 200 `{"dryRun":true,"wouldRemove":{"registryRecord":true,"schedules":1,"snapshotBackupIds":[2 ids],"backupObjects":4,"r2Objects":0,"threadBindings":0,"container":"warm"}}`; immediately after, `/status` → `warm` and `/debug schedules` shows the refresh alarm still pending.
rebuild dry-run — plan only, still warm	`[agent]` `POST /rebuild {"resource":"repo:jshttp/vary","dryRun":true}` → 200 plan: `from {state:"warm"}`, discards = the current snapshot stamp + ids + `backupObjects:4`, `reprovision {defaultRef:"master", provisioningTimeoutMs:600000}`, `keeps {registryRecord:true, threadBindings:7}`; `/status` after → `warm`, snapshot ids unchanged.
real rebuild — down→onboarding→warm with fresh snapshots	`[agent]` `POST /rebuild {"resource":"repo:jshttp/vary"}` → 202 `{…, backupObjectsDeleted:4, state:"onboarding"}`; `/status` polled → `onboarding` → `warm`; `/debug info` → NEW backup ids, fresh `provisionedAt`, `lastRestore:null` (a true re-provision, not a restore), refresh chain armed, thread bindings retained.
rebuild refused mid-engine	`[agent]` `POST /rebuild` on a resource in state `onboarding` → `409 {"error":"rebuild-refused: the engine is mid-flight (state onboarding) — …"}` (e.g. right after its onboard).
onboard installation check skips HONESTLY when the App is unconfigured	`[agent]` With GITHUB_APP_* unset: onboard → 202 carrying `warning:"github-app-not-configured: installation membership was NOT verified …"`; provisioning proceeds to `warm`. With the secrets set: onboard of a private repo IN the installation → 202 with NO warning and reaches `warm`; onboard of a repo NOT in the installation (e.g. `repo:jshttp/fresh`, public, App installed on the org only) → `403 {"error":"not-in-installation: the GitHub App cannot mint a token scoped to repo:jshttp/fresh — install the App on the repository first (github-token-mint-failed: HTTP 422 …)"}`; `/status` → `not onboarded`, `/residents` count unchanged (no slot consumed).
watchdog auto-rebuild after 3 down passes	`[agent]` `POST /debug {"op":"force-down","resource":"repo:jshttp/fresh"}` → `down (r2-restore-failed: injected …)`; `{"op":"run-watchdog"}` ×3 → passes 1–2 `action:"none"` (strikes accumulate), pass 3 → `action:"auto-rebuilt"`, reason `auto-rebuild: down for 3 watchdog passes (…)`, state `onboarding`; polled to `warm` with fresh snapshot ids. Provision-failure downs are ineligible by the reason pattern (code: `REHYDRATION_FAILURE_RE`).
43/#566: the raw stopped-container spawn refusal (`The container is not running, consider calling start()`) — the container ROLL a deploy causes, distinct from the `container_stopped` interruption — classifies as `runtime-replaced` through the shared wording (on the message and its cause chain), and does NOT broaden to a genuine crash (`container exited with unexpected exit code`) or the readiness probe (`the container is not listening`)	`[unit]` `src/execution/residentRefresh.test.ts::RUNTIME_REPLACEMENT_WORDING (a deploy that ROLLS…::classifies the stopped-container spawn refusal as a runtime replacement`, `::matches on the cause chain too (the resident walks selfAndCauses), and is case-insensitive`, `::does NOT broaden to genuinely-fatal container failures — a crash or the readiness probe stay ordinary failures`
43/#566: at the fail-fast boundary the bug tripped — a single container roll rides through as recoverable (0 infra failures, exactly one re-attach, the named outcome reaches the model); the bound still holds — a second roll with no success between is an `ExecInfraError` so a container that cannot come back still aborts	`[unit]` `src/execution/resident.test.ts::ResidentExecutor infra classification through ExecHealthTracker::a single container roll rides through as recoverable — 0 infra failures, re-attached once`, `::bound: a container that cannot come back still aborts — a second roll with no success between is infra`
Residents dash routes: `/residents[/]` index; `/residents/<owner>/<name>[/]` detail (lowercased); non-slug / traversal / extra segments / other paths → null (fall through)	`[unit]` `src/channels/residentsView.test.ts::parseResidentsRoute::*`
Residents index: one full-row link per resident to its detail page with state dot/reason/ref/short sha; state→tone mapping for every lifecycle state; empty state names `repo onboard`; hostile slug/reason render as text and a non-slug resource never becomes a link; shared nav marks Residents current	`[unit]` `web/src/pages/residents.test.ts::ResidentsIndexPage::*`, `src/channels/residentsView.test.ts::residentStateTone` (escaping red-verified via the seed-island test)
Residents index tab favicon = the fleet's worst tone: red over amber over green; green only when every resident is warm; grey for no residents and for an unknown/unreachable resident with nothing worse to show (a non-record entry counts as unknown); rendered by the shell from the seed with four distinct dot URIs	`[unit]` `src/channels/residentsView.test.ts::residentsFleetTone::*`, `src/channels/webShell.test.ts::renderShell::the residents index wears the fleet's worst dot from the seed: red over amber over green, grey when empty or unknown`
Resident detail: thread worktrees newest-attach first with sha linked to its commit (hex only), deps mechanism, user, times, evicted marker + `N live · M evicted` summary; "no thread worktrees" when none; hostile thread fields render as text	`[unit]` `web/src/pages/residents.test.ts::ResidentDetailPage::lists thread worktrees (ref, sha linked to its commit, deps mechanism, attach times), newest first, marking evicted ones`, `::says so when a resident has no thread worktrees, and never links a non-hex sha`
Attach records the sha on the binding; `/residents` live view carries `threads`	`[agent]` After a run attaches, the resident detail's row for that thread shows its short sha linked to the commit and the deps mechanism (e.g. `hardlink`).
Resident detail: state, ref, sha (linked to the GitHub commit only when a sha exists), lockfile hash, provisioned/refreshed, snapshot stamp + backup ids (or "no snapshot"), schedules, command table + effects, worktree TTL; failure reason + last refresh error for a down resident; `{error}` live view → "unreachable" grey; hostile command strings render as text; back link to `/residents`	`[unit]` `web/src/pages/residents.test.ts::ResidentDetailPage::*`
Residents handler: ignores non-/residents paths; reads the admin registry LIVE on every request (call count asserted); serves index/detail with the `/runs` CSP + `X-Frame-Options: DENY` + `no-store`; unknown slug → 404 "not onboarded"; non-GET → 405 `allow: GET`; no client → 503 naming `execution.resident`; upstream non-200 or throw → 502 with the upstream status/reason	`[unit]` `src/channels/residentsView.test.ts::createResidentsViewHandler::*`
Residents dash behind Access: with SSO configured, `/residents` without a valid `Cf-Access-Jwt-Assertion` → 403; with it → the live list of onboarded repos; a detail page links to the GitHub commit	`[agent]` Open `https://<PUBLIC_BASE_URL>/residents` signed in → rows match `repo list`; `curl -i https://<PUBLIC_BASE_URL>/residents` (no cookie/JWT) → 403.
end-to-end chat commands against the LIVE resident	`[agent]` tsx script driving `dispatch()` (real config-command path, provider poisoned to throw) with `execution.resident` at the real base URL and `RESIDENT_ADMIN_TOKEN` in env: `repo list` (non-admin) → `Resident repos (1/6): • jshttp/vary — warm · ref master · sha 1220b9c4 …`; `repo offboard jshttp/vary --dry-run` (admin) → the itemized plan reply ending "Nothing was changed"; `repo onboard acme/api` (non-admin) → 🚫 naming the admin; `repo rebuild jshttp/vary --dry-run` (admin) → the rebuild plan reply. Zero provider calls, zero status cards; `/status` after → `jshttp/vary` still `warm`.
op recognition — the explicit form belongs to the registry (`repo.test\|build`: optional ref, slug lowercased, hostile refs → the schema's named refusal), conservative NL forms recognized and translated into the same invocation, silent fallthrough on ambiguity	`[unit]` `src/core/commands/repo.test.ts::repo test / repo build …::*`, `src/core/operations.test.ts` — the explicit form is null there, NL forms with thread-repo inheritance, question/extra-clause phrasings → null, metacharacter refs → null, owner/name-shaped "on" token ambiguous, directives disable NL.
deterministic ask answers with ZERO model turns	`[unit]` `src/core/dispatcher.test.ts::deterministic ops fast-path …::F3…` (op result posted; fake provider asserts zero calls; `makeExecutor` never called). `[agent]` E2E vs the LIVE resident: `dispatch()` with a POISONED provider (throws if called), real `execution.resident` config + `RESIDENT_OPERATOR_TOKEN`; `"run the tests on master in jshttp/vary"` AND `repo test jshttp/vary master` each replied `✅ test passed on repo:jshttp/vary @ master (1220b9c4) in 7s` — provider calls 0, status cards 0.
only the model call is skipped — never the permission machinery	`[unit]` `src/core/dispatcher.test.ts::…::a user without coding-agent access is refused by the \`agentRun` gate…`(the registry's shared restricted line, op never executes),`::a canUseRepo refusal … names the repo…` — both with zero provider calls.
hostile op inputs never reach a shell	`[unit]` `src/core/dispatcher.test.ts::an explicit \`repo test` with a hostile ref is a NAMED refusal…`,` ::a natural-language ref with shell metacharacters falls through silently…`,` src/core/commands/repo.test.ts::repo test / repo build…::a hostile ref is `invalid_input` naming the argument before any backend…`;` [agent]`live:`POST /op {"op":"deploy"}`→`400 op must be one of test, build, status`;` `→`400` naming the ref pattern.
/op runs the table command in a disposable checkout with warm deps	`[agent]` `POST /op {"resource":"repo:jshttp/vary","op":"test","ref":"master"}` (operator bearer) → streamed `{ok:true, …, sha:"1220b9c4…", summary:"test passed on repo:jshttp/vary @ master (1220b9c4) in 5s", deps:"hardlink", reconciled:false, durationMs:<ms>}` — deps from the shared lockfile-keyed cache, no install (`durationMs` a fraction of provisioning).
status op = no checkout, lifecycle + sha + lastRefresh	`[agent]` `POST /op {"op":"status"}` → `{ok:true, state:"warm", ref:"master", sha:"1220b9c4…", lastRefreshAt:"<iso>", summary:"repo:jshttp/vary is warm on master @ 1220b9c4, …"}` — instant, no container exec.
op on a concurrently-attached ref leaves the thread worktree untouched; checkout deleted after	`[agent]` Attached `slack:u6iso` on `master` (worker3), planted `u6-marker.txt`; ran `/op test` on the same ref; after: thread `/exec` shows the marker intact, `git status --porcelain -uno` empty, and `ls -A /workspace/ops/ \| wc -l` → `0` (per-op checkout deleted). Evict the binding afterwards via backdate+sweep to release the pool user.
failing op is a RESULT, not an error	`[unit]` `src/core/dispatcher.test.ts::…::an op failure (tests fail) is posted as ❌…`, `src/execution/resident.test.ts::ResidentOperations.run::a failing op is a RESULT…`, `src/execution/executor.test.ts::LocalOperations::a failing test run is a RESULT…`; `[agent]` live: reconfigured vary's test command to `process.exit(1)` → `/op test` streamed `{ok:false, summary:"test failed (exit 1) on repo:jshttp/vary @ master (1220b9c4)", stderr:"u6 forced failure", exitCode:1}`; table restored and re-verified passing.
effects:mutating refused on the modelless path with a named reason	`[unit]` `src/core/dispatcher.test.ts::…::a mutating command-table entry is refused…` (mocked ops backend returning the refusal; 🚫 reply, zero provider calls); `[agent]` live via reconfigure: `POST /reconfigure {"effects":{"test":"mutating"}}` → `/op test` → `409 {"error":"op-refused: the \"test\" command-table entry is marked effects: mutating — …"}`; `{"effects":{}}` restored → `/op test` passes again.
non-onboarded / unknown-ref asks degrade correctly	`[unit]` `::a non-onboarded repo natural-language ask falls through to the agent path` (op attempted, agent then served), `::an explicit \`repo test` on a non-onboarded repo gets a named reply`;` [agent]`live:`/op`on`repo:jshttp/fresh`→`404 not onboarded`;` ref:"no-such-branch-u6"`→ in-body`` over the 200 stream.
op on a ref whose lockfile differs reconciles via the shared cache	`[agent]` Mechanism row: `/op` materializes deps through the EXACT `materializeThreadDeps` used by attach (same-key `cp -al` proven live above via `deps:"hardlink"`, `reconciled:false`; the differing-key scoped-install branch proven live in the attach checks above — `deps:"install"`, `reconciled:true`), and the op response carries `deps`/`reconciled` as evidence. Not separately exercisable on jshttp/vary (single branch, no committed lockfile → every ref shares one key).
ambiguous phrasing → agent path	`[unit]` `src/core/dispatcher.test.ts::…::ambiguous phrasing ("can you check the tests seem fine?") falls through to the agent path`, `::an explicit agent directive skips the natural-language fast-path…`.
LocalOperations (dev/CLI) is a real second implementation	`[unit]` `src/execution/executor.test.ts::LocalOperations` — status/existence, `npm test` pass+fail, `--if-present` build, ref-ignored honesty; wired as the local-execution default in the dispatcher (`defaultOperations`).
live zero-setup dispatch against the deployed resident	`[agent]` tsx script calling `dispatch()` with a scripted provider (one bash tool_use; no ANTHROPIC_API_KEY in the check environment) and the real resident config. (1) `agent:coding on <default branch> in <onboarded repo>: run node -e "console.log(1)" and report` on a fresh thread → the production default resolver extracts `{repo, ref}` from the text, the executor is the ResidentExecutor, the provider receives the resident system variant (ends `Target repository: <repo>. The worktree is already on this thread's bound branch …`), and the ONLY `[tool]` log line is the scripted command itself — zero setup commands; the exec runs in `/workspace/threads/<thread>/<ref>` as a pool user on that branch. (2) Fresh thread, repo but no ref (`agent:coding in <repo>: say hello`): with a `defaultRef` configured for the repo the run binds to it with no question and the ✅ title carries `(repo default — no branch named)` (item 30); with no `defaultRef` → the 🌿 ask-once question naming the repo, ZERO provider calls, no status card, and the thread answer `on <branch>` (history carrying the original ask) inherits the repo, `/attach` binds that branch, and the run executes in a fresh worktree as another pool user. Evict the bindings afterwards via `backdate-thread` + `sweep-now` to release the pool users.
45: ref fate from the pulls list — open wins, merged beats closed, none → no-pr; the REST body is parsed defensively (non-array → null, malformed elements dropped)	`[unit]` `deploy/cloudflare-resident/gc.test.ts::pullsFate`, `::parsePullsBody` (red-verified: module absent)
45: reclaim decision — gone/merged/closed + clean + idle → reclaim; default ref, open PR, no PR, unknown fate, busy, dirty → keep with the reason named; runtime down (tree gone) → reclaim	`[unit]` `gc.test.ts::reclaimDecision*`
45: live — a review thread bound to a PR branch is reclaimed within one refresh cycle of the PR merging (or its branch being deleted), its pool user freed, the binding kept with `evictedWhy`	`[agent]` On `repo:<owner>/<name>`: attach a thread on an open PR's head branch (`POST /attach {"threadKey":"cli:gc50","refHint":"<branch>"}`), merge/close the PR or delete the branch, then `POST /debug {"op":"reclaim-now","resource":"repo:<owner>/<name>"}` → `reclaimed:[{"threadKey":"cli:gc50","ref":"<branch>","why":"gone"\|"merged (pr merged)"}]`; `/debug threads` shows the binding `evicted:true`, `user:""`, `evictedWhy` set; the dash row reads `evicted <ts> · <why>`. Negative control: a thread bound to `main` (default ref) or to a branch with an OPEN PR is in `kept` with `why:"default-ref"` / `"pr-open"`. The unforced path is the same function at the end of every `onRefreshAlarm` (≤ 10 min when awake).
45: a dirty tree on a merged branch is kept, not destroyed	`[agent]` Attach on a PR branch, `/exec` `echo x >> README.md`, merge the PR, `reclaim-now` → the thread is in `kept` with `why:"dirty"`; the tree and its edit survive (`/exec git status --porcelain` shows `M README.md`). Force-detach the thread afterwards to free its pool user.
46: LRU pick — coldest by last activity (attach or provisioning, evicted bindings count); floor keeps a just-used resident; live worktree / non-warm / busy / unknown in-flight / failed view are each rejected by name; ties deterministic; empty fleet total	`[unit]` `gc.test.ts::pickEvictionCandidate*`
46: chat — `--evict-coldest` sends `evictColdest:true`; the 202 `evicted` block renders as a ♻️ line naming the resident + last use; a 429 with `rejected` renders per-resident bullets; the flag is refused on `repo reconfigure`	`[unit]` `src/core/commands/repo.test.ts::repo onboard::--evict-coldest opts the onboard into LRU eviction …`, `::an over-cap onboard with no eligible resident relays the per-resident reasons`, `::--evict-coldest is an onboard-only flag: reconfigure refuses it as an unknown option`
46: dash shows the eviction reason on an evicted thread row, as text	`[unit]` `web/src/pages/residents.test.ts::ResidentDetailPage::lists thread worktrees … marking evicted ones` (asserts `evicted <ts> · <why>` with a `<b>` payload staying text)
46: live — over the cap WITHOUT the flag → 429 naming the opt-in and nothing offboarded; WITH the flag → the coldest eligible resident is offboarded and the new one onboards; no eligible candidate → 429 with `rejected` reasons	`[agent]` (with the effective cap lowered to the current fleet size via item 49's `set-test-overrides`, so the existing fleet IS the full cap; fillers must be slugs of the installation's owner because the installation-scoped mint refuses others, and each is a billable container; the example fleet below is `<owner>/<repo>` + `jshttp/vary`) `repo onboard <owner>/<other>` → `⚠️ … HTTP 429 … resident cap reached (2/2) … evictColdest:true …`, `repo list` unchanged; `repo onboard <owner>/<other> --evict-coldest` while `<repo>` is active and `vary` is `degraded` → 429 with `rejected[]` naming both reasons (`active Nm ago (floor 60m)` / `N live worktree(s)`, `state degraded`); then `repo rebuild jshttp/vary`, leave it untouched for the 1 h floor, and `repo onboard <owner>/<other> --evict-coldest` → `🏗️ Onboarding … ♻️ Made room: evicted \`jshttp/vary` …`,` repo list`2/2 with the newcomer,`vary`'s` /status`→ 404.`evictColdest:"yes"`→`400 evictColdest must be a boolean`before any registry write; a duplicate onboard WITH`evictColdest:true`→`409 already onboarded`, nothing evicted (the flag is inert off the 429 path). The floor BOUNDARY itself is item 49's live row.
49: override parsing — lower-or-equal accepted, neither field = clear, raising above the constant / non-integers / zero cap / negatives → named 400 text	`[unit]` `deploy/cloudflare-resident/gc.test.ts::parseTestOverrides*` (red-verified)
49: effective limits — defaults with no override; only the named fields lowered; a record from another build ignored (`stale-build`); stored values above the constant clamped	`[unit]` `gc.test.ts::effectiveLimits*` (red-verified)
49: `repo list` names an active override (enforced vs default cap, floor, when set, how to clear); no warning without one	`[unit]` `src/core/commands/repo.test.ts::repo.list::an empty registry and an active test override keep their wording`
49: live — set `cap:1` on a 1-resident fleet → the next onboard 429s at `(1/1)`; `GET /residents` shows `cap:1, capDefault:6, testOverrides{…}`; read token on the op → 403; `cap:7` → 400; clear → `/residents` `cap:6`, no `testOverrides`; and with `floorS:600` + a fresh filler the boundary is provable in minutes: `--evict-coldest` at ~8 min → `rejected … active 8m ago (floor 10m)`, at ~11 min → the filler is evicted	`[agent]` `curl -X POST -H "Authorization: Bearer $ADMIN" .../debug -d '{"op":"set-test-overrides","cap":1,"floorS":600}'` then the probes above with `repo:<owner>/<small repo>` (`test:"true", build:"true"`) as the filler; finish with `-d '{"op":"set-test-overrides"}'` to clear and verify `/residents` (`cap:6`, `testOverrides:null`). Expected shapes: read token → `403 forbidden: admin scope required`; `{"cap":7}` → `400 cap must be an integer between 1 and 6 …`; a set → `200 {cap, capDefault:6, floorS, floorDefaultS:3600, override:{…, build:"<marker>"}}`; under the floor → `429 rejected:[{<filler>, "active Nm ago (floor Mm)"}, …]`, past it → `202 evicted:{resource:<filler>, lastActivityAt, …backupObjectsDeleted:4, errors:[]}`.
46: the flag alone never changes the atomic cap semantics — without 429 it is inert; the slot swap is one input-gated registry section and nothing is destroyed before the newcomer holds the slot	`[code review]` `handleOnboard`: the eviction branch is entered only on `result.status === 429 && evictColdest`; its only registry mutation is `registry.replace(victim, record)` (delete + put under the DO input gate, cap re-checked); `teardownResident` runs strictly after a successful swap.
50: `readonly` body field parses absent→false, boolean→itself, anything else→named 400	`[unit]` `src/execution/residentReadonly.test.ts::parseReadonly…`
50: read-only plan = no token, scrub, origin=mirror; writable plan = token, no scrub, origin=GitHub; mode switch on a live tree (both directions, and pre-field bindings count as writable); evicted prior never switches; reused read-only tree still scrubbed	`[unit]` `src/execution/residentReadonly.test.ts::planReadonlyAttach`
50: the bot sends `readonly: true` iff the agent's toolset is `readonly` (review), omits the field otherwise (coding)	`[unit]` `src/execution/factory.test.ts::a readonly-toolset agent attaches with readonly:true…`, `src/execution/resident.test.ts::sends readonly:true in the attach body…`
50: Worker wiring compiles against the pure plan (`attachThread(threadKey, refHint, readonly)`, `ensureThreadWorktree(binding, sha, originUrl, modeSwitch)`, `scrubThreadCredentials`)	`[unit]` resident `npm run typecheck`
50: live — `agent:review` on the resident: `/debug threads` shows the binding with `readonly: true`; in the worktree `test -f .git/github-credentials` → absent, `git config credential.helper` → unset, `git remote get-url origin` → `/workspace/mirror`, `git fetch origin` → permission denied / not a repository, `git diff origin/main...HEAD --stat` → works; then `agent:coding` in the same thread → `recreated: true`, `credentials: "ok"`, origin = GitHub	`[agent]` Run `agent:review` then `agent:coding` on an onboarded repo in one thread; inspect via `/debug threads` and `/exec` in that thread's worktree.
51: `sha` body field parses absent→null, full 40-hex lowercase→itself, anything else (7-char abbreviation, uppercase, `HEAD`, `main`, `../evil`, 39/41 chars, non-strings)→named 400	`[unit]` `src/execution/residentHead.test.ts::parseWantSha…`
51: the sha applies to the ref it was resolved for — refHint = bound ref or no refHint → kept; refHint ≠ sticky bound ref → dropped; no sha → null	`[unit]` `src/execution/residentHead.test.ts::wantShaForBinding`
51: fetch decision — missing ref → fetch (with or without a wanted sha); no wanted sha + ref present → no fetch; ref present at an OLDER tip than the wanted sha → fetch; tip already at the wanted sha → no fetch; unreadable tip + wanted sha → fetch	`[unit]` `src/execution/residentHead.test.ts::mirrorNeedsFetch`
51: the client sends `sha` in the attach body when an expected head is set, omits the field otherwise	`[unit]` `src/execution/resident.test.ts::sends sha in the attach body when an expected head is known…`
51: attach target after the fetch — the ref when it exists; the expected commit, detached, when the ref is gone but the mirror holds the commit; unknown-ref when no commit was named or the mirror lacks it	`[unit]` `src/execution/residentHead.test.ts::attachTarget::*`
51: executor selection forwards a resolved `headSha` as the attach body's `sha` (alongside `refHint` + `readonly`), no field when unresolved	`[unit]` `src/execution/factory.test.ts::a resolved headSha is sent as the attach body's sha…`
51: the dispatcher passes `RepoContext.headSha` into executor selection	`[unit]` `src/core/dispatcher.test.ts::passes the resolved repo, ref and PR head to executor selection`
51: Worker wiring compiles against the pure decision (`attachThread(threadKey, refHint, readonly, wantSha)`, `wantShaForBinding`, `mirrorNeedsFetchFor`)	`[unit]` resident `npm run typecheck`
51: live — push a commit to an open PR's branch, then within the same refresh cycle (< 10 min) `agent:review <PR url>` in a thread already bound to that branch → the status card's `resident · <branch>@<sha7>` names the NEW head, the review posts to the PR (no `ℹ️ Review not posted … reviewed head X is not the PR head Y`), and `/debug threads` shows the binding's `sha` = the new head	`[agent]` (post-deploy of resident + bot)
16b: clean-check decisions are unchanged in one spawn — missing tree → clean ("worktree missing"); a failed git probe → NOT clean with the first error line named; changes/unpushed counted with the exact pre-existing wording; PAM banners cannot shift a field; the git probes run inside `su`, never as root	`[unit]` `src/execution/residentCleanliness.test.ts`
53: `describeStepFailure` — the pnpm capture (real bytes: `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH` on stdout, the `[WARN]` on stderr) keeps the error AND the warning; both streams labelled; an empty stream drops its label; no output at all says `no output`; a timeout says `(timed out)`; a long stream keeps its END and marks the cut; the whole reason stays bounded; exit 0 throws	`[unit]` `src/execution/residentStepReport.test.ts`
53: `stepFailureLog` — names the step, carries both streams at the log budget, stays under 20 kB for a runaway step	`[unit]` `src/execution/residentStepReport.test.ts`
53: `imagePins` — exact global install → pinned; `@latest`, a range and a bare name → floating; a `FROM` with no tag or `:latest` → floating while `node:22-slim` is not; reads through a line continuation past glued shell separators; a comment naming `@latest` is not an instruction. Fence: all three Dockerfiles (resident, sandbox, bot) carry zero floating pins	`[unit]` `src/deploy/imagePins.test.ts`
53: the reported reason was never a cause — `pnpm install --frozen-lockfile` on a pnpm workspace's default branch inside `cloudflare/sandbox:0.13.0-next.751.1` with an unpinned `pnpm@latest` (11.x), as `worker1` via `su -s /bin/bash`: exit 0, stderr byte-identical to the stored `down` reason	`[agent]` container repro: run the install inside the base image as `worker1` and compare its stderr with the recorded reason
53: the pin removes the noise at its source — the same repro on the image built from the pinned Dockerfile: `pnpm 10.34.5` / `yarn 1.22.22` asserted at build, install exit 0 with stderr empty, still delegating to the repo's own `packageManager` pin	`[agent]` container repro on the pinned image: assert the tool versions at build, run the install as `worker1`, expect exit 0 and an empty stderr
53 live: after the resident deploy, `repo rebuild <owner>/<name>` either reaches warm or goes `down` with a reason that names the failing command's own error (both streams present)	`[agent]` post-deploy: rebuild a resident whose install had failed and read the reason on `/status`
54: `isDiskFullMessage` — the SDK's `ENOSPC` writeFile error (the refresh reason item 54 describes), git/cp `No space left on device`, npm `ENOSPC` → true; git config's errno-less exit-4 message, the stale-lock `File exists` message, near-miss words → false	`[unit]` `src/execution/residentDisk.test.ts::isDiskFullMessage …::*`
54: `parseDfFreeKiB` — the Available column of `df -Pk`; header-only, non-numeric, or an error line → `null`, never 0; `DF_FREE_ARGV` is `df -Pk /workspace`	`[unit]` `src/execution/residentDisk.test.ts::parseDfFreeKiB …::*`
54: `diskFullReason` names the step, keeps the message verbatim, appends the probe only when it answered; `isDiskFullReason` matches the exact `disk-full:` prefix; the floor is 128 MiB	`[unit]` `src/execution/residentDisk.test.ts::diskFullReason / isDiskFullReason …::*`
54: `planDiskFullRecovery` — clear → recycle; inside the 1 h cooldown → wait naming the minutes and "does not fit"; ops in flight → wait naming the count; dirty/unreadable tree → wait; cooldown checked first	`[unit]` `src/execution/residentDisk.test.ts::planDiskFullRecovery …::*`
54: `classifyRefreshFailure` — ENOSPC wording → `disk-full:` outright (over a kill signature too); the errno-less git-config exit 4 → disk-full only with a probe below the floor, else `git-setup-failed`; a kill signature with a low probe stays `refresh-interrupted`; no probe → the step's own failure	`[unit]` `src/execution/residentRefresh.test.ts::classifyRefreshFailure …::*` (red-verified: fails on a classifier without the disk-full class)
54: `disk-full-restart` re-arms at `INTERRUPTED_REARM_S`, uncapped like an image-stale restart	`[unit]` `src/execution/residentRefresh.test.ts::nextRefreshDelayS …::disk-full restart …`
54: `disk-full: …` is never a serviceable degraded reason (the bot goes cold without attaching)	`[unit]` `src/execution/residentState.test.ts::isServiceable::degraded stays cold …` (fence: the allow-list is exact — the test names the case so a widened list fails loudly)
54: the two git messages are two different failures — a full disk with a creatable inode is exit 4 `failed to write new configuration file <cfg>.lock`; a pre-existing `.lock` is exit 255 `could not lock config file <cfg>: File exists`	`[agent]` Linux tmpfs, no resident needed: `docker run --rm --tmpfs /t:size=64k alpine:3.20 sh -c 'apk add -q git; printf "[a]\n\tb = c\n" > /t/gitconfig; dd if=/dev/zero of=/t/fill bs=4k 2>/dev/null; git config --file /t/gitconfig safe.directory /x; echo exit=$?; rm -f /t/fill; : > /t/gitconfig.lock; git config --file /t/gitconfig safe.directory /x; echo exit=$?'` → `error: failed to write new configuration file /t/gitconfig.lock` / `exit=4`, then `error: could not lock config file /t/gitconfig: File exists` / `exit=255`.
54 live: a full disk is named, the card goes cold without an attach, and the resident recycles itself	`[agent]` On a warm resident with ONE attached thread and nothing else live: from that thread `/exec` `fallocate -l <free minus 64M> /tmp/fill` (as the thread user; `df -Pk /workspace` afterwards shows `Available` under 131072), then `POST /detach {force:true}` for the thread (no live tree left to protect), then `POST /debug {"op":"refresh-now"}`. Expect: `/status` → `{"state":"degraded","reason":"disk-full: fetch … ENOSPC … (/workspace: <n> KiB free)"}`; a Slack request naming the repo shows `resident degraded (disk-full: …) — using fresh sandbox` on its card and `wrangler tail switchboard-resident` shows NO `attachThread` for it; the tail shows `disk-full: recycling the container …`, then within ~2 min `/status` `restoring · rehydrating` → `warm`, `/debug info` → `lastRestore` fresh and `lastRefreshError` cleared; the fill file is gone with the disk. Repeat the fill within the hour → `degraded(disk-full: …)` stays, `/debug info` `lastRefreshError` ends `— container kept: recycled <n> min ago and the disk filled again …`. Then free the space (`rm` the fill from a re-attached thread, detach) → the next cycle goes `warm` without a recycle.
18: pnpm's `.pnpm` store stays hardlinked — never in the mutable-cache swap — while a `.cache` nested inside it, `.modules.yaml` and `.bin` still are (a pnpm-workspace attach ~190 s → tens of seconds, thread tree 2.6 GB → ~0.45 GB)	`[unit]` `src/execution/residentDepCache.test.ts::mutableCachePaths…::pnpm's \`.pnpm` store is package content, never a cache…`;` [agent]`fresh-thread attach on a pnpm-workspace resident: card duration and`du -xsh`of the thread's`node_modules/.pnpm` (hardlinked: link count > 1 on its files)
18: the one-fork materialization script gates each dir on src/dst exactly like the old spawns, combines the chown + harden walks into one traversal with the identical `-perm -g+w -o -perm -o+w` test, falls back from `cp -al` to plain copy, tags mechanisms/mutable listing/failed steps; the swap script replays `rm -rf`/`cp -R`/`chown -Rh` per path	`[unit]` `src/execution/residentDepCache.test.ts::depCacheScript…` + `::parseDepCacheScriptOutput` + `::mutableCacheSwapScript…`
5/7: an R2 SNAPSHOT upload that outlives its 5-min budget rejects with the transfer NAMED (`<what> timed out after <ms>ms`), a late loser never surfaces as an unhandled rejection; restores are judged by bytes instead (item 61)	`[unit]` `src/execution/residentRefresh.test.ts::withTimeout…`
45: the for-each-ref listing parses into the mirror's branch set (trimmed, blanks dropped, empty listing = empty set)	`[unit]` `deploy/cloudflare-resident/gc.test.ts::parseRefListing…`
Perf live: cold-wake restore and refresh-cycle timings improve (concurrent restore/snapshot pairs; reclamation fate lookups no longer serial) — compare `lastRestore.ms` and the tail's `refresh: … done in <ms>` before/after	`[agent]` post-deploy of the resident: compare both timings across the deploy
55: measurement — `parseDfKiB` reads total/used/free (header-only or non-numeric → null); `duArgv` orders mirror, `checkout/node_modules`, checkout, threads, homes; `parseDu` keeps `<KiB>\t<path>` lines only; `assembleDiskSample` itemizes by key, unread parts are null (never 0), `other` = used − Σ and never negative	`[unit]` `src/execution/residentDiskBudget.test.ts::measurement …::*`
55: reserve — staging is 0.6 × (mirror + deps + checkout), the floor the larger of 1 GiB and 5 % of the disk, unmeasured parts count 0 in staging while the floor stands	`[unit]` `src/execution/residentDiskBudget.test.ts::the reserve …::*`
55: projection — hardlink → `parts.checkout`; reconcile (lockfile diverged, install command present) → `+ RECONCILE_DEPS_RATIO × parts.deps` (strictly between a hardlink and a full copy); reuse → 0; a missing part → null, never a guess	`[unit]` `src/execution/residentDiskBudget.test.ts::projecting …::*`
55: `diskBudgetMb` caps free at `budget − used` (never negative; a budget above the disk is the disk; `capped` says whether it decided)	`[unit]` `src/execution/residentDiskBudget.test.ts::diskBudgetMb …::*`
55: admission — fits when `free − reserve ≥ projected` with the exact shortfall otherwise; a fresher `df` overrides the sample's free; in-flight commitments are deducted ONCE from the raw reading (a re-check after an eviction feeds `rawFreeAfterEviction`'s raw number, never a net one); the cap refuses; an unmeasured projection admits only while free clears the reserve; reuse is never refused	`[unit]` `src/execution/residentDiskBudget.test.ts::checkDiskAdmission …::*` (red-verified: with the reserve zeroed the install-thread case admits)
55: eviction order — coldest attach first, ties on key; the requesting thread, a busy tree, the default branch and a tree attached under 10 min are kept by name; an unparsable `lastAttachAt` is kept as recent	`[unit]` `src/execution/residentDiskBudget.test.ts::orderEvictionCandidates …::*`
55: the refusal text names projected (or UNMEASURED), free, the cap when one decided, the reserve and both terms, the shortfall, what was evicted with the bytes back, and every keep with its why; `isDiskPressureReason` matches the exact prefix	`[unit]` `src/execution/residentDiskBudget.test.ts::diskPressureReason …::*`
55: one unit everywhere — `formatGiB` (two decimals under 10 GiB, one above, `?` unknown) and `formatDiskGauge` (`used/total (pct)`)	`[unit]` `src/execution/residentDiskBudget.test.ts::formatting …`
55: eviction removes the pool user's pnpm store, `.cache`, `.npm`, `.yarn`, `.bun` in one `rm`	`[unit]` `src/execution/residentDiskBudget.test.ts::threadUserCacheCleanArgv …`
55: a `disk-pressure` 503 from `/attach` falls back to the cold sandbox with the whole refusal in the card note	`[unit]` `src/execution/factory.test.ts::makeExecutor resident selection::item 55: a warm probe then a disk-pressure attach (503…`
55: `repo list` appends `· disk <used>/<total> (<pct>%)` from `live.disk`; a malformed sample adds nothing	`[unit]` `src/core/commands/repo.test.ts::repo.list::item 55 …`
55: the watchdog line carries `· disk max <pct>% (<owner/name>)` for the fullest measured resident; absent/malformed gauges are skipped	`[unit]` `src/core/schedules.test.ts::watchdogFiring …::item 55 …`
55: `/residents` index row shows the gauge with the sample time on hover (none for an unmeasured resident); the detail page's Disk section shows gauge, free (under the cap when set), reserve + terms, headroom in trees, sample time, every component (thread trees at unique bytes, homes above 1 MiB), "not measured yet" before the first sample, and a malformed sample never renders NaN or markup	`[unit]` `web/src/pages/residents.test.ts::ResidentsIndexPage::item 55 …`, `::ResidentDetailPage::item 55 …` (×2)
55: the instance type is 4 vCPU / 12 GiB / 20 GB — the CPU is the decision, memory is what the vCPUs require, the disk is what the platform allows for that memory; it fits 16 hardlinked + 1 installing (and 12 + 2) trees with the reserve, is not over-provisioned, the pool maximum (16 + 2) does not fit even this ceiling, the previous 16 GB fit 10 + 1 — over the SAME reserve constants the admission uses	`[unit]` `deploy/cloudflare-resident/instanceSizing.test.ts::*`
60: `purge-bindings` selects only evicted bindings under a whole non-production namespace (or longer); a bare/partial/production prefix is refused by name; live bindings are listed as kept, never deleted	`[unit]` `src/execution/bindingPurge.test.ts::selectBindingsToPurge::*`
60 live: a load run leaves no rows behind	`[agent]` After `npm run load -- resident --resource repo:<slug> --threads 4 --hold 30 --cpu-seconds 5`, `POST /debug {"op":"threads","resource":"repo:<slug>"}` lists no `load:` thread keys; `POST /debug {"op":"purge-bindings","resource":"repo:<slug>","prefix":"slack:"}` answers 400 naming the production namespace.
55 live: the gauge appears and moves — after the resident deploy, `POST /debug {"op":"measure-disk","resource":"repo:<owner>/<name>"}` ($ADMIN) answers a `DiskSample` whose `parts.deps` is the deps store's size, `parts.checkout` the tree without it, `parts.homes.worker1` < 1 MiB; `GET /residents` shows the same under `live.disk`; `/residents/<owner>/<name>` shows the Disk section with "room for N more"; `repo list` in Slack shows `· disk x/y (pct%)`; attach a thread (any coding request naming the repo), wait ~5 s, re-read `/residents`: `parts.threads` gains the thread at its unique bytes and `used` grew by about that; after the run's `/detach` the entry disappears and `used` falls back	`[agent]` post-deploy: the probes in the criterion, in order
55 live: admission refuses legibly — on a resident with ONE attached thread, from that thread `/exec` `fallocate -l <free − reserve − 0.3G> /tmp/fill` (leave room for less than one hardlinked tree above the reserve; `df -Pk /workspace` confirms), then a coding request on a NEW thread naming the repo: the card reads `resident attach failed (… disk-pressure: need 0.5x GiB for a new tree (hardlink), but … free minus the … reserve (snapshot staging … + floor 1.00 GiB) leaves … — short by …; evicted nothing; kept 1: <the filled thread> (attached <n>m ago (floor 10m))) — using fresh sandbox`, `wrangler tail switchboard-resident` shows the same line, `/status` stays `warm` (no lifecycle flip); backdate the filled thread (`POST /debug {"op":"backdate-thread", threadKey, days:1}`) and repeat: the refusal now reads `evicted 1 idle tree(s) …` and the attach SUCCEEDS on the space it freed (the fill file went with the tree); `rm` nothing by hand — the eviction did it	`[agent]` post-deploy: the recipe in the criterion
56: the pre-step sweep is a root `sh -c` (never `su`), scoped by cwd to the tree (`readlink /proc/$p/cwd` under `/workspace/checkout` — a parallel store install elsewhere is never matched), names the survivors (`ps -o pid=,args=`) BEFORE `kill -KILL` (SIGKILL, never SIGTERM), waits bounded until EVERY matched pid is gone (`exit 1` after 5 s), and exits 0 silently when nothing matches	`[unit]` `src/execution/residentRefresh.test.ts::killStaleBuildProcessesCommand …::*` (red-verified: export absent)
56: an abandoned wait becomes the step's OWN timeout — `timedOut: true`, the observed exit (or -1 with "no exit status observed"), "killed" in the report — and `classifyRefreshFailure` files it `<step>-failed: exit <n> (timed out) …`, never an interruption	`[unit]` `src/execution/residentStepReport.test.ts::abandonedWaitStepResult …::*` (red-verified: export absent)
56 live: on a busy resident, `wrangler tail switchboard-resident` (or the observability log) shows `install: killing stale worker1 processes:` + the npm command line at most once after the deploy (the orphans the incident left), then a cycle that ends `refresh: … rebuild done in <ms>` with `/residents` `state: warm`, `lastRefreshError: null`; no later cycle logs `checkout-update failed … Directory not empty`	`[agent]` post-deploy: watch the tail across the first cycles
57: an installing marker for the target lockfile with no deps key → `rebuild` with install on a `keep-deps` clean, `why` names the resume; an installing marker for another key → the conservative full clean; a landed deps key wins over a leftover installing marker	`[unit]` `src/execution/residentRefresh.test.ts::planRefresh: a timed-out install resumes instead of starting over` (3 tests; the resume case is red without the planner branch)
57 live: after a cycle whose install timed out, the next cycle logs `refresh: … rebuild (install resumes — …)` and ends `rebuild done in <ms>` with `/residents` `state: warm`; every failed cycle logs `refresh: cycle failed — <reason>` in `wrangler tail switchboard-resident`	`[agent]` watch the tail across the next lockfile-moving merge on a repo whose cold install nears the budget; the Worker wiring (`INSTALLING_MARKER`, `readRefreshDisk`, the log line) is `deploy/*` code covered by `npm run typecheck` in `deploy/cloudflare-resident`
58: `planThreadDeps` — reused tree → nothing; same key → seed only; different key + install command → seed AND install (`why` names the reconcile); different key, no install command → nothing; `threadDepsMechanism` → `reconcile` whenever the install ran, else the seed's mechanism or `none`	`[unit]` `src/execution/residentDepCache.test.ts::planThreadDeps …::*` (5 tests; red-verified: exports absent)
58: the residents page counts lockfile-diverged trees at the reconcile cost (`N more (0.98 GiB each) lockfile-diverged (reconciling)` on the fixture), never at the full deps	`[unit]` `web/src/pages/residents.test.ts::ResidentDetailPage::item 55: the Disk section …`
58 live: an attach on a branch whose lockfile differs from main answers `deps: "reconcile"`, `reconciled: true`; the resident log shows `deps /workspace/threads/…: lockfile differs from the warm checkout — seeded from the shared cache, install reconciles the delta` and `attach …: disk admitted — reconcile <GiB> projected`; `du -xsk` of the tree is a fraction of the checkout's deps and the attach completes well inside the bot's wait (the baseline it replaces: `install 1.94 GiB projected`, 5+ min, `/attach` fetch failed at ~272 s)	`[agent]` after deploy, attach a read-only probe thread (operator token) with `refHint` set to a lockfile-diverged branch; record `deps`, `attachMs`, the log lines and `du` in the receipt
61: `judgeRestoreProgress` — bytes still growing → wait however long (past the old 300 s); no growth for `RESTORE_STALL_MS` → `stalled` naming the idle span and the bytes so far; no first byte within the window → `stalled` (the clock runs from the start); a null sample neither counts as growth nor resets the clock; past the hydrate-wide deadline (`RESTORE_MAX_MS`, shared by the pending wait and both restores; an explicit `deadlineMs` caps a late-starting restore by the hydrate's clock, not its own) → `capped` even while bytes arrive; the cap sits under the 30-min stale-mid-flight window; no samples yet → wait	`[unit]` `src/execution/residentRefresh.test.ts::judgeRestoreProgress …::*` (6 tests; red-verified: export absent)
61 live: a container swap (an image-changing deploy) is followed by `mirror restore: <GiB> in <s> (<MiB/s>)` and `checkout restore: <GiB> in <s> (<MiB/s>)` in the tail and `/residents` `warm` with `lastRestore {at, ms}` — including when the checkout restore runs past 300 s (the shape that forced it: 481 s); a stall reads `down(r2-restore-failed: checkout restore stalled: no bytes written for <s> (<GiB> after <s>))`	`[agent]` after the next image-changing deploy, read the tail around the first attach; the Worker wiring (`restoreWithProgress`, `pendingRestores`, `dirKiB`) is `deploy/*` code covered by `npm run typecheck` in `deploy/cloudflare-resident`
61 live (rebuild after a failed restore): a `down(r2-restore-failed: …)` reason ends `— container stopped so the transfer cannot land on a rebuild`; the rebuild that follows provisions on a fresh disk (`uptime` inside the container restarts) and the deps store's entry files read 444 after it (`find /workspace/deps/*/node_modules -type f -perm -u+w \| wc -l` → 0 from a read-only attach) — the shape this rules out read 644 on 45 290 of 45 291 files	`[agent]` post-deploy, on the next restore that stalls (or `POST /debug {"op":"force-down"}` + `/rebuild` on a resident whose restore is mid-flight); the Worker wiring (the two `stop()` exits, provisioning's `await-restores`) is `deploy/*` code covered by `npm run typecheck`
61: `backupTransferMode` — the four presigned inputs, exactly the names the SDK reads; all non-blank → `presigned` / `localBucket: false`; any absent, empty or non-string → `local` / `localBucket: true` with the missing names in env order; unrelated env ignored	`[unit]` `src/execution/residentBackupTransfer.test.ts::backupTransferMode …::*` (red-verified: module absent)
61 live (presigned transfers): after the R2 token is put and the resident deployed, `GET /healthz` answers `backupTransfer: "presigned"` (before: `"local"` with `backupTransferMissing`); the next refresh cycle's `backup.create` events no longer carry `provider: "local-binding"`; a container swap then restores through the container (`mirror restore: …`, `checkout restore: …` logged) — including a checkout over 1 GB with no `isolate exceeded its memory limit` and no manual rebuild	`[agent]` post-deploy; first a forced `stop-container` on a small resident, then the largest resident's next swap
61: content-addressed snapshots — the checkout archive excludes exactly the top-level `node_modules`; an entry backup record is keyed `resident:depsBackup:<key>` and a non-key is refused; the archive TTL is at least 180 days; `planDepsMaterialization` restores a recorded backup before installing while a complete entry or a running install still wins; after a sweep the archives to drop are exactly the evicted keys that have one	`[unit]` `src/execution/residentDepsStore.test.ts::entry backups: content-addressed snapshots of the store …::*` + `::planDepsMaterialization (hit / join / install)::a recorded entry backup is restored before anything is installed; a hit or a running install still wins` (red-verified: symbols absent)
61: `planWakeDepsBudget` — plenty left → the install keeps its budget and the restore is judged against the hydrate deadline; less left than the install budget → the install is bounded by the remainder; under `WAKE_DEPS_MIN_MS` (≥ 60 s) or past the deadline → skip	`[unit]` `src/execution/residentRefresh.test.ts::planWakeDepsBudget …::*` (red-verified: function absent)
61: a presigned restore is extracted, never left mounted — the staging mount is a sibling of the target under `/workspace`; the extract script prefers `unsquashfs` from the downloaded `.sqsh`, falls back to `cp -a` out of the mount, unmounts (overlay, then the `<backupId>_*` lowers under the SDK's mount root; a non-id is refused as a glob), removes the archive, and renames the tree in LAST; the unmount-all script takes every fuse mount under `/workspace` and the SDK's mount root deepest first, then the leftovers; on a real filesystem the cp branch reproduces the tree with its modes and leaves no staging dir; the resident image's build asserts `unsquashfs` with the same `command -v` probe the script uses (`unsquashfs -version` exits 1 on squashfs-tools 4.5 with no filesystem named, and failed the 1.2.0 image build)	`[unit]` `src/execution/residentRestoreExtract.test.ts::residentRestoreExtract (item 61: the SDK's presigned restore MOUNTS the archive; the resident extracts it onto ext4)::*` (red-verified: module absent)
61 live (extraction): after the release, a wake logs `checkout restore: … GiB in … s` followed by `checkout restore: extract: unsquashfs in … s` (or `extract: cp` on the previous image), `mount` inside the resident shows no fuse mount under `/workspace`, `/var/backups` holds no `.sqsh`, a second hydrate on the same disk passes `clean-before-restore`, and the disk sample's `mirror`/`checkout` parts are the tree sizes again	`[agent]` after the release: read the wake's log lines, then `mount`, `ls /var/backups` and the disk sample from inside the resident
61 live (content-addressed snapshots): after the release, a refresh cycle's checkout `backup.create` is a fraction of the previous size (the tree without its node_modules); `POST /debug {"op":"deps-backups"}` lists one record per warm key with `deps: backed up <key8>` logged once and never again for that key; a forced `stop-container` restores mirror + tree and then logs `deps: restored <key8> from backup in <ms>` with no install; a rebuild's provisioning restores the warm key's entry instead of installing	`[agent]` after the release: read the cycle's `backup.create` size, `/debug deps-backups`, and the log lines across a forced `stop-container` and a rebuild
59: store layout — entries under `/workspace/deps/<key>`, `.complete` inside the entry; a non-hex or wrong-length key is refused as a path segment	`[unit]` `src/execution/residentDepsStore.test.ts::deps store layout …::*` (red-verified: module absent)
59: `planDepsMaterialization` — complete → hit; in flight → join; neither → install; complete beats a stale in-flight memo	`[unit]` `src/execution/residentDepsStore.test.ts::planDepsMaterialization …::*`
59: `depsInstallSemaphoreSize` is `nproc`, and 1 for anything unreadable (`""`, an error line, `0`, null)	`[unit]` `src/execution/residentDepsStore.test.ts::depsInstallSemaphoreSize …::*`
59: the scratch clone is `--shared --no-checkout` of the mirror + `checkout --detach <sha>`; the commit script moves node_modules, renames staging → entry, writes `.complete` LAST, refuses an install that produced no node_modules, keeps the scratch on adoption, and REPLACES a marker-less entry (crash debris between rename and touch) instead of nesting inside it; run on a real filesystem the entry is complete, the scratch is gone, a second racer for the same key leaves the winner intact, and a re-install over debris ends complete with only the fresh tree	`[unit]` `src/execution/residentDepsStore.test.ts::the install scratch tree and the store commit::*`
59: `deps-harden` strips owner write from every installed file; a scratch with no node_modules is refused (exit 1, names the scratch) when the commit has a lockfile and gets an empty node_modules owned by the build user when it has none (`NO_LOCKFILE_KEY` = sha256 of no input), after which the commit script completes the entry	`[unit]` `src/execution/residentDepsStore.test.ts::deps-harden …::*` (red-verified: symbols absent, then `chown` group on macOS)
59: listing parse (entry/complete/kib/used + leftovers, garbage ignored); eviction order never names a protected key, debris first, then coldest; `planDepsEviction` keeps `DEPS_STORE_MAX_UNREFERENCED` (1) warmest spares and removes the rest plus leftovers; nothing to do → empty plan	`[unit]` `src/execution/residentDepsStore.test.ts::store listing and eviction …::*`
59: `depCacheScript` takes the store entry as the node_modules source while build dirs still come from the checkout; without the option it is byte-identical to before	`[unit]` `src/execution/residentDepCache.test.ts::depCacheScript with a store-backed node_modules source …::*`
59: the tool-cache swap makes its copies writable again (`chmod -R u+w` between `cp -R` and `chown`) — store entries are owner-read-only	`[unit]` `src/execution/residentDepCache.test.ts::mutableCacheSwapScript …::the copy is made writable again …`
59: `duArgv` measures the deps store BEFORE the checkout; `assembleDiskSample` reads `parts.deps` from it	`[unit]` `src/execution/residentDiskBudget.test.ts` (the layout fixture carries `depsStoreDir`)
59: `ResidentExecutor.attach` parses a heartbeat-streamed binding; a streamed refusal's in-body `status` (503 mirror-busy, 409 needs ref) is handled exactly like the real status; a pre-validation 404 without `status` is still not-onboarded	`[unit]` `src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream …::*` (red-verified: status read from HTTP only)
59 live: a lockfile-changing merge on main — the resident log shows `deps: installing <key8> at <sha8> in /workspace/deps/.scratch-…`, then `deps: installed <key8> in <ms>`, the cycle's `checkout-update` + `deps-materialize` under the lock and `refresh: … rebuild done`; `ls /workspace/deps` (via a read-only attach) shows the entry with `.complete`; the checkout's `node_modules/<pkg>/package.json` has link count ≥ 2	`[agent]` post-deploy: watch the tail across the next lockfile-changing merge
59 live: two threads attach on the SAME new key while it installs — one `deps: installing`, both attaches answer `deps: hardlink` after it, neither logs a second install; a thread on an OLDER branch attaches in seconds via the entry main used to have	`[agent]` (post-deploy)
59 live: item 58's shape (ship on a lockfile-diverged branch while the refresh installs) binds — the card shows the wait, never `fetch failed`; the `/attach` response is heartbeat whitespace then JSON	`[agent]` (post-deploy)
59 live: after the deploy onto a pre-store disk, the first attach on main logs `deps: adopting the checkout's node_modules as <key8>` once, and `/residents` `disk.parts.deps` equals `du -sk /workspace/deps`; a pnpm resident's `.pnpm` survives adoption hardlinked (link count on a `.pnpm` file ≥ 2)	`[agent]` (post-deploy)
63: the collector's offsets, bounds, sanitization and skew rule	`[unit]` `src/execution/residentStepTrace.test.ts::createStepTrace::*`
63: the bot re-validates the trace at the parse and grafts it under its attach / command span	`[unit]` `src/execution/residentTrace.test.ts::*`, `src/execution/resident.test.ts::ResidentExecutor.attach over a heartbeat stream …::carries the resident's step trace on the binding…`
The watchdog firing and the refresh cycle are roots of the resident's own log — `resident.watchdog` with `resident.check` children, `resident.refresh` with its `resident.<step>` children (tracing.md item 25)	`[agent]` live: `wrangler tail switchboard-resident` across one cron minute
63 live: an attach answer's `trace` names `mutex_wait`, `clone`/`worktree-clone`, `install` with plausible durations that sum to about `attachMs`	`[agent]` after deploy: `POST /attach` for an onboarded repo with the operator bearer and read `trace` in the answer
22: the incarnation id comes from the random source alone, and one predicate judges every lease — alive for the current incarnation inside its budget (the last instant included), dead for another incarnation however fresh, dead past its expiry for the current one	`[unit]` `src/execution/residentIncarnation.test.ts::mintIncarnationId …::`, `src/execution/residentIncarnation.test.ts::leaseIsDead …::` (red-verified: module absent)
22: `takeMutex` — an empty row is taken and records the current incarnation, the step and the budget as expiry; a live holder of the current incarnation makes the caller wait for what is left of its budget; a holder of another incarnation or an expired one is taken over at once with the reason named and the dead lease handed back (its tree with it); the row carries a tree only when the taker names one	`[unit]` `src/execution/residentIncarnation.test.ts::takeMutex …::*`
22: `releaseMutex` releases only the holder's own row; a row another holder took over, or no row, is left alone	`[unit]` `src/execution/residentIncarnation.test.ts::releaseMutex …::*`
22: the Worker mints the incarnation id at the isolate start and in `swapIncarnation` alone; `setResidentState` clears the memos without swapping, so a lifecycle transition never kills the transitioning cycle's own lease	`[unit]` `src/execution/residentIncarnation.test.ts::the incarnation boundary in the resident Worker …::*`
22: a dead deps holder's paths are its scratch tree and its staging dir, recovered from the scratch path its lease names	`[unit]` `src/execution/residentDepsStore.test.ts::the install scratch tree and the store commit::a dead attempt's paths are its scratch tree AND its staging dir, recovered from the scratch path its lease names; any other path names no attempt`
22: the in-flight row — a refresh or hydration lease of the current incarnation inside its budget is in flight, a previous incarnation's is dead, a hydration past the stale bound is dead the way the memo was judged, clearing one kind leaves the other; the refresh lease outlasts the stale bound; the mutex, in-flight and deps-lease keys are distinct and a deps lease key is built only from a lockfile key	`[unit]` `src/execution/residentIncarnation.test.ts::the in-flight row …::`, `src/execution/residentIncarnation.test.ts::storage keys …::`
22: every step plan called twice over the facts its first run leaves behind is `done` the second time with exactly one command issued — the fetch (per cycle and ref), the install (a complete entry), the build (the markers, through the refresh planner), the snapshot (the record at the stamp), the restore (the ready marker) and the wake's dependency view	`[unit]` `src/execution/residentStepPlan.test.ts::*::called twice…` (red-verified: module absent)
22: the fetch, install, restore and deps-view plans each decide from their one fact — this cycle's fetch record for this ref (another cycle's or another ref's is not this step's work), the complete store entry, the ready marker naming the stamp's sha, the checkout holding its view (and whether the entry must be produced first)	`[unit]` `src/execution/residentStepPlan.test.ts::planFetchMirror …::`, `src/execution/residentStepPlan.test.ts::planInstallDeps …::`, `src/execution/residentStepPlan.test.ts::planRestore …::`, `src/execution/residentStepPlan.test.ts::planMaterializeDeps …::`
22: the build plan over the refresh planner — facts already at the target → done; HEAD, deps and build markers all at the target → done; key unchanged → run without install on a keep-deps clean; key changed → run with install on a full clean, a resumed install keeping its partial tree	`[unit]` `src/execution/residentStepPlan.test.ts::planBuild …::*`
22: the snapshot step — no record or a record at another stamp → run, the record at this stamp → done; the commit decision commits over an unchanged record (or none on both reads) and answers `superseded` when the record appeared, vanished or changed since the read, never a throw	`[unit]` `src/execution/residentStepPlan.test.ts::planSnapshot + snapshotCommitDecision …::*`
22 live: `/debug info` shows the resident's `incarnation`, `mirrorMutex` null while nothing holds the mirror and `leases.refresh` naming that incarnation with `step: "refresh"` during a `refresh-now`, null again once the cycle ends; a Worker deploy mid-cycle leaves a `leases.refresh` whose `incarnation` differs from the new `incarnation`, and the next cycle runs without waiting	`[agent]` `POST /debug {"op":"info"}` before, during and after `POST /debug {"op":"refresh-now"}`; then `wrangler deploy` during a cycle and read `info` again

Resident repo environments ​

Behavior ​

Validation criteria ​

Resident repo environments

Behavior

Validation criteria