OpenSwitchboard

Appearance

Execution & sandboxes ​

Tools never touch the bot host: every bash/read_file/write_file runs through an Executor. Production uses per-thread Cloudflare Sandboxes behind the proxy Worker; a thread's follow-ups reuse its workspace.

  • Code: src/execution/ (executor, factory, cloudflareSandbox, sandboxErrors, sandboxEnv, e2b, resident, githubApp, sandboxKeepalive), deploy/cloudflare-sandbox/worker.ts + wrangler.jsonc + Dockerfile + docker-wrapper.sh
  • Docs: Execution and trust, AGENTS.md invariants 5, 6
  • Tests: src/execution/factory.test.ts (per-agent provisioning + resident selection), src/execution/resident.test.ts, src/execution/shellQuote.test.ts, src/execution/githubApp.test.ts, src/execution/githubIdentity.test.ts (the identity the process acts as), src/execution/bashTimeout.test.ts, src/execution/executor.test.ts, src/execution/e2b.test.ts, src/execution/cloudflareSandbox.test.ts, src/execution/sandboxErrors.test.ts, src/execution/sandboxEnv.test.ts (the body-only env reader the Worker uses), src/execution/sandboxKeepalive.test.ts (the exec keepalive + static guards that the sandbox Worker wires it, renders every failure through thrownText, reads the env map through envFromRequest from the body alone while the executor sends only the body, and that wrangler.jsonc rolls out in one wave), src/deploy/wranglerTemplate.test.ts (the sandbox template's instance type), src/deploy/sandboxDocker.test.ts (static: the sandbox image installs the Docker engine and the lazy-start wrapper, and the wrapper's shape); the deeper sandbox behaviors are infrastructure-dominated — the live checks below are their proof.

Behavior ​

  1. Per-thread isolation: one workspace/sandbox per threadKey; follow-ups reconnect. Losing a sandbox degrades gracefully — repos re-clone, nothing else is lost. A thread's container stays warm for 5 minutes of idle (SANDBOX_SLEEP_AFTER, src/execution/sandboxKeepalive.ts; the SDK default is 10 on 0.12.x, was 20 on 0.3.x): a follow-up inside that window lands on the same workspace, a later one gets a fresh container and re-clones — legible, and otherwise unchanged behaviour. Idle means idle: a running command never counts toward it (item 2).

  2. Command timeout is shell-level and structured: every /exec runs under coreutils timeout -k 10 <secs> inside the sandbox — 280s by default, raised per call by the bash tool's timeoutMs (item 12) up to the shared 20-min ceiling; a deadline kill is a genuine exit 124 (137 normalized to 124) with an actionable stderr line naming the limit and the timeoutMs knob. The SDK's COMMAND_TIMEOUT_MS (1240s, Dockerfile — 40s above the ceiling; changing it needs a container rebuild) is a pure backstop. The SDK's own timeout surfaces only a useless generic error, which is why the deadline is shell-level. Invariant — one in-flight exec never outlives the container's activity timeout: the Durable Object renews the activity clock every 60 s while a command runs (SwitchboardSandbox.exec → withActivityKeepalive, EXEC_KEEPALIVE_INTERVAL_MS), so sleepAfter (5 min) is idle time only and the shell-level timeout is the one deadline a command can hit. Why: the @cloudflare/containers base class renews the clock once per proxied fetch, BEFORE the fetch, and its alarm loop SIGTERMs the container the moment the clock reads expired with no in-flight awareness. Without the keepalive, a command whose budget equals sleepAfter is SIGTERMed at the budget exactly — before coreutils timeout can fire — the SDK answers Command execution failed (its wrapper for the container's Session terminated), and the next command starts a new container with /workspace empty.

  3. Heartbeat streaming: /exec responses send headers immediately, a whitespace heartbeat every 15s, then one JSON document — no hop ever sees an idle connection (idle drops lost results in transit). Failures arrive in-body in a dual shape (error + exitCode: 127/stderr) so old/new executors both render them; in-body errors are never blindly retried. The failure text is never empty: both Worker catches (the /exec stream and the /read//write handler) render the thrown value through thrownText (sandboxErrors.ts) — the SDK's own message when it has one, else sandbox exec failed with no message from the SDK (<name>[, code <code>] | no error name); the container may still be running a previous image while a Worker/image rollout is in progress — retry in a minute; the classifiers (isFleetBusyError, isRecycleError, recycledMidCommandMessage) keep reading the raw shape. On the other side, a success body has no error key at all, so the executor treats a PRESENT-but-empty error as the Worker's failure shape with its text missing — ExecInfraError, item 9 — never as a command exit. The failure this closes: a SandboxError built from a body with no message has message === ""; a shape.message ?? String(err) fallback keeps it (?? fires only on null/undefined), a truthy check lets {error: ""} through as a plain exit 127, the health tracker counts a success, and the model reports its shell "down". The document also carries durationMs, the command's wall time in the sandbox (tracing.md item 19); the bot does not read it yet.

  4. Session recovery: on SDK 0.3.x a container restart under a live Durable Object orphaned the SDK's cached session, and the Worker detected the stale-session error, reset the cached session, and retried once. SDK 0.12.x recovers by itself — the session id is persisted in DO storage, cleared in onStop, and the container recreates a missing session on the next exec — so resetDefaultSession is a fence that should never fire; it stays until the live container-restart row below retires it. The Worker's one remaining self-retry is a booting container (Container is starting. Please retry in a moment., isContainerStarting): nothing ran, so the same request is re-sent after 3 s. A failure whose command MAY have run — a session shell that exited, a container that stopped under the call — is never re-run; item 9 names it a recycle instead. The 0.3.x file-op retry (Failed to read/write file taken as possibly stale) is gone: 0.12.x file errors carry their real cause.

  5. GitHub identity: a GitHub App mints 1-hour installation tokens on demand, injected into sandboxes as GH_TOKEN — never stored on the bot host. Least-privilege by toolset: the token is minted scoped to what the agent may do — a readonly agent's sandbox (the review agent) gets a read-scoped token — exactly contents:read, pull_requests:read, issues:read (for the github_issue_list/get tools of github-tools.md), actions:read (workflow runs, jobs, logs and the Actions cache list: gh run list/view --log, gh cache list), checks:read (check-runs and check-suites: gh pr checks) and metadata:read (GitHub requires it on every installation token), every value read — so it can clone a private repo and read its PRs, issues and CI results but physically cannot comment/review/push/re-run from inside, while a full agent (coding) gets the write-scoped token it needs (resolveGithubToken(scope) in githubApp.ts, chosen by githubEnvs in factory.ts). The read set is an explicit subset of the installation's grant, never the grant itself: GitHub refuses a mint that asks for a permission the App does not hold, so every name in it must appear in the required permissions, and statuses (the legacy commit-status API — Actions reports through check-runs, so an Actions-built repository never needs it; a repository whose CI posts commit statuses would need it added to the set for those rows to show in gh pr checks) and workflows (a write-only grant) are deliberately not requested. Required app permissions: Contents, Pull requests, Issues (all RW) + Actions/Checks/Statuses (read), Workflows (RW). PRs are authored as <app>[bot]. The installation must cover every repo agents are asked about — with repository_selection: "selected", an uncovered repo 404s on every gh call (this repo itself included). Prefer "All repositories" on the org, or audit the selection when adding repos. Never in the command text: the sandbox Worker passes the credential through the SDK's per-exec env option (0.12.x applies it to that one command and restores the environment afterwards), so the command string the SDK logs never carries GH_TOKEN — 0.3.7 ignored the option and the Worker used an inline base64 export prefix that put the live token into every Command executed log line. The credential rides ONLY in the request body: CloudflareSandboxExecutor.call sends the resolved map as env: { NAME: value } in the JSON body on every route (/exec, /read, /write — one shape everywhere; the Worker uses it only for /exec), and the Worker reads it through envFromRequest (src/execution/sandboxEnv.ts, node-free, bundled into the Worker), which upper-cases names, keeps only shell identifiers (^[A-Z_][A-Z0-9_]*$, the resident's rule) with string values, and drops everything else — an env that is not a plain object yields nothing, never a throw; request headers are never read. Why: Workers Logs record every invocation's request headers and redact them by a NAME heuristic only; bodies are not recorded. Measured on a header transport: x-env-gh_token showed as REDACTED while a probe header x-env-PROBE_VAR: hello-from-env-option was visible in clear in the POST /exec invocation event — a variable with an innocent-looking name leaks. There is no header fallback: both ends use the body alone and no x-env-* header is sent or read (a body-only bot meeting a header-only Worker was a one-release transition — deploy all deploys the bot before the sandbox Worker, src/deploy/plan.ts — closed once every deployed sandbox Worker read the body). Freshness — a command that starts on a token finishes on it: the credential is resolved per COMMAND, never captured per run — CloudflareSandboxExecutor and E2BExecutor take resolveEnvs() and call it on every exec (the sandbox Worker already injects the env inline per command, so nothing persists in the sandbox) — and the mint cache reuses a token only while it has at least TOKEN_REUSE_MARGIN_MS (the 20-min command ceiling + 5 min) left, so the longest possible command cannot outlive the token it started with. Under a "5 minutes before expiry" rule the cache can hand a nearly spent token to a run whose first command then runs the full 20-min ceiling; the token expires under it, every later command in the run gets 401 Bad credentials, and the run produces no verdict.

  6. A sandbox Worker/image rollout never looks like a dead or silent sandbox. A bot deploy has a ~10–15 min deaf window (container image rollout); Socket Mode events during it are lost, not redelivered. A sandbox-Worker deploy rolls containers too, and a Worker and its image deploy as two artifacts: until the rollout finishes, the NEW Worker code can be handed a container instance still running the PREVIOUS image (Cloudflare's own words). In-flight thread sandboxes on old instances are SIGTERMed in any rollout mode (the grace period is 0) — recovered by (4), disk lost, named by item 9. What is closed and named is the window a NEW thread could fall into: (a) deploy/cloudflare-sandbox/wrangler.template.jsonc sets rollout_step_percentage: 100, so old-image instances are replaced in ONE wave and the window shrinks from minutes (the platform default [10, 100]) to seconds; (b) SwitchboardSandbox.onStart() compares the container's /api/version (client.utils.getVersion(), "unknown" on any failure) with the Worker's own @cloudflare/sandbox pin (imported from its package.json — the pin check:sandbox-pair holds equal to the image tag) and logs sandbox.version-skew container=<v> sdk=<pin> at warn — LOG ONLY, never destroy() there: onStart runs inside blockConcurrencyWhile, destroy() is unbounded and coalesced callers hang until eviction, a fresh placement during a gradual wave can land on the old image again, and the healthy-but-not-running state after a destroy takes the SDK's stale-state path, which can ctx.abort() the DO; (c) a command that reaches a skewed instance and fails without a message is named by thrownText (item 3) with the rollout hint — the Worker's exec is super.exec under the keepalive and retries nothing of its own. The shape this closes: a release deploys the sandbox Worker and its image together, a thread created seconds after the upload lands on a previous-image container, and every command fails with an empty message rendered as a silent exit 127 until a rollout wave happens to replace the instance.

  7. Per-agent provisioning: executor selection is context-aware — makeExecutor(opts, { threadKey, agent, repo?, ref? }) sees the resolved agent and the inferred repo/ref for resident environments, and returns { executor, note? } (the note is the named fallback reason the dispatcher shows on the status card). Agents declare the resources they need via AgentDef.resources; an agent that declares no repo (general) gets a null executor: no sandbox or workspace is created or reconnected, and no sandbox credential is required. Only repo-requiring agents (coding, review) provision the configured backend. A tool call reaching a null executor surfaces as a legible tool error (a wiring bug: tools without declared resources), never a crash.

  8. Resident backend: with execution.resident {baseUrl, tokenEnv} configured AND a resolved target repo (ctx.repo — populated by the repo resolver), the factory probes the resident's /status (short timeout; transport failures negative-cached 30s) and selects ResidentExecutor only on warm; any not-warm state falls back to the per-thread backend above with a named note (resident <state> (<reason>) — using fresh sandbox — a degraded resident is loud). not-onboarded (404) also falls back per-thread, but carries a note too — repo not onboarded as a resident — running in a cold per-thread sandbox; onboard it (\repo onboard <owner/name>`) for a warm, deps-ready environment` — so the user can tell coding ran cold instead of on a warm resident (never a silent surprise). Full contract and criteria in resident-repos.md.

  9. Infra-failure classification: an Executor returns a normal nonzero command exit as ordinary output (exit <n>: …, never a throw), but throws ExecInfraError when the exec transport itself fails — the sandbox unreachable (network), an HTTP/worker error, the sandbox's in-body "Command execution failed" / exitCode-127 signal, a PRESENT-but-empty in-body error ("" — the Worker's failure shape with its text missing, item 3; a success body carries no error key, and a bare exit 127 with no output and no error key stays an ordinary command result — foo 2>/dev/null is a legitimate silent 127), or a worktree still unavailable after a re-attach. When that in-body failure says the container went away under the command — a typed 0.12.x error (SessionTerminatedError, OperationInterruptedError, matched by NAME because the Durable Object RPC boundary keeps name/message and drops the prototype; isRecycleError) at any elapsed time, or a recycle-shaped text (0.12.x: Session '…' shell exited (exit code: n), The sandbox container stopped while the operation was pending., and The sandbox was destroyed while the operation was pending. — the disconnect a destroy() under a pending call produces; 0.3.x: Command execution failed, Session terminated, Session '…' not found) more than a minute into the attempt (per attempt — a retry gets its own clock) — the sandbox Worker rewords it as sandbox recycled mid-command after <n>s — the container was replaced and /workspace is empty; re-clone before continuing (<original>) (recycledMidCommandMessage) — still exit 127, never a faked exit 124: it IS an infra failure and the workspace really is gone, so the model re-clones instead of shortening a command that was never the problem. Both remote executors do this (CloudflareSandboxExecutor, ResidentExecutor); ExecInfraError extends Error, so message/instanceof Error callers are unchanged. This is the seam the runner's fail-fast reads (see run-loop.md — ExecHealthTracker + MAX_CONSECUTIVE_INFRA_FAILURES), so a wedged sandbox is told apart from a command the agent should keep handling. Two things are deliberately NOT infra: a resident's named runtime-replaced answer (a deploy swapped the resident isolate under one command — resident-repos.md item 43) is returned to the model as ordinary output the first time; only a second one with no successful op between becomes ExecInfraError, so a single deploy can never trip the breaker. And a full sandbox fleet (fleet-busy, item 14) is capacity: the executor waits for a slot and, when the wait is spent, throws ExecCapacityError — a separate class the tracker neither counts nor resets on — so a fleet at max_instances can never read as a dead sandbox. Every ExecInfraError the resident executor throws is classified for the spans around it (tracing.md item 2): a failed request transport (timeout when the deadline fired), a non-200 http with the status as the code, a worktree still gone after re-attach infra/attach, a repeated runtime replacement infra/runtime-replaced, an in-body exec error infra; a GitHub API failure is http with its status.

  10. Container toolchains: the sandbox image (deploy/cloudflare-sandbox/Dockerfile) carries git + gh + the JS toolchain (Node 22 from the cloudflare/sandbox:0.12.9 base, asserted ≥22 at build time — the 0.3.x base shipped Node 20 and needed a NodeSource upgrade layer; image and SDK move together) plus npm and pnpm installed globally with npm; the resident image (deploy/cloudflare-resident/Dockerfile) carries git + Node 24 + npm + bun + pnpm + yarn (classic; a Berry repo bootstraps through its committed yarnPath release) — every package manager the onboard-time detectCommands (resident-repos.md item 52) can name, so a detected install command always exists where it runs. Both images therefore satisfy this repo's engines: node >=22, so a review that falls back off the resident onto a fresh sandbox validates a Node-22 repo with the full suite loadable (on Node 20 the sandbox failed to load 15 suites: undici@8 needs Node 22's webidl.util.markAsUncloneable). pnpm was added to both so coding/build runs in pnpm repos no longer hit pnpm: command not found. Every toolchain version is pinned exactly and asserted at build time, and src/deploy/imagePins.ts + imagePins.test.ts fail the suite if a floating tag returns (resident-repos.md item 53): both images installed pnpm@latest, so the pnpm major tracked the last image build rather than a commit — a routine rebuild moved it 10 → 11, pnpm 11 stopped reading package.json's pnpm field, and a repo keeping its overrides/patchedDependencies/onlyBuiltDependencies there fails --frozen-lockfile. pnpm sits on the 10 line on purpose (10.x reads that field; pnpm ≥10 self-manages packageManager, so the pin is a floor a repo can raise for itself — which also means the version baked in is not necessarily the one that runs, and a repo with a packageManager field fetches its own at install time). Changing either image requires a container rebuild + redeploy to take effect.

  11. Hard-stop cancellation: Executor.exec(command, { signal }) carries the run's hard-stop AbortSignal. LocalExecutor kills the child process (the abort surfaces as an exit … line, never a throw); ResidentExecutor and CloudflareSandboxExecutor join it with their per-call deadline (execDeadline) so the bot-side request drops at once — the command inside the container keeps running until the release: the resident's /detach force kills the thread user's processes before evicting (resident-repos.md item 16a); the thread-sandbox Worker has no kill route, so there the command runs to its own timeout. E2BExecutor ignores the signal (its command API cannot cancel) — safe because the runner races every tool await against the signal and stops waiting regardless — and gains release(mode): "always" (a read-only run or a hard stop) kills the micro-VM, the only way to end an uncancellable command; "if-clean" keeps it for the thread (the idle timeout reclaims it). The dispatcher releases with "always" after any hard stop, so a hard stop always frees the resident pool user (/detach force) or the E2B sandbox. Every remote exec has a bot-side deadline of budget + EXEC_CALL_MARGIN_MS — ResidentExecutor and CloudflareSandboxExecutor run every send (headers AND the streamed body — /exec answers HTTP 200 at once and heartbeats until the outcome, so the body read is where a dead sandbox's wait sits) under execDeadline(budget + 30 s, signal), the budget being the command's clamped timeoutMs for /exec and BASH_TIMEOUT_MS for a file op; a Worker that never answers is an infra failure — ExecInfraError naming both numbers (sandbox worker /exec gave no answer within 330s (command budget 300s + 30s margin) — the sandbox may be gone; the command may still be running in it), which fail-fast (item 9) counts — never an indefinite wait. The deadline is per send: a fleet-busy wait (item 14) does not eat into it. execDeadline is a plain timer, not AbortSignal.timeout (Node runs that one on an internal timer no test can drive). Without it a run can wait indefinitely on one /exec: the sandbox container gone (There is no container instance…), the Worker still heartbeating, a client that passes only the hard-stop signal and reads the body with no deadline, and a runner that checks its wall clock only between steps — nothing fires until an operator hard-stops it.

  12. Per-call bash timeout (timeoutMs): the bash tool's input schema carries an optional integer timeoutMs, Claude Code-style — the tool description names the default and the max, which is how the model learns to ask for more time (a long probe dies at a flat 5-min cap, and npm test/npm run build brush it). Policy lives in ONE module, src/execution/bashTimeout.ts, bundled into both deploy Workers: default BASH_TIMEOUT_MS (5 min, unchanged), ceiling BASH_TIMEOUT_MAX_MS (20 min), floor 1s. Clipped to the run: the ceiling alone does not stop one command from eating a run (20 min is 80% of the review agent's 25-min budget — one first command can take all of it and leave ~4 min), so the runner hands tools the run's remaining wall clock (ToolContext.remainingMs, on the runner's own clock) and the bash tool caps every budget — requested or the 5-min default — at remaining − RUN_DEADLINE_RESERVE_MS (60 s), appending a [timeout clipped to …] line so the model knows why the command ended early; inside the reserve it refuses outright (exit 124: run budget exhausted …) instead of starting a command that cannot finish (bashBudgetWithinRun). The documented clamp rule (clampBashTimeout): a finite number is truncated and clamped into [1s, 20 min] — so 0/negative run at the 1s floor and a 25-min ask runs at 20 min — and anything else (absent, NaN, a string) falls back to the 5-min default. Every executor honors it: LocalExecutor maps it onto execFile's timeout (maxBuffer kept); E2BExecutor onto the SDK's commands.run({timeoutMs}); ResidentExecutor puts it in the /exec body only when the caller asked for one (an older resident sees the body it always did — attach's readonly/sha convention) and stretches its HTTP wait to timeoutMs + EXEC_CALL_MARGIN_MS (30s) so the resident's own streamed exit-124 answer wins the race against the client's transport deadline; CloudflareSandboxExecutor does the same body-only-when-asked, and the sandbox Worker sizes its coreutils timeout from it. Servers clamp with the same bounds and never trust the client's number (resident handleExec, sandbox Worker /exec — both via the shared clampBashTimeout). A deadline kill is exit 124 with a line naming the limit that fired and the timeoutMs knob (max included), so the model self-corrects instead of reading a generic abort. The resident's deterministic /op path keeps the flat 5-min budget — it has no caller knob. Back-compat & deploy ordering: no timeoutMs → byte-for-byte today's behavior; a resident Worker predating this change either ignores the field (very old — command still capped at its 5-min budget, answered legibly as exit 124 while the client clamp bounds the wait) or 400-rejects a value above 5 min (the immediate predecessor's parsePositiveInt), so the resident deploys before the bot, as always. Same for the sandbox Worker + its Dockerfile backstop (container rebuild).

  13. Every Worker reports the commit it was built from, and nothing is hand-bumped (without it a deploy's receipt has to be assembled from wrangler versions list plus a container image digest, because a Worker's /healthz cannot say which build is at the edge). The bot always could: npm run deploy writes build.json from the tree's HEAD (deploy/cloudflare/write-build.mjs) — and the release's publish-image job writes it from the release commit before building the bot's image, so a container from the published image (release-and-deploy.md items 21, 25) is stamped too — the image COPYs it, /healthz serves build: {commit, builtAt}, and deploy all's live gate compares it to the commit it deployed (slack-channel.md item 8). The three Worker SCRIPTS now do the same, and the resident's hand-edited const BUILD_MARKER = "perf53" is gone: its comment said "bump on every deploy-worthy change", it was bumped three times in total (gc51 → head52 → perf53) and then went unbumped across five deploys — so it could neither prove a deploy nor expire a stale test override, which is exactly what resident-repos.md item 49(c) (ignored:"stale-build <marker>") depends on. Mechanism: deploy/bin/build-stamp.mjs replaces bare wrangler deploy in each Worker-script npm run deploy, derives {commit, builtAt} from the tree (a -dirty suffix when it is not clean, because wrangler bundles the TREE and not the commit), and passes them as wrangler deploy --define SWITCHBOARD_BUILD_COMMIT/SWITCHBOARD_BUILT_AT; src/deploy/buildStamp.ts is the only reader (injectedBuildStamp), gating both reads behind typeof because an un-stamped bundle's identifiers do not exist at all — that path answers commit: "unknown" instead of throwing inside /healthz. --define and not --var: a CLI --var may replace the vars a Worker's wrangler.jsonc declares, and the resident's STATE_WORKER_URL is load-bearing (the watchdog records firings through it). A generated FILE cannot serve a Worker script either — committed, a deploy would dirty the tree (which deploy all refuses); gitignored, tsc and a fresh clone break on the missing import. Surfaces: resident GET /healthz → {ok, build} (was {ok, u}); memory GET /healthz → {ok, build, features}; sandbox gains GET /healthz → {ok, build} behind the same SANDBOX_TOKEN bearer as every other route — a build stamp does not justify a new unauthenticated surface on the Worker that proxies command execution.

  14. A full fleet is capacity, not a dead sandbox (fleet-busy): the per-thread sandbox Worker runs at most max_instances container instances (deploy/cloudflare-sandbox/wrangler.template.jsonc, 25 — billed per AWAKE instance, with ~15× headroom on the account's containers limits at the item-16 instance size); the thread past that ceiling gets no instance, so @cloudflare/sandbox 0.3.x cannot create a session and throws the bare Failed to create session: 503 (the platform's own wording is "no container instance that can be provided…"; 0.12.x throws a typed ContainerUnavailableError, code CONTAINER_UNAVAILABLE, keeping that wording as its message — the Worker takes the type by name first, isFleetBusyError, and the text second). Nothing ran and nothing is broken. The contract: (a) the Worker recognizes those messages with the shared isFleetBusy (src/execution/sandboxErrors.ts, node-free, bundled into the Worker) and NAMES the condition — /exec finishes its streamed body with {error: "fleet-busy: <explanation> (<sdk message>)", reason: "fleet-busy", stdout: "", stderr: <same>, exitCode: 127} (the item-3 dual shape kept for older executors), /read and /write answer HTTP 503 {error, reason: "fleet-busy"} — the same named-reason precedent as the resident's mirror-busy; re-sending is safe by construction because session creation fails before any command or file op starts. (b) The executor (CloudflareSandboxExecutor.call) matches the machine token only (reason === "fleet-busy", in-body or on a 503 — an OLDER Worker's bare message stays an ordinary in-body error, so a bot deployed ahead of its Worker changes nothing) and re-sends the IDENTICAL request — same route, body, headers, the envs resolved once per call — after 10 s, 20 s, then 30 s, until the total wait reaches min(the command's own timeoutMs — the 5-min default for a file op —, FLEET_BUSY_WAIT_MAX_MS = 5 min); a hard stop ends the wait at once. Every OTHER in-body error is still ExecInfraError after exactly one send. (c) When the wait is spent it throws ExecCapacityError (executor.ts; extends Error, NOT ExecInfraError) whose message names the wait: sandbox fleet busy — no free per-thread sandbox after waiting 300s (the fleet's max_instances is reached); try again in a few minutes. ExecHealthTracker leaves its consecutive count untouched on it. (d) The runner (run-loop.md item 7) does not count it toward fail-fast, emits a typed run_note of kind fleet_busy (parsed by runEventLines, a medium infra_failure finding in the friction analyzer), and returns the tool result as an error text the model can act on: ⏳ Sandbox fleet busy — <message>. Retry the command in a minute or finish with what you have. The shape this closes: a burst of cold runs exhausts max_instances, one run gets two identical Failed to create session: 503 errors seconds apart, the breaker reads them as a wedged sandbox, and the run aborts with a "dead sandbox" diagnosis for a fleet that was merely full. Deploy order: the sandbox Worker before the bot — an old Worker's plain in-body error still reads as infra (today's behaviour, unchanged).

  15. Every executor operation is an exec.* span (tracing.md item 17). The runner hands each tool call a TracingExecutor (src/execution/tracingExecutor.ts, a Decorator over the run's health-tracked executor): exec → exec.exec, readFile → exec.read_file, writeFile → exec.write_file, and exec.release / exec.move_to only when the wrapped executor has them — each a log-only span under the tool call's span carrying the backend and the per-call timeoutMs, never the command, the path or the output; a throw ends the span error and propagates unchanged. ExecutorSelection.backend names where the run's commands execute — local (the local executor, and the null executor an agent without a repo gets), e2b, sandbox (the per-thread Cloudflare sandbox), resident — so the spans of a resident-backed run and a cold run tell apart. The wrapper hands each exec.* span to the inner executor as opts.span (ExecTraceOptions on every Executor method), so a resident's or a sandbox's HTTP call is that span's http.client child (tracing.md item 21).

  16. The cold sandbox is the platform's largest predefined instance type: deploy/cloudflare-sandbox/wrangler.template.jsonc sets instance_type: "standard-4" — 4 vCPU / 12 GiB / 20 GB; a custom type can be no larger. A cold run clones, installs and checks a whole repository in one container with nothing warm, so the size is decided by the largest single command a cold thread must be able to run, not by the typical one: a large monorepo's typecheck alone needs more than 8 GiB (its own CI notes put the root program near 8 GB and each sub-program near 4 GB), and the previous standard-3 (2 vCPU / 8 GiB / 16 GB; a live probe read nproc 2, 8189 MiB, 15 GB on /) could not run it at all. The cost is per AWAKE instance — a sandbox sleeps after 5 idle minutes (item 1) — so the step is paid per active run-hour: 12 GiB against 8 of memory, and vCPU only while the cores are busy; the fleet ceiling (item 14) keeps ~15× headroom on the account's containers limits (6 TiB memory / 1500 vCPU against 25 × 12 GiB and 25 × 4 vCPU — the vCPU line is the tighter one). The resident reaches the same 4 / 12 GiB / 20 GB from the other direction (sixteen threads sharing one container, resident-repos.md item 55).

  17. Docker in the cold sandbox: the sandbox image installs Debian's docker.io engine and iptables in its apt layer, and docker on PATH is a wrapper (deploy/cloudflare-sandbox/docker-wrapper.sh, installed as /usr/local/bin/docker ahead of /usr/bin/docker) that starts the engine on first use: when /var/run/docker.sock is absent or docker info fails it enables net.ipv4.ip_forward (the default bridge needs it for containers to reach the network), starts dockerd in its own session (setsid -f, log at /var/log/dockerd.log), waits up to 40 s for docker info to answer — failing with a message that names the log — and then execs the real client with the caller's arguments. Every later docker call in the same warm container finds the engine running and passes straight through. Why in the image: a daemon installed at run time is lost with the container after its idle sleep (item 1) and costs minutes per run. Why its own session: every /exec runs under timeout -k 10 … bash -c (item 2) and its process group is reaped when the command returns, so a daemon started with nohup … & dies with the command that started it; a setsid daemon survives to the next one. Why it is safe to ship: the sandbox already runs as root with full capabilities inside its own Firecracker microVM, so a container a run starts lives inside that run's VM and shares its blast radius — the engine adds no privilege the run did not already have, and no privileged flag, capability or Worker config changes with it. What it unlocks: nektos/act in its default per-job-container mode, and repositories whose tests need containers (testcontainers, compose, image builds), which a cold sandbox could not run before.

Validation criteria ​

CriterionProof
13: resolveBuildStamp — an injected commit + timestamp survive; a -dirty suffix is preserved; a missing/blank/non-string commit is unknown and no timestamp is invented; both are trimmed[unit] src/deploy/buildStamp.test.ts
13: injectedBuildStamp with no --define present returns {commit:"unknown"} rather than throwing on the undeclared identifiers[unit] src/deploy/buildStamp.test.ts
13: buildStamp/defineArgs — the stamp carries the commit and build time, marks a dirty tree, and emits defines whose values are JS string LITERALS; a value containing a quote is escaped, not injected as code; the identifiers match the reader's[unit] src/deploy/buildStamp.test.ts
13: buildId — the identity a stored artifact compares against is the commit PLUS the build time, so two builds of one dirty tree and a redeploy of the same clean commit are told apart, while sibling isolates of one deployed version agree; no timestamp → the commit alone[unit] src/deploy/buildStamp.test.ts
13: spawnOutcome — exit 0/1 pass through; a spawn error and a SIGNAL kill (which reports status: null) each become exit 1 with the reason NAMED, so an interrupted deploy is not read as a failed one[unit] src/deploy/buildStamp.test.ts
13: memory /healthz carries build beside features, unknown in an un-stamped test bundle[unit] deploy/cloudflare-memory/worker.test.ts + runs.test.ts (workerd)
13: the define reaches all three bundles — wrangler deploy --dry-run --outdir per Worker contains the HEAD sha and zero leftover SWITCHBOARD_* identifiers (esbuild folds typeof … === "string" to true), and the resident's dry run still binds env.STATE_WORKER_URL (the --var hazard avoided)[agent] In each Worker directory run wrangler deploy --dry-run --outdir <dir>; grep the bundle for git rev-parse HEAD and for leftover SWITCHBOARD_ identifiers; the resident's dry-run output must still list STATE_WORKER_URL under its bindings.
13 live: after deploy each Worker's /healthz reports the deployed commit — resident and memory unauthenticated, sandbox with the bearer[agent] After a deploy, curl each Worker's /healthz (the sandbox with its bearer) and compare build.commit to the deployed commit.
≤280s commands succeed; >280s return exit 124 + timeout stderr[agent] In Slack: @switchboard agent:review run \sleep 45 && echo OK` then `sleep 320 && echo NO`, report raw results. Expect OKthenexit 124: command timed out in the sandbox after 280s…`.
Long commands stream heartbeats (no fetch failed)[agent] Covered by the same run — without heartbeats the 320s case surfaces as fetch failed.
Session recovery after container replacement[agent] Run a command in a thread, wrangler deploy the sandbox worker (replaces containers), run a follow-up in the same thread — it must succeed (repo re-clones), not 500 Session not found.
Thread workspace reuse[agent] agent:coding create file /workspace/marker.txt, then follow-up cat marker.txt — must print the content (same sandbox), unless the sandbox expired (then a legible re-clone story, not a crash).
2: the keepalive renews the activity clock once per interval while a command is pending, makes zero renew calls for a command that finishes inside one interval, stops on resolve AND on reject (the rejection propagates untouched), and swallows a renew that throws or rejects[unit] src/execution/sandboxKeepalive.test.ts::withActivityKeepalive::makes zero renew calls for a command that finishes inside one interval, ::renews once per interval while the command is pending, ::stops renewing once the command resolves, ::stops renewing once the command rejects, and the rejection propagates untouched, ::swallows a renew that throws or rejects — the command's own result still comes back
2: the renew interval (60 s) sits inside sleepAfter (5 min) with room to spare, in the Container class's own <n>[smh] grammar[unit] src/execution/sandboxKeepalive.test.ts::keepalive constants::the keepalive interval fits inside sleepAfter with room to spare, so a renew always lands before expiry, ::parseSleepAfterMs reads the Container class's s/m/h grammar and rejects anything else
2: the sandbox Worker WIRES the keepalive — sleepAfter = SANDBOX_SLEEP_AFTER, exec runs under withActivityKeepalive renewing via renewActivityTimeout, the /exec failure path classifies typed errors first and calls recycledMidCommandMessage (static guard; the Worker itself cannot run under vitest)[unit] src/execution/sandboxKeepalive.test.ts::sandbox Worker wiring (static)::the Durable Object's sleepAfter is the shared constant and exec runs under the keepalive, ::the /exec failure path names a mid-command recycle, typed errors first
9: the recycled-mid-command wording fires only for a recycle-shaped failure more than a minute into the attempt, or at any time for a typed recycle error; an early failure of any text, and a late failure of any other text, is returned unchanged; both the 0.3.x and the 0.12.x recycle texts are recognized[unit] src/execution/sandboxKeepalive.test.ts::recycledMidCommandMessage::leaves an early failure with any message alone, ::leaves a recycle-shaped failure alone inside the first minute — that is a real startup failure, not a recycle, ::names the recycle when a recycle-shaped failure arrives after more than a minute, ::never rewords an unrelated failure on timing alone — a late transport error keeps its own text, ::recognizes the 0.12.x recycle texts — a terminated session shell and a container stopped under a pending call, ::a typed recycle error (certain) is named at any elapsed time — the SDK is stating the container stopped; src/execution/sandboxKeepalive.test.ts::isRecycleError::takes the 0.12.x typed errors by NAME (the RPC boundary drops the prototype), ::falls back to the recycle-shaped texts, and rejects everything else
5: the credential rides in the exec env option and never in the command text — the Worker source carries env: envVars and no base64 export[unit] src/execution/sandboxKeepalive.test.ts::sandbox Worker wiring (static)::the credential goes through the exec env option, never through the command text; live: after the sandbox Worker deploys, a cold run's Command executed lines in the Worker logs carry no GH_TOKEN
5: the executor sends the resolved env as body.env on every route (/exec, /read, /write) and NO x-env-* header — the request carries only authorization, content-type, x-thread-key; an empty map is still env: {}; the fleet-busy re-send carries the same body, env included[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor env transport::sends the map in the body and NO x-env-* header on any route, ::an empty env is still the one body shape: \env: {}`, so the Worker reads one field on every route; src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor fleet-busy wait::re-sends the SAME route, body and headers after 10 s then 20 s and returns the eventual result`
5: envFromRequest — reads body.env alone (request headers are never read); body keys are upper-cased; names that are not shell identifiers and non-string values are dropped; an env that is not a plain object is ignored; a non-object body yields {} without throwing; no other body field is read as env[unit] src/execution/sandboxEnv.test.ts::envFromRequest::reads \env` from the body — an object of string values — and returns it as the env map, ::upper-cases body keys, so `gh_token` and `GH_TOKEN` are one variable, ::drops names that are not shell identifiers after upper-casing, ::drops non-string values — numbers, booleans, null, objects, arrays — and keeps the string ones, ::an `env` that is not a plain object (array, string, null, number, boolean) yields an empty map, ::a body that is not an object yields an empty map — never a throw, ::reads only `env` — the body's other fields are never env, and there is no header channel`
5: the sandbox Worker WIRES the body path (static) — envVars comes from envFromRequest( and the Worker source names no x-env- channel; the executor resolves the map once (const envs = await this.opts.resolveEnvs()), passes it as env: envs in the body, and its source names no x-env- either[unit] src/execution/sandboxKeepalive.test.ts::sandbox Worker wiring (static)::the Worker reads the env map through envFromRequest and names no x-env header channel, ::the executor sends the env in the body alone — no x-env-* header
5 live: after the sandbox Worker AND the bot deploy, a cold run's command still sees GH_TOKEN (gh auth status succeeds) and the POST /exec invocation event in Workers Logs lists authorization, content-type, x-thread-key and nothing beginning x-env- (the body, which now carries the credential, is not recorded)[agent] After both deploy: run a cold command that includes gh auth status, then read the POST /exec invocation event in Workers Logs filtered on the thread key and list its request headers.
4: a booting container's answer is recognized and nothing else is[unit] src/execution/sandboxErrors.test.ts::isContainerStarting::recognizes the 0.12.x boot-time answer and nothing else
3: thrownText — the SDK's message verbatim (trimmed) when present; a message-less error names the error name and code and says a rollout may be in progress; whitespace-only counts as empty; no name at all still yields a non-empty text[unit] src/execution/sandboxErrors.test.ts::thrownText::returns the SDK's message verbatim (trimmed) when it has one, ::a message-less error names the error name and code and says a rollout may be in progress, ::a whitespace-only message counts as empty, ::no name at all still yields a non-empty text that says so
9: the destroy-time disconnect text (The sandbox was destroyed while the operation was pending.) is recycle-shaped — reworded after a minute, kept inside it, and recognized by isRecycleError[unit] src/execution/sandboxKeepalive.test.ts::recycledMidCommandMessage::recognizes the destroy-time disconnect text as a recycle; src/execution/sandboxKeepalive.test.ts::isRecycleError::falls back to the recycle-shaped texts, and rejects everything else
3/9: the executor throws ExecInfraError after exactly one fetch on a PRESENT-but-empty in-body error; a body without an error key and exit 127 with empty output is the normal exit 127: result; exit 0 with no output is (no output)[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor in-body empty error::an empty-string error is ExecInfraError after exactly one fetch — the Worker's failure shape with its text missing, ::a body WITHOUT an error key and exit 127 with empty output is the normal \exit 127:` result — `foo 2>/dev/null` is legitimate, ::exit 0 with no error key and no output renders (no output)`
3/6: the sandbox Worker WIRES it (static; the Worker cannot run under vitest) — both catches call thrownText( and the .message ?? String(err) fallthrough is gone; onStart calls getVersion( and exec is super.exec under the keepalive with no this.destroy( anywhere in the Worker; wrangler.jsonc sets rollout_step_percentage to 100[unit] src/execution/sandboxKeepalive.test.ts::sandbox Worker wiring (static)::every failure text goes through thrownText — the empty-string fallthrough is gone, ::onStart logs the container/SDK version skew; exec is super.exec under the keepalive, with no retry of its own, ::wrangler.jsonc rolls the sandbox image out in one wave (rollout_step_percentage 100)
6 live: across the next sandbox Worker+image rollout, the Worker logs carry no empty-tailed sandbox.exec error … — lines; a thread that starts during the rollout either runs normally or sees the named sandbox exec failed with no message from the SDK (…rollout…) text, never a silent exit 127; any sandbox.version-skew container=… sdk=… line is the skew being seen[agent] wrangler tail <sandbox worker> through the deploy, start one cold review inside the first two minutes, read its run page
2 live: a command longer than sleepAfter completes in a per-thread sandbox and the workspace survives it[agent] After the sandbox Worker deploys, in a NON-resident thread: run \mkdir -p /workspace/probe && sleep 330 && echo OK` with timeoutMs 600000, then ls /workspace→OKthenprobe. Pre-fix (5-min sleepAfter, no keepalive) the first command would have died at 300 s with Command execution failedandprobe` would be gone.
Sandbox image Node satisfies the repo's engines pin (≥22), so fallback reviews load the full suite[agent] After a sandbox-worker deploy, force the per-thread path (any repo not onboarded as a resident, or a pruned ref) and run: @switchboard agent:review run \node --version` in <owner/repo>, then `npx vitest run --reporter=basic 2>&1 | tail -5`. Expect v22.xand 0 unloadable suites (pre-fix: 15 suites failed to load withwebidl.util.markAsUncloneable is not a function). Build-time backstop: the Dockerfile asserts node --version` is v22 or the image build fails.
Shell quoting through the timeout wrapper[unit] src/execution/shellQuote.test.ts — tests the exact shellQuote module the worker ships (extracted to deploy/cloudflare-sandbox/shellQuote.ts); the timeout-binary cases run on CI's Linux and skip on macOS.
GH App token minting & caching[unit] src/execution/githubApp.test.ts — mocked fetch: JWT shape, cache hit, mint-failure surfacing, GH_TOKEN/null fallbacks.
5: a cached token is reused only while it outlives the longest single command — the margin is ≥ BASH_TIMEOUT_MAX_MS + 5 min, a token inside it re-mints, one outside it is reused[unit] src/execution/githubApp.test.ts::the reuse margin covers the longest single command (BASH_TIMEOUT_MAX_MS) plus slack, ::re-mints when the cached token has less than the reuse margin left, ::reuses the cached token while it still has at least the reuse margin left
5: the sandbox credential is resolved per command, not captured at executor construction — each command carries the token current at its own start[unit] src/execution/cloudflareSandbox.test.ts::resolves the sandbox env on EVERY call, so each command carries the credential current at its start, ::resolves nothing at construction — building the executor mints no credential; src/execution/e2b.test.ts::each command carries the envs resolved at its start; src/execution/factory.test.ts::the sandbox credential is resolved per command, not captured at executor construction
12: a command's budget is clipped to the run's remaining wall clock minus the 60 s reserve — requested and default alike, with a clip line — and inside the reserve nothing runs; plenty of run left changes nothing[unit] src/tools/workspace.test.ts::plenty of run left changes nothing: the request passes through and no request stays no request, ::little run left clips a requested budget to what is left minus the reserve, and says so, ::little run left also clips the 5-minute default, which the executor would otherwise apply, ::inside the reserve nothing runs: the tool refuses legibly instead of starting a command that cannot finish; the runner hands tools the remaining wall clock: src/runner.test.ts::hands tools the run's remaining wall clock (maxMinutes at the start)
5: least-privilege token by toolset — the read scope mints exactly contents, pull_requests, issues, actions, checks, metadata, every value read; the write scope requests no restriction; the scopes cache separately; a readonly agent gets the read token and a full agent the write one[unit] src/execution/githubApp.test.ts::mints a READ-scoped token whose permissions are exactly contents, pull_requests, issues, actions, checks and metadata — all read, ::every permission the read scope requests is read — the request never carries write or admin, ::the default (write) scope requests NO permissions restriction — the full grant, ::read and write tokens cache in separate slots (one mint per scope); src/execution/factory.test.ts::a readonly agent's sandbox gets a READ-scoped token; a full agent gets WRITE
5: a review run's sandbox reads CI on the read token — gh run list -R <owner>/<repo> -L 1 and gh pr checks <n> succeed, gh run rerun <id> and gh pr comment are refused[agent] After the bot deploy, post agent:review on an open PR of a repository the installation covers and, from the run page's tool log, confirm the read commands answered 200 and a write returned 403 Resource not accessible by integration; also runnable from a dev checkout by minting resolveGithubToken("read") and driving gh with it as GH_TOKEN.
App installation covers the target repo[agent] agent:review run \gh repo view <owner/repo> --json name`` for each repo agents work on — must return JSON, not 404. Failure signature: "GitHub credentials don't have access" + 404.
Agents declaring no repo provision nothing, under every execution type[unit] src/execution/factory.test.ts::an agent declaring no repo gets a null executor with cloudflare configured (no sandbox call, no token needed), ::an agent declaring no repo gets a null executor with e2b configured (no API key needed), ::an agent declaring no repo creates no workspace directory in local mode
Repo-requiring agents still get the configured backend[unit] src/execution/factory.test.ts::a repo-requiring agent gets a LocalExecutor with a per-thread workspace (local), ::a repo-requiring agent gets the Cloudflare backend when configured, ::a repo-requiring agent with e2b configured but no API key still fails legibly; end-to-end: src/core/dispatcher.test.ts::a coding ask still selects the configured remote backend
Executor selection receives the resolved agent in its context[unit] src/core/dispatcher.test.ts::passes the resolved agent to executor selection
Resident selection: warm → resident; anything else → named per-thread fallback; outage circuit breaker[unit] src/execution/factory.test.ts::makeExecutor resident selection (all six scenarios); details in resident-repos.md
Infra failures throw ExecInfraError (distinct from a nonzero exit) so the runner can fail fast[unit] src/execution/executor.test.ts::ExecHealthTracker (classification + consecutive-failure counting); the runner fail-fast that consumes this seam is unit-tested in src/runner.test.ts (see run-loop.md)
A single resident runtime-replaced (one deploy) is NOT infra and leaves the consecutive-failure count at 0[unit] src/execution/resident.test.ts::ResidentExecutor infra classification through ExecHealthTracker …::a single runtime-replaced (one deploy) never counts toward fail-fast
14: isFleetBusy recognizes the SDK 0.3.x Failed to create session: 503, the platform's "no container instance" wording, and CONTAINER_UNAVAILABLE; a stale session, "Command execution failed", a file-op failure, a 500, and arbitrary text are NOT fleet-busy[unit] src/execution/sandboxErrors.test.ts::isFleetBusy::recognizes the SDK 0.3.x client's unparsed 503 from createSession, ::recognizes the platform's raw no-instance message, ::recognizes the newer SDKs' CONTAINER_UNAVAILABLE code, ::is NOT a stale session, a wedged sandbox, or arbitrary text
14: isFleetBusyError takes the 0.12.x ContainerUnavailableError by name or code and falls back to the recognized texts; a recycle, a stale session, or an unrelated failure is not fleet-busy; thrownShape reads an Error, a plain object, and a string alike[unit] src/execution/sandboxErrors.test.ts::isFleetBusyError / thrownShape::takes the 0.12.x typed error by name, whatever its text, ::takes the error code when a client surfaces it, and falls back to the recognized texts, ::is NOT a recycle, a stale session, or an unrelated failure, ::thrownShape reads an Error, a plain object, and a string the same way
10: the INSTALLED @cloudflare/sandbox under each Worker matches its pin — a nested install that drifted from the lockfile (the main checkout carried 0.12.9 against a 0.3.7 pin) fails check:sandbox-pair with the npm ci remedy; a fresh clone with nothing installed passes[unit] src/sandboxPairCheck.test.ts::check-sandbox-pair installMismatches::passes when the installed SDK equals the pin, and when nothing is installed yet (a fresh clone), ::fails an installed SDK that differs from the pin and says how to fix it, ::reads the version a Worker would bundle: its nested install first, else the root's; src/sandboxPairCheck.test.ts::the repository's sandbox-image Workers::pin @cloudflare/sandbox to exactly the image tag they build FROM, and the installed SDK matches
14: the Worker's answer shapes — reason: "fleet-busy", an error keeping the SDK's message as the cause, and the /exec dual form (error + exit 127 + stderr)[unit] src/execution/sandboxErrors.test.ts::the fleet-busy answer shapes the Worker sends::names the reason and the explanation, and keeps the SDK's own message as the cause, ::the /exec shape carries the dual in-body failure form (error + exit 127 + stderr) like every other exec failure; the Worker wiring (streamExec catch → fleetBusyExecAnswer, outer catch → 503 fleetBusyAnswer) is [agent]: fill the fleet (max_instances threads busy) and run one more cold command — the run page must show ⏳ Sandbox fleet busy and the run must continue, not Sandbox exec transport failed 2 times in a row
14: the executor re-sends the IDENTICAL request (route, body, headers) after 10 s then 20 s and returns the eventual result; a /read answered 503 fleet-busy waits the same way[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor fleet-busy wait::re-sends the SAME route, body and headers after 10 s then 20 s and returns the eventual result, ::a /read answered HTTP 503 with reason fleet-busy is the same wait (the default 5-min budget applies)
14: the wait is bounded by the command's own budget and by the 5-min cap, and then throws ExecCapacityError — not ExecInfraError — naming the seconds waited[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor fleet-busy wait::gives up once the total wait reaches the command's own budget and throws ExecCapacityError, not ExecInfraError, ::never waits longer than FLEET_BUSY_WAIT_MAX_MS (5 min) even for a 20-minute command; constants: src/execution/sandboxErrors.test.ts::the executor's bounded wait::* (3)
14: a hard stop during the wait ends it at once with no further send[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor fleet-busy wait::a hard stop during the wait ends it at once with no further send
14: every OTHER in-body error — including an OLD Worker's bare Failed to create session: 503 — is still ExecInfraError after exactly ONE send (nothing else is replayed)[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor fleet-busy wait::every OTHER in-body error is still ExecInfraError after exactly one send — nothing else is replayed, ::an OLD Worker's bare Failed-to-create-session 503 (no reason) still reads as infra — one send
14: ExecHealthTracker leaves the consecutive count (and the last infra error) untouched on ExecCapacityError[unit] src/execution/executor.test.ts::ExecHealthTracker::leaves the count untouched on ExecCapacityError (a full fleet is not a dead sandbox)
14: the runner does not fail fast on capacity errors, emits fleet_busy notes, and shows the model the ⏳ text[unit] src/runner.test.ts::fleet-busy capacity errors do not trip fail-fast::* (3) — see run-loop.md item 7
14: fleet_busy notes survive the saved-stream parser and count as a medium infra_failure finding[unit] src/core/runEventLines.test.ts::accepts every declared run_note kind, fleet_busy included (docs/reference/specs/execution.md item 14); src/core/runFriction.test.ts::infra_failure: a fleet_busy note is an infra finding too (capacity is friction), at medium severity — the run went on
14 live: with the fleet full, one more cold run waits and then proceeds (a slot frees) or ends with ⏳ Sandbox fleet busy — … after waiting 300s and a normal write-up — never the dead-sandbox abort; max_instances 25 is live on the deployed Worker[agent] wrangler deployments status / the containers dashboard shows 25; force the condition by running max_instances+1 concurrent cold reviews, read the last run's page
11: the shared clamp rule — a 25-min ask runs at 20 min, 0/negative at the 1s floor, NaN/absent/non-numeric at the 5-min default[unit] src/execution/bashTimeout.test.ts::clampBashTimeout (every branch)
11: the bash tool declares optional integer timeoutMs, documents default (300000) and max (1200000) in the tool text, clamps before the executor, and omitting it reproduces the pre-feature call exactly[unit] src/tools/workspace.test.ts::bash tool timeoutMs
11: LocalExecutor honors the per-call budget (execFile timeout, maxBuffer kept) and a deadline kill is exit 124 naming the limit + the timeoutMs knob — never a generic abort, never the 124 wording for an ordinary nonzero exit[unit] src/execution/executor.test.ts::LocalExecutor per-call timeout
11: ResidentExecutor sends timeoutMs in the /exec body only when asked (older-resident body compat), clamped, kept across the one re-attach retry[unit] src/execution/resident.test.ts::ResidentExecutor per-call timeout
11: E2BExecutor maps the budget onto commands.run({timeoutMs}) and renders the SDK's TimeoutError as exit 124 naming the limit[unit] src/execution/e2b.test.ts
11: CloudflareSandboxExecutor sends timeoutMs in the /exec body only when asked, clamped[unit] src/execution/cloudflareSandbox.test.ts
11: every send to the sandbox Worker has a bot-side deadline of budget + 30 s covering the headers AND the streamed body — a fetch that never answers, and a body that heartbeats but never completes (a gone container under a still-heartbeating Worker), are both abandoned at budget + margin with ExecInfraError naming both numbers; no timeoutMs → 330 s; a 20-min (or a clamped 25-min) ask → 1230 s; /read and /write wait the 5-min file-op budget + margin[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor per-send deadline …::a fetch that never answers is abandoned at budget + margin…, ::a body that heartbeats but never completes…, ::no timeoutMs → the 5-minute default budget…, ::a 20-minute timeoutMs → 20 min + 30 s…, ::/read and /write wait the file-op budget…
11: the ResidentExecutor bounds the body read too — a heartbeating resident body that never completes is an ExecInfraError (resident worker /exec request failed) at budget + margin, which fail-fast counts, not a raw TimeoutError[unit] src/execution/resident.test.ts::ResidentExecutor per-send deadline …::a heartbeating body that never completes is an ExecInfraError at the command budget + margin, not a raw TimeoutError or a hang
11: the hard stop still aborts the send at once (reported as the stop, not as a silent Worker); an answer inside the deadline is unaffected and nothing fires after it; the deadline is per SEND — a fleet-busy wait before the send does not eat into it[unit] src/execution/cloudflareSandbox.test.ts::CloudflareSandboxExecutor per-send deadline …::the hard-stop signal aborts at once…, ::an answer inside the deadline is unaffected…, ::the deadline is per SEND…
11: execDeadline fires exactly at its timeout under fake timers (a plain timer, not AbortSignal.timeout) with a TimeoutError reason, and a joined hard stop aborts it at once with the stop's reason, the timer never re-firing[unit] src/execution/executor.test.ts::execDeadline::*
11 live: a cold (sandbox) run whose container is destroyed mid-command — wrangler containers stop/delete the thread's instance while a long sleep runs — gets sandbox worker /exec gave no answer within … as an ExecInfraError at budget + 30 s instead of hanging; the card reads running bash (Ns) meanwhile, and the run ends through fail-fast or the finale, never by an operator stop[agent] human-gated.
11: resident Worker clamps the body's timeoutMs server-side with the shared bounds (never trusts the client) and its exit-124 stderr names the limit + knob; /op keeps the flat 5-min budget[agent] (Worker wiring; the clamp itself is the shared unit-tested clampBashTimeout) On a warm resident: run \sleep 330 && echo LONG` with timeoutMs 360000→LONG; the same command without timeoutMs→exit 124at 300s with a stderr line namingtimeoutMsand1200000`.
11: a >5-min command completes end-to-end through a live resident (HTTP wait = budget + 30s margin, heartbeat keeps hops alive)[agent] The 360s check above must return output, not a bot-side resident worker /exec request failed transport error.
11: sandbox Worker sizes coreutils timeout from the body's timeoutMs (clamped); Dockerfile backstop (1240s) stays above the ceiling — needs a container rebuild to take effect[agent] In a non-resident thread: run \sleep 300 && echo OK` with timeoutMs 360000→OK; without it → exit 124` at 280s.
15: exec/readFile/writeFile are exec.* spans under the tool span with backend and budget, never command/path/output; release/moveTo wrap only when present; a throw ends error and propagates[unit] src/execution/tracingExecutor.test.ts::TracingExecutor::*
15: the factory names the backend on every selection — local for the null and local executors, sandbox for the per-thread Cloudflare sandbox, resident for a warm resident[unit] src/execution/factory.test.ts::makeExecutor per-agent provisioning::an agent declaring no repo gets a null executor with cloudflare configured (no sandbox call, no token needed), ::a repo-requiring agent gets a LocalExecutor with a per-thread workspace (local), ::a repo-requiring agent gets the Cloudflare backend when configured, src/execution/factory.test.ts::makeExecutor resident selection::warm probe → ResidentExecutor, attached on open, with the resident discriminant set
16: the sandbox template's container is standard-4 — 4 vCPU / 12 GiB / 20 GB, the largest predefined type, rendered against the example profile and read back as JSON[unit] src/deploy/wranglerTemplate.test.ts::the sandbox Worker's container::the cold per-thread sandbox runs on standard-4 …
16 live: after the sandbox Worker deploys, a cold run's nproc; free -m; df -h / prints 4 / ~12 GiB (Mem: total ≈ 12 000 MiB) / ~20 GB (/ size ≈ 19–20 GB)[agent] In a NON-resident thread: @switchboard agent:review run nproc; free -m; df -h / in <owner/repo>, report raw output. The previous standard-3 read 2 / 8189 MiB / 15 GB. Then the shape that motivated the size: the large monorepo's root typecheck in a cold thread completes instead of dying on memory.
17: the sandbox Dockerfile installs docker.io and iptables in the apt layer without recommends, still drops the apt lists in that layer, and installs the wrapper as /usr/local/bin/docker (mode 755) ahead of the engine's own client[unit] src/deploy/sandboxDocker.test.ts::the cold sandbox image ships a Docker engine::installs docker.io and iptables in the apt layer, without recommends, ::still drops the apt lists in the same layer, ::installs the wrapper as /usr/local/bin/docker, ahead of the engine's client on PATH
17: the wrapper is plain sh that parses, starts dockerd under setsid -f and never as a plain background job, enables IPv4 forwarding before the start, starts only when the socket is missing or docker info fails, gives up after 40 s naming /var/log/dockerd.log, ends with exec "$REAL" "$@", and stays under 45 lines[unit] src/deploy/sandboxDocker.test.ts::the docker wrapper::* (7)
17 live: a cold run pulls and runs a container with working networking, and the engine survives to the next command[agent] After the sandbox Worker+image rollout, in a NON-resident thread: agent:coding run \docker run --rm alpine:3 wget -qO- https://registry.npmjs.org/pnpm | head -c 80`, then `docker ps`(or the same two commands through a direct/exec on the sandbox Worker). Expect the first to print the start of the registry's JSON ({"_id":"pnpm"…) after a few extra seconds for the daemon start, and the second to answer at once with the engine already running. Failure signatures: docker: command not found= an old image still serving;docker: the engine did not come up within 40 s — see /var/log/dockerd.log= the daemon failed to start (read the log in a follow-up); JSON absent withbad address/network is unreachable` = forwarding or the bridge is off.