Load harness

The repeatable measurement of OpenSwitchboard under many concurrent runs, and the receipt every capacity change is judged by: each change is a before/after number from these commands, never an estimate. The harness measures infrastructure — the bot process, the resident, the cold sandbox fleet, the Slack card path — never a model: the end-to-end command drives the bot with a scripted provider that never thinks.

• Code: src/load/aggregate.ts (samples → per-op percentiles, refusals by reason, D10 checks, the markdown receipt), src/load/history.ts (run-store paging + the concurrency sweep-line), src/load/runLoop.ts (the staggered N-thread driver with guaranteed teardown), src/load/reasons.ts (refusal tokens from client errors), src/load/residentLoad.ts, src/load/sandboxLoad.ts, src/load/e2eLoad.ts, src/load/cardsLoad.ts (the four drivers), src/load/scriptedProvider.ts (the OpenAI-compatible scripted model + the review/coding profile scripts), src/load/fakeSlack.ts (the rate-limited Slack Web API stand-in), src/execution/bindingPurge.ts (the resident's purge-bindings decision), src/channels/processMetrics.ts (/healthz.process), scripts/load.ts (the entrypoint behind npm run load)
• Tests: src/load/aggregate.test.ts, src/load/history.test.ts, src/load/runLoop.test.ts, src/load/reasons.test.ts, src/load/residentLoad.test.ts, src/load/sandboxLoad.test.ts, src/load/e2eLoad.test.ts, src/load/cardsLoad.test.ts, src/load/scriptedProvider.test.ts, src/load/fakeSlack.test.ts, src/execution/bindingPurge.test.ts, src/channels/processMetrics.test.ts
• Docs: how-to: run a load test

Behavior

1. One sample per operation, one summary per run. Every command records a Sample per operation it performs (op, start, duration, ok, and on failure the refusal's machine token as reason) and reduces them with summarize: per operation the count, successes, failures and p50/p95/p99/max latency — over the successful samples only, because a refusal answers in milliseconds and would flatter every percentile it joined; an operation with no success reports NaN, never 0. Failed samples are counted by reason (unnamed when the client threw without one). Percentiles are nearest-rank.
2. Checks are the plan's D10 lines. evaluateSlo(summary, spec) turns <op> p<p> ≤ N ms, zero <reason> and zero failed <op> rules into pass/fail lines carrying the actual number; a latency or zero-failures rule on an operation with no samples FAILS with no samples rather than passing vacuously. zero failed <op> is the structural form for a command whose failure reasons are open-ended (load -- e2e: a run's terminal status or an HTTP status), so an unanticipated reason can never pass by omission. Each command has its own spec in scripts/load.ts; the exit code is non-zero when any check fails.
3. The receipt is markdown. renderMarkdown writes the parameters, the per-op table, refusals by reason, each check as ✅/❌ with the actual and the limit, a Verdict: PASS|FAIL|no checks line, and free-text notes. Every command writes load-results/<command>-<runId>.json (the samples, the summary, the checks) and .md beside it; load-results/ is gitignored — receipts go on the tracking issue.
4. load:history reproduces the baseline. It pages the state Worker's POST /runs/list newest-first with the (finishedAt, id) cursor, drops the provisional start-of-run tombstones (finishedAt === startedAt, run-history.md item 27), and sweeps the half-open [startedAt, finishedAt) intervals for the peak number of simultaneously live runs (overall and per UTC day) plus the duration mean and percentiles. A run finishing exactly when another starts does not overlap it.
5. One driver, guaranteed teardown. runThreads starts N threads spread evenly over the stagger window (the first at once, the last at the window's end), runs each thread's setup → iterate… → teardown until the hold ends or the run is aborted, and ALWAYS runs teardown — with the setup or iteration error that ended the thread — so whatever setup acquired (a pool seat, a sandbox, a card) is released. A setup failure counts and never iterates; an iteration error ends its thread and counts; a teardown that throws counts and never stops the other teardowns; an abort signal ends every thread at its next iteration boundary.
6. The scripted provider never thinks. startScriptedProvider serves POST /chat/completions in the OpenAI wire shape the openaiCompat provider parses; the answer is the script step at index = the number of tool results already in the conversation, so one server serves fifty conversations with no per-conversation state. A tool step answers one tool_calls entry (call_<index>, JSON arguments) with finish_reason: tool_calls; a text step answers content with stop; past the script's end the answer is its trailing text (Done. when none); a tool the request did not declare is removed from the script for that request (the step index stays the tool-result count), so each declared tool is called exactly once. The coding profile reads, inspects, burns CPU (cpuBurnCommand: one core, pure compute, --cpu-seconds), writes under .load-harness/ in the worktree and answers; the review profile reads, inspects, burns and answers. Neither ends with a terminal tool by default (submit_pr_description, submit_verdict): a load run has no GitHub side effect; terminal: true opts in for a fixture repo.
7. The fake Slack enforces the published limits. FakeSlack answers chat.postMessage, chat.update and conversations.replies in Slack's shapes, refuses chat.update with ratelimited + retryAfterS when either budget is spent — a per-app token bucket at the Tier 3 rate (default 50 a minute) and a per-channel bucket at the coalescer's cited ~1 edit a second — and records for every accepted edit the lag from the frame's production time (frameAt) to acceptance, so a refused frame's lag keeps counting through its retries. stats() reports posts, updates, refusals, distinct cards edited, and lag p50/p95/p99/max.
8. load:resident drives the bot's own client. N ResidentExecutors with thread keys load:<runId>:<i>, each attaching (timed as attach) then looping the profile: read README.md, exec a short git command, exec-cpu the CPU burn, write under .load-harness/ (coding only; the review profile attaches read-only and never writes), a pause; teardown release("always") timed as detach. Refusals count by token (user-pool-exhausted, mirror-busy, disk-pressure). Guards (plan D9): the run refuses to start unless the operator GET /status reports exactly 0 in flight (unknown is refused); more than RESIDENT_LOAD_MAX_THREADS (16, one resident's pool) needs --override; and after the run the admin purge-bindings op removes the run's evicted bindings under load:<runId>: — a purge failure is reported in the receipt, never thrown, so the measurements always come back.
9. load:cards is a simulation over the real coalescer. The HTTP ingress status() handle is a no-op (http-ingress.md item 2), so an end-to-end run exercises zero card traffic; simulateCards instead runs N cards through coalesceStatus (the production floor, 3 s) against FakeSlack in virtual time: each card produces the dispatcher's 5 s heartbeat frames and a tool event every 7 s for the hold, then a terminal frame; the edit path mirrors the Slack adapter — one chat.update, on ratelimited wait Retry-After and re-send the same frame up to the WebClient's retry budget (10), give up after it; a retry whose frame has since been superseded is dropped as staleDropped (a real client would have spent it on a stale card). Deterministic and instant. What it shows at the published limits, fifty cards, ten minutes: edits refused, retries served first-come so most cards never get an edit at all (lag is measured over accepted frames only, so starvation hides in it — the surviving frames wait past the coalescer floor while the starved cards contribute nothing), and terminal frames lost — the adapter's done swallows the failed edit the same way, so a card past the retry budget stays a spinner forever. Lifting the limits gives every card sub-floor lag and every terminal frame: the limit, not the coalescer, is the ceiling. This is the number the plan's status budget (D3) has to fix.
10. load:e2e runs the whole bot. N threads each post POST /ingress synchronously ({text, channel: "load", thread: <runId>-<i>}, the bearer from SWITCHBOARD_LOAD_INGRESS_TOKEN) and record one run sample per request from the response's run receipt (ok iff completed; the terminal status is the reason otherwise; a transport failure is transport, a client timeout timeout). Between a thread's runs it pauses --pause; after a failed request it pauses at least failurePauseMs (5 s) so a dead endpoint is never hammered at full speed. While threads run, GET /healthz is sampled every 15 s for inFlight and the process metrics (item 12). The model is the scripted provider (npm run load -- provider, item 6) configured on the bot as an openai-compatible provider; the resident and sandbox Workers are whatever that bot is configured with. The receipt's check is zero failed run (item 2), and the run ends when the last thread does: the health sampler's sleep is raced against completion, never adding its interval to the tail.
11. load:sandbox proves the cold path and measures the fleet ceiling. N CloudflareSandboxExecutors with keys load:<runId>:sandbox:<i>; the first command per thread (first-exec) is what creates the container, so its latency is the cold start or the fleet-busy wait (execution.md item 14); then exec and exec-cpu per iteration. More than SANDBOX_LOAD_MAX_THREADS (25, the fleet's max_instances when written) needs --override. Sandboxes are not released explicitly; the Worker sleeps them after their idle window.
12. /healthz.process. The bot samples process.memoryUsage() and a monitorEventLoopDelay histogram (startProcessMetrics): {rssMb, heapUsedMb, eventLoopLagP99Ms}, the lag being the p99 over the window since the previous read (the histogram resets on each read, so consecutive polls read consecutive windows). Always present on the live /healthz; the harness reads it, and it is the only off-box view of the bot process under load.

Validation criteria

Criterion	Evidence
1: nearest-rank percentiles; per-op stats over successful samples only; NaN with no success; refusals by reason with unnamed	[unit] src/load/aggregate.test.ts::percentile — nearest-rank over a sorted array::*, ::summarize — per-operation latency and refusals by reason::*
2: latency and zero-reason checks pass/fail with the actual number; no samples fails	[unit] src/load/aggregate.test.ts::evaluateSlo — the D10 lines::*
3: the receipt carries parameters, the op table, refusals, checks and the verdict line	[unit] src/load/aggregate.test.ts::renderMarkdown — the receipt::*
4: tombstones dropped; sweep-line peak, first instant, per-day; half-open intervals; cursor paging bounded by maxPages	[unit] src/load/history.test.ts::*
5: staggered starts; teardown always runs with the ending error; setup/iteration/teardown failures counted; abort ends threads	[unit] src/load/runLoop.test.ts::runThreads::*
6: step = count of tool results; the OpenAI wire shape; undeclared tools skipped; profiles end in text and carry no terminal tool unless asked; the server serves /chat/completions and 404s the rest	[unit] src/load/scriptedProvider.test.ts::*
7: token buckets; per-channel and per-app refusals with Retry-After; lag counted from the frame's production; replies oldest-first; stats	[unit] src/load/fakeSlack.test.ts::*
8: the coding mix, read-only review, named refusals past the pool, the 0-in-flight preflight, the 16-thread guard, the purge (and a failed purge reported)	[unit] src/load/residentLoad.test.ts::runResidentLoad::*
8: the purge decision — evicted bindings under a whole non-production namespace only	[unit] src/execution/bindingPurge.test.ts::selectBindingsToPurge::*
9: one card is well-behaved; fifty on the pre-budget client lose terminal frames and starve; fifty on the budgeted client see no refusal, paint every card, land every terminal frame, and send the same edits whether or not the limits are lifted; today's seven-card shape is clean; deterministic	[unit] src/load/cardsLoad.test.ts::simulateCards::*
10: one synchronous ingress request per iteration with bearer, text and thread; run status as the outcome; transport errors; the failure pause	[unit] src/load/e2eLoad.test.ts::runE2eLoad::*
11: cold start as first-exec, fleet-busy named past the seats, the 25-thread guard	[unit] src/load/sandboxLoad.test.ts::runSandboxLoad::*
12: /healthz carries process when sampled; the sampler reads whole MiB and a fresh lag window	[unit] src/channels/health.test.ts::healthPayload — process metrics::*, src/channels/processMetrics.test.ts::startProcessMetrics::*
Live: load:resident at N = 16 against a quiet resident produces a receipt, leaves no bindings behind, and refuses to start while the resident has work in flight	[agent] In a quiet window: npm run load -- resident --resource repo:<slug> --threads 16 --hold 120 --cpu-seconds 20 → load-results/resident-*.md with attach/read/exec/exec-cpu/write/detach rows and purge: 16 binding(s) removed; /debug threads shows no load: keys afterwards. Start a real run on the same repo and re-run → refusing to start: … op(s) in flight.
Live: load:sandbox at N = 1 proves the current cold path	[agent] npm run load -- sandbox --threads 1 --hold 30 --cpu-seconds 5 → first-exec ok, no refusals.
Live: load:e2e against a local bot on the scripted provider completes runs with no model spend	[agent] npm run load -- provider --profile coding --cpu-seconds 10 in one shell; a bot started with a config naming providers.scripted: {type: openai-compatible, baseUrl: http://127.0.0.1:8089} and defaults.models.coding: scripted/any in another; npm run load -- e2e --ingress-url http://127.0.0.1:8080/ingress --healthz-url http://127.0.0.1:8080/healthz --text "agent:coding in <slug>: load harness" --threads 8 --hold 120 → every run sample completed, healthz samples with rss and lag in the notes.