This is the full developer documentation for microvms-agentd
# microvms-agentd
> Run AI agents in sandboxed AWS MicroVMs from the CLI or Python, JavaScript, TypeScript, and Rust SDKs.
Run AI agents in sandboxed MicroVMs
LAUNCH · RUN · COLLECT · CLEAN UP
Give an agent a copy of your project, let it edit files and run tools in a remote AWS Lambda MicroVM, and bring back the results. Use the `microvm` CLI or a Python, JavaScript/TypeScript, or Rust SDK. No local Docker daemon or hypervisor is needed.
## Start in 90 seconds
[Section titled “Start in 90 seconds”](#start-in-90-seconds)
With the CLI installed, AWS configured, and Bedrock model access enabled for your caller, start a coding agent from your project directory:
```bash
1
microvm agent-up --vm-name review --agent claude-code --project .
2
microvm agent-prompt --name review --agent claude-code \
3
"Review this project and write your findings to REVIEW.md."
4
microvm cp --name review vm:/workspace/REVIEW.md ./REVIEW.md
5
microvm terminate review --wait
```
The agent runs as a non-root user inside the VM, using a short-lived Bedrock token minted from your AWS credentials. Replace `claude-code` with `codex` in both commands to use Codex CLI.
**First time here?** [Install the CLI](/microvms-agentd/learn/tutorial/install/), then [configure AWS](/microvms-agentd/learn/tutorial/first-run/). You need AWS CLI v2, `gh` or `curl`, AWS credentials, Lambda MicroVMs access, an artifact bucket, and build/execution roles. The first image build takes several minutes; 90 seconds is the path to starting the workflow, not a promise that AWS setup or the build has finished. VMs, builds, and stored images create AWS charges.
## Pick your interface
[Section titled “Pick your interface”](#pick-your-interface)
| Use | Install | Start here |
| ----------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| Terminal | `cargo binstall microvms-cli --no-confirm` | [CLI and AWS setup](/microvms-agentd/learn/tutorial/first-run/) |
| Python 3.9+ | `pip install microvms` | [Python SDK](/microvms-agentd/learn/tutorial/from-code/#python) |
| Node 22.13+ | `npm install @theagenticguy/microvms` | [JavaScript / TypeScript SDK](/microvms-agentd/learn/tutorial/from-code/#javascript--typescript) |
| Rust | `cargo add microvms-core` | [Rust SDK](/microvms-agentd/learn/tutorial/from-code/#rust) |
The CLI install command needs `cargo-binstall`. [Installation](/microvms-agentd/learn/tutorial/install/) also covers standalone binaries and Cargo source builds. All interfaces can launch a VM, run commands, transfer files, stream output, and terminate it. The SDK guide includes complete programs you can save and run.
## Start with a command
[Section titled “Start with a command”](#start-with-a-command)
For a sandbox that needs no Bedrock model access, run `microvm quickstart` after AWS setup. It builds an image, runs hello-world, reports the result and estimated cost, and attempts cleanup. Then reuse a named image:
```bash
1
microvm build --name agent-tools
2
microvm run --image agent-tools --exec "uname -m"
```
The command prints `aarch64` from inside the VM. Use [project uploads](/microvms-agentd/learn/tutorial/run-a-project/) for your code and [custom images](/microvms-agentd/learn/operations/write-a-guest-dockerfile/) for dependencies.
## Choose the next task
[Section titled “Choose the next task”](#choose-the-next-task)
| Task | Guide |
| ------------------------------------------ | --------------------------------------------------------------------------------------------- |
| Run Claude Code or Codex against a project | [Coding agents in sandboxes](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) |
| Use a sandbox in your application | [SDK examples](/microvms-agentd/learn/tutorial/from-code/) |
| Keep working in the same VM | [Long-lived VMs](/microvms-agentd/learn/tutorial/long-lived-vm/) |
| Restrict internet access | [Networking](/microvms-agentd/learn/operations/configure-networking/) |
| Find a flag, response, or error | [CLI reference](/microvms-agentd/reference/) |
| Recover resources after failed cleanup | [Recovery](/microvms-agentd/learn/operations/recover-a-leaked-vm/) |
The VM separates the agent’s work from your local machine. `agent-up` enables outbound networking, and workloads can obtain the execution role’s credentials. Keep that role minimal. Internet isolation needs a VPC with no internet route; omitting `--egress` does not enforce it. See [Trust](/microvms-agentd/internals/trust/).
For automation, `microvm manifest` describes the installed CLI and [For agents](/microvms-agentd/agents/) explains JSON and streaming. Every page has a Markdown twin at its path ending in `.md`; [llms.txt](/microvms-agentd/llms.txt) indexes them. [Internals](/microvms-agentd/internals/) covers the daemon, wire protocol, and platform behavior.
# For agents
> Machine-readable contracts, automation rules, and documentation entry points.
**For an agent.** Start with `microvm manifest`. It returns commands, flags, response types, and exit codes as JSON without credentials or network access. `microvm manifest --dense` gives a compact command list. Prefer the installed binary’s contract over prose that may describe a different version.
## 1. Automation
[Section titled “1. Automation”](#1-automation)
* Use `--json`. Commands write one envelope to stdout and progress to stderr. `exec --stream` emits NDJSON events and the final envelope.
* Branch on error `code` and `exitCode`, not message text. `ERR_EXEC_FAILED` means the command in the VM failed; it does not imply an AWS failure.
* Read `leaked` after cleanup. Teardown is attempted by default, but failures and process termination can leave resources behind. `microvm ls --remote` compares the local ledger with AWS.
* Keep per-VM secrets out of shared images. The guest can access its execution role; user demotion does not hide it.
* No internet egress requires a VPC without an IGW or NAT gateway. Neither omitting `--egress` nor setting `--deny-egress` enforces isolation.
* Keep `agentd` as the image’s `CMD` and start workloads only after bootstrap.
## 2. Entry points
[Section titled “2. Entry points”](#2-entry-points)
| Task | Contract or guide |
| ------------------------- | ------------------------------------------------------------------------------------ |
| CLI integration | `microvm manifest`, [Reference](/microvms-agentd/reference/) |
| Direct daemon integration | `GET /v1/schema`, [Protocol](/microvms-agentd/internals/protocol/) |
| Rust, Python, Node | [Libraries](/microvms-agentd/learn/tutorial/from-code/) |
| First AWS run | [First-run tutorial](/microvms-agentd/learn/tutorial/first-run/) |
| Coding agents inside a VM | [Agents on Bedrock](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) |
| Network isolation | [Networking](/microvms-agentd/learn/operations/configure-networking/) |
| Image integration | [Embedding](/microvms-agentd/internals/embedding/) |
`microvm doctor` checks prerequisites before a build. `quickstart` creates billable AWS resources, runs a hello-world, and attempts cleanup. Reuse images with `--image` to avoid unnecessary builds and retention charges.
## 3. Repository work
[Section titled “3. Repository work”](#3-repository-work)
Read `CONTRIBUTING.md`. Run `mise run check` for local verification and `mise run docs:check` for documentation. Live AWS behavior needs a separate exercise of the changed path; `mise run live` is billable. Report when it has not been run, and verify cleanup independently afterward.
Edit `site/authored/` for user guides and top-level `docs/*.md` for contracts. `site/src/content/docs/` is generated. References generated from source contain commit-pinned citations that may be stale in the current checkout; check the source instead of treating prose as proof. Record new AWS measurements with date, region, and API version, retaining earlier observations.
## 4. Read next
[Section titled “4. Read next”](#4-read-next)
| Read this | Raw Markdown |
| ------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| [Learn](/microvms-agentd/learn/) | [`learn.md`](/microvms-agentd/learn.md) |
| [Install the CLI or an SDK](/microvms-agentd/learn/tutorial/install/) | [`learn/tutorial/install.md`](/microvms-agentd/learn/tutorial/install.md) |
| [Configure AWS and run your first sandbox](/microvms-agentd/learn/tutorial/first-run/) | [`learn/tutorial/first-run.md`](/microvms-agentd/learn/tutorial/first-run.md) |
| [Keep a VM running and work inside it](/microvms-agentd/learn/tutorial/long-lived-vm/) | [`learn/tutorial/long-lived-vm.md`](/microvms-agentd/learn/tutorial/long-lived-vm.md) |
| [Run a project through a VM](/microvms-agentd/learn/tutorial/run-a-project/) | [`learn/tutorial/run-a-project.md`](/microvms-agentd/learn/tutorial/run-a-project.md) |
| [Use the SDKs](/microvms-agentd/learn/tutorial/from-code/) | [`learn/tutorial/from-code.md`](/microvms-agentd/learn/tutorial/from-code.md) |
| [Configure networking](/microvms-agentd/learn/operations/configure-networking/) | [`learn/operations/configure-networking.md`](/microvms-agentd/learn/operations/configure-networking.md) |
| [Write a guest Dockerfile](/microvms-agentd/learn/operations/write-a-guest-dockerfile/) | [`learn/operations/write-a-guest-dockerfile.md`](/microvms-agentd/learn/operations/write-a-guest-dockerfile.md) |
| [Embed agentd in your own image](/microvms-agentd/learn/operations/embed-agentd-in-your-image/) | [`learn/operations/embed-agentd-in-your-image.md`](/microvms-agentd/learn/operations/embed-agentd-in-your-image.md) |
| [Run coding agents in sandboxed MicroVMs](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) | [`learn/operations/run-coding-agents-on-bedrock.md`](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock.md) |
| [Remote dev with code-server over port-forward](/microvms-agentd/learn/operations/remote-dev-with-code-server/) | [`learn/operations/remote-dev-with-code-server.md`](/microvms-agentd/learn/operations/remote-dev-with-code-server.md) |
| [Prefetch S3 content at image build](/microvms-agentd/learn/operations/prefetch-s3-at-build/) | [`learn/operations/prefetch-s3-at-build.md`](/microvms-agentd/learn/operations/prefetch-s3-at-build.md) |
| [Read the cost report](/microvms-agentd/learn/operations/read-the-cost-report/) | [`learn/operations/read-the-cost-report.md`](/microvms-agentd/learn/operations/read-the-cost-report.md) |
| [Debug a failed build](/microvms-agentd/learn/operations/debug-a-failed-build/) | [`learn/operations/debug-a-failed-build.md`](/microvms-agentd/learn/operations/debug-a-failed-build.md) |
| [Recover a leaked VM](/microvms-agentd/learn/operations/recover-a-leaked-vm/) | [`learn/operations/recover-a-leaked-vm.md`](/microvms-agentd/learn/operations/recover-a-leaked-vm.md) |
| [Configure the project file](/microvms-agentd/learn/operations/configure-the-project-file/) | [`learn/operations/configure-the-project-file.md`](/microvms-agentd/learn/operations/configure-the-project-file.md) |
| [Drive it from a script or an agent](/microvms-agentd/learn/operations/drive-it-from-a-script-or-an-agent/) | [`learn/operations/drive-it-from-a-script-or-an-agent.md`](/microvms-agentd/learn/operations/drive-it-from-a-script-or-an-agent.md) |
| [Run the live suite](/microvms-agentd/learn/operations/run-the-live-suite/) | [`learn/operations/run-the-live-suite.md`](/microvms-agentd/learn/operations/run-the-live-suite.md) |
| [Reference](/microvms-agentd/reference/) | [`reference.md`](/microvms-agentd/reference.md) |
| [microvm ack](/microvms-agentd/reference/commands/ack/) | [`reference/commands/ack.md`](/microvms-agentd/reference/commands/ack.md) |
| [microvm agent-prompt](/microvms-agentd/reference/commands/agent-prompt/) | [`reference/commands/agent-prompt.md`](/microvms-agentd/reference/commands/agent-prompt.md) |
| [microvm agent-up](/microvms-agentd/reference/commands/agent-up/) | [`reference/commands/agent-up.md`](/microvms-agentd/reference/commands/agent-up.md) |
| [microvm attach](/microvms-agentd/reference/commands/attach/) | [`reference/commands/attach.md`](/microvms-agentd/reference/commands/attach.md) |
| [microvm build](/microvms-agentd/reference/commands/build/) | [`reference/commands/build.md`](/microvms-agentd/reference/commands/build.md) |
| [microvm constants](/microvms-agentd/reference/commands/constants/) | [`reference/commands/constants.md`](/microvms-agentd/reference/commands/constants.md) |
| [microvm cost](/microvms-agentd/reference/commands/cost/) | [`reference/commands/cost.md`](/microvms-agentd/reference/commands/cost.md) |
| [microvm cp](/microvms-agentd/reference/commands/cp/) | [`reference/commands/cp.md`](/microvms-agentd/reference/commands/cp.md) |
| [microvm dockerfile](/microvms-agentd/reference/commands/dockerfile/) | [`reference/commands/dockerfile.md`](/microvms-agentd/reference/commands/dockerfile.md) |
| [microvm doctor](/microvms-agentd/reference/commands/doctor/) | [`reference/commands/doctor.md`](/microvms-agentd/reference/commands/doctor.md) |
| [microvm exec](/microvms-agentd/reference/commands/exec/) | [`reference/commands/exec.md`](/microvms-agentd/reference/commands/exec.md) |
| [microvm health](/microvms-agentd/reference/commands/health/) | [`reference/commands/health.md`](/microvms-agentd/reference/commands/health.md) |
| [microvm history](/microvms-agentd/reference/commands/history/) | [`reference/commands/history.md`](/microvms-agentd/reference/commands/history.md) |
| [microvm kill](/microvms-agentd/reference/commands/kill/) | [`reference/commands/kill.md`](/microvms-agentd/reference/commands/kill.md) |
| [microvm logs](/microvms-agentd/reference/commands/logs/) | [`reference/commands/logs.md`](/microvms-agentd/reference/commands/logs.md) |
| [microvm ls](/microvms-agentd/reference/commands/ls/) | [`reference/commands/ls.md`](/microvms-agentd/reference/commands/ls.md) |
| [microvm manifest](/microvms-agentd/reference/commands/manifest/) | [`reference/commands/manifest.md`](/microvms-agentd/reference/commands/manifest.md) |
| [microvm port-forward](/microvms-agentd/reference/commands/port-forward/) | [`reference/commands/port-forward.md`](/microvms-agentd/reference/commands/port-forward.md) |
| [microvm ps](/microvms-agentd/reference/commands/ps/) | [`reference/commands/ps.md`](/microvms-agentd/reference/commands/ps.md) |
| [microvm quickstart](/microvms-agentd/reference/commands/quickstart/) | [`reference/commands/quickstart.md`](/microvms-agentd/reference/commands/quickstart.md) |
| [microvm resume](/microvms-agentd/reference/commands/resume/) | [`reference/commands/resume.md`](/microvms-agentd/reference/commands/resume.md) |
| [microvm run](/microvms-agentd/reference/commands/run/) | [`reference/commands/run.md`](/microvms-agentd/reference/commands/run.md) |
| [microvm shell](/microvms-agentd/reference/commands/shell/) | [`reference/commands/shell.md`](/microvms-agentd/reference/commands/shell.md) |
| [microvm stdin](/microvms-agentd/reference/commands/stdin/) | [`reference/commands/stdin.md`](/microvms-agentd/reference/commands/stdin.md) |
| [microvm suspend](/microvms-agentd/reference/commands/suspend/) | [`reference/commands/suspend.md`](/microvms-agentd/reference/commands/suspend.md) |
| [microvm sync](/microvms-agentd/reference/commands/sync/) | [`reference/commands/sync.md`](/microvms-agentd/reference/commands/sync.md) |
| [microvm terminate](/microvms-agentd/reference/commands/terminate/) | [`reference/commands/terminate.md`](/microvms-agentd/reference/commands/terminate.md) |
| [microvm tunnel](/microvms-agentd/reference/commands/tunnel/) | [`reference/commands/tunnel.md`](/microvms-agentd/reference/commands/tunnel.md) |
| [The envelope](/microvms-agentd/reference/envelope/) | [`reference/envelope.md`](/microvms-agentd/reference/envelope.md) |
| [Exit codes](/microvms-agentd/reference/exit-codes/) | [`reference/exit-codes.md`](/microvms-agentd/reference/exit-codes.md) |
| [Response types](/microvms-agentd/reference/response-types/) | [`reference/response-types.md`](/microvms-agentd/reference/response-types.md) |
| [Wire schema](/microvms-agentd/reference/wire-schema/) | [`reference/wire-schema.md`](/microvms-agentd/reference/wire-schema.md) |
| [CLI](/microvms-agentd/reference/cli/) | [`reference/cli.md`](/microvms-agentd/reference/cli.md) |
| [Public API](/microvms-agentd/reference/public-api/) | [`reference/public-api.md`](/microvms-agentd/reference/public-api.md) |
| [RPC tools](/microvms-agentd/reference/rpc-tools/) | [`reference/rpc-tools.md`](/microvms-agentd/reference/rpc-tools.md) |
| [Internals](/microvms-agentd/internals/) | [`internals.md`](/microvms-agentd/internals.md) |
| [Networking](/microvms-agentd/internals/networking/) | [`internals/networking.md`](/microvms-agentd/internals/networking.md) |
| [AWS Lambda MicroVMs: measured platform behavior](/microvms-agentd/internals/platform/) | [`internals/platform.md`](/microvms-agentd/internals/platform.md) |
| [Wire protocol v1](/microvms-agentd/internals/protocol/) | [`internals/protocol.md`](/microvms-agentd/internals/protocol.md) |
| [The trust contract for a control daemon inside a Lambda MicroVM](/microvms-agentd/internals/trust/) | [`internals/trust.md`](/microvms-agentd/internals/trust.md) |
| [Embedding agentd in your own image and driving it from your own harness](/microvms-agentd/internals/embedding/) | [`internals/embedding.md`](/microvms-agentd/internals/embedding.md) |
| [Strategy: the trust and preparation contract for coding agents on microVMs](/microvms-agentd/internals/strategy/) | [`internals/strategy.md`](/microvms-agentd/internals/strategy.md) |
| [What agent harnesses require of this platform](/microvms-agentd/internals/harness-capabilities/) | [`internals/harness-capabilities.md`](/microvms-agentd/internals/harness-capabilities.md) |
| [Agent VMs: the L3 helpers over the primitives](/microvms-agentd/internals/agent-vms/) | [`internals/agent-vms.md`](/microvms-agentd/internals/agent-vms.md) |
| [Plan: full live coverage through the `microvm` CLI](/microvms-agentd/internals/cli-coverage-plan/) | [`internals/cli-coverage-plan.md`](/microvms-agentd/internals/cli-coverage-plan.md) |
| [Data flow](/microvms-agentd/internals/architecture/data-flow/) | [`internals/architecture/data-flow.md`](/microvms-agentd/internals/architecture/data-flow.md) |
| [Module map](/microvms-agentd/internals/architecture/module-map/) | [`internals/architecture/module-map.md`](/microvms-agentd/internals/architecture/module-map.md) |
| [System overview](/microvms-agentd/internals/architecture/system-overview/) | [`internals/architecture/system-overview.md`](/microvms-agentd/internals/architecture/system-overview.md) |
| [Processes](/microvms-agentd/internals/behavior/processes/) | [`internals/behavior/processes.md`](/microvms-agentd/internals/behavior/processes.md) |
| [State machines](/microvms-agentd/internals/behavior/state-machines/) | [`internals/behavior/state-machines.md`](/microvms-agentd/internals/behavior/state-machines.md) |
| [Dead code](/microvms-agentd/internals/analysis/dead-code/) | [`internals/analysis/dead-code.md`](/microvms-agentd/internals/analysis/dead-code.md) |
| [Ownership](/microvms-agentd/internals/analysis/ownership/) | [`internals/analysis/ownership.md`](/microvms-agentd/internals/analysis/ownership.md) |
| [Risk hotspots](/microvms-agentd/internals/analysis/risk-hotspots/) | [`internals/analysis/risk-hotspots.md`](/microvms-agentd/internals/analysis/risk-hotspots.md) |
| [Components](/microvms-agentd/internals/diagrams/architecture/components/) | [`internals/diagrams/architecture/components.md`](/microvms-agentd/internals/diagrams/architecture/components.md) |
| [Sequences](/microvms-agentd/internals/diagrams/behavioral/sequences/) | [`internals/diagrams/behavioral/sequences.md`](/microvms-agentd/internals/diagrams/behavioral/sequences.md) |
| [Dependency graph](/microvms-agentd/internals/diagrams/structural/dependency-graph/) | [`internals/diagrams/structural/dependency-graph.md`](/microvms-agentd/internals/diagrams/structural/dependency-graph.md) |
| [Business logic](/microvms-agentd/internals/insights/business-logic/) | [`internals/insights/business-logic.md`](/microvms-agentd/internals/insights/business-logic.md) |
| [Contract map](/microvms-agentd/internals/insights/contract-map/) | [`internals/insights/contract-map.md`](/microvms-agentd/internals/insights/contract-map.md) |
| [Debugging guide](/microvms-agentd/internals/insights/debugging-guide/) | [`internals/insights/debugging-guide.md`](/microvms-agentd/internals/insights/debugging-guide.md) |
| [Impact analysis](/microvms-agentd/internals/insights/impact-analysis/) | [`internals/insights/impact-analysis.md`](/microvms-agentd/internals/insights/impact-analysis.md) |
| [Tech debt](/microvms-agentd/internals/insights/tech-debt/) | [`internals/insights/tech-debt.md`](/microvms-agentd/internals/insights/tech-debt.md) |
| [Glossary](/microvms-agentd/glossary/) | [`glossary.md`](/microvms-agentd/glossary.md) |
## 5. Machine-readable documentation
[Section titled “5. Machine-readable documentation”](#5-machine-readable-documentation)
Append `.md` to a page path: `/reference/cli/` becomes `/reference/cli.md`. Fetch the individual page when its location is known. [llms.txt](/microvms-agentd/llms.txt) indexes the corpus; [llms-small.txt](/microvms-agentd/llms-small.txt) and [llms-full.txt](/microvms-agentd/llms-full.txt) provide bundles. [schema.json](/microvms-agentd/schema.json) is the generated daemon wire contract.
# Glossary
> The project's vocabulary, one entry each, in the narrow sense this project uses, with the page that develops the term.
## 1. How to read this page
[Section titled “1. How to read this page”](#1-how-to-read-this-page)
The terms are alphabetical, and each entry names the page that develops it. Several of them mean something looser in the wider literature, so each definition states the narrow sense this project uses and, where a neighboring term is easy to confuse with it, says which one it is.
## 2. Terms
[Section titled “2. Terms”](#2-terms)
### Ack
[Section titled “Ack”](#ack)
`POST /v1/exec/{id}/ack`, or `microvm ack`: the call that releases a finished exec’s buffered output and starts its collection clock. Output lives until the ack, so nothing a slow reader has not seen is destroyed; a second ack answers 409, because the first released it and a 200 with an empty body would read as “the command produced no output”. See [Protocol](/microvms-agentd/internals/protocol/).
### Agent token
[Section titled “Agent token”](#agent-token)
The bearer the daemon’s control API requires. It arrives inside the `runHookPayload`, is installed by the one-shot bootstrap, and every `/v1/` route except `/v1/health` and `/v1/schema` takes it as `Authorization: Bearer`. Comparison is constant-time over bytes, and the token never enters an exec’d child’s environment. It is not the proxy token, which the platform’s endpoint wants on the same request. See [Trust](/microvms-agentd/internals/trust/).
### Agent VM
[Section titled “Agent VM”](#agent-vm)
A MicroVM launched by `microvm agent-up` (or `AgentVm` in the Python and Node packages): an image carrying Claude Code and/or Codex CLI, egress on, a Bedrock bearer token installed as `/workspace/.agent-env` for the agent to source, and a marker `/workspace/.agent-vm.json` naming the agents and models it was provisioned with. `agent-prompt` runs an agent’s headless command in it as uid 1000. It is the one opinionated layer over the generic lifecycle, and its agent-specific detail lives in one dated profile table. See [Agent VMs](/microvms-agentd/internals/agent-vms/).
### Artifacts globs
[Section titled “Artifacts globs”](#artifacts-globs)
The `artifacts` patterns in `microvm.toml`. After a sync-mode run, members of the guest’s `/workspace` matching them come back into the local directory, including when the command failed, because a failing run’s report is the artifact CI most wants. See [Run a project through a VM](/microvms-agentd/learn/tutorial/run-a-project/).
### Baseline and peak
[Section titled “Baseline and peak”](#baseline-and-peak)
The two memory figures behind `minimumMemoryInMiB`. The request selects a size class whose baseline is billed while running and whose peak, four times the baseline, is provisioned from the start; the guest’s `/proc/meminfo` reports the peak. Nothing changes size during a run. See [Platform](/microvms-agentd/internals/platform/).
### Bearer token (Bedrock)
[Section titled “Bearer token (Bedrock)”](#bearer-token-bedrock)
A short-lived credential for Bedrock minted from the caller’s own AWS credentials: a SigV4 presign of `POST https://bedrock.amazonaws.com/?Action=CallWithBearerToken`, base64, prefixed `bedrock-api-key-`, valid for at most twelve hours. `agent-up` and `AgentVm.install_access` mint it in process and write it into the guest’s environment file; it is never printed, never an argument, and never in the launch payload. In the bindings it is a `BearerToken` with no constructor and one door, `expose()`. It is not the agent token and not the proxy token. See [Agent VMs](/microvms-agentd/internals/agent-vms/).
### Bootstrap, one-shot
[Section titled “Bootstrap, one-shot”](#bootstrap-one-shot)
The rule that governs the run hook. The first `/run` carrying a token installs it and answers 200. A later `/run` carrying the identical token also answers 200, because the platform may retry its own hook. A later `/run` carrying a different token answers 409 and changes nothing. Until bootstrap lands, every control route answers 503. See [Trust](/microvms-agentd/internals/trust/).
### Build role and execution role
[Section titled “Build role and execution role”](#build-role-and-execution-role)
Two of the three AWS prerequisites an image build needs, beside an S3 bucket for the code artifact. The repository’s Terraform stack creates exactly those three; the CLI reads them from `MICROVM_BUCKET`, `MICROVM_BUILD_ROLE_ARN`, and `MICROVM_EXECUTION_ROLE_ARN`, and `microvm doctor` names whichever is missing. See [Install](/microvms-agentd/learn/tutorial/install/).
### Byte cursor
[Section titled “Byte cursor”](#byte-cursor)
The offset an exec stream resumes from. `GET /v1/exec/{id}/stream?offset=N` yields exactly the bytes after N, so a reconnecting client receives what it has not seen; a reattach past the retained window gets an explicit `gap` event naming the missing range rather than a silent skip. Measured across a real suspend and resume, the held handle resumed contiguously. See [Protocol](/microvms-agentd/internals/protocol/).
### `clientToken`
[Section titled “clientToken”](#clienttoken)
The idempotency key on the service’s create calls, and a permanent one. A token derived from a stable resource identity replays forever: after an image is deleted and recreated under the same name, the service replays the original create as a no-op and the image sits in `CREATING` with its builds never scheduled. The client scopes a create token to a single build attempt. See [Platform](/microvms-agentd/internals/platform/).
### Conformance suite
[Section titled “Conformance suite”](#conformance-suite)
The live tier of verification, run against real MicroVMs by `mise run live`. It exercises the whole surface through the `microvm` CLI, costs money, and takes roughly a quarter of an hour, which is why `mise run check` is the offline definition of done and says nothing about it. See [CLI coverage plan](/microvms-agentd/internals/cli-coverage-plan/) and [Strategy](/microvms-agentd/internals/strategy/).
### Control API
[Section titled “Control API”](#control-api)
The daemon’s own routes under `/v1/`: exec, files, health, schema. They are distinct from the platform’s lifecycle hooks, whose paths the platform fixes and which stay unversioned. Control routes answer 503 before bootstrap, never 404 and never a dropped connection. See [Protocol](/microvms-agentd/internals/protocol/).
### Control plane and session plane
[Section titled “Control plane and session plane”](#control-plane-and-session-plane)
The two halves of the client. The control plane, in `microvms-core`, wraps the service API: build, launch, suspend, resume, terminate. The session plane talks to `agentd` through the VM’s authenticated endpoint: exec, streaming, files, port forwarding. The CLI and the Python and Node bindings are thin shells over both. See [System overview](/microvms-agentd/internals/architecture/system-overview/).
### Deployment invariant
[Section titled “Deployment invariant”](#deployment-invariant)
`ENTRYPOINT []` and `CMD ["/agentd"]`, with no init system and no other process started first. It is what makes “no in-VM workload runs before bootstrap completes” true, what makes an omitted `cwd` inherit the image `WORKDIR`, and what makes identity repair sound. The daemon cannot enforce it; whoever builds the image does. See [Trust](/microvms-agentd/internals/trust/).
### Detached exec
[Section titled “Detached exec”](#detached-exec)
An exec whose record lives in the daemon, keyed by a caller-minted `exec_id`, independent of any client connection. Start, poll, and ack are separate calls; a retried start returns the original exec without spawning a second child; a detached exec outlives the proxy token that started it, so start, rotate, poll, ack is a normal sequence. `microvm exec --detach` starts one and `--poll ` reads it back. See [Embedding](/microvms-agentd/internals/embedding/).
### Drift gate
[Section titled “Drift gate”](#drift-gate)
The offline check that compares the service constraints hardcoded in the client against the pinned botocore model, so a limit the service changes cannot stay wrong silently. `microvm constants` emits every constraint the client believes, for that gate. See [constants](/microvms-agentd/reference/commands/constants/).
### Egress
[Section titled “Egress”](#egress)
Outbound network traffic. `--egress` requests the managed internet connector; omitting it does not disable internet access. No internet egress requires a VPC without an IGW or NAT gateway, attached through a custom connector. `--deny-egress` only sets proxy variables. See [Networking](/microvms-agentd/internals/networking/).
### Envelope
[Section titled “Envelope”](#envelope)
The one JSON document every `microvm` command writes to stdout under `--json`, with progress on stderr. Success carries `type` and `data`; failure carries a stable `code`, a mapped `exitCode`, and `suggestions`. The one exception is `exec --stream`, which emits NDJSON events and the envelope last. See [The envelope](/microvms-agentd/reference/envelope/).
### Exit-code catalog
[Section titled “Exit-code catalog”](#exit-code-catalog)
The mapping from a failure’s stable `code` to the process exit status a caller branches `$?` on. It is part of the manifest and it is the contract; the prose beside each code is rewritten freely, so a matcher over the message breaks on a wording change that broke nothing. `ERR_EXEC_FAILED` means the sandbox worked and the command inside it exited non-zero. See [Exit codes](/microvms-agentd/reference/exit-codes/).
### Hooks: ready, validate, run
[Section titled “Hooks: ready, validate, run”](#hooks-ready-validate-run)
The platform’s lifecycle endpoints, served by the daemon under a prefix the platform fixes. `ready` and `validate` are build-time hooks: the build calls them in the snapshot VM to decide whether the snapshot it just produced is usable, before any instance exists. `run` fires at launch and delivers the `runHookPayload`. All three must answer on `AGENTD_PORT`, which must agree with the port in the create call (`--port`, default 9000). See [Platform](/microvms-agentd/internals/platform/).
### Identity repair
[Section titled “Identity repair”](#identity-repair)
What the daemon does at startup to the files that are supposed to be unique per machine. One image is snapshotted once and restored many times, so every VM shares the snapshot’s machine-id, hostname, boot\_id, and random seed; the daemon replaces them unless `AGENTD_REPAIR_IDENTITY` opts out, and `microvm health` reports what it did. See [Trust](/microvms-agentd/internals/trust/).
### `idlePolicy`
[Section titled “idlePolicy”](#idlepolicy)
The launch-time policy under which the platform suspends an idle VM and then terminates a suspended one after its window. Idleness is measured by inbound traffic through the endpoint proxy, so a computing workload with no inbound requests reads as idle, and a request from inside the guest cannot reset the timer. The keepalive is yours and it runs outside the VM. See [Platform](/microvms-agentd/internals/platform/).
### Image and image version
[Section titled “Image and image version”](#image-and-image-version)
A MicroVM image is built from a Dockerfile and carries a snapshot; a VM launches from an image version. `minimumMemoryInMiB` lives on the version, so the only way to learn a running VM’s memory from the API is to fetch its version. The snapshot carries a one-week minimum retention, so reuse an image with `--image` rather than rebuilding. See [Platform](/microvms-agentd/internals/platform/).
### `microvm manifest`
[Section titled “microvm manifest”](#microvm-manifest)
The command that prints the binary’s whole contract as JSON: every command, its flags, its response type, the envelope schema, and the exit-code catalog. It needs no credentials, no region, and no network. The Reference tier of this site is generated from it, and it outranks every page here. See [Reference](/microvms-agentd/reference/).
### `microvm.toml`
[Section titled “microvm.toml”](#microvmtoml)
The CLI’s configuration file. A typed flag beats the file, and the file beats the built-in default. It is where `artifacts` globs live and where an `image` can be pinned for sync mode. See [CLI](/microvms-agentd/reference/cli/).
### Name registry
[Section titled “Name registry”](#name-registry)
The local record `--vm-name` writes when a VM is kept, and `microvm attach` writes for a VM this machine did not launch. It lives in the CLI’s state directory, carries the endpoint, the agent token, and the MicroVM id, and resolves a `--name` with zero AWS calls. It is a local fact, so a VM launched elsewhere is addressed by the explicit triple until it is attached. See [CLI](/microvms-agentd/reference/cli/).
### Network connector
[Section titled “Network connector”](#network-connector)
The launch-time grant that gives a VM a network path, spelled as an ARN of the form `arn:aws:lambda::aws:network-connector:aws-network-connector:`, never as the bare name. `ALL_INGRESS` and `INTERNET_EGRESS` are the two the client uses; `SHELL_INGRESS` is what `microvm shell` requires. See [Platform](/microvms-agentd/internals/platform/).
### Profile table
[Section titled “Profile table”](#profile-table)
The one place agent-specific detail lives: for each agent the npm package, the default model (an inference-profile id), and the date and region it was last verified. Every default is overridable (`--claude-model`, `--codex-version`, `AgentSpec(model=..., cli_version=...)`), and the table is what bounds the churn the agent layer takes on. See [Agent VMs](/microvms-agentd/internals/agent-vms/).
### Proxy token
[Section titled “Proxy token”](#proxy-token)
The credential the platform’s endpoint proxy wants on every request: an `X-aws-proxy-auth` JWE scoped to one MicroVM id and one set of ports, beside an `X-aws-proxy-port` header naming the target port. It is minted by `CreateMicrovmAuthToken`, which returns a map of header names rather than a string, and it expires within sixty minutes, so the client refreshes it at thirty. It is not the agent token. See [Embedding](/microvms-agentd/internals/embedding/).
### `--reuse`
[Section titled “--reuse”](#--reuse)
The `build --project` flag that names the image after a content hash of the project’s dependency files and skips the build when an image of that name already exists. The lockfile is the identity: an edit to `uv.lock` alone moved the hash and produced a new image, while unchanged files answered in under a second with `reused: true`. See [Platform](/microvms-agentd/internals/platform/).
### `runHookPayload`
[Section titled “runHookPayload”](#runhookpayload)
The string passed to `RunMicrovm` and delivered to the daemon’s `/run` hook at launch, wrapped one JSON layer deeper than the caller wrote it. It carries the agent token, an optional launch environment, and the identity seed for a verified tunnel, and it is capped at 4096 bytes. It is the only per-VM secret channel: a secret baked into the image is shared by every VM restored from the snapshot. See [Platform](/microvms-agentd/internals/platform/).
### Snapshot
[Section titled “Snapshot”](#snapshot)
The captured state of the build VM after the `ready` and `validate` hooks succeed. Every VM launched from the image is a restore of that snapshot, so every byte in it is identical across VMs, which is what identity repair exists for and why no secret belongs in an image. See [Trust](/microvms-agentd/internals/trust/).
### Sync mode
[Section titled “Sync mode”](#sync-mode)
What `microvm run` becomes when its positional argument is a directory rather than a binary: a pack-run-collect round trip against an existing image. The tree is packed locally, uploaded to `/workspace`, the command runs there, and the artifacts come back. `microvm sync` pushes a directory into a running VM uploading only what changed. See [Run a project through a VM](/microvms-agentd/learn/tutorial/run-a-project/).
### Tar confinement
[Section titled “Tar confinement”](#tar-confinement)
How the daemon extracts an archive so that no member can write outside its target. The extraction root is opened once and every member is created relative to that descriptor with `openat2` under `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS`, so a member whose path would traverse a symlink is refused with 400. It needs Linux 5.6 or newer, and the guest kernel is 6.1. See [Protocol](/microvms-agentd/internals/protocol/).
### Two tiers of document
[Section titled “Two tiers of document”](#two-tiers-of-document)
The standing rule about reliability under Internals. The hand-written documents carry measured findings and design rationale and win any disagreement; the generated categories carry `path:line` citations pinned to a commit, which is their value and their expiry date. See [Internals](/microvms-agentd/internals/).
# Internals
> Protocol, trust boundaries, measured AWS behavior, and source analyses.
Use [Learn](/microvms-agentd/learn/) for walkthroughs and [Reference](/microvms-agentd/reference/) for command syntax. These pages explain contracts and design decisions.
| Document | Purpose |
| ------------------------------------------------------------------------ | ------------------------------------------------------- |
| [Platform](/microvms-agentd/internals/platform/) | Dated AWS measurements and corrections |
| [Networking](/microvms-agentd/internals/networking/) | VPC connectors and internet isolation |
| [Protocol](/microvms-agentd/internals/protocol/) | Authentication, execution, files, and streaming |
| [Trust](/microvms-agentd/internals/trust/) | Bootstrap, workload boundaries, credentials, and egress |
| [Embedding](/microvms-agentd/internals/embedding/) | Image contract and custom harness integration |
| [Agent VMs](/microvms-agentd/internals/agent-vms/) | Agent lifecycle and Bedrock integration |
| [Strategy](/microvms-agentd/internals/strategy/) | Scope and priorities |
| [Harness capabilities](/microvms-agentd/internals/harness-capabilities/) | Integration requirements and gaps |
Generated source analyses cover [architecture](/microvms-agentd/internals/architecture/system-overview/), [behavior](/microvms-agentd/internals/behavior/processes/), [risks](/microvms-agentd/internals/analysis/risk-hotspots/), [diagrams](/microvms-agentd/internals/diagrams/architecture/components/), and [maintenance](/microvms-agentd/internals/insights/impact-analysis/). The [CLI coverage plan](/microvms-agentd/internals/cli-coverage-plan/) is retained as history.
Source citations link to a commit, but the cited line may describe older code. Prefer the current executable contract and implementation when a page conflicts with them. AWS runtime observations have a date, region, and API version; later corrections retain the original measurement.
# Agent VMs: the L3 helpers over the primitives
> Status: specified and built 2026-09-10. This document is the specification the code in microvms-core/src/agents/ and the agent-up / agent-prompt commands implement, and the record…
Status: specified and built 2026-09-10. This document is the specification the code in `microvms-core/src/agents/` and the `agent-up` / `agent-prompt` commands implement, and the record of the scope decision it changed.
## The three layers, named
[Section titled “The three layers, named”](#the-three-layers-named)
The repository has always had two layers and never named them. This document names three.
**L1, the primitives.** The daemon’s wire routes (`exec`, `fs`, `health`, the lifecycle hooks) and the control-plane operations (`CreateMicrovmImage`, `RunMicrovm`, suspend, resume, terminate, the auth-token mint). Every one is generic, measured against the real service, and closed against the platform’s traps. `docs/PROTOCOL.md` and `docs/PLATFORM.md` own this layer.
**L2, the lifecycle.** `Sandbox` and `Session` in `microvms-core`, and the `microvm` commands over them: `run`, `build --reuse --project`, `exec`, `cp`, `sync`, named VMs, `attach`, `suspend`, `resume`, `terminate`. Still generic: nothing here knows what a coding agent is.
**L3, the agent VMs.** One call that gets you a VM with a coding agent installed, model access wired, running as a non-root user, and a second call that hands the agent a task. Everything L3 does is a composition of L2 calls that `examples/coding-agents-on-bedrock/` already performed in a shell script; what L3 adds is that the composition lives in the library, so the CLI, the bindings, and a harness call one function instead of re-deriving the seven-step recipe and its four measured traps.
## The rule this changes, and how the cost is bounded
[Section titled “The rule this changes, and how the cost is bounded”](#the-rule-this-changes-and-how-the-cost-is-bounded)
Until this document the rule was: platform code exposes only generic primitives, and agent-specific detail (CLI installs, model ids, credential wiring, config files) lives only in `examples/` and docs. `docs/HARNESS-CAPABILITIES.md` lists “harness provider classes” as an explicit non-goal, and `docs/STRATEGY.md` says “not an orchestrator”. The reason was churn: an agent CLI’s flags, config format, and model ids move on a vendor’s cadence, and a platform library that hardcodes them decays.
L3 crosses that line on purpose, for two agents, with the churn contained:
1. **Every agent-specific fact is data in one table**, `agents::profile`, not code spread across the crate. Each entry carries the date and version it was verified against, the way `docs/PLATFORM.md` dates a platform claim. Updating a profile is a one-file edit plus a live run.
2. **Every default is overridable at the call site.** The model id, the agent CLI’s install line, the headless command template, and the environment file are each parameters with a default; a caller whose vendor moved first passes the new value and does not wait for a release.
3. **The generic layer stays generic.** L1 and L2 gained one read-only accessor for L3 (`Sandbox::port`) and nothing else. `agents` depends on `control`, `session`, and `sandbox`; nothing depends on `agents`.
4. **This is still not an orchestrator.** L3 provisions one VM and runs one prompt in it. Scheduling, retries across VMs, multi-agent coordination, and turn loops stay outside this repository, per `docs/STRATEGY.md`.
The non-goals paragraph in `docs/HARNESS-CAPABILITIES.md` still holds for harness *provider classes* (a Harbor `BaseEnvironment`, an eve backend): those import the harness’s packages and live in its ecosystem. L3 is the layer such a class would call.
## Requirements
[Section titled “Requirements”](#requirements)
Written in the EARS shapes the rest of `spec/` uses. `AGENT-n` is the id.
* **AGENT-1.** The `agents` module shall expose exactly two agent profiles, `ClaudeCode` and `Codex`, as a closed enum, so a caller cannot name an agent the library has no recipe for.
* **AGENT-2.** For any non-empty set of profiles, `agents::dockerfile` shall produce a Dockerfile that starts with the client’s own agentd stanza lines (`FROM` the managed base’s `docker_ref`, `COPY agentd`, `chmod`), adds the union of the profiles’ install layers, creates uid and gid 1000 by appending to `/etc/passwd` and `/etc/group`, creates and hands `/workspace` to that uid, sets `WORKDIR /workspace`, and ends with the stanza’s `ENV`, `EXPOSE`, `ENTRYPOINT []`, `CMD ["/agentd"]` lines. The output shall pass every local guard `Sandbox::preflight` runs.
* **AGENT-3.** When a profile set is unchanged and the daemon binary is unchanged, the derived image name shall be unchanged, so `build --reuse` semantics hold: the name is `agent-vm--<12 hex of the artifact content hash>`.
* **AGENT-4.** `agents::bedrock::mint` shall produce a Bedrock bearer token from the caller’s AWS credential chain by SigV4 query presigning, with the lifetime the caller asks for, capped at the service’s 12-hour ceiling, and shall never write the token to a log, a `Debug` impl, or a command line.
* **AGENT-5.** `install_access` shall deliver model credentials to a running VM as files only (`/workspace/.agent-env` mode `0600`, `/workspace/.codex/config.toml` when Codex is present), through the authenticated file route, then run one root exec that hands `/workspace` to uid 1000. The environment file shall set `PATH` explicitly.
* **AGENT-6.** `install_access` shall write `/workspace/.agent-vm.json` naming the installed profiles and their models, so a later process that holds only the VM’s identifiers can learn which agent to prompt without a local record.
* **AGENT-7.** `prompt` shall run the agent’s headless command as uid 1000 and gid 1000, with `/workspace` as the working directory, sourcing the environment file first, and shall refuse locally (zero wire calls) a task string that is empty.
* **AGENT-8.** `agent-up` against a name that is already registered shall not build or launch; it shall attach, mint a fresh token, re-run `install_access`, and report `reused: true`. This is how a 12-hour token is refreshed on a long-lived VM.
* **AGENT-9.** `agent-up` shall launch with egress requested, since neither agent can reach Bedrock without it, and shall register the name only after the launch succeeds and the credentials are installed.
* **AGENT-10.** `agent-prompt` without `--agent` shall read the marker from AGENT-6 and choose the sole installed profile; when two are installed it shall refuse and name both.
* **AGENT-11.** Neither command shall introduce a new exit row; failures map onto the existing table (`ERR_PRECONDITION`, `ERR_INVALID_ARG`, `ERR_EXEC_FAILED`, `ERR_NAME_TAKEN`, and the wire rows).
## The profile table
[Section titled “The profile table”](#the-profile-table)
`agents/profile.rs`. Each row is a `const` with the fields below; the values are the ones `examples/coding-agents-on-bedrock/` measured working on 2026-09-02 in us-east-1.
| Field | ClaudeCode | Codex |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `id` | `claude-code` | `codex` |
| `install_lines` | `dnf install nodejs22 nodejs22-npm python3 git tar gzip which findutils procps-ng`; `npm install -g @anthropic-ai/claude-code` | the same `dnf` line; `npm install -g @openai/codex` |
| `default_model` | `global.anthropic.claude-opus-5` | `global.openai.gpt-5.6-sol` |
| `env` | `CLAUDE_CODE_USE_BEDROCK=1`, `ANTHROPIC_MODEL=`, `AWS_BEARER_TOKEN_BEDROCK=` | `OPENAI_API_KEY=` |
| `env_extra` | `CLAUDE_CODE_USE_BEDROCK=1`, `ANTHROPIC_MODEL=`, `AWS_BEARER_TOKEN_BEDROCK=` | `AWS_BEARER_TOKEN_BEDROCK=` (what Codex reads on this host), `OPENAI_API_KEY=` (the `env_key` the config declares) |
| `config_files` | none | `/workspace/.codex/config.toml`: provider `bedrock`, `model_reasoning_effort = medium` (Codex has no metadata for a Bedrock model id and otherwise sends none; a no-effort run declined a task once in five on 2026-09-10), `base_url = https://bedrock-runtime..amazonaws.com/openai/v1`, `web_search = disabled` (Codex advertises hosted web search by default and bedrock-runtime fails the turn), `env_key = OPENAI_API_KEY`, `wire_api = responses`, `model = ` |
| `headless_command(task)` | `claude -p --allowedTools Bash,Read,Edit,Write,Grep,Glob` | `codex exec --skip-git-repo-check -s workspace-write ` |
| `verified` | 2026-09-10, us-east-1, `@anthropic-ai/claude-code` latest on that date | 2026-09-10, us-east-1, `@openai/codex` 0.154.0, bedrock-runtime host |
Shared, not per profile: `HOME=/workspace`, `PATH=/usr/local/bin:/usr/bin:/bin`, `AWS_REGION=`, uid and gid 1000, `WORKDIR /workspace`. The `dnf` line is emitted once when both profiles are present.
The install lines pin nothing, so an image built today carries today’s CLI. That is a choice, not an oversight: the `--reuse` hash covers the Dockerfile text, and a pinned version would make every vendor release a Dockerfile edit and a rebuild. A caller who wants a pin passes `--claude-version` or `--codex-version`, which appends `@` to the npm install and therefore changes the hash.
## The Bedrock bearer token
[Section titled “The Bedrock bearer token”](#the-bedrock-bearer-token)
`agents/bedrock.rs` ports the `aws-bedrock-token-generator` recipe to Rust on the `aws-sigv4` crate core already carries, so the CLI drops its `uvx` dependency for this path and the token never touches a subprocess’s argv.
The recipe, verbatim from the reference implementation, is recorded in that file’s module docs with the source permalink and the date it was read. In outline: presign a `POST` to `https://bedrock.amazonaws.com/` (the host is fixed and not regional; the region enters only the credential scope) with `Action=CallWithBearerToken` in the query, service `bedrock`, signed header `host` only, `X-Amz-Expires` at the requested lifetime (default and ceiling 43200 seconds), then base64-encode the presigned URL with its scheme stripped and `&Version=1` appended, and prefix `bedrock-api-key-`. The unit test asserts the shape (prefix, base64 alphabet, the decoded URL’s host, action, and expiry) against fixed credentials and a fixed clock; the live conformance check asserts the token is accepted by the real service, because a presign that is one canonical byte off still looks like a token.
The token is a `BearerToken` newtype whose `Debug` prints its length only, following `.erpaval/solutions/best-practices/credential-structs-never-derive-debug.md`.
## The core API
[Section titled “The core API”](#the-core-api)
```rust
1
pub mod agents {
2
pub enum Agent { ClaudeCode, Codex } // AGENT-1
3
pub struct AgentSpec { agent: Agent, model: Option, cli_version: Option }
4
pub struct Profile { /* the table row */ }
5
impl Agent { pub fn profile(self) -> &'static Profile; pub fn as_str(self) -> &'static str; }
6
7
pub const AGENT_UID: u32 = 1000; pub const AGENT_GID: u32 = 1000;
8
pub const WORKDIR: &str = "/workspace";
9
pub fn dockerfile(specs: &[AgentSpec], base: &BaseImage, port: u16) -> String; // AGENT-2
10
pub fn image_stem(specs: &[AgentSpec]) -> String; // AGENT-3
11
12
pub mod bedrock {
13
pub struct BearerToken(..); // Debug = length only
14
pub struct Minted { token: BearerToken, expires_at: SystemTime }
15
pub async fn mint(region: &Region, lifetime: Duration) -> Result; // AGENT-4
16
pub fn mint_with(credentials, region, lifetime, now) -> Result; // pure, tested
17
}
18
19
pub struct BedrockAccess { region: Region, token: BearerToken }
20
pub struct GuestFile { path: String, contents: Vec, mode: &'static str }
21
pub fn provisioning_files(specs, access) -> Vec; // AGENT-5, AGENT-6
22
pub async fn install_access(session: &Session, specs, access) -> Result<(), Error>;
23
pub async fn installed_agents(session: &Session) -> Result, Error>; // AGENT-10
24
pub fn prompt_request(spec: &AgentSpec, task: &str, opts: &PromptOptions) -> Result; // AGENT-7
25
pub async fn prompt(session: &Session, spec, task, opts) -> Result;
26
27
pub struct AgentVm { sandbox: Sandbox, specs: Vec }
28
impl AgentVm {
29
pub fn new(sandbox: Sandbox, specs: Vec) -> Result; // refuses an empty set
30
pub fn claude_code(sandbox: Sandbox) -> Self; // the names the request used
31
pub fn codex(sandbox: Sandbox) -> Self;
32
pub fn image_request(&self, binary: Vec, artifact_uri, build_role_arn, size) -> CreateImageRequest;
33
pub fn launch_request(&self, image_identifier, execution_role_arn) -> RunRequest; // egress on
34
pub async fn build(&mut self, request) -> Result<&Image, Error>;
35
pub async fn launch(&mut self, request) -> Result<&Session, Error>;
36
pub async fn install_access(&self, access: &BedrockAccess) -> Result<(), Error>;
37
pub async fn prompt(&self, agent: Agent, task: &str, opts) -> Result;
38
pub async fn terminate(&mut self, opts: TeardownOpts) -> TeardownReport;
39
pub fn sandbox(&self) -> &Sandbox; pub fn session(&self) -> Option<&Session>;
40
}
41
}
```
The artifact upload stays the caller’s, exactly as it is for `Sandbox::build_image`: S3 is not in core’s dependency set, and `AgentVm::image_request` returns the request whose `code_artifact_uri` the caller fills before calling `build`. The free functions exist beside the struct because the CLI’s refresh and prompt paths hold an attached `Session` and no `Sandbox`; `AgentVm`’s methods delegate to them.
## The CLI surface
[Section titled “The CLI surface”](#the-cli-surface)
Two flat commands, in the tree beside `run` and `exec`. Flat rather than a nested `agent` group because the manifest generator, its cross-check test, and the docs site’s Reference generator all read one level of subcommands, and a nested group would be a second command grammar for two commands. `port-forward` set the hyphenated precedent.
**`microvm agent-up [BINARY] --vm-name NAME [--agent claude-code|codex]... [--claude-model ID] [--codex-model ID] [--claude-version V] [--codex-version V] [--project DIR] [--memory MIB] [--token-ttl-hours H] [--max-idle-sec S] [--suspended-sec S] [--auto-resume] [--max-duration-sec S] [--port P] [--state-dir DIR] `**
Builds the profile image under its content-hash name (reusing an existing one), launches a kept, named VM with egress, mints a Bedrock token, installs it, optionally uploads `--project`’s tree to `/workspace` (packed by the same `sync` module `run ` uses), and registers the name. Against a registered name it takes the AGENT-8 refresh path. `--agent` defaults to `claude-code`; both may be given. `--memory` defaults to 1024, the example’s measured choice for peaky agent sessions. Envelope type `microvm.agent`; keys `vmName`, `microvmId`, `endpoint`, `agentToken`, `imageIdentifier`, `imageName`, `imageReused`, `vmReused`, `agents` (`[{agent, model, cliVersion, headlessCommand}]`), `credentialExpiresAt` (epoch seconds), `workdir`, `project` (`{workdir, uploadedBytes, uploadedMembers}` or null), `agentd`.
**`microvm agent-prompt TASK [--agent A] [--timeout SEC] [--detach] [--exec-id ID] `**
Runs the headless agent over the task as uid 1000 in `/workspace`. Without `--agent`, reads the guest marker (AGENT-10). `--detach` starts and returns the exec id for `exec --poll`; the default waits up to `--timeout` (900 s, client-side; the daemon-side exec carries no budget, as `exec` does) and acks. On the refresh path a `--project` tree is uploaded before the credential install, so the same `chown` hands it to the agent. Envelope type `microvm.agent.prompt`; keys `execId`, `agent`, `model`, `phase`, `exitCode`, `stdout`, `stderr`, `truncated`. A non-zero agent exit earns `ERR_EXEC_FAILED` on a success envelope, as `exec` does.
Teardown is `microvm terminate NAME`, which already exists. Streaming is `microvm exec --stream --name NAME --user 1000 --group 1000 ''`, and the `agents[].headlessCommand` key in the `agent-up` envelope is there so a caller can do that without knowing the template.
## Verification
[Section titled “Verification”](#verification)
`mise run check` covers: the Dockerfile derivation against every preflight guard, the image-name stability property (same specs and binary give the same name; a version pin changes it), the token shape under fixed credentials and clock, the provisioning file set per profile combination, `prompt_request` field-by-field, the marker round trip through the session recorder, the CLI guards (both commands fail closed through the seam and name their door; a registered name skips the build door; an empty task is refused with zero doors), and the manifest count.
The live half, per `CLAUDE.md`’s rule, is `drive_agent_vm` in `conformance/run_rs.py`: `agent-up` with both profiles builds or reuses the image and launches; the marker names both; the credential file exports the variable each installed agent reads (names only, so no value reaches a log); `agent-prompt --agent claude-code` completes a Bash task with exit 0 and a tool-call in its transcript; `agent-prompt --agent codex` creates a file that a following `exec` can `cat`; a second `agent-up` against the same name reports `vmReused: true` and a later `credentialExpiresAt`; `terminate NAME` releases the name. That section needs Bedrock access to both default models in the conformance account and is the only part of the suite that does; it reports the model ids it used.
## The bindings
[Section titled “The bindings”](#the-bindings)
`microvms-py` and `microvms-js` carry the layer as `AgentVm`, `AgentSpec`, and `BearerToken`, plus four module functions for a caller who holds only a session: `installed_agents`, `install_agent_access`, `prompt_agent`, and `mint_bedrock_token` (`installedAgents`, `installAgentAccess`, `promptAgent`, `mintBedrockToken`). The sequence is the CLI’s, one method per step, and the upload stays the caller’s because S3 is not in the core’s dependency set:
```python
1
vm = microvms.AgentVm(
2
microvms.Region.us_east_1(),
3
[microvms.AgentSpec.claude_code(), microvms.AgentSpec.codex()],
4
)
5
image = vm.find_image(binary=agentd, build_role_arn=build_role)
6
if image is None:
7
name = vm.image_name(binary=agentd, build_role_arn=build_role)
8
s3.put_object(
9
Bucket=bucket,
10
Key=f"{name}.zip",
11
Body=vm.build_artifact(binary=agentd, build_role_arn=build_role),
12
)
13
image = vm.build_image(
14
binary=agentd,
15
code_artifact_uri=f"s3://{bucket}/{name}.zip",
16
build_role_arn=build_role,
17
).identifier
18
vm.launch(image_identifier=image, execution_role_arn=exec_role)
19
token = vm.install_access() # minted in process; token.expires_at
20
result = vm.prompt_sync("codex", "Create hello.py that prints hello, run it.")
21
vm.terminate(delete_image=False)
```
The binding holds the same lock the sandbox and every session it hands out hold, so a `terminate` and a session call cannot interleave; `vm.sandbox` and `vm.session` reach the same VM for suspend, resume, and file transfer. `terminate(delete_image=True)` deletes only an image the object itself built, the sandbox’s existing rule; a reused image is reported `image_deleted: false` with no failure, and the caller deletes it. The core’s `AgentVm` owns its sandbox, which one binding class cannot share, so the bindings drive the layer through the core’s free functions (`image_request_for`, `launch_request_for`, `install_access`, `prompt`, `spec_for`) with the specs kept beside the lock. No refusal lives in a binding: an unknown agent name, an empty or repeated spec set, a prompt for an agent the VM does not carry, a blank task, and a token lifetime past the ceiling are all the core’s messages. A `BearerToken` has no constructor and shows only its length; `expose()` is the one door to the text, for a caller writing it into an environment themselves.
The Python stub is regenerated from the compiled module (`mise run stubs`) and the Node `index.d.ts` from the napi surface, so both type surfaces follow the Rust.
## Phase 2, deliberately out of this change
[Section titled “Phase 2, deliberately out of this change”](#phase-2-deliberately-out-of-this-change)
* `--stream` on `agent-prompt`. The `exec --stream` path exists and the envelope publishes the command to stream.
* `microvm.toml` keys for the agent flags. `merge_config` is `run`-shaped and the agent flags would need their own precedence table.
* Other model access paths (a vendor API key, an AgentCore gateway). `BedrockAccess` is a struct rather than an enum until a second variant exists to name.
# Dead code
> The compiler already owns most of this question. rustc's unused_imports and dead_code are warn-by-default, Rust 1.97 added dead_code_pub_in_binary, and cargo clippy --all-targets…
The compiler already owns most of this question. `rustc`’s `unused_imports` and `dead_code` are warn-by-default, Rust 1.97 added `dead_code_pub_in_binary`, and `cargo clippy --all-targets --all-features` over all seven workspace members exits 0 with zero warnings — verified on a forced fresh re-check, not a warm cache, with every `.rs` file in `agentd`, `microvms-cli`, `microvms-core`, `microvms-js`, `microvms-py`, `model`, and `protocol` touched first. Nothing crate-private is dead, and no import is unused anywhere `rustc` can see. Restating that would add nothing.
What the compiler cannot decide is what this file covers:
* A `pub` item in `microvms-core` with zero in-repo callers may still be live API. The crate has three consumers (`microvms-cli`, `microvms-py`, `microvms-js`), and `protocol` is consumed by `agentd` and `microvms-core`.
* Items reached only through `#[pyclass]` / `#[pymethods]` / `#[pyfunction]` / `#[pymodule]` or `#[napi]` are dispatched by CPython and Node, never by a Rust caller. They read as uncalled and are not dead. `#[derive(Serialize, Deserialize)]` impls, trait impls satisfying a bound, and `#[test]` functions read the same way.
* napi-rs renames snake\_case Rust to camelCase JS, so a `#[napi]` export exercised only from `microvms-js/__test__/*.mjs` is invisible to a snake\_case search.
Method: a symbol is a candidate when no edge of kind `calls | references | instantiates | imports | implements | extends` targets it in the CodeGraph index (`.codegraph/codegraph.db`, 4,430 nodes / 16,638 edges), and no textual reference to it — or to its camelCase / `#[napi(js_name)]` / `#[pyo3(name)]` alias — exists anywhere in the git-tracked tree. `contains` edges are excluded because every symbol is `contains`-reachable from its own file node. The funnel: 1,398 zero-inbound Rust symbols → 551 after dropping test files and `#[cfg(test)]` modules → 465 after dropping two whole-file test-only modules → 112 after the alias-aware reference search → 2 after removing 110 trait and language-protocol members → **1** after hand-dropping a trait associated type ([`microvms-js/src/exec.rs:267`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/exec.rs#L267), `type Return = ()` inside `impl AsyncGenerator for ExecStream`) that the automated filter missed because it matched trait methods but not associated types. No dead-code analyzer is integrated in this repo (no `cargo-udeps`, `cargo-machete`, `vulture`, or `knip` in `mise.toml`, `Cargo.toml`, `deny.toml`, or `.github/workflows/`), so the index is the analyzer.
## Unreferenced exports
[Section titled “Unreferenced exports”](#unreferenced-exports)
| Symbol | Path | Last modified |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- |
| `SessionBuilder::with_timeout` | [`microvms-core/src/session/mod.rs:508`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L508) | 2026-08-15 |
**Confidence: high.** `git grep -n "with_timeout\|withTimeout" -- .` over the whole git-tracked tree returns exactly one line — the declaration. Every sibling on the same builder has a caller: `with_minter` ([`microvms-core/src/session/mod.rs:479`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L479)) from `microvms-cli/src/seam.rs`, `microvms-core/src/sandbox.rs`, and `microvms-core/tests/turmoil_client.rs`; `with_proxy_auth` ([`microvms-core/src/session/mod.rs:487`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L487)) from `microvms-core/tests/turmoil_client.rs`; `with_backend` ([`microvms-core/src/session/mod.rs:494`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L494)) from `microvms-cli/src/guards.rs` and `microvms-core/tests/turmoil_client.rs`; `with_port` ([`microvms-core/src/session/mod.rs:501`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L501)) from `microvms-cli/src/seam.rs` and `microvms-core/src/sandbox.rs`.
The field it writes is live — only the setter is unreached. `SessionBuilder::build` reads `self.timeout` at [`microvms-core/src/session/mod.rs:517`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L517) and [`:529`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L529), and `Session::run` back-fills a per-request `None` from it at [`microvms-core/src/session/mod.rs:119-120`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L119-L120). `Session::builder` is the sole constructor ([`microvms-core/src/session/mod.rs:230-231`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L230-L231)) and seeds every path with `DEFAULT_REQUEST_TIMEOUT` ([`microvms-core/src/session/mod.rs:237`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L237)).
**What would falsify this.** Three conditions hold; any one of them failing moves this row out of the table.
1. No unseen downstream consumer. `microvms-core` is a library crate, so a `pub` method is reachable by anything that depends on it. The package is published on crates.io and has language bindings. This analysis cannot establish the absence of downstream callers, so removing public API requires a compatibility decision; lack of local references is insufficient.
2. No binding re-exports it. `microvms-py/microvms.pyi` contains no `with_timeout`, and no `withTimeout` exists in `microvms-js`.
3. No host runtime dispatches to it. Its only attribute is `#[must_use]` — no `#[napi]`, no `#[pymethods]`.
**Related defect in the same construct.** The builder’s doc comment at [`microvms-core/src/session/mod.rs:229`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L229) reads “A builder, for the cases that need a port, a timeout, or a custom backend.” The port case and the backend case each have callers; the timeout case has none. The comment asserts a motivating case that does not exist in the tree.
## Unreferenced files
[Section titled “Unreferenced files”](#unreferenced-files)
*none*
All 200 git-tracked files resolve to an inbound reference. Every non-root `.rs` file has a `mod` declaration or is cargo-auto-discovered by one of the patterns `*/src/lib.rs`, `*/src/main.rs`, `*/build.rs`, `*/src/bin/*.rs`, `*/tests/*.rs`, `*/examples/*.rs`. Four files that a basename search calls orphans, each cleared against its real invocation site:
| File | Reached by |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvms-py/tests/test_stubs.py` | `pytest microvms-py/tests -q` at [`.github/workflows/ci.yml:308`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L308); pytest auto-discovers `test_*.py`, so no file names it |
| `microvms-js/__test__/support/decimal.mjs` | [`microvms-js/__test__/cost.mjs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/cost.mjs#L40) |
| `microvms-js/__test__/support/sse.mjs` | [`microvms-js/__test__/cost.mjs:41`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/cost.mjs#L41), [`microvms-js/__test__/errors.mjs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/errors.mjs#L40), [`microvms-js/__test__/exec.mjs:44`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/exec.mjs#L44), [`microvms-js/__test__/process.mjs:31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/process.mjs#L31), [`microvms-js/__test__/session.mjs:33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/session.mjs#L33) |
| `conformance/infra/main.tf` | `terraform -chdir=conformance/infra` at [`mise.toml:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L53), [`mise.toml:93`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L93), [`mise.toml:307`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L307), [`mise.toml:538`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L538) |
Two files are compiled only under `cfg(test)` and are live test code, not dead source: `microvms-cli/src/guards.rs` (inner `#![cfg(test)]` at [`microvms-cli/src/guards.rs:20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L20), plus `#[cfg(test)] mod guards;` at [`microvms-cli/src/main.rs:36-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L36-L37)) and `microvms-core/src/control/fake.rs` (`#[cfg(test)] pub(crate) mod fake;` at [`microvms-core/src/control/mod.rs:928-929`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L928-L929)).
## Dead imports
[Section titled “Dead imports”](#dead-imports)
| Path | Symbol | Imported from |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`microvms-cli/src/commands/lifecycle.rs:1168`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L1168) | `_DocsOnly` (alias of `ControlPlane`) | `microvms_core::control::ControlPlane`, re-bound from [`microvms-cli/src/commands/lifecycle.rs:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L70) |
| [`microvms-cli/src/commands/attached.rs:932`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L932) | `_DocsOnly` (alias of `ErrorKind`) | `microvms_core::ErrorKind`, re-bound from [`microvms-cli/src/commands/attached.rs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L40) |
**Confidence: high that nothing names `_DocsOnly`; do not delete either line on its own.** Both carry `#[allow(unused_imports, reason = …)]`, so `rustc` never reports them, and both are the only `allow` attributes of this family in the workspace. Neither is independently removable. Measured by copying the git-tracked tree to a scratch directory, deleting each three-line construct (doc comment, attribute, `use`), and rebuilding:
* `cargo clippy -p microvms-cli --all-targets` emits two new warnings — `unused import: ControlPlane` and `unused import: ErrorKind` — because the `_DocsOnly` re-export is what consumes the code-level import at [`microvms-cli/src/commands/lifecycle.rs:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L70) and [`microvms-cli/src/commands/attached.rs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L40). Under `-D warnings` that is a build failure.
* `cargo doc --no-deps -p microvms-cli` emits the same eight warnings with or without the constructs, and neither `ControlPlane` nor `ErrorKind` appears among them. The stated reason — “Re-exported so `[ControlPlane]` is nameable in this module’s docs” ([`microvms-cli/src/commands/lifecycle.rs:1166`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L1166)) — is not the mechanism. The intra-doc link at [`microvms-cli/src/commands/lifecycle.rs:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L10) resolves from the [`:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L70) import directly.
The two differ in whether the whole construct earns its place:
* `ControlPlane` is a trait, so its name never appears in an expression; the [`:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L70) import is what makes `[ControlPlane]` at [`microvms-cli/src/commands/lifecycle.rs:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L10) resolve. Removing the pair means removing that doc link. Load-bearing as a unit.
* `ErrorKind` is named nowhere in `microvms-cli/src/commands/attached.rs` except its import at [`:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L40) and the two `_DocsOnly` lines at [`:930-932`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L930-L932). The only documentation that links `[ErrorKind]` is the doc comment justifying the import that makes it resolvable. Removable as a unit — [`:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L40)’s `ErrorKind`, plus all three lines at [`:930-932`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L930-L932).
**Non-Rust surfaces, both clean.** `uvx ruff check --select F401,F811,F841` over all 17 tracked `.py` files plus `microvms-py/microvms.pyi` reports no findings; F401 is in the repo’s own selected set (`ruff.toml`) and [`mise.toml:91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L91) runs `ruff check .` across the whole repo. The eight `microvms-js/__test__/*.mjs` files have no linter, so their `import … from '…'` bindings were checked directly for non-comment uses: zero unused.
## See also
[Section titled “See also”](#see-also)
* [impact analysis](../insights/impact-analysis.md) — 7 shared source citations
* [contract map](../insights/contract-map.md) — 6 shared source citations
* [processes](../behavior/processes.md) — 5 shared source citations
* [tech debt](../insights/tech-debt.md) — 5 shared source citations
* [risk hotspots](risk-hotspots.md) — 4 shared source citations
# Ownership
> Per-author ownership analysis does not apply to this repository, and this file says so with numbers before it measures anything else.
Per-author ownership analysis does not apply to this repository, and this file says so with numbers before it measures anything else.
The whole history is 114 commits spanning 2026-08-05 to 2026-08-17 (`git rev-list --count HEAD`; `git log --reverse --date=short --pretty=format:%ad` for the bounds). `git log --pretty=format:'%an <%ae>' | sort | uniq -c` returns three author identities and only three: 56 commits from `alsaadoonlaith@gmail.com`, 31 from `9553966+theagenticguy@users.noreply.github.com`, and 27 from `bgagent@noreply.github.com`. The first two are the same human — `%cn`/`%ce` shows the 31 are committed by `GitHub `, which marks them as web-flow and merge-queue commits — so the history is **one human author (87 commits) and one automated agent identity (27)**. No `CODEOWNERS` file exists at the repository root, under `.github/`, or under `docs/`, so there is no declarative owner to check the git data against either.
A folder-by-author commit-share table over that history would report the same two numbers on every row and would name a bus factor of 1 that is true by construction rather than discovered. This file measures the two things that do vary across subsystems:
* **Churn** — how many times each subsystem has been edited. One “file-touch” is one file changed in one non-merge commit, from `git log --pretty=format: --name-only`, counted on the first path segment. The total across all history is 896. Counting the same paths with `git log --no-merges --oneline -- ` reproduces every per-file figure below exactly, so the two definitions agree.
* **Symbol density** — how much declared structure each subsystem holds, from `codegraph files --format grouped` over the 121 indexed files.
Where the two measures agree, a reader is looking at a subsystem that is both large and frequently revised. That is the concentration this repository actually has.
## Knowledge concentration by subsystem
[Section titled “Knowledge concentration by subsystem”](#knowledge-concentration-by-subsystem)
`Share` is the subsystem’s percentage of the 896 total file-touches. `Symbols` is the sum of declared symbols across that subsystem’s indexed files; a dash means the tree holds no symbol-bearing indexed files.
| Folder | Churn (file-touches) | Share | Symbols |
| ------------------------------- | -------------------- | ----- | ------- |
| `microvms-core/` | 149 | 17% | 1,572 |
| `microvms-cli/` | 137 | 15% | 708 |
| `docs/` | 104 | 12% | — |
| `./` (root config and README) | 100 | 11% | — |
| `microvms-py/` | 72 | 8% | 591 |
| `agentd/` | 71 | 8% | 727 |
| `clients/` (deleted, see below) | 70 | 8% | — |
| `microvms-js/` | 50 | 6% | 374 |
| `.github/` | 31 | 3% | — |
| `scripts/` | 26 | 3% | 170 |
| `.erpaval/` | 23 | 3% | — |
| `conformance/` | 22 | 2% | 96 |
| `protocol/` | 17 | 2% | 88 |
| `model/` | 10 | 1% | 104 |
| `spec/` | 6 | 1% | — |
Three readings of that table matter.
**`clients/` is the largest body of knowledge in this repository that no working tree contains.** It accrued 70 file-touches, 8% of all churn, and then left: `git log --oneline --diff-filter=D -- clients` returns `c4d396e`, “Retire the Python client: the discovery instrument becomes git history”. 28 files lived under `clients/python/`, 12 of them under `tests/` and 10 of those test modules covering SSE reconnect, proxy auth, pricing, sizing, and cost. [`mise.toml:150-153`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L150-L153) records what that suite was worth: 83 client-library tests against a fake daemon over a real loopback socket, and both suites passing against real AWS on the same commit — Python oracle 56/56, Rust CLI 38/38 — is what ended the oracle’s job. Recovering any of it requires `git show`, and nothing in the tree points a reader at that commit.
**The densest files are also the most-churned files.** `microvms-core/src/cost.rs` holds 242 symbols, the highest in the repository, and 7 commits. `agentd/src/exec.rs` 147, `agentd/src/fs.rs` 137, `microvms-cli/src/guards.rs` 123 with 13 commits, `microvms-py/src/cost.rs` 117, `microvms-js/src/cost.rs` 106, `microvms-core/src/sandbox.rs` 105 with 11 commits, `microvms-core/src/control/image.rs` 99 with 11, `microvms-core/src/control/ops.rs` 96 with 9, `microvms-core/src/session/mod.rs` 94. Six of those ten are in both the churn top-20 and the density top-10. A change to any of them is a change to a file that is simultaneously the largest and the least settled thing in its crate.
**`protocol/` and `model/` are the inverse case.** 17 and 10 file-touches, 88 and 104 symbols. `protocol/` is small and quiet, and it is also the crate both the daemon and every client compile against — `microvms-cli/tests/dependency_direction.rs` makes the direction `cli -> core -> protocol` a test rather than a convention. Low churn there is a property of the wire contract being stable, not of the code being unimportant.
## Where the knowledge lives outside one head
[Section titled “Where the knowledge lives outside one head”](#where-the-knowledge-lives-outside-one-head)
Bus factor 1 is the starting condition here, so the question worth asking is how much of what one person knows has been written down somewhere a second person can read. The answer is more than expected, in four layers.
**Measured platform behavior.** `docs/PLATFORM.md` carries 43 H2 headings, 42 of them distinct — the heading “A WebSocket reaches a guest server through the endpoint, and the proxy strips its own subprotocols” appears at both [`docs/PLATFORM.md:784`](/microvms-agentd/internals/platform/) and [`docs/PLATFORM.md:853`](/microvms-agentd/internals/platform/), so the file holds one duplicated section and a reader counting findings should count 42. Each is a finding that cannot be derived from this repository’s source, because it describes the AWS service rather than this code. Measurement dates run 2026-06-17 through 2026-08-16, with 22 references to 2026-08-15 alone. Two sections show why the file cannot be cheaply regenerated: [`docs/PLATFORM.md:46`](/microvms-agentd/internals/platform/) establishes that `runHookPayload` arrives wrapped rather than as the request body, which cost a full build-and-run cycle because the platform terminates the VM on the resulting 400 before the payload can be read; [`docs/PLATFORM.md:64`](/microvms-agentd/internals/platform/) fixes the `runHookPayload` ceiling at 4096 bytes and notes the service model states it twice, differently.
**Formal requirements.** `spec/core.symspec.json` holds 51 requirements, every one `status: approved`, keyed `TRAP` 13 / `STATE` 12 / `COST` 10 / `CLI` 6 / `ARCH` 5 / `BIND` 5, with `verificationMethod` distributed `test` 38 / `analysis` 9 / `inspection` 4, plus a state model and one waiver. Its `systemName` field gives the per-subsystem coverage: `microvms-core` 27, cost engine 10, CLI crate 8, language-bindings layer 3, JavaScript binding 1, Python binding 1, sizing model 1. `spec/agentd.symspec.json` adds 6, all `systemName: agentd`. The 13 `TRAP-*` requirements are `docs/PLATFORM.md`’s findings in enforceable form.
**Compounded lessons.** 12 files under `.erpaval/solutions/`, in four categories: `api-patterns/` 3, `architecture-patterns/` 2, `best-practices/` 4, `test-failures/` 3. They carry the failures that cost the most to rediscover — that `aws-config` with `default-features = false` cannot resolve credentials at all, that a byte-offset cursor is what separates a working stream reconnect from a broken one, that a deterministic simulator has two clocks and a spawned child obeys the wrong one.
**Executable gates.** [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301) defines `check`, the stated definition of done, as exactly eight tasks: `lint`, `security`, `test`, `schema:check`, `stubs:check`, `model:check`, `live:check`, `build`. Four of those are drift gates that keep a hand-maintained value honest against an independent source: `schema:check` ([`mise.toml:173`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L173)) asserts `docs/schema.json` still describes what the daemon serves, `stubs:check` ([`mise.toml:195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L195)) asserts `microvms-py/microvms.pyi` still describes the pyo3 surface, `model:check` ([`mise.toml:257`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L257)) asserts `microvms-core`’s hardcoded constants still match the pinned botocore service model, and `live:check` ([`mise.toml:288`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L288)) asserts the live tier’s own wiring, including `mise.toml` itself. A gate is stronger than a document because it fails rather than being unread.
### What that coverage does not reach
[Section titled “What that coverage does not reach”](#what-that-coverage-does-not-reach)
Three subsystems or artifacts sit outside it, each verifiable from the repository.
**Neither symspec gate runs in `check`.** [`mise.toml:207`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L207) and [`mise.toml:227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L227) are the two spec-verification tasks, and neither appears in `check`’s dependency list at [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301). Their own comments give the reason: `symspec` is a global npm install plus a downloaded embedding model, and [`mise.toml:227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L227) invokes the v5 CLI as `node ~/workplace/symspec/packages/symspec/dist/cli.mjs` — an absolute path into one developer’s home directory. The strongest externalization in the repository, 57 approved requirements, is therefore verified by a toolchain a second contributor does not have, and no unconditional gate reports when the requirements and the code diverge.
**`microvms-js` has no typings drift gate.** `.gitignore:29` ignores `microvms-js/index.d.ts`, and neither `mise.toml` nor `.github/workflows/ci.yml` mentions `index.d.ts` anywhere. The Python binding’s equivalent artifact is gated by `stubs:check`; the Node binding’s is generated, ignored, and unchecked, so a divergence between the Rust surface and the TypeScript surface shipped to consumers surfaces at a consumer’s keyboard.
**`protocol` and `agentd-model` carry zero formal requirements.** No `systemName` in either symspec file names them ([`protocol/Cargo.toml:2`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/Cargo.toml#L2) declares `protocol`, [`model/Cargo.toml:2`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L2) declares `agentd-model`). `protocol/` is the crate the daemon and every client both compile against, and `docs/PROTOCOL.md` states the wire contract must never change silently. The requirement set that would make a silent change fail does not exist for it; the compile error from a type change is the whole defense.
## Read first
[Section titled “Read first”](#read-first)
A second contributor becomes productive by reading these in this order. The first three come before any code change, because skipping `docs/PLATFORM.md` means re-learning its traps at the price of live AWS runs.
1. `README.md` — why the project exists. [`README.md:203`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/README.md#L203) names the origin: a daemon inside Harbor PR #2469, and the review rounds that argued for a verified stack instead.
2. `docs/PLATFORM.md` — every section, each dated and scoped to a region and API version. Read it before touching launch, cost, or hook code.
3. `docs/PROTOCOL.md` — wire protocol v1, and which paths the platform fixes versus which this project owns. `docs/schema.json` is its generated companion and `schema:check` keeps the two in agreement.
4. `docs/TRUST.md` — the boundary contract: what the daemon must refuse, and why the workload is untrusted by design.
5. `spec/core.symspec.json` — the 51 requirements, the 13 `TRAP-*` entries first, since each is a `docs/PLATFORM.md` finding with a `verificationMethod` attached.
6. `.erpaval/solutions/` — 12 lessons, ordered by whichever subsystem is about to be touched. Consulting them before a fix costs minutes; rediscovering one costs a session.
7. `mise.toml` — the command surface. `mise run check` is the local gate; `mise run live` is billable and manual.
8. `microvms-core/src/` — the largest crate at 39,097 lines and 1,572 symbols. `constants.rs` and `cost.rs` are where the measured platform values land in code.
9. `docs/STRATEGY.md` — scope, audience, and the labeling discipline every claim follows: measured, documented, vendor-claimed, or inferred.
## Single points of failure
[Section titled “Single points of failure”](#single-points-of-failure)
The entire codebase is effectively one owner (see intro).
The per-path shares below are still worth stating, because with only two identities in the history the split is between the human author and the `bgagent` automated identity, and which one holds a file predicts whether any person reviewed it. Read each percentage against its commit count: on a file with three commits a share above 70% carries little information, so every bullet names the count. Shares are computed with `git log --no-merges --pretty=format:%ae -- `.
* `mise.toml` — sole human author (71% of 17 commits). Bring both symspec gates inside `check` behind a pinned, repository-local toolchain so the requirement set is verified by the command a fresh clone can run, rather than by a path into one home directory.
* `microvms-cli/src/guards.rs` — sole human author (85% of 13 commits). At 123 symbols and the highest churn in the CLI crate, this file needs a second reader more than any other; pair a review of it with `.erpaval/solutions/test-failures/guards-that-passed-against-broken-code.md`, which records four ways its guards passed against broken code.
* `microvms-cli/src/cli.rs` — sole human author (85% of 13 commits). The command surface definition is the CLI’s contract with every consumer, so changes here belong behind the `microvms-cli/tests/manifest.rs` and `thinness.rs` assertions rather than behind review alone.
* `microvms-core/src/control/image.rs` — sole human author (82% of 11 commits). 99 symbols covering image and version creation, an area where [`docs/PLATFORM.md:725`](/microvms-agentd/internals/platform/) and [`docs/PLATFORM.md:747`](/microvms-agentd/internals/platform/) record two service refusals; keep those two sections and this file under one change.
* `microvms-core/src/control/ops.rs` — sole human author (78% of 9 commits). Cross-train a second reader here before the control-plane call surface grows again, since 96 symbols in one file is where a control-plane behavior change hides.
* `microvms-cli/src/commands/lifecycle.rs` — sole human author (75% of 12 commits). The lifecycle commands are the operator’s path to spending money, so every change here should cite the `docs/PLATFORM.md` section whose behavior it depends on.
* `microvms-core/src/sandbox.rs` — sole human author (73% of 11 commits). 105 symbols implementing the single-writer state machine that the `model/` crate’s stateright model also encodes; change the model in the same commit so the two descriptions cannot drift.
* `microvms-core/src/cost.rs` — `bgagent` automated identity (71% of 7 commits). The densest file in the repository at 242 symbols, holding a hand-pinned us-east-1 rate table that has already drifted once — [`microvms-core/src/cost.rs:1018-1021`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1018-L1021) records that `0.08` was the plausible-looking wrong value against the correct `dec!(0.0811111030)` — so keep `scripts/check-live-rates.py` in the billable tier and treat a rate edit as a measurement.
* `agentd/src/routes.rs` — `bgagent` automated identity (86% of 7 commits). This file splits the 20 daemon endpoints into the Bearer-guarded `control` router and the `open` router at [`agentd/src/routes.rs:48-56`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L48-L56), which makes it the repository’s authorization boundary; it should carry a named human reviewer on every change, since no commit on it currently does.
* `docs/reference/cli.md` — sole human author (100% of 9 commits). A hand-written reference for a surface that `microvm manifest` already emits machine-readably, so generate the overlapping sections or add a drift check, rather than maintaining two descriptions of one command set.
* `spec/core.symspec.json` — `bgagent` automated identity (100% of 2 commits). 51 requirements in a single file with no second copy and no gate inside `check`; approve changes to it the way a schema change is approved, and pair every edit with the `docs/PLATFORM.md` section it encodes.
* `scripts/check-model-drift.py` — `bgagent` automated identity (100% of 1 commit). [`scripts/check-model-drift.py:254`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L254) and [`:266`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L266) hold `PINNED_REGIONS` and `PINNED_SIZE_CLASSES` as deliberate hand-maintained copies, and [`:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L57) states they are the second reader for two values no AWS service model publishes; any change to the Rust constants has to land in this file in the same commit.
## See also
[Section titled “See also”](#see-also)
* [system overview](../architecture/system-overview.md) — 6 shared source citations
* [contract map](../insights/contract-map.md) — 4 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 4 shared source citations
* [module map](../architecture/module-map.md) — 3 shared source citations
* [state machines](../behavior/state-machines.md) — 3 shared source citations
# Risk hotspots
> Risk here is composed from two measured signals, because the one the default recipe reaches for is empty. Both gate-level static-analysis passes return zero findings on every…
Risk here is composed from two measured signals, because the one the default recipe reaches for is empty. Both gate-level static-analysis passes return zero findings on every file: `cargo clippy --all-targets --message-format=json` emits 0 diagnostics at `warning` or `error` level, and the semgrep pass the gate runs over all seven crates ([`mise.toml:112`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L112)) reports 0 results. So severity is measured instead as **test-tier reach**, using CodeGraph’s covering-test relation over the 4,430-node index. A symbol counts once, and only if it carries at least one inbound `calls`/`references`/`instantiates` dependent and sits outside any `#[cfg(test)]` region. **`error`** means no test file reaches it and its containing file holds zero in-file `#[test]` functions — no test tier in this repository reaches that symbol through a source-level edge. **`warn`** means no test file reaches it but the file does carry an in-file `#[cfg(test)]` module, so a same-file unit test may while no cross-file tier does. Trend is cross-sectional over the 30-day window: 78 live source files were touched across 114 commits, per-file commit counts have median 4.0 and population σ 2.7928, so `↑ rising` is 7 or more commits (19 files), `→ flat` is 2 to 6 (58 files), and `↓ falling` is 1 (1 file). The score is `2 × error + 0.5 × warn + 1 × (trend is rising)`, ties broken by commit count descending. Totals across the 75 scored files: 278 error-class and 654 warn-class findings.
Four limitations bound every number below. First, the repository’s whole history — 114 commits, 2026-08-05 to 2026-08-17 — fits inside the 30-day window, so there is no earlier baseline and the trend arrow compares a file against its peers, never against its own past. Second, the covering-test relation counts test *files*, and Rust’s dominant unit-test idiom here is an in-file `#[cfg(test)] mod tests`; 770 such tests exist across `microvms-core` (436), `microvms-cli` (172), `agentd` (130), `protocol` (17) and `model` (15), and none of them can satisfy a cross-file rule. A `warn` count is therefore a statement about tier reach, not about absence of testing — `microvms-core/src/sandbox.rs` returns an empty `codegraph affected -d 1` result while holding 27 unit tests of its own. Third, and most consequential for reading the table: **no source-level edge can cross the PyO3 or napi-rs FFI boundary**, so the 159 pytest functions in `microvms-py/tests/` and the 168 `test(` calls in `microvms-js/__test__/` cannot register as covering tests for the Rust code they exercise, and CodeGraph does not classify a `.mjs` file as a test at all. Every `error` count in a bindings file measures Rust-tier reach only; the residual risk is narrower than “untested” and is named per file in the drill-down. The index also resolves references by name, so a declaration whose name is short or collides with a widely used external type absorbs mentions that are not references to it. `Duration` ([`microvms-js/src/cost.rs:60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L60)) is the clearest instance: it is the only node of that name in the index, so every `Duration` in the workspace resolves onto it, and it is credited with coverage from `microvms-core/tests/turmoil_client.rs`, a file containing zero references to `microvms-js`. `live` ([`microvms-js/src/session.rs:274`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L274)) and `data` ([`microvms-py/src/exec.rs:236`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L236)) are likewise sole holders of their names. So per-symbol dependent counts below are what the index records rather than verified call sites, and the direction of the bias on `error` counts is toward understating them. Fourth, ownership carries no bus-factor information: one human authors 87 of 114 commits under two identities and a `bgagent` bot the other 27, every file has one or two distinct authors, and a `bgagent` top-owner share marks a file that arrived inside a large squashed commit rather than one with a second maintainer. Two files are excluded from scoring because they are test code that lives under `src/`: `microvms-cli/src/guards.rs` (`#![cfg(test)]` at [`microvms-cli/src/guards.rs:20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L20), declared at [`microvms-cli/src/main.rs:36-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L36-L37)) and `microvms-core/src/control/fake.rs` (declared at [`microvms-core/src/control/mod.rs:928-929`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L928-L929)). That exclusion matters: `guards.rs` is the joint-highest-churn source file in the repository at 13 commits, so a churn-only ranking puts a file that never ships in a binary at the top.
| File | Trend | Open findings | Top owner | Citation |
| ------------------------------ | --------- | ---------------- | --------------------- | ------------------------------------------ |
| `microvms-py/src/cost.rs` | ↑ rising | 0 warn, 66 error | Laith Al-Saadoon 57% | `microvms-py/src/cost.rs` (1,123 LOC) |
| `microvms-js/src/cost.rs` | → flat | 0 warn, 65 error | Laith Al-Saadoon 60% | `microvms-js/src/cost.rs` (1,038 LOC) |
| `microvms-js/src/session.rs` | → flat | 0 warn, 25 error | Laith Al-Saadoon 60% | `microvms-js/src/session.rs` (609 LOC) |
| `microvms-core/src/cost.rs` | ↑ rising | 93 warn, 0 error | bgagent 71% | `microvms-core/src/cost.rs` (4,127 LOC) |
| `microvms-py/src/exec.rs` | ↑ rising | 0 warn, 22 error | Laith Al-Saadoon 71% | `microvms-py/src/exec.rs` (631 LOC) |
| `microvms-js/src/exec.rs` | → flat | 0 warn, 19 error | Laith Al-Saadoon 60% | `microvms-js/src/exec.rs` (456 LOC) |
| `microvms-js/src/sandbox.rs` | → flat | 0 warn, 16 error | Laith Al-Saadoon 60% | `microvms-js/src/sandbox.rs` (623 LOC) |
| `microvms-js/src/process.rs` | ↓ falling | 0 warn, 12 error | Laith Al-Saadoon 100% | `microvms-js/src/process.rs` (541 LOC) |
| `microvms-py/src/session.rs` | → flat | 0 warn, 11 error | Laith Al-Saadoon 67% | `microvms-py/src/session.rs` (605 LOC) |
| `microvms-core/src/sandbox.rs` | ↑ rising | 40 warn, 0 error | Laith Al-Saadoon 73% | `microvms-core/src/sandbox.rs` (2,371 LOC) |
| `agentd/src/fs.rs` | ↑ rising | 38 warn, 0 error | bgagent 57% | `agentd/src/fs.rs` (2,628 LOC) |
| `microvms-py/src/sandbox.rs` | → flat | 0 warn, 10 error | Laith Al-Saadoon 67% | `microvms-py/src/sandbox.rs` (780 LOC) |
The shape of that list is the finding. Nine of the twelve rows are binding files, and the reason is structural rather than per-file: all 18 files under `microvms-py/src/` and `microvms-js/src/` — 3,856 and 3,773 LOC respectively — contain zero `#[cfg(test)]` modules and zero `#[test]` functions, while the other five crates hold 770 between them — an average of 19 per file in `microvms-core`, 11 in `microvms-cli`, 10 in `agentd`. The three non-binding rows (`microvms-core/src/cost.rs`, `microvms-core/src/sandbox.rs`, `agentd/src/fs.rs`) are the opposite case: heavily unit-tested files whose churn is rising and whose public surface no cross-file tier reaches.
## Per-file drill-down
[Section titled “Per-file drill-down”](#per-file-drill-down)
### 1. `microvms-py/src/cost.rs`
[Section titled “1. microvms-py/src/cost.rs”](#1-microvms-pysrccostrs)
**What’s there.** The PyO3 mirror of the cost engine, whose stated job is wrapping the core types “without loosening any of them (BIND-5)” by making the unsafe spellings absent rather than rejected — no `__float__`, `__int__`, `__index__`, `__add__` or any numeric dunder on `EstimatedUsd`, no `#[pyo3(transparent)]`, and no defaulting `Duration` constructor ([`microvms-py/src/cost.rs:2-24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L2-L24)). Concretely it is 111 symbols of `#[pyclass(frozen, from_py_object)]` wrappers: `PyDuration` ([`microvms-py/src/cost.rs:89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L89)), `PySizeClass` ([`microvms-py/src/cost.rs:488`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L488)), `PyRateTable` ([`microvms-py/src/cost.rs:576`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L576)), `PyLineItem` ([`microvms-py/src/cost.rs:313`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L313)) and `PyCostReport` ([`microvms-py/src/cost.rs:696`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L696)).
**Recent activity.** 7 commits in the 30-day window, `↑ rising` — at the rising threshold of 7 and joint-highest among all binding files.
**Owners.** Laith Al-Saadoon 57% (4 of 7 commits); `bgagent` 43% (3 of 7). Two identities, one of them a bot, so no second human reviewer is implied.
**Findings.** 66 error, 0 warn. All 66 of the file’s 66 symbols that carry inbound dependents are error-class, the highest count in the repository, and `codegraph affected microvms-py/src/cost.rs -d 1 --json` returns an empty `affectedTests` array against 2 dependents traversed in total. The most-depended-on uncovered symbols are `PySizeClass` (10 dependents, [`microvms-py/src/cost.rs:488`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L488)), the `seconds` getter (6 dependents, [`microvms-py/src/cost.rs:126`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L126)), `PyDuration` (5, [`microvms-py/src/cost.rs:89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L89)) and `PyRateTable` (5, [`microvms-py/src/cost.rs:576`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L576)). The mitigation is real and it is dynamic: `microvms-py/tests/test_cost.py` holds 39 pytest functions over 753 lines, and the generated stubs are gated by `stubs:check` ([`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195), listed in `check` at [`mise.toml:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L297)). What no tier covers is a Rust-level refactor of the absences the module docs enumerate — a `__float__` accidentally reintroduced on `EstimatedUsd` is caught only if a Python test happens to assert its absence, after a full native rebuild.
### 2. `microvms-js/src/cost.rs`
[Section titled “2. microvms-js/src/cost.rs”](#2-microvms-jssrccostrs)
**What’s there.** The napi-rs mirror of the same engine, where the entire BIND-5 requirement reduces to one decision: every type is a `#[napi]` **class** and never `#[napi(object)]`, because an object converts by structure and `{ amount: 1.5 }` would satisfy an `EstimatedUsd` parameter ([`microvms-js/src/cost.rs:2-12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L2-L12)). `valueOf`, `toJSON`, `Symbol.toPrimitive` and any `add` method are absent by design, so `Number(usd)` is `NaN` and the figure leaves only through `.amount` as a string ([`microvms-js/src/cost.rs:16-25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L16-L25)).
**Recent activity.** 5 commits, `→ flat`. It ranks second on score with no rising bonus, entirely on finding count.
**Owners.** Laith Al-Saadoon 60% (3 of 5); `bgagent` 40% (2 of 5).
**Findings.** 65 error, 0 warn, out of 66 symbols with inbound dependents — and the 66th is a measurement artifact, not a covered symbol. The one symbol credited with coverage is `Duration` ([`microvms-js/src/cost.rs:60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L60)), attributed to `microvms-core/tests/turmoil_client.rs`, which contains zero references to `microvms-js` in either spelling. `Duration` is the only node of that name in the entire index, so every `Duration` mention in the workspace — including every `std::time::Duration` — resolves onto this one napi class. The real figure for this file is 66 of 66. Highest-dependent uncovered symbols are `SizeClass` (27 dependents, [`microvms-js/src/cost.rs:422`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L422)), `Amount` (5, [`microvms-js/src/cost.rs:187`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L187)), the `CostReport::wrap` constructor (4, [`microvms-js/src/cost.rs:619`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L619)), `Total` (3, [`microvms-js/src/cost.rs:383`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L383)) and `all` (3, [`microvms-js/src/cost.rs:450`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L450)). `microvms-js/__test__/cost.mjs` exercises this surface with 42 `test(` calls over 733 lines, but CodeGraph does not classify `.mjs` as a test file, so those never appear in `affected` output for any path. This crate’s `index.d.ts` is gitignored (`.gitignore:29`) and no drift gate for it appears anywhere in `mise.toml`, unlike its Python twin whose stubs are checked at [`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195) — the one asymmetry on this list that is a gap in the gate rather than a limit of the measurement.
### 3. `microvms-js/src/session.rs`
[Section titled “3. microvms-js/src/session.rs”](#3-microvms-jssrcsessionrs)
**What’s there.** The control API of one running MicroVM on the Node side, where the binding inherits its exclusion guarantees from the core: `Sandbox` owns its `Session` by value, hands out only `Option<&Session>`, `Session` is not `Clone`, and no accessor exposes the agent token, so a second independent session against the same VM cannot be constructed ([`microvms-js/src/session.rs:4-12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L4-L12)). The mutex is tokio’s rather than `std`’s because every method is `async` and holds the guard across an `await`, reproducing the `&mut self` exclusion that `suspend`/`resume`/`terminate` require ([`microvms-js/src/session.rs:14-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L14-L18)).
**Recent activity.** 5 commits, `→ flat`.
**Owners.** Laith Al-Saadoon 60% (3 of 5); `bgagent` 40% (2 of 5).
**Findings.** 25 error, 0 warn, from 27 symbols with inbound dependents out of 33 total. The two the index ranks highest by dependent count are the lock-acquiring internals rather than the public methods — `Live::session` at 18 ([`microvms-js/src/session.rs:244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L244)) and the `live()` guard-taker at 17 ([`microvms-js/src/session.rs:274`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L274)) — though both names are short enough that those counts are inflated by the name-resolution caveat above; `into_request` (3, [`microvms-js/src/session.rs:142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L142)), `in_sandbox` (3, [`microvms-js/src/session.rs:264`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L264)) and `port` (3, [`microvms-js/src/session.rs:306`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L306)) follow. Those two functions are where the “held for exactly one method call and no more” invariant in the doc comment at [`microvms-js/src/session.rs:272-273`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L272-L273) actually lives, and a lock-scope regression in them is the class of defect a dynamic suite detects only as a hang. `codegraph affected microvms-js/src/session.rs -d 1` does return two tests — `agentd/tests/turmoil_transport.rs` and `microvms-core/tests/turmoil_client.rs` — reached through the core types this file wraps, not through the binding surface itself.
### 4. `microvms-core/src/cost.rs`
[Section titled “4. microvms-core/src/cost.rs”](#4-microvms-coresrccostrs)
**What’s there.** The cost engine proper, and the largest file in the workspace at 4,127 lines: rate data carried as types rather than prose because MicroVMs publishes no standalone pricing page, with two invariants enforced by shape instead of at runtime — “seconds are measured, dollars are estimated” via a `DurationP` enum whose every variant names its provenance (COST-1), and “unknown is not zero” via `Amount::Unpriced` as a distinct variant that forces a match arm (COST-3), promoting to `Total::AtLeast` so a floor cannot be read without its reasons (COST-4) ([`microvms-core/src/cost.rs:2-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2-L27)). Central types are `RateTable` with its `region`/`source_url`/`retrieved` provenance fields ([`microvms-core/src/cost.rs:849`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L849)), `CalendarDate` ([`microvms-core/src/cost.rs:209`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L209)), `LineItem` ([`microvms-core/src/cost.rs:1440`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1440)), `CostReport` ([`microvms-core/src/cost.rs:1478`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1478)) and `RunUsage` ([`microvms-core/src/cost.rs:1730`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1730)).
**Recent activity.** 7 commits, `↑ rising`.
**Owners.** `bgagent` 71% (5 of 7); Laith Al-Saadoon 29% (2 of 7). The only top-5 file whose top owner is the bot, which means most of its 4,127 lines landed inside large squashed commits.
**Findings.** 93 warn, 0 error — the largest warn count in the repository, and every one of them is a tier-reach statement rather than an absence of tests: the file carries 65 in-file `#[test]` functions, so no symbol qualifies for the error class. Highest-dependent uncovered symbols are `RateTable` (15 dependents, [`microvms-core/src/cost.rs:849`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L849)), `CalendarDate` (14, [`microvms-core/src/cost.rs:209`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L209)), `LineItem` (13, [`microvms-core/src/cost.rs:1440`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1440)), `today_utc` (12, [`microvms-core/src/cost.rs:251`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L251)), `CostReport` (11, [`microvms-core/src/cost.rs:1478`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1478)) and `RunUsage` (11, [`microvms-core/src/cost.rs:1730`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1730)). `codegraph affected microvms-core/src/cost.rs -d 1` reaches 8 test files, the broadest of any file on this list. Two specifics deserve attention: `today_utc` ([`microvms-core/src/cost.rs:251`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L251)) reads the wall clock and floors on integer division, so it is the file’s one ambient-time dependency and cannot be exercised deterministically by a `turmoil` tier that controls virtual time only; and `RateTable::retrieved` ([`microvms-core/src/cost.rs:849`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L849)) makes rate freshness a data property, which the separate `scripts/check-live-rates.py --twin-only` cross-check exists to verify ([`mise.toml:412-414`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L412-L414)) rather than any Rust test tier.
### 5. `microvms-py/src/exec.rs`
[Section titled “5. microvms-py/src/exec.rs”](#5-microvms-pysrcexecrs)
**What’s there.** One exec plus the SSE stream exposed as a Python iterator, which the module docs single out as the only shape on the surface that needed real work: `ExecStream::new` spawns a driver onto the shared runtime with an owned `ExecHandle` and a bounded `mpsc` sender, and `__next__` blocks on `recv` ([`microvms-py/src/exec.rs:4-15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L4-L15)). The channel bound is deliberately 1, because the daemon’s SSE body is the backpressure signal and an unbounded channel would buffer a fast producer’s whole output inside the binding — the failure the core’s byte-offset cursor exists to make unnecessary ([`microvms-py/src/exec.rs:11-15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L11-L15)); dropping the iterator drops the receiver, the next `send` fails, and the drive ends on `ControlFlow::Break` ([`microvms-py/src/exec.rs:18-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L18-L20)).
**Recent activity.** 7 commits, `↑ rising` — tied with `microvms-py/src/cost.rs` for the hottest binding file.
**Owners.** Laith Al-Saadoon 71% (5 of 7); `bgagent` 29% (2 of 7).
**Findings.** 22 error, 0 warn, covering all 22 of the file’s symbols that have inbound dependents (58 symbols total, so most of the surface has no recorded dependent at all). `codegraph affected microvms-py/src/exec.rs -d 1 --json` returns an empty `affectedTests` array. Ranked by dependents: `PyExecResult` (5, [`microvms-py/src/exec.rs:64`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L64)), a `wrap` constructor (5, [`microvms-py/src/exec.rs:78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L78)), the `seconds` getter (3, [`microvms-py/src/exec.rs:629`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L629)), `PyStdinAck` (2, [`microvms-py/src/exec.rs:173`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L173)), `data` (2, [`microvms-py/src/exec.rs:236`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L236)) and `PyExecHandle` (2, [`microvms-py/src/exec.rs:485`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L485)). The drop-and-break lifecycle described at [`microvms-py/src/exec.rs:18-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L18-L20) is the specific untested-at-Rust-tier surface worth attention, because a leaked driver task shows up as a hang or a stray thread rather than a failing assertion. `microvms-py/tests/test_exec.py` covers the behaviour dynamically with 29 pytest functions over 655 lines. `PyExecResult` keeping `exit_code` and `signal` as `Option` rather than sentinel integers ([`microvms-py/src/exec.rs:64-68`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L64-L68)) is the kind of distinction that a Python-only test can assert but no Rust tier here defends.
## Reproduction
[Section titled “Reproduction”](#reproduction)
Every number above traces to one of these, run from the repository root at commit `5e6f752a4c2a75f88522a38ce40d0a444f23ebc4`:
* Trend and ownership — `git log --since=30.days.ago --name-only --pretty=format:'---%H|%an|%ae'`, keeping `.rs`/`.py` paths outside `tests/`, `conformance/`, `scripts/`, `examples/`, `benches/` that still exist on disk (78 files; 15 touched paths have since been deleted, including the retired `clients/python/` package).
* Static-analysis baseline — `cargo clippy --all-targets --message-format=json` (0 diagnostics) and the semgrep invocation at [`mise.toml:112`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L112) (0 results).
* Covering-test relation — reverse traversal of `calls`/`references`/`instantiates` edges in `.codegraph/codegraph.db` at depth 1. The rule was validated against 126 markers harvested from 33 `codegraph explore` calls: depth 1 agrees on 124 of 126, and the 2 disagreements are test-glob edge cases (`microvms-js/__test__/*.mjs` is not treated as a test; `microvms-py/examples/typed_usage.py` is). Aligning on those reproduces the tool at 126/126.
* In-file unit-test census — count of `#[test]` and `tokio::test` attributes per `*/src/**/*.rs`.
* Per-file coverage cross-check — `codegraph affected -d 1 --json`. Depth matters: at the default `-d 5` every path returns the same saturated 19-file test set, so the default invocation is not a discriminator.
## See also
[Section titled “See also”](#see-also)
* [business logic](../insights/business-logic.md) — 6 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 6 shared source citations
* [contract map](../insights/contract-map.md) — 5 shared source citations
* [public api](../../reference/public-api.md) — 5 shared source citations
* [dead code](dead-code.md) — 4 shared source citations
# Data flow
> Two surfaces trigger work in this system and nothing else does: a CLI invocation, dispatched through a 17-arm exhaustive match (microvms-cli/src/main.rs:382-403), and a daemon…
Two surfaces trigger work in this system and nothing else does: a CLI invocation, dispatched through a 17-arm exhaustive match ([`microvms-cli/src/main.rs:382-403`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L382-L403)), and a daemon HTTP request, dispatched through a handler table walked from the same list `/v1/schema` publishes ([`agentd/src/routes.rs:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110)). The bindings re-enter the same `microvms-core` surfaces the CLI uses, so they add no distinct flow, and the daemon’s only recurring job is a 30-second expired-exec reaper rather than a request lifecycle ([`agentd/src/main.rs:61`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L61), [`agentd/src/exec.rs:951`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L951)).
The three flows below are ranked by how much of the client-to-daemon boundary each exercises, tie-broken by whether it is named after one of the system’s core verbs. Flow 1 is the only arm that launches a VM and the only one that touches all four actors. Flow 2 is the streaming read path, whose correctness rests on a byte-offset cursor that survives a reconnect through the endpoint proxy. Flow 3 is the file-transfer path, and it ends in the daemon’s one confined write.
Participants are the workspace crates named in `architecture/module-map.md` plus two external actors. `microvm CLI` is `microvms-cli`; `agentd` is the in-VM daemon; `AWS MicroVMs` is the control plane together with its endpoint proxy.
## Flow 1: microvm run — build, launch, bootstrap, exec, tear down
[Section titled “Flow 1: microvm run — build, launch, bootstrap, exec, tear down”](#flow-1-microvm-run--build-launch-bootstrap-exec-tear-down)
1. `commands::lifecycle::run` resolves region, size class, and image name, then requires every infra role before anything is created, so a missing role surfaces immediately rather than after a build ([`microvms-cli/src/commands/lifecycle.rs:121`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L121), guard at [`microvms-cli/src/commands/lifecycle.rs:141-166`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L141-L166)).
2. It opens a `Sandbox` through the library seam and races `launch_and_exec` against ctrl-c in a `tokio::select!`, with the sandbox owned outside the select so a cancelled launch still holds the identifiers teardown needs ([`microvms-cli/src/commands/lifecycle.rs:170-198`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L170-L198), recovery at [`microvms-cli/src/commands/lifecycle.rs:213-220`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L213-L220)).
3. `launch_and_exec` preflights the build request, uploads the artifact, then `Sandbox::build_image` issues `CreateMicrovmImage` and waits for the image to become usable ([`microvms-cli/src/commands/lifecycle.rs:302-307`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L302-L307), [`microvms-core/src/sandbox.rs:551`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551)).
4. `Sandbox::run` refuses a second bootstrap on the same sandbox, mints the agent token, and wraps it with the launch env in a typed `RunHookPayload` that checks its 4096-byte budget before any call ([`microvms-core/src/sandbox.rs:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648), refusal at [`microvms-core/src/sandbox.rs:652`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L652), payload at [`microvms-core/src/sandbox.rs:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L682)).
5. `ControlPlane::run_microvm` validates the identifier, the duration range, and the role ARN, splits ingress and egress connectors by intent, and puts the payload on the wire ([`microvms-core/src/control/microvm.rs:356`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L356)).
6. The platform calls the daemon’s run hook over loopback; `run_hook` unwraps the envelope, parses the inner payload, and installs the token once — an identical replay is 200 and a different token is 409 ([`agentd/src/routes.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178), verdicts at [`agentd/src/routes.rs:213-234`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L213-L234)).
7. `ControlPlane::wait_for_running` polls to RUNNING and fails fast on any terminal state; the client then polls unauthenticated `/v1/health` until `bootstrapped` ([`microvms-core/src/control/microvm.rs:435`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L435), [`microvms-core/src/session/mod.rs:342`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L342)). The sandbox marks the token installed only after RUNNING is observed ([`microvms-core/src/sandbox.rs:722-724`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L722-L724)).
8. The optional workload runs through `Session::run_sync` — start, wait, ack — and `tear_down` plus `attach_cost` then run however the select ended ([`microvms-core/src/session/mod.rs:408`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L408), [`microvms-cli/src/commands/lifecycle.rs:393`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L393), [`microvms-cli/src/commands/lifecycle.rs:443`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L443)).
## Flow 2: microvm exec –stream — SSE output on a byte-offset cursor
[Section titled “Flow 2: microvm exec –stream — SSE output on a byte-offset cursor”](#flow-2-microvm-exec-stream--sse-output-on-a-byte-offset-cursor)
1. `commands::attached::exec` attaches a session from the identifier triple, builds the start request under a caller-supplied or minted `exec_id`, starts the command, then branches to `stream_exec` ([`microvms-cli/src/commands/attached.rs:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L103), branch at [`microvms-cli/src/commands/attached.rs:163-165`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L163-L165)).
2. `stream_exec` drives `ExecHandle::for_each_event` with a `FnMut(ExecEvent) -> ControlFlow<()>` callback, writes one NDJSON line plus the raw bytes per event, and reports `nextOffset` from core’s cursor rather than its own tally ([`microvms-cli/src/commands/attached.rs:240`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L240), cursor read at [`microvms-cli/src/commands/attached.rs:281`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L281)).
3. `for_each_event` delegates to `for_each_event_async`, whose loop steps the `advance` state machine, reads the cursor off the machine, and reports `EndReason::Cut` when a body ends with no `exit` event ([`microvms-core/src/session/exec.rs:347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L347), loop at [`microvms-core/src/session/exec.rs:419-428`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L419-L428)).
4. `advance` re-attaches at the last good cursor with a fixed backoff on a retryable failure, and errors out past `max_reconnects` instead of looping forever ([`microvms-core/src/session/exec.rs:460`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L460), backoff and re-attach at [`microvms-core/src/session/exec.rs:487-491`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L487-L491)).
5. `ExecHandle::attach` issues `GET /v1/exec/{id}/stream?offset=N` with `accept: text/event-stream`, building its headers inside the request path so a mid-stream reconnect re-mints an expired token ([`microvms-core/src/session/exec.rs:591`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L591), mint at [`microvms-core/src/session/exec.rs:600`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L600)).
6. `ProxyAuth::headers` serves the cached proxy token, or takes the mint lock and re-checks freshness under it so two racing tasks do not burn two control-plane calls ([`microvms-core/src/session/proxy.rs:432`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L432), double check at [`microvms-core/src/session/proxy.rs:521-530`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L521-L530)).
7. The daemon’s `stream` handler snapshots the replay ring, reads the terminal marker after the snapshot, and sends the SSE body with a keepalive plus `x-accel-buffering: no` so a buffering proxy cannot batch a live stream into one delivery at exit ([`agentd/src/exec.rs:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L455), ordering at [`agentd/src/exec.rs:474-479`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L474-L479), header at [`agentd/src/exec.rs:489-491`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L489-L491)).
8. `build_stream` emits any `gap` first, drains the replayed backlog, then the live broadcast channel, and closes the body one step after the terminal `exit` event ([`agentd/src/exec.rs:560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L560), ending at [`agentd/src/exec.rs:604-615`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L604-L615)).
## Flow 3: microvm cp –tar — an archive into the one confined write path
[Section titled “Flow 3: microvm cp –tar — an archive into the one confined write path”](#flow-3-microvm-cp-tar--an-archive-into-the-one-confined-write-path)
1. `commands::attached::cp` resolves the direction from the `vm:` prefix before opening anything, so two local paths or two remote paths are refused by name rather than guessed at ([`microvms-cli/src/commands/attached.rs:805`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L805), resolver at [`microvms-cli/src/commands/attached.rs:902`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L902)).
2. It attaches through the helper every command in that file starts with, which resolves the region first because the region is what the proxy-token mint’s ARN is derived for ([`microvms-cli/src/commands/attached.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L73)).
3. The upload arm reads the local archive whole and sends it without inspecting it: the daemon’s extractor is the only one in the system, and a client-side check would be a second set of member rules that could disagree with it ([`microvms-cli/src/commands/attached.rs:812-835`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L812-L835), stated at [`microvms-cli/src/commands/attached.rs:798-804`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L798-L804)).
4. `Session::upload_tar` delegates to `files::upload_tar`, which builds `PUT /v1/fs/tar?path=...` with `content-type: application/x-tar` and the archive bytes as the body ([`microvms-core/src/session/mod.rs:444`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L444), [`microvms-core/src/session/files.rs:98`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L98)).
5. `Transport::request` prepends the proxy headers and the session’s bearer token to the caller’s own headers rather than replacing them, which is what keeps the content type on the request ([`microvms-core/src/session/mod.rs:106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L106), header assembly at [`microvms-core/src/session/mod.rs:88`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L88)).
6. `auth::require_token` guards the control router before the body is polled, answering 503 when no token is installed and 401 on a mismatch, then draining a bounded prefix of the rejected body so the client sees the status rather than a TCP reset ([`agentd/src/auth.rs:62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L62), verdicts at [`agentd/src/auth.rs:69-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L69-L80), applied at [`agentd/src/routes.rs:66-69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L66-L69)).
7. `fs::write_tar` refuses a relative extraction root, preflights free disk against that root before the body is spooled, then spools the body under the disk pacer ([`agentd/src/fs.rs:1433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1433), preflight at [`agentd/src/fs.rs:1459-1461`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1459-L1461), spool at [`agentd/src/fs.rs:872`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L872)).
8. `extract_into` runs under `spawn_blocking` and holds one confined directory handle for the whole extraction: ownership and xattrs are dropped, device and fifo members are refused, out-of-tree link targets are refused, and directory modes are replayed after all content lands. Success is 204 ([`agentd/src/fs.rs:621`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L621), refusals at [`agentd/src/fs.rs:702-707`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L702-L707) and [`agentd/src/fs.rs:742`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L742), deferred modes at [`agentd/src/fs.rs:810`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L810), dispatch and status at [`agentd/src/fs.rs:1479-1487`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1479-L1487)).
## See also
[Section titled “See also”](#see-also)
* [processes](../behavior/processes.md) — 13 shared source citations
* [sequences](../diagrams/behavioral/sequences.md) — 11 shared source citations
* [debugging guide](../insights/debugging-guide.md) — 9 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 9 shared source citations
* [business logic](../insights/business-logic.md) — 8 shared source citations
# Module map
> The workspace declares seven members (Cargo.toml:2-9), and the sections below run in the dependency order of the system-overview.md flowchart, bottom-up: the wire contract first…
The workspace declares seven members ([`Cargo.toml:2-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L2-L9)), and the sections below run in the dependency order of the `system-overview.md` flowchart, bottom-up: the wire contract first, then the two crates that compile against it, then the three surfaces over the client, then the checked model ([`docs/architecture/system-overview.md:79-97`](/microvms-agentd/internals/architecture/system-overview/)). Each crate’s file list is its `src/` tree ranked by size and by how many other files reference it; a crate’s `tests/` tier is excluded so the source files a reader is looking for are not crowded out, since `agentd/tests/turmoil_transport.rs` alone is 2,163 lines. Files belonging to no crate are collected under `Supporting code` at the end.
## protocol
[Section titled “protocol”](#protocol)
`protocol` is the wire contract expressed as Rust types, split into `exec`, `fs`, `health`, and `hook` ([`protocol/src/lib.rs:29-32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L29-L32)). The daemon and every Rust client of it compile against the same definitions, so a renamed field breaks compilation on whichever side has not caught up rather than surfacing as a consumer’s runtime bug ([`protocol/src/lib.rs:10-12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L10-L12)). Membership is decided by one rule — pure data that travels on the wire is admitted and machinery for making it travel is not, which is why the SSE event payloads live here and the stream that emits them does not ([`protocol/src/lib.rs:16-21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L16-L21)). Every type derives both halves of serde even where one side needs only one, because the missing half is what a client would otherwise hand-write, and `docs/schema.json` is generated from those same attributes under both contracts and byte-compared in CI ([`protocol/src/lib.rs:23-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L23-L27)).
* `protocol/src/exec.rs` (496 LOC)
* `protocol/src/lib.rs` (68 LOC)
* `protocol/src/hook.rs` (380 LOC)
* `protocol/src/health.rs` (259 LOC)
* `protocol/src/fs.rs` (92 LOC)
## agentd
[Section titled “agentd”](#agentd)
`agentd` is the in-VM daemon supplying the exec and file-transfer APIs AWS Lambda MicroVMs does not have ([`agentd/src/lib.rs:4-7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/lib.rs#L4-L7)). Its twelve modules divide by defect class rather than by HTTP surface: `state` owns the one-shot bootstrap, `auth` decides authorization before a body byte is read, `exec` owns idempotent start with ack-gated release, and `fs` owns streaming tar ([`agentd/src/lib.rs:31-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/lib.rs#L31-L42)). The trust boundary is the crate’s organizing fact — the platform’s own `/run` hook arrives from `127.0.0.1`, indistinguishable at the socket level from a request sent by a process inside the VM, so source-address filtering would reject a legitimate bootstrap and the one-shot property is the only defense left ([`agentd/src/lib.rs:11-16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/lib.rs#L11-L16)). `routes.rs` assembles the router by walking `surface_docs`, the same twenty-endpoint list `/v1/schema` publishes, so a documented route with no handler panics at startup, and each endpoint’s declared auth mode decides which of the two routers it joins ([`agentd/src/routes.rs:31-35`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L31-L35), [`agentd/src/routes.rs:48-58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L48-L58), [`agentd/src/routes.rs:371`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L371), [`docs/schema.json:497-1154`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/docs/schema.json#L497-L1154)).
* `agentd/src/exec.rs` (3798 LOC)
* `agentd/src/fs.rs` (2654 LOC)
* `agentd/src/schema.rs` (929 LOC)
* `agentd/src/routes.rs` (993 LOC)
* `agentd/src/identity.rs` (724 LOC)
* `agentd/src/state.rs` (592 LOC)
* `agentd/src/disk.rs` (435 LOC)
* `agentd/src/config.rs` (196 LOC)
## microvms-core
[Section titled “microvms-core”](#microvms-core)
`microvms-core` is the client library and the workspace’s largest crate, holding the control plane, the in-VM session client, the cost engine, and every trap closure ([`microvms-core/src/lib.rs:2-3`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L2-L3)). Its own doc comment splits it in two: `error`, `region`, `sizing`, `hooks`, and `constants` are the foundation, while `cost`, `control`, `session`, and `sandbox` are the product surface ([`microvms-core/src/lib.rs:59-65`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L59-L65)). Each of the seventeen measured platform findings is spent once here so no caller has to measure it again, and every closure is ranked on a strength ladder where S1 means the mistake cannot be written down at all ([`microvms-core/src/lib.rs:7-14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L7-L14), [`microvms-core/src/lib.rs:23-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L23-L40)). `cost.rs` is the largest file in the repository and carries the rule that makes the rest of it legible — unknown is not zero, so `Amount::Unpriced` is a distinct variant a consumer has to match on rather than a $0.00 line ([`microvms-core/src/cost.rs:22-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L22-L27)) — and the crate re-exports `protocol` so consumers name wire types through here instead of depending on the contract crate ([`microvms-core/src/lib.rs:79-81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L79-L81)). One module sits deliberately above the generic lifecycle: `agents` is the L3 layer, a dated two-row profile table (Claude Code, Codex), an `AgentVm` that derives an image, launches with egress, and provisions Bedrock access, and `agents::bedrock`, which mints the bearer token in process (`microvms-core/src/agents/mod.rs`, `docs/AGENT-VMS.md`). Its free functions (`image_request_for`, `launch_request_for`, `install_access`, `prompt`) are what the bindings drive, because their sandbox sits behind a lock one `AgentVm` cannot own.
* `microvms-core/src/cost.rs` (4156 LOC)
* `microvms-core/src/control/image.rs` (3812 LOC)
* `microvms-core/src/session/exec.rs` (1712 LOC)
* `microvms-core/src/control/microvm.rs` (2467 LOC)
* `microvms-core/src/sandbox.rs` (2643 LOC)
* `microvms-core/src/agents/mod.rs` (1182 LOC)
* `microvms-core/src/control/ops.rs` (2491 LOC)
* `microvms-core/src/control/mod.rs` (1737 LOC)
* `microvms-core/src/session/mod.rs` (1293 LOC)
## microvms-cli
[Section titled “microvms-cli”](#microvms-cli)
`microvms-cli` builds the `microvm` binary: twenty-eight subcommands in lifecycle order over `microvms-core`, and nothing the library does not do ([`microvms-cli/src/cli.rs:83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L83), [`microvms-cli/src/main.rs:2`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L2)). Thinness is checked three ways rather than intended — the direct dependency set contains none of twelve denylisted transport and signing crates, no source file here names a transport or a control-plane operation, and every AWS-touching command must fail when the library seam is made to refuse ([`microvms-cli/src/main.rs:10-13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L10-L13), [`microvms-cli/tests/thinness.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L66)). A coding agent is a first-class consumer, so `microvm manifest` emits the whole command tree with its option domains, exit codes, and envelope schema generated from the parser, and every command writes exactly one envelope object to stdout with progress on stderr ([`microvms-cli/src/main.rs:17-21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L17-L21)). There is no lib target, which is why the modules are declared in `main.rs`, and `guards.rs` — the crate’s largest file — holds the three guards that have to inject a refusing seam from inside the crate and so compiles only under `cfg(test)` ([`microvms-cli/src/main.rs:23-28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L23-L28), [`microvms-cli/src/guards.rs:12-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L12-L20)).
* `microvms-cli/src/guards.rs` (7192 LOC)
* `microvms-cli/src/cli.rs` (3024 LOC)
* `microvms-cli/src/exit.rs` (682 LOC)
* `microvms-cli/src/commands/attached.rs` (3037 LOC)
* `microvms-cli/src/commands/lifecycle.rs` (2620 LOC)
* `microvms-cli/src/render.rs` (1018 LOC)
* `microvms-cli/src/seam.rs` (620 LOC)
* `microvms-cli/src/envelope.rs` (593 LOC)
## microvms-py
[Section titled “microvms-py”](#microvms-py)
`microvms-py` is the PyO3 binding over `microvms-core`: a total, thin mapping where every public core constructor gets one binding constructor and no arithmetic or coercion surface the core does not have ([`microvms-py/src/lib.rs:6-11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/lib.rs#L6-L11)). No validation lives here — no range check, no state check, no region check, no size check — because a guard added in a binding is the copy every Python caller hits and the copy nothing else tests ([`microvms-py/src/lib.rs:12-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/lib.rs#L12-L18)). Four closures a binding could give away for free are each stopped by an absent surface rather than an added check: no `__float__` on a dollar amount, no `__new__` on a duration, no region string on any method, and the two hook timeouts as separate `#[pyclass]`es so transposing them is a `TypeError` before any Rust runs ([`microvms-py/src/lib.rs:20-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/lib.rs#L20-L40)). Methods are synchronous over the async core, blocking on one shared multi-thread tokio runtime with the GIL released first ([`microvms-py/src/lib.rs:42-46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/lib.rs#L42-L46)), and module membership is declared inside the `#[pymodule] mod` so the committed `microvms.pyi` is a function of this file and `mise run stubs:check` fails when the two disagree ([`microvms-py/src/lib.rs:98-101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/lib.rs#L98-L101)). `agents.rs` is the L3 layer as Python sees it: `AgentVm`, `AgentSpec`, and `BearerToken` over the same `Arc>` every session shares, driving the core’s free functions with the specs kept beside the lock (`microvms-py/src/agents.rs`).
* `microvms-py/src/cost.rs` (1124 LOC)
* `microvms-py/src/sandbox.rs` (854 LOC)
* `microvms-py/src/agents.rs` (674 LOC)
* `microvms-py/src/exec.rs` (631 LOC)
* `microvms-py/src/session.rs` (707 LOC)
* `microvms-py/src/errors.rs` (226 LOC)
* `microvms-py/src/lib.rs` (145 LOC)
* `microvms-py/src/hooks.rs` (117 LOC)
* `microvms-py/src/runtime.rs` (92 LOC)
## microvms-js
[Section titled “microvms-js”](#microvms-js)
`microvms-js` is the napi-rs binding over the same core under the same thin-mapping and no-validation rules as the Python side, plus one module the Python side has no twin for — `process`, the same exec seen as two byte streams for a consumer shaped like the AI SDK’s `SandboxProcess` ([`microvms-js/src/lib.rs:6-17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L6-L17), [`microvms-js/src/lib.rs:72-74`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L72-L74)). Its single most important decision is `#[napi]` classes rather than `#[napi(object)]` for anything carrying a closure: `#[napi(object)]` converts by structure, so `{ seconds: 3600 }` would satisfy a `RunHookTimeout` and `{ amount: 1.5 }` an `EstimatedUsd`, which is precisely the coercion those types exist to prevent ([`microvms-js/src/lib.rs:19-35`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L19-L35)). JS coerces more eagerly than Python, so the money type carries no `valueOf`, no `toJSON`, and no `Symbol.toPrimitive` — the figure comes out only through `.amount`, a string ([`microvms-js/src/lib.rs:39-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L39-L42)). Async maps straight through with no `block_on` bridge, at the cost of the one divergence from the Python twin — napi’s async rejection path is typed over its own closed `Status` enum, so a caller branches on `err.cause.message` rather than `err.code` ([`microvms-js/src/lib.rs:58-67`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L58-L67)) — and the generated `index.js`, `index.d.ts`, and `.node` addon are untracked, so they are absent from the list below (`.gitignore:27-29`). `agents.rs` is the L3 layer as JS sees it, the twin of the Python file over tokio’s mutex; `BearerToken` is a `#[napi]` class rather than an object because it carries a secret, so `JSON.stringify` gives `{}` and a look-alike object is rejected by napi’s conversion (`microvms-js/src/agents.rs`).
* `microvms-js/src/cost.rs` (1034 LOC)
* `microvms-js/src/session.rs` (675 LOC)
* `microvms-js/src/exec.rs` (458 LOC)
* `microvms-js/src/sandbox.rs` (680 LOC)
* `microvms-js/src/agents.rs` (609 LOC)
* `microvms-js/src/process.rs` (544 LOC)
* `microvms-js/src/region.rs` (139 LOC)
* `microvms-js/src/lib.rs` (98 LOC)
* `microvms-js/src/errors.rs` (158 LOC)
## model
[Section titled “model”](#model)
`model` builds the `agentd-model` crate, an executable specification rather than daemon code: a state machine whose reachable states stateright enumerates exhaustively, plus the safety properties the real daemon must uphold ([`model/Cargo.toml:2`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L2), [`model/src/lib.rs:3-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L3-L9)). It has one dependency and no edge to any workspace member, because it models the protocol instead of importing it ([`model/Cargo.toml:9-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L9-L10)). The question it settles is whether an in-VM process can hijack the unauthenticated `/run` bootstrap hook, and it prices the unenforced invariant instead of asserting it: `Config::attacker_before_bootstrap` toggles the assumption that no in-VM workload runs before bootstrap, so the model reports both that the attacker never obtains authority while the assumption holds and the concrete path by which it does once the assumption breaks ([`model/src/lib.rs:20-34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L20-L34)). `client.rs` is the deliberate sibling covering what `microvms-core`’s `Sandbox` may do from outside the VM, where `State::wire` counts the calls the client issued so a property can say no resume ever fires once `was_terminated` holds ([`model/src/client.rs:2-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L2-L9), [`model/src/client.rs:23-30`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L23-L30)).
* `model/src/client.rs` (982 LOC)
* `model/src/lib.rs` (658 LOC)
* `model/Cargo.toml` (17 LOC)
## Supporting code
[Section titled “Supporting code”](#supporting-code)
Verification tooling, generated surfaces, and requirements data. None of it is a workspace crate ([`Cargo.toml:2-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L2-L9)), and none of it is a module under the enumeration rule that skips tooling-only paths.
* `conformance/run_rs.py` (5055 LOC)
* `microvms-py/microvms.pyi` (1759 LOC)
* `scripts/check-model-drift.py` (1089 LOC)
* `spec/core.symspec.json` (1049 LOC)
* `scripts/check-live-rates.py` (625 LOC)
* `scripts/check-live-wiring.py` (450 LOC)
* `scripts/generate-py-stubs.py` (345 LOC)
* `scripts/check-lint-coverage.py` (292 LOC)
* `conformance/infra/main.tf` (281 LOC)
* `spec/microvms-core-kickoff.md` (175 LOC)
* `scripts/verify-clean.py` (589 LOC)
* `examples/coding-agents-on-bedrock/run.sh` (145 LOC)
* `spec/agentd.symspec.json` (117 LOC)
* `scripts/check-license-headers.py` (131 LOC)
* `examples/coding-agents-on-bedrock/Dockerfile` (51 LOC)
Related: [System overview](system-overview.md) · [Data flow](data-flow.md) · [Contract map](../insights/contract-map.md) · [Impact analysis](../insights/impact-analysis.md) · [Tech debt](../insights/tech-debt.md) · [CLI reference](../../reference/cli.md)
## See also
[Section titled “See also”](#see-also)
* [system overview](system-overview.md) — 8 shared source citations
* [business logic](../insights/business-logic.md) — 8 shared source citations
* [contract map](../insights/contract-map.md) — 8 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 6 shared source citations
* [dependency graph](../diagrams/structural/dependency-graph.md) — 5 shared source citations
# System overview
> AWS Lambda MicroVMs hands you an isolated Firecracker VM and no way to use it: there is no API to run a command inside one and no API to move a file into or out of one…
AWS Lambda MicroVMs hands you an isolated Firecracker VM and no way to use it: there is no API to run a command inside one and no API to move a file into or out of one ([`docs/PLATFORM.md:20-23`](/microvms-agentd/internals/platform/)). Every harness built on the service has to supply both itself. This repository is that supply — `agentd`, a static daemon baked into the VM image, plus the `microvm` CLI and the Rust, Python, and Node libraries that talk to it ([`README.md:11-15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/README.md#L11-L15)). The CLI and libraries are distributed through crates.io, PyPI, and npm; the daemon is a release binary. Selected crates opt into publishing despite the workspace default of `publish = false`. The audience is whoever builds a sandbox product on MicroVMs — an agent harness, a CI runner, a code-execution service.
The client’s real work is absorbing the platform’s surprises once. `docs/PLATFORM.md` records dated findings, many of which are traps in the specific sense that the platform’s answer points away from the cause: an unsupported region answers `AccessDeniedException` with a null message, a `clientToken` replay wedges an image in `CREATING` for fifteen hours with no error at all ([`microvms-core/src/lib.rs:7-14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L7-L14)). Each closure is ranked by strength — S1 inexpressible, S2 rejected locally before any call, S3 correct by default and overridable ([`microvms-core/src/lib.rs:23-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L23-L40)).
Seven crates carry that ([`Cargo.toml:2-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L2-L10)), and the seams follow defect classes rather than layers. `protocol` is the wire contract as types: pure data, serde plus schemars, no tokio, no axum, no base64 ([`protocol/src/lib.rs:16-21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L16-L21), 68 LOC). Both the daemon and the client compile against it, so a renamed field fails a build instead of a consumer’s runtime ([`agentd/Cargo.toml:11-15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L11-L15)). `agentd` is the daemon, twelve modules across 16,280 LOC ([`agentd/src/lib.rs:31-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/lib.rs#L31-L42), 45 LOC) — `state` owns the one-shot bootstrap, `auth` decides before a body byte is read, `exec` and `fs` own idempotent exec and streaming tar. Its router is assembled by walking the same endpoint list `/v1/schema` publishes, so a documented route with no handler panics at startup ([`agentd/src/routes.rs:29-35`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L29-L35), 993 LOC); there are twenty, split into a Bearer-guarded `control` router and an `open` one ([`agentd/src/routes.rs:51-59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L51-L59), [`agentd/src/routes.rs:110-140`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110-L140)). It runs as the container `CMD` on a current-thread runtime sized for a 512 MiB guest ([`agentd/src/main.rs:4-6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L4-L6), [`agentd/src/main.rs:24-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L24-L27)).
`microvms-core` is the client library and the largest crate — 39,097 LOC over 34 files, eleven modules, nine of which its own doc comment splits into foundation and product surface, with `agents` as the one layer above them ([`microvms-core/src/lib.rs:61-77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L61-L77)). `control` speaks hand-signed SigV4 rest-json because `lambda-microvms` has no SDK crate ([`microvms-core/src/control/mod.rs:2-3`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L2-L3)); `session` is the in-VM client, carrying proxy auth and the byte-offset cursor that makes an interrupted stream resumable ([`microvms-core/src/session/mod.rs:4-7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L4-L7)); `sandbox` keeps every lifecycle field private so the Z3 proofs are proofs about the code ([`microvms-core/src/sandbox.rs:11-17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L11-L17)); `cost` treats unpriced as a distinct variant rather than zero ([`microvms-core/src/cost.rs:22-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L22-L27)).
`microvms-cli` ships `microvm` with twenty-eight subcommands ([`microvms-cli/src/cli.rs:95-351`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L95-L351), 3,024 LOC), each invocation writing exactly one JSON envelope to stdout and progress to stderr ([`microvms-cli/src/envelope.rs:4-11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L4-L11)). It has no lib target ([`microvms-cli/Cargo.toml:21-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L21-L23)) and no second path to AWS: a twelve-crate denylist of HTTP clients, signers, and credential chains is asserted against `cargo metadata` ([`microvms-cli/tests/thinness.rs:49-96`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L49-L96)). `microvms-py` and `microvms-js` wrap the same core and never the CLI ([`microvms-py/Cargo.toml:22-26`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L22-L26), [`microvms-js/Cargo.toml:20-21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L20-L21)). Verification sits outside the product graph: `model` has one dependency and no workspace edge, modelling the protocol rather than importing it ([`model/Cargo.toml:12-13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L12-L13)), and `conformance/run_rs.py` drives the built CLI through 188 named checks against real AWS ([`conformance/run_rs.py:9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L9), 5,055 LOC). Start at [`agentd/src/lib.rs:9-29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/lib.rs#L9-L29) for the trust boundary, then [`microvms-core/src/lib.rs:21-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L21-L40) for the trap ladder.
## Stack
[Section titled “Stack”](#stack)
| Layer | Technology | Source |
| --------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Language | Rust, `edition = "2024"`, `resolver = "3"` | [`Cargo.toml:23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L23), [`Cargo.toml:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L11) |
| Toolchain and targets | `channel = "stable"`, `targets = ["aarch64-unknown-linux-musl", "x86_64-unknown-linux-musl"]` | [`rust-toolchain.toml:13-16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/rust-toolchain.toml#L13-L16) |
| Shipping artifact | `lto`, `codegen-units = 1`, `panic = "unwind"`, `strip`, `opt-level = "z"` | [`Cargo.toml:36-59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L36-L59) |
| Daemon HTTP | `axum = "0.8.9"`; `tower-http` `"0.6"` with `limit` + `catch-panic` | [`agentd/Cargo.toml:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L16), [`agentd/Cargo.toml:25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L25) |
| Async runtime | `tokio = "1.53"`, no `rt-multi-thread` in the daemon or the library | [`agentd/Cargo.toml:26`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L26), [`microvms-core/Cargo.toml:104`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L104) |
| AWS control plane | `reqwest = "0.13"` on `rustls`, `aws-sigv4 = "1.5"`, `aws-config = "1.10"` | [`microvms-core/Cargo.toml:76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L76), [`microvms-core/Cargo.toml:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L70), [`microvms-core/Cargo.toml:59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L59) |
| Wire schema | `schemars = "1.2.2"`, `default-features = false`, `derive` + `std` only | [`protocol/Cargo.toml:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/Cargo.toml#L16) |
| Money | `rust_decimal = "1.42"` with `serde-with-str` | [`microvms-core/Cargo.toml:46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L46) |
| CLI surface | `clap = "4.6.6"` with `derive`; `ratatui = "0.30.2"` | [`microvms-cli/Cargo.toml:55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L55), [`microvms-cli/Cargo.toml:59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L59) |
| Bindings | `pyo3 = "0.29"` with `abi3-py39`; `napi = "3"` with `napi5` + `async` + `web_stream` | [`microvms-py/Cargo.toml:38`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L38), [`microvms-js/Cargo.toml:44-48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L44-L48) |
| Verification tiers | `stateright = "0.31"`, `turmoil = "0.7.2"`, `proptest = "1.11"` | [`model/Cargo.toml:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L10), [`agentd/Cargo.toml:76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L76), [`agentd/Cargo.toml:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L73) |
| Live suite | PEP 723 inline script under `uv`, `boto3` + `httpx` | [`conformance/run_rs.py:1-5`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L1-L5) |
| Build gate | `mise run check` — lint, security, tests, schema, stubs, model drift, build | [`mise.toml:290-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L290-L301) |
## Module map
[Section titled “Module map”](#module-map)
## See also
[Section titled “See also”](#see-also)
* [impact analysis](../insights/impact-analysis.md) — 16 shared source citations
* [contract map](../insights/contract-map.md) — 10 shared source citations
* [dependency graph](../diagrams/structural/dependency-graph.md) — 9 shared source citations
* [module map](module-map.md) — 8 shared source citations
* [business logic](../insights/business-logic.md) — 8 shared source citations
# Processes
> Three initiator families drive every process here. The daemon's HTTP surface, assembled by walking one list so a route cannot be served unless it is documented …
Three initiator families drive every process here. The daemon’s HTTP surface, assembled by walking one list so a route cannot be served unless it is documented — [`agentd/src/routes.rs:36`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L36), dispatched through the exhaustive match at [`agentd/src/routes.rs:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110). The client library’s public lifecycle methods on `Sandbox` — [`microvms-core/src/sandbox.rs:551`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551), [`:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648), [`:755`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L755), [`:837`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L837), [`:935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L935). And the CLI’s command handlers, dispatched from [`microvms-cli/src/main.rs:63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L63).
The eight processes below carry the load-bearing behavior. Everything else, including the proxy-token mint that runs inside every request and the `microvm run` command that composes four of these processes, is listed under `## Minor flows` with its entry point.
## Image build and the stalled-build probe
[Section titled “Image build and the stalled-build probe”](#image-build-and-the-stalled-build-probe)
Entry point: [`microvms-core/src/sandbox.rs:551`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551)
1. `Sandbox::build_image` records the requested size class and hands the request to the control plane; the local guards live one level down because the create happens *after* the caller’s artifact upload — [`microvms-core/src/sandbox.rs:551`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551).
2. `ControlPlane::create_image` runs `preflight` before its own wire call, and delegates rather than keeping a second copy of the list, so the two call sites cannot drift — [`microvms-core/src/control/image.rs:158`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L158).
3. `preflight` is the whole guard list and is callable before the upload: image name, `require_workdir` under `inherit_workdir`, and for a supplied Dockerfile the matching `FROM`, the agreeing agentd port, a keepalive under the stream idle timeout, and a `CMD` that runs the daemon — [`microvms-core/src/control/image.rs:219`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L219).
4. The wire body injects the one architecture value and derives the one accepted OS capability from a boolean, mints the `clientToken` from a label rather than accepting one, then goes out through `send_with_retry` — [`microvms-core/src/control/image.rs:193`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L193).
5. `wait_for_image` refuses an empty identifier before the loop — an empty one collapses the URI onto the collection and polls the *listing* until the deadline — then polls `GetMicrovmImage`, returning on `Image::is_ready` and raising through `build_failure` on `Image::is_failed` — [`microvms-core/src/control/image.rs:296`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L296).
6. Once elapsed time passes `WaitOpts::stall_grace` — `DEFAULT_STALL_GRACE` is 240s at [`microvms-core/src/control/image.rs:37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L37) — the wait probes exactly once, tracked by a `probed` flag rather than re-armed — [`microvms-core/src/control/image.rs:319`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L319).
7. `probe_stalled_build` returns `Ok` for an unreadable build list and `Ok` for an empty one, because neither is evidence; only an all-`PENDING` list raises `ErrorKind::BuildWedged` naming the `clientToken` replay signature — [`microvms-core/src/control/image.rs:353`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L353).
8. Back in `build_image`, `image_exists` is set before anything is launched, so a teardown can name the image whether or not a VM ever ran from it — [`microvms-core/src/sandbox.rs:560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L560).
### Related
[Section titled “Related”](#related)
* [`microvms-core/src/control/image.rs:406`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L406)
* [`microvms-core/src/control/image.rs:489`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L489)
* [`microvms-core/src/control/artifact.rs:495`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L495)
* [`microvms-core/src/control/token.rs:101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L101)
* [`microvms-core/src/control/mod.rs:893`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L893)
* [`microvms-core/src/control/transport.rs:275`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L275)
## Launch and the RUNNING wait
[Section titled “Launch and the RUNNING wait”](#launch-and-the-running-wait)
Entry point: [`microvms-core/src/sandbox.rs:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648)
1. `Sandbox::run` refuses a second launch on the same sandbox before any call. This is STATE-3’s local half, and the refusal is here because a retry loop around a timed-out launch is the plausible mistake — [`microvms-core/src/sandbox.rs:652`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L652).
2. The image identifier comes from the request or from a previously built image; neither is a `Precondition` error naming both remedies — [`microvms-core/src/sandbox.rs:663`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L663).
3. An agent token is minted unless the caller supplied one, then `RunHookPayload::for_launch` re-checks the 4096-byte budget even though this code built the JSON, because neither the token nor the launch env is this crate’s — [`microvms-core/src/sandbox.rs:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L682).
4. `ControlPlane::run_microvm` closes the local guards: identifier, duration range, idle duration, pinned version, and the execution role ARN — [`microvms-core/src/control/microvm.rs:357`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L357).
5. Connector intents are split into separate ingress and egress members rather than concatenated, the combined count is checked against the `NetworkConnectorList` ceiling, and `RunMicrovm` goes out with a `clientToken` minted from a scope label — [`microvms-core/src/control/microvm.rs:378`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L378).
6. On acceptance the lifecycle moves to `Pending`, `image_exists` is set, and `suspended_window` is recorded from *this* request — the only place the value is knowable, since `GetMicrovm` does not return it — [`microvms-core/src/sandbox.rs:699`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L699).
7. `wait_for_running` delegates to `wait_for_state` with the terminal set as `fail_on`, so a VM that dies during startup raises through `reached_terminal_state` with both the state and `stateReason` — [`microvms-core/src/control/microvm.rs:436`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L436).
8. Reaching RUNNING sets `token_installed` and increments `bootstrap_count` (STATE-2, STATE-3), then a `ControlPlaneMinter` goes behind an `Arc` into `Session::builder` so minting happens inside every later request — [`microvms-core/src/sandbox.rs:722`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L722).
### Related
[Section titled “Related”](#related-1)
* [`microvms-core/src/control/microvm.rs:450`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L450)
* [`microvms-core/src/control/microvm.rs:486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L486)
* [`microvms-core/src/control/microvm.rs:165`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L165)
* [`microvms-core/src/sandbox.rs:1097`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1097)
* [`microvms-core/src/session/mod.rs:230`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L230)
* [`microvms-core/src/sandbox.rs:385`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L385)
## Suspend and resume with the launch-time window
[Section titled “Suspend and resume with the launch-time window”](#suspend-and-resume-with-the-launch-time-window)
Entry point: [`microvms-core/src/sandbox.rs:755`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L755)
1. `Sandbox::suspend` resolves the VM id through `require_microvm`, then refuses any lifecycle but RUNNING (STATE-5). Zero control-plane calls on the refusal is the observable a test asserts on — [`microvms-core/src/sandbox.rs:759`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L759).
2. `SuspendMicrovm` goes out **first**, and only then does the lifecycle move to `Suspending` — moving before the call would leave a throttled or dropped request stuck in a state neither suspend nor resume accepts, bricking the handle over one bad request — [`microvms-core/src/sandbox.rs:773`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L773).
3. `suspended_at` is stamped from the control plane’s own clock after the call and before the wait, because the `idlePolicy` window starts when the platform begins suspending — [`microvms-core/src/sandbox.rs:778`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L778).
4. `wait_for_state` is called with `SUSPEND_WANTED`, so TERMINATED is a wanted outcome that sets `was_terminated` rather than an error raised out of the middle of a teardown; anything else is `ErrorKind::Platform` — [`microvms-core/src/sandbox.rs:794`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L794).
5. `Sandbox::resume` refuses a terminated VM first (STATE-11), then any lifecycle but SUSPENDED (STATE-7) — [`microvms-core/src/sandbox.rs:842`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L842).
6. `require_open_suspended_window` compares elapsed time against the recorded window and refuses locally once it has passed; with no window recorded — the attach path — it returns `Ok`, because no default would be reliable — [`microvms-core/src/sandbox.rs:902`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L902).
7. `ResumeMicrovm` goes out, then `wait_for_state` waits for RUNNING with `DEAD_STATES` as `fail_on`. The terminal set is wrong here because SUSPENDED, the state the call was made from, is in it — [`microvms-core/src/sandbox.rs:865`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L865).
8. `Session::rebind` takes the endpoint the service just reported, invalidating the cached proxy token (STATE-8), and `suspended_at` is cleared so the next cycle measures its own window — [`microvms-core/src/sandbox.rs:881`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L881).
### Related
[Section titled “Related”](#related-2)
* [`microvms-core/src/control/microvm.rs:541`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L541)
* [`microvms-core/src/control/microvm.rs:549`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L549)
* [`microvms-core/src/session/mod.rs:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L314)
* [`microvms-core/src/session/proxy.rs:410`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L410)
* [`microvms-core/src/sandbox.rs:1043`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1043)
* [`agentd/src/routes.rs:276`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L276)
## Teardown
[Section titled “Teardown”](#teardown)
Entry point: [`microvms-core/src/sandbox.rs:935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L935)
1. `Sandbox::terminate` returns a `TeardownReport` rather than a `Result` and marks the sandbox torn down. It runs where a caller’s `finally` would, and an error raised from there would replace the real failure — [`microvms-core/src/sandbox.rs:936`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L936).
2. The session is dropped first, because its only remaining asset is a cached proxy token for a VM that is going away — [`microvms-core/src/sandbox.rs:941`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L941).
3. The lifecycle moves to `Terminating` and `was_terminated` is set **before** the call, so a terminate whose call fails still blocks a later resume instead of leaving the sandbox looking resumable — [`microvms-core/src/sandbox.rs:948`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L948).
4. `TerminateMicrovm` goes out; a failure is pushed onto both `failures` and `undeleted` rather than raised — [`microvms-core/src/sandbox.rs:951`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L951).
5. When `wait_for_terminated` was requested, `wait_for_state(["TERMINATED"])` runs. A timeout there is recorded as a failure but **not** a leak, because the platform accepted the terminate — [`microvms-core/src/sandbox.rs:967`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L967).
6. The image goes second, through `delete_image`’s retry loop; the identifier is checked before the loop so an invalid one costs one comparison rather than nineteen backoff sleeps, and the refusal is `false` rather than an error because this path must not raise — [`microvms-core/src/control/image.rs:1069`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L1069).
7. `try_delete_image` collects **every** page of versions before the first delete, drops all but the first version, deletes the image, and parses the readback for a failure spelling — a one-page read would leave an image nothing can delete — [`microvms-core/src/control/image.rs:1099`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L1099).
8. The build log group is handled **last**, and named rather than deleted: CloudWatch Logs is not in the crate’s dependency set, so the group lands in `undeleted` with a failure line saying why — [`microvms-core/src/sandbox.rs:1011`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1011).
### Related
[Section titled “Related”](#related-3)
* [`microvms-core/src/control/image.rs:81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L81)
* [`microvms-core/src/control/microvm.rs:563`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L563)
* [`microvms-core/src/sandbox.rs:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L297)
* [`microvms-core/src/sandbox.rs:364`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L364)
* [`microvms-cli/src/commands/lifecycle.rs:393`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L393)
* [`microvms-cli/src/ledger.rs:124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L124)
## Runtime hooks and one-shot bootstrap
[Section titled “Runtime hooks and one-shot bootstrap”](#runtime-hooks-and-one-shot-bootstrap)
Entry point: [`agentd/src/routes.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178)
1. `ready` and `validate` answer 200 during the image build, before any instance and therefore before any token exists. Gating them on bootstrap state would fail every build — [`agentd/src/routes.rs:244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L244).
2. Hook routes go into the unauthenticated `open` router while Bearer endpoints go into `control` behind the token guard, split at assembly by each endpoint’s declared auth — the platform has no credential to present and the hook prefix is fixed by the service — [`agentd/src/routes.rs:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L53).
3. `run_hook` answers 400 for a body that is not JSON and 400 for an envelope carrying no `runHookPayload`. Never 404: a client that maps 404 onto “missing file” would report a phantom absent artifact for a protocol error — [`agentd/src/routes.rs:182`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L182).
4. The `runHookPayload` member is parsed a *second* time, because the platform wraps the caller’s string — so `agent_token` sits one JSON parse deeper than the request body. A rejection returns the typed error’s own text, since this is the one route whose failure is invisible from outside the VM — [`agentd/src/routes.rs:199`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L199).
5. `AppState::bootstrap` takes the token lock through `recover`, installs into an empty slot, or compares constant-time against what is there. The launch env is installed under the same lock so a racer that loses the token cannot win the environment — [`agentd/src/state.rs:202`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202).
6. The three outcomes map to statuses: `Installed` and `AlreadyIdentical` are both 200 — the platform may retry its own hook, and 409 would fail a launch that is fine — while `Conflict` is 409. The log line carries the launch-env *count*, never the values — [`agentd/src/routes.rs:213`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L213).
7. `suspend`, `resume`, and `terminate` acknowledge. `resume` warns loudly when the token is absent, because measured behavior is that a suspend/resume preserves in-memory bootstrap state and exec records — [`agentd/src/routes.rs:276`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L276).
8. Every later control request runs `require_token`, which separates not-yet-bootstrapped (503) from a wrong credential (401) using `token_matches`’s three-valued answer, and drains a bounded prefix of a rejected body so hyper does not answer with a TCP RST — [`agentd/src/auth.rs:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L69).
### Related
[Section titled “Related”](#related-4)
* [`agentd/src/routes.rs:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110)
* [`agentd/src/routes.rs:371`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L371)
* [`agentd/src/state.rs:78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L78)
* [`agentd/src/state.rs:245`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L245)
* [`agentd/src/auth.rs:92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L92)
* [`agentd/src/routes.rs:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L148)
## Exec start and its detached waiter
[Section titled “Exec start and its detached waiter”](#exec-start-and-its-detached-waiter)
Entry point: [`agentd/src/exec.rs:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L331)
1. `start` rejects a malformed body and an empty `exec_id` with 400, then validates `timeout_sec` before anything is spawned — validating it in the waiter left a running child with nobody to reap it — [`agentd/src/exec.rs:343`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L343).
2. `build_command` assembles the child: one joined script to `sh -c` when `shell` is set, `env_clear()` so the daemon’s environment and the agent token cannot reach it, launch env then request env so the request’s copy of a key wins, an omitted `cwd` inheriting rather than defaulting to `/`, `process_group(0)`, and `/dev/null` on stdin unless stdin was requested — [`agentd/src/exec.rs:1040`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1040).
3. Idempotency is decided under the registry lock before the spawn, so two concurrent retries cannot both find the slot empty; a known id returns 200 with the existing entry’s phase and output untouched — [`agentd/src/exec.rs:366`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L366).
4. `spawn` starts the child and captures the pgid immediately, while `Child::id()` still answers — a lazy read would find nothing for exactly the fast-then-forking commands that most need killing — [`agentd/src/exec.rs:1119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1119).
5. The three pipes are taken out of the `Child`, a bounded broadcast channel is created, and `Shared` plus the registry entry are built. Taking stdin out is also why the daemon must drop its own copy on EOF, since `wait()` closes only what the `Child` still owns — [`agentd/src/exec.rs:1121`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1121).
6. A detached task runs `super_wait`, whose first phase selects `child.wait()` against both pipe pumps together. Concurrent draining is required: a child filling a 64 KiB pipe buffer blocks in `write` forever if nobody reads, and the timeout branch signals the whole process group before completing the wait — [`agentd/src/exec.rs:1235`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1235).
7. Phase two lingers after the child is gone so grandchildren still holding the write end are read, setting `writers_may_be_alive` when the deadline cuts the drain short. This is the case temp files got wrong — they were unlinked here, so anything a backgrounded process wrote afterward went to a file with no name — [`agentd/src/exec.rs:1258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1258); recorded as `.erpaval/solutions/best-practices/pipes-not-tempfiles-for-subprocess-output.md`.
8. The waiter writes the `terminal` marker **before** `result`, so a stream that sees `Finished` immediately can always find an exit event, then drops the daemon’s stdin copy and sends `Frame::Finished` — [`agentd/src/exec.rs:1182`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1182).
### Related
[Section titled “Related”](#related-5)
* [`agentd/src/exec.rs:1018`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1018)
* [`agentd/src/exec.rs:1351`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1351)
* [`agentd/src/exec.rs:1399`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1399)
* [`agentd/src/exec.rs:1438`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1438)
* [`agentd/src/exec.rs:951`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L951)
* [`agentd/src/state.rs:257`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L257)
## Exec stream attach and byte-offset reconnect
[Section titled “Exec stream attach and byte-offset reconnect”](#exec-stream-attach-and-byte-offset-reconnect)
Entry point: [`agentd/src/exec.rs:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L455)
1. `stream` refuses a non-integer offset with 400 and an unknown id with 404, so a client can tell a protocol mistake from an absent exec — [`agentd/src/exec.rs:460`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L460).
2. `Shared::attach` subscribes to the live channel and snapshots the replay ring under one lock. `publish` holds that same lock across its broadcast send, which is what makes the pair atomic — written as two statements it is a silent one-chunk hole that only appears under load — [`agentd/src/exec.rs:293`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L293).
3. `Log::since` returns the gap the requested offset fell into rather than papering it over, plus the cursor clamped forward to the ring’s start. Handing back a later window with no marker is the failure a cursorless attach has by construction — [`agentd/src/exec.rs:186`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L186).
4. The terminal marker is read **after** the snapshot, so an exec that finishes between the two is observed as finished rather than waiting forever on a `Finished` that was sent before the subscribe — [`agentd/src/exec.rs:479`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L479).
5. `Attach::take_chunk` drops the prefix already delivered, and emits a `gap` event when a live chunk lands past the cursor rather than a chunk at a discontinuous offset the client cannot reconcile — [`agentd/src/exec.rs:515`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L515).
6. The response carries `x-accel-buffering: no`, because a buffering proxy otherwise holds events until its own buffer fills, turning a live stream into a batch delivered at exit — [`agentd/src/exec.rs:489`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L489).
7. On the client, `ExecHandle::advance` advances its cursor only past bytes it handed to the consumer, and past a gap’s `to` — otherwise a reconnect asks for the evicted range again and is told about the same gap forever — [`microvms-core/src/session/exec.rs:526`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L526).
8. A body that ends with no exit event becomes `Reconnect` with `attempts + 1`, backed off from a fixed table and re-attached at the cursor; because the streaming path builds its own headers, that reconnect also re-mints an expired proxy token — [`microvms-core/src/session/exec.rs:569`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L569). The cursor is the whole mechanism: `.erpaval/solutions/architecture-patterns/byte-offset-cursor-is-what-makes-reconnect-work.md`.
### Related
[Section titled “Related”](#related-6)
* [`agentd/src/exec.rs:255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L255)
* [`agentd/src/exec.rs:560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L560)
* [`agentd/src/exec.rs:535`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L535)
* [`microvms-core/src/session/exec.rs:591`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L591)
* [`microvms-core/src/session/exec.rs:716`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L716)
* [`microvms-core/src/session/exec.rs:347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L347)
## Tar upload and confined extraction
[Section titled “Tar upload and confined extraction”](#tar-upload-and-confined-extraction)
Entry point: [`agentd/src/fs.rs:1433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1433)
1. `write_tar` reads `?path=` and refuses a relative root, which would otherwise resolve against the daemon’s own working directory — the image `WORKDIR`, and not something the caller can see — [`agentd/src/fs.rs:1439`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1439).
2. The disk guard preflights the extraction root before the body is spooled, so an upload aimed at a full filesystem is refused without first spending the disk and the wire time to receive it — [`agentd/src/fs.rs:1459`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1459).
3. `spool_body` writes the archive to an unlinked spool under the same guard. Pressure becomes 507 naming the actual free space; a body that dies on the wire, including the 413 the body-limit layer injects, becomes 400 because nothing on the daemon’s side failed — [`agentd/src/fs.rs:1463`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1463).
4. Extraction is handed to `spawn_blocking`, since `tar`’s reader is blocking rather than async — [`agentd/src/fs.rs:1479`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1479).
5. `Confined::open` holds the kernel’s half of the confinement for the whole extraction, and ownership and xattr preservation are switched off the way CPython’s `data` filter drops them — [`agentd/src/fs.rs:631`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L631).
6. Per member the PAX-aware accessors are used rather than their fixed-field header counterparts, then `resolve_member` classifies the destination as under the root, escaping, or naming the root itself — where only a directory is tolerated, since a file or link there would redirect every later member — [`agentd/src/fs.rs:679`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L679).
7. Character devices, block devices, and FIFOs are refused outright. Link targets are refused when absolute, and otherwise checked against the base depth their kind implies — a symlink resolves from its own parent, a hard link from the root — [`agentd/src/fs.rs:704`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L704).
8. Data members are counted against the byte cap, streamed through the confined descriptor, and paced against the disk after each one; directory and file modes are deferred and replayed deepest-first masked to `0o755`, because applying a `0o500` directory mode at creation blocks every write beneath it — [`agentd/src/fs.rs:810`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L810).
### Related
[Section titled “Related”](#related-7)
* [`agentd/src/fs.rs:274`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L274)
* [`agentd/src/fs.rs:228`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L228)
* [`agentd/src/fs.rs:198`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L198)
* [`agentd/src/fs.rs:825`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L825)
* [`agentd/src/fs.rs:872`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L872)
* [`agentd/src/fs.rs:106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L106)
## Minor flows
[Section titled “Minor flows”](#minor-flows)
* Daemon startup — entry at [`agentd/src/main.rs:14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L14). Builds a current-thread runtime capped at four blocking threads, runs identity repair before the listener binds, spawns the 30-second exec collector, then serves.
* Startup identity repair — entry at [`agentd/src/identity.rs:230`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L230). Mints one fresh id and reuses it so the machine id and the hostname agree; without a fresh id every id-derived step is reported failed rather than silently skipped.
* Graceful shutdown — entry at [`agentd/src/serve.rs:19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/serve.rs#L19). Selects SIGTERM against ctrl-c and drains in-flight requests, so a harness waiting on `/v1/exec/{id}` gets its status rather than a transport error.
* Exec poll — entry at [`agentd/src/exec.rs:407`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L407). Strictly read-only with respect to the registry; an acked entry reports no output, because repeating it would contradict the phase.
* Exec stdin write — entry at [`agentd/src/exec.rs:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L682). A separate endpoint from the output stream on purpose: multiplexing the write half onto the read connection makes reconnecting load-bearing for correctness rather than only for observation.
* Exec ack — entry at [`agentd/src/exec.rs:831`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L831). Releases the buffered output and starts the TTL clock; `acked_at` is what separates “still running” from “an earlier ack already took it”, both of which find an empty slot.
* Exec kill — entry at [`agentd/src/exec.rs:905`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L905). Signals the whole process group rather than the direct child, and answers `killed: false` rather than 500 when no pgid was ever captured.
* Expired-exec collection — entry at [`agentd/src/exec.rs:951`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L951). A plain function the daemon’s own loop calls, not a task spawned per request; only acked entries are eligible, since an unacked one holds output nobody read.
* Exec activity — entry at [`agentd/src/exec.rs:997`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L997). Reports whether anything is still producing and how many entries are registered, for an orchestrator outside the VM.
* Health — entry at [`agentd/src/routes.rs:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314). Version, bootstrap state, a disk reading that is null when unmeasurable, the identity-repair verdict, and the exec-activity pair.
* Schema publication — entry at [`agentd/src/routes.rs:361`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L361). Serves the same `surface_docs()` list the router was assembled from, unauthenticated so a client can negotiate versions before it holds a token.
* Version stamping — entry at [`agentd/src/routes.rs:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L148). Applied outside `route_layer` so it covers handler bodies, the auth middleware’s 401/503, the body-limit 413, and the 404 fallback alike.
* File read — entry at [`agentd/src/fs.rs:1092`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1092). Streams the bytes, or a 1-based inclusive line range with an `end_line` past EOF reading through rather than erroring; 404 only when the path is genuinely absent.
* File write — entry at [`agentd/src/fs.rs:1182`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1182). Not confined to a root, since the path is the caller’s; the mode is parsed before a single byte lands, because validating after writing left a file behind with the wrong permissions.
* Tar download — entry at [`agentd/src/fs.rs:1370`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1370). Refuses a non-directory with 400 and an absent one with 404, estimates the tree against both caps, then packs with symlinks preserved into a rewound spool.
* Proxy-token mint in the request path — entry at [`microvms-core/src/session/mod.rs:106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L106). `Transport::request` mints inside the path every request takes, because a token minted once at construction expires mid-run and the resulting rejection looks like a dead daemon.
* Control-plane send with retry — entry at [`microvms-core/src/control/transport.rs:275`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L275). Exponential backoff with jitter over five attempts, gated on `Error::retryable`; mutating calls are safe to retry because each carries a `clientToken`.
* Session readiness wait — entry at [`microvms-core/src/session/mod.rs:342`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L342). Polls unauthenticated health through connection errors, returns at once on a fatal one, and names the last retryable error in the timeout.
* Client-side wait-then-ack — entry at [`microvms-core/src/session/exec.rs:687`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L687). Returns the ack’s own result rather than a post-ack poll, because the poll reports `acked` with no output.
* CLI dispatch — entry at [`microvms-cli/src/main.rs:63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L63). Reads `--json` and `--dense` off the raw tokens before the parse so an argument error still produces an envelope, and returns `ExitCode` rather than calling `exit` so `Sandbox`’s drop warning still runs.
* CLI run — entry at [`microvms-cli/src/commands/lifecycle.rs:121`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L121). Build, launch, exec, report, tear down in one invocation, with the interrupt future passed in so the teardown guard is testable.
* CLI build — entry at [`microvms-cli/src/commands/lifecycle.rs:515`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L515). Builds without launching; `--reuse` content-keys the image name by hash, because recreating an image under a previously-used fixed name can serve a stale snapshot.
* CLI attached suspend — entry at [`microvms-cli/src/commands/lifecycle.rs:915`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L915). Spends one `GetMicrovm` to refuse locally from anything but RUNNING, which is how STATE-5’s local half holds on a path that did not send the launch.
* CLI attached resume — entry at [`microvms-cli/src/commands/lifecycle.rs:976`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L976). Skips the suspended-window check, since a process that did not send the launch cannot know `suspendedDurationSeconds`, and relies on `fail_on: DEAD_STATES` instead.
* CLI attached terminate — entry at [`microvms-cli/src/commands/lifecycle.rs:1017`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L1017). VM, then image, then the log group last, and never fails on a teardown failure — it reports the identifier, which is the only remedy for a resource that would not delete.
* CLI exec — entry at [`microvms-cli/src/commands/attached.rs:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L103). Four shapes over one subcommand — start and wait, `--stream`, `--stdin`, `--poll` — because they are one question asked at different points in an exec’s life.
* CLI exec stream — entry at [`microvms-cli/src/commands/attached.rs:240`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L240). Drives core’s callback loop rather than a `Stream`, so the crate needs no `futures-util` dependency, and reports `nextOffset` from core’s own cursor.
* CLI health — entry at [`microvms-cli/src/commands/attached.rs:475`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L475). Warns about a degraded identity and disk pressure on stderr while keeping exit 0, because the daemon’s contract is to serve anyway and draining is the operator’s decision.
* CLI ack — entry at [`microvms-cli/src/commands/attached.rs:600`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L600). Issues the ack on its own for a detached caller; both 409 shapes collapse onto `ERR_PROTOCOL` while the daemon’s `still_running` or `already_acked` detail rides in the message.
* CLI stdin — entry at [`microvms-cli/src/commands/attached.rs:638`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L638). Reads `-` from this process’s stdin before writing, and surfaces the daemon’s 409-versus-410 split through `data.kind`.
* CLI cp — entry at [`microvms-cli/src/commands/attached.rs:805`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L805). Resolves direction from the `vm:` prefix and inspects no archive, so the daemon’s confined extractor stays the only extractor in the system and the only one under test.
* CLI doctor — entry at [`microvms-cli/src/commands/doctor.rs:32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L32). Region, credentials, managed bases, infra, Terraform, and the binary’s ELF machine — the check that turns a host-architecture binary from a 45-minute mystery into a line of output.
* CLI ls — entry at [`microvms-cli/src/commands/local.rs:23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L23). Reads the local ledger and lists what this CLI created and could not confirm it deleted.
* CLI logs — entry at [`microvms-cli/src/commands/local.rs:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L148). Derives and names the build log group and exits `ERR_PRECONDITION` with the `aws logs` invocation, because `lines: []` is the wire shape for “the group exists and is empty”.
* CLI manifest — entry at [`microvms-cli/src/commands/local.rs:193`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L193). Derived from clap introspection and the exit table, so it cannot drift from what the binary accepts; a command with no response-type row fails `microvms-cli/tests/manifest.rs`.
* CLI constants — entry at [`microvms-cli/src/commands/local.rs:216`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L216). Emits `microvms_core::constants::as_json()` verbatim for comparison against the pinned botocore model, with `--emit-json` writing the bare object the drift gate reads.
* CLI dockerfile — entry at [`microvms-cli/src/commands/local.rs:251`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L251). Emits the stanza with both platform traps as comments — the `FROM` that must pair with `baseImageArn`, and the `WORKDIR` the managed al2023 base does not declare.
* CLI cost — entry at [`microvms-cli/src/commands/cost.rs:27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/cost.rs#L27). Renders a report from pinned rates, optionally beside the residency comparison, with unpriced line items kept distinct from zero.
## See also
[Section titled “See also”](#see-also)
* [debugging guide](../insights/debugging-guide.md) — 15 shared source citations
* [business logic](../insights/business-logic.md) — 14 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 14 shared source citations
* [data flow](../architecture/data-flow.md) — 13 shared source citations
* [sequences](../diagrams/behavioral/sequences.md) — 11 shared source citations
# State machines
> Four machines, each declared once as a Rust enum. Three of them are also declared formally — twice over for the VM lifecycle — and the formal declaration is the authority: the…
Four machines, each declared once as a Rust enum. Three of them are also declared formally — twice over for the VM lifecycle — and the formal declaration is the authority: the `model/` crate holds `stateright` models whose properties hold over every interleaving ([`model/src/lib.rs:433-517`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L433-L517), [`model/src/client.rs:546-699`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L546-L699)), and `spec/core.symspec.json` carries a five-variable state model with a machine-readable transition effect per requirement ([`spec/core.symspec.json:995-1041`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L995-L1041)).
The models are ordinary `cargo test` targets in the `agentd-model` crate ([`model/Cargo.toml:2`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L2)), driven by `.checker().spawn_bfs().join().assert_properties()` ([`model/src/lib.rs:528-534`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L528-L534), [`model/src/client.rs:710-716`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L710-L716)) and run by `cargo test --all` ([`mise.toml:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L148)). The Z3 pass over the symspec is a separate task, `--reachability-timeout-ms 5000` against the v5 CLI ([`mise.toml:227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L227)), with the daemon’s own requirements gated by `symspec check spec/agentd.symspec.json --strict` ([`mise.toml:207`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L207)).
Where a machine is mirrored across crates, the mirror is by convention rather than by a cargo dependency — `agentd-model` has no edge to `microvms-core` or to `agentd` ([`model/src/client.rs:58-59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L58-L59)) — so each mirror is named beside its diagram.
## Boot
[Section titled “Boot”](#boot)
The one-shot bootstrap. Two states, and the whole security argument rests on the fact that only the first writer can install a token: the platform’s own `/run` hook arrives from `127.0.0.1`, so it is indistinguishable at the socket level from a request sent by a process inside the MicroVM ([`model/src/lib.rs:11-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L11-L18)).
Entry is `Boot::Uninitialized`, the sole initial state ([`model/src/lib.rs:240`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L240)). Every edge is in the `Action::RunHook` arm ([`model/src/lib.rs:316-332`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L316-L332)):
* `Uninitialized --> Ready` on a first hook. The token and the principal who installed it are both recorded, and the response is `Ok` — [`model/src/lib.rs:320-323`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L320-L323).
* `Ready --> Ready` when the presented token equals the installed one. Answered `Ok`, because the platform may retry its own hook and telling it the VM is broken would fail a launch that is fine — [`model/src/lib.rs:326-328`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L326-L328).
* `Ready --> Ready` when the presented token differs. Answered `Conflict`; nothing is replaced — [`model/src/lib.rs:330`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L330).
A control request arriving while `Uninitialized` is answered `Unavailable`, not `Unauthorized` and never `NotFound`: clients map 404 onto “missing file”, so the wrong code turns a protocol error into a phantom absent artifact — [`model/src/lib.rs:334-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L334-L339). The daemon’s middleware makes the same three-way distinction, with `token_matches` returning `None` for “not bootstrapped” and `Some(false)` for “wrong credential” — [`agentd/src/auth.rs:69-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L69-L80), [`agentd/src/state.rs:245-249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L245-L249).
Mirrors:
* [`agentd/src/state.rs:119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L119) — the daemon stores no state enum. Its bootstrap state is `token: Mutex>>`, read through `is_bootstrapped()` ([`agentd/src/state.rs:237-239`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L237-L239)), so `None` is `Uninitialized` and `Some` is `Ready`. `Bootstrap { Installed, AlreadyIdentical, Conflict }` ([`agentd/src/state.rs:96-106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L96-L106)) is the outcome of an attempted install, not a state field, which is why its variants are this diagram’s edge labels. `AppState::bootstrap` decides all three under the token lock ([`agentd/src/state.rs:202-221`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202-L221)) and `POST /run` maps them to 200/200/409 ([`agentd/src/routes.rs:213-234`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L213-L234)).
* [`spec/agentd.symspec.json:11-114`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/agentd.symspec.json#L11-L114) — four of its six EARS requirements are this machine: install the agent token ([`spec/agentd.symspec.json:89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/agentd.symspec.json#L89)), accept an identical token ([`spec/agentd.symspec.json:21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/agentd.symspec.json#L21)), reject a differing token ([`spec/agentd.symspec.json:72`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/agentd.symspec.json#L72)), and reject a control request while the token is not installed ([`spec/agentd.symspec.json:56`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/agentd.symspec.json#L56)).
Two `always` properties hold over the whole reachable space: `bootstrap is one-shot` (`token_replacements == 0`, [`model/src/lib.rs:446-448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L446-L448)) and `control API is closed before bootstrap` ([`model/src/lib.rs:458-465`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L458-L465)). `attacker never authorized` ([`model/src/lib.rs:443-445`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L443-L445)) is stated unconditionally rather than consulting the config it discriminates, and the model reports both halves of the deployment invariant: held, the attacker never gains authority ([`model/src/lib.rs:527-534`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L527-L534)); broken, `stateright` returns the concrete path by which it does ([`model/src/lib.rs:540-558`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L540-L558)). One-shot survives even a racing in-VM process ([`model/src/lib.rs:562-569`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L562-L569)).
The launch environment travels in the same payload and is installed only on `Installed` ([`agentd/src/state.rs:210`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L210)), under the token lock, so a caller who loses the token cannot win the environment. It is deliberately never the same slot as the token, because the token’s security property is that it stays out of child environments ([`agentd/src/state.rs:128-134`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L128-L134)).
Bootstrap state survives a suspend and resume — measured, not inferred — so `resume` is not an edge of this machine ([`agentd/src/routes.rs:261-290`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L261-L290)).
Defined at: [`model/src/lib.rs:64-69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L64-L69)
## ExecPhase
[Section titled “ExecPhase”](#execphase)
Where one exec sits in its lifecycle. Output is held until the caller acks, which is what makes a retried poll safe ([`model/src/lib.rs:71-72`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L71-L72)).
Entry: `ExecStart(id)` for an unseen id pushes an entry at `Running` with `output_held: true`, `spawns: 1`, `starts: 1` — [`model/src/lib.rs:357-363`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L357-L363).
* `ExecStart(id)` on a *known* id increments `starts` only; it spawns nothing and touches no other field. That is the idempotency contract — [`model/src/lib.rs:350-366`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L350-L366). The daemon decides it under the registry lock before the spawn, so two concurrent retries cannot both find the slot empty — [`agentd/src/exec.rs:363-377`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L363-L377).
* `ExecPoll(id)` touches no field — [`model/src/lib.rs:367-369`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L367-L369). Read-only is a property of the step rather than of any reachable state, so it is asserted against the transition function directly ([`model/src/lib.rs:574-606`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L574-L606)) and the daemon handler carries the same rule ([`agentd/src/exec.rs:402-434`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L402-L434)).
* `ChildExit(id)` applies only from `Running`; from any other phase `next_state` returns `None` — [`model/src/lib.rs:403-411`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L403-L411). On the daemon the waiter sets `shared.terminal` *before* `shared.result`, so a stream that sees the finish immediately finds the terminal marker present — [`agentd/src/exec.rs:1173-1183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1173-L1183).
* `ExecAck(id)` applies only from `Exited` and clears `output_held`. From any other phase the response is `Conflict`, not a silent success that would drop output still being written — [`model/src/lib.rs:370-379`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L370-L379). The daemon answers 409 `ERROR_STILL_RUNNING` when the result slot is empty and `acked_at` is unset, and 409 `ERROR_ALREADY_ACKED` on a second ack; `acked_at` is marked while the slot lock is still held so a concurrent duplicate cannot misreport an acked exec as running — [`agentd/src/exec.rs:837-886`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L837-L886).
* `Collect` retains only entries whose phase is not `Acked`, so `Acked` is the one phase an entry can be collected from — [`model/src/lib.rs:380-398`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L380-L398). TTL collection on the daemon keeps any entry whose `acked_at` is `None`, however old, because collecting it would destroy output the caller never read — [`agentd/src/exec.rs:951-962`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L951-L962).
`kill` signals the whole process group and leaves the phase alone; the phase moves only when the child actually exits, so it is not a transition of this machine — [`agentd/src/exec.rs:905-940`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L905-L940).
Mirrors:
* [`protocol/src/exec.rs:24-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L24-L31) — `Phase { Running, Exited, Acked }`, doc comment “Mirrors `ExecPhase` in the model crate” ([`protocol/src/exec.rs:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L16)). `rename_all = "snake_case"` ([`:23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L23)) puts `running` / `exited` / `acked` on the wire, spelled once in `as_str` ([`:47-53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L47-L53)) with the closed set in `ALL` ([`:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L40)) so a binding publishing the list reads it from the type.
* [`agentd/src/exec.rs:1193-1201`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1193-L1201) — `phase_of(acked, finished)`. The daemon stores no phase field; it derives one from `acked_at.is_some()` and `result.is_some()`, asserted exhaustively at [`agentd/src/exec.rs:2375-2377`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L2375-L2377).
Three `always` properties hold over the whole reachable space: `output is never released before ack` ([`model/src/lib.rs:466-472`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L466-L472)), `a retried start never spawns twice` (`spawns == 1`, [`model/src/lib.rs:473-475`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L473-L475)), and `one exec entry per id` ([`model/src/lib.rs:476-481`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L476-L481)). The first is audited against itself rather than asserted: the collect predicate flags any entry it would remove while `output_held` still holds, and acking is the only thing that releases output, so a collected entry with held output is exactly an exec destroyed without its caller’s ack ([`model/src/lib.rs:380-395`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L380-L395)). Coverage properties confirm the checker reached `Acked` and a retried start ([`model/src/lib.rs:504-509`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L504-L509)).
Defined at: [`model/src/lib.rs:74-81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L74-L81)
## Lifecycle
[Section titled “Lifecycle”](#lifecycle)
One MicroVM’s whole life, as the client tracks it. Six states and no others, which is the point of the enum: a lifecycle held as a `String` would let `"RUNNING "` and `"Running"` both exist, and every guard would have to decide which it meant ([`microvms-core/src/sandbox.rs:91-95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L91-L95)). The state is a private field, and the five `Sandbox` methods are the only writers.
Entry is `Lifecycle::Pending` ([`microvms-core/src/sandbox.rs:486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L486)), matching the symspec’s `initial` ([`spec/core.symspec.json:996`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L996)) and the model’s sole init state ([`model/src/client.rs:287`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L287)).
Edge labels below are the model’s `Action` variants ([`model/src/client.rs:116-142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L116-L142)), which is the one vocabulary all three declarations share. Each row gives the symspec key, the symspec’s `stateEffect`, the model arm, and the client site:
* `LaunchAccepted` · `Pending --> Pending` · STATE-1 ([`spec/core.symspec.json:690`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L690)), `when vm_state = PENDING: image_exists := true` ([`:698`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L698)) · [`model/src/client.rs:374-384`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L374-L384) · [`microvms-core/src/sandbox.rs:696-706`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L696-L706). The lifecycle is set after the wire call returns, because acceptance *is* the call succeeding.
* `HookSucceeded` · `Pending --> Running` · STATE-2 ([`:371`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L371)), `... vm_state := RUNNING, token_installed := true, bootstrap_count := bootstrap_count + 1` ([`:379`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L379)) · [`model/src/client.rs:388-409`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L388-L409) · [`microvms-core/src/sandbox.rs:708-726`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L708-L726). This is the one place `bootstrap_count` increments (STATE-3, [`:881`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L881)).
* `SuspendRequested` · `Running --> Suspending` · STATE-4 ([`:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L103)), `when vm_state = RUNNING: vm_state := SUSPENDING` ([`:112`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L112)) · [`model/src/client.rs:427-443`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L427-L443) · [`microvms-core/src/sandbox.rs:769-778`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L769-L778). The assignment follows the call for the same reason: moving first would leave a throttled call stuck in a state neither suspend nor resume accepts, bricking the handle over one bad request.
* `SuspendComplete` · `Suspending --> Suspended` · STATE-6 ([`:550`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L550)), `when vm_state = SUSPENDING: vm_state := SUSPENDED` ([`:558`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L558)) · [`model/src/client.rs:446-453`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L446-L453) · [`microvms-core/src/sandbox.rs:794-795`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L794-L795).
* `ResumeRequested` + `ResumeComplete` · `Suspended --> Running` · STATE-7 ([`:668`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L668)), `when vm_state = SUSPENDED: vm_state := RUNNING` ([`:677`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L677)) · [`model/src/client.rs:456-501`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L456-L501) · [`microvms-core/src/sandbox.rs:859-883`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L859-L883). Nothing is re-delivered: no payload, no token, no bootstrap, because the in-memory token survived the freeze and re-delivering it would hit the daemon’s one-shot bootstrap and be refused ([`microvms-core/src/sandbox.rs:818-822`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L818-L822)). The session rebinds to the endpoint the service just reported, which drops the cached proxy token (STATE-8, [`:199`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L199)).
* `TerminateRequested` · `Pending`/`Running`/`Suspended` `--> Terminating` · STATE-9 ([`:571`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L571)), `when vm_state = PENDING or vm_state = RUNNING or vm_state = SUSPENDED: vm_state := TERMINATING, was_terminated := true` ([`:579`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L579)) · [`model/src/client.rs:504-511`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L504-L511) · [`microvms-core/src/sandbox.rs:944-949`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L944-L949). Recorded before the call, so a terminate whose call fails still marks the VM as one this client asked to destroy.
* `TerminateComplete` · `Terminating --> Terminated` · STATE-10 ([`:803`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L803)), `when vm_state = TERMINATING: vm_state := TERMINATED` ([`:811`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L811)) · [`model/src/client.rs:514-521`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L514-L521) · [`microvms-core/src/sandbox.rs:967-976`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L967-L976). Reached only when the optional `wait_for_state(&["TERMINATED"])` succeeds; when the wait fails the lifecycle stays at `Terminating` honestly, because the platform accepted the terminate and the VM is on its way out ([`microvms-core/src/sandbox.rs:977-982`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L977-L982)).
One edge exists in the client with no matching `stateEffect`: the suspend wait settles on `SUSPENDED` **or** `TERMINATED`, and both are states this client asked for. A VM the launch-time `idlePolicy` killed mid-suspension lands directly in `Terminated` and also sets `was_terminated`, which is what then stops a resume from being offered — [`microvms-core/src/sandbox.rs:790-809`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L790-L809). The symspec omits `SUSPENDING` as a terminate source, and that omission is correct rather than a gap: `suspend(&mut self)` ([`microvms-core/src/sandbox.rs:755`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L755)) holds the exclusive borrow across its own wait, so no caller can invoke `terminate(&mut self)` ([`:935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L935)) while the lifecycle sits in `Suspending`. `Suspending` is transient within one call, never a resting state a caller can act from.
Every guard refuses before any control-plane call is made, and the zero-call refusal is the assertion rather than the resulting state:
* `run` twice is refused on `bootstrap_count > 0 || microvm.is_some()` (STATE-3) — [`microvms-core/src/sandbox.rs:649-661`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L649-L661).
* `suspend` is refused unless the lifecycle is `Running` (STATE-5, [`spec/core.symspec.json:294`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L294), constraint at [`:302`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L302)) — [`microvms-core/src/sandbox.rs:758-767`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L758-L767).
* `resume` is refused when `was_terminated` or the lifecycle is `Terminated` (STATE-11, [`:448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L448), constraint at [`:456`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L456)) — [`microvms-core/src/sandbox.rs:840-848`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L840-L848) — and unless the lifecycle is `Suspended` (STATE-7) — [`:849-854`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L849-L854).
* `resume` past the launch-time `suspendedDurationSeconds` window is refused with `ErrorKind::WindowClosed` (STATE-12, [`:487`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L487)) — [`microvms-core/src/sandbox.rs:857`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L857), [`:902-926`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L902-L926). An absent window is *not* a closed one: with either the window or the stamp missing the check passes, because that is the attach path where this sandbox did not send the launch, and guessing a default would refuse a resume the service would honour ([`microvms-core/src/sandbox.rs:903-907`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L903-L907); see `.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md`).
* `suspended_at` is cleared on a successful resume, so the next cycle’s window is measured from the next suspend rather than accumulating every suspension into one total — [`microvms-core/src/sandbox.rs:884-887`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L884-L887).
`Lifecycle::as_str` maps each state to the uppercase name the service uses, which is also what an error message prints — [`microvms-core/src/sandbox.rs:112-123`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L112-L123). `Lifecycle::is_live` is true for `Pending`, `Running`, `Suspending`, `Suspended`, and is read only by the `Drop` warning about a VM still billing — [`microvms-core/src/sandbox.rs:125-131`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L125-L131), [`microvms-core/src/sandbox.rs:1060-1077`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1060-L1077).
Mirrors:
* [`spec/core.symspec.json:1004-1011`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L1004-L1011) — `vm_state`, an enum whose domain is exactly `PENDING`, `RUNNING`, `SUSPENDING`, `SUSPENDED`, `TERMINATING`, `TERMINATED`, beside the four other variables the `Sandbox` carries: `token_installed`, `image_exists`, `was_terminated`, `bootstrap_count` ([`:1012-1040`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/spec/core.symspec.json#L1012-L1040)). The `STATE-1`..`STATE-12` keys cited above are EARS sentences in the same document.
* [`model/src/client.rs:61-74`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L61-L74) — `VmState`, “Mirrors `microvms_core::sandbox::Lifecycle` by convention rather than by dependency” ([`:58-59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L58-L59)). Its transitions are driven by `Action` ([`:116-142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L116-L142)), each answered `Issued`, `RefusedLocally`, or `Ignored` ([`model/src/client.rs:99-108`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L99-L108)).
The three invariants Z3 proves over the symspec are restated as `stateright` `always` properties over every interleaving of the model’s actions: `bootstrap happens at most once` ([`model/src/client.rs:554-556`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L554-L556)), `no suspend call outside RUNNING` ([`model/src/client.rs:557-566`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L557-L566)), and `a terminated VM never reaches RUNNING` ([`model/src/client.rs:567-569`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L567-L569)). The second is asserted against the counter `suspends_outside_running` rather than against the resulting state, because a suspend from `Running` and one from `Suspended` both land in `Suspending`, so nothing in the post-state distinguishes them — the first attempt at this property passed while a twelve-step counterexample existed ([`model/src/client.rs:558-565`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L558-L565)). Wire-call counts are state variables for the same reason ([`model/src/client.rs:23-34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L23-L34)): “a resume after terminate is rejected” is satisfied by a client that calls, fails, and burns a poll timeout, so the property that matters is that no resume call ever fires once `was_terminated` holds ([`model/src/client.rs:584-589`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L584-L589)).
Model checking found a defect behind the third invariant that code reading had missed. A resume issued legally from `Suspended`, then a terminate, then the resume’s completion arriving late, put a `was_terminated` VM back in `Running` — STATE-11 broken by an interleaving no state-only gate catches. The fix makes a completion apply only while a resume is still in flight *and* the state is still `Suspended`, so the terminate wins, which is what the client does: `terminate` clears the session and the lifecycle before anything else — [`model/src/client.rs:175-184`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L175-L184), [`model/src/client.rs:483-501`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L483-L501), [`model/src/client.rs:509`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L509).
Each guard is proved falsifiable rather than merely green. Under `Config::guards_skipped` ([`model/src/client.rs:247-252`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L247-L252)) the client issues a suspend outside RUNNING, a resume after a terminate, and a resume with the window closed, and `stateright` hands back each path — [`model/src/client.rs:726-745`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L726-L745), [`:750-763`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L750-L763), [`:768-774`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L768-L774). Every `always` property has a `sometimes` property beside it ([`model/src/client.rs:647-697`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L647-L697)) so none can pass over a space that never reached the interesting state.
Defined at: [`microvms-core/src/sandbox.rs:97-110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L97-L110)
## StreamState
[Section titled “StreamState”](#streamstate)
The SSE output stream’s cursor machine. It tracks where an attach is, how many times it has dropped, and the byte offset a resume would ask for. Written as a generator over an explicit state machine rather than a hand-rolled `Stream` impl, because the reconnect logic is a loop with an `await` in the middle and expressing that as a `poll_next` would mean storing the in-flight attach as a pinned field — where a self-referential-future bug lives ([`microvms-core/src/session/exec.rs:290-294`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L290-L294)). The enum is private, so it has no mirror.
Entry is `Reconnect { cursor: options.offset, attempts: 0 }`, seeded identically by both drivers: `stream_with` at [`microvms-core/src/session/exec.rs:299-303`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L299-L303) and `for_each_event_async` at [`:412-415`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L412-L415). `for_each_event` ([`:347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L347)) delegates to the async form ([`:359`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L359)), so both consumers run one step function, `advance` — [`microvms-core/src/session/exec.rs:460-588`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L460-L588). `attempts` is zero for the first attach, which is why the backoff and the max-reconnect check are both skipped there ([`:739-740`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L739-L740)).
Out of `Reconnect`:
* a successful `attach` moves to `Attached` carrying the same cursor and attempt count — [`:491-498`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L491-L498).
* a retryable `attach` failure re-enters `Reconnect` with `attempts + 1`. A cut connection or a failed token mint says nothing about the exec, which is still running server-side — [`:499-507`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L499-L507).
* a fatal failure goes to `Done` with the error, because reconnecting can never succeed. A 404 on a collected entry is the case that matters — [`:508-511`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L508-L511).
* `attempts > options.max_reconnects` goes to `Done` with a retryable error naming the last good offset — [`:474-486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L474-L486).
* `attempts > 0` with `reconnect` off ends the stream without stepping the machine — [`:470-473`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L470-L473).
Out of `Attached`, on the next decoded `ExecEvent`:
* `Output` stays `Attached` and advances the cursor to `offset + data.len()`, only past bytes actually handed over — [`:519-539`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L519-L539).
* `Gap` advances the cursor to `to` unconditionally, so a reconnect does not ask for the evicted range again and receive the same gap forever. It then stays `Attached`, or goes to `Done` with a `WireKind::OutputGap` error when `options.error_on_gap` is set — [`:540-561`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L540-L561). `from` is inclusive and `to` exclusive, which is why `to` is where a cursor resumes ([`microvms-core/src/session/sse.rs:248-252`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L248-L252)).
* `Exit` goes to `Done`. A finished command always delivers this event, and its absence is the only thing distinguishing a cut connection from a finished command — the byte sequences are otherwise identical — [`:563-567`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L563-L567), [`microvms-core/src/session/sse.rs:253-255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L253-L255).
* a body that ends with no `Exit` event re-enters `Reconnect` with `attempts + 1`, or ends the stream when `reconnect` is off — [`:568-577`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L568-L577).
* a retryable read error re-enters `Reconnect`; a fatal one goes to `Done` — [`:578-584`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L578-L584). A parse failure is `ErrorKind::Protocol`, and `Error::retryable` is true only for `ErrorKind::Retryable` ([`microvms-core/src/error.rs:116-118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L116-L118)), so a proxy answering an error page is not retried `max_reconnects` times, refilling the buffer each pass — [`microvms-core/src/session/exec.rs:726-731`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L726-L731).
`Done` yields nothing and ends the stream — [`:468`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L468). `StreamState::cursor()` returns `None` for `Done` rather than a number: `Done` is reached from three different places, so any value invented there could shadow the last real cursor the caller already holds — [`:753-767`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L753-L767).
`for_each_event_async` reports which `Done` path was taken as `EndReason` ([`microvms-core/src/session/exec.rs:143-154`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L143-L154)). `EndReason` is a return classification, not a state the machine occupies, so it gets no diagram of its own. It is `Exited` when the terminal `Exit` event was delivered ([`:448-452`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L448-L452)), `Stopped` when the callback answered `ControlFlow::Break` ([`:442-447`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L442-L447)), and `Cut` when the body ended with no `Exit` event and reconnecting was refused ([`:420-428`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L420-L428)) — where the command’s outcome is unknown rather than zero, and a caller reporting success would pass a CI step on evidence it never received ([`:150-153`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L150-L153)). The returned cursor is read off the machine through `next.cursor()` rather than recomputed from the events, so a caller that resumes holds one cursor and not a second one that would agree until a gap arrived — [`:429-434`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L429-L434).
Defined at: [`microvms-core/src/session/exec.rs:738-751`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L738-L751)
## See also
[Section titled “See also”](#see-also)
* [business logic](../insights/business-logic.md) — 9 shared source citations
* [contract map](../insights/contract-map.md) — 8 shared source citations
* [debugging guide](../insights/debugging-guide.md) — 8 shared source citations
* [impact analysis](../insights/impact-analysis.md) — 8 shared source citations
* [processes](processes.md) — 6 shared source citations
# Plan: full live coverage through the `microvm` CLI
> **Implemented. This document is kept as history because the reasoning still applies; the numbers below are outdated.**
>
> All five waves shipped and the conformance flip landed. `conformance/run_rs.py` expresses every named check with none skipped. It no longer carries an `UNSUPPORTED` table or an `unsupported()` helper. Instead, the suite computes its own coverage line from what ran (`expressed = passed + failed`, denominator plus skips), so no document has to keep a quoted figure in step. As a result, the “38 of 72” below describes the state on the day the plan was written and is no longer accurate. The numerator is now complete. The denominator settled at **75** rather than 72, because two of the original 38 turned out to be weak readings off the launch envelope and were split into real checks (see `CHANGELOG.md`); it has since grown to **77** as the suite gained checks, which is why the paragraph below says to read the figure off a run.
>
> Read this for *why* each surface exists and what the constraints were. For what the suite covers today, run it, or read the summary block at the end of `conformance/run_rs.py`. That block derives the coverage number from the checks that ran instead of quoting a stored figure, so it stays accurate.
**Goal (as written, before implementation).** `conformance/run_rs.py` currently expresses 38 of 72 named checks and prints 34 as SKIP, each naming the missing subcommand. This plan adds the five CLI surfaces that close every SKIP, so the Rust suite alone gives live AWS coverage of everything the retired Python oracle covered. The daemon needs no changes. Every capability already exists in `microvms-core` (`session/files.rs`, `session/exec.rs` streaming and stdin, `protocol::Health`) and is exercised today by the fake-backed and turmoil tiers. The only missing piece is the CLI surface for each capability. The live rounds have twice found bugs in paths that had never been driven end-to-end against real AWS, which is why these paths need live coverage.
## The 34 SKIPs, grouped by the surface that closes them
[Section titled “The 34 SKIPs, grouped by the surface that closes them”](#the-34-skips-grouped-by-the-surface-that-closes-them)
| New surface | SKIPs closed | The checks, by name (from run\_rs.py’s UNSUPPORTED table) |
| ----------------------------------------------------- | ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvm cp` (+ `--tar`) | 13 | single file write / read / absent-is-404; tree created for the round trip; tar download / upload; symlink survived as symlink / still resolves; nothing escaped the extraction root; 4 hostile archives refused |
| `microvm exec --stream` | 5 | SSE reached us through the proxy; output complete and ordered; no gap for a small stream; terminal exit event carries the real code; exec survives being streamed and stays pollable |
| stdin (`exec --stdin` + `microvm stdin`) | 5 | write accepted; close accepted; child exits once stdin closes; round trip through the child; refusal when the command never asked |
| exec identity (`--exec-id`, `--poll`) + `microvm ack` | 6 | ack accepted; double-ack 409; unknown id 404; retried start accepted / spawned no second child; pre-suspend exec record survives resume |
| `microvm health` | 5 | identity repair completed every step / actually ran; 8 MiB cap trio (noisy exit 0, `truncated` flag, daemon survived — survival is the health probe) |
Totals reconcile: 13+5+5+6+5 = 34 (29 UNSUPPORTED entries + 4 hostile archives + the pre-suspend exec record noted inline at run\_rs.py:1034).
## Design constraints that carry over (not negotiable)
[Section titled “Design constraints that carry over (not negotiable)”](#design-constraints-that-carry-over-not-negotiable)
* **CLI-2/CLI-5 hold.** Every new subcommand goes through `microvms-core` only; no option may accept a value core rejects. `cp --tar` hands core an archive PATH, and core’s tar handling stays the only extractor. The hostile-archive checks assert core’s refusal through the CLI’s `data.kind`, so the CLI must not pre-validate archives; pre-validation would test the CLI’s copy of the guard rather than core’s.
* **The manifest is generated**, so each subcommand lands with its options, exit codes, and envelope keys appearing in `microvm manifest` for free. The manifest cross-check test enforces this structurally.
* **One envelope on stdout.** `exec --stream` is the hard case because its stream chunks are output, not progress reporting. Piped/`--json` mode emits one NDJSON line per event on stdout and the final envelope last; the envelope’s `data` carries the event count and exit. The manifest documents this as `responseType: exec.stream`, a deliberate, named exception to the one-envelope rule. TTY mode instead renders live via ratatui. The envelope-purity test gains a stream variant: every stdout line before the last parses as an event, and the last parses as the envelope.
* **Guard proofs per surface.** Each check that goes SKIP→live keeps run.py’s original falsification (e.g. the byte-scan for tar confinement, the wc -l tick-count for stdin EOF). To prove each guard still fires, break it deliberately, confirm the check fails, and then restore it.
* **agentToken handling.** The new attach-shaped commands (`cp`, `stdin`, `ack`, `health`, `exec --poll`) take the same `--endpoint/--agent-token/ --microvm-id` triple `exec` takes today, reusing `seam::attach_session`.
## Waves
[Section titled “Waves”](#waves)
**W1 — `microvm health` + exec identity (`--exec-id`, `--poll`) + `microvm ack`.** This is the smallest surface, it unblocks 9 checks, and `health` is needed by W3’s cap trio. This wave is CLI-only: core already exposes `Session::health`, `Session::exec(id)`, and `ExecHandle::{poll,ack}`, so `run_sync` needs no sibling path in core. Exit-code note: `--poll` on a running exec returns OK with `phase: running`, because polling a still-running exec is not a failure.
**W2 — `microvm cp`.** `cp vm:` / `cp vm: `, `--tar` for directory round trips (pack local→upload, download→unpack via core’s confined extractor), `--mode` for permissions. This wave closes 11 checks. The four hostile archives are driven by handing the CLI a pre-built malicious tar file; the expected failure is core’s ProtocolError surfacing as `data.kind: protocol_error` with exit 5.
**W3 — `microvm exec --stream`.** This wave emits NDJSON event lines plus the final envelope, as described above. It reuses core’s cursor-reconnect stream and exposes `--from-offset` for the resume-at-cursor check. It closes 7 checks, including the cap trio (with W1’s health).
**W4 — stdin: `microvm exec --stdin` (streams local stdin to the child, EOF on close) and `microvm stdin [--eof]` for the detached case.** This wave closes 7 checks, including the opt-in refusal: running `stdin` against an exec started without `--stdin` must surface 410/StdinClosed as `data.kind`.
**W5 — conformance flip + live proof.** Delete each closed entry from `UNSUPPORTED` in run\_rs.py and add the check bodies (the assertions are copyable from git history: `conformance/run.py` at commit `c4d396e^`). The suite’s summary line changes from “38 of 72” to “72 of 72”. Run `mise run live` twice. The first run shakes out the new paths; the live tier has caught a bug in every new transport path so far, so plan for one fix round. The second, green run records the marker. Update mise task description, README, and CHANGELOG coverage notes.
## Effort and order-of-magnitude cost
[Section titled “Effort and order-of-magnitude cost”](#effort-and-order-of-magnitude-cost)
Each wave is a bounded Act task with the existing packet discipline (W1/W3 small, W2/W4 medium). Live proof runs cost what today’s tier costs — roughly two short-lived 1 GiB MicroVMs per run, at cents per round, plus one extra shake-out round. Nothing here is speculative. Every check’s assertion text and falsification already exists in git history, and every core capability is already tested locally. The remaining work is the five CLI surfaces and the conformance flip.
## What this plan does not do
[Section titled “What this plan does not do”](#what-this-plan-does-not-do)
* It makes no daemon changes, adds no new wire routes, and changes nothing in the protocol crate.
* It adds no `microvm shell` or interactive TTY into the VM, because the platform’s shell-auth path stays unreachable (TRAP-11).
* It adds no parallel-exec orchestration or watch modes, because the suite needs determinism.
# Components
## Legend
[Section titled “Legend”](#legend)
| Node or edge | Citations |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CoreSeam` | trait [`microvms-cli/src/seam.rs:136`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L136); methods [`microvms-cli/src/seam.rs:138`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L138), [`:141`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L141), [`:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L148), [`:172`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L172); `AwsSeam` impl [`:179`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L179), [`:183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L183), [`:201`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L201), [`:225`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L225) |
| `Sandbox` | struct [`microvms-core/src/sandbox.rs:422`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L422); methods [`:551`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551), [`:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648), [`:755`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L755), [`:837`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L837), [`:935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L935) |
| `ControlPlane` | struct [`microvms-core/src/control/mod.rs:160`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L160); methods [`microvms-core/src/control/image.rs:157`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L157), [`microvms-core/src/control/microvm.rs:356`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L356), [`:435`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L435), [`:563`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L563), [`:581`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L581) |
| `Session` | struct [`microvms-core/src/session/mod.rs:184`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L184); methods [`:329`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L329), [`:342`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L342), [`:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L380), [`:408`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L408), [`:444`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L444) |
| `ExecHandle` | struct [`microvms-core/src/session/exec.rs:213`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L213); methods [`:228`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L228), [`:248`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L248), [`:285`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L285), [`:624`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L624), [`:654`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L654) |
| `Routes` | module of free functions, not a type: [`agentd/src/routes.rs:36`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L36), [`:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110), [`:371`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L371), [`:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178), [`:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314) |
| `AppState` | struct [`agentd/src/state.rs:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L110); methods [`:202`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202), [`:245`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L245), [`:257`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L257), [`:176`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L176), [`:183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L183) |
| `Confined` | struct [`agentd/src/fs.rs:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L297); methods [`:350`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L350), [`:416`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L416), [`:428`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L428), [`:448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L448), [`:535`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L535) |
| `CoreSeam --> ControlPlane` | [`microvms-cli/src/seam.rs:138`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L138), impl [`:179`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L179) |
| `CoreSeam --> Sandbox` | [`microvms-cli/src/seam.rs:141`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L141), impl [`:183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L183) |
| `CoreSeam --> Session` | [`microvms-cli/src/seam.rs:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L148), impl [`:201`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L201) |
| `Sandbox --> ControlPlane` | [`microvms-core/src/sandbox.rs:65-67`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L65-L67), [`:553`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L553), [`:696`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L696), [`:773`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L773), [`:859`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L859), [`:951`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L951) |
| `Sandbox --> Session` | [`microvms-core/src/sandbox.rs:70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L70), [`:535`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L535), [`:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648) |
| `Session --> ExecHandle` | [`microvms-core/src/session/mod.rs:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L380), [`:403`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L403) |
| `Session ..> Routes` | [`microvms-core/src/session/mod.rs:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L331), [`:382`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L382); [`microvms-core/src/session/files.rs:45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L45), [`:52`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L52) |
| `ExecHandle ..> Routes` | [`microvms-core/src/session/exec.rs:233`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L233), [`:592`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L592), [`:659`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L659) |
| `Routes --> AppState` | [`agentd/src/routes.rs:36`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L36) |
| `Routes --> Confined` | [`agentd/src/routes.rs:132-135`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L132-L135), [`agentd/src/fs.rs:1433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1433), [`:1480`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1480), [`:631`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L631) |
| `..>` dashed | the HTTP wire, not a crate dependency: the shared contract is the `protocol` crate, re-exported at [`microvms-core/src/lib.rs:77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L77) and [`agentd/src/routes.rs:18-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L18-L20), and the permitted directions are asserted by `microvms-cli/tests/dependency_direction.rs` |
| `+` prefix | the class-diagram marker for a listed member, not a Rust visibility claim: `Confined` and its methods are crate-private ([`agentd/src/fs.rs:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L297), [`:350`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L350)), as are `routes::run_hook` and `routes::health` ([`agentd/src/routes.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178), [`:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314)) |
## See also
[Section titled “See also”](#see-also)
* [impact analysis](../../insights/impact-analysis.md) — 10 shared source citations
* [processes](../../behavior/processes.md) — 9 shared source citations
* [business logic](../../insights/business-logic.md) — 9 shared source citations
* [sequences](../behavioral/sequences.md) — 8 shared source citations
* [data flow](../../architecture/data-flow.md) — 7 shared source citations
# Sequences
> Three processes that cross the client/daemon HTTP boundary. Each participant is one module or one external actor; each edge is one call site.
Three processes that cross the client/daemon HTTP boundary. Each participant is one module or one external actor; each edge is one call site.
## Exec start, stream, and ack
[Section titled “Exec start, stream, and ack”](#exec-start-stream-and-ack)
Participants:
* `microvm CLI` — the `exec`, `exec --stream`, and `ack` subcommands ([`microvms-cli/src/commands/attached.rs:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L103), [`microvms-cli/src/commands/attached.rs:240`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L240), [`microvms-cli/src/commands/attached.rs:609`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L609)).
* `core session` — `Session` plus `ExecHandle`, banded because they share one module ([`microvms-core/src/session/mod.rs:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L380), [`microvms-core/src/session/exec.rs:213`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L213)).
* `ProxyAuth` — the proxy-token cache whose mint sits inside the request path ([`microvms-core/src/session/proxy.rs:432`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L432), [`microvms-core/src/session/mod.rs:88`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L88)).
* `agentd exec routes` — `start`, `stream`, `ack` ([`agentd/src/exec.rs:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L331), [`agentd/src/exec.rs:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L455), [`agentd/src/exec.rs:831`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L831)).
* `Shared ring` — the replay ring plus the broadcast channel, keyed by exec id ([`agentd/src/exec.rs:220`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L220)).
* `child pgroup` — the spawned process group ([`agentd/src/exec.rs:1113`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1113)).
Edges in order:
1. `run(argv)` — [`microvms-cli/src/commands/attached.rs:152`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L152).
2. `headers()` — the mint runs inside `Transport::headers`, so every request re-checks freshness ([`microvms-core/src/session/mod.rs:92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L92), [`microvms-core/src/session/mod.rs:115`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L115)).
3. `POST exec/start` — [`microvms-core/src/session/mod.rs:382`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L382); the handle is built from the id the daemon confirmed ([`microvms-core/src/session/mod.rs:394`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L394)).
4. `spawn pgid` — the pgid is captured while `Child::id()` still answers ([`agentd/src/exec.rs:1113`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1113), [`agentd/src/exec.rs:1119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1119)).
5. `register entry` — the registry insert makes the id addressable ([`agentd/src/exec.rs:1141`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1141)).
6. `200 running` — [`agentd/src/exec.rs:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L380); a retried start returns the same 200 without a second child ([`agentd/src/exec.rs:366`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L366)).
7. `publish bytes` — `Capped::pump` into `Shared::publish`, which appends to the ring and fans out live under one lock ([`agentd/src/exec.rs:1366`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1366), [`agentd/src/exec.rs:255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L255)).
8. `GET ?offset=N` — `ExecHandle::attach` builds `/v1/exec/{id}/stream?offset=`, mints its own headers because the streaming path bypasses `Transport::request`, and is re-entered per reconnect ([`microvms-core/src/session/exec.rs:592`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L592), [`microvms-core/src/session/exec.rs:600`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L600), [`microvms-core/src/session/exec.rs:491`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L491)).
9. `attach(offset)` — subscribe-before-snapshot, enforced by one lock so the unsafe order is not expressible from the handler ([`agentd/src/exec.rs:474`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L474), [`agentd/src/exec.rs:293`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L293)).
10. `output events` — base64 `output` frames carrying the offset of their first byte ([`agentd/src/exec.rs:642`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L642)); a lagged or evicted range comes through as a typed `gap` ([`agentd/src/exec.rs:656`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L656)).
11. `bytes + cursor` — the cursor advances only past bytes handed over, and past a gap’s `to` ([`microvms-core/src/session/exec.rs:526`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L526), [`microvms-core/src/session/exec.rs:543`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L543)); the CLI writes an NDJSON line plus the raw bytes ([`microvms-cli/src/commands/attached.rs:268`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L268)).
12. `exit event` — the terminal marker is written before the result slot, so a stream that sees `Finished` always finds an exit event ([`agentd/src/exec.rs:535`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L535), [`agentd/src/exec.rs:1182`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1182)).
13. `POST exec/ack` — [`microvms-core/src/session/exec.rs:654`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L654); `wait_and_ack` returns the ack’s result rather than a post-ack poll ([`microvms-core/src/session/exec.rs:687`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L687)).
14. `released output` — the result slot is taken once and `acked_at` is set while the slot lock is still held ([`agentd/src/exec.rs:863`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L863), [`agentd/src/exec.rs:867`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L867)).
Stdin is a separate request, never multiplexed onto this connection ([`microvms-core/src/session/exec.rs:624`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L624), [`agentd/src/exec.rs:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L682)).
## Tar upload and extraction
[Section titled “Tar upload and extraction”](#tar-upload-and-extraction)
Participants:
* `microvm cp --tar` — resolves direction from the `vm:` prefix and inspects no archive ([`microvms-cli/src/commands/attached.rs:805`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L805), [`microvms-cli/src/commands/attached.rs:809`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L809)).
* `Transport` — `files::upload_tar` plus the shared send path ([`microvms-core/src/session/files.rs:98`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L98), [`microvms-core/src/session/mod.rs:106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L106)).
* `agentd fs routes` — `write_tar` ([`agentd/src/fs.rs:1433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1433)).
* `disk Guard` — the reserve-aware probe, the body spool, and the pacer ([`agentd/src/fs.rs:1454`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1454), [`agentd/src/fs.rs:872`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L872), [`agentd/src/disk.rs:170`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L170)).
* `Confined` — the `openat2`-based extractor, the one confined write path ([`agentd/src/fs.rs:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L297), [`agentd/src/fs.rs:621`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L621)).
* `VM filesystem` — the extraction root inside the guest.
Edges in order:
1. `upload_tar()` — [`microvms-cli/src/commands/attached.rs:829`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L829).
2. `PUT /v1/fs/tar` — content type `application/x-tar`; the client does not inspect the archive, so the daemon’s extractor stays the only implementation of the member rules ([`microvms-core/src/session/files.rs:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L103), [`microvms-core/src/session/files.rs:94`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/files.rs#L94)).
3. `preflight(root)` — run against the extraction root before the body is spooled, so an upload aimed at a full filesystem is refused without spending the wire time ([`agentd/src/fs.rs:1459`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1459)).
4. `disk reading` — a reading below the reserve becomes 507 naming the path ([`agentd/src/fs.rs:1460`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1460), [`agentd/src/fs.rs:106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L106)).
5. `spool body` — the archive lands in full before a single member is extracted ([`agentd/src/fs.rs:1463`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1463), [`agentd/src/fs.rs:872`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L872)).
6. `spool file` — spool pressure and a truncated body are distinct outcomes, 507 and 400 ([`agentd/src/fs.rs:1469`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1469), [`agentd/src/fs.rs:1475`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1475)).
7. `extract_into` — inside `spawn_blocking`, because `tar`’s reader is blocking ([`agentd/src/fs.rs:1479`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1479), [`agentd/src/fs.rs:621`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L621)).
8. `openat root` — one confined root held for the whole extraction, so a component that turns out to be a symlink stops the write instead of redirecting it ([`agentd/src/fs.rs:631`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L631), [`agentd/src/fs.rs:350`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L350)).
9. `create member` — `resolve_member` refuses an escaping path and a non-directory naming the root; device and fifo members are refused; an absolute link target is refused ([`agentd/src/fs.rs:679`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L679), [`agentd/src/fs.rs:704`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L704), [`agentd/src/fs.rs:726`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L726), [`agentd/src/fs.rs:783`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L783)).
10. `pace bytes` — checked after each member lands, and extraction is not transactional by design ([`agentd/src/fs.rs:803`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L803)).
11. `deferred modes` — replayed deepest-first after all content has landed, so a directory packed `0o500` does not block the writes into it ([`agentd/src/fs.rs:810`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L810), [`agentd/src/fs.rs:825`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L825)).
12. `members count` — [`agentd/src/fs.rs:1485`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1485).
13. `204 No Content` — [`agentd/src/fs.rs:1487`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1487).
14. `bytes uploaded` — [`microvms-cli/src/commands/attached.rs:835`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L835).
## Daemon bootstrap through the run hook
[Section titled “Daemon bootstrap through the run hook”](#daemon-bootstrap-through-the-run-hook)
Participants:
* `Sandbox` — the client lifecycle object outside the VM ([`microvms-core/src/sandbox.rs:675`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L675)).
* `ControlPlane` — the signed AWS client ([`microvms-core/src/control/microvm.rs:356`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L356)).
* `AWS lambda-microvms` — the service, which calls the hook over loopback inside the VM ([`agentd/src/routes.rs:168`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L168)).
* `agentd open router` — the unauthenticated half of the router, holding the lifecycle hooks ([`agentd/src/routes.rs:48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L48), [`agentd/src/routes.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178)).
* `AppState` — the one-shot token slot and the launch-environment map ([`agentd/src/state.rs:202`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202)).
* `Session` — the client bound to the reported endpoint with the same token ([`microvms-core/src/sandbox.rs:733`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L733)).
* `agentd auth guard` — `require_token`, applied as a `route_layer` over every control route ([`agentd/src/auth.rs:62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L62), [`agentd/src/routes.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L66)).
Edges in order:
1. `mint 32 bytes` — 32 bytes of `/dev/urandom` rendered as 64 hex characters, unless the caller supplied a token ([`microvms-core/src/sandbox.rs:675`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L675), [`microvms-core/src/sandbox.rs:1101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1101)).
2. `run_microvm()` — the payload is validated before the launch, so an over-ceiling one fails with a byte count rather than as a service `ValidationException` ([`microvms-core/src/sandbox.rs:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L682), [`microvms-core/src/sandbox.rs:696`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L696)).
3. `RunMicrovm` — [`microvms-core/src/control/microvm.rs:423`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L423).
4. `POST run hook` — unauthenticated by necessity: the platform has no credential to present, and its request arrives over loopback indistinguishably from an in-VM process ([`agentd/src/routes.rs:168`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L168), [`agentd/src/routes.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L178)). A body that is not JSON is 400, never 404 ([`agentd/src/routes.rs:187`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L187)).
5. `bootstrap(tok)` — the token and the launch environment arrive in one payload and are taken as two arguments, so no path can move a byte from the first into the second ([`agentd/src/routes.rs:213`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L213), [`agentd/src/state.rs:202`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202)). The env is installed only for the first caller ([`agentd/src/state.rs:210`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L210)).
6. `200 installed` — an identical replay is also 200, because the platform may retry its own hook; a different token is 409 ([`agentd/src/routes.rs:224`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L224), [`agentd/src/routes.rs:230`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L230)).
7. `wait RUNNING` — [`microvms-core/src/sandbox.rs:708`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L708).
8. `GetMicrovm` — polled until RUNNING, failing fast on a terminal state ([`microvms-core/src/control/microvm.rs:459`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L459), [`microvms-core/src/control/microvm.rs:465`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L465), [`microvms-core/src/control/microvm.rs:510`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L510)).
9. `RUNNING + url` — RUNNING is what reports the hook succeeded, so this is where `token_installed` and `bootstrap_count` move ([`microvms-core/src/sandbox.rs:722`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L722)).
10. `builder(token)` — the same minted token becomes the session bearer ([`microvms-core/src/sandbox.rs:733`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L733)).
11. `Bearer request` — the guard runs before the body is polled, and drains a bounded prefix on rejection ([`agentd/src/auth.rs:62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L62), [`agentd/src/auth.rs:87`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L87)).
12. `token_matches()` — constant-time comparison against the installed slot ([`agentd/src/auth.rs:75`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L75), [`agentd/src/state.rs:214`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L214)).
13. `503/401/pass` — three-valued: not-yet-bootstrapped is 503, a wrong credential is 401, and a match falls through to the handler ([`agentd/src/auth.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L73), [`agentd/src/auth.rs:77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L77)).
## See also
[Section titled “See also”](#see-also)
* [data flow](../../architecture/data-flow.md) — 11 shared source citations
* [processes](../../behavior/processes.md) — 11 shared source citations
* [business logic](../../insights/business-logic.md) — 10 shared source citations
* [debugging guide](../../insights/debugging-guide.md) — 9 shared source citations
* [components](../architecture/components.md) — 8 shared source citations
# Dependency graph
> Seven crates and the third-party crates that define each one's external interface. The workspace declares its members in Cargo.toml:2-10 under resolver = "3" (Cargo.toml:11), and…
Seven crates and the third-party crates that define each one’s external interface. The workspace declares its members in [`Cargo.toml:2-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L2-L10) under `resolver = "3"` ([`Cargo.toml:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L11)), and every member inherits `edition = "2024"` from `[workspace.package]` ([`Cargo.toml:22-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L22-L23)).
Internal nodes are plain rectangles; external crates are cylinders with a dashed stroke. Each external label carries the version `Cargo.lock` resolves, not the manifest’s requirement string.
## Legend (overflow)
[Section titled “Legend (overflow)”](#legend-overflow)
The diagram carries twenty nodes: the seven members and thirteen externals. Every other direct dependency is below. `Edges` is the number of internal-to-external edges the crate would draw — one per member that declares it. `Refs` counts path-qualified references across all members’ `src/`, matching only where the crate name starts a path segment, so `axum::http::` does not count toward `http`, `std::os::unix::` does not count toward `nix`, and `tower_http::` does not count toward either.
| Elided dependency | Edges | Refs | Declared by |
| -------------------------- | ----- | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| tracing 0.1.44 | 1 | 60 | [`agentd/Cargo.toml:52`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L52) |
| futures-util 0.3 | 2 | 15 | [`agentd/Cargo.toml:54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L54), [`microvms-core/Cargo.toml:105`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L105) |
| base64 0.23 | 2 | 13 | [`agentd/Cargo.toml:60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L60), [`microvms-core/Cargo.toml:110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L110) |
| napi-derive 3 | 1 | 10 | [`microvms-js/Cargo.toml:49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L49) |
| tempfile 3 | 1 | 10 | [`agentd/Cargo.toml:51`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L51) |
| aws-credential-types 1.3 | 1 | 6 | [`microvms-core/Cargo.toml:65`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L65) |
| zip 8.6 | 1 | 6 | [`microvms-core/Cargo.toml:93`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L93) |
| http 1.5.0 | 1 | 6 | [`microvms-core/Cargo.toml:82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L82) |
| tar 0.4.46 | 1 | 4 | [`agentd/Cargo.toml:43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L43) |
| rust\_decimal\_macros 1.40 | 1 | 4 | [`microvms-core/Cargo.toml:47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L47) |
| aws-config 1.10 | 1 | 3 | [`microvms-core/Cargo.toml:59-64`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L59-L64) |
| tower-http 0.6 | 1 | 3 | [`agentd/Cargo.toml:25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L25) |
| rust\_decimal 1.42 | 1 | 2 | [`microvms-core/Cargo.toml:46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L46) |
| tracing-subscriber 0.3 | 1 | 2 | [`agentd/Cargo.toml:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L53) |
| http-body-util 0.1 | 1 | 2 | [`agentd/Cargo.toml:37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L37) |
| bytes 1 | 1 | 2 | [`agentd/Cargo.toml:38`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L38) |
| thiserror 2.0.19 | 1 | 1 | [`microvms-core/Cargo.toml:32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L32) |
| sha2 0.11 | 1 | 1 | [`microvms-core/Cargo.toml:98`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L98) |
| backon 1.6 | 1 | 1 | [`microvms-core/Cargo.toml:86`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L86) |
| subtle 2.6 | 1 | 1 | [`agentd/Cargo.toml:44`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L44) |
| tokio-util 0.7 | 1 | 1 | [`agentd/Cargo.toml:55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L55) |
| napi-build 2 | 1 | 0 | [`microvms-js/Cargo.toml:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L66), build-dependency |
Thirteen seated plus twenty-two elided is the complete direct-dependency union: thirty-five distinct third-party crates across the seven manifests.
A low count does not mean a weak edge. `http` is called in exactly one file: all six references build the `http::Request` that `aws-sigv4` signs and that `reqwest` 0.13 consumes through `TryFrom`, at [`microvms-core/src/control/transport.rs:515`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L515), [`:531`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L531), [`:580`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L580), [`:1241`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L1241), [`:1282`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L1282), [`:1496`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L1496). `thiserror`, `sha2`, and `backon` each land at a single derive or call site, and `aws-config`’s whole surface is one `load()`.
Which thirteen got a node therefore follows architectural role rather than reference count: one per member for the runtime, the wire codec, the schema generator, the HTTP server, the syscall layer, the HTTP client, the request signer, the argument parser, the TUI, and each binding’s FFI crate. That seats `aws-sigv4` at 3 references and drops `tracing` at 60, which is the highest count on the page without a node. Both counts are published so the ranking is inspectable rather than implied by node placement.
`napi-build` draws an edge but has no `napi_build::` path reference because its whole surface is one call in a build script, `napi_build::setup()` at [`microvms-js/build.rs:12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/build.rs#L12). Omitting it does not warn — it produces undefined-symbol link failures ([`microvms-js/build.rs:8-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/build.rs#L8-L9)).
Dev-dependencies are out of the diagram’s scope, which elides the test tiers: `proptest` 1.11 and `turmoil` 0.7.2 appear in both [`agentd/Cargo.toml:72-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L72-L79) and [`microvms-core/Cargo.toml:112-126`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L112-L126), alongside `hyper` 1.11, `hyper-util` 0.1 and `tower` 0.5. `stateright` is on the diagram because it is a normal dependency of `model` ([`model/Cargo.toml:9-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L9-L10)), not a dev-dependency.
## Direction is asserted, not conventional
[Section titled “Direction is asserted, not conventional”](#direction-is-asserted-not-conventional)
`microvms-cli/tests/dependency_direction.rs` reads `cargo metadata`’s resolved graph and asserts the edges among `microvms-cli`, `microvms-core`, `microvms-py`, and `microvms-js` as **equalities** rather than absences. The reason is written into the file: `assert!(no edge from A to B)` passes when A has no dependencies at all, which is what a stub crate looks like, so an equality is what fails both for a binding that grows an edge to the CLI and for a binding that never grows its edge to core ([`microvms-cli/tests/dependency_direction.rs:9-14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L9-L14)). The same file asserts `microvms-cli` exposes no `lib` target, making “nothing a binding needs lives in the CLI” a property rather than a request ([`microvms-cli/tests/dependency_direction.rs:16-21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L16-L21), [`microvms-cli/Cargo.toml:10-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L10-L20)).
`microvms-core` depending on the CLI would make every consumer of the library, both bindings included, carry `clap`, `ratatui`, and a multi-thread tokio runtime ([`microvms-cli/tests/dependency_direction.rs:63-65`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L63-L65)).
## The CLI’s dependency set is a denylist under test
[Section titled “The CLI’s dependency set is a denylist under test”](#the-clis-dependency-set-is-a-denylist-under-test)
`microvms-cli` takes the maintained crates it needs, thirteen direct dependencies today ([`microvms-cli/Cargo.toml:35-128`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L35-L128)), and nothing polices that count: the manifest’s own comment states the rule as “dependencies are otherwise welcome” ([`microvms-cli/Cargo.toml:25-34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L25-L34)). What is under test is the hazard. [`microvms-cli/tests/thinness.rs:49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L49) holds `const FORBIDDEN: [&str; 12]`, naming `reqwest`, `hyper`, `hyper-util`, `http`, `aws-config`, `aws-sdk-s3`, `aws-sdk-sts`, `aws-sigv4`, `aws-credential-types`, `aws-smithy-runtime`, `rusoto_core`, and `ureq`, and `no_direct_dependency_is_a_second_path_to_aws` ([`microvms-cli/tests/thinness.rs:96`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L96)) reads the manifest through `cargo metadata` and fails if any of them appears as a normal or dev dependency. The earlier six-crate allowlist, and the `RETIRED` record of `futures-util` leaving it, were removed with it: a cap on the manifest asserted a size, while the denylist asserts the property CLI-2 names, that every AWS call goes through `microvms-core`.
## Absent edges that carry weight
[Section titled “Absent edges that carry weight”](#absent-edges-that-carry-weight)
* **`microvms-cli` has no `protocol` edge.** The wire types are reached through `microvms_core::protocol::`, core’s re-export, so the CLI has one door to everything below it ([`microvms-cli/Cargo.toml:48-51`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L48-L51)). Confirmed in the source: `microvms-cli/src` contains no bare `protocol::` path — every one of the 25 references is qualified through core, as at [`microvms-cli/src/commands/attached.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L178).
* **Both bindings do have a direct `protocol` edge**, and it is live rather than vestigial: [`microvms-py/src/session.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L73) and [`microvms-js/src/session.rs:85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L85) name `protocol::health::Health` directly, and both build `protocol::exec::StartRequest` ([`microvms-py/src/session.rs:338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L338), [`microvms-js/src/session.rs:143`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L143)). Core’s public signatures already return these types, so a binding that mapped them without naming the crate would re-declare their fields, which is the drift `protocol` was extracted to prevent ([`microvms-py/Cargo.toml:27-32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L27-L32), [`microvms-js/Cargo.toml:22-25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L22-L25)).
* **`agentd` reaches no AWS crate and no HTTP client.** Its 18 direct dependencies ([`agentd/Cargo.toml:10-70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L10-L70)) contain no `reqwest` and no `aws-*`; it is a server, and the crate that talks to AWS is `microvms-core`, which runs on the developer host rather than in the MicroVM image ([`microvms-core/Cargo.toml:56-58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L56-L58)).
* **`agentd` does not declare `http` either**, though it uses those types constantly. Every one of its `http::` references is qualified through axum’s re-export — `axum::http::StatusCode` ([`agentd/src/routes.rs:6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L6)), `axum::http::HeaderMap` ([`agentd/src/auth.rs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L40)), `axum::http::header::CONTENT_TYPE` ([`agentd/src/exec.rs:1582`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1582)) — so the daemon carries no second path to the `http` version axum already fixes.
* **`model` has no workspace edge at all.** [`model/src/client.rs:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L58) says it mirrors `microvms_core::sandbox::Lifecycle` “by convention rather than by dependency”. Its single dependency is `stateright` ([`model/Cargo.toml:9-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L9-L10)), which is why it appears on the diagram with exactly one edge.
* **`microvms-core` declares no `axum`.** The `axum::serve::Listener` named at [`microvms-core/src/session/http.rs:15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/http.rs#L15) is a module comment drawing an analogy, not an import.
## Shared substrate
[Section titled “Shared substrate”](#shared-substrate)
Four dependencies are declared by more than one member. Each gets one edge on the diagram, sourced at the member with the most references, so a single edge is not a claim of exclusivity. Per-member reference counts:
| Dependency | agentd | microvms-core | microvms-cli | microvms-py | microvms-js | protocol |
| ----------- | ------ | ------------- | ------------ | ----------- | ----------- | -------- |
| tokio | 119 | 183 | 49 | 9 | 11 | — |
| serde\_json | 19 | 114 | 85 | — | — | 31 |
| serde | 6 | 59 | 7 | — | — | 27 |
| schemars | 1 | — | — | — | — | 4 |
The feature sets differ where the role differs, and the manifests say why. `microvms-cli` takes `rt-multi-thread` because it is the process and therefore the thing entitled to choose a runtime; `microvms-core` deliberately carries no runtime feature at all, because a library does not choose its caller’s ([`microvms-cli/Cargo.toml:65-69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L65-L69), [`microvms-core/Cargo.toml:101-104`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L101-L104)). `microvms-js` takes only `time` and `sync` because napi owns the runtime ([`microvms-js/Cargo.toml:50-53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L50-L53)), while `microvms-py` takes `rt-multi-thread` for one runtime blocked on with the GIL released ([`microvms-py/Cargo.toml:39-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L39-L42)).
`schemars` is pinned to the identical version with the identical two features in both crates that carry it, and `preserve_order` must stay off, because the committed `docs/schema.json` is byte-compared in CI and key order therefore has to be a function of the types rather than of derive order ([`protocol/Cargo.toml:12-16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/Cargo.toml#L12-L16), [`agentd/Cargo.toml:61-70`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L61-L70)).
## Version pins the manifests argue for
[Section titled “Version pins the manifests argue for”](#version-pins-the-manifests-argue-for)
Several externals carry a version floor with a defect behind it rather than a preference:
* `tar = "0.4.46"` — at least 0.4.45 is required because RUSTSEC-2026-0068 fixed a PAX size-header desync that let one archive parse differently across extractors ([`agentd/Cargo.toml:41-43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L41-L43)).
* `tower-http = "0.6"` and not 0.7 — axum 0.8.9 pins `^0.6.8` internally, so 0.6 keeps one version of the middleware types in the tree. Its `catch-panic` layer is load-bearing: the daemon is the only channel into the VM, so a panic that kills a connection makes the VM unreachable for good ([`agentd/Cargo.toml:17-25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L17-L25)). That layer is implemented with `catch_unwind`, which is why the release profile overrides `panic` back to `"unwind"` ([`Cargo.toml:39-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L39-L57)).
* `nix = "0.31"` — a caret on 0.31 is the widest safe range because nix is pre-1.0 and every minor is a breaking change. Four features, each named for a call site: `signal` for the exec kill path, `fs` for the `statvfs` disk-pressure guard, `mount` for the `MS_BIND` that shadows the read-only procfs `boot_id`, `hostname` for `sethostname` ([`agentd/Cargo.toml:45-50`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L45-L50)).
* `aws-config` keeps `default-https-client` **on** — the credential chain does its own HTTP for IMDS, SSO, and STS through smithy’s client and panics at `load()` without one. The price is two HTTP stacks, smithy for credentials and reqwest for service calls; both sit on rustls, so it is one TLS implementation ([`microvms-core/Cargo.toml:50-64`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L50-L64)).
* `aws-sigv4` rather than a generated SDK — `lambda-microvms` has no aws-sdk-rust crate, so the choice was between vendoring smithy codegen and signing 24 rest-json operations by hand ([`microvms-core/Cargo.toml:50-52`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L50-L52)).
* `reqwest` with `rustls` and not `default-tls`, because the daemon ships to an aarch64 musl target where a native-tls build needs an OpenSSL the image does not carry. `json` is off deliberately, so the error path can read a raw body and see an `AccessDeniedException` whose message field is null ([`microvms-core/Cargo.toml:71-81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L71-L81)).
* `zip = "8.6"` with `deflate` only — 9.0 exists only as `9.0.0-pre2`, and a pre-release in a shipping manifest is a version that can change under you; the other compressors are C libraries that would otherwise have to build for the musl target ([`microvms-core/Cargo.toml:87-93`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L87-L93)).
* `rust_decimal` with `serde-with-str` — the ARM rates are figures like `0.0000276944` and summing a few thousand in binary floating point drifts toward a bill nobody can reproduce ([`microvms-core/Cargo.toml:40-46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L40-L46)).
* `napi` at `napi5` rather than `napi4`, forced rather than chosen: napi 3.12’s `web_stream` declares only `napi4`, but its `ReadableStream` finalizer calls `napi_add_finalizer`, which `napi-sys` gates behind `napi5` ([`microvms-js/Cargo.toml:37-43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L37-L43)).
* `pyo3` with `abi3-py39` and **no** `extension-module` feature. That feature is deprecated, and enabling it disables libpython linking for every target in the workspace, breaking `cargo test` with undefined `_PyExc_*` symbols; maturin sets `PYO3_BUILD_EXTENSION_MODULE` itself for a wheel build ([`microvms-py/Cargo.toml:33-38`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L33-L38), [`microvms-py/pyproject.toml:5-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/pyproject.toml#L5-L9)).
## Crate-type asymmetry between the bindings
[Section titled “Crate-type asymmetry between the bindings”](#crate-type-asymmetry-between-the-bindings)
`microvms-py` is `crate-type = ["cdylib", "rlib"]`: the cdylib is what Python imports, the rlib is what lets `tests/` and doctests link the crate, and cdylib alone produces E0432/E0463 for anything that tries to `use` it ([`microvms-py/Cargo.toml:13-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L13-L19)). `microvms-js` is `cdylib` only, because a Node addon is loaded by the runtime and nothing in this workspace links it — the smoke test drives the built `.node` through `node --test` rather than through `cargo test` ([`microvms-js/Cargo.toml:12-17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L12-L17), [`microvms-js/package.json:13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L13)).
Neither crate publishes. `publish = false` is inherited workspace-wide ([`Cargo.toml:26-32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.toml#L26-L32)), the bindings restate it ([`microvms-py/Cargo.toml:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L11), [`microvms-js/Cargo.toml:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L10)), and the npm side says the same thing with `"private": true` ([`microvms-js/package.json:5`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L5)).
## See also
[Section titled “See also”](#see-also)
* [impact analysis](../../insights/impact-analysis.md) — 12 shared source citations
* [system overview](../../architecture/system-overview.md) — 9 shared source citations
* [contract map](../../insights/contract-map.md) — 8 shared source citations
* [tech debt](../../insights/tech-debt.md) — 8 shared source citations
* [business logic](../../insights/business-logic.md) — 6 shared source citations
# Embedding agentd in your own image and driving it from your own harness
> The platform has no exec API: a MicroVM exposes one HTTPS endpoint and forwards it to whatever the image's CMD is listening on. Every harness that wants to run commands inside a…
The platform has no exec API: a MicroVM exposes one HTTPS endpoint and forwards it to whatever the image’s `CMD` is listening on. Every harness that wants to run commands inside a VM therefore ships a daemon in its task image, and before agentd each harness wrote its own — evaluation harnesses and session servers each carry a several-hundred-line stdlib Python daemon baked into their images (`docs/HARNESS-CAPABILITIES.md`, gap 2). agentd supersedes those daemons. This document is the recipe for appending it to an arbitrary task image, and the orientation a harness client needs to drive it over the published wire protocol. The protocol itself is in `docs/PROTOCOL.md` and, machine-readably, at `GET /v1/schema` on any running daemon; nothing here duplicates either.
## The recipe
[Section titled “The recipe”](#the-recipe)
`microvm dockerfile` prints the stanza that wraps a base image with agentd — the same Dockerfile the default `microvm build` bakes, emitted by the same generator ([`microvms-core/src/control/artifact.rs:145`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L145)), so appending your own layers to it *is* the default build plus your layers.
```plaintext
1
microvm dockerfile --workdir /workspace > Dockerfile
2
# edit: insert your RUN layers between the chmod line and the ENV lines
3
microvm build ./agentd --dockerfile Dockerfile --name my-task-image
```
The stanza’s comments name the two platform constraints a hand-written wrapper hits, both enforced by microvms-core before any AWS call:
1. **The `FROM` must match the managed base’s `docker_ref`.** The build runs the Dockerfile on top of the base that `baseImageArn` names, and a mismatch builds against a base none of the measured platform behaviour applies to — so `require_matching_from` refuses it ([`microvms-core/src/control/artifact.rs:228-244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L228-L244)).
2. **A `WORKDIR` is required when the base declares none.** The managed al2023 base, like most public ARM64 bases, leaves `WorkingDir` empty, so “inherit the image WORKDIR” inherits `/` and every relative path in your commands resolves somewhere you did not mean ([`microvms-core/src/control/artifact.rs:196-220`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L196-L220)).
The worked example is [`examples/coding-agents-on-bedrock/Dockerfile`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/examples/coding-agents-on-bedrock/Dockerfile): the stanza’s lines, plus `dnf install` and `npm install -g` layers that put two coding-agent CLIs in the image, plus a `/workspace` WORKDIR. Any task image is the same shape — take the stanza, add the layers your workload needs, keep the daemon lines intact.
Two lines in the stanza are load-bearing and must survive your edits. `ENTRYPOINT []` plus `CMD ["/agentd"]` is the deployment invariant the trust boundary rests on: it guarantees no task workload runs before the platform’s run hook lands, and it is what makes an omitted `cwd` inherit the image `WORKDIR` ([`microvms-core/src/control/artifact.rs:132-144`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L132-L144), `docs/PROTOCOL.md`, “Trust boundary”). A base image that starts its own background process before bootstrap breaks the invariant, and enforcing it belongs to whoever builds the image — the daemon cannot.
One thing never goes in the image: a secret. The image becomes a shared snapshot, so every VM launched from it sees the same bytes; per-VM credentials travel through `runHookPayload` at launch instead ([`microvms-core/src/control/artifact.rs:15-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L15-L23)).
## The wire contract a harness client implements
[Section titled “The wire contract a harness client implements”](#the-wire-contract-a-harness-client-implements)
The full route table, request shapes, and the defect-driven rules are in `docs/PROTOCOL.md`; the same contract is served as JSON Schema at `GET /v1/schema`, unauthenticated, so a client can fetch it before it holds a token. What follows is the shape of the client, not the contract itself.
**Bootstrap.** The platform delivers your `runHookPayload` string to the daemon’s `/run` hook; agentd parses it as JSON and installs `agent_token` ([`agentd/src/routes.rs:166-216`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L166-L216)). The install is one-shot: a replay of the identical token answers 200 (the platform may retry its own hook), a different token answers 409 and changes nothing. Until it lands, every control route answers 503 — not 404, not a dropped connection — so a client can distinguish “not yet bootstrapped” from “broken” ([`agentd/src/auth.rs:62-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L62-L80)). The payload is capped at 4096 bytes ([`microvms-core/src/constants.rs:61`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L61)).
**Auth.** Every `/v1/` route except `/v1/health` and `/v1/schema` takes `Authorization: Bearer ` — the same token the payload delivered. Comparison is constant-time over bytes ([`agentd/src/auth.rs:28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L28)).
**Exec.** The client mints the `exec_id` and sends it in `POST /v1/exec/start`. That is what makes a retry safe: a start carrying a known id returns success without spawning a second child, decided under the registry lock ([`agentd/src/exec.rs:364-367`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L364-L367)), so a harness whose process died between sending the start and reading the answer sends the identical start again and gets the original exec. `GET /v1/exec/{id}` polls, read-only, repeatable. `POST /v1/exec/{id}/ack` releases the buffered output and starts the collection clock; a second ack is 409, because the first released it and a 200 with an empty body would read as “the command produced no output”. Output lives until the ack, so nothing a slow reader has not seen is destroyed. `POST /v1/exec/{id}/kill` signals the process group, SIGTERM then SIGKILL after a grace period ([`agentd/src/exec.rs:900-931`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L900-L931)). `POST /v1/exec/{id}/stdin` writes to a child that was started with `stdin: true` and carries the explicit EOF signal; an exec that never asked for stdin answers 409.
**Streaming.** `GET /v1/exec/{id}/stream?offset=N` follows output as SSE from a byte cursor. A reconnecting client passes the offset it read to and receives exactly what it has not seen; a reattach past the retained window gets an explicit `gap` event naming the missing byte range rather than silently skipping ([`agentd/src/exec.rs:436-524`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L436-L524)). The stream ends with a typed `exit` event, which is what distinguishes a finished command from a cut connection — the reason this is SSE and not a chunked byte stream.
**Files.** `PUT`/`GET /v1/fs/file` move one file, streamed, with a mode applied at open. `PUT`/`GET /v1/fs/tar` move directory trees; extraction is confined by lexical resolution with symlink and bomb defenses and member/size caps ([`agentd/src/fs.rs:4-41`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L4-L41)), and a write that would push the filesystem under the disk reserve is refused with 507 naming the real free space ([`agentd/src/fs.rs:66-91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L66-L91)).
**Health.** `GET /v1/health` is unauthenticated and reports version, bootstrap state, disk pressure, and the identity-repair flags — the conditions that are reasons to drain a VM rather than schedule more work onto it.
## The proxy-token reality
[Section titled “The proxy-token reality”](#the-proxy-token-reality)
The daemon’s endpoint sits behind the platform’s proxy, and the proxy wants two headers on every request: `X-aws-proxy-auth` carrying a minted JWE, and `X-aws-proxy-port` naming which allowed port this request targets — omitting the second is rejected in a way that reads like a bad token ([`microvms-core/src/session/proxy.rs:5-13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L5-L13)). The token comes from `CreateMicrovmAuthToken`, and the response’s `authToken` is a **map of header name to value**, not a string; read it as a string and every request fails.
The service caps a token at sixty minutes ([`microvms-core/src/session/proxy.rs:63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L63)). That is not a choice, and it is shorter than a long agent run, so a client that mints once at construction expires mid-run with a rejection indistinguishable from a dead daemon. The pattern that works is minting inside the request path with a refresh interval well under the ceiling — this repo’s clients refresh at half of it, thirty minutes, so a request in flight across the rollover still holds a token with about thirty minutes of life ([`microvms-core/src/session/proxy.rs:29-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L29-L37)). A mint failure is retryable; treat it that way, because a control-plane throttle at minute thirty must not kill a healthy run.
Token rotation costs nothing on the daemon side. All exec state — the records, the buffered output, the stream cursors — lives in the daemon, keyed by `exec_id`, so a detached exec started under one proxy token is polled and acked under the next one. Start, rotate, poll, ack is a normal sequence, not a recovery path. This is a tested contract, not an inference: the live suite’s `reattach after token rotation` section starts a detached exec, drops every piece of client state except the endpoint, the agent token, and the MicroVM id, reattaches under freshly minted proxy tokens, and asserts that the output produced *before* the reattach comes back whole — nothing buffered under one token is lost to the next (`conformance/run_rs.py`, `drive_token_rotation`).
## The idle keepalive is yours, and it must run outside the VM
[Section titled “The idle keepalive is yours, and it must run outside the VM”](#the-idle-keepalive-is-yours-and-it-must-run-outside-the-vm)
The platform measures idleness by inbound traffic through the endpoint proxy and suspends a VM whose window elapses without any. Your harness — the orchestrator outside the VM — owns the keepalive: poll `GET /v1/health` on an interval well under the launch’s `maxIdleDurationSeconds`, and each poll is the inbound traffic that resets the timer. Measured, both halves: a polled VM outlives its idle window and the same VM suspends once the polling stops (`docs/PLATFORM.md`, “An outside poll of `/v1/health` does reset the idle timer”; asserted every live run by `conformance/run_rs.py`, `drive_idle_keepalive`).
An in-guest keepalive **cannot** work, and it is worth knowing why before someone builds one: the endpoint proxy terminates *outside* the VM and forwards over loopback, so a request a guest process sends to the daemon’s own port is generated on the far side of the meter and never crosses it. A guest-side keepalive route would answer 200 and change nothing, and the failure would surface as a suspend during exactly the long run it was added to protect (`docs/HARNESS-CAPABILITIES.md`, gap 6). Neither does in-guest *work*: a VM running a multi-hour exec with no outside traffic is suspended mid-work at the idle window. The process survives — suspend is a freeze, not a kill — but nothing external can reach it until someone resumes it.
`/v1/health` is the right route for the poll: unauthenticated, one small request, and it carries `busy` and `execs` so the poll is informed rather than unconditional — an orchestrator can stop keeping a drained VM alive instead of billing it to the duration ceiling.
## What the hand-rolled daemons needed, and where agentd covers it
[Section titled “What the hand-rolled daemons needed, and where agentd covers it”](#what-the-hand-rolled-daemons-needed-and-where-agentd-covers-it)
The two daemon shapes this supersedes are described in `docs/HARNESS-CAPABILITIES.md`; neither project is a dependency of this repo, so the rows are the generic needs.
| Need | Who had it | agentd |
| ---------------------------------------------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Start/poll/ack exec that outlives an auth-token ceiling | evaluation harnesses | caller-minted `exec_id`, idempotent start, read-only poll, explicit ack, TTL only after ack (`agentd/src/exec.rs`) |
| Idempotent start under retry | evaluation harnesses | a known id returns success without spawning a second child ([`agentd/src/exec.rs:364-367`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L364-L367)) |
| Per-exec env, cwd, user/group, timeout | evaluation harnesses | in the wire protocol and applied by the daemon; the child’s environment starts empty, so the token never leaks into it |
| File and directory-tree transfer with tar fidelity | evaluation harnesses | streamed file routes plus confined tar extraction (`agentd/src/fs.rs`) |
| Per-instance credential bootstrap, no secret in the shared image | both | one-shot `runHookPayload` bootstrap with replay semantics ([`agentd/src/routes.rs:166-216`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L166-L216)) |
| Lifecycle hooks answered so the platform can manage the VM | session servers | ready/validate/run/suspend/resume/terminate all served ([`agentd/src/routes.rs:112-118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L112-L118)) |
| A liveness probe cheaper than an exec | session servers | unauthenticated `GET /v1/health` |
| Live output streaming with resume | neither had it | SSE with byte-cursor resume and explicit gap events ([`agentd/src/exec.rs:436-524`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L436-L524)) |
## Coding agents over the daemon
[Section titled “Coding agents over the daemon”](#coding-agents-over-the-daemon)
The one opinionated layer this repo ships over the recipe above is `docs/AGENT-VMS.md`: `microvm agent-up` derives a Dockerfile from your agentd stanza plus three layers (Node 22 with `nodejs22-npm`, `npm install -g` of Claude Code and/or Codex, a uid 1000), launches with egress, mints a Bedrock bearer token from the caller’s own credentials, and installs it as `/workspace/.agent-env` for the agent to source; `microvm agent-prompt` runs the agent headless as that user. A harness that already embeds agentd gets the same steps from the bindings’ `AgentVm`, or piecewise from `install_agent_access` and `prompt_agent` over any `Session`, so the credential file, the demotion, the `PATH` line, and the read-back-the-effect discipline are the library’s rather than each harness’s to rediscover.
## Configuration knobs
[Section titled “Configuration knobs”](#configuration-knobs)
Every `AGENTD_*` variable is read at startup by `Config::from_env` ([`agentd/src/config.rs:116-152`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L116-L152)); an unset or unparseable value keeps the default rather than refusing to boot, because a daemon that will not start strands the VM with no way in. Set them as `ENV` lines in your Dockerfile — the stanza already sets the first two.
| Variable | Default | What it bounds |
| -------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AGENTD_PORT` | `9000` | the port the control API and hooks listen on ([`agentd/src/config.rs:15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L15)) |
| `AGENTD_LOG` | `info` | the tracing filter, standard `EnvFilter` syntax ([`agentd/src/main.rs:91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L91)) |
| `AGENTD_MAX_BODY_BYTES` | 512 MiB | largest request body accepted on the wire ([`agentd/src/config.rs:17-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L17-L19)) |
| `AGENTD_MAX_OUTPUT_BYTES` | 8 MiB | per-stream cap on captured exec output; exceeding it truncates and marks the result ([`agentd/src/config.rs:25-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L25-L27)) |
| `AGENTD_OUTPUT_LINGER_SECS` | `5` | how long to keep reading pipes after the child exits, for grandchildren holding them ([`agentd/src/config.rs:28-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L28-L31)) |
| `AGENTD_EXEC_TTL_SECS` | `900` | how long an acked exec entry is retained before collection ([`agentd/src/config.rs:32-33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L32-L33)) |
| `AGENTD_STREAM_BUFFER_BYTES` | 1 MiB | bytes of recent output kept for stream replay; a reattach past it gets a gap event ([`agentd/src/config.rs:41-45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L41-L45)) |
| `AGENTD_STREAM_CHANNEL_CAPACITY` | `256` | slots in an exec’s live fan-out channel; a lagging subscriber re-reads the ring instead of losing output ([`agentd/src/config.rs:46-49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L46-L49)) |
| `AGENTD_SSE_KEEPALIVE_SECS` | `15` | interval between SSE keep-alive comments, so a silent exec does not look like a dead connection ([`agentd/src/config.rs:50-53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L50-L53)) |
| `AGENTD_MAX_STDIN_WRITE_BYTES` | 1 MiB | largest single decoded stdin write ([`agentd/src/config.rs:54-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L54-L57)) |
| `AGENTD_DISK_RESERVE_BYTES` | 256 MiB | free bytes a write target must keep; a write that would cross it is refused with 507. Zero disables the guard ([`agentd/src/config.rs:63-69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L63-L69)) |
| `AGENTD_REPAIR_IDENTITY` | `true` | whether to replace image-derived identity at startup, because N VMs restored from one snapshot share machine-id, hostname, and boot\_id. `0`/`false`/`no`/`off` opt out ([`agentd/src/config.rs:70-78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L70-L78)) |
# What agent harnesses require of this platform
> This document maps the sandbox contracts of three agent harnesses onto this platform's current surface, and ranks the gaps. The harnesses are Harbor (agent evaluation), Omnigent…
This document maps the sandbox contracts of three agent harnesses onto this platform’s current surface, and ranks the gaps. The harnesses are Harbor (agent evaluation), Omnigent (server-managed agent sessions), and the Vercel Sandbox / eve shape (agent-tool sandboxes). None of them becomes a dependency: their contracts were read from their source and public docs, and what this platform ships stays generic: capabilities any workload can use, never integrations that track a harness’s release cadence.
Sources: Harbor read at `harbor-framework/harbor` (local checkout, `feat/lambda-microvm-environment`); Omnigent read at PR #2217’s branch; Vercel Sandbox from vercel.com/docs/sandbox (retrieved 2026-08-14, SDK v2.9.2) and eve.dev/docs/sandbox; this repo read on `main` at d80150a.
## The three contracts, compressed
[Section titled “The three contracts, compressed”](#the-three-contracts-compressed)
**Harbor** drives a sandbox through one abstract class: exec (shell command, cwd, per-exec env dict, run-as-user with HOME/groups fixup, timeout with process-group kill and code 124, full stdout/stderr capture), file transfer (single files with mode, directory trees with tar fidelity: modes, symlinks, empty dirs), lifecycle (build-from-Dockerfile with content-addressed reuse, readiness signal, terminate/suspend), and per-instance credential bootstrap that never bakes a secret into a shared image. Commands must be startable idempotently under retry and outlive any auth-token ceiling, which is why Harbor’s own MicroVM provider hand-rolled a start/poll/ack daemon. Optional tiers Harbor degrades around: multi-container, GPUs, Windows, dynamic network policy, live output streaming.
**Omnigent** needs lifecycle more than exec. Its host dials out to the server over a WebSocket tunnel, so the sandbox needs no inbound API at all; what it needs is: create (or reserve-an-id-then-create), idempotent terminate, suspend-to-snapshot that preserves the whole process tree, idempotent resume under the same identity, cheap liveness, a per-launch env/secret channel delivered before the workload starts, and a published lifetime cap so launch-token TTLs can be derived above it. Its provider plus a deploy shim hand-rolled a lifecycle-hooks HTTP server, a 16 KB `runHookPayload` env channel, idle-policy derivation math, and NotFound/Conflict-to-success mappings, all of which become deletable if the platform supplies those natively. An exec daemon also lets Omnigent use its simpler exec-model launcher, where repo clone and config injection are shared framework code instead of in-image shell scripts.
**Vercel Sandbox / eve** is an API shape rather than a consumer: named, persistent-by-default sandboxes (stop auto-snapshots; the next call resumes), blocking and detached exec with a durable command id, replayable buffered output, live log streaming, kill with signal, sudo, cwd/env, batch file writes as gzipped tarballs, a `node:fs/promises`-shaped metadata surface, declared ports mapping to public URLs, SNI-domain and CIDR egress policy applied to the live session, and snapshot/fork. Agent frameworks layered on it need far less: eve’s backend adapter is about ten session methods (run, spawn with byte streams and kill, read/write file, remove, resolve path, set network policy, stop), and the AI SDK’s canonical sandbox tool needs only create, blocking exec with captured output, and teardown.
## Where the platform already meets them
[Section titled “Where the platform already meets them”](#where-the-platform-already-meets-them)
The daemon and client cover more of these contracts than any of the three providers’ hand-rolled daemons did, with proofs behind each behavior:
* Idempotent detached exec: caller-minted exec id, retry-safe start, read-only poll, explicit ack, TTL only after ack, so unread output is never destroyed (`agentd/src/exec.rs`). This is precisely the start/poll/ack model Harbor’s provider built, plus SSE streaming with byte-cursor resume and explicit gap events, which Harbor’s daemon lacks.
* Per-exec `env`, `cwd`, `user`/`group`, `timeout_sec`, `stdin`, and shell vs argv mode are all in the wire protocol and applied by the daemon ([`protocol/src/exec.rs:116`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L116), [`agentd/src/exec.rs:999-1019`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L999-L1019)), and exposed by the Rust, Python, and Node clients.
* Kill with SIGTERM-grace-SIGKILL to the process group; per-command timeout enforced daemon-side when requested.
* File transfer: streamed single-file read/write with mode-at-open, tar upload/download with a confined extraction path (lexical resolution, symlink and bomb defenses, member/size caps), disk-pressure refusal with the real numbers (`agentd/src/fs.rs`).
* Per-VM secret bootstrap through `runHookPayload` with one-shot semantics and traffic ordering guaranteed by the platform; the token never enters a child’s environment.
* Suspend/resume that preserves memory, filesystem, token, running processes, and exec records (measured, `docs/PLATFORM.md`); local refusal of illegal transitions with zero billable calls.
* Teardown that never raises, reports leaked identifiers, and a local ledger (`microvm ls`) that records leaks before attempting deletes.
* Image builds from a Dockerfile with local pre-flight of the two platform traps (FROM/base agreement, WORKDIR requirement) and clientToken replay protection.
* A machine-readable manifest, one JSON envelope per invocation, stable error codes, and append-only exit codes: the agent-friendly CLI surface none of the three harnesses’ providers had to build against before.
## The gaps, ranked
[Section titled “The gaps, ranked”](#the-gaps-ranked)
Ranked by how many harnesses need it, times how much hand-rolled code it deletes, over the cost of building it here.
**1. Expose per-exec env (and user) through the CLI.** The daemon applies `env` per request and the bindings expose it; the CLI hardcodes `env: HashMap::new()` ([`microvms-cli/src/commands/lifecycle.rs:677-707`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L677-L707)). Every harness passes env per exec (Harbor merges three layers of it on every call), and the PATH failure the coding-agents example documents is this gap biting a real workload. `exec --env KEY=VALUE` (repeatable) plus `--user`/`--group` makes the CLI equal to the bindings. Smallest change, highest reach.
**2. Ship the platform daemon as the reusable answer to “no exec API”.** Harbor and Omnigent each carry a several-hundred-line stdlib Python daemon baked into task images. agentd already does everything those daemons do, better tested. What is missing is packaging: a documented recipe (and a `Dockerfile` stanza helper) for appending agentd to an arbitrary task image, so a harness provider is a thin client over the published wire protocol instead of a daemon author. The coding-agents example is the seed; this is its generalization.
**3. Image name resolution and content-addressed reuse in the CLI.** `run --image` passes the identifier verbatim to the service, which rejects bare names (“Malformed ARN”); both Harbor’s provider and our example resolve ARNs by listing, and both key image names to content hashes to avoid the stale-snapshot-on-name-reuse hazard. Resolve names client-side and offer `build --reuse` keyed on artifact content hash.
**4. A per-launch environment channel. Shipped.** Omnigent’s whole hooks-server shim existed because the platform offers no per-launch env vars; the `runHookPayload` is the only per-VM secret channel and it carried exactly one token. The run hook now accepts an optional `env` map in the same payload and the daemon applies it as the base environment of every later exec, with the per-request `env` winning on a shared key. `RunRequest::with_launch_env`, `run --launch-env KEY=VALUE`, and both bindings expose it. Two things the design pinned rather than left open: the token never becomes part of that base environment, proven by a test that asserts a child’s whole environment equals the launch map; and only the first successful bootstrap sets it, so a caller who cannot win the token cannot rewrite the environment either. The payload budget is 4096 bytes and not 4 KB of headroom — it is shared with the token, and `microvms-core` refuses an over-budget payload locally, naming the env’s share of it, since AWS’s own answer arrives after the call and botocore does not check. Credential-scale material still belongs on the file path or a role.
**5. Session-lifetime alignment for long execs. Shipped.** Harbor’s daemon exists partly because commands must outlive the 60-minute proxy-token ceiling. Our detached exec already survived it by design — state lives in the daemon; a re-minted token reattaches — and the design is now the tested contract this item asked for. The live suite’s `reattach after token rotation` section (`conformance/run_rs.py`, `drive_token_rotation`) starts a detached exec, reattaches from nothing but the three identifiers a harness would have persisted — each attach minting a fresh proxy token, which is what a rotation *is* — and asserts under the check name `no output produced before the reattach was lost` that bytes buffered under one token are read whole under the next. What it deliberately does not do is wait the real hour: an expired token being refused is the platform’s property, not this contract, and the mechanism the survival rests on is fully exercised without it. The doc half is `docs/EMBEDDING.md`, “The proxy-token reality”, which names start, rotate, poll, ack as a normal sequence rather than a recovery path.
**6. Idle-signal correctness for outbound-tunnel workloads. Shipped, with the naive half ruled out.** The platform measures idleness only by inbound endpoint traffic. Omnigent’s host holds an outbound tunnel and receives none, so auto-suspend can freeze a VM mid-turn; multi-hour agent runs past 400 minutes are the case that hurts.
The parenthetical above — “a trivial periodic authenticated request” — is the thing that does not work, and the reason is measured rather than argued. The endpoint proxy terminates *outside* the VM and forwards over loopback (`docs/PLATFORM.md`), so a request an in-VM process sends to the daemon’s port is generated on the far side of the meter and never passes through it. A keepalive route inside the guest would answer 200 and change nothing, discovered as a suspend during exactly the long run it was added to protect.
So `GET /v1/health` now carries `busy` and `execs`, and the consumer is an orchestrator outside the VM whose own poll *is* the inbound traffic. That also keeps the assertion repeated and explicitly the caller’s, which is what rules out the daemon self-keepaliving: a hung process would otherwise bill silently to the 8-hour ceiling. `busy` is “producing”, not “unfinished” — an exited exec awaiting an ack reads false — and `execs` counts every registered entry so a caller can tell a drained VM from one holding output nobody read. The one thing this section left unmeasured — that a poll from outside does in fact reset the timer — has since been measured twice over: by hand in `docs/PLATFORM.md` (“An outside poll of `/v1/health` does reset the idle timer”), and on every live run by `conformance/run_rs.py`’s `drive_idle_keepalive`, which runs a VM to the edge of a 60-second idle window while polling and watches it survive under the check name `a VM polled from outside outlives its idle window` — then stops polling and watches the same VM suspend, which is the control that proves the survival was the polling.
**7. An eve backend adapter (separate package, later).** Ten session methods over the Node binding makes this platform a pinnable eve backend: real VMs where the consolidator today accepts a pure-JS bash interpreter. Worth doing as its own repo once 1-3 land; it depends on eve’s types, so it can never live here.
## Explicit non-goals
[Section titled “Explicit non-goals”](#explicit-non-goals)
* **Vercel wire compatibility.** The valuable seam is eve’s adapter, not Vercel’s REST surface. Snapshot/fork-on-stop, the heart of Vercel’s persistence model, needs snapshot-to-image, which is the standing AWS platform ask (`docs/STRATEGY.md`), not something this client can build.
* **Multi-container.** `ResourcesList.max = 1` is a platform constant. Harbor treats single-container as an accepted tier; Omnigent needs one container; nothing here changes.
* **Harness provider classes.** The Harbor `BaseEnvironment` subclass, the Omnigent launcher, and the eve backend all import their harness’s packages, so they live in those ecosystems (or standalone adapter repos), never here. This repo’s deliverable is the daemon, the clients, and the published behavior they can rely on. `docs/AGENT-VMS.md` now carries two agent profiles (Claude Code, Codex) as the L3 layer such a class would call: one function that builds, launches, and provisions the VM, one that hands the agent a task. The non-goal still holds for the class itself, because it imports the harness’s packages and this repo does not.
* **GPUs, Windows, dynamic network policy.** Not offered by the platform; harnesses that need them reject the environment up front, which is the correct degradation.
## What this changes next
[Section titled “What this changes next”](#what-this-changes-next)
Items 1 and 3 are CLI work measured in hours and unblock every harness equally. Item 2 is documentation plus a small helper. Item 7 waits for the first three.
Items 4 and 6 have shipped, and the three design decisions this section predicted they would need were the right three. Payload budget: the token and the env share 4096 bytes, checked locally before the launch because neither AWS nor botocore gives a caller a signal in time. Env precedence: the launch env is the base and the per-request map is overlaid, which leaves the existing per-request contract unchanged for anyone who sends no launch env. Busy semantics: producing rather than unfinished, reported to an orchestrator *outside* the VM, because the guest-side keepalive this document floated cannot work against a proxy that terminates outside the guest. Item 5 has shipped too, and it cost no code: the reattach-after-token-rotation path always worked by design, and what this document asked for — a conformance check plus a doc section — is exactly what landed. `drive_token_rotation` and `drive_idle_keepalive` in `conformance/run_rs.py` are the checks; `docs/EMBEDDING.md` carries the operator-facing halves of both, the rotate-mid-run sequence and the outside-poll keepalive ownership.
Reading `readTextFile` from the AI SDK sandbox contract while item 4 was being built turned up one gap this document had missed. That method takes 1-based inclusive `startLine`/`endLine` and returns through EOF when `endLine` is past the end, and `GET /v1/fs/file` had no way to express it — a harness implementing it over this daemon would read whole files and slice them client-side, which on a multi-megabyte file is the transfer this route exists to avoid. The route now takes `start_line` and `end_line` with exactly those semantics, still streamed, with the un-ranged read byte-identical to what it was.
# Business logic
## What counts as business logic here, and why the shape is unusual
[Section titled “What counts as business logic here, and why the shape is unusual”](#what-counts-as-business-logic-here-and-why-the-shape-is-unusual)
There are no users, no orders, and no billing accounts in this codebase. The domain rules are **trap closures**: rules that exist because someone measured AWS Lambda MicroVMs behaving in a way that points away from its own cause, and paid for that measurement once so no caller pays again. A rule here is a refusal that costs a second, standing in for a service answer that costs a build cycle, a poll timeout, or an hour spent auditing a correct IAM policy.
**Scope.** Everything in this file is application-layer. There is no database, no ORM, and no migration, so there are no DB-side invariants to surface. There is no UI, so there is no form-validation layer. Request validation performed by AWS is out of scope *except* where this client duplicates it deliberately — which it does in most places, for a measured reason: read 2026-08-07 out of botocore’s `validate.py`, `VALIDATED_METADATA_ATTRS` is `{'required', 'min', 'document', 'union'}`, so `max`, `pattern`, and `enum` violations are serialized, sent, and answered with a `ValidationException` ([`microvms-core/src/constants.rs:11-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L11-L23)). And this client never reaches that validator at all: it signs with `aws-sigv4` and sends with `reqwest`, so **every** model constraint including `min` is enforced by this crate or by nothing ([`microvms-core/src/constants.rs:25-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L25-L31)). Deleting a local guard on the assumption that the SDK covers the model reopens the constraint with no visible failure.
### The rules are formal requirements first, code second
[Section titled “The rules are formal requirements first, code second”](#the-rules-are-formal-requirements-first-code-second)
`spec/core.symspec.json` carries 51 EARS requirements for `microvms-core`, every one at `status: approved`, in six families:
| Family | Count | What it governs |
| ---------------------- | ----- | --------------------------------------------- |
| `TRAP-1` … `TRAP-13` | 13 | closures over platform behavior that misleads |
| `STATE-1` … `STATE-12` | 12 | the VM lifecycle |
| `COST-1` … `COST-10` | 10 | cost honesty |
| `CLI-1` … `CLI-6` | 6 | the binary’s surface and its exit contract |
| `ARCH-1` … `ARCH-5` | 5 | crate boundaries |
| `BIND-1` … `BIND-5` | 5 | what a language binding may not weaken |
`spec/agentd.symspec.json` adds 6 requirements, all `status: draft`, covering the daemon’s bootstrap and control-token ladder — accept a control request whose token equals the installed one, reject a differing one, reject any control request while none is installed, install on first bootstrap, accept an identical replay, reject a different token.
The spec’s `stateModel` names five variables: `vm_state` over `PENDING / RUNNING / SUSPENDING / SUSPENDED / TERMINATING / TERMINATED`, plus `token_installed`, `image_exists`, `was_terminated`, and `bootstrap_count` bounded `0..3`, all `frame: stable`. Three lifecycle invariants over that model are proved in Z3 by `mise run spec:core`, whose recorded run reports 3 constraints proved under hypotheses, 0 violated, 0 unknown ([`mise.toml:224-226`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L224-L226)) — that task needs a symspec v5 CLI at an absolute path, so it is deliberately outside `mise run check` ([`mise.toml:222-227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L222-L227)). The runnable half is `stateright`, which restates the same three over every interleaving in [`model/src/client.rs:554-569`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L554-L569) and passes under `cargo test -p agentd-model`. One waiver exists, `GTWR_R6_MISSING_UNITS` against TRAP-5, because the linter’s unit list does not include bytes.
### The strength ladder, and how to read the failure-mode column
[Section titled “The strength ladder, and how to read the failure-mode column”](#the-strength-ladder-and-how-to-read-the-failure-mode-column)
Every closure is ranked, strongest first, at [`microvms-core/src/lib.rs:21-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L21-L40):
* **S1, inexpressible** — the mistake cannot be written down: a closed enum, a newtype with no conversion, an absent parameter. An S1 closure cannot regress without a compile error.
* **S2, expressible but rejected** — the mistake can be written, and the client refuses it locally before any control-plane call, with an error naming the `docs/PLATFORM.md` finding. Weaker, because the guard is code that can regress, but the cost is seconds rather than a build cycle. Every boundary where a bare integer or string still has to be judged lands here.
* **S3, correct by default and overridable** — weakest, because it protects only the caller who accepts the default. An S3 closure must state what the override costs. There is exactly one.
Two conventions follow. **Every error message names its finding** rather than restating the constraint, because the guards exist so a reader can reach the measurement ([`microvms-core/src/lib.rs:42-48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L42-L48)). **Every guard has a demonstrated way to fail** — a named plausible edit that must turn a specific test red; “delete the feature and the test fails” does not count ([`microvms-core/src/lib.rs:50-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L50-L57)).
## Validations
[Section titled “Validations”](#validations)
All row citations are the enforcement site. Where a rule is S1, the “failure mode” column says what a caller cannot write rather than what they are told.
### Control-plane request shapes
[Section titled “Control-plane request shapes”](#control-plane-request-shapes)
Nine `require_*` functions in one module, each guarding a member the pinned service model constrains and the SDK does not check.
| Rule | Domain | Citation | Failure mode |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `maximumDurationInSeconds` outside `1..=28800` is refused | Launch | [`microvms-core/src/control/mod.rs:477-494`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L477-L494), constant at [`microvms-core/src/constants.rs:263`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L263) | `ERR_INVALID_ARG` saying a longer session needs a second VM, not a larger number. 28800 is eight hours and the hard ceiling on any one VM’s life |
| `idlePolicy.maxIdleDurationSeconds` under 60 is refused | Launch | [`microvms-core/src/control/mod.rs:748-778`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L748-L778), constant at [`microvms-core/src/constants.rs:241-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L241-L258) | `ERR_INVALID_ARG`. The model states no maximum and the client adds none; the bound that ends a VM’s life is `maximumDurationInSeconds` |
| A `Version` value that is empty, over 2048 characters, or carries whitespace anywhere is refused | Image build, Launch | [`microvms-core/src/control/mod.rs:496-542`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L496-L542), constants at [`microvms-core/src/constants.rs:121`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L121), [`:128`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L128) | Three separate `ERR_INVALID_ARG` messages. The pattern is `[^\s]+`, so a version pasted with a trailing newline satisfies “non-empty” and fails; the message names the character it found |
| A `NonBlankString` member (`codeArtifact.uri`, `baseImageArn`, `nameFilter`, `imageVersion`, `buildId`) that is empty, over 2048 characters, or carries whitespace is refused | Image build | [`microvms-core/src/control/mod.rs:544-596`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L544-L596), constants at [`microvms-core/src/constants.rs:141`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L141), [`:145`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L145) | `ERR_INVALID_ARG` naming the character. A blank `nameFilter` rides in the query string, where it either 400s or filters differently from what was meant |
| An identifier that is empty or over 256 characters is refused | Every operation | [`microvms-core/src/control/mod.rs:598-651`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L598-L651), constants at [`microvms-core/src/constants.rs:174`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L174), [`:183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L183) | `ERR_INVALID_ARG`. An empty identifier is the case that pays for this guard: ten of these members are URI parameters, so an empty one collapses `/microvms/` onto the listing and a `DELETE` on a collapsed path is worse |
| A `RoleArn` under 20 characters, over 2048, or off-pattern is refused | Image build, Launch | [`microvms-core/src/control/mod.rs:653-708`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L653-L708), constants at [`microvms-core/src/constants.rs:209`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L209), [`:212`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L212), [`:225`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L225) | Three messages. The short case says a value that short is almost always a role *name*; the pattern case names the twelve account digits |
| A port of 0 is refused; there is no ceiling branch | Image build, Session | [`microvms-core/src/control/mod.rs:710-746`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L710-L746), constants at [`microvms-core/src/constants.rs:232`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L232), [`:239`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L239) | `ERR_INVALID_ARG`. Zero means “let the kernel choose” to a listener and is not a port the platform can forward to. `PortNumber.max` equals `u16::MAX`, so a ceiling branch would be unreachable — pinned instead by [`microvms-core/src/constants.rs:1244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L1244) |
| A tag key that is empty, over 128 characters, or off-pattern is refused; a tag value over 256 or off-pattern is refused | Image build | [`microvms-core/src/control/mod.rs:780-846`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L780-L846), constants at [`microvms-core/src/constants.rs:186`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L186), [`:189`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L189), [`:206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L206) | `ERR_INVALID_ARG` naming the offending key. An empty tag *value* is legal and an empty key is not, and the two ceilings differ by 2x |
| An image name that is empty, over 64 characters, or outside `[a-zA-Z0-9-_]+` is refused | Image build | [`microvms-core/src/control/mod.rs:848-878`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L848-L878), constants at [`microvms-core/src/constants.rs:104`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L104), [`:112`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L112) | Three messages, because the pattern message (“no dots, no slashes”) misleads for a 70-character name containing neither |
| More than 10 network connectors on a launch is refused | Networking | [`microvms-core/src/control/microvm.rs:391-399`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L391-L399), constant at [`microvms-core/src/constants.rs:290`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L290) | `ERR_INVALID_ARG`. The image-level egress list caps at **1**, not 10 ([`microvms-core/src/constants.rs:292-304`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L292-L304)), pinned by [`microvms-core/src/constants.rs:897`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L897) |
`ControlPlane::run_microvm` runs the identifier, duration, idle-duration, version, and role-ARN guards before it builds a wire body ([`microvms-core/src/control/microvm.rs:356-374`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L356-L374)).
### Image build and Dockerfile agreement
[Section titled “Image build and Dockerfile agreement”](#image-build-and-dockerfile-agreement)
Five guards compare what this client sends against what the caller’s Dockerfile declares. Every one of them defends against the same failure shape: a build that succeeds, a daemon that logs that it started, and an image that still lands in `CREATE_FAILED` naming nothing.
| Rule | Domain | Citation | Failure mode |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `inherit_workdir` against a base that declares no `WorkingDir` and a Dockerfile that sets none is refused | Image build | [`microvms-core/src/control/artifact.rs:234-262`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L234-L262) | `ERR_INVALID_ARG`. Measured 2026-08-05: `al2023-minimal`, `python:3.12-slim`, and `node:20-slim` all leave `WorkingDir` empty, so inheritance inherits `/` and every relative path resolves somewhere the caller did not mean |
| A Dockerfile whose `FROM` is not the selected base image’s `docker_ref` is refused | Image build | [`microvms-core/src/control/artifact.rs:524-548`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L524-L548) | `ERR_INVALID_ARG`. The build runs the Dockerfile on top of `baseImageArn`, so a mismatch builds against a base none of the measured platform behavior describes |
| A Dockerfile whose `AGENTD_PORT` disagrees with the `hooks.port` this client sends is refused; an absent variable is checked against the daemon’s own default of 9000 | Image build | [`microvms-core/src/control/artifact.rs:305-348`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L305-L348) | `ERR_INVALID_ARG`. Silence is not neutral: the daemon keeps its default for an unset variable, so the absent variable produces the failure for a caller who never typed a port |
| A Dockerfile `AGENTD_SSE_KEEPALIVE_SECS` at or above the client’s stream idle timeout is refused | Image build | [`microvms-core/src/control/artifact.rs:405-445`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L405-L445) | `ERR_INVALID_ARG`. Equality is refused too, since an interval equal to the timeout races. The failure it prevents reports the client’s own 60s as though it were the keepalive interval |
| A Dockerfile with no `CMD`, or with a non-empty `ENTRYPOINT`, is refused | Image build | [`microvms-core/src/control/artifact.rs:483-522`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L483-L522) | `ERR_INVALID_ARG`. Weak-form on purpose: it does not check that the `CMD` names a copied path. The unenforceable half — a base image starting its own process before bootstrap — stays with whoever builds the image |
| The daemon entry in the build artifact carries mode `0o755` explicitly | Image build | [`microvms-core/src/control/artifact.rs:1-13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L1-L13), [`:36-39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L36-L39) | Structural. A non-executable binary produces an image whose `CMD` fails, and the symptom is a run-hook timeout that says nothing about permissions |
| The agent token has no path into the build artifact | Image build | [`microvms-core/src/control/artifact.rs:15-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L15-L23) | Unwritable (S1). `build_artifact` has no parameter that could carry one. The artifact becomes a shared image snapshot, so a per-VM secret in it is a secret shared with every VM; a test scans the produced zip’s raw bytes rather than reviewing the API |
### Trap closures — control plane
[Section titled “Trap closures — control plane”](#trap-closures--control-plane)
| Rule | Domain | Citation | Failure mode |
| ------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| TRAP-1: an image-create or run token is derived from a per-attempt nonce; there is no caller-supplied token parameter | Idempotency | [`microvms-core/src/control/token.rs:101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L101), [`:111`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L111), [`:120-148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L120-L148) | Unwritable (S1). The parameter does not exist. Minted at [`microvms-core/src/control/image.rs:187-190`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L187-L190) and [`microvms-core/src/control/microvm.rs:415`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L415) |
| TRAP-1: the scope label is truncated at its **tail**, never its head, and the nonce is never truncated | Idempotency | [`microvms-core/src/control/token.rs:55-71`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L55-L71), [`:150`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L150) | Silent truncation of the label only. 64-byte scope plus an 8-byte hex nonce stays under the 128-character `clientToken` ceiling ([`microvms-core/src/constants.rs:423`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L423)) |
| TRAP-2: an image in `CREATING` past the stall grace with builds listed, non-empty, and **every** build still `PENDING` fails the wait | Image build | [`microvms-core/src/control/image.rs:318-324`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L318-L324), [`:338-390`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L338-L390) | `ERR_BUILD_WEDGED`, naming the `clientToken` replay signature. A `clientToken` is a permanent idempotency key, so a replayed create is a no-op: the image sits in `CREATING`, cannot be deleted, and its only version cannot be dropped. Two images were wedged this way for \~15 hours |
| TRAP-3: guest identity repair is a `bool` intent; the client injects the one accepted enum value `["ALL"]` | Image build | [`microvms-core/src/control/image.rs:181-185`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L181-L185), [`microvms-core/src/constants.rs:276-280`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L276-L280) | Unwritable (S1). There is no capability list a caller can populate, and no way to ask for `CAP_SYS_ADMIN` alone |
| TRAP-4: a connector is an enumerated intent that derives a fully-qualified ARN for the request region | Networking | [`microvms-core/src/control/connector.rs:39-47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L39-L47), [`:60-83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L60-L83) | Unwritable (S1). Two intents (`AllIngress`, `Egress`), no free-form string. `ConnectorIntent::ALL` at [`:54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L54) is the complete set a test can enumerate |
| TRAP-5: a `runHookPayload` over 4096 bytes is refused locally before any control-plane call | Launch | [`microvms-core/src/control/microvm.rs:161-185`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L161-L185), constant at [`microvms-core/src/constants.rs:83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L83) | `ERR_INVALID_ARG` naming the service-model ceiling. Inclusive, measured 2026-08-07: 4096 passes, 4097 fails. Bytes, not characters. `docs/STRATEGY.md`, `docs/TRUST.md`, and the model’s own documentation string all claim 16 KB ([`microvms-core/src/constants.rs:97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L97)), which is wrong by 4x in the dangerous direction — the shape `RunMicrovmRequestRunHookPayloadString` is the authority |
| TRAP-6: a region outside the five that carry MicroVMs is refused before the first control-plane call | Region | [`microvms-core/src/region.rs:38-63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L38-L63), [`:137-164`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L137-L164) | S1 for a held `Region`, S2 at the `FromStr` boundary. `ERR_INVALID_ARG` naming the null-message `AccessDeniedException` finding |
| TRAP-8: a VM reaching a state in `fail_on` before the wanted one fails the wait with state **and** `stateReason` attached | Launch | [`microvms-core/src/control/microvm.rs:461-466`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L461-L466), [`:482-500`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L482-L500) | `ERR_LAUNCH_DIED`. Fails fast rather than polling to the deadline. Both facts, because either alone is unactionable: the state says the VM is gone, the reason is the only evidence that survives it |
| TRAP-10: a `minimumMemoryInMiB` that is not one of the five documented baselines is refused locally | Sizing | [`microvms-core/src/sizing.rs:25-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L25-L31), [`:146-161`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L146-L161) | S1 for a held `SizeClass`, S2 at `from_baseline_mib`. Refused, never snapped to a neighbour: the two plausible service behaviors for 1500 differ in both the memory the guest gets and the rate it is billed at |
| TRAP-11: `CreateMicrovmShellAuthToken` is never called and `SHELL_INGRESS` is never requested | Networking | [`microvms-core/src/control/connector.rs:16-28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L16-L28), [`microvms-core/src/control/mod.rs:27-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L27-L31) | Unwritable (S1). No enum variant renders it and no method on `ControlPlane` calls it. The test counts the calls a full lifecycle makes rather than asserting a refusal |
| Two hook-timeout families cannot be interchanged: run/resume/suspend/terminate cap at 60s, ready/validate at 3600s | Hooks | [`microvms-core/src/hooks.rs:56-82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L56-L82), [`:84-105`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L84-L105), constants at [`microvms-core/src/constants.rs:268`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L268), [`:271`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L271) | Unwritable across families (S1) — no `From`, no shared trait. S2 within a family: `ERR_INVALID_ARG` naming **both** ceilings, because the caller who hits it nearly always picked a build-hook number |
| A hook port outside `1..=65535` is refused | Hooks | [`microvms-core/src/hooks.rs:141-149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L141-L149) | `ERR_INVALID_ARG` naming the model range and version |
| An architecture other than `ARM_64` cannot be requested | Image build | [`microvms-core/src/control/image.rs:168-172`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L168-L172), [`microvms-core/src/constants.rs:282-287`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L282-L287) | Unwritable (S1). The enum has one value, so the field is injected rather than accepted — a field could only ever express a request AWS rejects, after the upload |
| `ENABLED` on all six hooks is a typed enum value, not a `&str` literal | Image build | [`microvms-core/src/control/mod.rs:880-909`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L880-L909) | Compile error. The literal appeared six times with no constant naming either value, so a typo in one was a `ValidationException` on a call made after the artifact upload |
### Trap closures — in-VM session
[Section titled “Trap closures — in-VM session”](#trap-closures--in-vm-session)
| Rule | Domain | Citation | Failure mode |
| ----------------------------------------------------------------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| TRAP-7: the proxy token is read out of the `authToken` **map**, never as a string | Session | [`microvms-core/src/control/microvm.rs:268-283`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L268-L283), [`:300-326`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L300-L326), [`microvms-core/src/session/proxy.rs:14-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L14-L19) | S1: `ProxyToken` exposes no `as_str`, no `Display`, no `Deref`; the auth value comes out through `auth_value()`, which names the header it reads ([`microvms-core/src/session/proxy.rs:177`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L177)). A missing key is `WireKind::AuthTokenMint`, which is retryable |
| TRAP-7: every endpoint request sends **both** `X-aws-proxy-auth` and `X-aws-proxy-port` | Session | [`microvms-core/src/control/microvm.rs:328-337`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L328-L337), [`microvms-core/src/session/proxy.rs:428-434`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L428-L434), [`:436-461`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L436-L461) | Structural: `headers()` returns a two-element array. One without the other is rejected indistinguishably from a bad token, so the header that is wrong is not the header the error mentions |
| A WebSocket handshake carries the same two facts as three subprotocols, minted through the same cache | Session | [`microvms-core/src/session/proxy.rs:39-52`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L39-L52), [`:463-496`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L463-L496) | Structural. The browser `WebSocket` constructor cannot set a header, so the platform moves both facts into `Sec-WebSocket-Protocol` and strips all three before forwarding. A second token path would be a second place TRAP-9 has to be got right |
| TRAP-9: the token is minted inside the request path, through exactly one mint function | Session | [`microvms-core/src/session/proxy.rs:498-506`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L498-L506), [`:508-560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L508-L560) | Structural: `headers`, `headers_for_port`, and `subprotocols` all reach the control plane through `token_for`. A cache miss is two conditions — stale, or out of scope for the requested port |
| A refresh interval at or above the 60-minute ceiling is refused at construction | Session | [`microvms-core/src/session/proxy.rs:356-375`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L356-L375), constants at [`:107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L107), [`:109-111`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L109-L111) | S2, `ERR_INVALID_ARG`. `DEFAULT_REFRESH_AFTER` is 30 minutes — **half** the ceiling, not just under it, because refreshing at fifty-nine minutes puts the expiry inside the window between building the headers and the proxy validating them |
| A mint asks for every port already cached plus the new one | Session | [`microvms-core/src/session/proxy.rs:532-545`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L532-L545) | A superset rather than a replacement. Measured 2026-08-15: a token minted for the agent port does not authorize 8080, and reusing it produces `403 Access to port denied` — whose WebSocket form is an unreasoned 1006 |
### Cost inputs
[Section titled “Cost inputs”](#cost-inputs)
| Rule | Domain | Citation | Failure mode |
| --------------------------------------------------------------------------------------------------------------- | ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| COST-1: every duration carries a `measured` or `projected` provenance label; there is no unlabelled constructor | Cost | [`microvms-core/src/cost.rs:419-425`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L419-L425) | Unwritable (S1). `DurationP` is an enum whose every variant names its provenance; no `From`, no `Default`, both pinned by `compile_fail` doctests at [`:395-408`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L395-L408) |
| COST-2: an estimated dollar amount has no coercion to a bare float | Cost | [`microvms-core/src/cost.rs:546-578`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L546-L578) | Unwritable (S1). Private field, no `From`, no `Into`, no `Deref`. Three `compile_fail` doctests at [`:519-545`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L519-L545), each pinning its own error code |
| COST-3: an unpriced quantity is a distinct `Unpriced` variant carrying a reason, never zero dollars | Cost | [`microvms-core/src/cost.rs:614-625`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L614-L625) | S1 by exhaustive `match`. Zero is a claim about the bill; unpriced is a claim about the documentation |
| COST-6: `gb_decimal` is the only place an `f64` becomes a `Decimal`, and it is fallible | Cost | [`microvms-core/src/cost.rs:138-152`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L138-L152) | `ERR_INVALID_ARG`. A negative size would render as a credit; `NaN`, an infinity, and a magnitude past 28 digits have no decimal reading. `EstimatedUsd::new` takes a `Decimal` so it cannot become a third boundary ([`:553-561`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L553-L561)) |
| COST-9: a rate catalog whose ARM compute line is missing is rejected rather than substituted | Cost | [`microvms-core/src/cost.rs:1191-1199`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1191-L1199), [`:1200-1258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1200-L1258), [`:1268-1302`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1268-L1302) | S1 for direct construction — the rate fields are private and there are exactly two doors. S2 at `from_catalog`, which refuses four ways: a missing ARM line whose x86 sibling is present, a missing line with no sibling, a restated unit, and two products where there was one. The ARM message names the x86 rate it will not substitute and the \~18% error that substituting would introduce |
| A calendar date arriving from outside the crate is validated against its month’s real length | Cost | [`microvms-core/src/cost.rs:232-241`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L232-L241) | S2. `2026-02-30` would otherwise yield a day number for March 2nd and an age two days out |
### Daemon authorization
[Section titled “Daemon authorization”](#daemon-authorization)
| Rule | Domain | Citation | Failure mode |
| ---------------------------------------------------------------------------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A control request whose token equals the installed agent token is accepted | Daemon auth | [`agentd/src/state.rs:241-249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L241-L249), [`agentd/src/auth.rs:75-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L75-L79) | Constant-time byte comparison via `subtle::ConstantTimeEq` ([`agentd/src/auth.rs:28-33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L28-L33)) |
| A control request whose token differs is rejected | Daemon auth | [`agentd/src/auth.rs:78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L78) | 401 |
| A control request while no token is installed is rejected **distinguishably** from a bad credential | Daemon auth | [`agentd/src/state.rs:241-249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L241-L249), [`agentd/src/auth.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L73), [`:76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L76) | 503, never 401 and never 404 — a client maps 404 onto “file not found”, turning a protocol error into a phantom missing artifact |
| A bootstrap request presenting a token while none is installed installs it | Daemon auth | [`agentd/src/state.rs:202-212`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202-L212), [`agentd/src/routes.rs:214-223`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L214-L223) | 200 |
| A bootstrap request presenting a token **identical** to the installed one is accepted | Daemon auth | [`agentd/src/state.rs:213-215`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L213-L215), [`agentd/src/routes.rs:224-229`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L224-L229) | 200. The platform may retry its own hook, and answering 409 would fail a launch that is fine |
| A bootstrap request presenting a **different** token is refused and changes nothing | Daemon auth | [`agentd/src/state.rs:216-218`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L216-L218), [`agentd/src/routes.rs:230-233`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L230-L233) | 409, installed token unchanged |
| A malformed run-hook body, an absent `runHookPayload`, or a payload that does not parse are each refused | Daemon auth | [`agentd/src/routes.rs:182-211`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L182-L211) | 400, never 404. The token and the payload carrying it are never logged; the refusal is loggable because `RunHookError` names a key or a shape and never a value |
| The `Authorization` header is parsed and compared on **raw bytes**, never decoded as UTF-8 first | Daemon auth | [`agentd/src/auth.rs:40-47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L40-L47) | No token extracted, or a mismatch. Never a crash — proved over arbitrary header bytes by [`agentd/tests/proptest_tar.rs:867`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L867), [`:905`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L905), [`:931`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L931), [`:955`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L955) |
| Authorization is decided **before** the request body is polled, and a rejected body is drained under a cap | Daemon auth | [`agentd/src/auth.rs:62-89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L62-L89), cap at [`agentd/src/config.rs:20-24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L20-L24) | An unauthenticated caller cannot make the daemon allocate. Draining lets a client’s pooled connection survive an error response; draining without a cap is itself a denial-of-service vector |
| An unmatched path falls through to the 404 fallback rather than being answered 401 | Daemon auth | [`agentd/src/routes.rs:61-76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L61-L76), [`:78-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L78-L79) | 404. `route_layer` applies the guard only to matched routes: answering 401 for a typo sends a client chasing credentials |
| `ready`, `validate`, `suspend`, and `terminate` hooks answer 200 without regard to bootstrap state | Daemon lifecycle | [`agentd/src/routes.rs:237-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L237-L258), [`:292-295`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L292-L295) | Always 200. `ready` and `validate` are image-*build* hooks called before any instance exists; gating them on a token fails the build rather than the run |
### Daemon filesystem and resources
[Section titled “Daemon filesystem and resources”](#daemon-filesystem-and-resources)
Confinement here is **two layers**, and the second is the kernel’s.
| Rule | Domain | Citation | Failure mode |
| -------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A tar member path that is rooted, carries a prefix component, or pops past depth zero is refused | File transfer | [`agentd/src/fs.rs:198-216`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L198-L216), refusal at [`:682`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L682) | Member rejected, 400. Resolution is lexical component walking, never `realpath`/`canonicalize`. A leading `/` is refused rather than stripped: CPython strips it from a member name, but a caller learns nothing from a rewrite |
| Every member is created relative to one root descriptor with `openat2` and `RESOLVE_BENEATH \| RESOLVE_NO_SYMLINKS \| RESOLVE_NO_MAGICLINKS` | File transfer | [`agentd/src/fs.rs:45-67`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L45-L67) | `ELOOP` for a symlink component, `EXDEV` for a resolution leaving the root; both become a 400 naming the member. This is the layer the lexical one cannot cover: issue #15’s `V/a/..` case is judged in-tree by name and lands one level shallower on disk |
| An absolute symlink or hard-link target is refused outright | File transfer | [`agentd/src/fs.rs:228-231`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L228-L231), [`:726-729`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L726-L729) | Member rejected naming “absolute link target” |
| A relative link target must resolve under the extraction root, from its own base depth | File transfer | [`agentd/src/fs.rs:218-245`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L218-L245) | Member rejected. A symlink resolves from its own directory, a hard link from the archive root — different bases, both confirmed against CPython 3.14’s `_get_filtered_attrs` |
| An in-tree symlink is created as a symlink with its target preserved verbatim | File transfer | [`agentd/src/fs.rs:20-43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L20-L43) | Accepted. Refusing every link member would break `upload_dir` for any skills tree or test directory containing one — a worse outcome than the traversal hole it guarded |
| Device and FIFO members are refused; member count and total uncompressed size are capped | File transfer | [`agentd/src/fs.rs:41-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L41-L42), [`:664`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L664), [`:777`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L777), caps at [`agentd/src/config.rs:37-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L37-L40) | Member or archive rejected, 413. Boundary behavior proved exactly by [`agentd/tests/proptest_tar.rs:776`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L776) |
| The extraction root must be absolute | File transfer | [`agentd/src/fs.rs:1433-1448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1433-L1448) | 400. A relative root resolves against the daemon’s own working directory, which the caller cannot see |
| A write that would take the filesystem below the configured reserve is refused before it starts | Disk | [`agentd/src/disk.rs:66-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L66-L80), [`agentd/src/fs.rs:97-106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L97-L106), [`:804`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L804), reserve at [`agentd/src/config.rs:63-69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L63-L69) | 507 naming the actual free space. Not 500: a 500 is indistinguishable from a daemon defect, so a client retries it — correct for a defect and actively harmful for a full disk. Zero disables the guard |
| Every buffer, output capture, stdin write, linger, TTL, and stream window is bounded | Resources | [`agentd/src/config.rs:11-71`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L11-L71) | Truncation with a marker, or a bounded refusal. A stdin write that blocks past its timeout gives up rather than pinning a connection for the life of the VM |
Confinement is deliberately asymmetric, and the reasoning was argued with a reviewer ([`agentd/src/fs.rs:4-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L4-L18)). The single-file routes `PUT`/`GET /v1/fs/file` are **not** confined to a root. The same bearer token authorizes `POST /v1/exec/start`, which runs arbitrary commands as root by design, so a root prefix would add no security while breaking real behavior: harnesses write credentials into home directories, drop config into `/etc`, and stage scratch in `/tmp`. The confinement that matters is on `PUT /v1/fs/tar`, where member paths come out of an uploaded archive rather than from a caller who named them. That gap is where the entire traversal class lives.
### CLI parser closures
[Section titled “CLI parser closures”](#cli-parser-closures)
| Rule | Domain | Citation | Failure mode |
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| CLI-5: `--memory` is a closed value set over the five documented baselines | CLI | [`microvms-cli/src/cli.rs:231-250`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L231-L250), reasoning at [`:4-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L4-L19) | Unparseable (S1 at the parser). 1500 never reaches a handler. The difference between refusing it at the parser and refusing it in core is a build cycle |
| CLI-5: `--region` is a closed value set over the five MicroVM regions | CLI | [`microvms-cli/src/cli.rs:266-284`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L266-L284) | Unparseable. `--unlisted-region` is the named way out, declared `conflicts_with = "region"` once on a flattened struct so the relationship cannot be forgotten on the twelfth command ([`:298-331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L298-L331)) |
| The domains are spelled out rather than generated, and a test asserts the enum equals the size table | CLI | [`microvms-cli/src/cli.rs:13-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L13-L19), [`microvms-cli/tests/manifest.rs:90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L90) | A domain computed at runtime is invisible to `--help`, to shell completion, and to the manifest’s `choices` field. A sixth size class that does not reach `cli.rs` fails the test rather than shipping unreachable |
| No `--client-token`, `--capabilities`, `--connector`, or `--architecture` flag exists | CLI | [`microvms-cli/src/cli.rs:21-29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L21-L29) | Unwritable. Core has no such parameter, so there is nothing to forward. Absence asserted by [`microvms-cli/tests/thinness.rs:426`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L426) and the manifest cross-check |
| A `microvm cp --mode` conflicts with `--tar`; `--poll` conflicts with every writing flag; `--detach` conflicts with the shapes that must not return early | CLI | [`microvms-cli/src/cli.rs:660`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L660), [`:680`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L680), [`:805`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L805) | Unparseable, pinned by [`microvms-cli/src/cli.rs:1214`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1214), [`:1266`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1266) |
## Invariants
[Section titled “Invariants”](#invariants)
### VM lifecycle (the twelve STATE requirements)
[Section titled “VM lifecycle (the twelve STATE requirements)”](#vm-lifecycle-the-twelve-state-requirements)
Enforced in `microvms-core/src/sandbox.rs`, whose `Lifecycle` enum is the spec’s `vm_state` verbatim and which carries the other four spec variables beside it. Every one of those fields is private and every mutation happens in one of the five lifecycle methods, which is what makes the Z3 and `stateright` proofs proofs about *this struct’s* reachable states ([`microvms-core/src/sandbox.rs:9-17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L9-L17), [`:96-110`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L96-L110), [`:504-520`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L504-L520)).
| Invariant | Where enforced | Citation |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| STATE-1: an accepted launch moves the lifecycle to PENDING and records the image as existing | Application, `Sandbox::run`, after the wire call | [`microvms-core/src/sandbox.rs:698-706`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L698-L706) |
| STATE-2: the platform reporting a successful run hook is what marks the lifecycle RUNNING and the token installed — not the launch call | Application, `Sandbox::run`, after the wait | [`microvms-core/src/sandbox.rs:720-724`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L720-L724) |
| STATE-3: the agent token is installed at most once per VM lifetime | Application, both sides | [`microvms-core/src/sandbox.rs:649-661`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L649-L661) (a second `run` on one sandbox is refused); [`agentd/src/state.rs:202-221`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L202-L221) (the daemon’s one-shot bootstrap) |
| STATE-4: a suspend accepted from RUNNING moves the lifecycle to SUSPENDING before the wait, and after the call | Application, `Sandbox::suspend` | [`microvms-core/src/sandbox.rs:769-778`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L769-L778) |
| STATE-5: no suspend call is issued while the lifecycle is not RUNNING | Application, before the wire | [`microvms-core/src/sandbox.rs:758-767`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L758-L767) |
| STATE-6: the platform reporting suspension complete marks the lifecycle SUSPENDED; a VM that dies while suspending is recorded as terminated instead | Application, `Sandbox::suspend` | [`microvms-core/src/sandbox.rs:790-809`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L790-L809), wanted set at [`microvms-core/src/control/microvm.rs:650`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L650) |
| STATE-7: a resume is issued only from SUSPENDED, reuses the installed token, and re-delivers **no** run-hook payload | Application, `Sandbox::resume` | [`microvms-core/src/sandbox.rs:849-854`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L849-L854), [`:859`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L859) |
| STATE-8: a completed resume invalidates the cached proxy token, through the endpoint the service just reported | Application, `Session::rebind` | [`microvms-core/src/sandbox.rs:875-883`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L875-L883), [`microvms-core/src/session/mod.rs:314-320`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L314-L320) |
| STATE-9: an accepted terminate moves the lifecycle to TERMINATING and records the VM as terminated — **before** the call | Application, `Sandbox::terminate` | [`microvms-core/src/sandbox.rs:945-949`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L945-L949) |
| STATE-10: the platform reporting termination complete marks the lifecycle TERMINATED | Application, `Sandbox::terminate`, only when a wait was asked for | [`microvms-core/src/sandbox.rs:972-976`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L972-L976) |
| STATE-11: a terminated VM never returns to RUNNING, checked before the window check and before any call | Application, `Sandbox::resume` | [`microvms-core/src/sandbox.rs:840-848`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L840-L848) |
| STATE-12: a resume past the launch-time suspended window is refused with the elapsed window named | Application, before `ResumeMicrovm` | [`microvms-core/src/sandbox.rs:856-857`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L856-L857), [`:898-926`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L898-L926) |
| The suspended-window clock is stamped after the suspend call and before the wait, and cleared on a successful resume | Application, `Sandbox` | [`microvms-core/src/sandbox.rs:775-778`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L775-L778), [`:884-887`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L884-L887) |
| The three Z3-proved invariants hold over every interleaving: bootstrap at most once, no suspend outside RUNNING, a terminated VM never reaches RUNNING | `stateright` model | [`model/src/client.rs:554-569`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L554-L569) |
| A locally refused call costs **zero** wire calls — resume after terminate, resume with the window closed, and the payload count matching the launch count are all checked as counters, not as end states | `stateright` model | [`model/src/client.rs:584-598`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L584-L598), [`:623-640`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L623-L640) |
| The installed token is never replaced and survives a suspend/resume cycle | `stateright` model | [`model/src/client.rs:599-616`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L599-L616) |
| `image_exists` is true exactly when a launch was accepted, and a bootstrapped token implies one | `stateright` model | [`model/src/client.rs:570-582`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L570-L582) |
### Daemon
[Section titled “Daemon”](#daemon)
| Invariant | Where enforced | Citation |
| ---------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The launch environment is installed only on the *first* bootstrap, under the token lock | Application, `AppState::bootstrap` | [`agentd/src/state.rs:196-210`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L196-L210). A racer that loses the token cannot win the environment: the winner’s map is in place before any other caller can observe a token installed |
| The agent token and the launch environment are separate parameters, so no code path can move a byte from one into the other | Application, by signature | [`agentd/src/state.rs:187-194`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L187-L194), [`agentd/src/routes.rs:166-177`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L166-L177) |
| An exec id already present in the registry returns success **without spawning a second child**, decided under the registry lock | Application, `agentd` exec | [`agentd/src/exec.rs:363-377`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L363-L377) |
| A poisoned lock is recovered rather than propagated, and the soundness argument is per-lock | Application, `agentd` state | [`agentd/src/state.rs:8-66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L8-L66), [`:76-92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L76-L92) |
| An arbitrary archive never escapes its root and never panics | `proptest`, 256 cases | [`agentd/tests/proptest_tar.rs:566`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L566) |
| A symlink cannot redirect a member that arrives after it | `proptest` | [`agentd/tests/proptest_tar.rs:600`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L600) |
| Plain members and in-tree symlinks are always accepted | `proptest` | [`agentd/tests/proptest_tar.rs:720`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/proptest_tar.rs#L720) |
| The attacker is never authorized; bootstrap is one-shot; only the installed token is accepted; the control API is closed before bootstrap | `stateright` model | [`model/src/lib.rs:443-465`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L443-L465) |
| Output is never released before an ack; a retried start never spawns twice; there is one exec entry per id | `stateright` model | [`model/src/lib.rs:466-481`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L466-L481) |
| Every safety property has a `sometimes` property beside it, so a green run cannot mean a state space that never reached the interesting states | `stateright` model | [`model/src/lib.rs:482-515`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L482-L515) |
### Cross-crate and cost
[Section titled “Cross-crate and cost”](#cross-crate-and-cost)
| Invariant | Where enforced | Citation |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| COST-4: an unpriced line routes the whole total to a lower-bound variant that names its unpriced items | Application, one `Total::of` | [`microvms-core/src/cost.rs:706-751`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L706-L751). `Total::AtLeast` holds the floor *beside* the reasons, and `Add` is implemented only `EstimatedUsd + EstimatedUsd`, so summing an `Amount` is a compile error |
| COST-5: each compute line item is computed from the size-class **baseline**, never from the peak the guest reports | Application, by reachability | [`microvms-core/src/cost.rs:1584-1616`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1584-L1616), accessors at [`microvms-core/src/sizing.rs:184-205`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L184-L205). `compute_lines` reaches only `baseline_gb`/`baseline_vcpu`; reading the peak would overstate the memory line exactly 4x |
| COST-7: a rate table older than 90 days attaches a staleness warning to every report computed from it | Application, on the report | [`microvms-core/src/cost.rs:90-95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L90-L95), [`:961-987`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L961-L987), [`:1510-1512`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1510-L1512). Carried on the report rather than logged, so a library caller with a log filter and a CLI writing only stderr do not each lose it |
| COST-8: the one-week minimum retention floor applies to every snapshot storage line item | Application, a field on the rate row | [`microvms-core/src/cost.rs:104-109`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L104-L109), [`:1627-1657`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1627-L1657) |
| COST-10: every duration in a plan estimate is marked `projected` | Application, by type | [`microvms-core/src/cost.rs:1871-1880`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1871-L1880), [`:1908-1913`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1908-L1913). `PlanUsage` fields are bare `f64` seconds, so there is no field a `Measured` duration could be written into |
| ARCH-2: protocol drift between client and daemon fails compilation | Cargo dependency graph | [`microvms-cli/tests/dependency_direction.rs:179`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L179) |
| ARCH-3 / ARCH-4 / BIND-1: `cli -> core -> protocol`, bindings depend only on core, core depends on neither | Test over `cargo_metadata` | [`microvms-cli/tests/dependency_direction.rs:68`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L68), [`:95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L95), [`:219`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L219) |
| ARCH-5: the CLI exports no library target at all | Test over `cargo_metadata` | [`microvms-cli/tests/dependency_direction.rs:126`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L126) |
| CLI-2: the CLI reaches the control plane and the endpoint proxy only through core, and the guard names *which* seam door was entered | Injected refusing seam | [`microvms-cli/src/guards.rs:403`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L403), [`:487`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L487); source scan at [`microvms-cli/tests/thinness.rs:426`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L426) |
| The CLI’s direct dependency set contains none of the twelve denylisted transport and signing crates | Test over `cargo_metadata` | [`microvms-cli/tests/thinness.rs:96`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L96) |
| Only the envelope module and two named exceptions in `main` write to stdout | Source scan | [`microvms-cli/tests/thinness.rs:503`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L503) |
| CLI-4: one JSON envelope per invocation on stdout, on success, on failure, and on a stream that died before its first event | Spawned-binary test | [`microvms-cli/tests/exit_codes.rs:154`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L154), [`:198`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L198), [`:233`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L233) |
| BIND-5: both bindings preserve provenance-labelled durations, estimate-typed dollars, and the distinct `Unpriced` value | Application, by absent constructors | [`microvms-py/src/cost.rs:9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L9), [`:23-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L23-L27), [`:220-241`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L220-L241); [`microvms-js/src/cost.rs:20-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L20-L31), [`:52-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L52-L57). `new Duration(3600)` is a `TypeError`, `Amount.usd` is null for an unpriced line, and `to_json`/`to_dict` omit the key entirely rather than emitting a null anything permissive sums as zero |
| Every constant in `constants::as_json` is checked against the pinned botocore service model by the build gate (TRAP-12), and the key set is pinned by a test | Build gate plus a key-set test | [`microvms-core/src/constants.rs:33-46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L33-L46), [`:589`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L589), [`:693`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L693) |
| `DEAD_STATES` is a strict subset of `TERMINAL_STATES`, and `SUSPENDED` is terminal but not dead | Application, pinned by test | [`microvms-core/src/constants.rs:448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L448), [`:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L455), [`:878`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L878) |
| The model-backed and tolerated image-ready state sets are disjoint | Application, pinned by test | [`microvms-core/src/constants.rs:431`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L431), [`:441`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L441), [`:941`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L941) |
| A ledger file is removed only when nothing is outstanding; leaked identifiers are recorded **before** the delete is attempted | Application, on disk | [`microvms-cli/src/ledger.rs:1-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L1-L22) |
| CLI-3: every failure class exits with its own integer and `ERR_*` string, distinct from `ERR_UNEXPECTED` | Spawned-binary test plus a classification test | [`microvms-cli/src/exit.rs:78-101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L78-L101), [`microvms-cli/tests/exit_codes.rs:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L29), [`microvms-cli/src/guards.rs:2949`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L2949), [`:3063`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L3063); published table cross-checked at [`microvms-cli/tests/manifest.rs:161`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L161) |
| Retryability is derived from the error kind rather than stored, so the two cannot drift | Application, one `matches!` | [`microvms-core/src/error.rs:111-118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L111-L118), mapping at [`:358-397`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L358-L397) |
The lifecycle is deliberately **runtime-checked rather than typestate** ([`microvms-core/src/sandbox.rs:19-32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L19-L32)). A `Sandbox` returning a `Suspended` handle would make STATE-5’s wrong call a compile error, which is strictly stronger on the ladder. But a type whose Rust identity changes on every transition cannot be one `#[pyclass]`, so it would be re-erased into a runtime-checked enum at the binding boundary — and the binding’s copy is the one most callers hit. The part of the typestate idea that costs nothing is kept: the check happens **before** the wire call, and the test asserts the control-plane call count, which is the observable that distinguishes the two designs.
## Calculations
[Section titled “Calculations”](#calculations)
| Calculation | Inputs | Output | Citation |
| ---------------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Compute cost for a phase, as two separate line items | size class, labelled duration, rate table, phase | vCPU-seconds and GB-seconds line items with estimated dollars | [`microvms-core/src/cost.rs:1584-1616`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1584-L1616) |
| Snapshot storage for a hold, with the retention floor applied | phase, GB, labelled hold, rate table | GB-months line item, note naming the floor when it applied | [`microvms-core/src/cost.rs:1627-1657`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1627-L1657) |
| Snapshot transfer (write on suspend, read on launch or resume) | phase, line, GB, cycle count, rate table | GB line item, no time component | [`microvms-core/src/cost.rs:1661-1683`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1661-L1683) |
| Per-GB-month storage rate, derived from the API’s per-GB-hour quote | catalog entry USD per GB-hour | Decimal USD per GB-month | [`microvms-core/src/cost.rs:1235-1242`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1235-L1242), [`:88`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L88) |
| A report’s total | every line item’s phase and amount | `Total::Exact`, or `Total::AtLeast` with named unpriced lines | [`microvms-core/src/cost.rs:726-751`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L726-L751) |
| Residency ratio: how many times more a running VM costs than a suspended one | two cost reports | Decimal multiplier | [`microvms-core/src/cost.rs:1987-1991`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1987-L1991) |
| Per suspend/resume cycle cost | size class baseline GB, write rate, read rate | `EstimatedUsd` | [`microvms-core/src/cost.rs:1993-1999`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1993-L1999) |
| Break-even suspended hold | size class, rate table | Decimal seconds | [`microvms-core/src/cost.rs:2013-2027`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2013-L2027) |
| Rate table age and staleness verdict | retrieval date, today | days, bool, optional warning text | [`microvms-core/src/cost.rs:961-987`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L961-L987) |
| Proleptic-Gregorian day number, for date subtraction without a date crate | year, month, day | `i64` days since 1970-01-01 | [`microvms-core/src/cost.rs:279-291`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L279-L291) |
| Exact seconds from a `Duration`, without a lossy float step | `std::time::Duration` | `Decimal` seconds | [`microvms-core/src/cost.rs:120-124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L120-L124) |
| Idempotency token assembly | verb, scope label, 8 random bytes | `--<16 hex>` | [`microvms-core/src/control/token.rs:120-148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L120-L148) |
| Connector ARN | intent, region | fully-qualified ARN string | [`microvms-core/src/control/connector.rs:60-83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L60-L83) |
| Available bytes on a write target’s filesystem | path | `u64` bytes | [`agentd/src/disk.rs:66-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L66-L80) |
### Compute cost per phase
[Section titled “Compute cost per phase”](#compute-cost-per-phase)
Both figures read the **baseline**, never the peak ([`microvms-core/src/cost.rs:1593-1597`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1593-L1597)). `vcpu_quantity = baseline_vcpu × seconds`, priced at `rates.vcpu_second()`. `memory_quantity = baseline_gb × seconds`, priced at `rates.gb_second()`. They are two line items rather than one blended GB-second because that is how the pricing page prices them, and a blended figure cannot be reconciled against a Cost Explorer breakdown that keeps them apart ([`microvms-core/src/cost.rs:940-944`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L940-L944)). The guest reports the peak and bursts to it, but the peak is charged only for the seconds above baseline actually consumed; this client cannot observe those seconds, so the peak is left out rather than guessed at. The 2 GB class reports 8 GB in the guest ([`microvms-core/src/sizing.rs:81-86`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L81-L86)), so reading the peak would overstate the memory line exactly 4x.
A suspended VM gets **no compute line at all**, rather than a compute line multiplied by zero. A zeroed line would reappear the moment someone changed how a duration is derived ([`microvms-core/src/cost.rs:1827-1835`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1827-L1835)).
### Snapshot storage with the retention floor
[Section titled “Snapshot storage with the retention floor”](#snapshot-storage-with-the-retention-floor)
`billed_seconds = max(held_seconds, floor_seconds)` where the floor is one week ([`microvms-core/src/cost.rs:1635-1637`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1635-L1637)). Then `quantity = gb × billed_seconds / SECONDS_PER_MONTH`, priced at `rates.storage_gb_month()`. `SECONDS_PER_MONTH` is `2628000`, which is `730 × 3600` — AWS’s own month. It is spelled out because 30-day and calendar-month conventions both give plausible-looking answers that disagree with the worked examples by a few percent ([`microvms-core/src/cost.rs:74-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L74-L80)).
When the floor applies, the note quotes the day count off the rate row rather than dividing by 86,400 beside the message. The rate-row field is the only thing that knows how long the window is, so a division written beside the message would keep saying “7-day” after a rate row moved to a fortnight ([`microvms-core/src/cost.rs:1639-1648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1639-L1648), [`:918-927`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L918-L927)). Not applying the floor would understate the one line item that dominates a create-and-destroy suite by four orders of magnitude: a 2 GB image deleted after sixty seconds still bills about four cents ([`microvms-core/src/cost.rs:104-109`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L104-L109)).
### Break-even suspended hold
[Section titled “Break-even suspended hold”](#break-even-suspended-hold)
The least trivial formula in the module ([`microvms-core/src/cost.rs:2013-2027`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2013-L2027)). `running_per_sec = baseline_vcpu × vcpu_rate + baseline_gb × gb_rate`. `storage_per_sec = baseline_gb × storage_gb_month / SECONDS_PER_MONTH`. `churn = baseline_gb × (write_rate + read_rate)`.
The solve is **piecewise**, because the storage charge behaves differently on each side of the minimum-retention window. Inside the window the storage charge is a constant, so the equation is linear in the hold and the candidate is `(churn + floor_sec × storage_per_sec) / running_per_sec`. Past the window, storage grows with the hold and the slope changes, so the answer is `churn / (running_per_sec − storage_per_sec)` instead. Solving only one branch returns a number in the wrong regime.
This is the figure a pool scheduler needs, and a bare “100x cheaper” headline does not show it. Below the break-even hold, suspending and resuming costs more than leaving the VM running, so the conclusion the comparison supports is “avoid churn” rather than “avoid residency” ([`microvms-core/src/cost.rs:1993-1995`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1993-L1995)).
### Why the sizing table is data, not arithmetic (TRAP-13)
[Section titled “Why the sizing table is data, not arithmetic (TRAP-13)”](#why-the-sizing-table-is-data-not-arithmetic-trap-13)
Every documented peak is exactly four times its baseline, which makes `baseline × 4` look like the obvious simplification. The sizing module must not compute it that way. The regularity belongs to AWS’s current table rather than to the service’s contract, so a sixth row that broke the pattern would get the pattern applied to it, reporting a burst ceiling the service does not offer ([`microvms-core/src/sizing.rs:13-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L13-L23)).
So `SIZE_CLASSES` ([`microvms-core/src/sizing.rs:64-99`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L64-L99)) is the only place any of the twenty numbers appears, and every accessor reads a row out of it through one lookup. To make the guard falsifiable, `row_in` and `class_for_baseline_in` take the table as a **parameter** ([`microvms-core/src/sizing.rs:247`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L247), [`:255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L255)) so a test can drive the accessors over a table whose peak is *not* 4x its baseline. A test against the shipped table could not tell a lookup from an arithmetic derivation, because every shipped peak is 4x.
### Two float boundaries, and only two (COST-6)
[Section titled “Two float boundaries, and only two (COST-6)”](#two-float-boundaries-and-only-two-cost-6)
`seconds_of` ([`microvms-core/src/cost.rs:120-124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L120-L124)) is exact rather than a lossy conversion: a `Duration` is a whole-seconds count plus a nanosecond remainder, both integers, and the nanosecond division is by a power of ten.
`gb_decimal` ([`microvms-core/src/cost.rs:138-152`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L138-L152)) is the **only** place an `f64` becomes a `Decimal`. It goes through the float’s decimal *string* rather than its binary value, because `Decimal::try_from(0.1f64)` would carry the binary error into every downstream figure. It is fallible rather than lossy: `NaN`, an infinity, and a magnitude past 28 digits have no decimal reading, and a money figure derived from one of them would be a number nobody could reconcile. `EstimatedUsd::new` deliberately takes a `Decimal` and not an `f64`, so it cannot become a third boundary ([`microvms-core/src/cost.rs:553-561`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L553-L561)).
### The day-number formula
[Section titled “The day-number formula”](#the-day-number-formula)
`CalendarDate::day_number` ([`microvms-core/src/cost.rs:279-291`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L279-L291)) is an era-based proleptic-Gregorian conversion so that date subtraction needs no date crate. The year is shifted to be March-based, which puts February’s variable length last; `719468` is the day number of 1970-01-01 in the era count. Rate-table age is `today.days_since(retrieved)` over that number ([`microvms-core/src/cost.rs:962-965`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L962-L965)).
## Policy and gates
[Section titled “Policy and gates”](#policy-and-gates)
* **Absent parameters are the primary policy mechanism.** The strongest closures are things that do not exist: no `client_token` parameter (TRAP-1), no capability list (TRAP-3), no `SHELL_INGRESS` variant and no `mint_shell_auth_token` method (TRAP-11), no architecture field, no conversion between hook-timeout families, no `f64` accessor on a dollar figure (COST-2), no unlabelled duration constructor (COST-1), no `Measured` field on a plan (COST-10). Where a requirement is about an impl being *absent*, the check is a program that fails to build — as a `compile_fail` doctest with a **pinned error code**, because a bare `compile_fail` passes for any build failure including a typo in the test. [`microvms-core/src/cost.rs:395-408`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L395-L408), [`:519-545`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L519-L545).
* **Local refusal before the wire, always.** Every S2 guard fires before the first control-plane call, and where the distinction is observable the test asserts the control-plane **call count** rather than just the error. [`microvms-core/src/sandbox.rs:29-32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L29-L32); [`model/src/client.rs:584-589`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/client.rs#L584-L589) states it as a model property.
* **One guard list, runnable before the artifact upload.** `create_image` runs after the caller has already uploaded the artifact, so a locally-refusable request refused from inside it has cost the caller the S3 PUT. `ControlPlane::preflight` is that same list extracted as a pure function of the request, and `create_image` delegates to it rather than keeping a copy, so the two cannot drift. A caller who skips preflight loses only the upload. [`microvms-core/src/control/image.rs:144-158`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L144-L158), [`:206-266`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L206-L266); asserted with a zero-call count at [`microvms-cli/src/guards.rs:1468`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L1468).
* **Every refusal names its measurement.** A guard’s error message cites the `docs/PLATFORM.md` section rather than restating the constraint, because the guards exist so a reader can reach the measurement. [`microvms-core/src/lib.rs:42-48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L42-L48).
* **The one S3 escape hatch, and what it costs.** `Region::unlisted` accepts a region this client has not seen carry MicroVMs, because AWS adds regions faster than the list is re-read and a client that refuses a region AWS just launched in is its own kind of wrong. The override costs the diagnostic: if the region does not carry MicroVMs, the first control-plane call answers `AccessDeniedException` with a null message and the caller spends an hour reading a correct IAM policy. It is a visible enum **variant** rather than a hidden flag, so a reader of a call site can see someone opted in, and a supported name handed to it comes back as its proper variant so nothing downstream handles two spellings. [`microvms-core/src/region.rs:38-63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L38-L63), [`:94-113`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L94-L113).
* **The region list can be wrong in two directions, and one direction is worse.** A *missing* region refuses a launch AWS would have accepted — the safer direction, still wrong, and what `unlisted` is for. An *extra* region is worse, because it re-opens the null-message trap for a name nothing will reject. No API answers the question: `get_available_endpoints` returns an empty list while `get_available_regions` returns all 34 Lambda regions, so the list is kept by hand and keeping it right is the whole correctness condition. `eu-central-1` does not carry MicroVMs and is one of three regions measured returning the null-message denial (2026-08-07). [`microvms-core/src/region.rs:20-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L20-L31).
* **A best-effort probe only raises when it has the evidence.** TRAP-2’s stall probe fires once, past the grace, and raises only when builds are listed, the list is non-empty, and **every** build is `PENDING`. A listing failure returns `Ok(())`, so the wait continues and the caller gets a plain timeout. Unknown is not empty, and a wedge claim made on a throttled API call sends the reader after the wrong cause. The state field read is `buildState`, not `state`, and the deserializer refuses the other spelling. [`microvms-core/src/control/image.rs:338-390`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L338-L390).
* **Fail-fast state sets are per-call-site, not global.** `wait_for_state` takes `fail_on` as a parameter because different callers fail on different states. Suspend *wants* `SUSPENDED` and tolerates `TERMINATED`; resume must pass the *dead* states only, because failing on `SUSPENDED` would fail every resume — that is the state the call is made from. This is why `constants.rs` carries both `TERMINAL_STATES` and `DEAD_STATES`. [`microvms-core/src/control/microvm.rs:440-456`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L440-L456), [`microvms-core/src/sandbox.rs:860-873`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L860-L873).
* **Token minting lives inside the retry path, and a mint failure is retryable.** A proxy token capped at sixty minutes and minted once at construction expires mid-trial, and the rejection is indistinguishable from a dead daemon. Refresh is at **half** the ceiling rather than just under it, because refreshing at fifty-nine minutes puts the expiry inside the window between building the headers and the proxy validating them. A control-plane throttle at minute thirty must not kill a trial that is otherwise healthy. [`microvms-core/src/session/proxy.rs:21-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L21-L37), [`:107-111`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L107-L111), [`:547-560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L547-L560).
* **Credentials never reach a log line, by construction rather than by care.** `RunHookPayload` and both `ProxyToken` types have hand-written `Debug` impls that print the byte count or the header names instead of the value. Because `RunMicrovmRequest` keeps its derive, every struct and error chain that formats one inherits the behavior. The values that do carry a credential — the header pairs and the subprotocol strings — are returned to a caller and stored on nothing, so there is no type whose `Debug` could leak one; a caller that logs what it was handed is outside what the module can prevent, and both accessors say so. [`microvms-core/src/control/microvm.rs:70-82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L70-L82), [`:285-298`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L285-L298), [`microvms-core/src/session/proxy.rs:54-60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L54-L60). The daemon logs the launch-env variable **count**, never the keys or values ([`agentd/src/routes.rs:214-221`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L214-L221)).
* **Each authorization failure has its own status code.** The daemon answers 503 while no token is installed, 401 for a wrong token, 409 for a bootstrap conflict, and 400 for a malformed hook. It never answers 404, because clients map 404 onto “file not found” and turn a protocol error into a phantom missing artifact. [`agentd/src/auth.rs:69-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L69-L79).
* **A retried bootstrap of the identical token is success, not a conflict.** The platform may retry its own hook, and answering 409 there would fail a launch that is fine. [`agentd/src/state.rs:94-106`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L94-L106), [`agentd/src/routes.rs:224-229`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L224-L229).
* **The bootstrap route is unauthenticated on purpose, and its defense is arity.** The platform has no credential to present, and its request arrives over loopback indistinguishably from an in-VM process. The defense is that the route can only succeed once. [`agentd/src/routes.rs:166-171`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L166-L171).
* **Build hooks are ungated on purpose.** `ready` and `validate` are image-*build* hooks called before any instance exists and therefore before any token has been delivered. Gating them on bootstrap state fails the build rather than the run, which is a confusing place to discover the mistake. [`agentd/src/routes.rs:237-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L237-L258), [`microvms-core/src/control/mod.rs:880-886`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L880-L886).
* **Health is reachable before bootstrap, and `busy` is the orchestrator’s signal rather than a keepalive.** A client needs the contract before it holds a token. And the platform measures idleness by inbound traffic through the endpoint proxy, which terminates *outside* the guest, so a request an in-VM process sends to this port never reaches the thing counting traffic — an in-guest keepalive route would keep nothing alive and would be discovered as broken by a multi-hour run auto-suspending mid-work. So an orchestrator outside the VM polls, which is real inbound traffic, and reads `busy` to decide whether to keep polling. The assertion of liveness is repeated and is the caller’s. [`agentd/src/routes.rs:297-340`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L297-L340).
* **A resume that finds no installed token is logged loudly rather than treated as routine.** Measured 2026-08-05 in us-east-1: the in-memory agent token, the filesystem, exec records, and backgrounded processes all survive a suspend/resume cycle, and the endpoint URL is unchanged. So the normal case is a VM that needs nothing, and the absence of a token would mean the resume behaved like a cold start. The one thing that does change is the guest’s view of time: it observes the whole suspension as a single jump, so any timeout, lease, or TLS session a running command holds expires at once. [`agentd/src/routes.rs:260-290`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L260-L290).
* **A poisoned lock is recovered rather than propagated, and the reasoning is per-lock.** The daemon is the only channel into the VM — no SSH, no supervisor, no console — so `.expect()` on a poisoned mutex converts one handler bug into a permanently unreachable VM. The `token` lock is sound in the strong sense: every write is a whole-value assignment, and recovery cannot *install* a token, so poisoning is not a bootstrap bypass. `launch_env` is the same argument and carries nothing that authorizes anything. The `execs` lock is sound for a narrower reason — the map is not left internally corrupt by a panic in a caller’s closure, but one exec entry may be semantically inconsistent, which limits the blast radius to one exec id. [`agentd/src/state.rs:8-66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L8-L66), [`:76-92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L76-L92).
* **A panic is caught at the outermost layer, and that is narrow on purpose.** Without the catch, a panicking handler drops the connection and the client sees a transport error it cannot distinguish from a dead VM; with it the client gets a 500 and the connection survives. It does not undo the panic, and any `std::sync::Mutex` the handler held is now poisoned — which is why it pairs with the recovery above. [`agentd/src/routes.rs:86-95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L86-L95).
* **The version header is stamped outside `route_layer`, so it covers every response.** Handler bodies, the 401/503 the auth middleware returns before a handler runs, the 413 the body-limit layer injects, and the 404 fallback. A version header a client only sometimes receives is one it cannot use as a precondition. [`agentd/src/routes.rs:78-85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L78-L85).
* **The extractor-level body limit is disabled and the wire-level layer is the real cap.** `DefaultBodyLimit` does not apply to bodies consumed as a stream, so keeping both would silently truncate JSON control bodies at 2 MiB while leaving tar uploads unbounded. [`agentd/src/routes.rs:71-76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L71-L76).
* **Teardown never raises, and order matters.** `Sandbox::terminate` returns a `TeardownReport` rather than a `Result`. It runs where a caller’s `finally` would, and an error raised there replaces the original failure, which is the one worth reading. The order is VM, then image (retrying, because an image in `CREATING` refuses deletion), then the log group **last**, because the service can recreate a group deleted before its image. The log group is *named* rather than deleted: CloudWatch Logs is not in this crate’s dependency set. [`microvms-core/src/sandbox.rs:45-53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L45-L53), [`:932-935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L932-L935), [`:986-1010`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L986-L1010).
* **There is no `Drop` that tears down.** Rust has no context manager and `Drop` cannot await. A blocking `Drop` would deadlock inside a runtime and a spawning one would race process exit. So `Drop` only warns, naming the id, and the rule is that a caller calls `terminate` explicitly. [`microvms-core/src/sandbox.rs:55-60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L55-L60), [`:1060-1061`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L1060-L1061).
* **Leaked identifiers are recorded before the delete is attempted, not after.** Recording after loses the identifier when the process dies inside the call, which is exactly the interrupt case the ledger exists for. The file is removed only when nothing is outstanding, because a leftover file is how `microvm ls` knows there is something to tell the operator about. For a wedged image and a service-created log group the identifier **is** the remedy; there is no second way to find them. [`microvms-cli/src/ledger.rs:1-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L1-L22); CLI-6’s teardown-on-interrupt guarded at [`microvms-cli/src/guards.rs:824`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L824), [`:917`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L917).
* **Exec idempotency is opt-in, which deliberately inverts TRAP-1’s shape.** The default is a generated exec id, because `microvm exec` is one shot and an id reused by accident means the second invocation is answered from the first’s record. The *stable* id is the flag, and what it buys is a retry that is safe across the caller’s own restart. This differs from a control-plane `clientToken`, whose replay wedges an image permanently and which this CLI does not have at all. [`microvms-cli/src/cli.rs:634-652`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L634-L652), [`agentd/src/exec.rs:363-377`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L363-L377); both halves guarded at [`microvms-cli/src/guards.rs:2004`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L2004), [`:2045`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L2045).
* **Local constants are checked against the pinned service model in the build gate (TRAP-12), and the key names are a contract with a script.** `constants::as_json` publishes every hardcoded constraint keyed with the names `scripts/check-model-drift.py` reads, and the key set is pinned by a test — because a rename here does not fail compilation, it makes a check silently stop comparing. The two values no model states, `MICROVM_REGIONS` and `SIZE_CLASSES`, are compared against pinned literals in the script instead, since a value compared only against itself passes by construction. The gate hard-fails when `MODEL_API_VERSION` disagrees with the service directory it resolves, rather than skipping: a constraint checked against a different API version is a constraint that was not checked. [`microvms-core/src/constants.rs:33-46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L33-L46), [`:52-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L52-L57), [`:589`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L589), [`:693`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L693).
* **A disk write is refused before it starts rather than after ENOSPC.** ENOSPC arrives after the filesystem is already full, so by then every other writer in the VM is broken too, including the ones that cannot report anything. It also arrives as a generic io error, so the caller cannot distinguish “the disk is full” from “the daemon is broken” — and retrying is correct for the second case and makes the first worse. [`agentd/src/disk.rs:4-30`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L4-L30).
* **A rate table’s staleness warning is a fallback rather than the primary defence.** The warning can only say that nobody has looked; a drift check against the Pricing API is what tells you whether a rate moved. Ninety days is the same order as the interval at which AWS has historically restructured Lambda pricing, and the cost of the warning when nothing changed is one line of output. [`microvms-core/src/cost.rs:50-56`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L50-L56), [`:90-95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L90-L95).
* **A rate table is all-or-nothing, and a fetched one is authoritative on rates while still hand-read on rules.** A partial table would price a run at less than it costs with no way for the caller to see which field was left stale. The catalog prices line items and says nothing about the one-week storage minimum, a per-request charge, a billing increment, or a free tier, so the retention floor is carried from the constant rather than from the fetch — dropping it there would understate a create-and-destroy suite by four orders of magnitude. [`microvms-core/src/cost.rs:1191-1199`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1191-L1199), [`:1253-1258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1253-L1258).
* **Four `WireKind`s collapse onto one exit code deliberately, and `Unauthorized` is not one of them.** A shell cannot act differently on “the daemon rejected the request on its merits” in four flavours, and a caller that can reads `data.kind`. But a 401’s remedy is a credential rather than a wait, so it maps to `ERR_CREDENTIALS`: retrying a 401 forever and failing a launch that was 200 ms from ready are the two mistakes the classification exists to prevent. A `match` on a closed enum has no ordering to get wrong. [`microvms-core/src/error.rs:358-397`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L358-L397), [`microvms-cli/src/exit.rs:8-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L8-L9), [`:41`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L41).
## See also
[Section titled “See also”](#see-also)
* [impact analysis](impact-analysis.md) — 23 shared source citations
* [contract map](contract-map.md) — 22 shared source citations
* [debugging guide](debugging-guide.md) — 17 shared source citations
* [processes](../behavior/processes.md) — 14 shared source citations
* [public api](../../reference/public-api.md) — 13 shared source citations
# Contract map
> When module A passes something to module B, what is B really expecting?
When module A passes something to module B, what is B really expecting?
## What counts as a contract here
[Section titled “What counts as a contract here”](#what-counts-as-a-contract-here)
Three tiers, all of which cross a boundary a single `cargo build` cannot fully police:
1. **A Rust type or constant declared in one workspace crate and named by at least one other.** The dependency edges are `cli -> core -> protocol`, `bindings -> core`, `agentd -> protocol`, asserted as *equalities* over the metadata by [`microvms-cli/tests/dependency_direction.rs:68-125`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L68-L125) — a violation is a test failure, not a convention.
2. **A shape that crosses a language boundary**, where no compiler checks either side: the HTTP/SSE wire format, the `--json` envelope the Python conformance suite parses, the generated Python stub, the generated Node declarations.
3. **A contract stated as a machine-checked assertion rather than as a type**: the generated `docs/schema.json`, the 51 EARS requirements in `spec/core.symspec.json`, and the cross-language agreement tests such as `every_rate_byte_matches_the_python_literal` ([`microvms-core/src/cost.rs:2179-2196`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2179-L2196)).
Ranking is by distinct consumer *file* count, measured with `rg -l` over qualified paths and `rg -o … | uniq -c` over occurrences. `agentd` counts as one consumer of every `protocol` type, because [`agentd/src/exec.rs:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90) re-exports the whole `protocol::exec` surface in one block. Every client-side consumer reaches the same types through `microvms_core::protocol::…`, because [`microvms-core/src/lib.rs:77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L77) re-exports the crate — which is what lets `microvms-cli` name wire types while its allowlisted dependency set ([`microvms-cli/tests/thinness.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L66)) contains six entries and none of them is `protocol`.
### Gate coverage, which is not symmetric
[Section titled “Gate coverage, which is not symmetric”](#gate-coverage-which-is-not-symmetric)
Three surfaces in this repo are generated from Rust and consumed by something that cannot read Rust. Two have a regenerate-then-diff gate; one does not.
| Surface | Producer | Gate |
| --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docs/schema.json` | `agentd/src/bin/schema.rs` | [`mise.toml:168-173`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L168-L173), `cargo run -p agentd --bin schema -- --check` |
| `microvms-py/microvms.pyi` + `microvms-py/py.typed` | `scripts/generate-py-stubs.py` | [`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195), `mise run stubs:check` |
| `microvms-js/index.d.ts` | `napi build --platform` ([`microvms-js/package.json:12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L12)) | **none** |
`microvms-js/index.d.ts` is gitignored at `.gitignore:29` alongside `index.js` and `*.node`, with the rationale at `.gitignore:23-26` (a platform-specific binary must not be committed). The declarations file is not a binary and shares the exclusion anyway, so nothing in `mise tasks` regenerates and diffs it. [`mise.toml:181-189`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L181-L189) spells out why the Python side is gated — `microvms-py/py.typed` promises a type checker the package is typed, so a stale stub “degrades to confidently wrong” rather than to unchecked. The Node package makes the same promise via `"types": "index.d.ts"` ([`microvms-js/package.json:7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L7)) with no equivalent check. There is no `--check` mode in either toolchain: `git diff --exit-code` after regeneration is the whole state of the art.
***
## microvms\_core::ErrorKind — the coarse failure taxonomy
[Section titled “microvms\_core::ErrorKind — the coarse failure taxonomy”](#microvms_coreerrorkind--the-coarse-failure-taxonomy)
**Producer:** [`microvms-core/src/error.rs:126-159`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L126-L159) (enum), [`:166-180`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L166-L180) (`ALL`), [`:187-203`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L187-L203) (`code`)
**Consumer(s):**
* [`microvms-cli/src/exit.rs:140-156`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L140-L156) — `Exit::for_kind`, a total match that turns a fourteenth kind into a compile error rather than a fall-through to `ERR_UNEXPECTED`.
* [`microvms-cli/src/exit.rs:486-503`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486-L503) — asserts the CLI’s `EXIT_TABLE` and `ErrorKind::ALL` are the same thirteen classes with byte-identical code strings.
* [`microvms-py/src/errors.rs:129-145`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L129-L145) — `exception_for`, one Python exception type per kind.
* [`microvms-js/src/errors.rs:143-149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L143-L149) — `error_codes()`, enumerated from `ErrorKind::ALL` rather than transcribed.
* Raise sites across core: [`microvms-core/src/sandbox.rs:68`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L68), [`microvms-core/src/hooks.rs:156`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L156), [`microvms-core/src/sizing.rs:266`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L266), [`microvms-core/src/cost.rs:2124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2124), plus `control/{artifact,image,microvm,mod,transport}.rs` and `session/{exec,http,mod,proxy,sse}.rs`.
* The daemon’s own uses: [`agentd/src/disk.rs:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L69), [`agentd/src/exec.rs:776`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L776), [`agentd/src/fs.rs:1107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L1107), [`agentd/src/identity.rs:411`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L411).
Count: 36 non-declaring files, from `rg -l '\bErrorKind\b' --type rust --glob '!microvms-core/src/error.rs'`.
**Shape:**
```rust
1
pub enum ErrorKind {
2
/// No handler claimed this — a bug in this crate, not the platform.
3
Unexpected,
4
/// Refused locally, before any AWS call. Every trap closure lands here.
5
InvalidArg,
6
/// Transient. Run the identical request again.
7
Retryable,
8
/// An identity is wrong or absent; waiting will not fix it.
9
Credentials,
10
/// The daemon rejected the request on its merits.
11
Protocol,
12
/// The image build was never scheduled — the `clientToken` replay signature.
13
BuildWedged,
14
/// The MicroVM reached a terminal state before RUNNING; read `stateReason`.
15
LaunchDied,
16
/// The launch-time suspended window passed, so there is nothing to resume.
17
WindowClosed,
18
/// A control-plane failure with no more specific class.
19
Platform,
20
/// A client-side deadline elapsed. The VM and the exec are untouched.
21
Timeout,
22
/// Interrupted after launch; teardown ran and any leak is named in the payload.
23
Interrupted,
24
/// A prerequisite is missing.
25
Precondition,
26
/// The sandbox worked and the command in it exited non-zero.
27
///
28
/// Its own class because it is the one failure that means nothing is wrong with
29
/// the platform, the credentials, or this client — a CI caller needs to tell
30
/// "your tests failed" from "we never got a VM", and one shared class cannot say
31
/// both.
32
ExecFailed,
33
}
```
**Assumptions consumers make:**
* **The mapping to exit integers is injective, and consumers rely on that.** [`microvms-cli/src/exit.rs:512-525`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L512-L525) asserts no two kinds collapse onto one exit row, and names the plausible edit it exists to catch (routing `Precondition` to `InvalidArg`).
* **The `ERR_*` string is the branch key, not the integer.** [`microvms-core/src/error.rs:182-186`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L182-L186) states it: a shell reads `$?`, an agent parsing `--json` reads `code` and should never keep an integer table.
* **`ALL` is in exit-code order, and two independent hand-written tables depend on that order.** [`microvms-core/src/error.rs:434-452`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L434-L452) and [`microvms-cli/src/exit.rs:406-433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L406-L433) both spell the thirteen codes out as literals, deliberately: “a generated list would agree with a typo” ([`microvms-core/src/error.rs:430-431`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L430-L431)).
* **Retryability is derived, never stored.** [`microvms-core/src/error.rs:116-118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L116-L118) reads the kind; [`microvms-core/src/error.rs:399-417`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L399-L417) keeps a second, test-only table so the two can be compared rather than trusted.
* **The Python exception hierarchy is one-to-one with the kinds and rooted at one base**, so `except MicrovmError` catches everything ([`microvms-py/src/errors.rs:4-9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L4-L9)).
* **Node callers cannot read `.code`.** [`microvms-js/src/errors.rs:16-38`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L16-L38) records the measured collapse: `code="ERR_INVALID_ARG"` on a sync export, `code="GenericFailure"` through a Promise rejection. The contract is `err.cause.message`, which is exactly the `ERR_*` string on every path ([`microvms-js/src/errors.rs:70-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L70-L80)). Restates `.erpaval/solutions/api-patterns/napi-async-collapses-error-codes.md`.
**Drift risk:** adding a fourteenth kind is forced into three exhaustive matches ([`microvms-core/src/error.rs:188`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L188), [`microvms-cli/src/exit.rs:141`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L141), [`microvms-py/src/errors.rs:130`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L130)) but **not** into `ALL`, so a variant added without an `ALL` entry compiles and silently vanishes from `error_codes()` ([`microvms-js/src/errors.rs:143-149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L143-L149)) and from every consumer that enumerates the catalog. The cross-check at [`microvms-cli/src/exit.rs:486-492`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486-L492) catches it only when an `EXIT_TABLE` row is added in the same change. Mitigation: assert `ErrorKind::ALL.len()` against a literal alongside the thirteen spelled codes at [`microvms-core/src/error.rs:434-452`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L434-L452).
## microvms\_core::Region — the closed region set, S1 closure
[Section titled “microvms\_core::Region — the closed region set, S1 closure”](#microvms_coreregion--the-closed-region-set-s1-closure)
**Producer:** [`microvms-core/src/region.rs:44-63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L44-L63) (enum), [`:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L73) (`MICROVM_REGIONS`), [`:107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L107) (`unlisted`), [`:137-146`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L137-L146) (`FromStr`)
**Consumer(s):**
* [`microvms-cli/src/cli.rs:33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L33) — the `--region` value, parsed into the enum at the CLI edge.
* [`microvms-cli/src/commands/doctor.rs:17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L17), [`microvms-cli/src/guards.rs:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L29), [`microvms-cli/src/seam.rs:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L29)
* [`microvms-py/src/region.rs:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/region.rs#L11), [`microvms-py/src/sandbox.rs:309`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/sandbox.rs#L309)
* [`microvms-js/src/region.rs:8`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/region.rs#L8), [`microvms-js/src/sandbox.rs:59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/sandbox.rs#L59), [`microvms-js/src/lib.rs:46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/lib.rs#L46)
* [`microvms-core/src/lib.rs:81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L81) (re-export), [`microvms-core/src/cost.rs:850`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L850), [`microvms-core/src/sandbox.rs:465`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L465), plus `control/{artifact,connector,image,microvm,mod,transport}.rs`
* [`microvms-core/tests/live_pagination.rs:59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/live_pagination.rs#L59), [`microvms-core/tests/live_versions.rs:37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/live_versions.rs#L37)
Count: 20 non-declaring files, from `rg -l '\bRegion\b' --type rust --glob '!microvms-core/src/region.rs'`.
**Shape:**
```rust
1
pub enum Region {
2
UsEast1,
3
UsEast2,
4
UsWest2,
5
EuWest1,
6
ApNortheast1,
7
/// A region this client has not seen carry MicroVMs.
8
///
9
/// **You are opting into the null-message trap.** If this region does not run
10
/// MicroVMs, the first control-plane call answers `AccessDeniedException` with a
11
/// null message and you will spend the next hour reading an IAM policy that is
12
/// correct. Constructible only through [`Region::unlisted`], which says so at
13
/// the call site.
14
///
15
/// It exists because AWS adds regions faster than this list is re-read, and a
16
/// client that refuses a region AWS has just launched in is its own kind of
17
/// wrong. The override costs exactly the diagnostic above.
18
Unlisted(String),
19
}
```
**Assumptions consumers make:**
* **`Unlisted` is a visible variant, not a hidden flag**, so a `match` over regions cannot forget the case exists and a reader of a call site can see that someone opted into the trap ([`microvms-core/src/region.rs:40-43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L40-L43)).
* **`Region` is not `Copy`.** It carries a `String` in `Unlisted`, so it derives `Clone, Debug, Eq, Hash, PartialEq` only ([`microvms-core/src/region.rs:44`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L44)). Consumers that hold a region across an `async` boundary clone it; `RateTable::region()` returns `&Region` for the same reason ([`microvms-core/src/cost.rs:868-870`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L868-L870)).
* **The region label is priced, not cosmetic.** [`microvms-core/src/cost.rs:864-867`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L864-L867) measures the consequence: a Tokyo caller reading the us-east-1 table understates snapshot write by 22.6%, and staleness checking would never surface it.
* **The five-region list is measurement-backed and cannot be model-checked.** [`scripts/check-model-drift.py:52-58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L52-L58) states that no service model names the regions and that the two botocore calls that look like substitutes disagree with each other, so `PINNED_REGIONS` in that script is a deliberate second reader rather than a self-comparison.
**Drift risk:** AWS launching a sixth MicroVM region leaves every caller on the `Unlisted` path, which works but discards the null-message diagnostic that is the whole reason the enum exists. Mitigation: the region list is the one constant with no model to check it against, so re-read it whenever `docs/PLATFORM.md` gains a dated region finding and update both [`microvms-core/src/region.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L73) and `scripts/check-model-drift.py`’s `PINNED_REGIONS` in the same commit.
## microvms\_core::session::Session — the in-VM control API handle
[Section titled “microvms\_core::session::Session — the in-VM control API handle”](#microvms_coresessionsession--the-in-vm-control-api-handle)
**Producer:** [`microvms-core/src/session/mod.rs:183-188`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L183-L188) (struct), [`:190-224`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L190-L224) (constructors), [`:466-474`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L466-L474) (`SessionBuilder`)
**Consumer(s):**
* [`microvms-py/src/session.rs:8`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L8), [`microvms-py/src/exec.rs:482`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/exec.rs#L482), [`microvms-py/src/runtime.rs:12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/runtime.rs#L12), [`microvms-py/src/sandbox.rs:632`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/sandbox.rs#L632)
* [`microvms-js/src/session.rs:7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L7), [`microvms-js/src/exec.rs:319`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/exec.rs#L319), [`microvms-js/src/process.rs:190`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/process.rs#L190), [`microvms-js/src/sandbox.rs:28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/sandbox.rs#L28)
* [`microvms-cli/src/seam.rs:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L18), [`microvms-cli/src/guards.rs:28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L28), [`microvms-cli/src/commands/attached.rs:39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L39), [`microvms-cli/tests/thinness.rs:242`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L242)
* [`microvms-core/src/sandbox.rs:535`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L535) (`session()`), [`:648`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L648) and [`:837`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L837) — `run` and `resume` hand back `&mut Session`; plus `microvms-core/src/control/ops.rs` and `microvms-core/src/session/{http,proxy}.rs`
* [`microvms-core/tests/turmoil_client.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L66)
Count: 17 non-declaring files, from `rg -l '\bSession\b' --type rust --glob '!microvms-core/src/session/mod.rs'`.
**Shape:**
```rust
1
/// The control API of one running MicroVM.
2
pub struct Session {
3
transport: Arc,
4
endpoint: String,
5
port: u16,
6
}
```
**Assumptions consumers make:**
* **Constructing a session does not probe the VM.** [`microvms-core/src/session/mod.rs:203-206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L203-L206) makes this explicit: “do I have a session” and “is the VM up” are different questions with different answers during a launch, so a probing constructor would conflate them.
* **`agent_token()` is readable but never printed.** [`microvms-core/src/session/mod.rs:191-200`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L191-L200) makes it public because a reattaching caller needs it; the hand-written `Debug` at [`:454-463`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L454-L463) drops it. Restates `.erpaval/solutions/best-practices/credential-structs-never-derive-debug.md`.
* **`Session::direct` is a supported shape, not a test escape hatch** ([`microvms-core/src/session/mod.rs:217-222`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L217-L222)) — the conformance path and every local-binary test go through it, so proxy-auth headers being absent is a valid state rather than a bug.
* **Token minting happens inside the request path**, which is what makes a long run survive the 60-minute proxy-token lifetime; `SessionBuilder::with_minter` / `with_proxy_auth` decide the schedule and the latter wins ([`microvms-core/src/session/mod.rs:476-490`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L476-L490)).
* **The HTTP backend is a replaceable seam** ([`microvms-core/src/session/mod.rs:492-497`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L492-L497)), which is what lets `microvms-core/tests/turmoil_client.rs` drive the real client under simulated network faults. Restates `.erpaval/solutions/api-patterns/axum-listener-trait-enables-turmoil.md`.
**Drift risk:** the port a session was built with is also the port its proxy token is scoped to ([`microvms-core/src/session/mod.rs:499-504`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L499-L504)), so a consumer that changes the agent port without rebuilding the session gets a token scoped to the old port and a 401-shaped failure that reads as a credential problem. Mitigation: keep `with_port` the only way to set it, so the scope and the header are assigned from one value.
## microvms\_core::WireKind — the fine taxonomy, where 400 and 404 stay different
[Section titled “microvms\_core::WireKind — the fine taxonomy, where 400 and 404 stay different”](#microvms_corewirekind--the-fine-taxonomy-where-400-and-404-stay-different)
**Producer:** [`microvms-core/src/error.rs:218-268`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L218-L268) (enum), [`:272-286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L272-L286) (`ALL`), [`:292-308`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L292-L308) (`as_str`), [`:315-331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L315-L331) (`status`), [`:343-356`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L343-L356) (`from_status`), [`:366-397`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L366-L397) (`error_kind`)
**Consumer(s):**
* [`microvms-cli/src/envelope.rs:321-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L321-L339) — writes `data.kind` from it.
* [`microvms-cli/src/exit.rs:336-365`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L336-L365) — keys the remedy suggestion on it where two conditions share an exit code.
* [`microvms-cli/src/exit.rs:534-560`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L534-L560) — pins which five collapse onto `ERR_PROTOCOL`.
* [`microvms-cli/src/guards.rs:2470`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L2470)
* [`microvms-py/src/errors.rs:161-164`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L161-L164) — sets `.wire_kind` on the raised exception.
* [`microvms-js/src/errors.rs:70-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L70-L80), [`:152-158`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L152-L158) — the cause’s cause, and `wire_kinds()`.
* [`microvms-core/src/lib.rs:79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L79) (re-export), plus `control/{microvm,transport}.rs` and `session/{exec,files,http,mod,proxy,sse}.rs`
* [`microvms-core/tests/turmoil_client.rs:63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L63)
Count: 15 non-declaring files, from `rg -l '\bWireKind\b' --type rust --glob '!microvms-core/src/error.rs'`.
**Shape:**
The thirteen variants are listed at [`microvms-core/src/error.rs:219-268`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L219-L268). The load-bearing member is the status table, because it is where a consumer’s 400-versus-404 distinction is either preserved or lost:
```rust
1
pub fn from_status(status: u16) -> Option {
2
match status {
3
400 => Some(WireKind::ProtocolError),
4
401 => Some(WireKind::Unauthorized),
5
404 => Some(WireKind::NotFound),
6
408 => Some(WireKind::RequestTimeout),
7
409 => Some(WireKind::Conflict),
8
410 => Some(WireKind::StdinClosed),
9
413 => Some(WireKind::TooLarge),
10
503 => Some(WireKind::NotBootstrapped),
11
s if s >= 500 => Some(WireKind::ServerError),
12
_ => None,
13
}
14
}
```
**Assumptions consumers make:**
* **There is deliberately no generic 4xx fallback.** [`microvms-core/src/error.rs:336-342`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L336-L342) names the defect a fallback would reintroduce — a 4xx mapped to `NotFound` made a protocol typo look like a missing file, and it hid for a full review round. Asserted at [`microvms-core/src/error.rs:519-529`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L519-L529): 402, 403, 405, 418, 429, 451 must all resolve to `None`.
* **5xx *does* fall back, and 503 is the one exception.** [`microvms-core/src/error.rs:535-540`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L535-L540) — “come back in a moment” is not “the daemon broke”.
* **`status()` and `from_status()` are inverses wherever both are defined**, asserted at [`microvms-core/src/error.rs:546-556`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L546-L556), and four variants deliberately have no status (`Transport`, `AuthTokenMint`, `ExecTimeout`, `OutputGap` — [`:326-329`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L326-L329)).
* **The `as_str` strings are Python exception class names, not a re-spelling.** [`microvms-core/src/error.rs:288-291`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L288-L291) — the conformance oracle compares against them, and [`conformance/run_rs.py:187-196`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L187-L196) reads them out of `data.kind`.
* **`None` is information.** [`conformance/run_rs.py:189-193`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L189-L193) states that an absent `data.kind` means the client refused before any call. The CLI preserves that by inserting the key only when a wire kind exists ([`microvms-cli/src/envelope.rs:326-328`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L326-L328)), and [`microvms-core/src/error.rs:562-567`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L562-L567) asserts a local reject carries none.
* **Exactly five variants are retryable**, named as literals so a sixth added by mistake fails a test rather than a retry loop ([`microvms-core/src/error.rs:485-501`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L485-L501)).
**Drift risk:** a new variant added to `WireKind` reaches `error_kind()` as a compile error (the match is closed) but reaches `from_status` silently — a status the daemon starts using with no row in that table maps to `None` and surfaces as something other than the daemon’s own decision. Mitigation: `status()`/`from_status()` inverse test at [`microvms-core/src/error.rs:546-556`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L546-L556) catches it only if the new variant declares a status, so declare one whenever the daemon does.
## microvms\_core::sandbox::Sandbox — the product surface and its state machine
[Section titled “microvms\_core::sandbox::Sandbox — the product surface and its state machine”](#microvms_coresandboxsandbox--the-product-surface-and-its-state-machine)
**Producer:** [`microvms-core/src/sandbox.rs:422-442`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L422-L442) (struct), [`:97-131`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L97-L131) (`Lifecycle`), [`:551-935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L551-L935) (the transitions)
**Consumer(s):**
* [`microvms-cli/src/commands/lifecycle.rs:6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L6), [`microvms-cli/src/guards.rs:27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L27), [`microvms-cli/src/ledger.rs:94`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L94), [`microvms-cli/src/seam.rs:9`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L9), [`microvms-cli/tests/thinness.rs:240`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L240)
* [`microvms-py/src/sandbox.rs:6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/sandbox.rs#L6), [`microvms-py/src/session.rs:27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L27), [`microvms-py/src/runtime.rs:12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/runtime.rs#L12)
* [`microvms-js/src/sandbox.rs:6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/sandbox.rs#L6), [`microvms-js/src/session.rs:7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L7), [`microvms-js/src/region.rs:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/region.rs#L11)
Count: 14 non-declaring files, from `rg -l '\bSandbox\b' --type rust --glob '!microvms-core/src/sandbox.rs'`.
**Shape:**
```rust
1
/// The symspec's `vm_state`, verbatim.
2
///
3
/// Six states and no others, which is the S1 half of this module: a lifecycle held as a
4
/// `String` would let `"RUNNING "` and `"Running"` both exist, and every guard below would
5
/// have to decide which it meant.
6
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
7
pub enum Lifecycle {
8
/// The initial state, and the state a launch is accepted into (STATE-1).
9
Pending,
10
/// The run hook answered with a success status and the token is installed (STATE-2).
11
Running,
12
/// A suspend was accepted and the platform has not yet reported it complete (STATE-4).
13
Suspending,
14
/// The platform reported suspension complete (STATE-6).
15
Suspended,
16
/// A terminate was accepted (STATE-9).
17
Terminating,
18
/// The platform reported termination complete (STATE-10).
19
Terminated,
20
}
```
```rust
1
pub struct Sandbox {
2
control: Arc,
3
image: Option,
4
microvm: Option,
5
session: Option,
6
7
// ── the symspec's five variables ─────────────────────────────────────────
8
lifecycle: Lifecycle,
9
token_installed: bool,
10
image_exists: bool,
11
was_terminated: bool,
12
bootstrap_count: u32,
```
**Assumptions consumers make:**
* **Every field is private; the contract is the accessor set.** `lifecycle()`, `token_installed()`, `image_exists()`, `was_terminated()`, `bootstrap_count()` at [`microvms-core/src/sandbox.rs:500-520`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L500-L520) are named after the symspec’s five state variables, so a consumer asserting against the formal model reads them rather than reconstructing state.
* **`Suspended` is still billing.** `Lifecycle::is_live` at [`microvms-core/src/sandbox.rs:126-131`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L126-L131) includes `Pending | Running | Suspending | Suspended`, which is what a `Drop` warning is for. That is a *different* question from `constants::TERMINAL_STATES` ([`microvms-core/src/constants.rs:448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L448)), which lists `SUSPENDED`/`SUSPENDING` as states a launch wait must stop on.
* **The suspended window is local knowledge, not readable from the platform.** [`microvms-core/src/sandbox.rs:435-436`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L435-L436) — `GetMicrovm` does not return it, so a consumer that reconstructs a `Sandbox` from a `GetMicrovm` response cannot answer “is the resume window still open”.
* **`terminate` returns a report and never raises.** [`microvms-core/src/sandbox.rs:332-334`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L332-L334), [`:935`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L935) — it runs where a `finally` would, so a consumer must inspect `TeardownReport::leaked()` ([`:364-366`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L364-L366)) rather than trusting the absence of an error.
* **`undeleted` carries identifiers, not a boolean**, because “a leak nobody can name is a leak nobody can clean up” ([`microvms-core/src/sandbox.rs:336-348`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L336-L348)), and the build log group lands there unconditionally: this crate cannot delete it.
* **`image_deleted: Option`** distinguishes “deletion was not asked for” from “deletion failed” ([`microvms-core/src/sandbox.rs:355`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L355)). Restates `.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md`.
* **`Debug` omits the agent token** ([`microvms-core/src/sandbox.rs:444-448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L444-L448)), so a consumer logging a sandbox does not leak the credential.
**Drift risk:** the six `Lifecycle` variants and the six `MICROVM_STATES` wire strings are two readers of one AWS fact, and [`microvms-core/src/constants.rs:341-347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L341-L347) states plainly that a wire string cannot be exhaustively matched. A state AWS respells fails the model gate (`scripts/check-model-drift.py`) and a subset test, but does not fail to compile. Mitigation: keep `scripts/check-model-drift.py` in `mise run check` — it needs no network and no credentials because the model is a file inside botocore.
## protocol::exec::StreamKind — one offset space, two channels
[Section titled “protocol::exec::StreamKind — one offset space, two channels”](#protocolexecstreamkind--one-offset-space-two-channels)
**Producer:** [`protocol/src/exec.rs:76-99`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L76-L99)
**Consumer(s):**
* [`agentd/src/exec.rs:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90) (re-exported into the daemon’s own namespace)
* [`microvms-core/src/session/sse.rs:243`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L243) — the `ExecEvent::Output` payload field.
* [`microvms-cli/src/commands/attached.rs:354-355`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L354-L355) — the two-arm map to `"stdout"` / `"stderr"`.
* [`microvms-py/src/session.rs:600-603`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L600-L603) — published as `streamKinds` from `StreamKind::ALL`.
* [`microvms-js/src/session.rs:582-586`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L582-L586) — the same list, built as a JSON array.
* [`microvms-js/src/process.rs:172-174`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/process.rs#L172-L174) — `is_stderr`.
* [`microvms-core/tests/turmoil_client.rs:850`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L850)
**Shape:**
```rust
1
/// Which pipe a streamed chunk came from. Both share one offset space, so a
2
/// client holds one cursor rather than two that can disagree about ordering.
3
#[derive(Clone, Copy, Debug, Deserialize, Eq, JsonSchema, PartialEq, Serialize)]
4
#[serde(rename_all = "snake_case")]
5
pub enum StreamKind {
6
Stdout,
7
Stderr,
8
}
9
10
impl StreamKind {
11
/// Both kinds, in the order the daemon documents them.
12
///
13
/// Same reason as [`Phase::ALL`]: the bindings publish this closed set, and a
14
/// list they spell themselves is a list the enum can outgrow.
15
pub const ALL: [StreamKind; 2] = [StreamKind::Stdout, StreamKind::Stderr];
16
17
/// The wire spelling — the exact string serde writes under `rename_all` above.
18
pub const fn as_str(self) -> &'static str {
19
match self {
20
StreamKind::Stdout => "stdout",
21
StreamKind::Stderr => "stderr",
22
}
23
}
24
}
```
**Assumptions consumers make:**
* **One shared byte-offset space across both pipes.** [`protocol/src/exec.rs:76-78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L76-L78) states it, and every reconnect depends on it: a client resumes with a single `?offset=N` ([`protocol/src/exec.rs:171-178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L171-L178)). Restates `.erpaval/solutions/architecture-patterns/byte-offset-cursor-is-what-makes-reconnect-work.md`.
* **A gap is attributed to the stream a *later* frame named**, not to one the gap frame carried — [`microvms-js/src/process.rs:159-169`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/process.rs#L159-L169) names this as the field a reader would guess wrong. So a consumer demultiplexing into two channels cannot attribute a gap without look-ahead.
* **`is_stderr` is a boolean predicate, not an exhaustive match.** [`microvms-js/src/process.rs:172-174`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/process.rs#L172-L174) is `matches!(stream, StreamKind::Stderr)`. A third variant would be classified as stdout with no compile error. Every other consumer matches exhaustively.
* **`as_str` and serde’s `rename_all` must agree**, asserted for every variant at [`protocol/src/exec.rs:318-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L318-L339), with `ALL` held complete by wildcard-free matches rather than by a length check ([`protocol/src/exec.rs:313-316`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L313-L316)).
**Drift risk:** a third channel added to `StreamKind` compiles against [`microvms-js/src/process.rs:172`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/process.rs#L172) and silently routes to stdout. Mitigation: rewrite `is_stderr` as a total match returning the channel, so the Node binding fails to build.
## protocol::exec::Phase — the exec lifecycle, and its two redundant name tables
[Section titled “protocol::exec::Phase — the exec lifecycle, and its two redundant name tables”](#protocolexecphase--the-exec-lifecycle-and-its-two-redundant-name-tables)
**Producer:** [`protocol/src/exec.rs:22-54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L22-L54)
**Consumer(s):**
* [`agentd/src/exec.rs:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90) (re-export)
* [`microvms-core/src/session/exec.rs:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L69) (`ExecResult::phase`), [`:86-91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L86-L91) (`done()`), [`:264-269`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L264-L269), [`:689`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L689)
* [`microvms-cli/src/commands/attached.rs:442-448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L442-L448) (`phase_name`)
* [`microvms-py/src/session.rs:596-599`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L596-L599) — published as `phases` from `Phase::ALL`
* [`microvms-js/src/session.rs:577-581`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L577-L581) — the same list
* [`microvms-core/tests/turmoil_client.rs:954`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L954)
**Shape:**
```rust
1
#[derive(Clone, Copy, Debug, Deserialize, Eq, JsonSchema, PartialEq, Serialize)]
2
#[serde(rename_all = "snake_case")]
3
pub enum Phase {
4
/// Child spawned, still running (or its pipes still held by a grandchild).
5
Running,
6
/// Child exited and output is buffered and readable.
7
Exited,
8
/// Caller acked; output has been released and the entry awaits collection.
9
Acked,
10
}
11
12
impl Phase {
13
/// Every phase, in lifecycle order.
14
///
15
/// Public because a client that publishes the closed set — both bindings do, in
16
/// their `session_constants` — needs the list from the type rather than a spelled-out
17
/// copy that goes stale the first time a phase is added. The round-trip test below
18
/// holds `ALL` complete by exhaustive match.
19
pub const ALL: [Phase; 3] = [Phase::Running, Phase::Exited, Phase::Acked];
20
21
/// The wire spelling — the exact string serde writes under `rename_all` above.
22
///
23
/// Here rather than in each client because two bindings each grew their own
24
/// three-arm match over this enum; a variant renamed on the wire must change
25
/// exactly one table, and the test below is what keeps this one equal to serde's.
26
pub const fn as_str(self) -> &'static str {
27
match self {
28
Phase::Running => "running",
29
Phase::Exited => "exited",
30
Phase::Acked => "acked",
31
}
32
}
33
}
```
**Assumptions consumers make:**
* **`Exited` and `Acked` both mean finished.** [`microvms-core/src/session/exec.rs:86-91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L86-L91) defines `done()` as `Exited | Acked`, so a consumer must not treat `Acked` as an error state.
* **`Acked` means the output is already gone.** [`microvms-core/src/session/exec.rs:681-693`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L681-L693) returns the *ack* response rather than a post-ack poll, because a poll after the ack reports `acked` with no output — named a “silent empty-output bug” at [`:685`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L685).
* **The daemon keeps a separate exit marker precisely because an ack takes the `Outcome`.** [`agentd/src/exec.rs:92-99`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L92-L99) — after an ack, `result` is `None` again and is no longer usable as “has this exec finished?”, so a stream attaching then would wait on a channel that never carries another message.
* **`as_str` is meant to be the only phase-name table, and two consumers spell their own anyway.** The bindings comply ([`microvms-py/src/session.rs:596-599`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L596-L599), [`microvms-js/src/session.rs:577-581`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L577-L581)), but [`microvms-core/src/session/exec.rs:264-269`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L264-L269) and [`microvms-cli/src/commands/attached.rs:442-448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L442-L448) each hand-write all three strings. Those matches are exhaustive, so a new *variant* is a compile error — a renamed *wire spelling* is not, because `as_str` and serde would move together under the test at [`protocol/src/exec.rs:318`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L318) while the two copies keep emitting the old string.
* **`ALL` is in lifecycle order**, and the CLI’s `phase_name` deliberately avoids `Debug` because `Debug` emits `Running` where the wire carries `running` ([`microvms-cli/src/commands/attached.rs:437-441`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L437-L441)).
**Drift risk:** renaming a phase on the wire (a `#[serde(rename)]` on a variant) updates `as_str` under compiler pressure from [`protocol/src/exec.rs:318`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L318) but leaves the two hand-spelled tables emitting the old string, so the CLI envelope and a core timeout message would disagree with the daemon’s own JSON. Mitigation: route both call sites through `Phase::as_str` and delete the local tables.
## protocol::health::Health — the liveness answer, with two defaulted fields
[Section titled “protocol::health::Health — the liveness answer, with two defaulted fields”](#protocolhealthhealth--the-liveness-answer-with-two-defaulted-fields)
**Producer:** [`protocol/src/health.rs:10-89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L10-L89) (`Health`), [`:92-102`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L92-L102) (`DiskHealth`)
**Consumer(s):**
* [`agentd/src/routes.rs:19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L19) — `pub use protocol::health::{DiskHealth, Health};`
* [`microvms-core/src/session/mod.rs:329`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L329) — `pub async fn health(&self) -> Result`
* [`microvms-py/src/session.rs:59-85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L59-L85) — `PyHealth::wrap`
* [`microvms-js/src/session.rs:84-97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L84-L97) — `Health::wrap`
* [`microvms-core/tests/turmoil_client.rs:876`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L876)
**Shape:**
Quoted in full, because the field doc comments *are* the contract here — every one of them names a monitor behaviour a consumer would otherwise get wrong.
```rust
1
/// `GET /v1/health` response.
2
#[derive(Debug, Deserialize, JsonSchema, Serialize)]
3
pub struct Health {
4
// The daemon's own version, distinct from `PROTOCOL_VERSION`. `Cow` so the
5
// daemon reports its `CARGO_PKG_VERSION` borrowed while a client deserializes
6
// into an owned string.
7
//
8
// Not a doc comment, deliberately: schemars publishes doc comments as
9
// `description` and `docs/schema.json` is byte-compared, so adding one here is a
10
// schema change. The field was undocumented before the extraction and stays so.
11
pub version: Cow<'static, str>,
12
pub bootstrapped: bool,
13
/// Free space on the daemon's working filesystem, and the reserve it is judged
14
/// against.
15
///
16
/// Reported so disk pressure is something an orchestrator *watches* rather than
17
/// something it discovers from a failed write. anthropics/claude-code#59856
18
/// filled two 10 GB disks to 100% with never-collected session directories and
19
/// the first symptom was `useradd: No space left on device` — by which point
20
/// every writer in the sandbox was already broken. A number on a health endpoint
21
/// is what makes that curve visible while there is still time to act.
22
///
23
/// `None` when free space could not be measured, which is deliberately distinct
24
/// from zero: unmeasurable is not full, and a monitor that conflated them would
25
/// page on a missing `statvfs`.
26
pub disk: Option,
27
/// Whether any startup identity repair step failed. True means the VM is serving
28
/// with a value from the shared image still in place — a duplicate machine-id or
29
/// boot_id — which is a security-relevant condition an operator may want to
30
/// drain the VM over, but is never a reason for the daemon to refuse to serve.
31
pub identity_degraded: bool,
32
/// False when identity repair was switched off by config. Distinguished from a
33
/// repair that ran and found nothing so a monitor can tell "opted out" from
34
/// "nothing to do".
35
pub identity_repaired: bool,
```
The two defaulted fields, and the reason the asymmetry is deliberate ([`protocol/src/health.rs:44-88`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L44-L88)):
```rust
1
/// `#[serde(default)]`, unlike every field above it, and the asymmetry is not an
2
/// oversight. The daemon is baked into an image while the client is installed
3
/// separately, so a current client routinely talks to a daemon from whenever that
4
/// image was built — and a required field would make `health()` fail outright
5
/// against a daemon that predates it, turning a missing signal into an
6
/// unreachable VM. False is also the right absence: a daemon that cannot say
7
/// whether it is busy has not asserted that it is.
8
#[serde(default)]
9
pub busy: bool,
10
/// How many execs are registered, in any phase.
11
///
12
/// Alongside `busy` because the two answer different questions and a monitor
13
/// wants both: `busy: false, execs: 0` is a fresh or drained VM, while
14
/// `busy: false, execs: 7` is a VM holding seven unacked results that somebody
15
/// still has to collect. Terminating the second loses output nobody read.
16
///
17
/// Defaulted for the same reason as `busy`: a client routinely talks to a daemon
18
/// baked into an older image, and zero is the honest reading of a daemon that
19
/// does not report a count.
20
#[serde(default)]
21
pub execs: usize,
22
}
```
```rust
1
pub struct DiskHealth {
2
/// Bytes available to an unprivileged writer, from `statvfs` `f_bavail`.
3
pub available_bytes: u64,
4
/// Bytes that must stay free before a write is refused. Zero means the guard is
5
/// disabled.
6
pub reserve_bytes: u64,
7
/// Whether a write would be refused right now. Precomputed rather than left to
8
/// the client, so every consumer applies the same comparison the write path does.
9
pub under_pressure: bool,
10
}
```
**Assumptions consumers make:**
* **`busy: false` is not an assertion of idleness.** `busy` and `execs` are the only `#[serde(default)]` fields ([`protocol/src/health.rs:75`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L75), [`:87`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L87)), and the reason at [`:68-74`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L68-L74) is that the daemon is baked into an image while the client installs separately — so `false`/`0` is also what a pre-feature daemon returns. A required field would turn a missing signal into an unreachable VM.
* **Polling from outside the VM is the keepalive, and the daemon must not self-keepalive.** [`protocol/src/health.rs:46-61`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L46-L61) — the platform measures idleness by inbound traffic through a proxy that terminates outside the guest, so in-guest traffic cannot reset the idle timer.
* **`disk: None` is not `disk: 0`.** Asserted at [`protocol/src/health.rs:111-128`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L111-L128): unmeasurable is not full, and a monitor that conflated them would page on a missing `statvfs`. Both bindings preserve the distinction by flattening into three `Option`s ([`microvms-py/src/session.rs:77-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L77-L79), [`microvms-js/src/session.rs:89-91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L89-L91)).
* **`busy` and `execs` answer different questions.** [`protocol/src/health.rs:78-86`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L78-L86) — `busy: false, execs: 7` is a VM holding seven unacked results, and terminating it loses output nobody read. Asserted at [`:134-151`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L134-L151).
* **`version` is deliberately undocumented.** [`protocol/src/health.rs:12-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L12-L18) is a `//` comment, not a doc comment, because schemars publishes doc comments as `description` and `docs/schema.json` is byte-compared — adding one is a schema change.
* **The Node binding narrows `execs: usize` to `i64`** ([`microvms-js/src/session.rs:95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L95)), which is what `#[napi]` can express; a count above `i64::MAX` is not reachable.
**Drift risk:** a new `Health` field without `#[serde(default)]` makes `health()` fail outright against a daemon baked into an older image, which reads to a caller as an unreachable VM rather than a version skew. Mitigation: default every field added after the first release, and take `busy`’s doc comment ([`protocol/src/health.rs:68-74`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L68-L74)) as the rule.
## protocol::exec::StartRequest — the exec start body
[Section titled “protocol::exec::StartRequest — the exec start body”](#protocolexecstartrequest--the-exec-start-body)
**Producer:** [`protocol/src/exec.rs:103-139`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L103-L139)
**Consumer(s):**
* [`agentd/src/exec.rs:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90) (re-export; the daemon’s extractor target)
* [`microvms-core/src/session/mod.rs:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L380) — `pub async fn run(&self, req: protocol::exec::StartRequest)`
* [`microvms-cli/src/commands/lifecycle.rs:859`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L859) — `pub fn start_request(spec: StartSpec<'_>) -> microvms_core::protocol::exec::StartRequest`
* [`microvms-py/src/session.rs:338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L338), [`:400`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L400) — two construction sites
* [`microvms-js/src/session.rs:142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L142) — `fn into_request(self, command: Either>) -> protocol::exec::StartRequest`
**Shape:**
```rust
1
pub struct StartRequest {
2
/// Caller-minted idempotency key. Harbor retries, and a retry must not
3
/// produce a second child.
4
pub exec_id: String,
5
/// argv when `shell` is false, or the script when it is true.
6
pub command: Vec,
7
#[serde(default)]
8
pub shell: bool,
9
/// Omitted means inherit the daemon's working directory. See the module docs.
10
#[serde(default)]
11
pub cwd: Option,
12
#[serde(default)]
13
pub env: HashMap,
14
/// Numeric uid to demote to. Optional; omitted means run as the daemon's own
15
/// user.
16
#[serde(default)]
17
pub user: Option,
18
#[serde(default)]
19
pub group: Option,
20
/// Wall-clock budget. Validated before the child spawns — the predecessor
21
/// raised on a bad value inside the waiter thread, by which point the child
22
/// was already running and became an orphan.
23
#[serde(default)]
24
pub timeout_sec: Option,
```
**Assumptions consumers make:**
* **`exec_id` is the idempotency key, and a retry must not spawn twice** ([`protocol/src/exec.rs:105-107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L105-L107)). A consumer that mints a fresh id on retry loses that.
* **`stdin: false` is the default and matters.** [`protocol/src/exec.rs:128-138`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L128-L138) — a child holding an open stdin pipe nobody writes to blocks forever the first time it reads, and `/bin/sh`, `git`, and any tool that probes for input behave differently against a pipe than against `/dev/null`. Writing without it is a 409 ([`microvms-js/src/session.rs:111-112`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L111-L112)).
* **`command` is never split on whitespace.** [`microvms-py/src/session.rs:566-572`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L566-L572) states the rule: splitting turns a path containing a space into two arguments nobody meant; `shell=True` is how a caller asks for a script.
* **`timeout_sec` is validated before the spawn**, not inside the waiter — the predecessor raised late and orphaned a running child ([`protocol/src/exec.rs:123-126`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L123-L126)).
* **Every defaulted field can be omitted**, asserted by round-tripping a body carrying only `exec_id` and `command` ([`protocol/src/exec.rs:359-367`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L359-L367)). Both bindings expose all of them as optional ([`microvms-js/src/session.rs:101-113`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L101-L113)).
* **An absent `env` and an empty `env` are the same thing** — a plain `HashMap` rather than an `Option`, matching the same decision on the run hook ([`protocol/src/hook.rs:48-53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L48-L53)).
**Drift risk:** adding a required field to `StartRequest` makes the daemon reject every body a pinned client sends. Mitigation: `#[serde(default)]` on every field but `exec_id` and `command`, and the omit-everything test at [`protocol/src/exec.rs:359-367`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L359-L367) as the guard.
## protocol::exec::PollResponse and the flattened Outcome
[Section titled “protocol::exec::PollResponse and the flattened Outcome”](#protocolexecpollresponse-and-the-flattened-outcome)
**Producer:** [`protocol/src/exec.rs:229-236`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L229-L236) (`PollResponse`), [`:56-74`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L56-L74) (`Outcome`)
**Consumer(s):**
* [`agentd/src/exec.rs:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90) (re-export)
* [`microvms-core/src/session/exec.rs:74-82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L74-L82) — `impl From for ExecResult`
* [`microvms-cli/src/guards.rs:1941`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L1941) — the expected envelope shape is written out rather than serialized from `PollResponse`, “which is the whole point”.
* [`microvms-cli/src/commands/attached.rs:1089`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1089), [`:1093`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1093) — constructs `Outcome` values for its render tests.
* [`microvms-core/tests/turmoil_client.rs:952`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L952)
**Shape:**
```rust
1
#[derive(Debug, Deserialize, JsonSchema, Serialize)]
2
pub struct PollResponse {
3
pub exec_id: String,
4
pub phase: Phase,
5
#[serde(skip_serializing_if = "Option::is_none")]
6
#[serde(flatten)]
7
pub result: Option,
8
}
```
```rust
1
/// Captured output and exit status of a finished exec.
2
#[derive(Clone, Debug, Default, Deserialize, JsonSchema, Serialize)]
3
pub struct Outcome {
4
/// Exit code, or `None` when the child died to a signal.
5
pub exit_code: Option,
6
/// Signal number that killed the child, when one did.
7
pub signal: Option,
8
pub stdout: String,
9
pub stderr: String,
10
/// Set when either stream hit `max_output_bytes` and was cut. An explicit
11
/// flag rather than a sentinel string in the output: a marker inside the
12
/// bytes is indistinguishable from output that happens to contain it.
13
pub truncated: bool,
14
/// Set when the post-exit linger deadline expired with the pipes still open,
15
/// meaning some grandchild is alive and may write more that nobody will see.
16
/// Reported rather than hidden, because a harness that sees empty output from
17
/// a command it knows produced some needs to be able to tell why.
18
pub writers_may_be_alive: bool,
19
}
```
**Assumptions consumers make:**
* **An absent outcome is normal, not an error.** [`protocol/src/exec.rs:369-374`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L369-L374) names the failure a stricter reading would cause: a client that read a running exec’s missing outcome as an error would fail on every poll before the first one that mattered. A running poll serializes to exactly `{"exec_id":"e1","phase":"running"}` ([`:382`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L382)).
* **`truncated` is a flag, never a sentinel string in the bytes** ([`protocol/src/exec.rs:65-67`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L65-L67)) — a marker inside the output is indistinguishable from output that happens to contain it.
* **`writers_may_be_alive` is reported rather than hidden** ([`protocol/src/exec.rs:69-73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L69-L73)), so a harness seeing empty output from a command it knows produced some can tell why.
* **`exit_code: None` means a signal killed the child**, so a consumer must read `signal` before concluding failure ([`protocol/src/exec.rs:60-62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L60-L62); [`microvms-core/src/session/exec.rs:93-96`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L93-L96)).
* **`ExecResult` is a thin wrapper and renames one field.** [`microvms-core/src/session/exec.rs:61-82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L61-L82) maps `result` to `outcome`, deliberately not re-modelling the shape so the two cannot disagree.
**Drift risk:** `#[serde(flatten)]` means schemars inlines `Outcome`’s fields into `PollResponse` and emits **no `$defs` entry for `Outcome`** — the generated `PollResponse` definition in `docs/schema.json` even carries *Outcome’s* doc comment as its `description`. A generated client reading `docs/schema.json` therefore has no `Outcome` type to name, and a new field on `Outcome` appears as a new optional field on `PollResponse` with no signal that it belongs to the finished-exec half. Mitigation: `agentd/tests/schema_artifact.rs` already asserts `definition_collisions == []`; extend it to assert the expected `$defs` key set so a flatten added or removed shows up as a named failure.
## The `/v1/exec/{id}/stream` SSE contract — three payloads and three event names
[Section titled “The /v1/exec/{id}/stream SSE contract — three payloads and three event names”](#the-v1execidstream-sse-contract--three-payloads-and-three-event-names)
**Producer:** [`protocol/src/exec.rs:180-206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L180-L206) (`OutputEvent`, `GapEvent`, `ExitEvent`), [`:256-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L256-L258) (`EVENT_*`), [`:171-178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L171-L178) (`StreamQuery`)
**Consumer(s):**
* [`agentd/src/exec.rs:73-78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L73-L78) — the daemon imports `EVENT_EXIT`, `EVENT_GAP`, `EVENT_OUTPUT` and re-exports the payload types at [`:87-90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87-L90).
* [`microvms-core/src/session/sse.rs:272`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L272) — matches on `protocol::exec::EVENT_OUTPUT` / `EVENT_GAP`, and deserializes each payload into `ExecEvent` ([`microvms-core/src/session/sse.rs:240-256`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L240-L256)).
* [`microvms-cli/src/commands/attached.rs:253`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L253), [`:1027`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1027) — renders `ExitEvent` into the NDJSON stream.
* [`microvms-core/tests/turmoil_client.rs:855`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L855), [`:870`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L870) — drives all three event names under simulated faults.
**Shape:**
```rust
1
/// One `output` SSE event.
2
#[derive(Debug, Deserialize, JsonSchema, Serialize)]
3
pub struct OutputEvent {
4
pub offset: u64,
5
pub stream: StreamKind,
6
pub output: String,
7
}
8
9
/// One `gap` SSE event: the byte range a lagging or late subscriber lost.
10
#[derive(Debug, Deserialize, JsonSchema, Serialize)]
11
pub struct GapEvent {
12
pub from: u64,
13
pub to: u64,
14
}
15
16
/// The terminal `exit` SSE event. Emitted before the stream ends, so a client
17
/// that sees the body close without one knows the connection failed rather than
18
/// the command finishing.
19
#[derive(Debug, Deserialize, JsonSchema, Serialize)]
20
pub struct ExitEvent {
21
pub exit_code: Option,
22
pub signal: Option,
23
pub truncated: bool,
24
pub writers_may_be_alive: bool,
25
/// Total bytes published, so a client can assert it saw all of them.
26
pub offset: u64,
27
}
```
```rust
1
pub const EVENT_OUTPUT: &str = "output";
2
pub const EVENT_GAP: &str = "gap";
3
pub const EVENT_EXIT: &str = "exit";
```
**Assumptions consumers make:**
* **The absence of an `exit` event is the signal, not silence.** [`protocol/src/exec.rs:195-197`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L195-L197) and [`microvms-core/src/session/sse.rs:253-255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L253-L255) — a raw byte stream cannot distinguish a finished command from a dropped connection, so the terminal typed event is what makes the difference observable. Restates `.erpaval/solutions/architecture-patterns/byte-offset-cursor-is-what-makes-reconnect-work.md`.
* **`GapEvent.from` is inclusive and `to` is exclusive**, so `to` is where a cursor resumes ([`microvms-core/src/session/sse.rs:249-252`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L249-L252)). Nothing in the wire type says this — it is a client-side convention documented only at the consumer.
* **An unknown event name is dropped, not raised; a bad base64 payload is raised.** [`microvms-core/src/session/sse.rs:260-269`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L260-L269) splits the two deliberately: one bad frame must not end a live stream, but silently dropping output a caller asked for is the failure the whole protocol is shaped to prevent.
* **`ExitEvent.offset` is a total, so a client can assert it saw every byte** ([`protocol/src/exec.rs:204-205`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L204-L205)).
* **The event names are constants because a typo on either side is a stream that carries events nobody dispatches** ([`protocol/src/exec.rs:251-255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L251-L255)).
* **`ExecEvent` is not `Clone`**, because `ExitEvent` is not, and adding the derive would be an edit to a crate the consumer does not own ([`microvms-core/src/session/sse.rs:236-238`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L236-L238)).
* **`StreamQuery.offset` absent means 0**, i.e. everything still in the replay window ([`protocol/src/exec.rs:174-177`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L174-L177)) — not “everything the command ever wrote”. The window is `stream_replay_bytes: 1048576` in `docs/schema.json`.
**Drift risk:** a fourth event name added on the daemon side is dropped silently by `microvms-core/src/session/sse.rs` (`Ok(None)`), which is the correct degradation for an old client but means a *new* client failing to dispatch a name it should handle looks identical. Mitigation: the `EVENT_*` constants are the single source; a consumer adding dispatch should match on the constant, and `microvms-core/tests/turmoil_client.rs` should gain a case per name.
## The microvm –json envelope — the contract the conformance oracle reads
[Section titled “The microvm –json envelope — the contract the conformance oracle reads”](#the-microvm-json-envelope--the-contract-the-conformance-oracle-reads)
**Producer:** [`microvms-cli/src/envelope.rs:311-318`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L311-L318) (`ok`), [`:321-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L321-L339) (`error`), [`:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L66) (`API_VERSION`)
**Consumer(s):**
* [`conformance/run_rs.py:164-219`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L164-L219) — the `Envelope` dataclass, which reads every failure field directly so a missing key is a `KeyError` rather than a `None` that flows into a passing assertion ([`:168-172`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L168-L172)).
* [`conformance/run_rs.py:222-249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L222-L249) — `KindError`, carrying kind, code, and exit code so a check can assert at whichever granularity it means.
* [`conformance/run_rs.py:284-302`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L284-L302) — cross-checks the process exit code against the envelope’s own `exitCode`, because they are two independent renderings of one decision.
* [`conformance/run_rs.py:1685-1710`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L1685-L1710) — the offline self-test’s frozen envelope fixtures.
* [`microvms-cli/tests/exit_codes.rs:96`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L96), [`microvms-cli/tests/manifest.rs:205`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L205).
**Shape:**
```rust
1
pub fn ok(kind: &str, data: Map) -> Value {
2
json!({
3
"status": "ok",
4
"apiVersion": API_VERSION,
5
"type": kind,
6
"data": Value::Object(data),
7
})
8
}
```
```rust
1
pub fn error(failure: &CliError) -> Value {
2
let mut data = failure.data.clone();
3
// The fine-grained daemon status, for the consumer the exit code is too coarse for.
4
// Inserted rather than replacing whatever `data` already holds, so a teardown's leaked
5
// identifiers and the kind coexist on one failure.
6
if let Some(wire) = failure.wire_kind {
7
data.insert("kind".to_string(), json!(wire.as_str()));
8
}
9
json!({
10
"status": "error",
11
"apiVersion": API_VERSION,
12
"error": failure.message,
13
"code": failure.code(),
14
"exitCode": failure.exit.as_u8(),
15
"finding": failure.finding(),
16
"suggestions": failure.suggestions,
17
"data": Value::Object(data),
18
})
19
}
```
**Assumptions consumers make:**
* **Every failure key is unconditional.** [`microvms-cli/src/envelope.rs:20-25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L20-L25) — `finding` is present and empty when no measured finding applies, `suggestions` is an empty array and `data` an empty object rather than absent, because “the consumer that forgets reads `undefined` as ‘no finding’ for a failure that had one”. [`conformance/run_rs.py:168-172`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L168-L172) takes the CLI at its word and reads them directly.
* **Exactly one JSON object reaches stdout, except on the streaming path.** [`microvms-cli/src/envelope.rs:4-11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L4-L11) — progress goes to stderr always, and [`microvms-cli/tests/thinness.rs:503`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L503) asserts no module but `envelope` and two `main` exceptions writes to stdout. [`conformance/run_rs.py:252-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L252-L258) types a second document as an `EnvelopeError`, deliberately distinct from a protocol result, because it means the binary is wrong.
* **The streaming exception is a different discriminant, not a relaxed rule.** [`microvms-cli/src/envelope.rs:33-49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L33-L49) — a streamed exec emits NDJSON and its final envelope’s `type` is `microvm.exec.stream`, never `microvm.exec`, and that envelope is written **compact** because “the last line is the envelope” is only true if the envelope is one line.
* **`--quiet` cannot buy silence about a leak.** [`microvms-cli/src/envelope.rs:13-18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L13-L18) — only `progress` is suppressed; a stale rate table and a leaked resource still reach `warn`. [`conformance/run_rs.py:268-271`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L268-L271) relies on this to pass `--quiet` on every invocation.
* **`data.kind` is the only place the daemon’s fine status survives.** [`microvms-cli/src/envelope.rs:27-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L27-L31) names `conformance/run_rs.py` as the consumer that needs it, because `ERR_PROTOCOL` covers five `WireKind`s.
* **`apiVersion` bumps on a meaning change, not on a new command.** [`microvms-cli/src/envelope.rs:62-65`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L62-L65) — adding a command changes `microvm manifest`, not this.
**Drift risk:** the envelope is hand-built with `json!` and has no generated schema, so a renamed key breaks the Python oracle at runtime rather than at build time — and the offline half of that suite ([`conformance/run_rs.py:1685-1710`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L1685-L1710)) carries frozen fixtures that would need the same edit. Mitigation: `./conformance/run_rs.py --self-test` is free and offline and is already in `mise run check`’s neighbourhood; keep the fixtures and the `json!` literals edited in one commit.
## Other contracts
[Section titled “Other contracts”](#other-contracts)
* **`microvms_core::SizeClass`** — [`microvms-core/src/sizing.rs:112-119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L112-L119), five closed baselines with `ALL` at [`:132`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L132), `DEFAULT = Mib2048` at [`:129`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L129), and the one S2 boundary at `from_baseline_mib` ([`:146`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L146)). 13 consumer files across cli, py, js, and core’s cost engine.
* **`microvms-cli::Exit` and `EXIT_TABLE`** — [`microvms-cli/src/exit.rs:78-102`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L78-L102), [`:173-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173-L258). Fourteen append-only rows; `#[repr(u8)]` with explicit discriminants so an inserted variant cannot silently renumber the contract ([`:74-77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L74-L77)). 9 consumer files.
* **`microvms-cli::CliError`** — [`microvms-cli/src/exit.rs:266-278`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L266-L278). Carries `wire_kind`, `suggestions`, and a `data` map so a partial result stays machine-readable on the failure path. 9 consumer files.
* **`microvms_core::session::ExecEvent` / `ExecResult`** — [`microvms-core/src/session/sse.rs:240-256`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L240-L256), [`microvms-core/src/session/exec.rs:67-72`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L67-L72). The client-side view of the wire types, 8 and 7 consumer files.
* **`microvms_core::sandbox::TeardownReport`** — [`microvms-core/src/sandbox.rs:335-361`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L335-L361). Returned where a `finally` would run; `image_deleted: Option` separates “not asked for” from “failed”. 5 consumer files.
* **`docs/schema.json`** — generated by `agentd/src/bin/schema.rs`, gated by [`mise.toml:168-173`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L168-L173). 20 routes, 21 `$defs`, `protocol_version: "1"`, `definition_collisions: []`, plus a `limits` object publishing every operative cap. [`agentd/tests/schema_artifact.rs:149-328`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L149-L328) probes the real router against it.
* **`microvms-py/microvms.pyi` + `microvms-py/py.typed`** — generated by `scripts/generate-py-stubs.py`, gated by [`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195). 1476 lines, with the do-not-edit header at [`microvms-py/microvms.pyi:1-8`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/microvms.pyi#L1-L8). The script pins `maturin@1.14.1` because maturin 1.15.0 moved `generate-stubs` output into the module’s package dir, so bumping that pin breaks `mise run stubs:check`.
* **`microvms-js/index.d.ts`** — generated by `napi build --platform` ([`microvms-js/package.json:12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L12)), gitignored at `.gitignore:29`, **no drift gate**. 1075 lines, declared as the package’s `"types"` at [`microvms-js/package.json:8`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L8).
* **`pinned_rates()` and its Python twin** — [`microvms-core/src/cost.rs:1011-1026`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1011-L1026) against [`scripts/check-live-rates.py:119-145`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L119-L145). A deliberate second copy: [`:112-118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L112-L118) states that importing the values would compare a table against itself. `verify_twin` ([`:148-211`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L148-L211)) reads the Rust literals as text, so a reflow of `pinned_rates()` is a named exit 1. `every_rate_byte_matches_the_python_literal` ([`microvms-core/src/cost.rs:2179-2196`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2179-L2196)) checks scale as well as value.
* **`microvms-core/src/constants.rs` against the botocore service model** — 40+ constants (`MODEL_API_VERSION = "2025-09-09"` at [`:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L57), `MAX_RUN_HOOK_PAYLOAD_BYTES = 4096` at [`:83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L83)) read back out of the shipped `lambda-microvms` model by `scripts/check-model-drift.py`. `DOCUMENTED_RUN_HOOK_PAYLOAD_BYTES = 16_384` ([`:97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L97)) is retained as the contradicted prose figure the check caught.
* **`session_constants`, which diverges between the two bindings** — [`microvms-py/src/session.rs:578-604`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L578-L604) publishes 7 keys; [`microvms-js/src/session.rs:574-606`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L574-L606) publishes 10, adding `wsSubprotocol`, `wsAuthSubprotocolPrefix`, `wsPortSubprotocolPrefix`. Nothing asserts the two dictionaries agree.
* **`protocol::hook::RunHookEnvelope` / `RunHook`** — [`protocol/src/hook.rs:26-30`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L26-L30), [`:45-54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L45-L54). The platform wraps the caller’s string, so the payload is one `serde_json` parse deeper than the body (measured 2026-08-05, [`:20-25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L20-L25)). `RunHook::parse` ([`:119-150`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L119-L150)) is hand-walked so no refusal quotes a value, because the payload carries the agent token ([`:56-63`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L56-L63)); unknown keys are ignored on purpose, since a 400 here terminates the VM ([`:108-114`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L108-L114)).
* **`protocol::fs::FsQuery` / `FileReadQuery`** — [`protocol/src/fs.rs:17-24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L17-L24), [`:38-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L38-L57). `path` missing is 400 and never 404, because clients map 404 onto `FileNotFoundError` ([`:13-16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L13-L16)); `mode` is a string so `0644` and `644` both parse ([`:21-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L21-L23)); line ranges are 1-based inclusive on both ends with `end_line` past EOF reading through, verbatim from the AI SDK harness contract ([`:34-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L34-L37)).
* **`protocol::exec::ErrorBody` and the ten `ERROR_*` slugs** — [`protocol/src/exec.rs:245-249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L245-L249), [`:266-286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L266-L286). A client branches on `error` plus the status code and never on `detail` ([`:240-244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L240-L244)). The fs routes answer `text/plain` instead ([`protocol/src/fs.rs:5-6`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L5-L6)), and `ErrorBody` gets **no `$defs` entry** in `docs/schema.json`.
* **`protocol::PROTOCOL_VERSION` and `VERSION_HEADER`** — [`protocol/src/lib.rs:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L58), [`:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L66). [`protocol/src/lib.rs:40-54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L40-L54) states what a client must do on each mismatch case, including that a differing `daemon_version` under the same `protocol_version` must not be treated as an error.
* **`spec/core.symspec.json` and `spec/agentd.symspec.json`** — 51 EARS requirements held as an id-keyed object (each with `key`, `patternType`, `priority`, `sentence`, `systemName`, `verificationMethod`) plus a `stateModel` whose five variables are mirrored field-for-field by [`microvms-core/src/sandbox.rs:428-433`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L428-L433).
* **The workspace dependency edges** — [`microvms-cli/tests/dependency_direction.rs:68-125`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L68-L125) asserts them as equalities, not as `assert!(no edge)`, because a stub crate with no dependencies passes a negative assertion ([`:11-12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L11-L12)). [`microvms-cli/tests/thinness.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L66) holds the CLI’s six-entry allowlist, each with a paragraph, and [`:145-213`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L145-L213) asserts the direct dependency set is exactly that.
## See also
[Section titled “See also”](#see-also)
* [impact analysis](impact-analysis.md) — 40 shared source citations
* [business logic](business-logic.md) — 22 shared source citations
* [public api](../../reference/public-api.md) — 22 shared source citations
* [debugging guide](debugging-guide.md) — 16 shared source citations
* [tech debt](tech-debt.md) — 13 shared source citations
# Debugging guide
> Something is broken. This tells you where to look first.
Something is broken. This tells you where to look first.
Almost every failure mode below was found against real AWS after the offline tiers were green, so the operational knowledge is already written down in four places rather than in anyone’s head:
* `docs/PLATFORM.md` — measured platform behavior, each entry carrying a date, a region, and an API version. Contradictions are appended rather than deleted, so the file reads as a log of surprises.
* `EXIT_TABLE` in [`microvms-cli/src/exit.rs:173-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173-L258) — fourteen rows. Each carries an integer, an `ERR_*` code, a `meaning` (what to do next), and a `finding` (the `docs/PLATFORM.md` section that measured it). The `finding` column turns an exit code into a documentation lookup.
* Trap messages in the library. `microvms-core` writes the finding into the message itself, so the failure explains itself without a doc lookup — see [`microvms-core/src/control/image.rs:371-386`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L371-L386).
* `.erpaval/solutions/` — twelve compounded lessons, each the conclusion of a real debugging session.
No `.rs` or `.py` file in the workspace carries a `TODO`, `FIXME`, `HACK`, `INCIDENT`, or `POSTMORTEM` marker; the repo convention is that comments record constraints and defects defended against rather than narration. The four sources above are the history.
Two operational facts before you start. `mise run check` is the free offline gate — lint, security, all six Rust tiers, schema and stub freshness, model drift, live wiring, and the release cross-compile ([`mise.toml:290-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L290-L301)). `mise run live` is BILLABLE, takes about fifteen minutes against real AWS, and is never a first debugging step ([`mise.toml:428-429`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L428-L429)); after any live run, teardown is verified separately by `mise run live:verify-clean` ([`mise.toml:416-426`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L416-L426)), because the service creates log groups under `/aws/lambda-microvms/` that outlive `terraform destroy` ([`docs/PLATFORM.md:195-201`](/microvms-agentd/internals/platform/)).
## Failure-mode index
[Section titled “Failure-mode index”](#failure-mode-index)
| Symptom | Likely surface | First check | Citation |
| --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `AccessDeniedException` whose `message` field is null | Not IAM. The region does not price MicroVMs — only five do. The service model’s `endpointPrefix` is `lambda`, so a client constructs and resolves for any region and the first API call is the only reporter | Read the `message` field. A real denial names the principal and the action; this one is `None`. Then `microvm doctor`, whose region check runs first for exactly this reason | [`docs/PLATFORM.md:146-168`](/microvms-agentd/internals/platform/), [`microvms-core/src/region.rs:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L73), [`microvms-cli/src/exit.rs:340-344`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L340-L344) |
| Image stuck in `CREATING`; builds never start; `updatedAt` never advances past `createdAt` | The `clientToken` replay. A `clientToken` is a permanent idempotency key, so a create whose token repeats an earlier one is replayed as a no-op. The image cannot be deleted (`CREATING` forbids it) and its only version cannot be dropped either. Two were wedged about 15 hours | `ListMicrovmImageBuilds` — every build `PENDING` is the signature. Record the identifier and build under a fresh `--name`; waiting does not help | [`docs/PLATFORM.md:549-565`](/microvms-agentd/internals/platform/), [`microvms-core/src/control/image.rs:366-386`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L366-L386), [`microvms-cli/src/exit.rs:354-357`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L354-L357) |
| A VM reaches a terminal state before `RUNNING`; the client reports a connection error | A lifecycle hook failed. `PENDING → RUNNING → SUSPENDING/SUSPENDED → TERMINATING → TERMINATED`; anything terminal before `RUNNING` died during startup, and the platform terminates it before forwarding any traffic | `GetMicrovm`’s `stateReason` — the only evidence that outlives the VM. The client already puts the state and the reason both in the message | [`docs/PLATFORM.md:186-193`](/microvms-agentd/internals/platform/), [`microvms-core/src/control/microvm.rs:482-499`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L482-L499), [`microvms-cli/src/exit.rs:216-221`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L216-L221) |
| `CREATE_FAILED` with a fully green build log, every docker layer succeeding, and no error line anywhere | The guest’s `AGENTD_PORT` disagrees with the create call’s `hooks.port`. The build-time `ready` and `validate` hooks are dialled on the create call’s port, so a daemon listening elsewhere answers none of them. An unset `AGENTD_PORT` is the same failure, with nothing in the Dockerfile to point at | Fetch `GetMicrovmImageVersion` and compare `hooks.port` against the Dockerfile’s `ENV AGENTD_PORT`. `GetMicrovmImage` structurally cannot say why | [`docs/PLATFORM.md:1328-1350`](/microvms-agentd/internals/platform/), [`agentd/src/config.rs:116-120`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L116-L120), [`.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md:19-24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md#L19-L24) |
| A 45-minute build ends as `Ready hook invocation timed out after PT5M`, saying nothing about architecture | A host-architecture daemon binary. MicroVMs are ARM64-only, so an x86-64 `CMD` cannot exec and surfaces only as the hook never answering | `microvm doctor --binary `. It reads twenty bytes of ELF header and compares `e_machine` against `0xB7`; a script or wrapper is caught as “not an ELF binary” | [`microvms-cli/src/commands/doctor.rs:8-15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L8-L15), [`microvms-cli/src/commands/doctor.rs:25-29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L25-L29), [`microvms-cli/src/commands/doctor.rs:422-436`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L422-L436) |
| Every failed build reports `reason=unknown` and the log group holds nothing at all | The build role’s log permissions, not a silent service. Logs go to `/aws/lambda-microvms/`, not the plausible `/aws/lambda/microvms/*`. The caller’s own policy is discarding the evidence | `microvm logs ` names the group; an empty group beside `reason=unknown` is the prefix signature. Unknown alone is not the same as empty | [`docs/PLATFORM.md:567-575`](/microvms-agentd/internals/platform/), [`microvms-core/src/control/image.rs:55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L55), [`agentd/src/main.rs:84-87`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L84-L87) |
| A build says `The container image build failed.` and nothing else | `stateReason` lives on the **build** only. `GetMicrovmImage` reports `CREATE_FAILED` with no reason member at all, and `ListMicrovmImageVersions` reported `null` across three separate failures | `ListMicrovmImageBuilds`, and expect a **list**: each failed version produced two builds, one per Graviton generation, with identical reasons. Then read `snapshotBuild`’s shape — absent means the Dockerfile broke before anything installed, `codeInstallSizeInBytes` alone means code installed and the daemon never became ready | [`docs/PLATFORM.md:577-605`](/microvms-agentd/internals/platform/), [`docs/PLATFORM.md:1283-1303`](/microvms-agentd/internals/platform/) |
| Every control request answers 503 | Not bootstrapped. The run hook has not landed, so the control API is closed. Deliberately not 401 (which sends a client chasing credentials) and never 404 (which clients map onto “file not found”) | `GET /v1/health` — unauthenticated on purpose so it answers in exactly this window — and read `bootstrapped`. `NotBootstrapped` is retryable: the platform is about to deliver the token | [`agentd/src/auth.rs:69-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L69-L80), [`microvms-core/src/error.rs:244-248`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L244-L248), [`protocol/src/health.rs:20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L20) |
| Requests answer 401 after a successful launch | The presented bearer is not the one the run hook installed. Fatal — waiting does not fix it, and it is classified `ERR_CREDENTIALS` rather than `ERR_PROTOCOL` for that reason | Read the `run` envelope’s `agentToken`. The `ERR_CREDENTIALS` remedy differs for this case and for an unresolvable credential chain, so read the `suggestions` line rather than assuming | [`agentd/src/auth.rs:74-79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L74-L79), [`microvms-core/src/error.rs:220-221`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L220-L221), [`microvms-cli/src/exit.rs:337-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L337-L339) |
| `resume` is refused and no flag reopens the window | The launch-time `idlePolicy` terminated the suspended VM once `suspendedDurationSeconds` passed. The client refuses before calling `ResumeMicrovm` | The error names the elapsed time and the window. A longer window is set at launch with `--suspended-sec`; no call extends the current one. The policy is also readable from `GetMicrovm`, which returns all three members in `RUNNING` and in `SUSPENDED` alike | [`microvms-core/src/sandbox.rs:900-925`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L900-L925), [`docs/PLATFORM.md:612-621`](/microvms-agentd/internals/platform/), [`docs/PLATFORM.md:652-674`](/microvms-agentd/internals/platform/) |
| A multi-hour agent run is auto-suspended while it is busy | Idleness is measured by inbound traffic through the endpoint proxy, and that proxy terminates outside the VM. Traffic a guest process sends to the daemon’s own port never crosses the thing doing the measuring, so an in-VM keep-alive is not implementable | Poll `GET /v1/health` from **outside** the VM. Measured: a polled VM stayed `RUNNING` through 311 seconds against a 60-second window while an unpolled control suspended at 66 | [`docs/PLATFORM.md:623-650`](/microvms-agentd/internals/platform/), [`agentd/src/routes.rs:314-338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314-L338) |
| A long-running trial dies mid-flight with what looks like a dead daemon | An expired proxy token. The service caps a JWE at sixty minutes, shorter than a long agent run, and the rejection is indistinguishable from a daemon that died | Minting happens inside the request path and `DEFAULT_REFRESH_AFTER` is thirty minutes — half the ceiling rather than marginally under it, so a request in flight across the rollover still holds about thirty minutes of life. A mint failure is retryable on purpose | [`docs/PLATFORM.md:501-514`](/microvms-agentd/internals/platform/), [`microvms-core/src/session/proxy.rs:21-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L21-L37), [`microvms-core/src/session/proxy.rs:111`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L111) |
| Writes are refused with **507** naming byte counts | Disk pressure past the configured reserve. 507 rather than 500 deliberately: a 500 is indistinguishable from a daemon defect, so a client retries it, which is correct for a defect and actively harmful for a full disk | Read the free-space numbers in the response body, then `GET /v1/health` → `disk.under_pressure`. `disk: null` means unmeasurable, which is deliberately not zero | [`agentd/src/fs.rs:97-124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L97-L124), [`agentd/src/disk.rs:142-159`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/disk.rs#L142-L159), [`protocol/src/health.rs:21-34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L21-L34) |
| An exec result carries `truncated: true` | The per-stream output cap. Default 8 MiB, sized well under a 512 MiB baseline VM, because an OOM-killed daemon is unrecoverable — there is no supervisor inside the VM to restart it | `AGENTD_MAX_OUTPUT_BYTES` against the volume the command emits. Past the cap the daemon keeps reading and discarding rather than stopping, so the writer never blocks in the kernel | [`agentd/src/config.rs:25-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L25-L27), [`agentd/src/config.rs:87`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L87), [`protocol/src/exec.rs:66-73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L66-L73) |
| An exec result carries `writers_may_be_alive: true` and the output looks cut short | A grandchild still holds the inherited pipe past `output_linger`. EOF arrives when the *last* writer closes, so a command that backgrounds a server or a log tailer keeps writing after the direct child exits | `AGENTD_OUTPUT_LINGER_SECS` (default 5). Under a simulated-time test this flag is a false positive — see the two-clocks incident below | [`agentd/src/config.rs:28-31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L28-L31), [`agentd/src/exec.rs:1259-1296`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L1259-L1296), [`.erpaval/solutions/best-practices/pipes-not-tempfiles-for-subprocess-output.md:11-27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/best-practices/pipes-not-tempfiles-for-subprocess-output.md#L11-L27) |
| A stream delivers a `gap` event with `from`/`to` offsets | The subscriber lagged the bounded broadcast channel, or the replay ring evicted the bytes. Classified `ERR_PLATFORM` and not retryable — the bytes are gone. The gap is a typed event rather than a log line precisely so a cursor cannot advance silently past dropped data | Re-GET from the last offset actually received. Then check `AGENTD_STREAM_CHANNEL_CAPACITY` and `AGENTD_STREAM_BUFFER_BYTES` against the output rate | [`agentd/src/exec.rs:617-630`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L617-L630), [`microvms-core/src/error.rs:265-267`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L265-L267), [`microvms-core/src/error.rs:395`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L395) |
| A stdin write answers 409, or 410, or 408 — and the three mean different things | 409 `Conflict` is “you did not ask for stdin”, fixed at start time. 410 `StdinClosed` is a lifecycle fact: EOF already arrived or the child stopped reading, and a retry never succeeds. 408 `RequestTimeout` is the child not draining within the write timeout — retryable, and some bytes may already have landed | Read `data.kind`, not the exit code: all three collapse onto `ERR_PROTOCOL` except 408, which is `ERR_RETRYABLE`. The daemon keeps its stdin handle open across a 408 so a retry can succeed | [`microvms-core/src/error.rs:229-243`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L229-L243), [`agentd/src/config.rs:58-62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L58-L62), [`protocol/src/exec.rs:276-286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L276-L286) |
| A tar upload answers 400 naming one member | That member violated the data-filter contract — an escaping path, a symlink out of the root, a refused type. The refused name travels with the refusal, because a 400 saying only “bad archive” sends the caller re-reading their whole tree | Read the member name in the body. 413 is a different answer (over `max_tar_members` or `max_tar_bytes`) and 507 a third (the filesystem filled partway through) | [`agentd/src/fs.rs:126-142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L126-L142), [`agentd/src/fs.rs:158-182`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L158-L182), [`agentd/src/config.rs:37-40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L37-L40) |
| A `nameFilter` listing returns an empty first page while matches exist | `maxResults` is applied **before** `nameFilter`, so the service pages over the unfiltered collection and then filters the page. Measured: `nameFilter=bonk&maxResults=1` over 22 images took 26 pages to yield the 10 matches, and page one held zero items | The only termination condition is an absent `nextToken`. A loop that stops on an empty `items` finds nothing at all. `nameFilter` is also a substring match, so the exact-name comparison has to happen client-side across every page | [`docs/PLATFORM.md:704-723`](/microvms-agentd/internals/platform/) |
| `403 AccessDeniedException` naming an image resource that exists and is permitted | The ARN separator. Customer image ARNs use a **colon** before the name; the slash form is evaluated by IAM as a resource no policy matches, so the answer is a permissions message about a resource that is fine | Read the separator before widening any policy. An *unencoded* slash is a third failure: the raw `/` splits into extra path segments and the gateway answers 404 with an HTML body | [`docs/PLATFORM.md:747-767`](/microvms-agentd/internals/platform/) |
| A 5xx on `GetMicrovmImage` with an nginx HTML body | The gateway in front of the service, not the service. Observed once where an immediate hand-signed repeat of the identical URL answered 403 five times out of five | Retry past a 5xx on this operation and never past a 4xx — a 4xx is the answer | [`docs/PLATFORM.md:769-776`](/microvms-agentd/internals/platform/) |
| A poll against a running exec returns `phase: running` with no partial stdout | Polling is terminal-only by design. A detached exec does survive the 60-minute proxy-token ceiling — 450 of 450 ticks recovered across the boundary, the straddling pair a nominal 10-second gap — but its output is readable only at the end | Stream the exec, or have the command write to a file and fetch that file. A 75-minute detached exec also needs outside polling, or the idle window suspends it regardless of how healthy the exec is | [`docs/PLATFORM.md:1192-1227`](/microvms-agentd/internals/platform/) |
| A client sees a transport error it cannot tell from a dead VM | A panicking handler. Without the outermost `CatchPanicLayer` the panic reaches hyper and the connection drops; with it the client gets a 500 and the connection survives. It does not undo the panic — any `std::sync::Mutex` the handler held is now poisoned | Grep the daemon log for `recovering a poisoned lock`. Locks recover rather than propagate, because `.expect()` on a poisoned token lock closes the whole control API forever | [`agentd/src/routes.rs:86-101`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L86-L101), [`agentd/src/state.rs:73-92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/state.rs#L73-L92), [`agentd/tests/panic_guard.rs:11-26`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/panic_guard.rs#L11-L26) |
| A connection is refused a second or two after the VM reaches `RUNNING` | Expected. The endpoint proxy path is not wired up the instant the state flips. Classified `Transport`, retryable because it says nothing about the daemon’s state | Retry. If it persists past a few attempts, go to the terminal-state row and read `stateReason` | [`microvms-core/src/error.rs:251-256`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L251-L256) |
| `resume` returns 200 but the control API stays closed | The VM resumed without an installed token, which contradicts the measured suspend/resume behavior — the in-memory token, the filesystem, exec records, and backgrounded processes all survive a normal cycle | Grep the daemon log for `resumed WITHOUT an installed token`. That line means the resume behaved like a cold start and every in-flight exec record is gone | [`agentd/src/routes.rs:276-290`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L276-L290), [`docs/PLATFORM.md:418-451`](/microvms-agentd/internals/platform/) |
| A Node caller reads `err.code` and gets `GenericFailure` | napi-rs types the async path over its own closed `Status` enum, so a custom code survives a synchronous throw and is collapsed on a Promise rejection. Nearly every binding method is async | Read `err.cause.message` for the `ERR_*` code and `err.cause.cause.message` for the fine-grained wire kind | [`.erpaval/solutions/api-patterns/napi-async-collapses-error-codes.md:11-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/api-patterns/napi-async-collapses-error-codes.md#L11-L19) |
| `terraform destroy` reports success and the account is still billing | The service creates `/aws/lambda-microvms/` itself, so Terraform never owns it. Separately, an image refuses deletion while its VM is still terminating, so one teardown pass is not enough | `mise run live:verify-clean` asks the account directly and separates leak / standing / pending. `microvm ls` alarms on every run whose ledger has a non-empty `leaked` list | [`docs/PLATFORM.md:195-201`](/microvms-agentd/internals/platform/), [`scripts/verify-clean.py:7-28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/verify-clean.py#L7-L28), [`microvms-cli/src/main.rs:222-246`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L222-L246) |
## Log and error surfaces
[Section titled “Log and error surfaces”](#log-and-error-surfaces)
| Surface | Where it emits | What to grep for | Citation |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Daemon structured log | JSON to **stdout**, which is where the platform’s CloudWatch capture reads from. Level from `AGENTD_LOG`, defaulting to `info`; targets on | `agentd listening` for the bind line with `addr` and `version`; `recovering a poisoned lock` for a handler panic; `exec exceeded its timeout and its process group was signalled`; `process group survived SIGTERM; escalating to SIGKILL`; `ignoring unparseable configuration value` for a bad `AGENTD_*` | [`agentd/src/main.rs:84-97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L84-L97), [`agentd/src/main.rs:79`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L79) |
| Daemon `warn` sites | The same stdout JSON stream. 38 `warn`/`error` calls, all in the daemon — `agentd/src/fs.rs` 19, `agentd/src/exec.rs` 14, `agentd/src/routes.rs` 13, and one to four each in `serve`, `main`, `identity`, `config`, `state`, `disk` | `tar member refused`, `archive over cap`, `refusing a write: the target filesystem is under the disk reserve`, `exec stream subscriber lagged`, `spawn failed`, `bootstrap refused: a different token is already installed`, `run hook body is not JSON` | [`agentd/src/fs.rs:161-178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L161-L178), [`agentd/src/exec.rs:624`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L624), [`agentd/src/routes.rs:186-231`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L186-L231) |
| Build log group | `/aws/lambda-microvms/`, created by the service. Not `/aws/lambda/microvms/*` | An *empty* group beside `reason=unknown` is the IAM-prefix signature, not a silent service | [`microvms-core/src/control/image.rs:55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L55), [`microvms-core/src/control/image.rs:76-81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L76-L81) |
| `microvm logs ` | A success that names the group and hands you the read: `data.tailCommand` is the working `aws logs tail` invocation (AWS CLI v2 only — the subcommand does not exist in v1), and `data.lines` is explicitly `null`, never `[]`, because an empty list reads as “there are no logs” when this client did not read the group. CloudWatch is not in the transport’s dependency set; the read runs under your own identity, granted by the stack’s `logs_read_policy_arn` | `data.logGroup`, `data.tailCommand`, `data.tailRequires`, `data.streams` | [`microvms-cli/src/commands/local.rs:228`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L228) |
| `GET /v1/health` | The daemon, on the unauthenticated router so it answers during the pre-bootstrap window | `bootstrapped`, `disk.under_pressure`, `disk.available_bytes`, `identity_degraded`, `identity_repaired`, `busy`, `execs`. `disk: null` means unmeasurable, not zero | [`agentd/src/routes.rs:314-338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314-L338), [`protocol/src/health.rs:11-45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L11-L45) |
| `GET /v1/schema` and `docs/schema.json` | The daemon serves the same list the router is assembled from, so a route cannot be served unless it appears in the list and a listed route with no handler panics at startup | The 20 endpoints and their `auth` field, which is what splits the Bearer-guarded router from the open one | [`agentd/src/routes.rs:36-59`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L36-L59), [`agentd/src/routes.rs:110-139`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L110-L139) |
| `microvms-agentd-version` response header | Stamped on **every** response by middleware applied outside `route_layer` — handler bodies, the auth middleware’s 401/503, the body-limit layer’s 413, and the 404 fallback | The header’s presence. A version header a client only sometimes receives is one it cannot use as a precondition | [`agentd/src/routes.rs:142-164`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L142-L164) |
| Daemon error-body slug | The response body of a failing control route, as `{"error": "", "detail": "..."}` | The slug paired with the status: `malformed_request`, `unknown_exec`, `spawn_failed`, `still_running`, `already_acked`, `stdin_not_requested`, `stdin_closed`, `stdin_write_timeout`, `stdin_write_too_large`, `stdin_write_failed` | [`protocol/src/exec.rs:266-286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L266-L286) |
| Exec result flags | The poll and stream payloads | `truncated` (the per-stream cap was hit) and `writers_may_be_alive` (the linger deadline expired with a writer still holding the pipe). Both are explicit rather than inferred from a short log | [`protocol/src/exec.rs:66-73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L66-L73) |
| SSE event names | The `/v1/exec/{id}/stream` frames | `output`, `gap`, `exit`. A `gap` frame carries `from`/`to` byte offsets and is the only honest report of lost bytes | [`protocol/src/exec.rs:256-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L256-L258), [`agentd/src/exec.rs:656-658`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L656-L658) |
| CLI failure envelope | Exactly one JSON object on **stdout**. `finding`, `suggestions`, and `data` are unconditional keys — present and empty, never absent | `code` for the `ERR_*` string, `exitCode` (which must agree with `$?`), `finding` for the `docs/PLATFORM.md` section, `data.kind` for the daemon-chosen status the exit code collapses, `data.leaked` for identifiers teardown could not delete | [`microvms-cli/src/envelope.rs:320-339`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L320-L339), [`microvms-cli/src/exit.rs:260-278`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L260-L278) |
| CLI human failure rendering | stdout on the plain path. Deterministic: sorted data keys, byte-identical across two renders | First line `error ERR_*: `; then `see docs/PLATFORM.md, ''`; then `hint:` lines; then `: ` per sorted data key | [`microvms-cli/src/envelope.rs:345-377`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L345-L377) |
| CLI progress and warnings | **stderr**, always, so stdout stays exactly one document. `--quiet` suppresses progress and never a warning | A leak warning survives `--quiet`, because a leak nobody is told about is the one thing silence must not buy | [`microvms-cli/src/envelope.rs:481-493`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L481-L493), [`microvms-cli/tests/exit_codes.rs:154-190`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L154-L190) |
| `exec --stream` NDJSON | The one invocation allowed more than one object on stdout: every line before the last is an event, the last is the envelope, under its own discriminant `microvm.exec.stream` | Branch on `type`. A streamed exec that fails before any event writes exactly one document | [`microvms-cli/tests/exit_codes.rs:232-278`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L232-L278) |
| Exit code in `$?` | The process. Fourteen rows, 0 through 13, append-only, `#[repr(u8)]` with explicit discriminants so a variant inserted in the middle cannot silently renumber the contract | The integer, then `Exit::row()`’s `meaning` and `finding`. Thirteen distinct non-zero codes; no two rows share one | [`microvms-cli/src/exit.rs:78-102`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L78-L102), [`microvms-cli/src/exit.rs:173-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173-L258) |
| Run ledger on disk | One JSON file per invocation under `$MICROVM_STATE_DIR`, else `~/.microvm/runs`. Written **before** each delete is attempted, and its file is refused deletion while `leaked` is non-empty | `leaked` — the operator’s to-do list. For a `CREATING` image and a service-created log group the identifier *is* the remedy, because there is no second way to find them. A write failure is swallowed, so an unwritable state dir costs the `ls` entry and nothing else | [`microvms-cli/src/ledger.rs:1-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L1-L22), [`microvms-cli/src/ledger.rs:37-49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L37-L49), [`microvms-cli/src/seam.rs:450-459`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L450-L459) |
| `microvm ls` | stdout. Rows marked as alarms plus a trailing count | “N run(s), M with something still billing” | [`microvms-cli/src/main.rs:209-247`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/main.rs#L209-L247) |
| `microvm doctor` | A **success** envelope with `ok: false` plus exit `ERR_PRECONDITION`, because the check succeeded — it found what was wrong | `checks[]` per named check. Advisory checks do not fail the run; the fatal ones do | [`microvms-cli/src/commands/doctor.rs:62-83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L62-L83) |
| `mise run live:verify-clean` | stdout, exit 0 clean and 1 leaked | Three outcomes, not two: **leak** (still billing and nothing intends to keep it), **standing** (the Terraform stack, possibly on purpose), **pending** (a delete in flight — re-run in a minute) | [`scripts/verify-clean.py:7-28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/verify-clean.py#L7-L28), [`mise.toml:416-426`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L416-L426) |
| Guest OOM counters | In-guest, readable with no extra privileges | `dmesg`, and `/sys/fs/cgroup/memory.events` → `oom`, `oom_kill`, `oom_group_kill`. Poll these rather than discovering a kill after the fact | [`docs/PLATFORM.md:375-393`](/microvms-agentd/internals/platform/) |
## First-checks ladder
[Section titled “First-checks ladder”](#first-checks-ladder)
Cheapest first. Steps 1 through 6 cost nothing and make no billable AWS call. Step 10 spends money.
1. **Read the exit integer, then its row.** The integer is coarse by design; the row carries the `meaning` (what to do next) and the `finding` (which measurement explains it). `ERR_RETRYABLE` means run the identical command again, `ERR_CREDENTIALS` means waiting never helps, and `ERR_EXEC_FAILED` means the sandbox worked and your command exited non-zero — which is the one non-zero exit that says nothing is wrong with the platform. [`microvms-cli/src/exit.rs:173-258`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173-L258)
2. **Read the envelope’s `finding`, `suggestions`, and `data.kind`.** All three keys are always present. `data.kind` is the distinction the exit code deliberately collapses: five wire kinds share `ERR_PROTOCOL`, so a 400 and a 409 arrive with the same integer and different `data.kind`. Two failures sharing `ERR_CREDENTIALS` also get different `suggestions`, so read the line rather than assuming which one you have. [`microvms-cli/src/exit.rs:336-365`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L336-L365)
3. **Run `microvm doctor --binary target/aarch64-unknown-linux-musl/release/agentd`.** It is the only command that must work with nothing configured, and its check order is the diagnosis order: region first (a wrong region produces the null-message denial that reads as IAM), then whether the credential chain resolves at all — which spends no API call, so `doctor` cannot fail on a throttle — then the three Terraform outputs by name, then whether the stack is actually applied, then the managed bases, then the binary’s architecture last because it is the one failure that costs a full build cycle. [`microvms-cli/src/commands/doctor.rs:36-60`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L36-L60)
4. **Do not trust `terraform.tfstate` on disk as evidence the stack exists.** A destroyed stack leaves the file behind with an empty resource list, which is exactly the state that produces “bucket does not exist” three minutes into a build. `doctor` asks `terraform output` instead, which needs no credentials. [`microvms-cli/src/commands/doctor.rs:333-341`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/doctor.rs#L333-L341)
5. **Run `microvm ls` before anything else touches the account.** A non-empty `leaked` list from an earlier invocation is both a bill and a clue — the ledger is written before each delete is attempted, so the identifiers survive a process that died inside the call. [`microvms-cli/src/ledger.rs:11-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/ledger.rs#L11-L22)
6. **Run `mise run check`.** It is offline, free, and the definition of done: lint, security, all six Rust tiers, `schema:check`, `stubs:check`, `model:check`, `live:check`, and the release cross-compile. A drifted generated artifact — the served schema, the Python stub, a hardcoded API constraint against botocore’s model — fails here rather than in production. [`mise.toml:290-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L290-L301)
7. **If the VM is reachable: `GET /v1/health`.** One call answers six questions. `bootstrapped` false plus 503s everywhere means the run hook has not landed; `disk.under_pressure` means writes are about to be refused with 507; `identity_degraded` means this VM still shares a value from the image with every sibling restored from the same snapshot; `busy` and `execs` say whether an outside poll should keep the VM alive. [`agentd/src/routes.rs:314-338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L314-L338)
8. **Read the daemon’s own log in `/aws/lambda-microvms/`, raising `AGENTD_LOG` if `info` is not enough.** The daemon writes JSON to stdout and the platform captures it from there. If that group is empty while a build reports `reason=unknown`, the cause is the build role’s log prefix and not the service. [`agentd/src/main.rs:84-97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/main.rs#L84-L97)
9. **If a build failed: fetch the reason from `ListMicrovmImageBuilds`, and read `snapshotBuild`’s shape beside it.** `GetMicrovmImage` structurally cannot say why, and `ListMicrovmImageVersions` said nothing across three measured failures. Expect a list of builds, one per Graviton generation. A missing `snapshotBuild` means the Dockerfile broke before anything installed; `codeInstallSizeInBytes` alone with no snapshots means code installed and the daemon never became ready, which points at the daemon rather than the build. [`docs/PLATFORM.md:577-605`](/microvms-agentd/internals/platform/), [`docs/PLATFORM.md:1283-1303`](/microvms-agentd/internals/platform/)
10. **Only now spend money: `mise run live`, then `mise run live:verify-clean`.** The live tier is billable and takes about fifteen minutes, and it is the only tier that can catch a fake more forgiving than the real daemon. Teardown reporting success and the account being clean are different questions, so the leak check runs independently of the code that did the cleanup, and expect to run `--delete` more than once because an image refuses deletion while its VM is still terminating. [`mise.toml:428-429`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L428-L429), [`scripts/verify-clean.py:7-28`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/verify-clean.py#L7-L28)
## Known incident patterns
[Section titled “Known incident patterns”](#known-incident-patterns)
These recur. Each is recorded in `.erpaval/solutions/` or in `docs/PLATFORM.md` with a date.
* **The green run that measured nothing:** the most common pattern in this project’s history. The first OOM probe allocated with `python3`, which `amazonlinux:2023-minimal` does not have; it reported `command not found` with exit 127 and every downstream check passed. The second allocated with `dd` into `/dev/shm`, which is tmpfs and capped near half of RAM, so `dd` stopped at 64 MiB against a 1 GiB request and exited 0. Signal: a suite that passes while a condition you expected to observe never appears. Mitigation: assert on the verdict computed from the input, never on the absence of failure. [`docs/PLATFORM.md:395-412`](/microvms-agentd/internals/platform/)
* **Containment without a verdict:** removing the `?` from `parts.pop()?` in the tar path-resolution loop turned `../x` into `x` instead of an error, and the whole proptest suite passed because a filesystem walk cannot see it — the archive landed entirely inside the root. Signal: a policy test that only checks where files ended up. Mitigation: compute the expected status from the generated member and assert on it, which makes the same break shrink to a one-member archive. [`.erpaval/solutions/test-failures/proptest-and-dst-tiers-need-verdict-assertions.md:11-33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/test-failures/proptest-and-dst-tiers-need-verdict-assertions.md#L11-L33)
* **The guard never watched failing:** five distinct shapes found in one session. A bare `compile_fail` block passes for any build error including a typo in the doctest, so each is pinned to a measured rustc error code. A fake that models the failure *event* cannot catch lateness, because a client refreshing too late presents a token with no life left rather than an expired one — the fake had to measure the remaining margin. Uniform proptest draws almost never land in the narrow band where a rounding bug lives. And a guard can require the very divergence it should catch. Signal: a guard you have never seen red. Mitigation: break the invariant, watch that specific test fail, restore. [`.erpaval/solutions/test-failures/guards-that-passed-against-broken-code.md:13-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/test-failures/guards-that-passed-against-broken-code.md#L13-L37)
* **The ordering defect no guard can see:** every guard fired and every refusal test passed, yet the S3 upload ran *before* the guards refused, so a rejected request still cost a PUT. A test asserting “the bad request is refused”, or even “zero control-plane calls”, stays green across the broken ordering, because an upload is not a wire call and had no recorder. Signal: a side effect with no channel to assert absence through. Mitigation: give the side effect its own recorder — a separate `uploads` vec, deliberately not mixed into `calls` — and assert it is empty on a request the library refuses. Falsification is a pure reorder, and both call sites need the break run separately. [`.erpaval/solutions/best-practices/ordering-defects-need-their-own-recorder-channel.md:9-30`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/best-practices/ordering-defects-need-their-own-recorder-channel.md#L9-L30)
* **The fake more forgiving than the real server:** 310 fake-backed tests were green over a client whose auth-header injection replaced the request headers, stripping content-type, where the real daemon’s typed extractor answered 400. Separately, the `run` envelope published a null `agentToken` through 139 CLI tests because nothing round-tripped the envelope into a second command. Signal: a live tier failing a request every local tier accepts. Mitigation: run the live tier before trusting a transport that has only ever spoken to fakes, because a fake that accepts what the real server rejects converts integration bugs into production bugs. [`.erpaval/solutions/test-failures/guards-that-passed-against-broken-code.md:39-47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/test-failures/guards-that-passed-against-broken-code.md#L39-L47)
* **Two clocks in one test:** under a virtual-time simulator everything inside the simulation runs on the paused clock and a spawned child does not — measured here, 2 real seconds of child sleep elapsed while the simulation advanced 30 virtual ones. A server-side deadline measured against real work therefore expires in milliseconds of wall time: `output_linger` is a virtual 5 seconds against a real pipe drain, so the waiter abandons a still-writing child and the exec reports missing output with `writers_may_be_alive` set. Signal: simulator-induced truncation that looks like a daemon defect. Mitigation: never pace a child with `sleep` in a simulated test — make it block on `read` released by an explicit stdin write, so the harness is the clock. Loosening the assertion to match would encode the artifact as expected behavior. [`.erpaval/solutions/test-failures/simulated-time-and-real-children-are-two-clocks.md:13-39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/test-failures/simulated-time-and-real-children-are-two-clocks.md#L13-L39)
* **The absent value that is not neutral:** an agreement guard has to decide separately what a missing value means, and the deciding question is what the *consumer* does with absence. A consumer that errors out makes absence safe to pass; a consumer with a silent fallback makes absence a disagreement in disguise, because the fallback is a third value nobody wrote down. Signal: a constant whose docstring justifies it in terms of another component’s default — “four times the daemon’s fifteen-second SSE keepalive”. Mitigation: grep docstrings for “twice the”, “four times”, “matching the”, “same as the daemon’s”; and treat an *unparseable* value wherever the absent one lands, since `env_parse` warns and keeps the default. [`.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md:12-37`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md#L12-L37)
* **A constructor that only ever ran under a fake:** `aws-config` built with `default-features = false` looks right when you have hand-rolled an HTTP client, and it is wrong — the credential chain does its own HTTP for IMDS, SSO, and STS, and `load()` panics with `"a http_client is required"` before asking any credential question. Nothing caught it because all 300 tests constructed through the injectable transport, and the one constructor that talks to the world had no test at all. Signal: a `new()` that touches the real environment with no test calling it. Mitigation: `default-https-client` stays on ([`microvms-core/Cargo.toml:59-62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L59-L62)), and one test constructs the real transport and accepts either `Result` flavor — a panic is the bug ([`microvms-core/src/control/transport.rs:899-908`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L899-L908)). [`.erpaval/solutions/api-patterns/aws-config-needs-its-own-http-client.md:11-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/api-patterns/aws-config-needs-its-own-http-client.md#L11-L22)
* **The credential in a derived `Debug`:** three of six token-carrying types in this workspace leaked secrets through `#[derive(Debug)]` while their three siblings hand-wrote redaction — the invariant was known and still missed half its sites, because a derive is the default and nothing flags it. Signal: any struct holding a token, an `Authorization` header, or a hook payload. Mitigation: hand-write `Debug` printing names and lengths only, add a per-type guard that formats with `{:?}` and asserts the secret absent, and redact *all* header values rather than an allowlist. [`.erpaval/solutions/best-practices/credential-structs-never-derive-debug.md:10-20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/best-practices/credential-structs-never-derive-debug.md#L10-L20)
* **The golden figure taken from the plan:** the plan pinned a 2 GB break-even at about 1357s and the oracle prints 1371.2916483478837. A golden test built from the plan’s number would have been the one check that agreed with a plausible wrong answer, since the port and its test derived from the same mistaken source. Signal: a pinned figure or output-contract string whose provenance is a document rather than an execution. Mitigation: capture every golden by running the oracle, paste it verbatim, and cite it. [`.erpaval/solutions/best-practices/run-the-oracle-never-rederive-goldens.md:10-19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/best-practices/run-the-oracle-never-rederive-goldens.md#L10-L19)
* **The tokio traps in the exec path:** `child.id()` returns `None` once the child has been polled to completion, so a pgid read lazily in the kill path yields `None`, the group signal never goes out, and a kill test asserting only on the HTTP status still passes while the process tree survives. Privilege demotion uses `Command::uid()/gid()` and never `pre_exec`, because running interpreted code between fork and exec is unsafe with threads. And a `std::sync::Mutex` guard is never held across an await. Signal: a kill that reports success over a live process tree. Mitigation: capture the pgid immediately after spawn and assert on the observable kill outcome. [`.erpaval/solutions/best-practices/pipes-not-tempfiles-for-subprocess-output.md:32-47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/best-practices/pipes-not-tempfiles-for-subprocess-output.md#L32-L47)
* **The simulator that needs two specific bounds:** making the serve path generic over `axum::serve::Listener` costs nothing in production and buys deterministic network simulation — but omitting `L::Addr: Debug` fails with an E0277 saying `Serve is not a future`, which points nowhere near the missing bound, and `turmoil::Builder::enable_tokio_io()` is required whenever the served code registers a signal handler or graceful shutdown panics inside the host. Signal: either of those two errors while adding a simulated test. [`.erpaval/solutions/api-patterns/axum-listener-trait-enables-turmoil.md:36-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/api-patterns/axum-listener-trait-enables-turmoil.md#L36-L42)
* **The security control that breaks the platform:** both the platform’s lifecycle hooks and the harness’s control requests arrive from `127.0.0.1`, because the endpoint proxy terminates outside the VM and forwards over loopback. A source-address rule rejecting loopback callers on the bootstrap route therefore rejects the platform’s own legitimate bootstrap and breaks every launch; an attempt at one broke 39 tests, and those failures were reporting a real defect. Mitigation: the one-shot bootstrap is the only available defense on that route, and its sufficiency is checked in the `model/` crate. [`docs/PLATFORM.md:463-485`](/microvms-agentd/internals/platform/)
* **The probe that looks like an attack:** the daemon receives raw TLS handshake bytes on its plaintext port before bootstrap — `code 400, message Bad request version ("\x13\x01\x13\x02...")` is a ClientHello reaching a plaintext HTTP server. Something in the platform’s path probes the port with TLS first. Signal: that line in the logs. Mitigation: none needed — the correct response is a 400 and a debug-level log, and it must not take the listener down. [`docs/PLATFORM.md:487-499`](/microvms-agentd/internals/platform/)
* **The hostile header that killed the handler:** `hmac.compare_digest` raises `TypeError` on a `str` containing non-ASCII, so `Bearer tökén` took down a handler thread and the client got `RemoteDisconnected` rather than a status it could act on. Any caller controls that header. Mitigation: the comparison runs on raw bytes and never decodes, and authorization is decided before a single body byte is read — buffering first let an unauthorized request force a 256 MB allocation on a VM whose baseline can be 512 MiB. [`agentd/src/auth.rs:2-14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/auth.rs#L2-L14)
* **Silent until the disk is already full:** `anthropics/claude-code#59856`, cited by number in source, filled two 10 GB disks to 100% and the first symptom was `useradd` failing rather than the workload — by which point every writer in the sandbox was broken, including the ones that cannot report anything. Mitigation: refuse a write that would cross a reserve before it starts, answer 507 with the actual free bytes, and put the number on `/v1/health` so the curve is visible while there is still time. [`agentd/src/config.rs:98-105`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L98-L105), [`protocol/src/health.rs:21-34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L21-L34)
## See also
[Section titled “See also”](#see-also)
* [impact analysis](impact-analysis.md) — 18 shared source citations
* [business logic](business-logic.md) — 17 shared source citations
* [contract map](contract-map.md) — 16 shared source citations
* [processes](../behavior/processes.md) — 15 shared source citations
* [tech debt](tech-debt.md) — 11 shared source citations
# Impact analysis
> A **high-impact surface** here is a definition with a high inbound reference count measured across the whole tree — all seven Rust crates, the Python conformance driver, the two…
A **high-impact surface** here is a definition with a high inbound reference count measured across the whole tree — all seven Rust crates, the Python conformance driver, the two Python gate scripts, the CI workflow, and the committed artifacts. Reference count rather than public-export count, because this workspace’s public API is small and its couplings are wide: the widest surface below is named by 29 files, and the narrowest of the eight by 22.
Two properties of the tree make reference count the right criterion, and the same two explain why a `cargo`-only import graph understates the blast radius:
* **Three of the eight surfaces reach a consumer through string matching rather than through the type system.** `constants::as_json()` is read by a Python script that looks its keys up by name ([`microvms-core/src/constants.rs:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L40)); `pinned_rates()`’s decimal literals are parsed out of the Rust source by another Python script ([`scripts/check-live-rates.py:148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L148)); the conformance oracle asserts on `WireKind`’s rendered strings ([`conformance/run_rs.py:191`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L191)). When one of those couplings changes, compilation still succeeds and the corresponding check silently stops comparing.
* **Two surfaces have a generated artifact downstream.** `docs/schema.json` is generated from the protocol types and byte-compared ([`agentd/tests/schema_artifact.rs:39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L39)), and the CLI manifest is generated from the exit table and the clap tree ([`microvms-cli/src/manifest.rs:34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L34)).
Each surface below carries a **Gate** line naming the check that catches a breaking change to it, because the gate is what decides whether a mistake is caught in 45 seconds or in production. One surface has no gate, and it is called out as such.
The `Touch on change` column answers whether a consumer needs an edit, or at minimum a deliberate read, in the same commit. `yes` means an edit is required. `likely` means it compiles but a reviewer has to look. `no` means only a behavioral change reaches it.
Where a surface has more consumers than are useful to list, the top rows are the ones with the highest reference count and a `(N more …)` row summarizes the rest.
## Protocol wire types
[Section titled “Protocol wire types”](#protocol-wire-types)
Defined at: [`protocol/src/lib.rs:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L29)-[`:32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L32) (the module list), with the shapes in [`protocol/src/exec.rs:24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L24) (`Phase`), [`protocol/src/fs.rs:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/fs.rs#L18) (`FsQuery`), [`protocol/src/health.rs:11`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L11) (`Health`), [`protocol/src/hook.rs:46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/hook.rs#L46) (`RunHook`), and the two published constants at [`protocol/src/lib.rs:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L58) (`PROTOCOL_VERSION`) and [`:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L66) (`VERSION_HEADER`).
Gate: a shape change is a **compile error in four crates by design** ([`microvms-core/Cargo.toml:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L29) states that a field renamed in `protocol/` must break core’s build), and the generated document is byte-compared by [`agentd/tests/schema_artifact.rs:39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L39), wired into the unconditional local gate as [`mise.toml:168 [tasks."schema:check"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L168).
29 files reference the crate, 121 references in total. Exactly four crates declare the dependency: [`agentd/Cargo.toml:15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/Cargo.toml#L15), [`microvms-core/Cargo.toml:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L29), [`microvms-js/Cargo.toml:25`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/Cargo.toml#L25), [`microvms-py/Cargo.toml:32`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/Cargo.toml#L32). `microvms-cli` deliberately declares none — it names the wire types through core’s re-export ([`microvms-cli/Cargo.toml:48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L48)-[`:49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L49), [`microvms-core/src/lib.rs:77`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/lib.rs#L77)), which is what keeps the CLI’s direct dependency set at six.
| Downstream | Type | Touch on change | Citation |
| --------------------------------------- | ------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docs/schema.json` (committed artifact) | config | yes | [`agentd/src/bin/schema.rs:44`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/bin/schema.rs#L44) writes it, resolving the path at [`:99`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/bin/schema.rs#L99); [`agentd/tests/schema_artifact.rs:39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L39) byte-compares it |
| `agentd/src/schema.rs` | direct import | yes | [`:51`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L51) re-exports `PROTOCOL_VERSION`; [`:286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L286) merges the `$defs` rendered from these types |
| `agentd/src/exec.rs` | direct import | yes | [`:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L73) imports the `ERROR_*` and `EVENT_*` names; [`:87`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L87) re-exports the exec types |
| `microvms-cli/src/commands/attached.rs` | indirect | yes | 22 `protocol::` references through core’s re-export, e.g. [`:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L178) `Phase::Running`, [`:253`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L253) `ExitEvent`, [`:442`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L442) `phase_name` |
| `microvms-core/src/session/exec.rs` | direct import | yes | 16 references; [`:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L69) and [`:71`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L71) are public struct fields of `protocol::exec::Phase` / `Outcome`; [`:118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/exec.rs#L118) aliases `StdinResponse` |
| `microvms-core/src/session/sse.rs` | direct import | yes | [`:272`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L272), [`:295`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L295), [`:304`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L304) dispatch on `EVENT_OUTPUT` / `EVENT_GAP` / `EVENT_EXIT`; [`:255`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L255) wraps `ExitEvent` |
| `microvms-core/src/session/mod.rs` | direct import | yes | [`:329`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L329) and [`:345`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L345) return `protocol::health::Health`; [`:380`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/mod.rs#L380) takes `protocol::exec::StartRequest` |
| `agentd/src/routes.rs` | direct import | yes | [`:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L18)-[`:20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L20) re-export `VERSION_HEADER`, `Health`/`DiskHealth`, and `HOOK_PREFIX`/`RunHook`/`RunHookEnvelope`/`RunHookError` |
| `microvms-js/src/session.rs` | direct import | yes | [`:142`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L142) builds a `protocol::exec::StartRequest`; [`:577`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/session.rs#L577) enumerates `Phase::ALL` so a new phase appears without an edit |
| `microvms-py/src/session.rs` | direct import | yes | [`:338`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L338) and [`:400`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L400) build `StartRequest`; [`:598`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/session.rs#L598) maps `Phase::ALL` through `Phase::as_str` |
| `agentd/src/fs.rs` | direct import | likely | [`:95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/fs.rs#L95) re-exports `FileReadQuery` and `FsQuery`; touched when the fs query shape moves |
| `microvms-js/index.d.ts` | config | likely | a build product, **gitignored** at `.gitignore:29` and regenerated by the `napi build --platform` step at [`.github/workflows/ci.yml:341`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L341); nothing compares it against the crate |
| `microvms-core/tests/turmoil_client.rs` | test | yes | 10 references; [`:848`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L848) and [`:861`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L861) synthesize real `OutputEvent` / `ExitEvent` frames against `EVENT_OUTPUT` ([`:855`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L855)) and `EVENT_EXIT` ([`:870`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L870)) |
| `agentd/tests/schema_artifact.rs` | test | yes | [`:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L58) is the staleness check; [`:71`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L71) renames `"exec_id"` to `"execId"` to prove it can fail; [`:431`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L431) pins the event names to `["output", "gap", "exit"]` |
| `conformance/run_rs.py` | test | likely | [`:1692`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L1692) embeds the exact `exit` event object as a fixture; [`:1990`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L1990) asserts the last NDJSON record is the `exit` event |
| `model/src/lib.rs` | indirect | likely | [`:74 ExecPhase`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/src/lib.rs#L74) is a deliberate independent mirror of `Phase`, declared as such at [`protocol/src/exec.rs:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/exec.rs#L16); there is no compile edge, so it drifts silently |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes)
* **Every type derives both halves of serde, and the pairing is a requirement rather than a convenience.** [`protocol/src/lib.rs:23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/lib.rs#L23) states the rule: a type carrying one half is a type the other side has to hand-write. Adding a `Serialize`-only type reopens the drift class the crate was extracted to close.
* **The schema is generated under two serde contracts and their `$defs` are merged, so a serde attribute can fail the build on a name collision rather than on the field.** [`agentd/src/schema.rs:242`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L242) builds a `for_serialize` generator and [`:245`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L245) a `for_deserialize` one; [`:286 merge_definitions`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L286) reports a name whose content differs instead of resolving it, and [`:321`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L321) publishes `definition_collisions` into the document, asserted empty at [`:764`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/schema.rs#L764).
* **A doc comment on a wire field is a schema change.** [`protocol/src/health.rs:17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/protocol/src/health.rs#L17) records that the `version` field’s comment is deliberately a `//` and not a `///`, because schemars publishes doc comments as `description` and the artifact is compared byte for byte.
## The Error kind / wire\_kind taxonomy
[Section titled “The Error kind / wire\_kind taxonomy”](#the-error-kind--wire_kind-taxonomy)
Defined at: [`microvms-core/src/error.rs:43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L43) (`Error`), [`:127`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L127) (`ErrorKind`, 13 variants enumerated at [`:166`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L166)), [`:219`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L219) (`WireKind`, 13 variants enumerated at [`:272`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L272)).
Gate: [`microvms-core/src/error.rs:433 every_kind_carries_its_python_err_code`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L433) — every kind must carry an `ERR_*` code — plus [`:459 no_two_kinds_share_a_code`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L459) and, across the crate boundary, [`microvms-cli/src/exit.rs:486 the_exit_table_and_cores_error_kinds_are_the_same_thirteen_classes`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486). The Python side is gated by [`mise.toml:179 [tasks."stubs:check"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179); the Node side is not gated at all.
The two types are two contracts serving two different consumers: `ErrorKind` answers which exit code applies, and `WireKind` answers which status the daemon chose ([`microvms-core/src/error.rs:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L18)). `microvms-core/src/error.rs` itself holds 140 references; the heaviest consumers follow.
| Downstream | Type | Touch on change | Citation |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvms-cli/src/exit.rs` | direct import | yes | 37 references; [`:140 Exit::for_kind`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L140) is an exhaustive match over `ErrorKind`, so a 14th kind is a compile error here |
| `microvms-py/src/errors.rs` | direct import | yes | 18 references; [`:129 exception_for`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L129) matches every kind onto one of the 13 `create_exception!` classes declared at [`:33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L33)-[`:117`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/errors.rs#L117) (one base plus thirteen) |
| `microvms-cli/src/envelope.rs` | direct import | yes | 9 references; [`:321 error()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L321) emits the failure envelope and [`:327`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L327) writes `data.kind` from the wire kind |
| `microvms-js/src/errors.rs` | direct import | yes | [`:70 code_chain`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L70) is the single conversion out to JS; the module docs at [`:35`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L35) and [`:43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L43) fix the contract as `err.cause.message` for the code and `err.cause.cause.message` for the wire kind |
| `microvms-core/src/session/http.rs` | direct import | yes | [`:126`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/http.rs#L126) is the sole non-test caller of `WireKind::from_status`, so the status table’s shape is this file’s contract |
| `microvms-core/src/control/transport.rs` | direct import | likely | 29 references; [`:38`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L38) imports `ErrorKind` and [`:130`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L130), [`:156`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L156), [`:222`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L222) are control-plane raise sites; [`:306`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L306) records that `WireKind` is the daemon’s discipline and has no role here |
| `microvms-core/src/control/image.rs` | direct import | likely | 29 references, all classifying at the point of raise; [`:284`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L284)-[`:285`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L285) document `BuildWedged`, `Platform`, and `Timeout` as three distinct build outcomes |
| `microvms-core/src/control/microvm.rs` | direct import | likely | 23 references; [`:303`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L303)-[`:304`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/microvm.rs#L304) record that a missing proxy-auth key is `ErrorKind::Retryable` via `WireKind::AuthTokenMint` because minting sits inside the retry path |
| `microvms-core/src/sandbox.rs`, `session/mod.rs`, `control/mod.rs`, `session/proxy.rs`, `control/artifact.rs`, `session/exec.rs`, `cost.rs`, `session/sse.rs`, `session/files.rs` | direct import | likely | (9 more direct imports, 4-17 references each, all raise sites under `microvms-core/src/`) |
| `microvms-cli/src/guards.rs` | test | yes | 23 references; the classification half of the exit catalogue, inducing each failure at the seam ([`:71`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L71), [`:108`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L108), [`:723`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L723)) |
| `conformance/run_rs.py` | test | yes | [`:191`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L191) documents `data.kind` as a `microvms_core::WireKind` and [`:226`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L226) asserts `Conflict` and `NotFound` are distinguishable by exception type |
| `microvms-core/tests/turmoil_client.rs` | test | yes | 7 references; [`:452`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L452) and [`:726`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L726) assert `WireKind::Transport`, [`:781`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L781) and [`:1383`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/tests/turmoil_client.rs#L1383) assert `WireKind::AuthTokenMint` |
| `microvms-py/tests/test_smoke.py` | test | yes | [`:412`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/tests/test_smoke.py#L412) asserts one exception per kind under one shared base; [`:275`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/tests/test_smoke.py#L275) asserts `wire_kind is None` for a local reject |
| `microvms-js/__test__/smoke.mjs` | test | yes | [`:345`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L345) asserts exactly thirteen `ERR_*` codes are enumerable, one per `ErrorKind` ([`:347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L347)) |
| `microvms-cli/src/seam.rs`, `commands/lifecycle.rs` | direct import | likely | 8 and 5 references on the classify-and-report path — [`microvms-cli/src/seam.rs:291`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L291), [`:306`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L306), [`:316`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L316); [`microvms-cli/src/commands/lifecycle.rs:191`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L191), [`:729`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L729), [`:741`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L741) |
| `agentd/src/fs.rs`, `agentd/src/exec.rs`, `agentd/src/disk.rs`, `agentd/src/identity.rs`, `agentd/tests/turmoil_transport.rs` | indirect | no | these are `std::io::ErrorKind`, not core’s — the name collides but the type does not |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-1)
* **`ErrorKind` is derived from `WireKind` and must never be passed in beside it.** [`microvms-core/src/error.rs:77 Error::wire`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L77) computes the kind from the wire kind precisely so the two cannot disagree, and [`:366 WireKind::error_kind`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L366) is the mapping. An `Error::wire_with_kind` escape hatch would let a 401 be classified as retryable and retried forever.
* **`from_status` has no generic 4xx fallback, and that absence is a tested invariant.** [`microvms-core/src/error.rs:343`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L343) maps the statuses explicitly, and [`:519 no_generic_four_hundred_fallback_can_produce_a_protocol_error`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L519) asserts the unmapped ones resolve to `None`. A generic fallback would make a protocol typo look like a missing file, which is the defect named in [`docs/PROTOCOL.md:49`](/microvms-agentd/internals/protocol/). 5xx statuses do fall back, to `ServerError`, with 503 excepted as `NotBootstrapped` ([`microvms-core/src/error.rs:539`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L539)).
* **The retryable set is derived from the kind, never stored, and the test compares it against an independently restated table.** [`microvms-core/src/error.rs:116 Error::retryable`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L116) reads `ErrorKind::Retryable`; [`:471 retryable_agrees_with_the_python_exception_contract`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L471) compares that against a `#[cfg(test)]` restatement, and [`:485 exactly_five_wire_kinds_are_retryable`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L485) pins the cardinality. A sixth added by mistake fails there rather than in a non-terminating retry loop.
## The exit table
[Section titled “The exit table”](#the-exit-table)
Defined at: [`microvms-cli/src/exit.rs:78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L78) (`Exit`, `#[repr(u8)]` with explicit discriminants) and [`:173`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173) (`EXIT_TABLE: [ExitRow; 14]`).
Gate: [`microvms-cli/tests/exit_codes.rs:29 every_locally_reachable_row_exits_with_its_own_integer_and_code`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L29) drives real spawned binaries, and [`microvms-cli/tests/manifest.rs:161`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L161) cross-checks the published table against what the binary actually exits. Inside the crate, [`microvms-cli/src/exit.rs:486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486) asserts the table and core’s `ErrorKind` describe the same thirteen classes, and [`:512`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L512) asserts the mapping is injective.
The table is append-only. Its 14 rows are the contract three consumers read: a shell reading `$?`, an agent reading the `--json` envelope’s `code`, and the conformance oracle reading `exitCode`. 34 files reference `Exit` or `EXIT_TABLE`, 219 references in total.
| Downstream | Type | Touch on change | Citation |
| ----------------------------------------------------------------------------------------------------- | ------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `microvms-cli/src/manifest.rs` | direct import | yes | [`:85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L85) publishes every row as `exitCodes`, asserted at [`:336 the_manifest_carries_all_fourteen_exit_rows`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L336) |
| `microvms-cli/src/main.rs` | direct import | yes | the process exit path; `Exit::as_u8` ([`microvms-cli/src/exit.rs:109`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L109)) is what reaches `$?` |
| `microvms-cli/src/envelope.rs` | direct import | yes | the failure envelope carries `exitCode`, `code`, and `finding` off the row |
| `microvms-core/src/error.rs` | indirect | yes | `ErrorKind::ALL` ([`:166`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L166)) and `ErrorKind::code` ([`:187`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/error.rs#L187)) are the other half; [`microvms-cli/src/exit.rs:486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486) asserts the two describe the same thirteen classes |
| `microvms-cli/src/commands/lifecycle.rs`, `attached.rs`, `local.rs`, `cost.rs`, `doctor.rs`, `mod.rs` | direct import | likely | (6 more direct imports under `microvms-cli/src/commands/`; every command constructs a `CliError` carrying an `Exit`, the shape declared at [`microvms-cli/src/exit.rs:78`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L78)) |
| `microvms-cli/tests/exit_codes.rs` | test | yes | [`:29`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L29) asserts integer, code, and finding together over every locally reachable row; [`:122`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L122) pins the shared argument-error code; [`:286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L286) pins the streaming exception |
| `microvms-cli/tests/manifest.rs` | test | yes | [`:161 the_published_exit_table_agrees_with_what_the_binary_exits`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L161) |
| `microvms-cli/src/guards.rs` | test | yes | the classification half — induces the rows an invocation cannot reach without an account, asserting the row directly ([`:948`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L948) `Exit::Interrupted`, [`:1183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L1183) `Exit::Precondition`) |
| `conformance/run_rs.py` | test | yes | [`:287`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L287) cross-checks the process exit code against the envelope’s own `exitCode`, and [`:302`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L302) names CLI-3 as the claim that they agree; [`:384`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L384) repeats it on the streaming path |
| `docs/PLATFORM.md` | config | likely | rows carry a `finding` naming a section of it ([`microvms-cli/src/exit.rs:62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L62), [`:127`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L127), [`:312`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L312); the module docs at [`:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L10) state each platform code names a different finding) |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-2)
* **The table is indexed by the discriminant, so `Exit::row` is infallible — and the same indexing means a reordered row silently returns a neighbour’s data.** [`microvms-cli/src/exit.rs:118`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L118) returns `&EXIT_TABLE[self.as_u8() as usize]` ([`:119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L119)), and [`:452 the_table_is_indexed_by_the_exit_integer`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L452) pins the correspondence. The explicit `#[repr(u8)]` discriminants exist because a variant inserted mid-enum renumbers everything after it, which for this type means silently rewriting the published contract.
* **The rows are spelled out rather than generated from `ErrorKind::code`, on purpose.** [`microvms-cli/src/exit.rs:170`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L170) records the reason: a generated table would agree with a typo. The cross-check at [`:486`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L486) is what makes the duplication safe, and [`:512 the_kind_to_exit_mapping_is_injective`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L512) keeps two kinds from collapsing onto one row — the named temptation being `Precondition` and `InvalidArg`, which need separate rows because one is fixed by editing a flag and the other by applying a Terraform stack.
* **Five `WireKind`s collapse onto one exit row on purpose, and `data.kind` preserves the distinction.** [`microvms-cli/src/exit.rs:534 the_five_protocol_wire_kinds_collapse_and_the_others_do_not`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L534) pins the collapsing set. Widening the exit table to split them would break the append-only rule; narrowing `data.kind` would leave the conformance oracle unable to tell them apart ([`conformance/run_rs.py:191`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L191)).
## constants.rs and its JSON emission
[Section titled “constants.rs and its JSON emission”](#constantsrs-and-its-json-emission)
Defined at: [`microvms-core/src/constants.rs:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L57)-[`:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L455) (the constants, from `MODEL_API_VERSION` to `DEAD_STATES`) and [`:589`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L589) (`as_json`).
Gate: [`mise.toml:229 [tasks."model:check"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L229), which runs `./scripts/check-model-drift.py` ([`:257`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L257)) and compares every emitted key against the pinned botocore service model. It sits in `check` rather than `live` because the model is a file inside botocore — no network, no credentials. Inside the crate, [`:693 as_json_carries_every_key_the_drift_gate_reads`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L693) and [`:752 the_emitted_values_are_the_measured_ones`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L752) hold the key set and the literals.
Every value here is transcribed from the botocore service model for `lambda-microvms`, API version `2025-09-09` ([`:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L57)). The JSON key names are a contract with a Python script, so renaming a key is a breaking change the compiler accepts — the module states the coupling at [`:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L40). 24 files reference the module, 182 references in total.
| Downstream | Type | Touch on change | Citation |
| ------------------------------------------------------------ | ------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scripts/check-model-drift.py` | config | yes | [`:95 RUST_SOURCE_ARGV`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L95) reads the object through `microvm constants --emit-json` ([`:103`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L103)), which [`:45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L45) names as the only client; [`:149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L149) is the key list, spelled as `as_json()`’s keys |
| `microvms-cli/src/commands/local.rs` | direct import | yes | [`:219`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L219) calls `as_json()`; [`:223`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L223) prints the bare object as the one non-envelope stdout write in the binary, and [`:206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L206) records that the keys are the gate’s contract |
| `microvms-core/src/control/mod.rs` | direct import | yes | [`:483`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L483) checks `MAX_DURATION_SEC`; [`:491`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L491) and [`:518`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L518) name `MODEL_API_VERSION` in the refusal text |
| `microvms-core/src/control/token.rs` | direct import | yes | [`:47`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L47) imports `MAX_CLIENT_TOKEN_LEN` and [`:139`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L139) enforces it; [`:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/token.rs#L69) records that the ceiling is measured against the worst legal scope because the run token folds a full ARN in |
| `microvms-core/src/control/image.rs` | direct import | yes | [`:91`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L91)-[`:92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L92) read both ready-state sets; [`:171`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L171) and [`:185`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L185) read `ARCHITECTURES[0]` and `CAPABILITIES[0]` |
| `microvms-core/src/hooks.rs` | direct import | yes | [`:40`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L40) imports both hook-timeout ceilings; [`:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L58) and [`:86`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L86) are the two newtypes’ `MAX_SECS` |
| `microvms-core/src/control/transport.rs` | direct import | yes | [`:47 const API_PATH_VERSION = crate::constants::MODEL_API_VERSION`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L47) — the request path is built from it, and [`:43`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L43) says it is read rather than written again |
| `microvms-core/src/sizing.rs`, `microvms-core/src/region.rs` | indirect | yes | `as_json` reaches into `crate::sizing::SIZE_CLASSES` at [`microvms-core/src/constants.rs:660`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L660) and `MICROVM_REGIONS` at [`microvms-core/src/constants.rs:651`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L651), so editing either table changes the gate’s payload |
| `microvms-core/src/sandbox.rs` | direct import | likely | [`:870`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L870) reads `DEAD_STATES` on the launch guard |
| `microvms-cli/src/commands/lifecycle.rs` | direct import | likely | [`:988`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L988) reads `DEAD_STATES`; [`:974`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L974) records that failing fast on it beats burning the poll budget |
| `microvms-cli/tests/manifest.rs` | test | yes | [`:229 constants_emit_json_writes_the_bare_object_the_drift_gate_reads`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L229); [`:297`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L297) asserts the command is listed rather than hidden |
| `microvms-cli/src/commands/local.rs` (own tests) | test | yes | [`:399`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L399) asserts the parsed output equals `microvms_core::constants::as_json()` |
| `docs/PLATFORM.md`, `docs/TRUST.md`, `docs/STRATEGY.md` | config | likely | [`docs/PLATFORM.md:64`](/microvms-agentd/internals/platform/) documents the 4096-byte ceiling and [`docs/PLATFORM.md:92`](/microvms-agentd/internals/platform/) records that a commit “correcting” it to 16384 fails a test; [`docs/TRUST.md:315`](/microvms-agentd/internals/trust/) and [`docs/STRATEGY.md:89`](/microvms-agentd/internals/strategy/) restate it |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-3)
* **These guards exist because botocore does not enforce the limits itself.** [`microvms-core/src/constants.rs:14`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L14) records the measurement: `VALIDATED_METADATA_ATTRS` is `{'required', 'min', 'document', 'union'}`, so `max`, `pattern`, and `enum` violations reach the wire. Deleting a guard on the assumption that the SDK validates the model reopens all of them.
* **`DEAD_STATES` is a strict subset of `TERMINAL_STATES`, and `SUSPENDED` must stay out of it.** [`microvms-core/src/constants.rs:448`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L448) lists four terminal states including `SUSPENDED`, [`:455`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L455) lists two dead ones, and [`:878 every_dead_state_is_also_a_terminal_state`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L878) asserts the containment. [`:452`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L452) gives the reason: `SUSPENDED` means death when it occurs before `RUNNING` and is also an ordinary waypoint on the resume path, so a resume that failed fast on it would fail on every resume.
* **The two image-ready sets must stay disjoint, or the gate reports a tolerated spelling as model-backed.** [`microvms-core/src/constants.rs:431`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L431) is checked against the model exactly and [`:441`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L441) exists because the service has answered differently across API versions; [`:941 the_model_and_tolerated_ready_states_do_not_overlap`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L941) keeps them apart.
## The size-class table
[Section titled “The size-class table”](#the-size-class-table)
Defined at: [`microvms-core/src/sizing.rs:68`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L68) (`SIZE_CLASSES`, 5 rows / 20 numbers) and [`:113`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L113) (`SizeClass`).
Gate: [`scripts/check-model-drift.py:266 PINNED_SIZE_CLASSES`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L266) is a deliberate literal twin compared against the emitted table, reached through [`mise.toml:229 [tasks."model:check"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L229). [`mise.toml:245`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L245) records why a twin is the only possible check here: the sizing table is measurement-backed, so the service model can say nothing about it and client-versus-client is the only comparison available. In-crate, [`microvms-core/src/sizing.rs:273 the_documented_table_carries_the_measured_rows`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L273) pins the rows.
`minimumMemoryInMiB` selects a class whose two numbers differ by 4x; it does not size a VM directly ([`docs/PLATFORM.md:236`](/microvms-agentd/internals/platform/)). The table is the only place any of the twenty numbers appears ([`microvms-core/src/sizing.rs:20`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L20)). 35 files reference the surface, 435 references in total — `codegraph callers SizeClass` reports 36 inbound callers.
| Downstream | Type | Touch on change | Citation |
| ----------------------------------------------------------------------------- | ------------- | --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `microvms-core/src/cost.rs` | direct import | yes | the rate arithmetic multiplies `baseline_gb()` ([`microvms-core/src/sizing.rs:195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L195)) and never the peak |
| `microvms-cli/src/cli.rs` | direct import | yes | [`:240 MemoryMib`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L240) is the clap `ValueEnum` mirror and [`:255 size_class()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L255) the exhaustive mapping; [`:1034`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1034) asserts the flag domain is exactly the documented table |
| `microvms-py/src/cost.rs`, `microvms-js/src/cost.rs` | direct import | yes | [`microvms-py/src/cost.rs:488 PySizeClass`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L488) and [`microvms-js/src/cost.rs:422 SizeClass`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L422) each wrap the core type ([`microvms-js/src/cost.rs:41`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L41)) over the same five rows |
| `microvms-core/src/control/image.rs` | direct import | yes | the `resources` list on the build request |
| `microvms-core/src/constants.rs` | direct import | yes | [`:660`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L660) flattens every row into the drift gate’s JSON payload |
| `scripts/check-model-drift.py` | config | yes | [`:266 PINNED_SIZE_CLASSES`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L266); [`:247`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L247) records that a value compared only against itself passes by construction |
| `microvms-core/src/control/mod.rs` | direct import | likely | the request builders take a `SizeClass` rather than an integer |
| `microvms-cli/src/commands/lifecycle.rs`, `microvms-cli/src/commands/cost.rs` | direct import | likely | each converts the `--memory` flag to a class and keeps it a class all the way down |
| `microvms-js/src/sandbox.rs`, `microvms-py/src/sandbox.rs` | direct import | likely | the build entry points take an `Option` size ([`microvms-js/src/sandbox.rs:244`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/sandbox.rs#L244), [`microvms-py/src/sandbox.rs:464`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/sandbox.rs#L464)); [`microvms-js/src/sandbox.rs:233`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/sandbox.rs#L233) records that an off-table baseline stays refused because the only way to hold a `SizeClass` is to have parsed one |
| `microvms-cli/src/render.rs` | direct import | no | the rendering takes a report, not a class; the references are under `#[cfg(test)]` from [`:394`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/render.rs#L394) |
| `microvms-cli/tests/manifest.rs` | test | yes | [`:90 every_published_domain_is_the_domain_the_parser_enforces`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L90) feeds the published `--memory` domain back to the parser |
| `microvms-py/tests/test_smoke.py`, `microvms-js/__test__/smoke.mjs` | test | yes | [`microvms-js/__test__/smoke.mjs:302`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L302)-[`:304`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L304) asserts 1500 is refused and names TRAP-10; [`microvms-py/tests/test_smoke.py:153`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/tests/test_smoke.py#L153) asserts the refusal surfaces as `ERR_INVALID_ARG` |
| `docs/PLATFORM.md` | config | likely | [`:236`](/microvms-agentd/internals/platform/) is the finding the table transcribes; [`:244`](/microvms-agentd/internals/platform/) states the four-times pairing and [`:259`](/microvms-agentd/internals/platform/) that billing follows the baseline |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-4)
* **Every shipped peak is exactly 4x its baseline, and nothing may compute it.** [`microvms-core/src/sizing.rs:15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L15) names computing the peak as the one thing the module must not do. The regularity comes from AWS, not from this codebase, so a sixth row breaking the pattern would silently get the pattern applied. The guard is testable because the lookups are table-parameterized ([`:247 row_in`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L247), [`:255 class_for_baseline_in`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L255)), and [`:298 a_peak_that_is_not_four_times_its_baseline_is_read_not_computed`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L298) drives them over a table a computed implementation cannot answer.
* **An off-table baseline is refused rather than snapped, and the refusal is a billing decision.** [`microvms-core/src/sizing.rs:146 from_baseline_mib`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L146) rejects anything not in the table, asserted at [`:360 an_off_table_baseline_is_refused_naming_the_finding`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L360). A request of 1500 has two plausible readings — round up, or take it literally — and they differ in both the memory the guest gets and the rate it is billed at. The proptest sampler at [`:427 plausible_baseline_mib`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L427) exists because a uniform `u32` draw almost never lands where snapping is even possible.
* **Two of the numbers are each both a baseline and a peak, so a caller echoing back a `MemTotal` gets a different class silently.** [`microvms-core/src/sizing.rs:384 a_peak_that_is_not_also_a_baseline_is_refused`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L384) records the overlap, and it is why `Display` ([`:226`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L226)) always names both numbers — asserted at [`:410 display_names_the_baseline_and_the_peak_together`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sizing.rs#L410). A `Display` that named one would let someone budget for memory they are not billed for.
## The region list
[Section titled “The region list”](#the-region-list)
Defined at: [`microvms-core/src/region.rs:45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L45) (`Region`) and [`:73`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L73) (`MICROVM_REGIONS: [Region; 5]`).
Gate: [`scripts/check-model-drift.py:254 PINNED_REGIONS`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L254) is the literal twin, compared through [`mise.toml:229 [tasks."model:check"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L229); in-crate, [`microvms-core/src/region.rs:176 the_five_supported_regions_are_the_measured_ones`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L176) and [`microvms-cli/src/cli.rs:1061 the_region_domain_is_exactly_the_five_measured_regions_and_excludes_eu_central_one`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1061) hold both ends. No service model states the set — this list is maintained by hand, and the two botocore calls that look like substitutes disagree with each other ([`microvms-core/src/region.rs:21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L21)).
46 files reference the surface, 388 references in total; `codegraph callers Region` reports 60 inbound callers, the highest of any surface here.
| Downstream | Type | Touch on change | Citation |
| ----------------------------------------------------------------------------------------------------------------------- | ------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvms-py/src/region.rs` | direct import | yes | [`:32 PyRegion`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/region.rs#L32) is a hand-ported closed class with one static constructor per region |
| `microvms-js/src/region.rs` | direct import | yes | [`:35 Region`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/region.rs#L35) with a factory per region and deliberately no constructor, asserted by [`microvms-js/__test__/smoke.mjs:267`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L267) |
| `microvms-cli/src/cli.rs` | direct import | yes | [`:273 RegionArg`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L273) is the clap mirror and [`:287 region()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L287) the exhaustive mapping |
| `microvms-core/src/cost.rs` | direct import | yes | a `RateTable` is region-scoped ([`:849`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L849)), and the region is what a caller reads back ([`microvms-js/__test__/smoke.mjs:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L331)) |
| `microvms-core/src/control/transport.rs` | direct import | yes | [`:432`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L432) and [`:480`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L480) build the AWS config and the endpoint host from `region.as_str()` ([`microvms-core/src/region.rs:83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L83)) |
| `microvms-core/src/constants.rs` | direct import | yes | [`:50`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L50) imports `MICROVM_REGIONS`; [`:651`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L651) publishes it in the gate’s payload; [`:575`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/constants.rs#L575) records that it is explicitly not model-backed |
| `scripts/check-model-drift.py` | config | yes | [`:254 PINNED_REGIONS`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L254); [`:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-model-drift.py#L57) explains why the two measurement-backed values each need a second reader |
| `microvms-cli/src/seam.rs` | direct import | likely | [`:341 resolve_region`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L341) and the `CoreSeam` methods are region-parameterized |
| `microvms-core/src/control/mod.rs` | direct import | likely | [`:183 ControlPlane::new`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L183) takes a `Region` rather than a string |
| `microvms-core/src/control/connector.rs`, `control/microvm.rs`, `control/artifact.rs`, `control/image.rs`, `sandbox.rs` | direct import | likely | (5 more direct imports under `microvms-core/src/`, 2-8 references each) |
| `microvms-js/src/sandbox.rs`, `microvms-py/src/sandbox.rs` | direct import | likely | `create`/`new` takes a `Region` object rather than a string, which is what keeps the closure |
| `microvms-cli/src/commands/doctor.rs` | direct import | yes | lists the supported names and falls back to `Region::UsEast1` |
| `microvms-cli/src/guards.rs` | test | likely | the injected seams are region-parameterized ([`:82`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L82), [`:89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L89), [`:98`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/guards.rs#L98)) |
| `microvms-py/tests/test_smoke.py`, `microvms-js/__test__/smoke.mjs` | test | yes | [`microvms-js/__test__/smoke.mjs:251`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L251) asserts the five names; [`microvms-js/__test__/smoke.mjs:232`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L232) and [`microvms-py/tests/test_smoke.py:270`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/tests/test_smoke.py#L270) each assert `eu-central-1` is refused, [`microvms-py/tests/test_smoke.py:264`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/tests/test_smoke.py#L264) naming the 2026-08-07 removal |
| `microvms-cli/src/cli.rs` (tests) | test | yes | [`:1061`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1061) asserts the flag domain equals the measured five; [`:1102`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1102) asserts the unlisted escape hatch conflicts with the closed set |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-5)
* **The correctness condition runs in both directions, and an extra entry causes more damage than a missing one.** [`microvms-core/src/region.rs:24`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L24) states both cases. A missing region refuses a launch AWS would have accepted, which is recoverable — `Region::unlisted` ([`:107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L107)) exists for that case. An extra region reopens the null-message trap for a name nothing will reject ([`docs/PLATFORM.md:146`](/microvms-agentd/internals/platform/)), which is why [`microvms-core/src/region.rs:196 eu_central_one_is_refused_naming_the_null_message_trap`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L196) names that specific value and why four separate tests across three languages repeat it.
* **`unlisted()` normalizes a supported name back to its variant, so there is never a second spelling of one region.** [`microvms-core/src/region.rs:107`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L107), asserted at [`:243 the_escape_hatch_normalises_a_supported_name_to_its_variant`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L243) and mirrored in the Node binding at [`microvms-js/__test__/smoke.mjs:262`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/__test__/smoke.mjs#L262). Removing the normalization would make `unlisted("us-east-1")` an unequal value that every downstream `match` has to handle twice.
* **`supported()` is the single reader of the five spellings, and both `FromStr` and `unlisted` go through it.** [`microvms-core/src/region.rs:119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L119). A second lookup table added anywhere, including in a binding, could drift from this one; [`:219 each_supported_region_round_trips_through_its_wire_name`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/region.rs#L219) keeps `as_str` and the parse path on one table.
## The pinned cost rate table
[Section titled “The pinned cost rate table”](#the-pinned-cost-rate-table)
Defined at: [`microvms-core/src/cost.rs:1011`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1011) (`pinned_rates`), returning the `RateTable` declared at [`:849`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L849), with the five decimal literals at [`:1016`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1016)-[`:1023`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1023).
Gate: two, running at different times. [`microvms-core/src/cost.rs:2180 every_rate_byte_matches_the_python_literal`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2180) compares each field against a literal in the offline tier, and `./scripts/check-live-rates.py --twin-only` cross-checks the script’s own pinned copy against the Rust source — offline and free, per [`mise.toml:411`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L411). The billable half, [`mise.toml:395 [tasks."live:rates"]`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L395), compares both against the live AWS Pricing API; it sits in `live` rather than `check` because it needs network and credentials ([`:402`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L402)).
The figures were read from the Lambda pricing page on 2026-08-07 in us-east-1 ([`microvms-core/src/cost.rs:992`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L992)). One of them, `storage_gb_month`, is derived rather than read. 22 files reference the surface, 156 references in total.
| Downstream | Type | Touch on change | Citation |
| ---------------------------------------- | ------------- | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scripts/check-live-rates.py` | config | yes | [`:121 PINNED`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L121) restates all five figures; [`:133 TWIN_PATH`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L133) and [`:134 TWIN_FN`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L134) point at `pinned_rates`, and [`:148 verify_twin`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L148) parses the `dec!()` literals out of the Rust source |
| `microvms-cli/src/commands/cost.rs` | direct import | yes | the `cost` command’s table |
| `microvms-py/src/cost.rs` | direct import | yes | [`:576 PyRateTable`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L576) and [`:590 pinned()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/src/cost.rs#L590) — the only pinned door, with deliberately no rates-taking constructor |
| `microvms-js/src/cost.rs` | direct import | yes | [`:501 RateTable`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L501), [`:514 pinned()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L514); [`:908`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L908), [`:960`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L960), [`:982`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/cost.rs#L982) default to `cost::pinned_rates` when no table is passed |
| `microvms-cli/src/commands/lifecycle.rs` | direct import | likely | [`:449`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L449) imports `pinned_rates` and `run_report`; [`:470`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L470)-[`:473`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L473) price a completed run |
| `microvms-cli/src/render.rs` | direct import | likely | [`:399`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/render.rs#L399) reads `retrieved()` ([`microvms-core/src/cost.rs:878`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L878)) for the report header; the remaining uses are under `#[cfg(test)]` from [`:394`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L394) |
| `microvms-core/src/cost.rs` (own tests) | test | yes | [`:2180`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2180) pins all five figures as literals; [`:2205`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2205) asserts the GB-month derivation as `dec!(0.0001111111) * dec!(730)` |
| `conformance/run_rs.py` | test | likely | asserts the `cost` command and the run envelope each report a labelled estimate |
| `docs/PLATFORM.md` | config | yes | [`:293`](/microvms-agentd/internals/platform/), [`:295`](/microvms-agentd/internals/platform/), and [`:299`](/microvms-agentd/internals/platform/) carry the same figures; [`:304`](/microvms-agentd/internals/platform/)-[`:306`](/microvms-agentd/internals/platform/) carry the GB-hour → GB-month derivation. [`microvms-core/src/cost.rs:57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L57) and [`:992`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L992) both point here, so the two change in one commit |
| `mise.toml` | config | no | [`:395`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L395) wires `live:rates` to the script; [`:411`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L411) records that `--twin-only` runs first on that path |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-6)
* **Renaming `pinned_rates` breaks the twin check by name, not by compilation.** [`scripts/check-live-rates.py:134`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L134) finds the function by the literal string `"pub fn pinned_rates()"`, and [`:180`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/check-live-rates.py#L180) is the error raised when it cannot — an error that explicitly instructs the reader to repoint `TWIN_FN` rather than delete the check. The script’s pinned figures are a deliberate second copy ([`mise.toml:411`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L411)), because a drift check that imported the values it checks would compare a table against itself.
* **Money is always a `Decimal`, and the pinned figures carry ten significant digits.** The literals at [`microvms-core/src/cost.rs:1016`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1016)-[`:1023`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L1023) are `dec!()` values, not floats. Summing a few thousand per-second ARM rates in binary floating point drifts toward a bill nobody can reproduce, and [`docs/PLATFORM.md:1230`](/microvms-agentd/internals/platform/) works the example figures at full precision.
* **`storage_gb_month` is derived, and the code and the platform doc both record the earlier wrong value.** [`microvms-core/src/cost.rs:2208`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/cost.rs#L2208) holds `dec!(0.08)` in the test that proves the current figure is not it, and [`docs/PLATFORM.md:304`](/microvms-agentd/internals/platform/)-[`:306`](/microvms-agentd/internals/platform/) records that $0.08 per GB-month understated every stored GB by 1.37% against $0.0001111111 per GB-hour at AWS’s own 730-hour month. `CatalogLine` ([`:1038`](/microvms-agentd/internals/platform/)) checks the unit the API reports for exactly this reason: if AWS restated storage per GB-month, the number would change by 730x and every downstream arithmetic check would still pass, because they all read the same table.
## The CLI manifest
[Section titled “The CLI manifest”](#the-cli-manifest)
Defined at: [`microvms-cli/src/manifest.rs:34`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L34) (`build`), reading `Cli::command()`, [`microvms-cli/src/exit.rs:173 EXIT_TABLE`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L173), [`microvms-cli/src/commands/mod.rs:104 RESPONSE_TYPES`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/mod.rs#L104), and [`microvms-cli/src/envelope.rs:66 API_VERSION`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L66).
Gate: three independent directions in one file — [`microvms-cli/tests/manifest.rs:46 every_command_the_manifest_lists_is_one_the_binary_routes`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L46), [`:90 every_published_domain_is_the_domain_the_parser_enforces`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L90), and [`:161 the_published_exit_table_agrees_with_what_the_binary_exits`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L161) — plus [`conformance/run_rs.py:816`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L816), which reads the manifest at runtime and drives every command it lists.
The manifest is generated and never hand-maintained ([`microvms-cli/src/manifest.rs:4`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L4)). Because it is built from the same tables the binary runs on, it cannot drift from the binary’s actual behavior, and that guarantee is what makes it useful to an agent.
| Downstream | Type | Touch on change | Citation |
| ------------------------------------------ | ---------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `microvms-cli/src/commands/local.rs` | direct import | yes | [`:193`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L193) is the `manifest` command handler, calling `crate::manifest::build()` at [`:196`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L196); [`:186`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L186) records that a command added without a `RESPONSE_TYPES` row fails `microvms-cli/tests/manifest.rs` rather than shipping undescribed |
| `microvms-cli/src/commands/mod.rs` | indirect | yes | [`:104 RESPONSE_TYPES`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/mod.rs#L104) is the one table the manifest reads rather than introspects; [`:263 response_type`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/mod.rs#L263) is called from every command module |
| `microvms-cli/src/cli.rs` | runtime dispatch | yes | the whole clap tree is the input, so a flag added to any command appears in the manifest without an edit here, and a `--stream` added elsewhere changes `alternateResponse` ([`microvms-cli/src/manifest.rs:75`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L75)) |
| `microvms-cli/src/exit.rs` | direct import | yes | [`:85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L85) publishes all 14 rows as `exitCodes` |
| `microvms-cli/src/envelope.rs` | direct import | likely | [`:30`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L30) imports `API_VERSION`, published as the manifest’s `apiVersion` at [`:81`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L81) and emitted on every envelope ([`microvms-cli/src/envelope.rs:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L314), [`:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L331)) |
| `conformance/run_rs.py` | test | yes | [`:816`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L816) calls `microvm manifest` and [`:817`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L817)-[`:819`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L819) assert the suite drives every command it lists; [`:751`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L751) takes a fixture value out of the manifest rather than writing it down; [`:205`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L205) reads `apiVersion` back |
| `microvms-cli/tests/manifest.rs` | test | yes | [`:46`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L46), [`:90`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L90), [`:161`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L161) as above; [`:195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L195) asserts a bare invocation emits JSON |
| `microvms-cli/src/manifest.rs` (own tests) | test | yes | [`:265`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L265) asserts the command list equals the clap tree exactly; [`:426`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L426) asserts every command declares a response type and its keys; [`:450`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L450) asserts every command publishes a summary from its doc comment |
| `docs/reference/cli.md` | config | likely | [`:3`](/microvms-agentd/reference/cli/) states twenty-eight subcommands and cites the clap enum in `microvms-cli/src/cli.rs`, so it restates by hand what the manifest generates |
### Blast-radius notes
[Section titled “Blast-radius notes”](#blast-radius-notes-7)
* **The command count is asserted at 17 in three places, so adding a command is a three-file change.** [`microvms-cli/src/commands/mod.rs:104`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/mod.rs#L104) declares `RESPONSE_TYPES: [(&str, &str, &[&str]); 17]`, [`microvms-cli/src/manifest.rs:280`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L280) asserts it, and [`microvms-cli/tests/manifest.rs:51`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/manifest.rs#L51) asserts it again with the breakdown — “the lifecycle six, the attached five, and the local six”. The triple assertion is deliberate: it is what keeps `RESPONSE_TYPES` from becoming the hand-maintained artifact generation forbids ([`microvms-cli/src/manifest.rs:13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L13)).
* **`choices: null` and `choices: []` mean different things, and a boolean flag must publish neither.** [`microvms-cli/src/manifest.rs:401 a_free_text_parameter_reports_a_null_domain`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L401) asserts free text reports `null` ([`:417`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L417)), and [`:464`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L464) records that publishing clap’s `["true", "false"]` for a `SetTrue` flag would put a `choices` array on all nineteen flags — making `choices` useless as the field a reviewer scans to find the genuinely closed sets ([`:137`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L137)).
* **`exec --stream` is the one documented exception to the one-envelope-per-invocation rule, and it is published as a machine-readable fact rather than as prose.** [`microvms-cli/src/manifest.rs:75`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L75) emits `alternateResponse` keyed off the flag’s presence, and [`:292 only_exec_publishes_an_alternate_streaming_response`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L292) asserts no other command claims one and that the streaming discriminant differs from the normal one — because a consumer branching on `type` cannot otherwise tell which parse applies. [`microvms-cli/tests/exit_codes.rs:286`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L286) asserts the binary really publishes it.
## Other notable surfaces
[Section titled “Other notable surfaces”](#other-notable-surfaces)
* [`agentd/src/routes.rs:371 surface_docs()`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L371) — the single route list the router ([`:31`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L31)), the `/v1/schema` handler ([`:346`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/routes.rs#L346)), the schema binary, and five assertions in `agentd/tests/schema_artifact.rs` all walk ([`agentd/tests/schema_artifact.rs:149 every_documented_route_is_served_by_the_router`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L149), [`:207 every_bearer_route_answers_503_before_bootstrap`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L207), [`:249`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L249), [`:291`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L291), [`:349`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/tests/schema_artifact.rs#L349)). A route absent from it does not exist.
* [`microvms-core/src/hooks.rs:48 RunHookTimeout`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L48) / [`:54 BuildHookTimeout`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L54) — two newtypes with no conversion between them and separate `MAX_SECS` ([`:58`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L58) = 60, [`:86`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/hooks.rs#L86) = 3600), so a 3600-second build timeout cannot reach a field capped at 60. Mirrored in `microvms-py/src/hooks.rs` and `microvms-js/src/hooks.rs`.
* [`microvms-cli/src/seam.rs:136 CoreSeam`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/seam.rs#L136) — the trait every CLI command reaches AWS through, and the injection point the test suite substitutes at. [`microvms-cli/tests/thinness.rs:426 no_shipping_source_line_names_an_operation_or_reaches_past_the_seam`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L426) asserts no shipping source line reaches past it, and [`:457 the_scan_cut_cannot_hide_production_code`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L457) guards the scan itself.
* [`microvms-core/src/control/transport.rs:245 Transport`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L245) and `microvms-core/src/control/mod.rs:112 Clock` — the two `Send + Sync` trait seams `ControlPlane` is constructed over ([`microvms-core/src/control/mod.rs:183`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L183)), with `microvms-core/src/control/fake.rs` as the recording implementation.
* `microvms-cli/Cargo.toml`’s six-name direct dependency set — asserted as an exact equality by [`microvms-cli/tests/thinness.rs:145 the_direct_dependency_set_is_exactly_the_allowed_one`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L145) against the `ALLOWED` table at [`:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L66), and the absence of a `lib` target asserted by [`microvms-cli/tests/dependency_direction.rs:126 the_cli_exports_no_library_target_at_all`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/dependency_direction.rs#L126). Both are manifest-shaped invariants a dependency addition trips.
* **The Node binding’s typed surface has no drift gate.** `microvms-js/index.d.ts` is gitignored (`.gitignore:29`) and untracked, and the `bindings` CI job builds the addon and runs `node --test` ([`.github/workflows/ci.yml:341`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L341)) without comparing the generated declarations against the crate. The Python side is gated — [`.github/workflows/ci.yml:319`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L319) runs `./scripts/generate-py-stubs.py --check`, and [`:314`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/generate-py-stubs.py#L314)-[`:317`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/generate-py-stubs.py#L317) records why a stale stub is worse than a stale schema: `py.typed` ships in the wheel beside it, so a stale stub leaves a caller confidently wrong rather than unchecked. The failure mode on the Node side is the same class with no detector: a renamed or removed method changes `index.d.ts` on the next build, and nothing fails until a TypeScript consumer’s call reaches a method the addon no longer exports.
## See also
[Section titled “See also”](#see-also)
* [contract map](contract-map.md) — 40 shared source citations
* [business logic](business-logic.md) — 23 shared source citations
* [public api](../../reference/public-api.md) — 21 shared source citations
* [debugging guide](debugging-guide.md) — 18 shared source citations
* [system overview](../architecture/system-overview.md) — 16 shared source citations
# Tech debt
**The marker pass returns zero rows, and that is a fact about the convention rather than about the debt.** A case-sensitive grep for `\bTODO\b`, `\bFIXME\b`, `\bHACK\b`, and `\bXXX\b` across every tracked `.rs` and `.py` file produces no output. Widened to every tracked file type, the only hits are two documents describing the absence. Case-insensitive over `.rs` and `.py`, the single hit is the byte literal `b"xxxxxxxx"` in a stream test ([`agentd/src/exec.rs:3234`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L3234)). The repo states the convention that replaced markers: comments record constraints and defects defended against, never narration. So the substitute for a `TODO` here is a doc-comment heading at the debt site that names the decision, its cost, and what would flip it — 345 such headings, carried by 77 of the 90 tracked `.rs` files. Reading “no markers” as “no debt” would invert the finding.
**Methodology.** Marker-grepping being empty, this register was assembled from six passes, each executed against source this session: (1) the marker grep above, kept as a negative result; (2) declined and open scope in `docs/STRATEGY.md`; (3) the twelve compounded lessons in `.erpaval/solutions/`, read for residual risk rather than for history; (4) version pins across `Cargo.toml`, `Cargo.lock`, `deny.toml`, `mise.toml`, `microvms-js/package.json`, `microvms-py/pyproject.toml`, and `.github/workflows/`, on the principle that every pin is a deferred upgrade with an owner; (5) gate-coverage differencing — what `mise run check` runs against what CI runs, and what each omits; (6) prose-versus-code differencing, checking whether each acceptance paragraph still describes the code beneath it. Passes 5 and 6 produced the findings the earlier passes did not.
**What this register is not about.** `mise run check` — lint, security, all six Rust tiers, `schema:check`, `stubs:check`, `model:check`, `live:check`, and the `aarch64-musl` cross-build ([`mise.toml:290-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L290-L301)) — is green as of 2026-08-25. Nothing below is a failing build or a broken test. This is structural and deferred-maintenance debt: gates that exist in one place and not another, pins with a manual tracking obligation, one number written down in several places, and rationale paragraphs that have outlived the code they explain.
**Deliberate is distinguished from accidental throughout.** Rows marked *deliberate* are documented refusals with a stated cost — [`docs/STRATEGY.md:117-136`](/microvms-agentd/internals/strategy/) declines a turn-boundary suspend protocol, process-tree fork, AgentCore exec/PTY parity, and, in three words, “**Not an orchestrator.**” A declined item is still debt in the sense that it is capability a reader may expect and will not find, but it is not rot, and calling it rot would misprice it. The accidental rows are the ones where nobody chose the current state.
Ranking is `cost-to-fix × consequence-of-leaving`, so a row rises when the remedy is real work *and* leaving it costs something continuously. Cheap-and-consequential rows rank above expensive-and-bounded ones. Category vocabulary is closed — `marker`, `wrong abstraction`, `error handling`, `dead code adjacent`, `deprecated pattern`, `version pin`, `duplicated logic`, `missing tests` — and no `marker` row appears, because there are no markers.
## Ranked register
[Section titled “Ranked register”](#ranked-register)
| Rank | Debt item | Category | Cost to fix | Citation |
| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | `spec:core`, the 51-requirement Z3 tier with a five-variable state model and an unbounded-reachability check, runs as `node ~/workplace/symspec/packages/symspec/dist/cli.mjs` — an absolute path into one developer’s home directory. It is in neither `check`’s `depends` nor CI. The strongest claim the project makes about itself is verified on one machine, last recorded green 2026-08-08. The remedy lives outside this repo: symspec v5 is published nowhere, and the 0.1.0 package on npm cannot parse `core.symspec.json` at all. | missing tests | L | [`mise.toml:209-227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L209-L227), [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301), [`.github/workflows/ci.yml:377-385`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L377-L385) |
| 2 | The `model/` crate’s verification engine is `stateright = "0.31"`, resolving to 0.31.0 — pre-1.0 by its own book, with no upstream release in roughly thirteen months. Not deprecated and no successor exists, so this is dependency liveness rather than a defect: a load-bearing proof tier sits on an unmaintained-looking crate with no migration target. Cost is L because the only real remedies are vendoring or replacing the engine. | version pin | L | [`model/Cargo.toml:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L10) |
| 3 | The Node binding has no lockfile of any kind — no `package-lock.json`, `pnpm-lock.yaml`, or `yarn.lock` exists anywhere in the tree. Its one devDependency is a caret range, and CI does not read that manifest at all: it runs `npx -y -p @napi-rs/cli@3`, a floating major with auto-install, inside the job that builds a shipped artifact. The Rust side commits `Cargo.lock` and the repo treats a committed lockfile as a supply-chain control. Dependabot watches npm weekly, but a caret range with no lockfile leaves it nothing to pin below 4.0.0. | version pin | M | [`microvms-js/package.json:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L18), [`.github/workflows/ci.yml:343`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L343), [`.github/dependabot.yml:21-23`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/dependabot.yml#L21-L23) |
| 4 | `cargo test --all` runs zero tests over the two binding crates. Measured this session: `cargo test -p microvms-py -p microvms-js` builds three test binaries and reports `running 0 tests` for each, against 7,616 lines of `src/`. The workflow states it — “`cargo test --workspace` reports 0 tests for these crates” — and the coverage is real but CI-only, under pytest and `node:test`. So the local definition of done compiles the bindings and asserts nothing about them. | missing tests | M | [`.github/workflows/ci.yml:299-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L299-L301), [`mise.toml:146-148`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L146-L148), [`.github/workflows/ci.yml:302-344`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L302-L344) |
| 5 | The committed `Cargo.lock` at `5e6f752` pins `h2` 0.4.15, which carries a RUSTSEC advisory patched in 0.4.16. `cargo deny check` runs in `mise run security` and in CI, so the gate is red at HEAD; it is green here only because of an uncommitted `cargo update -p h2` → 0.4.19 in the working tree. That update also moved `windows-sys` 0.61.2 → 0.52.0 in three places, a transitive downgrade nobody asked for — evidence that a targeted lockfile bump is not a targeted change. | version pin | S | [`Cargo.lock:1576-1577`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/Cargo.lock#L1576-L1577), [`mise.toml:115`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L115), [`.github/workflows/ci.yml:173-177`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L173-L177) |
| 6 | A stale rationale is holding a test off a working code path. `exit_codes.rs` states in the present tense that core’s `aws-config` is pinned `default-features = false`, that `ControlPlane::new` therefore panics with “a http\_client is required”, that the classified `ERR_CREDENTIALS` path is “currently unreachable”, and closes “Restore the credential version once core’s manifest is fixed.” The manifest is fixed: `default-https-client` is in the feature list and the manifest comment records the repair, and the regression test exists. So a security-adjacent exit row has unit classification coverage and no process-level coverage, for a reason that no longer holds. | missing tests | S | [`microvms-cli/tests/exit_codes.rs:138-149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L138-L149), [`microvms-cli/tests/exit_codes.rs:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L18), [`microvms-core/Cargo.toml:52-64`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L52-L64), [`microvms-core/src/control/transport.rs:906`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/transport.rs#L906) |
| 7 | *Deliberate.* There is no `Sandbox::attach`, so `suspend`, `resume`, and `terminate` address the control plane directly rather than through the type `run` and `build` use. Refused on a stated ground: an attach constructor would manufacture a second initial state that neither `spec/core.symspec.json`’s state model nor `model/src/client.rs`’s `init_states` enumerates, so both proof suites would silently stop covering it. The residual cost is that the attached lifecycle path lies outside both verification surfaces, and the module lists in full what it gives up. `microvm attach` (#66) does not reopen this: it is a CLI-level registry write that adopts a `NameRecord`, and the SDK still has no attach constructor. | wrong abstraction | L | [`microvms-cli/src/commands/lifecycle.rs:15-54`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L15-L54), `model/src/client.rs`, `spec/core.symspec.json` |
| 8 | *Deliberate.* Every platform claim rests on one architecture and one region. `aarch64-unknown-linux-musl` is the shipping target because Lambda MicroVMs are ARM64-only, `agentd`’s tiers run on ubuntu only because the guest is a Unix process model, and the strategy memo says coverage “is still one region” with the adoption-measurement half of its publish action open. Correct given the substrate; the cost is that a platform-behaviour change outside us-east-1 is invisible to every tier. | missing tests | L | [`rust-toolchain.toml:10-12`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/rust-toolchain.toml#L10-L12), [`.github/workflows/ci.yml:56-62`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L56-L62), [`docs/STRATEGY.md:106-115`](/microvms-agentd/internals/strategy/) |
| 9 | `mise run check` has no vulnerability tier. Its `depends` list is `lint, security, test, schema:check, stubs:check, model:check, live:check, build`; `vuln` — grype, trivy, osv-scanner — is absent, and those three are the scanners `deny.toml` names as the owners of advisory scanning. So the only local CVE coverage is whatever `cargo deny` incidentally provides, and the delegated lane runs in CI or on a manual invocation. | missing tests | S | [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301), [`mise.toml:131-144`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L131-L144), [`deny.toml:50-55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/deny.toml#L50-L55) |
| 10 | *Deliberate.* Every identity-repair failure in the daemon is logged and then ignored, because the daemon is the only channel into the VM and refusing to serve would strand a VM with work in it. The module lists five things it cannot do, including a `boot_id` bind mount that needs `CAP_SYS_ADMIN` and is refused outright in a container that did not ask for it. The accepted consequence is named in the strategy memo: an unrepaired identity produces “VM-generated keys that repeat across sandboxes, which is a security bug rather than a performance regression.” | error handling | M | [`agentd/src/identity.rs:27-49`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L27-L49), [`agentd/src/identity.rs:51-55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L51-L55), [`docs/STRATEGY.md:86-94`](/microvms-agentd/internals/strategy/) |
| 11 | `mise.toml` floats twelve tool versions to `latest` — `uv`, `ruff`, `semgrep`, `betterleaks`, `syft`, `grype`, `trivy`, `osv-scanner`, `terraform`, `lefthook`, `cargo-deny`, `actionlint` — while CI installs the same scanners at an exact version verified by `sha256sum -c -`. The local gate and the CI gate share a name and run different binaries, so a finding can appear in one and not the other, and neither result reproduces later. `rust = "stable"` floats too, with its reason recorded. | version pin | M | [`mise.toml:20-33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L20-L33), [`.github/workflows/ci.yml:150-151`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L150-L151), [`.github/workflows/ci.yml:201-206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L201-L206), [`.github/workflows/ci.yml:227-228`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L227-L228) |
| 12 | `deny.toml`’s `[advisories]` block asserts a delegation its own config version cannot express: “Advisory scanning is grype/trivy/osv-scanner’s lane … Running RustSec here too would give findings a fourth place to be silenced.” Under `version = 2` cargo-deny has no key that turns vulnerability scanning off — the v1 `vulnerability = "allow"` field was removed. Verified this session with cargo-deny 0.20.2: `cargo deny check advisories` reports `advisories ok` and exits 0, so the check is live. That mismatch is why row 5’s advisory surfaced as a `cargo deny` failure rather than as a scanner finding. | deprecated pattern | S | [`deny.toml:50-57`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/deny.toml#L50-L57), [`mise.toml:115`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L115) |
| 13 | maturin’s version is pinned at three sites that must move together, and the next release breaks the gate. The stub generator pins `maturin@1.14.1` exactly, CI installs `uvx maturin@1.14` twice (a minor pin that floats the patch), and `pyproject.toml` names the exact patch a fourth time in prose. maturin 1.15.0 moved `generate-stubs` output into the module’s package directory, so the pin is load-bearing: bumping it breaks `mise run stubs:check`. The pin is correct; the debt is that it is a manual tracking obligation with no alarm, and neither maturin nor `pyo3-stub-gen` offers a check mode. | version pin | S | [`scripts/generate-py-stubs.py:84-89`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/scripts/generate-py-stubs.py#L84-L89), [`.github/workflows/ci.yml:306`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L306), [`.github/workflows/ci.yml:337`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L337), [`microvms-py/pyproject.toml:36-42`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-py/pyproject.toml#L36-L42) |
| 14 | The two bindings are enforced asymmetrically over one core. `microvms-py/microvms.pyi` and `py.typed` are committed and gated by `stubs:check` inside `check`, and a typed consumer is checked against the built wheel under `ty@0.0.72`. `microvms-js/index.d.ts` is gitignored, no `tsconfig.json` exists anywhere in the tree, and no gate reads the declarations — yet `package.json` still advertises `"types": "index.d.ts"` to downstream consumers. Not a staleness row, since `napi build` regenerates the file every run; what is unverified is whether the generated types are *usable*. The fix is one typed consumer, not a diff gate. | missing tests | S | `.gitignore:23-29`, [`microvms-js/package.json:7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L7), [`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195), [`.github/workflows/ci.yml:335-340`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L335-L340) |
| 15 | The daemon’s default port, 9000, is written down three times and no assertion connects them. `microvms-core` declares `DEFAULT_AGENT_PORT` twice — once in `control/`, once in `session/` — each documented as matching the daemon, and `agentd` writes the literal itself in `Config::default`. The two tests that touch these constants each compare a value to its *own* module’s constant, so both stay green if either moves alone. This is the exact shape the repo’s own lesson names (“a comment explaining a number in terms of a value owned elsewhere”), and the repo has already invented the remedy: a compile-time `const` block that makes disagreement a build error. | duplicated logic | S | [`microvms-core/src/control/mod.rs:96-97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L96-L97), [`microvms-core/src/session/proxy.rs:82-83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L82-L83), [`agentd/src/config.rs:84`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L84), [`microvms-cli/src/commands/attached.rs:1148-1160`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1148-L1160) |
| 16 | The two bindings’ error contracts diverge, and nothing checks that the divergence stays where it is. napi types the async path over its own closed `Status` enum, so a custom `ERR_*` code survives a synchronous return and collapses to `GenericFailure` through a Promise rejection — measured with a probe addon. Nearly every method is async, so the Node rule is `err.cause.message` while the Python binding’s `.code` is reliable everywhere. Upstream-forced and thoroughly documented; the debt is that each suite asserts only its own contract, so no gate would catch the two surfaces drifting apart. | duplicated logic | M | [`microvms-js/src/errors.rs:10-45`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L10-L45), [`.erpaval/solutions/api-patterns/napi-async-collapses-error-codes.md:11-17`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/api-patterns/napi-async-collapses-error-codes.md#L11-L17), `microvms-py/tests/test_errors.py`, `microvms-js/__test__/errors.mjs` |
| 17 | The guard over the CLI’s dependency allowlist asserts `reason.len() > 25` and nothing else about any reason string, under a comment claiming “a new one cannot be added silently.” Length is not meaning: the guard catches an empty justification and cannot catch one that has stopped being true. That is the enforcement ceiling for every prose-accepted debt in this register, and this repo has already shipped one reason that went stale under it. | missing tests | S | [`microvms-cli/tests/thinness.rs:212-218`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L212-L218), [`microvms-cli/tests/thinness.rs:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L53), [`microvms-cli/tests/thinness.rs:66`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L66) |
| 18 | *Deliberate.* `microvm logs` names an image’s build log group and prints the working `aws logs tail` invocation that reads it (a success since the 0.6.0 ruling on #79; AWS CLI v2 only), still refusing to read CloudWatch itself. The reader was refused on three grounds, the decisive one being that the transport is single-service by construction — a `const` signing name and one `endpoint_for(region)` — so a CloudWatch reader in core would give the CLI a second path to AWS, which is what the thinness guard exists to forbid. The read grant ships as the Terraform stack’s `logs_read_policy_arn`. | wrong abstraction | M | [`microvms-cli/src/commands/local.rs:228`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L228) (the handler), `microvms-cli/src/commands/local.rs` `logs_succeeds_with_the_tail_command_and_refuses_to_imply_it_is_empty` (the pin) |
| 19 | Action pinning is inconsistent inside one file. Two actions are pinned to a commit SHA with the version in a trailing comment; every other `uses:` across both workflows is a mutable tag, including `aquasecurity/trivy-action@v0.36.0` inside the job whose entire output is supply-chain assurance. The same job hash-verifies four downloaded binaries, so the discipline exists and stops at the action boundary. | version pin | S | [`.github/workflows/ci.yml:174`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L174), [`.github/workflows/ci.yml:181`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L181), [`.github/workflows/ci.yml:216`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L216) |
| 20 | Three rationale blocks describe code that is no longer there. The action-version header names `checkout@v5`, `upload-artifact@v6`, and `setup-node@v5` and argues at length that “checkout is on v5 rather than v7 … v5 is the smallest version that satisfies the actual requirement” — the file uses `@v7` for all three, so the paragraph argues against the line below it. A CI comment counts “the CLI’s five test targets” where `microvms-cli/tests/` holds four. The live workflow says the suite “reports the 34 checks it cannot express as SKIP”, which the suite’s own source contradicts by marking that list permanently empty. | dead code adjacent | S | [`.github/workflows/ci.yml:29-39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L29-L39), [`.github/workflows/ci.yml:94`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L94), [`.github/workflows/live-conformance.yml:94`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/live-conformance.yml#L94), [`conformance/run_rs.py:428-432`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L428-L432) |
| 21 | The shipping binary is linked two different ways and only one of them is ever proven per environment. `.cargo/config.toml` selects `linker = "rust-lld"` for the aarch64-musl target, with a comment explaining that the alternative fails naming a missing `cc`. CI installs `gcc-aarch64-linux-gnu` and overrides the choice through `CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_LINKER`, and the environment variable wins. So the cross-build gate never exercises the configuration the repo ships, and the local build never exercises CI’s. | missing tests | S | [`.cargo/config.toml:1-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.cargo/config.toml#L1-L10), [`.github/workflows/ci.yml:358-365`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L358-L365) |
| 22 | *Deliberate.* The conformance suite keeps `Results.skipped` and its `skip()` primitive with no live caller — the list is documented as permanently empty and the primitive is exercised only by the offline self-test probe. Retained on the argument that a suite which removed its own ability to report a gap is a suite whose next gap is silent. Real dead code with a stated reason to stay. | dead code adjacent | S | [`conformance/run_rs.py:424-437`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L424-L437), [`conformance/run_rs.py:497-511`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L497-L511), [`conformance/run_rs.py:2159`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/conformance/run_rs.py#L2159) |
## Explicit markers
[Section titled “Explicit markers”](#explicit-markers)
There are no explicit markers in this tree. This section records the absence, the command that establishes it, and the convention that occupies the space a marker would.
The grep, verbatim:
```plaintext
1
grep -rnE '\b(TODO|FIXME|HACK|XXX)\b' --include='*.rs' --include='*.py' .
```
Zero lines. Widened to every tracked file type — excluding `.git`, `target`, `.codegraph`, `node_modules`, and `.venv` — the only two hits are prose *about* the absence. A case-insensitive sweep over `.rs` and `.py` returns exactly one line, and it is not a marker:
* `shared.publish(StreamKind::Stdout, b"xxxxxxxx").await;` — [`agentd/src/exec.rs:3234`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/exec.rs#L3234)
In place of markers, debt is accepted at the site under a doc-comment heading that names the decision. There are 345 such headings, carried by 77 of the 90 tracked `.rs` files. Quoted verbatim, the ones whose subject is a refusal, a limit, or an accepted cost:
* ``//! # There is no `Sandbox::attach`, and adding one would cost the proofs`` — [`microvms-cli/src/commands/lifecycle.rs:15`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L15)
* `//! # What this module cannot do, honestly` — [`agentd/src/identity.rs:27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L27)
* `//! # Why a failure is never fatal` — [`agentd/src/identity.rs:51`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/identity.rs#L51)
* `/// # The command choice is constrained by a core defect, and that is recorded rather than hidden` — [`microvms-cli/tests/exit_codes.rs:138`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L138)
* `/// # Adding a reader to core was assessed and refused, and not on grounds of size` — [`microvms-cli/src/commands/local.rs:92`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L92)
* `/// # Why this fails rather than returning an empty list` — [`microvms-cli/src/commands/local.rs:85`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/local.rs#L85)
* `//! # A dollar string's trailing zeros may differ from the Python oracle's, and that is accepted` — [`microvms-cli/src/render.rs:27`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/render.rs#L27)
* `//! # Runtime-checked rather than typestate, deliberately` — [`microvms-core/src/sandbox.rs:19`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/sandbox.rs#L19)
* `//! # Where the code lands, and why it is in two places rather than one` — [`microvms-js/src/errors.rs:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/src/errors.rs#L10)
* `/// # Output arrives as text plus a byte count, not as base64, and that is a deliberate limit` — [`microvms-cli/src/commands/attached.rs:331`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L331)
* `//! # TRAP-11: what is deliberately absent` — [`microvms-core/src/control/connector.rs:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/connector.rs#L16)
* `/// # What is deliberately not here` — [`microvms-core/src/control/mod.rs:261`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L261)
* `//! # What is deliberately *not* an option` — [`microvms-cli/src/cli.rs:21`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L21)
* `//! # Hand-rolled rather than a dependency` — [`microvms-core/src/session/sse.rs:16`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/sse.rs#L16)
Suppression files follow the same rule, stated in one of them:
* `# Accepted findings, each with its reason. An ignore without a reason is a` / `# finding someone silenced; an ignore with one is a decision someone made.` — [`.trivyignore.yaml:2-3`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.trivyignore.yaml#L2-L3)
* `reason = "optional rust_decimal feature, never enabled; not compiled into any artifact"` — [`osv-scanner.toml:13`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/osv-scanner.toml#L13)
Declined scope is written the same way, at document level rather than at a call site:
* `**Not an orchestrator.**` — [`docs/STRATEGY.md:136`](/microvms-agentd/internals/strategy/)
* `**Not fork.**` — [`docs/STRATEGY.md:127`](/microvms-agentd/internals/strategy/)
* `**Not a turn-boundary suspend protocol.**` — [`docs/STRATEGY.md:119`](/microvms-agentd/internals/strategy/)
* `**Not AgentCore parity on exec and PTY.**` — [`docs/STRATEGY.md:133`](/microvms-agentd/internals/strategy/)
* `> **Implemented. This document is kept as history because the reasoning still` / `> applies; the numbers below are outdated.**` — [`docs/CLI-COVERAGE-PLAN.md:3-4`](/microvms-agentd/internals/cli-coverage-plan/)
One consequence of the convention is worth stating plainly: an acceptance paragraph is searchable only if you know the phrasing, whereas `TODO` is one token. Fourteen `#[allow(` attributes exist in the tracked Rust tree and every one carries a `reason =` string arguing its specific case, which is the convention working. Row 20 above is the convention failing, and nothing distinguishes the two states without reading both the prose and the code.
## Pattern-level smells
[Section titled “Pattern-level smells”](#pattern-level-smells)
### The enforcement stops exactly at the crate boundary the project most wants adopted
[Section titled “The enforcement stops exactly at the crate boundary the project most wants adopted”](#the-enforcement-stops-exactly-at-the-crate-boundary-the-project-most-wants-adopted)
Every asymmetry in this register lands on the same two crates. The bindings are the surface the strategy memo exists to make adoptable, and they are the least-gated code in the workspace. `cargo test --all` runs zero tests against 7,616 lines of them. The Node side has no lockfile, advertises type declarations no gate reads, and gets its build toolchain from a floating major installed by `npx -y`. Neither binding’s manifest is read by any test, so a dependency either could re-add — the CLI has a named guard against exactly this — would fail nothing. The Python side is the counter-example and shows the shape of the fix: a committed stub, a `py.typed` marker, a regenerate-and-diff gate inside `check`, and a typed consumer checked from outside the repo’s own layout against a built wheel. That is four mechanisms on one binding and roughly zero on its twin, over one shared core.
Shows up in:
* [`.github/workflows/ci.yml:299-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L299-L301) — the workspace test runner reporting zero tests for both crates, stated in the file
* [`microvms-js/package.json:18`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L18) and [`.github/workflows/ci.yml:343`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L343) — a caret range with no lockfile, and CI bypassing the manifest entirely
* `.gitignore:23-29` and [`microvms-js/package.json:7`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-js/package.json#L7) — declarations advertised to consumers, gitignored, and read by nothing
* [`mise.toml:179-195`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L179-L195) and [`.github/workflows/ci.yml:335-340`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L335-L340) — the four mechanisms the Python side has
* [`microvms-cli/tests/thinness.rs:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L53) — a manifest guard that exists for the CLI and has no binding equivalent
Cost: M. The cheapest single item is one `tsconfig.json` and one typed `.mts` consumer, which closes row 14. A committed lockfile for `microvms-js` plus dropping `-y -p @napi-rs/cli@3` in favour of the manifest closes row 3 and is nearly as cheap.
### Debt is accepted in prose, and prose is the only thing enforcing it
[Section titled “Debt is accepted in prose, and prose is the only thing enforcing it”](#debt-is-accepted-in-prose-and-prose-is-the-only-thing-enforcing-it)
The acceptance-paragraph convention is genuinely better than a `TODO`: each paragraph names the cost, what would have to change for the answer to flip, and often where the counter-argument lives. What almost none of them have is a test that goes red when the acceptance stops being true. The measurable ceiling is one assertion: the dependency-allowlist guard checks `reason.len() > 25` and nothing else, under a comment claiming a new entry “cannot be added silently.” Length is not meaning. Three sites in this register prove the mechanism fails in practice — an action-version paragraph arguing against the line below it, a test comment deferring work on a defect that has been repaired, and a workflow describing SKIP behaviour the suite’s own source marks permanently empty. The pattern is not that the reasoning is wrong. It is that prose degrades silently while code does not, and this repo has already built the fix elsewhere: a compile-time `const` block, an exhaustive-match round-trip, a named `RETIRED` array whose failure message points at the replacement API.
Shows up in:
* [`microvms-cli/tests/thinness.rs:212-218`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L212-L218) — `reason.len() > 25`, the only assertion over any acceptance reason
* [`microvms-cli/tests/exit_codes.rs:138-149`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/exit_codes.rs#L138-L149) against [`microvms-core/Cargo.toml:52-64`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/Cargo.toml#L52-L64) — a rationale outliving the defect it describes, with a test still deferred on it
* [`.github/workflows/ci.yml:29-39`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L29-L39) against [`.github/workflows/ci.yml:69`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L69) — thirty lines arguing for a version the next line does not use
* [`microvms-cli/src/commands/attached.rs:1148-1160`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1148-L1160) — the remedy the repo already invented: a `const` block, so disagreement is a build error rather than a test failure
* [`microvms-cli/tests/thinness.rs:53`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/tests/thinness.rs#L53) — the other remedy: a named retirement whose failure message carries the replacement
Cost: M per site, and the work is adding one assertion beside an existing paragraph rather than changing behaviour.
### The local gate and the CI gate share one name and are two different gates
[Section titled “The local gate and the CI gate share one name and are two different gates”](#the-local-gate-and-the-ci-gate-share-one-name-and-are-two-different-gates)
`mise run check` is documented as the definition of done, and it is not a subset or a superset of CI — the two overlap partially and each holds ground the other does not. `check` omits the vulnerability tier entirely, so grype, trivy, and osv-scanner never run locally even though `deny.toml` names them as the owners of advisory scanning. `check` also omits the bindings, whose only tests live in a CI job. Going the other way, `check` runs `live:check`, `model:check`, and `stubs:check`. Where both do run the same gate, they run different binaries: `mise` resolves twelve tools including every scanner to `latest`, while CI installs the same scanners at exact versions verified against a SHA-256. Even the shipping binary is linked differently — the repo selects `rust-lld` and CI overrides it with `aarch64-linux-gnu-gcc` through an environment variable that takes precedence. A contributor who runs `check` green and a reviewer who reads a green CI badge are looking at two different claims.
Shows up in:
* [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301) — `check`’s `depends`, with no `vuln` and no binding suite
* [`mise.toml:20-33`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L20-L33) against [`.github/workflows/ci.yml:201-206`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L201-L206) — `latest` locally, exact version plus hash in CI, for the same scanners
* [`.cargo/config.toml:8-10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.cargo/config.toml#L8-L10) against [`.github/workflows/ci.yml:358-365`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L358-L365) — two linkers for one artifact, the env var winning
* [`deny.toml:50-55`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/deny.toml#L50-L55) — advisory scanning delegated to three scanners that `check` does not run
* [`.github/workflows/ci.yml:302-344`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L302-L344) — the binding tier that exists only here
Cost: M. Adding `vuln` to `check` is one line and would make the local gate slower and honest; reconciling the linker choice is a comment or a removed environment variable. Reconciling the tool versions is the expensive half, because pinning `mise` tools reintroduces the manual refresh obligation row 13 already carries.
### One number, written down in several places, guarded in one of them
[Section titled “One number, written down in several places, guarded in one of them”](#one-number-written-down-in-several-places-guarded-in-one-of-them)
The repo has diagnosed this class itself and written down the detection method: a comment explaining a number in terms of a value owned elsewhere is the signature, and the greps are “twice the”, “four times”, “matching the”, “same as the daemon’s”. Two pairs found that way got guards — the Dockerfile `AGENTD_PORT` against the client’s port, and the SSE keepalive interval against the client’s stream idle timeout, the latter refusing equality as well as excess. Running the same grep now still returns hits. The clearest is the default port itself: `microvms-core` declares `DEFAULT_AGENT_PORT = 9000` in two separate modules, each doc’d as matching the daemon, and `agentd` writes the literal a third time in `Config::default`. The two tests that mention these constants each compare a value to its own module’s constant, so both stay green if either moves alone. Nothing anywhere compares core’s number to the daemon’s.
Shows up in:
* [`microvms-core/src/control/mod.rs:96-97`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/mod.rs#L96-L97) and [`microvms-core/src/session/proxy.rs:82-83`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/session/proxy.rs#L82-L83) — the same constant declared twice in one crate
* [`agentd/src/config.rs:84`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/agentd/src/config.rs#L84) — the third copy, as a bare literal in the type that owns the truth
* [`microvms-core/src/control/artifact.rs:395-430`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/artifact.rs#L395-L430) — the pair that *was* guarded, with equality refused and the failure naming both numbers
* [`microvms-cli/src/commands/attached.rs:1148-1160`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L1148-L1160) — the strongest available remedy, a compile-time bound rather than a runtime assertion
* [`.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md:39-48`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.erpaval/solutions/architecture-patterns/an-absent-value-is-not-a-neutral-one.md#L39-L48) — the repo’s own detection method, which found the last two and has not been re-run
Cost: S. One `const` block asserting the two core constants against each other, and one test comparing core’s default to `agentd`’s `Config::default()`.
### The strongest verification tiers are the least reachable, and the newest one is unmaintained upstream
[Section titled “The strongest verification tiers are the least reachable, and the newest one is unmaintained upstream”](#the-strongest-verification-tiers-are-the-least-reachable-and-the-newest-one-is-unmaintained-upstream)
This project verifies itself unusually hard: a stateright model over every reachable state, proptest confinement properties, turmoil fault simulation, a panic guard, a schema-artifact check, and two Z3-backed symspec requirement documents. Reachability is inversely correlated with strength. The tiers inside `cargo test` run everywhere. The six-requirement daemon spec needs a global npm install plus a downloaded embedding model and sits outside `check`. The 51-requirement core spec — the only one with a state model and an unbounded-reachability tier — runs from an absolute path inside one developer’s home directory, is in neither `check` nor CI, and its CI job was deleted rather than pointed at the registry package, on the sound argument that a green job labelled “requirements” verifying six of fifty-seven is worse than no job. Underneath the tier that *is* always reachable, the engine is pre-1.0 with no upstream release in thirteen months. Both exclusions are argued well and both leave the same hole: the claims this project most wants believed are checked by one person on one machine, or by a crate with no maintenance signal.
Shows up in:
* [`mise.toml:209-227`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L209-L227) — the absolute home-directory path, the cooperative-cancellation timeout, and the “verified 2026-08-08” note
* [`mise.toml:197-207`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L197-L207) — the 0.1.0 task, scoped to six requirements, also outside `check`
* [`mise.toml:292-301`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/mise.toml#L292-L301) — `check`’s `depends`, with neither `spec` nor `spec:core`
* [`.github/workflows/ci.yml:377-385`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/.github/workflows/ci.yml#L377-L385) — the deleted job and the stated condition for its return
* [`model/Cargo.toml:10`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/model/Cargo.toml#L10) — the engine, pinned at a pre-1.0 version with no successor
Cost: L. The remedy for the spec tier lives outside this repo — publishing symspec v5 to a registry, or vendoring its `dist/cli.mjs` — and the owner’s stated preference is to publish. The remedy for the engine is to keep watching it, since vendoring or replacing a model checker is a larger project than the tier it serves.
## See also
[Section titled “See also”](#see-also)
* [impact analysis](impact-analysis.md) — 16 shared source citations
* [contract map](contract-map.md) — 13 shared source citations
* [business logic](business-logic.md) — 11 shared source citations
* [debugging guide](debugging-guide.md) — 11 shared source citations
* [processes](../behavior/processes.md) — 10 shared source citations
# Networking
**No internet egress requires a VPC without an internet gateway (IGW) or NAT gateway.** Attach that VPC through a custom network connector. Audit the selected subnet routes for other internet paths, including IPv6 gateways, transit networks, peering, and proxies. Private services and VPC endpoints remain reachable if your VPC configuration permits them.
Omitting the managed `INTERNET_EGRESS` connector does not disable internet access on the default MicroVM network. `--deny-egress` only sets proxy variables; a workload can ignore them. Neither is an isolation boundary.
## Create a VPC connector
[Section titled “Create a VPC connector”](#create-a-vpc-connector)
The [Lambda core API](https://docs.aws.amazon.com/lambda/latest/lambda-core/Welcome.html) manages connectors separately from the `lambda-microvms` API. Use a current boto3 release to create and manage connectors. This package accepts existing connector ARNs at launch; it does not manage the connectors’ lifecycle.
Provision the VPC, subnets, security groups, and a Lambda connector operator role first. The operator role needs permissions to manage the connector’s network interfaces; it is separate from the VM execution role. The following example creates a real AWS resource:
```python
1
import os
2
import time
3
import uuid
4
5
import boto3
6
7
core = boto3.client("lambda-core", region_name="us-east-1")
8
connector = core.create_network_connector(
9
Name="isolated-microvms",
10
ClientToken=str(uuid.uuid4()),
11
OperatorRole=os.environ["CONNECTOR_OPERATOR_ROLE_ARN"],
12
Configuration={
13
"VpcEgressConfiguration": {
14
"SubnetIds": [os.environ["ISOLATED_SUBNET_ID"]],
15
"SecurityGroupIds": [os.environ["CONNECTOR_SECURITY_GROUP_ID"]],
16
"NetworkProtocol": "IPv4",
17
"AssociatedComputeResourceTypes": ["MicroVm"],
18
}
19
},
20
)
21
arn = connector["Arn"]
22
deadline = time.monotonic() + 900
23
while True:
24
status = core.get_network_connector(Identifier=arn)
25
if status.get("State") == "ACTIVE":
26
break
27
if status.get("State") != "PENDING":
28
raise RuntimeError(status.get("StateReason", status.get("State")))
29
if time.monotonic() >= deadline:
30
raise TimeoutError(f"Connector still pending: {arn}")
31
time.sleep(5)
32
print(arn)
```
Creation is asynchronous and there is no built-in boto3 waiter. Preserve the client token when retrying the same creation request. After updates, inspect `LastUpdateStatus` as well as `State`; an active connector can have a failed configuration update. Delete an unused connector explicitly with `delete_network_connector(Identifier=arn)` and confirm completion.
## Attach it at launch
[Section titled “Attach it at launch”](#attach-it-at-launch)
```bash
1
microvm run --image my-image \
2
--egress-network-connector "$CONNECTOR_ARN" \
3
--exec "echo hello"
```
Persist the list in `microvm.toml` as `egress-network-connectors = ["ARN"]`. Explicit flags replace that configured list. The flag is repeatable up to the API’s connector limit. Do not combine it with `--egress`, which selects the managed internet connector. Python launch methods accept `egress_network_connectors=[arn]`; Node launch options use `egressNetworkConnectors: [arn]`. The AWS wire field is `egressNetworkConnectors` on `RunMicrovm`.
An attached connector ARN is not proof of isolation. The package does not audit its VPC, route tables, or security groups. Verify public destinations are unreachable from the launched VM and private destinations behave as intended. Custom connectors and VPC resources are managed separately from VM cleanup.
The VM execution role remains available through guest metadata. Restrict that role even when the VPC has no internet route; see [Trust](./trust.md).
## Evidence
[Section titled “Evidence”](#evidence)
Reviewed 2026-09-16 against boto3/botocore 1.43.95, Lambda core API version `2026-04-30`, and MicroVM API version `2025-09-09`. Connector configuration and state transitions above come from AWS documentation and SDK models. The example passed boto3 parameter validation with stubbed responses; it was not live-executed during this documentation refresh. Earlier default-network measurements are retained in [Platform](./platform.md).
# AWS Lambda MicroVMs: measured platform behavior
> A compact record of runtime observations. Unless stated otherwise, measurements used us-east-1, MicroVM API 2025-09-09, and an ARM64 al2023-minimal guest. Dates apply to…
A compact record of runtime observations. Unless stated otherwise, measurements used us-east-1, MicroVM API `2025-09-09`, and an ARM64 `al2023-minimal` guest. Dates apply to observations, not guarantees about current service behavior. Original experiments and full narratives remain in git history.
For current operation shapes, see the [AWS MicroVM documentation](https://docs.aws.amazon.com/lambda/latest/microvm-api/Welcome.html). For internet isolation, use [Networking](./networking.md): a VPC without an IGW or NAT gateway is required; connector omission and proxy variables do not seal the default network. The September 2026 correction below distinguishes the separate Lambda core API from the MicroVM API.
## The service provides no exec and no file transfer
[Section titled “The service provides no exec and no file transfer”](#the-service-provides-no-exec-and-no-file-transfer)
Lambda MicroVMs have no addressable command-execution or file-transfer API; `agentd` supplies those operations. The service does offer a programmatically usable PTY through `CreateMicrovmShellAuthToken` (measured 2026-08-15), which corrects the earlier claim that its shell was console-only. A PTY does not provide detached exec IDs, separated output streams, or exit-status records.
## Hooks are served under a fixed prefix, and two of them are build-time
[Section titled “Hooks are served under a fixed prefix, and two of them are build-time”](#hooks-are-served-under-a-fixed-prefix-and-two-of-them-are-build-time)
Measured 2026-08-05. Hooks are `POST /aws/lambda-microvms/runtime/v1/` for `ready`, `validate`, `run`, `resume`, `suspend`, and `terminate`. `ready` and `validate` run during image build, before token delivery, and must succeed without bootstrap. The model allows 3600-second build-hook timeouts and 60-second runtime-hook timeouts.
## `runHookPayload` arrives wrapped, not as the body
[Section titled “runHookPayload arrives wrapped, not as the body”](#runhookpayload-arrives-wrapped-not-as-the-body)
Measured 2026-08-05. The platform sends an outer JSON object containing the string supplied to `RunMicrovm`:
```json
1
{"runHookPayload": "{\"agent_token\": \"...\"}"}
```
Decode both layers. Reading `agent_token` from the outer object fails the run hook; AWS can terminate the VM before forwarding traffic. Read `stateReason` from `GetMicrovm` when launch fails.
## The `runHookPayload` ceiling is 4096 bytes, and the service model states it twice, differently
[Section titled “The runHookPayload ceiling is 4096 bytes, and the service model states it twice, differently”](#the-runhookpayload-ceiling-is-4096-bytes-and-the-service-model-states-it-twice-differently)
Measured 2026-08-07; model rechecked 2026-09-16 with botocore 1.43.95. The inclusive limit is **4096 bytes of the serialized payload**. The model’s member documentation still says 16,384, but its referenced shape says 4096. The client checks the smaller, measured limit before calling AWS.
| Payload size | Measured response with an invalid image identifier |
| ------------ | ----------------------------------------------------- |
| 4096 bytes | Passed length validation, then rejected the image ARN |
| 4097 bytes | Rejected `runHookPayload` length |
This corrects the earlier 16 KB claim. The budget includes the token, launch environment, identity material, and JSON escaping. See the [AWS RunMicrovm documentation](https://docs.aws.amazon.com/lambda/latest/microvm-api/API_RunMicrovm.html) for current shape constraints.
## Calling an unpriced region returns `AccessDeniedException` with a null message
[Section titled “Calling an unpriced region returns AccessDeniedException with a null message”](#calling-an-unpriced-region-returns-accessdeniedexception-with-a-null-message)
Measured 2026-08-07 with `ListMicrovms`. `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1` succeeded. `eu-central-1`, `ap-southeast-2`, and `sa-east-1` returned `AccessDeniedException` with a null message. This can resemble an IAM problem; a null message alone is not proof of its cause.
The shared Lambda endpoint resolver can list ordinary Lambda regions that do not support MicroVMs. SDK endpoint availability is not a MicroVM service availability check. The package validates its known regions and permits an explicit `--unlisted-region` override.
## Network connectors are ARNs
[Section titled “Network connectors are ARNs”](#network-connectors-are-arns)
Measured 2026-08-05. Managed connectors use ARNs such as `arn:aws:lambda::aws:network-connector:aws-network-connector:ALL_INGRESS`. The bare value `ALL_INGRESS` fails with `Malformed network connector ARN`. `INTERNET_EGRESS`, `HTTP_INGRESS`, and `SHELL_INGRESS` use the same form.
**Correction, 2026-09-16:** custom VPC connectors are created by the separate `lambda-core` service and attached through `egressNetworkConnectors`. The old claim that omitting a managed connector disables egress was disproved in September. Use a VPC without an IGW or NAT gateway for internet isolation; see [Networking](./networking.md).
## `CreateMicrovmAuthToken` returns a header map
[Section titled “CreateMicrovmAuthToken returns a header map”](#createmicrovmauthtoken-returns-a-header-map)
Measured 2026-08-05. `authToken` is a header map, not a string. Read `authToken["X-aws-proxy-auth"]`, and send `X-aws-proxy-port` with the target port. Preserve the header-map contract rather than assuming the response can never contain another header.
## MicroVM states, and terminal states reached before `RUNNING`
[Section titled “MicroVM states, and terminal states reached before RUNNING”](#microvm-states-and-terminal-states-reached-before-running)
Typical progression is `PENDING → RUNNING → SUSPENDING → SUSPENDED`, with resume and termination transitions. Poll for the desired state and stop on terminal states, reporting `stateReason`. A VM that terminates before `RUNNING` failed startup; continuing to poll hides the useful error. Use the [AWS GetMicrovm documentation](https://docs.aws.amazon.com/lambda/latest/microvm-api/API_GetMicrovm.html) for the complete state enum.
## The build log group survives Terraform
[Section titled “The build log group survives Terraform”](#the-build-log-group-survives-terraform)
Measured 2026-08-05. AWS creates `/aws/lambda-microvms/` outside the Terraform stack. `terraform destroy` does not remove it. Verify log groups separately after VM/image cleanup.
## Root in the guest is not enough: `sethostname` and bind mounts need `additionalOsCapabilities`
[Section titled “Root in the guest is not enough: sethostname and bind mounts need additionalOsCapabilities”](#root-in-the-guest-is-not-enough-sethostname-and-bind-mounts-need-additionaloscapabilities)
Measured 2026-08-06 with `al2023-1`. Without `additionalOsCapabilities`, writing `/etc/machine-id` succeeded while `sethostname` and the bind mount over `/proc/sys/kernel/random/boot_id` returned `EPERM`. Requesting `["ALL"]` made all three succeed in that run; `ALL` is the model’s only capability value.
**Later evidence, 2026-09-12:** the daemon and child capability bounding sets lacked `CAP_SYS_ADMIN` and `CAP_NET_ADMIN`, including with identity repair requested. Do not treat the earlier success or `["ALL"]` as a portable privilege guarantee. Inspect `identity_degraded` on health and the current capability mask. The metadata section below records the later measurements.
## `minimumMemoryInMiB` selects a *baseline*, and the guest reports the *peak*
[Section titled “minimumMemoryInMiB selects a baseline, and the guest reports the peak”](#minimummemoryinmib-selects-a-baseline-and-the-guest-reports-the-peak)
Measured 2026-08-07 with `al2023-1`. A 512 MiB baseline produced `MemTotal: 2037648 kB`; 2048 MiB produced `8209056 kB`. These match the documented size classes:
| Baseline memory / vCPU | Provisioned ceiling memory / vCPU |
| ---------------------- | --------------------------------- |
| 0.5 GiB / 0.25 | 2 GiB / 1 |
| 1 GiB / 0.5 | 4 GiB / 2 |
| 2 GiB / 1 | 8 GiB / 4 |
| 4 GiB / 2 | 16 GiB / 8 |
| 8 GiB / 4 | 32 GiB / 16 |
The service team confirmed in August 2026 that the ceiling is provisioned at launch; there is no resize event. AWS documents billing at the requested baseline while running, plus consumption above it. This corrects the earlier inference that the reported peak was the billing floor. Memory-pressure tests must use the guest’s ceiling, not the requested baseline. Swap was absent.
## What actually costs money
[Section titled “What actually costs money”](#what-actually-costs-money)
Queried 2026-08-07 from AWS Pricing in us-east-1, `ServiceCode="AWSLambda"`. These are dated USD rates, not a current invoice:
| Usage | Rate |
| ------------------------- | ------------ |
| ARM vCPU-second | 0.0000276944 |
| ARM memory GiB-second | 0.0000036667 |
| Snapshot read GiB | 0.0015467699 |
| Snapshot write GiB | 0.0037977138 |
| Snapshot storage GiB-hour | 0.0001111111 |
The API also returned non-ARM compute rates; MicroVMs support `ARM_64` only. Rates existed in five regions. us-east-2/us-west-2 matched us-east-1; eu-west-1 and ap-northeast-1 were higher. Regional usage types have prefixes that must be removed before comparing the same dimension.
Image storage has a one-week minimum. Running idle VMs still incur baseline charges; suspended VMs incur snapshot storage and transitions incur reads and writes. Data transfer is separate. Server-side build compute billing remains unverified and is reported as unpriced, not zero. The old $0.08/GiB-month storage estimate was rounded low; the API rate gives $0.081111103 at 730 hours. Use `microvm cost` and `scripts/check-live-rates.py` rather than copying rates from this record.
## Seeing an OOM: the process case works, the VM case is still unmeasured
[Section titled “Seeing an OOM: the process case works, the VM case is still unmeasured”](#seeing-an-oom-the-process-case-works-the-vm-case-is-still-unmeasured)
Measured 2026-08-07. `dmesg` was readable and `/sys/fs/cgroup/memory.events` exposed `oom`, `oom_kill`, and `oom_group_kill`, all zero in the tested VM. No actual OOM was induced: the first probe required an absent Python interpreter; a second hit the `/dev/shm` limit instead of RAM pressure. A guest-wide OOM’s `stateReason` therefore remains unmeasured.
The daemon remained reachable while processing 64 MiB of output and reported `truncated: true`. Unit tests cover signal reporting, but do not establish what AWS reports after a guest-wide OOM.
## Suspend/resume is a freeze and restore, not a stop and start
[Section titled “Suspend/resume is a freeze and restore, not a stop and start”](#suspendresume-is-a-freeze-and-restore-not-a-stop-and-start)
Measured 2026-08-05, `al2023-1`, 1024 MiB baseline, held suspended for 45 seconds. Token, files, exec records, unread output, background process, and endpoint URL all survived. A one-second ticker had a 51-second gap, then advanced six times in six seconds after resume.
Resume continues frozen memory and processes; it does not require bootstrap again. Wall-clock leases and credentials can expire during suspension. These observations corrected the earlier claim that an in-memory token was lost.
## Traffic ordering around the `/run` hook
[Section titled “Traffic ordering around the /run hook”](#traffic-ordering-around-the-run-hook)
AWS documents that external traffic is forwarded only after `/run` returns HTTP 200. This permits launch-time secret delivery without baking secrets into a shared image. It does not protect bootstrap from a process already running inside the guest. See [Trust](./trust.md).
## The platform’s own hook arrives over loopback
[Section titled “The platform’s own hook arrives over loopback”](#the-platforms-own-hook-arrives-over-loopback)
Measured 2026-08-04. Lifecycle hooks and proxied control requests both arrived from `127.0.0.1` on ephemeral ports. A loopback-address filter cannot distinguish AWS from a guest process, and rejecting loopback rejects legitimate bootstrap. Use the one-shot bootstrap contract and prevent workloads from starting before it completes.
## Something probes the port with TLS before bootstrap
[Section titled “Something probes the port with TLS before bootstrap”](#something-probes-the-port-with-tls-before-bootstrap)
Measured 2026-08-04. TLS ClientHello bytes reached the daemon’s plaintext port before bootstrap and produced HTTP 400. The source component was not identified. Reject malformed traffic without terminating the listener.
## Endpoint authentication
[Section titled “Endpoint authentication”](#endpoint-authentication)
AWS documents proxy JWEs scoped to a VM, allowed ports, and an expiry of at most 60 minutes. Clients must mint fresh credentials for later requests. Measured 2026-08-15 against a listener on port 8080:
| `allowedPorts` | HTTPS request to 8080 |
| ----------------------------------------------- | ---------------------------- |
| `[{"port":9000}]` | 403, `Access to port denied` |
| `[{"port":9000},{"port":8080}]` | 200 |
| `[{"allPorts":{}}]` | 200 |
| `[{"range":{"startPort":8000,"endPort":9100}}]` | 200 |
These are tagged-union wire forms, with one member per item. A permitted port with no listener returned 502. WebSocket failures instead appeared as close code 1006 without a reason; use an authenticated HTTPS request to diagnose port scope versus an unavailable listener.
## `clientToken` is a permanent idempotency key
[Section titled “clientToken is a permanent idempotency key”](#clienttoken-is-a-permanent-idempotency-key)
Measured 2026-08-02. Reusing a content-derived create token after deleting an image replayed the original creation rather than scheduling new builds. Two images remained `CREATING` for roughly 15 hours, with builds stuck `PENDING` and unchanged timestamps.
Use a fresh token for a new logical build, retaining it only for retries of that attempt. Detect stalled builds with `ListMicrovmImageBuilds` after a grace period. This observation concerns MicroVM image creation; it is not a claim about every AWS service’s idempotency lifetime.
## Build logs go to `/aws/lambda-microvms/`
[Section titled “Build logs go to /aws/lambda-microvms/\”](#build-logs-go-to-awslambda-microvmsimage-name)
Measured 2026-08-05. The log prefix is `/aws/lambda-microvms/`, not `/aws/lambda/microvms/`. Build roles need CloudWatch permissions on the correct group and ECR access for private source images. Incorrect logging permissions can hide the underlying container error.
## An image build is three VMs and three log streams, and `logStream` is an exact name
[Section titled “An image build is three VMs and three log streams, and logStream is an exact name”](#an-image-build-is-three-vms-and-three-log-streams-and-logstream-is-an-exact-name)
Measured August 2026. A build used a docker-build VM and snapshot VMs for Graviton 3 and 4; application startup logs came from the snapshot VMs. Default logging used a per-image group and separate randomly named streams.
The API’s configured `logStream` is an exact name, so all build phases write to that stream. This client adds `/<16 hex>` to a configured prefix for each create attempt and returns the resolved name. User prefixes are capped at 495 characters to fit the 512-character shape; `:` and `*` are forbidden. The configured group must be writable by the build role.
## A failed build’s `stateReason` lives on the **build**, not on the version or the image
[Section titled “A failed build’s stateReason lives on the build, not on the version or the image”](#a-failed-builds-statereason-lives-on-the-build-not-on-the-version-or-the-image)
Measured 2026-08-15 across three failed builds. `GetMicrovmImage` had no reason field; version summaries returned `stateReason: null`; build summaries from `ListMicrovmImageBuilds` contained the reason. Follow `latestFailedImageVersion`, list its builds, and inspect every failed build.
Observed reasons included `The container image build failed.` and `Ready hook invocation timed out after PT5M`. CloudWatch logs provide the container-level detail; a summary reason does not replace them.
## `idlePolicy`
[Section titled “idlePolicy”](#idlepolicy)
AWS documents idleness as inbound endpoint traffic, not guest CPU activity. Set `maxIdleDurationSeconds`, `suspendedDurationSeconds`, and `autoResumeEnabled` deliberately. The suspended timeout can terminate a VM before a later manual resume; the maximum VM duration also applies.
Measured 2026-08-15: `GetMicrovm` echoed all three policy fields unchanged in `RUNNING` and `SUSPENDED`. The older claim that the suspended timeout existed only in requests was wrong. External health polls kept a VM running; guest loopback requests do not traverse the endpoint’s idle accounting.
## Pagination cursors are URL-safe base64, and the padding still has to be encoded
[Section titled “Pagination cursors are URL-safe base64, and the padding still has to be encoded”](#pagination-cursors-are-url-safe-base64-and-the-padding-still-has-to-be-encoded)
Measured 2026-08-15 over 28 cursors. Tokens were 688–800 bytes of URL-safe base64, including `=` padding in six samples. An encoded `%3D` request succeeded; the otherwise identical raw `=` request returned HTTP 400 with a null message. Treat cursors as opaque and percent-encode query values before signing. The absence of `+` or `/` does not make encoding optional.
## `maxResults` is applied before `nameFilter`, so a page can be empty while matches remain
[Section titled “maxResults is applied before nameFilter, so a page can be empty while matches remain”](#maxresults-is-applied-before-namefilter-so-a-page-can-be-empty-while-matches-remain)
Measured 2026-08-15. With 22 images, ten matching `nameFilter=bonk`, and `maxResults=1`, the first page was empty and the complete listing took 26 pages. Follow `nextToken` until absent even when a page contains no items. `nameFilter` is a substring filter; exact-name lookup must compare names across the complete listing.
## A second `CreateMicrovmImage` under an existing name is refused, so a client without `UpdateMicrovmImage` cannot make a second version
[Section titled “A second CreateMicrovmImage under an existing name is refused, so a client without UpdateMicrovmImage cannot make a second version”](#a-second-createmicrovmimage-under-an-existing-name-is-refused-so-a-client-without-updatemicrovmimage-cannot-make-a-second-version)
Measured 2026-08-15. Creating an existing image name returned HTTP 400, `ValidationException: A MicroVM image with the name '' already exists in this account`. Use `UpdateMicrovmImage` to create another version. Its PUT request requires `codeArtifact`, `baseImageArn`, and `buildRoleArn`. A client limited to create calls cannot produce a multi-version image.
## The image ARN separator is a colon, and the slash form fails as `AccessDeniedException`
[Section titled “The image ARN separator is a colon, and the slash form fails as AccessDeniedException”](#the-image-arn-separator-is-a-colon-and-the-slash-form-fails-as-accessdeniedexception)
Measured 2026-08-15. Customer image ARNs use `arn:aws:lambda:::microvm-image:`. The colon form returned 200; an encoded slash form returned 403 `AccessDeniedException`. An unencoded slash created extra path segments and returned HTML 404.
One transient gateway 502 with an HTML body preceded consistent 403 responses. Do not interpret a gateway error as service validation, or widen IAM solely because a malformed ARN produced an authorization error.
## Most public ARM64 base images have no WORKDIR
[Section titled “Most public ARM64 base images have no WORKDIR”](#most-public-arm64-base-images-have-no-workdir)
Measured 2026-08-05. The inspected `al2023-minimal`, `python:3.12-slim`, and `node:20-slim` images left `WorkingDir` empty. Set `WORKDIR` explicitly and ensure the workload user can write there.
## A WebSocket reaches a guest server through the endpoint, and the proxy strips its own subprotocols
[Section titled “A WebSocket reaches a guest server through the endpoint, and the proxy strips its own subprotocols”](#a-websocket-reaches-a-guest-server-through-the-endpoint-and-the-proxy-strips-its-own-subprotocols)
Measured in two independent runs on 2026-08-15. Both reached a guest echo server with this offered subprotocol list:
```text
1
lambda-microvms
2
lambda-microvms.authentication.
3
lambda-microvms.port.
```
One run obtained credentials through the built Node binding’s `Session.connect_subprotocols` and `connect_headers`, verifying the helper as well as the protocol. Text frames round-tripped in order. An 899-byte auth subprotocol remained token-legal without escaping.
The proxy consumed its three subprotocols; none reached the guest. A fourth application subprotocol did reach it and could be negotiated. When the guest selected none, the client still observed `lambda-microvms`, supplied by the proxy. Client-visible `ws.protocol` alone is therefore not evidence of guest negotiation. HTTPS similarly stripped the proxy auth and port headers.
Missing credentials, missing marker, and wrong-port tokens all produced opaque 1006 closes. Diagnose using the HTTPS status as described under endpoint authentication.
## Binary frames survive a port-scoped WebSocket, and an upgrade cannot be replayed over HTTPS
[Section titled “Binary frames survive a port-scoped WebSocket, and an upgrade cannot be replayed over HTTPS”](#binary-frames-survive-a-port-scoped-websocket-and-an-upgrade-cannot-be-replayed-over-https)
Measured 2026-08-29 against a guest echo server on 8090. A real port-scoped `wss://` connection returned 101 and preserved binary frames byte-for-byte, including `00 FF FE 80 7F 00` and a 300-byte extended-length frame.
Forwarding an upgrade through an ordinary HTTPS request returned 400 and never reached the guest; a tunnel must perform a real WebSocket handshake. For `/v1/tcp?port=5432`, scope the proxy token to the daemon’s listening port (default 9000): the daemon makes the onward connection inside the guest. Scoping the token to 5432 instead produced the same opaque 1006 failure.
## The guest kernel is 6.1, which `openat2` needs
[Section titled “The guest kernel is 6.1, which openat2 needs”](#the-guest-kernel-is-61-which-openat2-needs)
Measured 2026-08-14 with `al2023-1`: kernel `6.1.166-24.303.amzn2023.aarch64`. Tar extraction uses `openat2` with `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS`, available since Linux 5.6. Unsupported kernels fail extraction rather than silently weakening confinement.
## A WebSocket reaches a guest server through the endpoint, and the proxy strips its own subprotocols
[Section titled “A WebSocket reaches a guest server through the endpoint, and the proxy strips its own subprotocols”](#a-websocket-reaches-a-guest-server-through-the-endpoint-and-the-proxy-strips-its-own-subprotocols-1)
The independent 2026-08-15 run confirmed the same handshake, stripping, application negotiation, and opaque failure behavior. Its observations are consolidated in the earlier WebSocket section; this heading remains for existing links.
## An outside poll of `/v1/health` does reset the idle timer, and the control half proves it
[Section titled “An outside poll of /v1/health does reset the idle timer, and the control half proves it”](#an-outside-poll-of-v1health-does-reset-the-idle-timer-and-the-control-half-proves-it)
Measured 2026-08-15 with two VMs running detached `sleep 300`, a 60-second idle timeout, and a 900-second suspended timeout. The VM polled through its endpoint every roughly 20 seconds remained `RUNNING` through 311 seconds. The unpolled control was `SUSPENDED` at 66 seconds and remained so.
External polling keeps a busy VM alive; guest work alone does not. Poll health at an interval below the idle timeout when preserving an active exec is the caller’s intention. A local ledger watcher does not have this effect.
## The 4096-byte `runHookPayload` ceiling is on the whole string, env map included
[Section titled “The 4096-byte runHookPayload ceiling is on the whole string, env map included”](#the-4096-byte-runhookpayload-ceiling-is-on-the-whole-string-env-map-included)
Measured 2026-08-15 using a payload containing both `agent_token` and an `env` map. Exactly 4096 serialized bytes passed length validation; 4097 failed before image resolution, matching the token-only measurement. Validate the whole serialized string after combining fields.
## The shell endpoint is a real PTY over a WebSocket, and it is programmatically drivable
[Section titled “The shell endpoint is a real PTY over a WebSocket, and it is programmatically drivable”](#the-shell-endpoint-is-a-real-pty-over-a-websocket-and-it-is-programmatically-drivable)
Measured 2026-08-15. A programmable shell requires `HTTP_INGRESS` plus `SHELL_INGRESS`. `ALL_INGRESS` alone cannot mint shell credentials. Combining `ALL_INGRESS` with `SHELL_INGRESS` launched a VM but failed later at token creation, so validate that combination before launch.
`CreateMicrovmShellAuthToken` returns a proxy header map and has no `allowedPorts` parameter. Connect to the VM’s WebSocket endpoint with the marker and authentication subprotocols; no port subprotocol is required. Ordinary HTTPS with the shell token returned 502.
| Message | Meaning |
| ------------------------------------------------------------ | ------------------------------------- |
| Initial text `{"type":"session_init","session_id":""}` | Session identifier |
| Binary frames | Raw terminal input/output |
| Text `{"type":"resize","cols":120,"rows":40}` | Resize; `stty size` reported `40 120` |
| Close 1000, `shell exited` | Shell finished |
The session was a root PTY with job control; Ctrl-C produced status 130. Unknown control messages became literal shell input rather than errors. There is no structured per-command exit-status channel. This corrects the original claim that the shell could not be driven programmatically; the package now exposes it through `microvm shell`.
## Tagging works on images and not on MicroVMs, and `RunMicrovm` takes no tags
[Section titled “Tagging works on images and not on MicroVMs, and RunMicrovm takes no tags”](#tagging-works-on-images-and-not-on-microvms-and-runmicrovm-takes-no-tags)
Measured 2026-08-15. Image tags could be created, accumulated, listed, and removed. `GetMicrovmImage` echoed them. Attempts to tag a running MicroVM by ARN or bare ID failed. `RunMicrovm` has no tags field in the 2025-09-09 model, still true in the 2026-09-16 SDK refresh. Do not assume image tags provide per-instance compute attribution.
## Build introspection returns snapshot sizes and a chipset generation, not logs
[Section titled “Build introspection returns snapshot sizes and a chipset generation, not logs”](#build-introspection-returns-snapshot-sizes-and-a-chipset-generation-not-logs)
Measured 2026-08-15. `ListMicrovmImageBuilds` requires an image identifier and version and returned two builds, for Graviton generations 3 and 4. `GetMicrovmImageBuild` added `snapshotBuild` sizes: 579080192 memory bytes, 2357084160 code-install bytes, and 24297472 disk-snapshot bytes for the tested image. It did not return logs or `stateReason`; use build summaries and CloudWatch for failure details.
`GetMicrovmImageVersion` echoes build configuration, including resources, hooks, connectors, and base version. `state` (build outcome) and `status` (launch eligibility) are separate. Use snapshot dimensions for estimates and preserve absent dimensions as unknown rather than zero.
## The managed base image has two versions, and its versions are bare integers
[Section titled “The managed base image has two versions, and its versions are bare integers”](#the-managed-base-image-has-two-versions-and-its-versions-are-bare-integers)
Measured 2026-08-15. The managed-image listing returned `al2023-1`, with versions `"0"` and `"1"`. The version readback of a derived image normalized its base version to `"1.0"`. This is a dated listing, not a claim that AWS will always offer one image or two versions. Discover current versions and retain their service-provided strings rather than treating every version as an integer or a common semantic-version format.
## Every field `GetMicrovm` returns for a running VM
[Section titled “Every field GetMicrovm returns for a running VM”](#every-field-getmicrovm-returns-for-a-running-vm)
Measured 2026-08-15. A healthy response contained `microvmId`, `state`, `endpoint`, `imageArn`, `imageVersion`, `executionRoleArn`, `idlePolicy`, `maximumDurationInSeconds`, `startedAt`, `ingressNetworkConnectors`, and `egressNetworkConnectors`.
`runHookPayload` was not echoed. `stateReason` was absent on the healthy VM. Memory sizing belongs to the image version, not the instance response. The idle policy echoed `autoResumeEnabled`; the package exposes it through `--auto-resume`. See [AWS GetMicrovm](https://docs.aws.amazon.com/lambda/latest/microvm-api/API_GetMicrovm.html) for the complete current response shape, including optional fields.
## A detached exec survives the 60-minute proxy-token ceiling
[Section titled “A detached exec survives the 60-minute proxy-token ceiling”](#a-detached-exec-survives-the-60-minute-proxy-token-ceiling)
Measured 2026-08-15. A 75-minute detached exec produced all 450 expected ten-second ticks and exited zero without truncation. The tick gap across the 60-minute credential boundary was ten seconds. Fresh client processes minted new proxy credentials; the guest exec record survived independently.
The VM had a 30-minute idle window and external traffic every eight minutes. Without that traffic, idle suspension would remain possible. Polling a running exec returned its phase without partial stdout; use streaming or a file for progress. Recorded compute plus snapshot-read estimate: about $0.16.
## `INACTIVE` is a real retire: `RunMicrovm` refuses the version, pinned or not
[Section titled “INACTIVE is a real retire: RunMicrovm refuses the version, pinned or not”](#inactive-is-a-real-retire-runmicrovm-refuses-the-version-pinned-or-not)
Measured 2026-08-16. After setting the sole image version to `INACTIVE`, both pinned and unpinned launches returned 404 `No active version found`. Readback still showed `state: SUCCESSFUL`, `status: INACTIVE`. Restoring `ACTIVE` allowed launch. `UpdateMicrovmImageVersion` changes eligibility without deleting the version. Effects on already-running VMs were not measured.
## A launch with no `executionRoleArn` **succeeds**, so there is no free `RunMicrovm` probe
[Section titled “A launch with no executionRoleArn succeeds, so there is no free RunMicrovm probe”](#a-launch-with-no-executionrolearn-succeeds-so-there-is-no-free-runmicrovm-probe)
Measured 2026-08-16. Omitting `executionRoleArn` created a real VM; the field is optional in the service model. Missing a required-looking field is not a safe dry run. Invalid image identifiers can bracket earlier validation, but every probe must account for the possibility of creating a billable resource.
## `GetMicrovmImageBuild`’s `snapshotBuild` is absent on a container-build failure and partial on a hook timeout
[Section titled “GetMicrovmImageBuild’s snapshotBuild is absent on a container-build failure and partial on a hook timeout”](#getmicrovmimagebuilds-snapshotbuild-is-absent-on-a-container-build-failure-and-partial-on-a-hook-timeout)
Measured 2026-08-16 across successful and deliberately failed images. Preserve this optional structure and its optional fields:
| Outcome | `snapshotBuild` |
| ----------------------- | ----------------------------------------------------- |
| Successful | Memory, code-install, and disk-snapshot sizes present |
| Ready-hook timeout | Only `codeInstallSizeInBytes` present |
| Container build failure | Entire member absent |
Filling missing values with zero erases the distinction between an unbuilt image and installed code whose daemon never became ready. Both tested failures appeared in builds for Graviton 3 and 4.
## `baseImageVersion` is accepted, validated, and normalised on the way back
[Section titled “baseImageVersion is accepted, validated, and normalised on the way back”](#baseimageversion-is-accepted-validated-and-normalised-on-the-way-back)
Measured 2026-08-16. `baseImageVersion: "999"` for `al2023-1` failed with HTTP 400 before creating an image. A build pinned to `"1"` read back as `"1.0"`. Discover valid request versions through the managed-version listing; do not compare echoed strings literally with listing strings.
An unpinned build also reports a base version, so readback alone does not prove the caller pinned one. Retain the original request for reproducibility.
## A guest listening on the wrong port fails the build with a clean build log
[Section titled “A guest listening on the wrong port fails the build with a clean build log”](#a-guest-listening-on-the-wrong-port-fails-the-build-with-a-clean-build-log)
Measured 2026-08-16, `al2023-1`, 8192 MiB baseline. A mismatch between `AGENTD_PORT` and `hooks.port` produced `CREATE_FAILED` despite successful Docker layers and clean startup logs: AWS called hooks on the wrong port. The same happens when the daemon defaults to 9000 but the client selects another port. Compare version readback with the Dockerfile; the client now rejects known mismatches before building.
## A baked environment layer removes the guest’s env init, measured with `build --project`
[Section titled “A baked environment layer removes the guest’s env init, measured with build --project”](#a-baked-environment-layer-removes-the-guests-env-init-measured-with-build---project)
Measured 2026-09-02, `al2023-1`, 1024 MiB baseline, a Python project with one `attrs` dependency:
| Operation | Time |
| ----------------------------------------------- | --------------------- |
| First `build --project --reuse` | 126.6 s |
| Same dependency files, reused image | 0.48 s |
| Lockfile-only edit, new build | 125.1 s |
| Fresh VM importing from baked environment | 11.13–12.44 s running |
| Plain VM installing dependencies then importing | 31.85 s running |
The lockfile edit changed the image hash and installed dependency version. This small sample saved roughly 20 seconds per launch; it is not a general benchmark. The earlier description of a launch without `--egress` as having no network was incorrect: the measurement showed dependency reuse, not network isolation.
Exec starts with a minimal environment. In this run no `PATH` or `HOME` was present; uv downloaded another interpreter despite one being installed. Use the baked venv’s absolute executable path or supply the required environment.
## A VM launched without the egress connector still has outbound network
[Section titled “A VM launched without the egress connector still has outbound network”](#a-vm-launched-without-the-egress-connector-still-has-outbound-network)
Measured 2026-09-11, 2026-09-12, and 2026-09-13, us-east-1, API `2025-09-09`, `al2023-1`, baselines 512 and 1024 MiB. Launches omitting `egressNetworkConnectors` reached public destinations:
| Destination | Result without managed egress connector |
| -------------------------------------- | --------------------------------------- |
| `example.com` | 200 |
| `github.com` | 200 |
| `sts.amazonaws.com` | 302 |
| `pypi.org` | 200 |
| `pypi.org` with an invalid HTTPS proxy | curl exit 7, HTTP status `000` |
The proxy-variable result is the basis for `--deny-egress`; it changes client behavior without removing the network path.
**Correction, 2026-09-16:** the earlier model review incorrectly concluded that no VPC control existed because it inspected only `lambda-microvms`. Boto3/botocore 1.43.95 also exposes `lambda-core` API `2026-04-30`, whose `CreateNetworkConnector` accepts VPC subnets and security groups. AWS documents attaching the active connector ARN through `RunMicrovm.egressNetworkConnectors`. No internet egress requires a VPC without an IGW or NAT gateway, with no alternative internet path. These are SDK/documentation findings; a VPC-isolated launch was not live-measured in this refresh. See [Networking](./networking.md).
## The guest reaches the execution role’s credentials through MMDS, and no in-guest block works
[Section titled “The guest reaches the execution role’s credentials through MMDS, and no in-guest block works”](#the-guest-reaches-the-execution-roles-credentials-through-mmds-and-no-in-guest-block-works)
Measured 2026-09-11 and 2026-09-12, `al2023-1`, 512 MiB baseline. `169.254.169.254` served the execution role through Firecracker MMDS:
| Request | Response |
| ---------------------------- | --------------------------------------- |
| Credential GET without token | 401 |
| IMDSv2 token PUT | 200 |
| Credential GET with token | 200, full temporary credential document |
| Token PUT as uid 1000 | 200 |
Both managed-egress and connector-less VMs behaved alike. Credential values were neither printed nor retained.
The daemon and child reported `CapBnd 00000000a80425fb`, lacking `CAP_NET_ADMIN` and `CAP_SYS_ADMIN`, including with identity repair requested. After installing `iproute`, route/rule/link changes still returned `EPERM`; relevant `/proc/sys` writes were read-only. No tested in-guest metadata block worked. This limits the earlier August capability observation.
Use least privilege on the execution role. VPC internet isolation does not remove metadata credentials. The conformance role permits CloudWatch logging and its policy is checked by `drive_platform_posture`.
# Wire protocol v1
> The lifecycle hook paths are fixed by the platform, so they stay unversioned. Everything this project owns lives under /v1/. Every response carries a microvms-agentd-version…
The lifecycle hook paths are fixed by the platform, so they stay unversioned. Everything this project owns lives under `/v1/`. Every response carries a `microvms-agentd-version` header.
## Routes
[Section titled “Routes”](#routes)
Hooks are served under `/aws/lambda-microvms/runtime/v1/`, abbreviated `HOOKS` below. That prefix is fixed by the service, so a daemon serving a bare `/run` never gets bootstrapped.
| Route | Auth | Purpose |
| --------------------------------------------- | -------------------- | ------------------------------------------------------------------------------------ |
| `POST HOOKS/ready` | none (platform hook) | image-build readiness probe |
| `POST HOOKS/validate` | none (platform hook) | image-build validation probe |
| `POST HOOKS/run` | none (platform hook) | one-shot token bootstrap from `runHookPayload`, plus the optional launch environment |
| `POST HOOKS/suspend` | none (platform hook) | acknowledged and logged |
| `POST HOOKS/resume` | none (platform hook) | acknowledged; signals in-memory state loss |
| `POST HOOKS/terminate` | none (platform hook) | acknowledged; begins graceful shutdown |
| `POST /v1/exec/start` | bearer | start a command under a caller-minted `exec_id` |
| `GET /v1/exec/{id}` | bearer | poll status and output; never mutates |
| `GET /v1/exec/{id}/stream?offset=` | bearer | follow output as SSE from a byte offset |
| `POST /v1/exec/{id}/stdin` | bearer | write to a child’s stdin, or signal EOF |
| `POST /v1/exec/{id}/ack` | bearer | release output, enter TTL collection |
| `POST /v1/exec/{id}/kill` | bearer | signal escalation to the process group |
| `GET /v1/procs` | bearer | process accounting: every exec’s group and its live pids, read from `/proc` |
| `GET /v1/tcp?port=&identity=` | bearer | WebSocket relay to `127.0.0.1:` in the guest; close codes carry the outcome |
| `PUT /v1/fs/tar` | bearer | streaming tar upload and confined extraction |
| `GET /v1/fs/tar?path=` | bearer | streaming tar download |
| `PUT /v1/fs/file` | bearer | write one file |
| `GET /v1/fs/file?path=&start_line=&end_line=` | bearer | read one file, or a 1-based inclusive line range of it |
| `GET /v1/health` | none | liveness, version, bootstrap state, exec-activity, hook observations |
| `GET /v1/schema` | none | this contract as a JSON Schema document: every route, shape, status, and limit |
**Every hook invocation is recorded and reported on `/v1/health`.** The daemon records each invocation in memory — the hook’s name and the daemon’s clock, in arrival order — before the handler does any work or chooses a status, and `GET /v1/health` reports the log as `hooks` (`[{hook, fired_at}]`, oldest first) beside `hooks_dropped`, a count of invocations past the log’s cap. A run hook that arrives malformed or answers 409 still leaves a `run` observation: the record is of the invocation, not the verdict. The platform writes no CloudWatch logs for the validate hook, so this record is the only trace that hook leaves anywhere. Build-time hooks (`ready`, `validate`) fire in the snapshot VM before the snapshot is taken, so their in-memory records ride the memory image into every VM launched from it and are visible on the first health poll.
One caveat is part of this contract rather than a footnote. The hook routes are unauthenticated and arrive over loopback indistinguishably from an in-VM process (see “Trust boundary” below), so a hostile workload can forge ADDITIONAL observations by posting the hook paths itself. It cannot remove or alter real ones: the daemon records before it responds, and the capped log keeps the *earliest* entries — the platform’s real lifecycle firings, which arrive before anything runs in the guest — while later spam is dropped and counted in `hooks_dropped`. Read an observation as “the daemon saw this hook fire”, never as “only the platform could have fired it”. Both fields are `#[serde(default)]` like `busy` and `execs`, and for the same reason: an older daemon omits them, and empty/zero is the honest reading.
## Rules that exist because a defect proved them necessary
[Section titled “Rules that exist because a defect proved them necessary”](#rules-that-exist-because-a-defect-proved-them-necessary)
Each of these rules comes from a real bug found during the Harbor PR #2469 integration. They are part of the protocol contract, so implementations must follow them.
**Bootstrap is one-shot, and a replay of the identical token succeeds.** A first `/run` installs the token and returns 200. A later `/run` carrying the same token returns 200, because the platform may retry its own hook and must not be told the VM is broken. A later `/run` carrying a different token returns 409 and changes nothing. The model in `model/` checks this over every interleaving, including a racing in-VM caller.
**Control routes answer 503 before bootstrap.** They do not answer 404, and they do not drop the connection.
**A missing or malformed body key is 400, never 404.** Clients map 404 onto “file not found”. Because of that mapping, returning 404 for a protocol typo makes the client believe an artifact is absent when it is not. One defect went undetected this way.
**Authorization is decided before any body byte is read.** An unauthenticated caller must not be able to make the daemon allocate. Rejected requests still drain a small body so pooled client connections keep working; larger ones close.
**Token comparison happens on bytes, in constant time.** Comparing `str` values raises on non-ASCII input in some languages. Any caller controls the header, so a non-ASCII header value could crash the connection.
**No exception on the parse, auth, or routing path may drop a connection.** A catch-all returns 500. Raw TLS handshake bytes get a 400 and a debug log, since something in the platform’s path probes the port with TLS first.
**`cwd` is omitted when unset.** When the client sends no working directory, the daemon emits no `cd` prefix, and the child inherits the daemon’s own working directory. Because the daemon is the container `CMD`, that directory is the image `WORKDIR`. Forcing `/` breaks prebuilt-image tasks and defeats any harness that discovers the image workdir with `pwd`.
**Exec is idempotent on a caller-minted `exec_id`.** A retried `/exec/start` returns success without spawning a second child. Polling is read-only. Output lives until the caller acks, and only acked entries are collected. The Python predecessor unlinked output files at child exit, which destroyed anything a backgrounded grandchild wrote afterward.
**A shell wraps the command only when the caller asks for one.** An argv array execs directly. `shell: true` wraps in `sh -c` with the command as a single argument. The predecessor’s brace-group wrapper turned empty and comment-terminated commands into syntax errors and let an unbalanced `}` escape the group.
**Tar extraction mirrors the CPython `data` filter contract.** In-tree symlinks are preserved, because harnesses legitimately pack them. Absolute link targets are refused. Relative targets must resolve under the root, using `normpath` semantics rather than `realpath`. Symlinks resolve relative to their own directory, while hard links resolve against the archive root. Member count and total size are capped. Modes are applied after content lands.
**A symlink an archive wrote cannot redirect a later member, and the kernel is what enforces that.** The `normpath` rule above judges a member at the depth its name implies, and that is not the same as the depth the write reaches once a symlink is in the path. So the daemon opens the extraction root once and creates every member relative to that descriptor with `openat2`, using `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS`. A member whose path would traverse a symlink is refused with 400 rather than written somewhere else. `openat2` needs Linux 5.6 or newer. On an older kernel the syscall answers `ENOSYS` and extraction answers 500, which is a refusal rather than a silent fall back to the weaker check.
**Bodies stream to disk, and caps are enforced on the wire.** The predecessor buffered whole archives in memory on a VM whose baseline can be 512 MiB, where an OOM-killed daemon is unrecoverable. It also measured archive size inside the gzip `with` block, where the stream is unflushed. There, `tell()` reported 10 bytes for a 327-byte archive, so the size guard almost never fired.
**Output is bounded, and truncation is marked explicitly.** A post-exit linger deadline bounds how long the daemon waits on grandchildren still holding the pipe.
**A grandchild outlives its exec unless the caller asks otherwise, and `/v1/procs` is how anyone finds it.** Each exec runs in its own process group, and until now only `/v1/exec/{id}/kill` ever signalled that group: a command that backgrounded something and exited left it running with nothing able to enumerate it (issue #157, measured 2026-09-11 — a ticker still counting six seconds after its exec reported exit 0). `GET /v1/procs` answers `{procs: [{exec_id, pgid, started_at, child_exited, reap, pids}]}`, one entry per registered exec in any phase. `pids` are the live pids whose process group is `pgid`, read from `/proc//stat` inside the guest, so the route needs no `ps` in the image; zombies are not live and are not listed, and an entry whose `pgid` was never captured lists none. `child_exited` is read from the same write-once marker `busy` uses, never from the polled result, so an acked exec still reads exited. `started_at` is epoch seconds on the daemon’s clock, like a hook’s `fired_at`. The row worth reading is `child_exited: true` with a non-empty `pids`: that is the survivor, and its `exec_id` is what the kill route takes.
`reap_group_on_exit: true` on `/v1/exec/start` is the opt-in that closes the gap at the source. When set, the daemon runs the kill route’s SIGTERM-then-SIGKILL escalation against the group as soon as the child’s exit is observed and before the linger begins, so the survivors die, the pipe closes, and `writers_may_be_alive` reads false because the linger saw EOF rather than its deadline. A timed-out exec is already escalated and is not escalated twice. The escalation’s grace ends as soon as `/proc` shows no live member of the group — not when `killpg(pgid, 0)` fails, because the daemon is PID 1 in the guest and a signalled grandchild stays a zombie that still answers that probe. The flag defaults to false and the default is the contract above: a backgrounded grandchild that keeps writing is a feature for the caller who did not set it. `reap` on a procs entry echoes what the start request asked.
## The launch environment
[Section titled “The launch environment”](#the-launch-environment)
The run-hook payload may carry an `env` map alongside `agent_token`. It becomes the base environment of every later exec.
```json
1
{"runHookPayload": "{\"agent_token\": \"...\", \"env\": {\"KEY\": \"VALUE\"}}"}
```
**The per-request `env` wins on a key both set.** The daemon applies the launch env first and the request’s own map second, so a launch env is a default for the whole VM and a request is the specific thing happening now. A caller who never sends a launch env sees no change: the child’s environment is the request’s map and nothing else, exactly as before.
**The agent token never becomes part of it.** `env_clear()` still runs, so the daemon’s own environment reaches no child, and the launch env and the token are separate parameters through the whole install path — there is no field a refactor could forward one into. Proven by a test that spawns `/usr/bin/env` with a launch env installed and asserts the child’s environment is *exactly* that map: an extra variable of any name fails it, which is what a leak would look like.
**Only the first successful bootstrap sets it.** A replay of the identical token answers 200 and leaves the installed env alone, and a conflicting token answers 409 and leaves it alone. Without that, a caller who cannot win the token could still rewrite the environment every later child runs in.
**Every value is a string, and a malformed `env` is 400 naming the problem.** A non-object `env`, or a value that is a number or a nested object, is refused with a body that names the key or the shape. The body never quotes a value, because the payload carries the token.
**Unknown payload keys are ignored.** A 400 at this hook makes the platform terminate the VM before forwarding any traffic, so a newer client sending a field this daemon has never heard of must still be able to bootstrap it. Forward compatibility here is the difference between an ignored field and a dead launch.
**The token and the env share one 4096-byte budget, measured in UTF-8 bytes and inclusive** (`PLATFORM.md`). The daemon cannot enforce what the platform already rejected — an over-ceiling payload never reaches the guest — so the check is on the *client* side, in `microvms-core`’s `RunHookPayload::for_launch`, and it fires before any control-plane call. botocore does not enforce the ceiling either, so without that local check there is no signal at all until AWS answers with a `ValidationException` on a member the caller did not know they were filling. The refusal names the byte count, the ceiling, and how much of it the env is, because “4142 bytes, ceiling 4096” alone does not say whether to shorten the token or drop a variable. One bearer token fits with room to spare; a set of AWS session credentials does not, and that is what makes this ceiling reachable in practice.
## Line-ranged text reads
[Section titled “Line-ranged text reads”](#line-ranged-text-reads)
`GET /v1/fs/file` takes optional `start_line` and `end_line`. The semantics are the AI SDK harness contract’s `readTextFile`, copied rather than chosen, because this route is what that method is implemented on top of:
**Both bounds are 1-based and inclusive.** `start_line=2&end_line=4` is three lines. `start_line` absent means 1 and `end_line` absent means through EOF.
**An `end_line` past the last line reads through EOF without an error.** Lines 1..1000 of a twelve-line file is a 200 carrying twelve lines, never a 416. A `start_line` past the last line is an empty 200, not a 404: the file is there and the window is empty, which is a different fact from the file being absent.
**A line owns its terminating newline.** Lines 1..2 and lines 3..5 concatenate back into the file rather than losing a separator at the seam. A last line with no trailing newline has none to own and is returned as it is.
**`start_line=0` and `end_line < start_line` are 400.** Neither is 416: 416 is about a range the file cannot satisfy, and both of these are ranges no file could, so a client sent to look at the file would be looking in the wrong place. A non-integer bound is also 400, and it does not masquerade as the missing-`path` refusal.
**A range still streams.** The read is filtered chunk by chunk and stops reading once the window closes, so lines 1..5 of a large file cost the first chunk. Nothing buffers a file to slice it, for the same reason nothing buffers an upload: an OOM-killed daemon in a MicroVM is unrecoverable.
**With no range the response is byte-identical to what it always was.** The un-ranged read hands back the reader stream untouched, so the path every existing caller uses does not acquire the range feature’s bug surface.
## Idle policy, and why liveness is a field rather than a route
[Section titled “Idle policy, and why liveness is a field rather than a route”](#idle-policy-and-why-liveness-is-a-field-rather-than-a-route)
The platform measures idleness by inbound traffic through the endpoint proxy (`PLATFORM.md`, “`idlePolicy`”). A workload holding an outbound connection, or one simply computing for hours, receives none, so auto-suspend can freeze a VM mid-work. Multi-hour agent runs are the real case.
**A guest-side request cannot fix this, and the daemon does not pretend otherwise.** The endpoint proxy terminates *outside* the VM and forwards over loopback — measured, `PLATFORM.md`, “The platform’s own hook arrives over loopback”. A request an in-VM process sends to the daemon’s own port therefore never reaches the thing counting traffic. A “keep myself alive” route would be a keepalive that keeps nothing alive, and it would be discovered as broken by a multi-hour run auto-suspending mid-work, which is the failure it was added to prevent.
**So `GET /v1/health` carries `busy` and `execs`, for an orchestrator outside the VM.** The orchestrator’s own poll *is* the inbound traffic, and `busy` is what makes that poll informed rather than unconditional. The assertion of liveness is therefore repeated and is explicitly the caller’s, which is the property that rules out the daemon self-keepaliving: a hung process would then bill to the 8-hour `maximumDurationInSeconds` ceiling with nobody having asked.
**`busy` means producing, not unfinished.** An exec whose child has exited and whose result is waiting to be acked is not busy — nothing is running, and holding a VM alive at baseline billing for a command that is over is the mistake this distinction exists to prevent. `execs` counts every registered entry in any phase, so `busy: false` with a non-zero count is a VM holding unacked output somebody still has to collect before terminating it.
Both fields default to `false` and `0` when absent, unlike every other field on the response. The daemon is baked into an image while a client is installed separately, so a current client routinely talks to a daemon from whenever that image was built; a required field would make a health call fail outright against an older daemon, turning a missing signal into an unreachable VM.
## Streaming and stdin
[Section titled “Streaming and stdin”](#streaming-and-stdin)
Both features serve one consumer, an agent harness running inside the VM. The harness emits output for minutes and may need a prompt written to it. Polling serves neither need well, because it re-sends the whole buffer each time and truncates at the output cap.
**The stream is a read-only view of the exec.** An exec is a server-side record keyed by its caller-minted `exec_id`. Attaching, detaching, or dropping a connection must not affect the command. Both views must keep working, so poll returns the buffer, stream follows it, and neither disturbs the other.
**Resume is by byte offset.** `?offset=N` yields exactly the bytes after N, so a client that reconnects can pick up where it left off. For comparison, E2B’s reattach takes no offset, so a reconnecting E2B client loses everything produced during the gap. A reattach past the retained window gets an explicit `gap` event naming the missing range. Without that event, the client would keep streaming and never learn that bytes were skipped.
**SSE is used because it can carry a typed terminal event.** A raw chunked byte stream cannot distinguish a finished command from a dropped connection, because the bytes are identical in both cases. The stream therefore emits a typed `exit` event carrying the status and *then* ends. Keep-alive comments fill silences, so an agent harness thinking for two minutes does not look like a dead connection.
**stdin is opt-in and a separate request.** A command that does not ask for stdin gets `Stdio::null()`, so nothing inherits a surprise descriptor. Writing to a command that did not request stdin returns 409. Writes go to `POST /v1/exec/{id}/stdin` and are never multiplexed onto the output connection. Because the two connections are separate, a dropped attach cannot corrupt stdin. EOF is an explicit signal rather than inferred, because a child reading stdin cannot exit until the daemon drops its own handle. `Child::wait()` drops the child’s copy of the handle, not the daemon’s.
## Reconnect-at-cursor across a real suspend, measured
[Section titled “Reconnect-at-cursor across a real suspend, measured”](#reconnect-at-cursor-across-a-real-suspend-measured)
The claim above — that resume by byte offset is what distinguishes this from a backend whose reattach loses the gap — was asserted from the daemon’s side only. The cut under unit test is a response body ending without an exit frame, which is the condition the reconnect keys on, but nothing had put a real platform suspend across a live exec.
Measured 2026-08-15, us-east-1, through `Session.spawn` in `microvms-js` against a guest ticking once a second and also appending each tick to a file, so the guest’s own record is an oracle independent of the stream under test:
| | Observation |
| ----------------------------- | --------------------------------------------------------------- |
| Before the suspend | `tick-1`, `tick-2`, `tick-3` delivered on the handle’s `stdout` |
| Suspend / resume | `SUSPENDED`, held \~20s, `RUNNING` |
| The handle already held | Resumed at **`tick-4`** — contiguous, no restart, no duplicate |
| Gaps recorded | **None**, on either the held handle or a fresh one |
| A fresh handle at `offset: 0` | Replayed **200 of 200** ticks, indices contiguous `1..200` |
| The guest’s own file | Agreed with the stream at every point |
So the property holds against the thing it was designed for, and it holds in both shapes: the handle a caller was already reading recovers at its cursor, and a *new* handle for the same `exec_id` replays the whole output across the suspension. The exec itself is untouched by the freeze, which is the `PLATFORM.md` finding this depends on.
The other three `ExecProcess` properties were measured on the same VM. `stdout` and `stderr` arrive as separate streams carrying only their own bytes, from one interleaved SSE channel with a per-frame discriminator. `wait()` reports the daemon’s real exit code — 0 for a success, 42 for `exit 42` — because it reads the exec record rather than inferring from the stream ending. `kill()` terminates the process group and a second call succeeds rather than 404ing, and `wait()` afterwards reports `signal: 15` with no exit code, so a killed build cannot read as passing.
## Credentials for a caller’s own connection
[Section titled “Credentials for a caller’s own connection”](#credentials-for-a-callers-own-connection)
`Session::connect_headers(port)` and `connect_subprotocols(port)` hand a caller what it needs to open its own connection to some other port on the same VM — the header pair for HTTPS, the three subprotocols for a WebSocket.
**Both mint a token scoped to the port they name, and that is load-bearing rather than an implementation detail.** The service scopes a proxy token at mint time, so a token minted for the agent port answers 403 `Access to port denied` for any other port and close code 1006 with no reason on a WebSocket. An implementation that reused the session’s cached token would therefore return a *correct-looking* port value behind a credential that does not authorize it — which is what this client did until the live run of 2026-08-15 caught it. See `PLATFORM.md`, “`allowedPorts` is a union of three forms”.
The scope is extended rather than replaced: a mint for a new port asks for every port already covered plus that one, so warming the cache for a workload port does not cost the session access to its own. One extra control-plane call per new port, cached thereafter, and `proxy_mint_count()` is the observable that says so.
**A caller debugging a failed WebSocket should retry the same port over HTTPS with `connect_headers`.** Every handshake failure is 1006 and none of them says why; the HTTPS request distinguishes 403 (wrong scope) from 502 (right scope, nothing listening).
## Trust boundary
[Section titled “Trust boundary”](#trust-boundary)
The platform’s `/run` hook arrives from `127.0.0.1` and is indistinguishable at the socket level from a request sent by a process inside the VM (measured; see `PLATFORM.md`). Filtering by source address therefore cannot separate the platform from an in-VM process, so it provides no protection here.
The remaining defenses, all checked in `model/`, are the following:
1. Bootstrap is one-shot, so a losing racer never replaces the winner’s token.
2. A post-bootstrap hijack attempt is refused at the hook with 409 and at the control API with 401.
3. The agent token never enters an exec’d child’s environment.
One risk remains. The design assumes the daemon is the container `CMD` and that the harness issues its first exec only after readiness. The daemon does not enforce this invariant. A base image that starts its own background process before bootstrap breaks it. `model/` includes that configuration and reports the counterexample path, so the consequence of breaking the invariant is a checked result rather than a prediction. Enforcing the invariant is the responsibility of whoever builds the image, not of this daemon.
# Strategy: the trust and preparation contract for coding agents on microVMs
> Provide a reusable guest daemon, client libraries, and measured AWS behavior for teams building on Lambda MicroVMs. Scheduling, pooling, and task routing belong to consumer…
Provide a reusable guest daemon, client libraries, and measured AWS behavior for teams building on Lambda MicroVMs. Scheduling, pooling, and task routing belong to consumer applications.
## Diagnosis
[Section titled “Diagnosis”](#diagnosis)
A workload and its control daemon share the guest. Bootstrap, credential handling, filesystem operations, and lifecycle recovery therefore need explicit contracts. Image preparation is also a recurring cost: dependencies can be built once and reused across launches.
The platform changes independently of this package. SDK models describe request shapes; only runtime measurements establish service behavior. [Platform](./platform.md) records those observations, with dates and corrections.
## Guiding policy
[Section titled “Guiding policy”](#guiding-policy)
Keep shared behavior in core and use thin CLI/Python/Node adapters. Validate requests before billable operations, preserve meaningful errors, and test failure paths. State limits directly: guest root is not isolated from the daemon, metadata exposes the execution role, and internet isolation requires a correctly configured VPC connector.
## Coherent actions
[Section titled “Coherent actions”](#coherent-actions)
1. Maintain the [wire protocol](./protocol.md) and [trust contract](./trust.md), with model, unit, property, and live conformance coverage appropriate to each claim.
2. Reuse images by build inputs and dependency lockfiles. Keep per-run code and credentials out of shared snapshots.
3. Maintain published CLI and language packages from one implementation. Refresh API references with boto3 and check serializer/SDK parity.
4. Keep documentation short: tutorials explain tasks, generated references describe contracts, and platform notes retain dated evidence.
The agent helpers are a deliberate convenience layer over core primitives; [Agent VMs](./agent-vms.md) defines their bounded scope.
## What we are deliberately not doing
[Section titled “What we are deliberately not doing”](#what-we-are-deliberately-not-doing)
* A scheduler, pool manager, or general orchestrator.
* Guest-side process-tree cloning as a substitute for provider snapshots.
* A new turn-boundary suspend protocol: consumers can use existing lifecycle calls and endpoint health polling.
* Competing with AgentCore by duplicating its managed execution product.
## The AWS ask, and the honest bet
[Section titled “The AWS ask, and the honest bet”](#the-aws-ask-and-the-honest-bet)
Reusable snapshots of running VM state require provider support. Existing suspend/resume preserves a single VM, but does not establish a reusable clone API. Prefer measured capabilities over predictions about future AWS features.
## How we would know this worked
[Section titled “How we would know this worked”](#how-we-would-know-this-worked)
Consumers can integrate the daemon without rediscovering bootstrap or network assumptions, reuse environments across launches, and upgrade packages without surprises between the CLI and language bindings. Live coverage remains scoped to the regions and paths actually exercised.
## What the first draft got wrong
[Section titled “What the first draft got wrong”](#what-the-first-draft-got-wrong)
The earlier strategy prioritized a turn-boundary suspend convention even though existing hooks and lifecycle APIs already covered it. It also mixed human interactive idle time with headless batch workloads. The lesson is to measure the target workload and check existing platform capabilities before adding an abstraction. Full research notes remain in git history.
# The trust contract for a control daemon inside a Lambda MicroVM
> agentd authenticates control requests inside a VM that may run hostile workloads. It does not isolate itself from a root workload. This document states the guarantees, deployment…
`agentd` authenticates control requests inside a VM that may run hostile workloads. It does not isolate itself from a root workload. This document states the guarantees, deployment assumptions, and limits. [Protocol](./protocol.md) defines the wire behavior; [Platform](./platform.md) records dated AWS evidence.
## The threat model
[Section titled “The threat model”](#the-threat-model)
A guest process can connect to the daemon over loopback, including lifecycle hook paths. A process without the agent token must not gain control through an authentication bypass or a second bootstrap. The token lives in daemon memory and is not written to disk or logged by the daemon.
An authorized caller can execute as root. A root workload can potentially read daemon memory with `ptrace` or `/proc//mem`, change files, or exhaust guest resources. User demotion is a convenience, not a separate sandbox. The model and authentication tests do not prove isolation from guest root.
## What the platform gives you for free
[Section titled “What the platform gives you for free”](#what-the-platform-gives-you-for-free)
AWS documents two relevant properties:
* Endpoint requests require a proxy credential scoped to the VM and permitted ports, with a maximum lifetime of 60 minutes. This credential is separate from the agent token. Clients refresh it for later requests.
* External traffic starts only after the run hook returns HTTP 200. This protects launch-time bootstrap from external traffic, but not from a process already running inside the guest.
Port scope was measured on 2026-08-15, us-east-1, API `2025-09-09`: a token for 9000 could not access 8080. Guest loopback traffic bypasses this proxy.
## Why source-address filtering is wrong, not merely weak
[Section titled “Why source-address filtering is wrong, not merely weak”](#why-source-address-filtering-is-wrong-not-merely-weak)
Measured 2026-08-04, us-east-1, API `2025-09-09`: platform hooks and proxied control requests arrived from `127.0.0.1`. A loopback filter cannot identify AWS and rejecting loopback rejects legitimate bootstrap. The platform does not present an authentication credential to the run hook.
## The five defenses that remain
[Section titled “The five defenses that remain”](#the-five-defenses-that-remain)
1. **One-shot bootstrap.** The first valid run hook installs the token and returns 200. An identical replay returns 200; a different token returns 409 without modifying state. Replays must remain idempotent because a failed run hook can cause AWS to terminate the VM. Implemented in `agentd/src/state.rs` and `agentd/src/routes.rs`.
2. **Constant-time comparison on bytes.** Bootstrap and request guards compare equal-length byte strings with `subtle::ct_eq`, avoiding Unicode decoding errors. Token length remains observable. Implemented in `agentd/src/auth.rs`.
3. **Authorization before request-body processing.** Protected routes reject unauthorized requests before parsing or buffering their body. A bounded drain (64 KiB by default) reduces connection resets; excess data closes the connection rather than causing an unbounded allocation.
4. **Explicit child environments.** Exec uses `env_clear()` and adds only the launch/request environment. The installed agent token is never implicitly inherited. Caller-supplied environment values are intentionally available to child processes. User changes use `Command::uid`/`gid`; avoid Rust `pre_exec` closures in a multithreaded process because inherited locks can deadlock after fork. Implemented in `agentd/src/exec.rs`.
5. **Distinct status codes.** Protected routes return 503 before bootstrap and 401 for an invalid token afterward. Unknown routes return 404. `/v1/health` and `/v1/schema` remain unauthenticated; health exposes bootstrap state so callers can check readiness.
`model/` explores bootstrap interleavings and tests both compliant and broken deployments. Unit and conformance tests exercise the implementation. The model does not cover identity repair, filesystem confinement, or all Linux process behavior.
## The unenforced invariant
[Section titled “The unenforced invariant”](#the-unenforced-invariant)
**Run the daemon as the image’s `CMD`, and start workloads only after bootstrap and readiness.** Use `ENTRYPOINT []` and `CMD ["/agentd"]`. Review the base image and startup behavior for processes that could run before the daemon.
A pre-existing hostile process could win the first run-hook request and install its own token. One-shot bootstrap then preserves the wrong principal. The daemon cannot verify the image’s full startup history. The model includes this misconfiguration and confirms an attacker can win that race.
## Identity repair for derived VMs
[Section titled “Identity repair for derived VMs”](#identity-repair-for-derived-vms)
Restored images can share files and cached userspace state. Firecracker VMGenID and Linux kernel reseeding do not repair identifiers already stored on disk or cached by applications. Optional identity repair runs during bootstrap and reports its result through health.
`agentd/src/identity.rs` uses a fresh 128-bit seed to:
* Rewrite `/etc/machine-id` and set the hostname.
* Remove `/var/lib/systemd/random-seed` rather than sharing its snapshot value.
* Attempt to shadow `/proc/sys/kernel/random/boot_id` with a bind mount.
* Remove configured cached identity files, including `/var/lib/dbus/machine-id` by default.
Repair cannot revoke values a process already read, update arbitrary application caches, or override missing kernel capabilities. A bind mount is namespace-local. Failures leave the daemon serving and set `identity_degraded`; callers that require repaired identity must check it. Opting out is reported separately through `identity_repaired`.
The August 2026 measurement succeeded with `additionalOsCapabilities: ["ALL"]`. September measurements found a restricted capability set even when repair was requested. Those observations are both retained in [Platform](./platform.md); requesting `ALL` is not proof that every repair operation succeeded.
## Tunnel identity: proving which VM answered
[Section titled “Tunnel identity: proving which VM answered”](#tunnel-identity-proving-which-vm-answered)
`microvm run --keep --identity` provisions keys for `microvm tunnel --verify-identity`. A Noise KK handshake runs inside the WebSocket and terminates in the daemon. The host supplies the VM seed and host public key through one-shot bootstrap; both sides pin the other’s public key. Subsequent tunnel data is encrypted with ChaCha20-Poly1305, beyond the proxy’s TLS termination. See `protocol/src/identity.rs`.
The local name record stores the host secret and VM public pin; it does not retain the VM secret. A stolen record can authorize the same tunnels its agent token already permits, but does not provide the VM key needed to impersonate it. Protect local records accordingly.
This proves key possession by the endpoint, not that the guest remains uncompromised. A root workload may read daemon keys from memory. It also does not authenticate arbitrary plain HTTP or unverified tunnel traffic.
## The execution role is the boundary
[Section titled “The execution role is the boundary”](#the-execution-role-is-the-boundary)
Measured 2026-09-11 and 2026-09-12, us-east-1, API `2025-09-09`: the guest could retrieve the execution role’s temporary credentials from Firecracker MMDS at `169.254.169.254`. Both root and uid 1000 could do so, with and without the managed internet connector. An empty child environment does not hide metadata credentials.
Grant the execution role only permissions every workload may use. The conformance role grants CloudWatch logging; its policy is checked during live verification. Deliver additional workload credentials with their own scope and lifetime instead of broadening the shared VM role.
Tested in-guest metadata blocks failed: the capability set lacked `CAP_NET_ADMIN`, and route/rule/link changes returned `EPERM`. Relevant sysctl paths were read-only. No tested guest-side block was effective. VPC internet isolation does not remove metadata access or make an overprivileged role safe.
## What this contract does not cover
[Section titled “What this contract does not cover”](#what-this-contract-does-not-cover)
**Internet isolation.** No internet egress requires a VPC without an internet gateway or NAT gateway, attached with a custom VPC connector. Check subnet routes for other paths, including IPv6, transit networks, and proxies. [Networking](./networking.md) gives the boto3 setup. Connector lifecycle belongs to the separate Lambda core API; a MicroVM launch supplies its ARN.
Omitting `--egress` only omits the managed connector. Default-network tests still reached public sites. `--deny-egress` sets proxy variables and cannot constrain a workload that ignores them. The package does not audit custom connector routing; its posture values are conservative:
| `egressPosture` | Meaning |
| --------------- | -------------------------------------------------------------- |
| `open` | Managed internet connector requested |
| `unsealed` | Internet isolation has not been established by the client |
| `best-effort` | Guest proxy variables discourage outbound HTTP clients |
| `sealed` | Retained for stored labels; not inferred from a launch request |
**Credential rotation.** The serialized `runHookPayload` has a measured 4096-byte limit shared by the token, environment, and identity material. It is delivered once and is not a rotation channel. The client validates the full payload before calling AWS. Use an authenticated session or an external credential broker for subsequent delivery and refresh.
**Workload confinement.** There is no seccomp or user-namespace boundary between workload and daemon. Single-file APIs intentionally accept arbitrary guest paths because the same authorized principal can execute as root. Tar uploads are confined because member names originate in an archive and may not be paths its uploader intended; see [Protocol](./protocol.md).
# Learn
> Tutorials that take you from an empty machine to a project running inside a Lambda MicroVM, and task-shaped pages for operating one.
Run a coding agent or your own tools in a remote AWS MicroVM. Pick the path that gets you to a working sandbox; you do not need to read every tutorial in order.
## Start here
[Section titled “Start here”](#start-here)
| Your starting point | Next step |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| I have not installed anything | [Install the CLI](/microvms-agentd/learn/tutorial/install/) |
| I need AWS resources or my first sandbox | [First run](/microvms-agentd/learn/tutorial/first-run/) |
| I want an agent to work on my project | [Run Claude Code or Codex in a sandbox](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) |
| I want to use Python, Node, or Rust | [Run a sandbox from code](/microvms-agentd/learn/tutorial/from-code/) |
| I already have a VM running | [Keep working by name](/microvms-agentd/learn/tutorial/long-lived-vm/) |
| I want to upload code and collect outputs | [Run a project](/microvms-agentd/learn/tutorial/run-a-project/) |
With the CLI installed and AWS configured, the quickstart takes less than 90 seconds to read and start. The first image build takes several minutes. VMs and images create AWS charges. The first-run guide covers prerequisites, expected output, image reuse, and cleanup.
## More tasks
[Section titled “More tasks”](#more-tasks)
These guides assume the CLI is installed and AWS is configured.
| Page | Answers |
| ----------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [Write a guest Dockerfile](/microvms-agentd/learn/operations/write-a-guest-dockerfile/) | How to start from the generated stanza, and which traps cost a server-side build cycle |
| [Embed agentd in your own image](/microvms-agentd/learn/operations/embed-agentd-in-your-image/) | How to append the daemon to a task image your own harness drives |
| [Run coding agents on Bedrock](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) | How `agent-up` and `agent-prompt` run Claude Code and Codex CLI headless in a VM with no vendor API key |
| [Remote dev with code-server](/microvms-agentd/learn/operations/remote-dev-with-code-server/) | How to reach VS Code in a browser through `port-forward`, on a VM that suspends when you leave |
| [Prefetch S3 content at image build](/microvms-agentd/learn/operations/prefetch-s3-at-build/) | How to bake an S3 prefix into the snapshot so a launched VM makes no S3 call |
| [Configure networking](/microvms-agentd/learn/operations/configure-networking/) | Attach a VPC connector and restrict internet access |
| [Read the cost report](/microvms-agentd/learn/operations/read-the-cost-report/) | What each line means, why a total may read “at least”, and how to plan with `microvm cost` |
| [Debug a failed build](/microvms-agentd/learn/operations/debug-a-failed-build/) | Where the reason lives, how to read the build log, and what `ERR_BUILD_WEDGED` means |
| [Recover a leaked VM](/microvms-agentd/learn/operations/recover-a-leaked-vm/) | What `ls` and `history` say you left behind, and how to ask the account directly |
| [Configure the project file](/microvms-agentd/learn/operations/configure-the-project-file/) | Every `microvm.toml` key, which source wins, and what the loader refuses |
| [Drive it from a script or an agent](/microvms-agentd/learn/operations/drive-it-from-a-script-or-an-agent/) | The one-envelope rule, the exit codes, the manifest, and the streaming exception |
| [Run the live suite](/microvms-agentd/learn/operations/run-the-live-suite/) | What `mise run check` proves, what `mise run live` adds, what it costs, and how to leave the account clean |
## Automation and cleanup
[Section titled “Automation and cleanup”](#automation-and-cleanup)
Every command takes `--json` and then writes exactly one JSON envelope on stdout; progress goes to stderr. A success envelope carries `type` and `data`. A failure envelope carries a stable `code`, an `exitCode` that matches `$?`, a `finding` naming the section of [Platform](/microvms-agentd/internals/platform/) that measured the behavior, and `suggestions`. Branch on `code`, never on the `error` text. The one exception is `exec --stream`, which writes NDJSON events and the envelope last, under its own `type`. [Drive it from a script or an agent](/microvms-agentd/learn/operations/drive-it-from-a-script-or-an-agent/) develops this.
Teardown is attempted by default. `microvm run` builds an image, launches a VM, runs your command, reports the cost, and attempts cleanup. Inspect `leaked` for failures; interruption can leave resources behind. `--keep` opts out and hands you the identifiers you have just taken responsibility for. The image is the durable artifact: its snapshot has a one-week minimum retention, so deleting it early saves nothing and reusing it with `--image` is the economical habit.
If you are an AI agent rather than a person reading a page, start with `microvm manifest`. It prints every command, flag, response type, and exit code this binary accepts, as JSON, on a machine with no credentials and no network. [For agents](/microvms-agentd/agents/) names which surface answers which question. Come back here for the worked paths.
# Configure networking
> Attach a VPC connector and understand which settings enforce internet isolation.
No internet egress requires a VPC without an internet gateway or NAT gateway, with no alternative internet route. Create a VPC network connector, wait for it to become active, and attach its ARN at launch:
```bash
1
microvm run --image my-image \
2
--egress-network-connector "$CONNECTOR_ARN" \
3
--exec "echo hello"
```
[Networking](/microvms-agentd/internals/networking/) gives the boto3 setup and validation steps. The package does not audit the VPC’s routes. Omitting `--egress` only omits the managed internet connector; `--deny-egress` only sets proxy variables that a workload can ignore. Keep the execution role minimal because its credentials remain available inside the guest.
# Configure the project file
> Every key microvm.toml accepts, where run and doctor look for it, which source wins when a flag and the file disagree, and what the loader refuses.
```toml
1
image = "ci-image"
2
exec = "pytest -q"
3
memory = 4096
4
region = "us-west-2"
5
egress = true
6
auto-resume = true
7
shell = true
8
max-idle-sec = 120
9
suspended-sec = 300
10
max-duration-sec = 7200
11
artifacts = ["dist/**", "*.log"]
12
log-group = "/aws/lambda-microvms/ci-builds"
13
log-stream = "ci-image"
14
15
[env]
16
RUST_LOG = "debug"
17
CI = "1"
```
Every knob in the file already exists as a `run` flag. The file adds no capability, only persistence, so `microvm run` in a configured project needs zero flags. At the end of this page you will have a `microvm.toml` that pins your project’s launch, validated by `doctor`, and you will know which source won for each knob.
## 1. Where it is read
[Section titled “1. Where it is read”](#1-where-it-is-read)
`run` and `doctor` look for `./microvm.toml` beside the invocation when `--config` is not given. `--config ` reads that file instead, and its absence is `ERR_CONFIG`, because a path you typed and got wrong must not silently become “no config”; the implicit default’s absence means a project configured by flags, which is not an error. `--no-config` ignores any `microvm.toml`, even a malformed one, so flags and built-in defaults apply. `build` reads no config file, which is why `build --log-stream` requires `build --log-group` on the command line.
Most field names match their flags; the repeatable connector flag uses the plural array key `egress-network-connectors`.
## 2. Every key
[Section titled “2. Every key”](#2-every-key)
| Key | The flag it persists | Notes |
| --------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `image` | `run --image` | Launch this existing image instead of building one. An ARN or a name. |
| `binary` | `run [BINARY]` | The daemon binary to bake in. A relative path resolves against the config file’s directory, never the process cwd. |
| `exec` | `run --exec` | The shell command to run in the VM. |
| `memory` | `run --memory` | Baseline MiB. Validated against the same closed set as the flag: `512`, `1024`, `2048`, `4096`, `8192`. |
| `region` | `run --region` | An unlisted region is refused at load with the remedy named: pass `--unlisted-region` on the command line. |
| `egress` | `run --egress` | Request the managed internet connector; false does not isolate networking. |
| `egress-network-connectors` | `run --egress-network-connector` | Existing VPC connector ARNs. Explicit flags replace the configured list; conflicts with `egress`. |
| `deny-egress` | `run --deny-egress` | Set advisory proxy variables; does not isolate networking. |
| `shell` | `run --shell` | Launch shell-capable, so `microvm shell` can attach later. |
| `auto-resume` | `run --auto-resume` | Let the platform resume a suspended VM on an incoming request. |
| `max-idle-sec` | `run --max-idle-sec` | Suspend after this much inbound-traffic idleness. |
| `suspended-sec` | `run --suspended-sec` | Terminate after this long suspended. |
| `max-duration-sec` | `run --max-duration-sec` | Hard ceiling on the VM’s life. Refused outside 1 through 28800, eight hours being the platform’s ceiling. |
| `[env]` | `run --launch-env` | The launch environment, as a table. Merged per key, with a `--launch-env` pair winning on a shared key. A key containing `=` or an empty key is refused. |
| `artifacts` | none | Globs for `run `: which files to bring back from the VM’s synced working directory. Deliberately no flag spelling. Each glob must compile. |
| `log-group` | `run --log-group` | The CloudWatch log group build logs go to. Validated against the platform’s group-name shape; a colon usually means an ARN was pasted. |
| `log-stream` | `run --log-stream` | A stream-name prefix inside `log-group`, which it requires. Capped at 495 characters and refused when it carries `:` or `*`. |
`binary` resolves relative to the file because `--config /repo/microvm.toml` from another directory is the flag’s flagship case, and a `target/agentd` resolved against wherever the caller stands is either a miss or a different binary that happens to share the name. Two Windows path shapes that mean two things at once, a rooted path with no drive and a drive with no root, are refused rather than guessed.
## 3. Which source wins
[Section titled “3. Which source wins”](#3-which-source-wins)
A typed flag beats the file, and the file beats the built-in default. “Typed” is read off the parse rather than off the value, so `--memory 2048` overrides a file that says `4096` even though 2048 is also the default. The merge happens in one place and its outcome is on the `run` envelope as `resolvedConfig`: each knob’s winning value and the source it came from, `flag`, `config`, `env`, or `default`. `env` appears only on the region, the one knob whose chain continues past the file into `$AWS_REGION` and `$AWS_DEFAULT_REGION`. `configPath` names the file that was read.
The `[env]` table merges per key, so a project pinning `RUST_LOG` is not discarded because you passed `--launch-env CI=1`; the flag wins its own key and the rest of the table survives. One pairing rule: a typed `BINARY` positional with no typed `--image` suppresses the file’s `image`, because `run` builds exactly when the merged image is absent, and a file that silently won that pair would run your tests against a stale pinned image. A directory positional does not suppress it, because sync mode launches and the pinned image is exactly what `run .` wants.
## 4. What the loader refuses
[Section titled “4. What the loader refuses”](#4-what-the-loader-refuses)
Unknown keys are refused by name. A typo silently ignored is a config you believe is applied and is not; `memroy = 4096` launching a 2 GB VM is the failure this closes. A value outside the matching flag’s domain is refused with the flag’s own vocabulary, so `memory = 1500` cannot load for the same reason `--memory 1500` cannot parse. Every domain violation is reported at once rather than first-wins, because the file arrives as a unit.
A refused file is `ERR_CONFIG` (exit 15), its own row rather than `ERR_INVALID_ARG`, because the remedies differ: an invalid argument is fixed by editing the command line, and a broken config file is fixed by editing, or `--no-config` bypassing, a file the invocation may never have named. The refusal is local, before any billable call.
## 5. Validate it with doctor
[Section titled “5. Validate it with doctor”](#5-validate-it-with-doctor)
```bash
1
microvm doctor --config microvm.toml
```
`doctor` validates the file through the same loader `run` uses, and the config check is its first line: fatal on a broken file, an advisory pass on an absent one. The two commands cannot disagree about a file, because there is one loader.
## 6. The shipped examples
[Section titled “6. The shipped examples”](#6-the-shipped-examples)
Each example directory carries a `microvm.toml` that parses through the real loader, and a test pins that the coding-agents file still does. [coding-agents-on-bedrock](https://github.com/laithalsaadoon/microvms-agentd/blob/main/examples/coding-agents-on-bedrock/microvm.toml) pins `memory = 1024` with the sizing rule in its comments; [code-server-remote-dev](https://github.com/laithalsaadoon/microvms-agentd/blob/main/examples/code-server-remote-dev/microvm.toml) sets `egress`, `shell`, `auto-resume`, `max-idle-sec`, and an `[env]` table; [s3-prefetch-at-build](https://github.com/laithalsaadoon/microvms-agentd/blob/main/examples/s3-prefetch-at-build/microvm.toml) pins `egress = false` to prove a point. [Run a project through a VM](/microvms-agentd/learn/tutorial/run-a-project/) is where `artifacts` earns its place, and [`run`](/microvms-agentd/reference/commands/run/) lists every flag the file persists.
# Debug a failed build
> Where a failed image build's reason lives, how to read the build log, what a wedged image looks like, and which failures the client now refuses before the billable call.
```bash
1
microvm doctor --binary ./agentd # every prerequisite, and the binary's architecture
2
microvm logs # the build log group and the command that reads it
3
microvm history # what was asked, and what the platform said back
```
A server-side build cycle is minutes long and its error surfaces point away from their causes. At the end of this page you will know which surface carries a build’s reason, how to read the log, what `ERR_BUILD_WEDGED` means for the image name, and which of these failures the client refuses locally now.
## 1. Read the exit code and the envelope first
[Section titled “1. Read the exit code and the envelope first”](#1-read-the-exit-code-and-the-envelope-first)
Every failure envelope carries a stable `code`, an `exitCode` that matches `$?`, a `finding` naming the section of [Platform](/microvms-agentd/internals/platform/) that measured the behavior, and `suggestions`, with the first suggestion the one most likely to help. Under a human rendering the first line is `error ERR_*: `, then `see docs/PLATFORM.md, ''`, then `hint:` lines. [Exit codes](/microvms-agentd/reference/exit-codes/) lists every row with its meaning.
The codes a build produces:
* `ERR_PRECONDITION` (exit 12): a prerequisite is missing; run `microvm doctor`.
* `ERR_INVALID_ARG` (exit 2): the request was refused locally, before any AWS call. The Dockerfile checks below land here.
* `ERR_BUILD_WEDGED` (exit 6): the image build was never scheduled, the `clientToken` replay signature.
* `ERR_LAUNCH_DIED` (exit 7): the MicroVM reached a terminal state before `RUNNING`; read `stateReason`.
* `ERR_CREDENTIALS` (exit 4): an identity is wrong or absent, and waiting will not fix it. An `AccessDeniedException` with a null message is the unsupported-region signature, and the suggestion says to check `--region` first.
## 2. Before building again
[Section titled “2. Before building again”](#2-before-building-again)
`microvm doctor` checks every prerequisite and says which one is wrong, in diagnosis order: the config file, the region, whether the credential chain resolves, the bucket and roles by name, whether the Terraform stack is applied (it asks `terraform output`, because a destroyed stack leaves `terraform.tfstate` behind with an empty resource list), the managed bases, and with `--binary` the daemon’s architecture last, because that is the failure that costs a full build cycle.
Then build the Dockerfile locally under arm64 with `docker buildx build --platform linux/arm64`. A `dnf` typo is free to find there. [Write a guest Dockerfile](/microvms-agentd/learn/operations/write-a-guest-dockerfile/) has the rules.
## 3. Where the reason lives
[Section titled “3. Where the reason lives”](#3-where-the-reason-lives)
A failed build’s `stateReason` lives on the build record, and nowhere else. `GetMicrovmImage` reports `CREATE_FAILED` and structurally cannot say why; `ListMicrovmImageVersions` reports `FAILED` with a null reason; `ListMicrovmImageBuilds` carries the sentence. Expect a list: each failed version had two builds, one per chipset generation, with the same reason. `GetMicrovmImage`’s `latestFailedImageVersion` names which version to ask about.
Two observed reasons, worth reading for how much they vary:
```text
1
The container image build failed. (a RUN exiting non-zero)
2
Ready hook invocation timed out after PT5M (a daemon that never became ready)
```
Read `snapshotBuild`’s shape beside the reason. Absent means the Dockerfile broke before anything installed. `codeInstallSizeInBytes` alone with no snapshots means code installed and the daemon never became ready, which points at the daemon or its `CMD` rather than at the build. The first reason names the failure without naming the cause inside the container, so the build log group is the only place the failing command’s own output appears.
## 4. Read the build log
[Section titled “4. Read the build log”](#4-read-the-build-log)
```bash
1
microvm logs
```
`logs` names an image’s build log group and prints the `aws logs tail` command that reads it. The group is `/aws/lambda-microvms/`, created by the service, and its envelope carries `logGroup`, `tailCommand`, and `tailRequires`: the printed command needs AWS CLI v2, because `aws logs tail` does not exist in v1, and an identity granted the Terraform stack’s `logs_read_policy_arn`. `lines` is explicitly `null`, never `[]`, because the CLI did not read the group itself.
An empty group beside every failure reading `reason=unknown` is the IAM-prefix signature. The build role must be granted `/aws/lambda-microvms/*`; a policy on the plausible `/aws/lambda/microvms/*` produces builds with no logs at all, and the caller’s own policy is what discarded the evidence.
One build is three VMs writing three streams: docker-build, then a snapshot pass per chipset generation, and the snapshot VMs are the ones that start the app, so the daemon’s own startup lines land there. `--log-group` on `run` and `build` sends the logs to a group of your own, and `--log-stream` names a stream prefix inside it; the client appends `/<16 hex>` per build attempt, because the wire member is an exact stream name and a fixed one would collapse every build’s streams into one. The resolved exact name is on the `build` envelope as `logStream`. A configured group still has to be somewhere the build role can write.
## 5. `ERR_BUILD_WEDGED` and the `clientToken`
[Section titled “5. ERR\_BUILD\_WEDGED and the clientToken”](#5-err_build_wedged-and-the-clienttoken)
A `clientToken` is a permanent idempotency key. After an image is deleted and recreated under the same name with a token derived from that name, the service replays the original create as a no-op: the image sits in `CREATING` with its builds never scheduled, and `ListMicrovmImageBuilds` shows every build `PENDING` with `updatedAt` never advancing past `createdAt`. An image in `CREATING` cannot be deleted, and its only version cannot be deleted because it is the last one. Two images were wedged this way for roughly fifteen hours before the service timed them out.
Waiting does not help. The suggestion on the envelope is the remedy: record the identifier and build under a fresh `--name`. The client’s defaults are shaped by this trap. `run` and `build` default to a per-invocation image name, and `--reuse` keys the name to a hash of the build inputs, so unchanged inputs reuse their image and changed inputs get a fresh name and a fresh build. Recreating an image under a previously used fixed name can also serve a stale snapshot, which is the same hazard class.
## 6. The green log that fails
[Section titled “6. The green log that fails”](#6-the-green-log-that-fails)
A guest whose `AGENTD_PORT` disagrees with the create call’s port fails the build with `CREATE_FAILED`, a fully populated and green build log, every docker layer succeeding, the daemon’s own `agentd listening` line with the wrong address, and no error line anywhere. The build-time hooks are dialled on the create call’s port, so a daemon listening elsewhere answers none of them. An unset `AGENTD_PORT` is the same failure when the client has moved off 9000, with nothing in the Dockerfile to point at. Both halves are refused locally now, before the billable call, so a build that reaches the service does not have this failure.
## 7. The daemon that never became ready
[Section titled “7. The daemon that never became ready”](#7-the-daemon-that-never-became-ready)
`Ready hook invocation timed out after PT5M` after a long build, saying nothing about architecture, is a host-architecture daemon binary. MicroVMs are ARM64-only, so an x86-64 `CMD` cannot exec and surfaces only as the hook never answering. `microvm doctor --binary ` reads the ELF header and answers before the build starts; a script or wrapper is caught as not an ELF binary. The same reason appears when an `ENTRYPOINT` swallows the `CMD`, when a start wrapper exits before handing off to the daemon, and when a prefetch or install at app start runs past the five-minute observed ceiling.
## 8. The VM that died before `RUNNING`
[Section titled “8. The VM that died before RUNNING”](#8-the-vm-that-died-before-running)
`ERR_LAUNCH_DIED` means a lifecycle hook failed after the image built. `GetMicrovm`’s `stateReason` is the only evidence that outlives the VM, and the client puts the state and the reason both in the message. The envelope’s `finding` points at the `runHookPayload` measurements, and the suggestion points at `microvm logs `, because the hook wrote to the build log group. A connection refused a second or two after the VM reaches `RUNNING` is a different thing: the endpoint proxy is not wired the instant the state flips, and that one is `ERR_RETRYABLE`.
The [debugging guide](/microvms-agentd/internals/insights/debugging-guide/) has the full failure-mode index, with a citation for every row.
# Drive it from a script or an agent
> The one-envelope rule, the stable codes and exit integers a script branches on, the manifest that describes the surface, the streaming exception, and the token-lean output for a consumer paying per token.
```bash
1
microvm manifest # the whole surface, as JSON, with no credentials
2
microvm --json run --exec "make test" # one envelope on stdout, progress on stderr
3
microvm --dense ls # tab-separated, token-lean
```
Every command is built to be driven by something that is not a person at a terminal. At the end of this page you will parse every command the same way, branch on stable codes, discover the surface without parsing help text, and know the one place stdout carries more than one object.
## 1. One envelope on stdout
[Section titled “1. One envelope on stdout”](#1-one-envelope-on-stdout)
`--json` is one of the three [global flags](/microvms-agentd/reference/#3-global-flags), so `microvm --json ls` and `microvm ls --json` are the same invocation. It emits the typed JSON envelope on stdout instead of human output and wins over every other format, including an interactive terminal. Progress always goes to stderr, and `--quiet` suppresses progress but never a warning, so a leaked resource is still reported in quiet mode.
A success envelope carries `status`, `apiVersion`, `type`, and `data`. `type` is the discriminant to branch on first, and each command’s `data` keys are published in the manifest as `responseKeys`:
```json
1
{
2
"status": "ok",
3
"apiVersion": "1",
4
"type": "microvm.run",
5
"data": {}
6
}
```
A failure envelope carries `status`, `apiVersion`, `error`, `code`, `exitCode`, `finding`, `suggestions`, and `data`. Every field is always present, so a consumer never guards against a missing key:
```json
1
{
2
"status": "error",
3
"apiVersion": "1",
4
"error": "human readable, may be reworded between releases",
5
"code": "ERR_PROTOCOL",
6
"exitCode": 5,
7
"finding": "",
8
"suggestions": [],
9
"data": { "kind": "Conflict" }
10
}
```
Branch on `code`, never on `error`. `data` carries partial results on the failure path, most importantly `leaked`, the identifiers a teardown could not delete. `data.kind` carries the daemon’s own status name when the exit code is coarser than the failure: several wire kinds collapse onto `ERR_PROTOCOL`, because a shell branching on `$?` cannot act differently on a 400 than on a 409, and a consumer that can reads `data.kind`. A request rejected locally reports no `data.kind`, because nothing reached the daemon. `apiVersion` is bumped when a field’s meaning changes, never when a command is added. [Envelope](/microvms-agentd/reference/envelope/) and [Response types](/microvms-agentd/reference/response-types/) are the generated references.
## 2. The exit codes
[Section titled “2. The exit codes”](#2-the-exit-codes)
`exitCode` in the envelope matches `$?`, and the table is append-only because consumers branch on it. Rows are split by what the caller should do next. `ERR_RETRYABLE` (3) means run the identical command again; `ERR_CREDENTIALS` (4) means fix an identity, and waiting will not help. `ERR_TIMEOUT` (10) means a client-side deadline elapsed and the VM and the exec are untouched, so a poll is safe. `ERR_EXEC_FAILED` (13) means the sandbox worked and your command exited non-zero, which is the one non-zero exit that says nothing is wrong with the platform, the credentials, or the CLI; a CI caller needs to tell “your tests failed” from “we never got a VM”. `ERR_PRECONDITION` (12) means run `microvm doctor`. `ERR_NAME_TAKEN` (14), `ERR_CONFIG` (15), and `ERR_SYNC` (16) are refused locally with no AWS call made. `ERR_UNEXPECTED` (1) is a bug in this CLI and never the platform. clap’s own usage errors exit 2, which is `ERR_INVALID_ARG`, so a caller reading `$?` sees the same number either way. [Exit codes](/microvms-agentd/reference/exit-codes/) has every row with its `finding`.
## 3. Discover the surface
[Section titled “3. Discover the surface”](#3-discover-the-surface)
`microvm manifest` emits the whole command surface, its exit codes, and its envelope schema, generated from the CLI’s own argument tree rather than written down. It is always JSON, and it needs no credentials, no region, and no network, so it doubles as a liveness check. Each command’s entry carries its parameters with `type`, `default`, `choices`, `required`, and `positional`, its `responseType` and `responseKeys`, and for `exec` an `alternateResponse` naming when it applies. The `conventions` list is the contract in prose. `microvm manifest --dense` prints one line per command with its parameters.
The agent layer’s two commands follow the same rules: `agent-up --json` is one `microvm.agent` envelope whose `agents` list carries each installed agent’s model and the exact `headlessCommand` `agent-prompt` will run, and `agent-prompt --json` is a `microvm.agent.prompt` envelope, `exec`’s keys plus `agent` and `model`. An agent that declines a task and exits 0 is an `ok` envelope, so read the effect back with `exec` rather than trusting the prompt’s exit alone.
`microvm constants --emit-json` emits every service constraint this client believes, unwrapped by an envelope, for the drift gate that compares them against the pinned service model.
## 4. The one exception: `exec --stream`
[Section titled “4. The one exception: exec --stream”](#4-the-one-exception-exec---stream)
`exec --stream` is the one invocation that writes more than one object to stdout. Under `--json` it emits NDJSON, one event object per line, and then the envelope as the final line:
```json
1
{"event":"output","stream":"stdout","offset":0,"bytes":12,"text":"hello world\n","lossy":false}
2
{"event":"exit","exitCode":0,"signal":null,"truncated":false,"writersMayBeAlive":false,"offset":12}
3
{"status":"ok","apiVersion":"1","type":"microvm.exec.stream","data":{"execId":"x-1","events":2,"bytes":12,"nextOffset":12,"gaps":0,"exitCode":0,"truncated":false}}
```
Three things keep the two contracts distinguishable. The discriminant differs: a streamed exec’s final envelope has `type` `microvm.exec.stream`, so a consumer learns which parse applies from the field it already reads first. The manifest publishes it, as `exec`’s `alternateResponse` with `when: "--stream"`. And the envelope is written compact once a stream has started, because “the last line is the envelope” is only true if the envelope is one line.
Event kinds are `output` (with `stream`, `offset`, `bytes`, `text`, `lossy`), `gap` (with `from` and `to`, the only report of lost bytes), and `exit`. The envelope’s keys summarize the stream rather than repeating it: `events` and `bytes` let a caller assert it read everything, and `nextOffset` is where `--from-offset` would resume. A stream cut before its exit event reports `exitCode: null` rather than `0`, because zero would turn a truncated stream into a passing build, and the command exits `ERR_EXEC_FAILED`. A stream that fails part-way through gets the failure envelope as its compact last line. The stream chunks are the command’s output, so they cannot go to stderr; `microvm exec --stream build.sh > log` has to write the log.
## 5. Token-lean output
[Section titled “5. Token-lean output”](#5-token-lean-output)
`--dense` is the second [global flag](/microvms-agentd/reference/#3-global-flags): token-lean output, for a consumer paying per token. It renders tab-separated text, one field per column; a dense failure is the code, then the message, tab-separated, so field one is always the code. `--json` wins over `--dense`, and `--dense --json` together emit the compact one-line JSON document rather than the pretty one. Neither depends on whether stdout is a terminal; without either, a terminal gets a human rendering and a pipe gets plain text.
## 6. Retries and idempotency
[Section titled “6. Retries and idempotency”](#6-retries-and-idempotency)
Pass your own `--exec-id` on an `exec` you may have to retry: a start carrying a known id returns the original exec without spawning a second child, so a caller whose process died between sending the start and reading the answer sends the identical start again. `ERR_RETRYABLE` means exactly that, run the identical command again. `build --reuse` makes a repeated build idempotent through a content-hash image name, and `run` defaults to a per-invocation image name because reusing one is how a `clientToken` replay wedges an image. A `--vm-name` is local state, refused with `ERR_NAME_TAKEN` before any call if a live VM holds it.
## 7. Extracting a field without jq
[Section titled “7. Extracting a field without jq”](#7-extracting-a-field-without-jq)
The shipped scripts read envelope fields with a python3 one-liner, which every machine that runs the examples already has:
```bash
1
jqr() { python3 -c "import json,sys; print(json.load(sys.stdin)['data']$1)"; }
2
LAUNCH=$(microvm run --json --keep --image "$IMAGE_ARN")
3
EP=$(echo "$LAUNCH" | jqr "['endpoint']")
4
TOK=$(echo "$LAUNCH" | jqr "['agentToken']")
5
ID=$(echo "$LAUNCH" | jqr "['microvmId']")
```
**For an agent.** Prefer `microvm manifest` to any page here wherever the two could disagree; it is generated from the source of truth. Read `type` before `data`, `code` before `error`, and `data.leaked` on every failure. Treat `ERR_EXEC_FAILED` as the workload’s result and every other non-zero exit as a problem with the run itself. Use `--exec-id` on anything you might retry, and `--dense` when you are paying per token and do not need the envelope’s structure.
[For agents](/microvms-agentd/agents/) names which surface answers which question, and the daemon’s own machine surface is `GET /v1/schema`, committed as the [wire schema](/microvms-agentd/reference/wire-schema/).
# Embed agentd in your own image
> Append the daemon to an arbitrary task image so your own harness can drive it over the published wire protocol, and the pointers a harness client needs.
```bash
1
microvm dockerfile --workdir /workspace > Dockerfile
2
# edit: insert your RUN layers between the chmod line and the ENV lines
3
microvm build --dockerfile Dockerfile --name my-task-image
```
The platform has no exec API: a MicroVM exposes one HTTPS endpoint and forwards it to whatever the image’s `CMD` is listening on. Every harness that runs commands inside a VM therefore ships a daemon in its task image, and before agentd each one wrote its own. At the end of this page you will have an image with agentd as its `CMD` and know which surfaces your client reads. The reasoning behind each rule is in [Embedding](/microvms-agentd/internals/embedding/).
## 1. The recipe
[Section titled “1. The recipe”](#1-the-recipe)
`microvm dockerfile` prints the stanza that wraps a base image with agentd, emitted by the same generator the default `microvm build` uses, so appending your layers to it is the default build plus your layers. Any task image is the same shape: take the stanza, add the layers your workload needs, keep the daemon lines intact. [Write a guest Dockerfile](/microvms-agentd/learn/operations/write-a-guest-dockerfile/) walks the stanza line by line.
## 2. The lines that must survive your edits
[Section titled “2. The lines that must survive your edits”](#2-the-lines-that-must-survive-your-edits)
`ENTRYPOINT []` plus `CMD ["/agentd"]` is the deployment invariant the trust boundary rests on: no task workload runs before the platform’s run hook lands, and an omitted `cwd` inherits the image `WORKDIR`. The `FROM` must match the managed base’s registry ref, and a `WORKDIR` is required because the managed base declares none. Both are enforced by the client before any AWS call. No secret goes in the image; per-VM credentials travel through `runHookPayload` at launch.
## 3. What your client implements
[Section titled “3. What your client implements”](#3-what-your-client-implements)
The full route table and request shapes are in [Protocol](/microvms-agentd/internals/protocol/), and the same contract is served as JSON Schema at `GET /v1/schema` on any running daemon, unauthenticated, so a client can fetch it before it holds a token. The committed copy is the [wire schema](/microvms-agentd/reference/wire-schema/). The shape of a client, in one line each:
* **Bootstrap.** The platform delivers your `runHookPayload` to the daemon’s `/run` hook, and agentd installs `agent_token` from it once. Until then every control route answers 503, so a client can tell “not yet bootstrapped” from “broken”.
* **Auth.** Every `/v1/` route except `/v1/health` and `/v1/schema` takes `Authorization: Bearer `.
* **Exec.** The client mints the `exec_id`, which is what makes a retry safe: a start carrying a known id returns the original exec without spawning a second child. Poll is read-only; ack releases the buffered output; output lives until the ack.
* **Streaming.** Server-sent events from a byte cursor, with an explicit `gap` event when a reattach falls past the retained window and a typed `exit` event that distinguishes a finished command from a cut connection.
* **Files.** One file per request, streamed, or a directory tree as tar with confined extraction.
* **Health.** `GET /v1/health` is unauthenticated and reports version, bootstrap state, disk pressure, and identity repair.
* **The proxy.** Every request crosses the platform’s endpoint proxy, which wants two headers, and the token it wants is capped at sixty minutes, so mint inside the request path and refresh well under the ceiling. A detached exec is polled and acked under the next token without loss.
* **The keepalive is yours.** Idleness is measured by inbound traffic through the proxy, which terminates outside the VM, so an outside poll of `/v1/health` resets the timer and a guest-side request cannot.
## 4. Get the daemon binary yourself
[Section titled “4. Get the daemon binary yourself”](#4-get-the-daemon-binary-yourself)
The CLI provisions its own version’s release asset. For a build you manage, fetch and verify it the same way:
```bash
1
gh release download --repo laithalsaadoon/microvms-agentd --pattern agentd
2
gh attestation verify agentd --repo laithalsaadoon/microvms-agentd
3
chmod +x agentd
```
Pass it as the positional argument to `build` or `run`, or as `$MICROVM_AGENTD`. `microvm doctor --binary ./agentd` checks that it is the ARM64 build before a host-architecture mistake costs a build cycle. [Harness capabilities](/microvms-agentd/internals/harness-capabilities/) maps what the hand-rolled daemons needed onto what agentd covers.
# Prefetch S3 content at image build
> Bake an S3 prefix into the image snapshot during the build, so every VM launched from it starts with the data on disk and makes no S3 call at all.
```bash
1
# from the repo root, with the first-run prerequisites in place:
2
PREFETCH_URI=s3://my-bucket/models/ bash examples/s3-prefetch-at-build/run.sh
3
# public bucket, zero credentials:
4
PREFETCH_URI=s3://a-public-bucket/prefix/ PREFETCH_NO_SIGN=1 bash examples/s3-prefetch-at-build/run.sh
```
A fresh VM’s first S3 call carries a measured five-to-ten-second penalty, and a workload that starts by pulling model weights, a dataset, or a toolchain pays that plus the whole transfer on every launch. This recipe moves both to image-build time. At the end of this page you will have an image whose snapshot already holds the prefix, and a VM launched from it without `--egress` that reads the data from disk.
## 1. Why it works
[Section titled “1. Why it works”](#1-why-it-works)
An image build boots the image in a snapshot VM, calls the build-time hooks against the daemon, and captures the memory and disk snapshot only after they answer. So anything that completes before the daemon answers those hooks is inside the snapshot. The wrapper here runs `aws s3 sync` first and only then hands the process to agentd, strictly ordered: the daemon cannot answer a hook until the sync is done, and the snapshot cannot be captured until the daemon answers.
Two shapes of the same rule, observed in [Platform](/microvms-agentd/internals/platform/). The model allows build hooks an hour, and an observed failure reads `Ready hook invocation timed out after PT5M`, so plan the transfer to fit well inside five minutes and measure before assuming the full hour is reachable. And one build runs the snapshot pass once per chipset generation, so the prefetch downloads twice per image; a launch restores the snapshot rather than re-running `CMD`, so it never prefetches, which is the entire point.
## 2. The Dockerfile pieces
[Section titled “2. The Dockerfile pieces”](#2-the-dockerfile-pieces)
The base and the daemon lines are the same as every other guest Dockerfile. AWS CLI v2 runs the prefetch, and AL2023 packages it as `awscli-2`. The URI and the sign mode are `ENV` rather than `ARG` because they are read at app start, in the snapshot VM, and because baking them in is what keys the image to its content:
```dockerfile
1
RUN dnf install -y awscli-2 \
2
&& dnf clean all
3
4
ENV PREFETCH_URI=""
5
ENV PREFETCH_NO_SIGN="0"
```
The start wrapper is materialized with `printf`, because the build context `microvm build` uploads carries exactly the Dockerfile and the daemon binary, so there is no file beside them to `COPY`:
```dockerfile
1
RUN printf '%s\n' \
2
'#!/usr/bin/env bash' \
3
'set -euo pipefail' \
4
'if [ -n "${PREFETCH_URI:-}" ]; then' \
5
' echo "prefetch: ${PREFETCH_URI} -> /opt/prefetch"' \
6
' args=(s3 sync --no-progress "${PREFETCH_URI}" /opt/prefetch)' \
7
' if [ "${PREFETCH_NO_SIGN:-0}" = "1" ]; then args+=(--no-sign-request); fi' \
8
' aws "${args[@]}"' \
9
' chown -R 1000:1000 /opt/prefetch' \
10
' du -sh /opt/prefetch' \
11
'else' \
12
' echo "prefetch: PREFETCH_URI is empty, skipping"' \
13
'fi' \
14
'exec /agentd' \
15
> /start.sh && chmod 0755 /start.sh
```
The `CMD` becomes `["/start.sh"]` with `ENTRYPOINT []` kept. `run.sh` rewrites the two `ENV` lines with your values before building, so each distinct URI is a different Dockerfile hash, a different image name, and a fresh build, and re-running with the same URI reuses the already-prefetched image in seconds.
## 3. Fail loud, never empty
[Section titled “3. Fail loud, never empty”](#3-fail-loud-never-empty)
The wrapper is `set -e`. A failed sync means agentd never starts, the ready hook times out, and the build fails with a named reason instead of producing an image whose snapshot silently lacks the data. The failure signature is `stateReason: Ready hook invocation timed out after PT5M` on the build record, and the wrapper’s own log lines are in the build log group; `microvm logs ` prints the `aws logs tail` command that reads them. An empty `PREFETCH_URI` skips the prefetch with a log line, so the Dockerfile also builds as checked in, and `run.sh` refuses to run without a URI so the demo cannot pass by fetching nothing.
## 4. Credentials at build time
[Section titled “4. Credentials at build time”](#4-credentials-at-build-time)
The prefetch runs inside the build’s snapshot VMs, under whatever network and credentials the build environment provides. What the repository has established: the build’s docker-build VM has outbound network, because every `dnf install` in the examples runs there. Whether the snapshot VMs have the same egress, and whether the build role’s credentials are visible to the app process there, has not been measured.
`PREFETCH_NO_SIGN=1` for a public bucket is the zero-assumption mode. For a private bucket, the AWS CLI walks its default chain: grant the build role read on the prefix and try it, and if the build fails with the `PT5M` signature and the wrapper’s log shows a credential or network error, that is a real finding about the snapshot VMs. The conservative fallback is to run the sync as a docker `RUN` step instead, which executes in the docker-build VM where outbound network is demonstrated and reaches the launched VM through the same disk snapshot.
## 5. Do not prefetch secrets
[Section titled “5. Do not prefetch secrets”](#5-do-not-prefetch-secrets)
The snapshot is shared by every VM launched from the image, so anything prefetched into it is readable by all of them. Prefetch data, models, and toolchains; deliver tokens and keys at launch through `runHookPayload` or `microvm cp` after bootstrap, the way [Run coding agents on Bedrock](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) does.
## 6. Prove it at launch
[Section titled “6. Prove it at launch”](#6-prove-it-at-launch)
```bash
1
microvm run --json --keep --image "$IMAGE_ARN" --config microvm.toml
2
microvm exec "du -sh /opt/prefetch && ls -la /opt/prefetch | head -20" \
3
--user 1000 --group 1000 "${ATTACH[@]}" --timeout 60
```
The example’s `microvm.toml` pins `egress = false` and `memory = 512`. The data is already present, but `egress = false` does not disable internet access; see [Networking](/microvms-agentd/internals/networking/). Running the check as uid 1000 also proves the wrapper’s `chown` did its job.
## 7. Rebuilds and cost
[Section titled “7. Rebuilds and cost”](#7-rebuilds-and-cost)
The prefetched bytes live in the image snapshot, which bills storage with a one-week minimum retention, and every launch pays a snapshot read scaled by its size. So prefetch what the workload reads at startup, and never the whole bucket. Delete retired images with `aws lambda-microvms delete-microvm-image`. [Read the cost report](/microvms-agentd/learn/operations/read-the-cost-report/) has the snapshot line items.
# Read the cost report
> What each line of a cost report means, where the rates come from, why a total may read "at least", and how to plan a run with microvm cost before spending anything.
```bash
1
microvm cost --estimate --memory 1024 --running-sec 1800 --image-gb 2
2
microvm cost --estimate --compare --memory 2048 --hold-sec 28800 --cycles 4
3
microvm cost --estimate --memory 2048 --running-sec 3600 --max-cost 1.50 --on-breach abort
```
Every `run` reports a cost estimate, and `microvm cost` produces the same report over durations you supply. At the end of this page you will know what each figure is built from, what it means when a total says “at least”, and how to put a budget on a run before it launches.
## 1. Where the figures come from
[Section titled “1. Where the figures come from”](#1-where-the-figures-come-from)
Rates are pinned, dated, and per region, and only the ARM rates apply, because the service’s architecture enum has one member. Dollar figures are estimates derived from published rates, never an invoice. `mise run live:rates` checks the pinned table against the AWS Pricing API, and the report carries a staleness note when the pinned date is old.
Anything the engine cannot price is reported as unpriced with a reason, and an unpriced line omits `usd` rather than reporting zero. The server-side image build is the usual case: AWS has not published whether a build bills as compute, so `--build-sec` puts an unpriced line on the report.
## 2. What bills
[Section titled “2. What bills”](#2-what-bills)
Compute bills per second while the VM is `RUNNING`, as separate vCPU and memory line items. The baseline you request with `--memory` is the floor you pay for every running second, the VM is provisioned at four times it from the start, and usage above the baseline bills by what is consumed. Idle time while `RUNNING` bills at baseline, so suspension is the only way to stop paying.
Snapshots bill three ways. Image storage bills per GB-hour with a one-week minimum retention, so a 2 GB image deleted sixty seconds after creation still bills about a week of storage. A suspended VM pays snapshot storage only, about $0.16 a month for a 2 GB VM against roughly $100 a month left running at baseline. Each suspend/resume cycle pays a snapshot write plus a read, about $0.011 for a 2 GB VM, so the thing to avoid is cycling constantly; a long suspension is cheap.
Rates vary by region. `us-east-2` and `us-west-2` match `us-east-1` on every line; `eu-west-1` and `ap-northeast-1` are higher, most of all on the snapshot dimensions, so a design that leans on a suspended pool should price in its own region. Data transfer bills at standard AWS rates and is not on the report. The measurements are in [Platform](/microvms-agentd/internals/platform/), under “What actually costs money”.
## 3. A total may be a lower bound
[Section titled “3. A total may be a lower bound”](#3-a-total-may-be-a-lower-bound)
A total over any unpriced line is a different kind of total. It renders as `at least $X`, and under `--json` the budget verdict carries `basis: "lower-bound"` where a fully priced report carries `basis: "exact"`. A verdict from a lower bound says so in both directions: a breach detected from a floor has already been exceeded by an unknown margin, and an under-budget floor proves nothing, so the text says which lines are unpriced beside the figure.
## 4. Plan before you spend
[Section titled “4. Plan before you spend”](#4-plan-before-you-spend)
`--estimate` treats the durations as a plan rather than as timings, and every duration on such a report is labelled projected, so an estimate cannot print as a report of something that ran. `--running-sec`, `--suspended-sec`, and `--build-sec` are the phases; `--image-gb` adds storage with its one-week minimum retention; `--cycles` counts suspend/resume cycles, each paying a snapshot write plus a read.
`--compare` also prints running versus suspended for the same hold, with the break-even, over `--hold-sec` (default 3600).
`--max-cost` is a budget in USD the report’s total is checked against, and `--on-breach` says what a breach does: `warn` warns and exits 0, `abort` aborts with `ERR_PRECONDITION` (exit 12). The pair is required together, because whether a breach of a lower-bound total should stop a script is the caller’s judgement and has no default. Under `--json` the `budget` key carries `maxUsd`, `onBreach`, `basis`, `breached`, and `overageAtLeastUsd`.
## 5. Size for the bill
[Section titled “5. Size for the bill”](#5-size-for-the-bill)
`--memory` selects a size class, and the guest reports the class’s peak in `/proc/meminfo`:
| Baseline (billed while running) | Peak (provisioned ceiling) |
| ------------------------------- | -------------------------- |
| `512` MiB, 0.25 vCPU | 2 GB, 1 vCPU |
| `1024` MiB, 0.5 vCPU | 4 GB, 2 vCPU |
| `2048` MiB, 1 vCPU (default) | 8 GB, 4 vCPU |
| `4096` MiB, 2 vCPU | 16 GB, 8 vCPU |
| `8192` MiB, 4 vCPU | 32 GB, 16 vCPU |
For peaky workloads such as builds, test runs, and agent sessions, pick a low minimum and let the peaks ride the always-present headroom, which bills only by what is consumed. Guest swap is absent, so pressure past the peak goes straight to the OOM killer. The value is also a `microvm.toml` key, `memory`, validated against the same closed set.
## 6. On the run envelope
[Section titled “6. On the run envelope”](#6-on-the-run-envelope)
A `run` envelope carries the report under `cost`, and the human rendering ends with a `cost:` line. `buildSeconds` and `runningSeconds` beside it are the measured phases the report was built from. Because the image snapshot’s one-week minimum applies whether or not you keep the image, reusing it with `--image` is the economical habit; [your first run](/microvms-agentd/learn/tutorial/first-run/) shows the loop.
# Recover a leaked VM
> What the local ledger says you left behind after an interrupted run, how to terminate it by id, and how to ask the account directly instead of trusting a teardown message.
```bash
1
microvm ls # what this CLI created and could not confirm it deleted
2
microvm history # what was asked of one VM and what the platform reported
3
microvm terminate --wait # stop paying
4
mise run live:verify-clean # ask the account, independently of the code that cleaned up
```
Teardown reporting success and the account being clean are different questions. At the end of this page you will know what the ledger says you left behind, how to release it, and how to confirm the account is clean without trusting any message that says so.
## 1. Ask the ledger
[Section titled “1. Ask the ledger”](#1-ask-the-ledger)
```bash
1
microvm ls
```
`ls` lists what this CLI created and could not confirm it deleted. It reads the local ledger rather than asking AWS, deliberately: the question it answers is “what did I leave behind”, and the resources worth asking about are the ones a killed process never got to report, which no listing call can attribute back to a command that died. Each run’s ledger is one JSON file under `~/.microvm/runs` (or `--state-dir`, or `$MICROVM_STATE_DIR`), written before each delete is attempted, so the identifiers survive a process that died inside the call. A run with a non-empty `leaked` list is marked as an alarm, the human rendering prints `LEAKED (still billing): `, and the trailing count reads “N run(s), M with something still billing”.
That is a ledger, not the account, and the output says so: every `ls` envelope carries `source: "local-ledger"`, and the header names the state directory it read. An entry can outlive the resource it names, because the ledger records what this CLI could not confirm it deleted, which is a different question from what exists. `microvm ls --remote` asks that second question through the same control plane every other command uses, listing the account’s MicroVMs and images and judging each identifier in an entry’s `leaked` list: the entry is `live` when one of them is still listed alive, `gone` when every one is a MicroVM id or image ARN the listings no longer carry, and `unjudged` when one of them is something those two listings can’t see at all, a `/aws/lambda-microvms/...` log group above all. A record from another region is `unjudged` too. Whatever is alive that no entry names is reported under `remote.unknownToLedger`, which is how a sibling client’s leftovers show up. `microvm ls --remote --prune` removes the files of `gone` entries and lists them in `pruned`; a `live` or `unjudged` record is never removed, because for a leaked log group the ledger file is the only pointer there is.
`--watch` re-reads the ledger on an interval until Ctrl-C, with `--interval-sec` (default 2) and `--max-refreshes`. It makes zero platform calls, and in particular never polls `/v1/health`, which is the call that resets a VM’s idle timer, so watching keeps nothing alive and bills for nothing. `--watch` and `--remote` do not combine.
Run `ls` before anything else touches the account after an interruption. A `leaked` list is both a bill and a clue.
## 2. Read one VM’s history
[Section titled “2. Read one VM’s history”](#2-read-one-vms-history)
```bash
1
microvm history
```
`history` prints what was asked of one MicroVM and what the platform reported back, from the local per-VM history appended by `run`, `exec`, `suspend`, `resume`, and `terminate`. The record survives terminate on purpose, because a caller attesting over a run needs it precisely after the VM is gone. It shows what the daemon and the control plane reported, never what a process inside the guest did between execs.
## 3. Terminate by id
[Section titled “3. Terminate by id”](#3-terminate-by-id)
```bash
1
microvm terminate --wait
2
microvm terminate --delete-image --image-identifier --image-name
```
`terminate` takes the MicroVM id as its positional argument, or a registered name, which is resolved locally. `--wait` waits for `TERMINATED` rather than returning as soon as the call is accepted. `--delete-image` also deletes the image: for a VM this state directory launched with `run --keep`, the run record already names the image and its name, so `--image-identifier` and `--image-name` are overrides; for a VM launched elsewhere, pass them, and the CLI names the build log group, `/aws/lambda-microvms/`, which the service created and Terraform never owns. The teardown envelope carries `leaked` and `undeletedLogGroups`, so a partial success is still machine readable.
An image refuses deletion while its VM is still terminating, so one pass is sometimes not enough. Deleting the image early also saves nothing, because its snapshot has a one-week minimum retention; a leaked image is a small bill and a leaked running VM is the one to hurry for.
## 4. Ask the account, never the teardown
[Section titled “4. Ask the account, never the teardown”](#4-ask-the-account-never-the-teardown)
```bash
1
mise run live:verify-clean
2
./scripts/verify-clean.py --delete
```
`live:verify-clean` queries the account directly and is independent of the code that did the cleanup, which is the point: `terraform destroy` once reported nine resources destroyed while six service-created log groups survived, because Terraform never owned them. It reports three outcomes rather than two. A **leak** is something still costing money that nothing intends to keep: a live MicroVM, an image, a log group. **Standing** is the Terraform stack, which you may keep applied on purpose. **Pending** is a deletion still in flight, where the right response is to re-run in a minute. Exit 0 when nothing leaked, 1 otherwise. `--delete` removes the leaks and leaves the stack to `terraform destroy`, and expect to run it more than once, because an image refuses deletion while its VM is still terminating.
It only recognizes resources under this project’s own name prefixes, so anything else in the account is untouched. It needs the repository (`scripts/verify-clean.py` runs under `uv`) and credentials for the account. [Run the live suite](/microvms-agentd/learn/operations/run-the-live-suite/) puts it at the end of every billable run.
## 5. Why a VM leaks
[Section titled “5. Why a VM leaks”](#5-why-a-vm-leaks)
`run` tears down by default, and an interruption after launch is `ERR_INTERRUPTED` (exit 11): teardown ran, and any leak is named in the envelope’s `data.leaked`. A process killed before it could report leaves the ledger entry that `ls` reads. A run under `--keep` is a leak you asked for, and its name is released when the terminate is accepted, so a `--vm-name` refused with `ERR_NAME_TAKEN` (exit 14) means a live VM still holds it.
Two things that look like leaks are not. A VM suspended past its `--suspended-sec` window was terminated by the platform, and a resume is refused with `ERR_WINDOW_CLOSED` (exit 8) because there is nothing to resume. And a build log group under `/aws/lambda-microvms/` outliving `terraform destroy` is expected; it is why `verify-clean` exists.
## 6. A name registered on another machine
[Section titled “6. A name registered on another machine”](#6-a-name-registered-on-another-machine)
The registry is local, so a name registered elsewhere is unknown here and fails with `ERR_PRECONDITION` naming the directory it looked in. The MicroVM id is accepted directly by every identifier-taking command. To make the name work here, adopt the record:
```bash
1
ssh other cat ~/.microvm/runs/names/dev.json | microvm attach --from -
```
**For an agent.** Read `data.leaked` on every failure envelope before doing anything else, and run `microvm ls --remote --json` at the start of a session that follows an interrupted one. Use `remote.entries[].status` to sort the entries, then read the `leaked` list of every entry that isn’t `gone`: a `live` entry’s list names what still bills, and an `unjudged` entry’s list names what the two listings can’t see, which is where a service-created log group shows up. A `gone` entry names only MicroVM ids and image ARNs the account no longer carries, so its `leaked` list can be skipped. A leaked identifier is the remedy for a `CREATING` image or a service-created log group, because there is no second way to find them; that’s why `--prune` leaves an `unjudged` record alone. Never conclude the account is clean from a teardown envelope; `mise run live:verify-clean` is the check, and exit 1 from it means something still bills.
# Remote dev with code-server over port-forward
> Run VS Code in a browser against a MicroVM through microvm port-forward, on a named VM that suspends when you close the tab and resumes when you come back.
```bash
1
# from the repo root, with the first-run prerequisites in place:
2
bash examples/code-server-remote-dev/run.sh
3
# then open http://127.0.0.1:8080
```
The script builds an image carrying code-server on first use, launches a named VM, starts the IDE inside it, and forwards local port 8080. Re-running it reattaches to the same VM instead of launching a second one, and Ctrl-C stops the forward and nothing else. At the end of this page you will have a disposable, suspendable dev box whose idle time costs roughly snapshot storage instead of compute.
## 1. The image
[Section titled “1. The image”](#1-the-image)
`Dockerfile` starts from the platform’s `al2023-1` base pair, pinned by digest, carries the daemon exactly as the client’s default Dockerfile does, then installs code-server from its release RPM. The RPM bundles its own Node, so the image adds only git and the shell basics beside it. The version is pinned by an `ARG` default; edit it to bump, and the content-hash-keyed image name makes the edited file build fresh under a new name.
## 2. The launch: named, kept, suspendable
[Section titled “2. The launch: named, kept, suspendable”](#2-the-launch-named-kept-suspendable)
```bash
1
microvm run --keep --vm-name code-server-dev --image "$IMAGE_ARN" --config microvm.toml
```
`--vm-name` registers the name in the local registry, so everything after the launch addresses the VM as `--name code-server-dev`. `microvm.toml` carries the choices that make this a dev box rather than a batch runner:
```toml
1
memory = 2048
2
egress = true
3
shell = true
4
auto-resume = true
5
max-idle-sec = 900
6
7
[env]
8
HOME = "/workspace"
9
PATH = "/usr/local/bin:/usr/bin:/bin"
```
`memory = 2048` keeps the default baseline and its 8 GiB ceiling, because an IDE with language servers is a steadier workload than a peaky agent session. `egress = true` because git, package registries, and the extension marketplace all need outbound network. `shell = true` launches shell-capable, so `microvm shell --name code-server-dev` can open a real PTY beside the IDE. `auto-resume` and `max-idle-sec` wire the suspend loop below. The `[env]` table sets `HOME` somewhere uid 1000 can write settings and extensions, and an explicit `PATH`, because the daemon spawns execs with a minimal environment.
## 3. Start the IDE, detached and demoted
[Section titled “3. Start the IDE, detached and demoted”](#3-start-the-ide-detached-and-demoted)
```bash
1
microvm exec "code-server --bind-addr 127.0.0.1:8080 --auth none /workspace" \
2
--detach --user 1000 --group 1000 --name "$VM_NAME" --region "$REGION" \
3
--json >/dev/null
4
5
microvm exec "timeout 60 bash -c 'until echo > /dev/tcp/127.0.0.1/8080; do sleep 1; done' 2>/dev/null" \
6
--name "$VM_NAME" --region "$REGION" --json >/dev/null
```
`--detach` starts code-server and returns; the second exec waits for the listener with bash’s `/dev/tcp`, so nothing has to be installed for the wait. `--user 1000` because an IDE hands a terminal to whoever reaches it, and that terminal should not be root’s.
`--auth none` is sound here because three layers already gate the path. The local listener binds `127.0.0.1`, so nothing off your machine reaches the forward. Every request the forward relays crosses the endpoint proxy, which requires an auth token scoped to this MicroVM and port set, and `port-forward` mints and attaches it for you. Inside the guest, code-server binds loopback only. A password prompt on top would authenticate nothing the proxy token has not already authenticated.
## 4. The forward
[Section titled “4. The forward”](#4-the-forward)
```bash
1
microvm port-forward 8080:8080 --name code-server-dev
```
`port-forward` serves a guest port on localhost so a browser here reaches a server in the VM. The ports are `LOCAL[:GUEST]`, and a single number uses it on both sides. `--bind` defaults to `127.0.0.1`, deliberately; `--max-connections` stops after serving that many connections instead of running until Ctrl-C.
## 5. The suspend/resume loop is what makes this cheap
[Section titled “5. The suspend/resume loop is what makes this cheap”](#5-the-suspendresume-loop-is-what-makes-this-cheap)
Suspend is a freeze and restore. The filesystem, running processes, and the endpoint URL all survive a suspend/resume cycle, so unsaved buffers and the terminal you left open come back exactly as you left them.
The loop wires itself. An open editor tab holds a live WebSocket, which is inbound traffic, and inbound traffic resets the platform’s idle timer. So the VM stays `RUNNING` while you work, suspends about fifteen minutes after you close the tab, and, because the config sets `auto-resume`, resumes on the next request through the endpoint. Reload the tab with the forward still up, or re-run the script, and you are back. A suspended VM bills roughly snapshot storage alone.
One ceiling to plan around: the platform caps any single VM’s life at eight hours (`--max-duration-sec`, refused above 28800 before any call). This is a work-session dev box, so get work out before the ceiling with git from inside the IDE, or from outside:
```bash
1
microvm cp --tar vm:/workspace ./workspace-backup.tar --name code-server-dev
```
## 6. A shell beside it
[Section titled “6. A shell beside it”](#6-a-shell-beside-it)
```bash
1
microvm shell --name code-server-dev
```
A real PTY with job control, signals, and resize, for anything the editor’s own terminal is the wrong tool for. It works while the browser tab is closed.
## 7. Teardown
[Section titled “7. Teardown”](#7-teardown)
```bash
1
microvm suspend code-server-dev # pause it now
2
microvm terminate code-server-dev # tear it down
```
`suspend`, `resume`, and `terminate` take the MicroVM id as their positional argument, and the registered name stands in for it. The image persists deliberately, because its snapshot has a one-week minimum retention; delete it with `aws lambda-microvms delete-microvm-image` when you are done with the recipe. The measurements behind the loop, the eight-hour ceiling, and the proxy’s port-scoped tokens are in [Platform](/microvms-agentd/internals/platform/).
# Run coding agents in sandboxed MicroVMs
> Give Claude Code or Codex a project in its own VM, run a task through Bedrock, and retrieve the results.
Run Claude Code or Codex in an AWS Lambda MicroVM with a copy of your project. The agent can edit files and run tools as a non-root user in `/workspace`. Both profiles call Amazon Bedrock using a short-lived token minted from your AWS credentials.
## Start with one agent
[Section titled “Start with one agent”](#start-with-one-agent)
First [install the CLI](/microvms-agentd/learn/tutorial/install/) and [configure AWS](/microvms-agentd/learn/tutorial/first-run/#configure-aws). You need AWS CLI v2, MicroVMs permissions, the artifact bucket and build/execution roles, and Bedrock invocation permissions with access to your chosen model. With those prerequisites ready, you can start this workflow in 90 seconds. The first image build takes several minutes; later launches reuse it when its inputs are unchanged.
From your project directory:
```bash
1
microvm agent-up --vm-name review --agent claude-code --project .
2
microvm agent-prompt --name review \
3
"Review this project and write your findings to REVIEW.md."
4
microvm cp --name review vm:/workspace/REVIEW.md ./REVIEW.md
5
microvm terminate review --wait
```
The file download verifies that the agent produced an artifact and saves it locally. A successful model response alone does not prove that files were changed. `agent-up` keeps the VM running; `terminate --wait` waits for termination. Its image stays available for the next agent VM.
## Choose an agent and model
[Section titled “Choose an agent and model”](#choose-an-agent-and-model)
| Agent | Launch flag | Default Bedrock model | Override |
| ----------- | --------------------- | -------------------------------- | ------------------------- |
| Claude Code | `--agent claude-code` | `global.anthropic.claude-opus-5` | `--claude-model MODEL_ID` |
| Codex | `--agent codex` | `global.openai.gpt-5.6-sol` | `--codex-model MODEL_ID` |
Omitting `--agent` installs Claude Code. Repeat it to put both agents in the same VM, then select one on every prompt:
```bash
1
microvm agent-up --vm-name dev --agent claude-code --agent codex --project .
2
microvm agent-prompt --name dev --agent codex \
3
"Create hello.py that prints hello from a microvm, run it, and show the output."
4
microvm cp --name dev vm:/workspace/hello.py ./hello.py
5
microvm terminate dev --wait
```
Use `--claude-version VERSION` or `--codex-version VERSION` on a fresh launch to pin the npm package. A changed version pin produces a different image name. To install a different agent set or package version, launch a new VM under a new name, or terminate the old VM first.
## Move code and results
[Section titled “Move code and results”](#move-code-and-results)
The following examples address a running VM named `review`; run `agent-up` again if you terminated it after the quickstart.
`--project .` uploads a copy into `/workspace`. It skips `.git`, `target`, `node_modules`, and `.venv`; other files are included. Install project dependencies in the guest as needed. The agent image already includes Node.js 22, npm, Python 3, Git, and basic shell tools.
While a named VM is running, upload an updated project with `microvm agent-up --vm-name review --project .`. This refreshes credentials and uploads files into the existing workspace. It does not rebuild the VM or reinstall the agent packages.
Copy individual outputs with `microvm cp`, as in the quickstart. For a directory of generated outputs, download an archive:
```bash
1
microvm cp --name review --tar vm:/workspace/dist ./dist.tar
```
The local destination is a tar file. Downloading all of `/workspace` also includes `.agent-env`, which contains the Bedrock bearer token; select your output files or output directory when collecting results.
## Long tasks and VM lifetime
[Section titled “Long tasks and VM lifetime”](#long-tasks-and-vm-lifetime)
`agent-prompt` waits and collects output by default, with a 900-second timeout. Use `--timeout SECONDS` for another task budget. To return immediately, start a detached task and poll its exec ID (this example uses `jq`):
```bash
1
ID=$(microvm agent-prompt --name review --detach --json \
2
"Run the test suite and write a summary to TESTS.md." | jq -r .data.execId)
3
microvm exec --name review --poll "$ID"
4
# After the exec finishes and you have collected its output:
5
microvm ack --name review "$ID"
```
Repeat the poll until the task finishes. Add `--agent` when the VM carries both agents. A detached task keeps the same task timeout.
| Setting | Default | Effect |
| -------------------- | ------- | --------------------------------------------------------------------- |
| `--memory` | `1024` | 1 GiB billing baseline, 4 GiB guest ceiling |
| `--max-idle-sec` | `600` | Suspend after this many seconds without inbound traffic |
| `--suspended-sec` | `600` | Terminate after this many seconds suspended |
| `--max-duration-sec` | `3600` | Maximum VM lifetime; upper limit is 28800 seconds (8 hours) |
| `--auto-resume` | off | Let an inbound request wake a suspended VM |
| `--token-ttl-hours` | `12` | Maximum Bedrock token lifetime; signing credentials may expire sooner |
Choose lifetime settings on the initial `agent-up`. For multi-hour work, raise `--max-duration-sec` and keep inbound traffic active by polling `microvm health --name review` at intervals below `--max-idle-sec`. An idle suspended VM terminates after `--suspended-sec` even with `--auto-resume`. A busy process inside the guest does not count as inbound traffic. [Suspend and resume](/microvms-agentd/learn/tutorial/long-lived-vm/) explains the lifecycle in more detail.
If credentials expire while the VM is still alive, refresh them:
```bash
1
microvm agent-up --vm-name review
```
This reads the installed-agent marker and rewrites the credential files. It keeps the existing agents and models, builds nothing, and does not extend the VM’s lifetime. For a model change on an installed agent, name that agent explicitly with its model override; include both agent flags when keeping both profiles. Refresh does not install missing packages.
## Isolation and credentials
[Section titled “Isolation and credentials”](#isolation-and-credentials)
The agent runs as uid/gid 1000 in the guest. Claude Code’s headless profile permits `Bash,Read,Edit,Write,Grep,Glob`; Codex uses `workspace-write`. The token is installed after launch in `/workspace/.agent-env` with mode `0600`, so it is absent from the shared image snapshot.
The agent VM has internet egress to reach Bedrock. VM isolation does not make it a network-isolated workload. Keep the execution role minimal: guest processes can access its credentials. For workloads requiring no internet egress, use the general sandbox flow with an existing VPC connector in a VPC without an internet gateway or NAT gateway. Omitting `--egress` does not block traffic, and `--deny-egress` is advisory. See [Networking](/microvms-agentd/internals/networking/).
## Cleanup, automation, and SDKs
[Section titled “Cleanup, automation, and SDKs”](#cleanup-automation-and-sdks)
Always terminate a kept VM when finished:
```bash
1
microvm terminate review --wait
```
Add `--delete-image` when you also want to delete its image. Snapshot storage has a one-week minimum charge even if you delete earlier. Inspect cleanup reports and `data.leaked` on failures; `microvm ls --remote` helps find remaining resources. See [Costs](/microvms-agentd/learn/operations/read-the-cost-report/).
Use `--json` to automate the same flow: `agent-up` returns a `microvm.agent` envelope and `agent-prompt` returns `microvm.agent.prompt`. [Script and agent integration](/microvms-agentd/learn/operations/drive-it-from-a-script-or-an-agent/) covers errors, polling, and streaming.
For programmatic use, start with the complete [Python, Node.js/TypeScript, and Rust examples](/microvms-agentd/learn/tutorial/from-code/). The Python and Node packages also expose `AgentVm` for building agent images, launching, installing credentials, and prompting. The [Agent VMs specification](/microvms-agentd/internals/agent-vms/) documents those methods, profiles, and provisioning details. The [shell example](https://github.com/laithalsaadoon/microvms-agentd/tree/main/examples/coding-agents-on-bedrock) shows the individual build, launch, file-transfer, and exec steps.
# Run the live suite
> What mise run check proves and cannot prove, what mise run live adds and costs, how to run the conformance suite by hand, and how to confirm the account is clean afterwards.
```bash
1
mise run install # once per clone: the git hooks
2
mise run check # every local gate. Offline, free, no AWS.
3
mise run live # the real-AWS suites. BILLABLE, about fifteen minutes. Deliberate only.
4
mise run live:verify-clean
```
This page is for contributors. At the end of it you will know what `mise run check` proves, why a green `check` says nothing about the platform, what `mise run live` runs and what it costs, and how to leave the account clean.
## 1. Two tiers, split by cost
[Section titled “1. Two tiers, split by cost”](#1-two-tiers-split-by-cost)
`mise run check` is the definition of done for a change: lint, security, every Rust test tier, the schema and stub freshness gates, the model-drift gate against the pinned botocore service model, the publish-set check, the live tier’s own wiring check, and the release cross-compile. It creates no AWS resources. Tool installation, security rules, and advisory updates can require network access. The pre-push hook and CI both run it.
`mise run live` is the set of suites that talk to real AWS. It creates real MicroVMs in your account and costs money whether it passes or fails, so it is never wired to a hook or to push. A gate that spends money on every push is a gate people disable with `--no-verify`, and `--no-verify` also skips the checks worth having. The hook does print an advisory when the daemon has changed since the last recorded live run.
Never pipe a gate into `head` or `tail`. The pipeline exits with the pager’s status, so a failing tier reads as success. Run it bare, or read `${PIPESTATUS[0]}`.
## 2. Why a green check is necessary and never sufficient
[Section titled “2. Why a green check is necessary and never sufficient”](#2-why-a-green-check-is-necessary-and-never-sufficient)
The local gate proves the code agrees with itself; only a live run can prove it agrees with AWS. This repository’s history is a list of things every local test passed while being wrong: the null-message unsupported-region trap, the `clientToken` replay that wedges an image, the proxy-token port scoping, and an id-prefix defect where fixtures spelled MicroVM ids `mvm-*`, the real service spells them `microvm-*`, and a resolution path keyed on the fixture prefix passed every unit test and refused every real VM on its first live run. A fixture convention is not a service fact.
So a feature that touches the platform surface is not verified until it has run against real AWS, and the closing discipline is four steps: `mise run check` green; a live exercise of the new path itself, the full suite or a targeted round trip with the real binary; a permanent named check in `conformance/run_rs.py`, so the live tier covers the surface on every future run; and `mise run live:verify-clean` afterwards. A pull request that skips the second step says so in its body, in plain words, as an unverified claim. Purely local changes (rendering, docs, `ls`, `history`, `cost`) are exempt; when in doubt, it is not exempt.
## 3. Before a live run
[Section titled “3. Before a live run”](#3-before-a-live-run)
```bash
1
mise run live:infra # apply the conformance stack: bucket, build role, execution role
2
mise run build # the aarch64 daemon
3
mise run build:cli # the host-architecture microvm binary
```
Rebuild `target/release/microvm` explicitly: `check` does not build it, and a live run against a stale binary verifies nothing. `mise run live` depends on all three, so it does them for you; a targeted round trip by hand does not. `build --reuse` makes repeat image builds nearly free, which is what makes a targeted exercise cheap to repeat.
## 4. Run it
[Section titled “4. Run it”](#4-run-it)
```bash
1
mise run live
```
In order: `live:paths` (the pagination cursor encoding and the colon image ARN against the real signer; read-only and free), `live:versions` (the version and build operations, including the one `PATCH` this client sends; costs one short VM), `live:conformance-rs` (the conformance suite through the real CLI against real VMs), then `live:rates` (the pinned rate table against the AWS Pricing API; free, and placed after the billable suite so a drifted rate cannot abort it mid-flight). The leak check runs in a shell trap so it fires on the failure path too, and on success the task records the commit as live-verified in a per-clone marker the pre-push hook reads.
To run the suite by hand:
```bash
1
terraform -chdir=conformance/infra init
2
terraform -chdir=conformance/infra apply
3
cargo build --release -p agentd --target aarch64-unknown-linux-musl
4
cargo build --release -p microvms-cli
5
conformance/run_rs.py \
6
--binary target/aarch64-unknown-linux-musl/release/agentd \
7
--microvm-binary target/release/microvm
```
`--binary` is the aarch64 daemon baked into the image, and `--microvm-binary` is the host CLI under test. Read the check count off the run’s summary block rather than from any prose. `--keep` skips teardown and leaks everything, so use it only while debugging a failure you cannot reproduce otherwise.
`./conformance/run_rs.py --self-test` is the offline half. It drives the envelope-to-exception mapping against a stub `microvm`, touches no account, and belongs in any change to `conformance/`.
## 5. What it costs
[Section titled “5. What it costs”](#5-what-it-costs)
The suite creates an S3 artifact, a real image build (with a timeout of up to forty-five minutes), and a running MicroVM. The image is the floor: its snapshot has a one-week minimum retention, so a 2 GB image built and deleted in one run still bills about a week of storage, roughly four cents. `live:versions` launches one short VM of its own. `live:paths` and `live:rates` are free. [Read the cost report](/microvms-agentd/learn/operations/read-the-cost-report/) has the rates.
## 6. Afterwards
[Section titled “6. Afterwards”](#6-afterwards)
```bash
1
mise run live:verify-clean
2
mise run live:destroy
```
Verify teardown independently, and do not trust a success message. The scripts delete the MicroVM, the image, and the log group in `finally`, and `terraform destroy` handles the stack, yet the service creates `/aws/lambda-microvms/` itself, so Terraform never owns that log group and `destroy` reports success while the group survives. Six leaked that way before anyone noticed. `live:verify-clean` asks the account directly and separates leak, standing, and pending; [Recover a leaked VM](/microvms-agentd/learn/operations/recover-a-leaked-vm/) explains the three. `live:destroy` tears the Terraform stack back down when you are done.
## 7. Platform claims need a date, a region, and an API version
[Section titled “7. Platform claims need a date, a region, and an API version”](#7-platform-claims-need-a-date-a-region-and-an-api-version)
A live run that shows the platform behaving differently from [Platform](/microvms-agentd/internals/platform/) is a finding, and findings are appended rather than corrected. Every entry there carries when it was measured, in which region, under which API version, and whether it is this project’s measurement or AWS documentation. If you contradict an existing entry, add your measurement with its date so the drift is visible.
## 8. The docs site
[Section titled “8. The docs site”](#8-the-docs-site)
`mise run docs:check` builds and gates the site: install, sync, the brace gate, the build, the typecheck, and the probes over `dist/`. It is out of `check` because it needs a `pnpm install`, and a gate that fails on a fresh clone is a gate people learn to skip. Treat a green `check` as saying nothing about the site.
# Write a guest Dockerfile
> Start from the stanza microvm dockerfile prints, add your layers, build it locally under arm64, and avoid the traps that each cost a server-side build cycle.
```bash
1
microvm dockerfile --workdir /workspace > guest.Dockerfile
2
# add your RUN layers between the chmod line and the ENV lines
3
docker buildx build --platform linux/arm64 -t guest-check -f guest.Dockerfile .
4
microvm build --reuse --name my-task-image --dockerfile guest.Dockerfile
```
A server-side build cycle is roughly three minutes plus, on some failures, an image name you cannot reuse. Each item below cost a real cycle to find. At the end of this page you will have a Dockerfile that builds locally under arm64 and passes every check the client runs before it makes a billable call.
## 1. Start from the generated stanza
[Section titled “1. Start from the generated stanza”](#1-start-from-the-generated-stanza)
`microvm dockerfile` prints the Dockerfile that a default build bakes, generated by the same code that builds it, with the platform constraints annotated inline. `--from` sets the image ref for the `FROM` line and defaults to the managed al2023 base’s pair, `--port` (default `9000`) is the port agentd listens on inside the guest, and `--workdir` is a working directory to create and set, which the help calls strongly recommended. The output opens with comment lines naming the constraints below; stripped of those, the stanza is this:
```dockerfile
1
FROM public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
2
COPY agentd /agentd
3
RUN chmod 0755 /agentd
4
RUN mkdir -p /workspace
5
WORKDIR /workspace
6
ENV AGENTD_PORT=9000
7
ENV AGENTD_LOG=info
8
EXPOSE 9000
9
ENTRYPOINT []
10
CMD ["/agentd"]
```
Append your `RUN` layers after the `chmod` line and pass the result to `--dockerfile`. When you pass `--dockerfile` you are replacing the default, so you carry its daemon lines yourself.
## 2. Build it locally under arm64 first
[Section titled “2. Build it locally under arm64 first”](#2-build-it-locally-under-arm64-first)
```bash
1
docker buildx build --platform linux/arm64 -t guest-check -f guest.Dockerfile .
```
A `dnf` typo or a missing package is free to find here and expensive to find server-side. With `qemu-aarch64` binfmt registered this builds the real target architecture on an x86 host. Two errors in the repository’s own example Dockerfile were caught this way in seconds.
## 3. The rules the client checks before any call
[Section titled “3. The rules the client checks before any call”](#3-the-rules-the-client-checks-before-any-call)
**The `FROM` must match the managed base’s registry ref.** The build runs your Dockerfile on top of the base the create call names, and a mismatch builds against a base none of the measured platform behavior applies to, so the client refuses it locally. A `@sha256:` digest suffix on that same ref is accepted and is what the shipped examples pin.
**Set a `WORKDIR` explicitly.** `al2023-minimal`, like most public ARM64 bases, leaves `WorkingDir` empty, so an omitted `--cwd` on every later `exec` resolves against `/`. The client requires one when the base declares none. Check that the user your workload runs as can write to it; a root-owned `WORKDIR` under a non-root workload fails at the first write.
**`ENV AGENTD_PORT` must agree with the client’s port.** The platform dials its build-time `ready` and `validate` hooks on the port from the create call, so a guest listening elsewhere answers neither and the build fails with `CREATE_FAILED` after a completely clean build log. The daemon’s own `agentd listening` line appears, with the wrong address, and no error line follows. The client refuses the disagreement locally, including the case where the Dockerfile names no port while you have moved the client off the default with `--port`: an unset variable leaves the daemon on 9000 rather than on your port.
**Keep `ENTRYPOINT []` and `CMD ["/agentd"]`.** That pair is the trust boundary. It guarantees no workload runs before the platform’s run hook lands and the token arrives, and it is what makes an omitted `cwd` inherit the image `WORKDIR`. An `ENTRYPOINT` that swallows `CMD` produces a VM whose daemon never starts, and the symptom is a run-hook timeout that mentions neither. A base image that starts its own background process before bootstrap breaks the invariant too, and enforcing it belongs to whoever builds the image. [Trust](/microvms-agentd/internals/trust/) is the threat model.
**Leave `AGENTD_SSE_KEEPALIVE_SECS` alone** unless you also raise the client’s stream idle timeout. The client treats sixty seconds of silence as a dead connection because the daemon’s keepalive is fifteen; a longer interval makes healthy streams look dead. This too is refused locally.
## 4. On a -minimal base
[Section titled “4. On a -minimal base”](#4-on-a--minimal-base)
Spell weak dependencies off as `--setopt=install_weak_deps=0`. The integer is what works; `False` is not accepted.
`useradd` is not in the minimal base, and pulling `shadow-utils` in for one line is a bigger image for no more correctness. The examples append to `/etc/passwd` directly and hand the working directory to that uid, because `WORKDIR` is where every exec lands and a root-owned one fails a demoted workload at its first write:
```dockerfile
1
RUN echo "agent:x:1000:1000::/workspace:/bin/bash" >> /etc/passwd \
2
&& echo "agent:x:1000:" >> /etc/group
3
RUN mkdir -p /workspace && chown 1000:1000 /workspace
4
WORKDIR /workspace
```
The daemon itself stays root; demotion is per command, with `exec --user 1000 --group 1000`. [Run coding agents on Bedrock](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) shows why that uid is not decoration.
## 5. No secret goes in the image
[Section titled “5. No secret goes in the image”](#5-no-secret-goes-in-the-image)
The image becomes a shared snapshot, so every VM launched from it sees the same bytes. Per-VM credentials travel through the platform’s one-shot `runHookPayload` at launch: the agent token always, and anything you add with `run --launch-env KEY=VALUE`, which shares the payload’s 4096-byte ceiling. Larger material goes in after bootstrap with `microvm cp --mode 0600` over the authenticated channel.
## 6. The build context holds two members
[Section titled “6. The build context holds two members”](#6-the-build-context-holds-two-members)
The artifact `microvm build` uploads carries exactly the Dockerfile and the `agentd` binary, so there is no file beside them for a `COPY` to find. A script the image needs is materialized in the Dockerfile with `printf`; [Prefetch S3 content at image build](/microvms-agentd/learn/operations/prefetch-s3-at-build/) shows a start wrapper written that way. `build --project` is the one widening: it adds the project’s manifest and lockfile pair at the archive root, so `COPY pyproject.toml uv.lock ./` finds them.
## 7. The daemon’s knobs
[Section titled “7. The daemon’s knobs”](#7-the-daemons-knobs)
Every `AGENTD_*` variable is read at startup, and an unset or unparseable value keeps the default rather than refusing to boot, because a daemon that will not start strands the VM with no way in. Set them as `ENV` lines. The stanza already sets `AGENTD_PORT` and `AGENTD_LOG`; the others bound request bodies, captured output, stream replay, stdin writes, the disk reserve, and the identity repair that gives each VM restored from one snapshot its own machine-id, hostname, and boot\_id. The table with defaults is in [Embedding](/microvms-agentd/internals/embedding/).
## 8. Working examples
[Section titled “8. Working examples”](#8-working-examples)
Three Dockerfiles in the repository respect all of the above: [coding-agents-on-bedrock](https://github.com/laithalsaadoon/microvms-agentd/tree/main/examples/coding-agents-on-bedrock) adds Node, Python, and two agent CLIs; [code-server-remote-dev](https://github.com/laithalsaadoon/microvms-agentd/tree/main/examples/code-server-remote-dev) installs an IDE from a release RPM; [s3-prefetch-at-build](https://github.com/laithalsaadoon/microvms-agentd/tree/main/examples/s3-prefetch-at-build) replaces `CMD` with a wrapper that fetches before it hands off to the daemon.
When a build fails anyway, [Debug a failed build](/microvms-agentd/learn/operations/debug-a-failed-build/) says where the reason lives. The measurements behind every rule above are in [Platform](/microvms-agentd/internals/platform/).
# Configure AWS and run your first sandbox
> Set up AWS once, give an agent a project in a MicroVM, and copy its results back.
Give a coding agent a copy of your project in its own AWS Lambda MicroVM. It can inspect files, edit code, and run commands in `/workspace`; you choose which results to bring back to your machine.
With the [CLI installed](/microvms-agentd/learn/tutorial/install/) and AWS configured, you can start the workflow below in 90 seconds. A first image build takes several minutes, and the agent’s task takes additional time. These commands create billable AWS resources.
## Configure AWS
[Section titled “Configure AWS”](#configure-aws)
You need AWS CLI v2, your normal AWS credential configuration, and a region where your account can use Lambda MicroVMs. Supported region names are `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, and `ap-northeast-1`. Use `gh` or `curl` for the CLI’s automatic daemon download.
Image builds need an S3 artifact bucket, a build role, and an execution role in your account. Your caller needs permission to manage MicroVMs, upload the artifact, and pass those roles. For coding agents, it also needs Bedrock model invocation permissions and access to the chosen model. The default Claude Code profile uses `global.anthropic.claude-opus-5`; the [agent guide](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) covers Codex and model overrides.
Export your existing infrastructure values, replacing the examples:
```bash
1
export AWS_REGION=us-east-1
2
export MICROVM_BUCKET=your-artifact-bucket
3
export MICROVM_BUILD_ROLE_ARN=arn:aws:iam::123456789012:role/microvm-build
4
export MICROVM_EXECUTION_ROLE_ARN=arn:aws:iam::123456789012:role/microvm-execution
```
Use `AWS_PROFILE` if your credentials are in a named profile. These examples use Bash-compatible shell syntax; in PowerShell, set environment variables with `$env:NAME = "value"`. Matching `--bucket`, `--build-role-arn`, `--execution-role-arn`, and `--region` flags are also available.
If you need the bucket and roles, the repository includes a Terraform example. With Git and Terraform 1.6+ installed, run this once:
```bash
1
git clone https://github.com/laithalsaadoon/microvms-agentd.git
2
cd microvms-agentd
3
terraform -chdir=conformance/infra init
4
terraform -chdir=conformance/infra apply -var="region=us-east-1"
5
export AWS_REGION=us-east-1
6
export MICROVM_BUCKET=$(terraform -chdir=conformance/infra output -raw s3_bucket)
7
export MICROVM_BUILD_ROLE_ARN=$(terraform -chdir=conformance/infra output -raw build_role_arn)
8
export MICROVM_EXECUTION_ROLE_ARN=$(terraform -chdir=conformance/infra output -raw execution_role_arn)
```
This is an example infrastructure stack; it does not grant your caller Bedrock model access. Keep the guest execution role minimal because the workload can access its credentials. The stack also creates a separate build-log reader policy for the operator.
```bash
1
microvm doctor
```
`doctor` reports setup findings and remedies. Check those before launching; it does not prove that your chosen Bedrock model is available to your account.
## Run an agent on your project
[Section titled “Run an agent on your project”](#run-an-agent-on-your-project)
Change to the project directory you want the agent to review, then run:
```bash
1
microvm agent-up --vm-name review --agent claude-code --project .
2
microvm agent-prompt --name review \
3
"Review this project and write your findings to REVIEW.md."
4
microvm cp --name review vm:/workspace/REVIEW.md ./REVIEW.md
5
microvm terminate review --wait
```
`agent-up` prepares the image, starts the VM, uploads your project, and installs a short-lived Bedrock token. `agent-prompt` runs as uid 1000 in `/workspace`. The copy command retrieves the actual file the agent wrote. Your project upload skips `.git`, `target`, `node_modules`, and `.venv`; other files are included. Dependencies excluded from the upload may need to be installed inside the VM.
`agent-up` keeps the VM until you terminate it or its lifetime policy expires. The default is a one-hour maximum, suspension after ten minutes of inbound inactivity, and termination after ten minutes suspended. Its image remains for reuse on the next launch.
The VM has outbound access to reach Bedrock. Omitting `--egress` on a general `run` does not disable outbound traffic. For no egress, use a custom VPC connector in a VPC without an internet gateway, NAT gateway, or other internet route; `--deny-egress` only sets proxy variables that workloads can bypass. See [Networking](/microvms-agentd/internals/networking/).
## Run a command without an agent
[Section titled “Run a command without an agent”](#run-a-command-without-an-agent)
For a hello-world with automatic VM and image cleanup:
```bash
1
microvm quickstart
```
For repeated CLI runs or the [SDK examples](/microvms-agentd/learn/tutorial/from-code/), build a general-purpose image once:
```bash
1
microvm build --name agent-tools --json
2
microvm run --image agent-tools --exec "uname -m"
```
The guest prints `aarch64`. Each `run` tears down its VM by default, while the existing image stays available. The CLI accepts an image name; SDKs need its ARN. Copy `data.imageIdentifier` from the build’s JSON into `MICROVM_IMAGE` for the SDK examples:
```bash
1
export MICROVM_IMAGE='paste-the-image-ARN-here'
```
Replace the example value with the exact ARN the build returned. With `--reuse`, `build` appends a content hash to the name; use the returned `imageIdentifier` or complete `imageName` in CLI commands instead of the `agent-tools` prefix.
## Check cleanup and cost
[Section titled “Check cleanup and cost”](#check-cleanup-and-cost)
`--json` returns structured results. On cleanup failures, inspect `data.leaked`; `microvm ls --remote` helps find remaining resources. Cost totals are estimates and are lower bounds when some items are unpriced. Images have a one-week minimum retention charge, so reusing them avoids repeated builds and snapshots.
Next: [run both coding agents](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/), [use an SDK](/microvms-agentd/learn/tutorial/from-code/), or [keep a VM and work by name](/microvms-agentd/learn/tutorial/long-lived-vm/).
# Use the SDKs
> Install the Python, JavaScript, TypeScript, or Rust SDK; run commands and coding agents in sandboxed MicroVMs.
Use the SDKs to give your agent a disposable Linux workspace: upload inputs, run tools, read their output, download results, and terminate the VM. `Sandbox` runs your own commands; `AgentVm` adds Claude Code and Codex through Amazon Bedrock.
With AWS configured and an existing image, pick a language below and run your first command in about 90 seconds. One-time AWS setup and image builds take longer.
## Before you start
[Section titled “Before you start”](#before-you-start)
Complete [AWS setup](/microvms-agentd/learn/tutorial/first-run/) and export `AWS_REGION` and `MICROVM_EXECUTION_ROLE_ARN`. You need Lambda MicroVMs access and AWS credentials on the machine running the SDK. SDKs use the normal AWS credential chain; they do not read the CLI’s `microvm.toml` configuration.
Use the [CLI](/microvms-agentd/learn/tutorial/install/) to build an image containing `agentd`:
```sh
1
microvm build --name agent-tools --json
2
export MICROVM_IMAGE='paste data.imageIdentifier from the result'
```
Use the image **ARN**, in the same account and region as your credentials. The CLI resolves image names; these SDK calls take the ARN directly. The CLI handles daemon provisioning and artifact upload. If your team has already built a compatible image, set `MICROVM_IMAGE` to its ARN and skip the build.
The shell examples use Bash or Zsh. In PowerShell, set environment variables with `$env:NAME='value'`.
## Python
[Section titled “Python”](#python)
Requires CPython 3.9+. Install in a virtual environment:
```sh
1
python3 -m venv .venv
2
. .venv/bin/activate
3
python -m pip install microvms
```
On Windows, activate with `.venv\Scripts\Activate.ps1`. Save this as `hello.py`:
```python
1
import os
2
import sys
3
from microvms import Region, Sandbox
4
5
image = os.environ["MICROVM_IMAGE"]
6
role = os.environ["MICROVM_EXECUTION_ROLE_ARN"]
7
vm = Sandbox(Region.parse(os.environ.get("AWS_REGION", "us-east-1")))
8
try:
9
session = vm.run(image_identifier=image, execution_role_arn=role)
10
result = session.run_sync(["echo", "hello from a sandbox"])
11
print(result.stdout, end="")
12
print(result.stderr, end="", file=sys.stderr)
13
if not result.ok:
14
raise RuntimeError(f"Command exited with {result.exit_code}")
15
finally:
16
cleanup = vm.terminate()
17
if cleanup.failures or cleanup.undeleted:
18
print("Cleanup needs attention:", cleanup.to_dict(), file=sys.stderr)
```
```sh
1
python hello.py
```
Expected output: `hello from a sandbox`.
Methods are synchronous. `session.run_sync()` starts a command, waits for completion, returns stdout/stderr and an exit code, and acknowledges the saved output. Nonzero command exits are results; library failures raise exceptions with `code`, `kind`, `wire_kind`, and `retryable` attributes.
## JavaScript / TypeScript
[Section titled “JavaScript / TypeScript”](#javascript--typescript)
Requires Node.js 22.13.0+. The package includes TypeScript declarations.
```sh
1
npm install @theagenticguy/microvms
```
Save this as `hello.mjs`:
```js
1
import { Region, Sandbox } from '@theagenticguy/microvms';
2
3
const imageIdentifier = process.env.MICROVM_IMAGE;
4
const executionRoleArn = process.env.MICROVM_EXECUTION_ROLE_ARN;
5
if (!imageIdentifier || !executionRoleArn) {
6
throw new Error('Set MICROVM_IMAGE and MICROVM_EXECUTION_ROLE_ARN first');
7
}
8
const vm = await Sandbox.create(Region.parse(process.env.AWS_REGION ?? 'us-east-1'));
9
try {
10
const session = await vm.run({ imageIdentifier, executionRoleArn });
11
const result = await session.runSync(['echo', 'hello from a sandbox']);
12
process.stdout.write(result.stdout);
13
process.stderr.write(result.stderr);
14
if (!result.ok) process.exitCode = 1;
15
} finally {
16
const cleanup = await vm.terminate();
17
if (cleanup.failures.length || cleanup.undeleted.length) {
18
console.error('Cleanup needs attention:', cleanup);
19
process.exitCode = 1;
20
}
21
}
```
```sh
1
node hello.mjs
```
Expected output: `hello from a sandbox`.
`runSync()` returns a Promise, like other methods that contact AWS or the guest. It starts a command, waits, and acknowledges the saved output. A nonzero exit is a result; an async library failure rejects with its `ERR_*` code in `error.cause.message`.
Both examples terminate the VM in `finally`, including when a command fails. Cleanup reports failures through `failures` and `undeleted`. By default it returns when termination is accepted; use `terminate(wait_for_terminated=True)` in Python or `terminate({ waitForTerminated: true })` in JavaScript to observe completion. The image remains available for reuse.
## Run a coding agent
[Section titled “Run a coding agent”](#run-a-coding-agent)
`AgentVm` installs model access after launch and runs the agent as UID/GID 1000 in `/workspace`. Each task can have its own VM and files. This example asks Claude Code to write and run a Python script, then downloads that specific file to your machine.
First configure [Bedrock access](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/). Prepare an agent image once, keeping its ARN for later SDK launches. The command below starts a temporary VM, so terminate it after capturing the image ARN. For Python users:
```sh
1
MICROVM_AGENT_IMAGE="$(microvm agent-up --vm-name sdk-image --agent claude-code --json |
2
python -c 'import json, sys; print(json.load(sys.stdin)["data"]["imageIdentifier"])')"
3
export MICROVM_AGENT_IMAGE
4
microvm terminate sdk-image --wait
```
For Node users, replace the `python -c ...` part with `node -pe 'JSON.parse(require("node:fs").readFileSync(0, "utf8")).data.imageIdentifier'`.
### Python agent
[Section titled “Python agent”](#python-agent)
Save as `agent.py`:
```python
1
import os
2
import sys
3
from microvms import AgentVm, Region
4
5
image = os.environ["MICROVM_AGENT_IMAGE"]
6
role = os.environ["MICROVM_EXECUTION_ROLE_ARN"]
7
vm = AgentVm(Region.parse(os.environ.get("AWS_REGION", "us-east-1")))
8
try:
9
session = vm.launch(image_identifier=image, execution_role_arn=role)
10
vm.install_access()
11
result = vm.prompt_sync(
12
"claude-code",
13
"Create /workspace/hello.py that prints hello from a sandbox. Run it.",
14
)
15
print(result.stdout, end="")
16
print(result.stderr, end="", file=sys.stderr)
17
if not result.ok:
18
raise RuntimeError(f"Agent exited with {result.exit_code}")
19
with open("hello-from-agent.py", "wb") as artifact:
20
artifact.write(session.download_file("/workspace/hello.py"))
21
finally:
22
cleanup = vm.terminate()
23
if cleanup.failures or cleanup.undeleted:
24
print("Cleanup needs attention:", cleanup.to_dict(), file=sys.stderr)
```
```sh
1
python agent.py
```
### JavaScript agent
[Section titled “JavaScript agent”](#javascript-agent)
Save as `agent.mjs`:
```js
1
import { writeFile } from 'node:fs/promises';
2
import { AgentVm, Region } from '@theagenticguy/microvms';
3
4
const imageIdentifier = process.env.MICROVM_AGENT_IMAGE;
5
const executionRoleArn = process.env.MICROVM_EXECUTION_ROLE_ARN;
6
if (!imageIdentifier || !executionRoleArn) {
7
throw new Error('Set MICROVM_AGENT_IMAGE and MICROVM_EXECUTION_ROLE_ARN first');
8
}
9
const vm = await AgentVm.create(Region.parse(process.env.AWS_REGION ?? 'us-east-1'));
10
try {
11
const session = await vm.launch({ imageIdentifier, executionRoleArn });
12
await vm.installAccess();
13
const result = await vm.promptSync(
14
'claude-code',
15
'Create /workspace/hello.py that prints hello from a sandbox. Run it.',
16
);
17
process.stdout.write(result.stdout);
18
process.stderr.write(result.stderr);
19
if (!result.ok) throw new Error(`Agent exited with ${result.exitCode}`);
20
await writeFile('hello-from-agent.py', await session.downloadFile('/workspace/hello.py'));
21
} finally {
22
const cleanup = await vm.terminate();
23
if (cleanup.failures.length || cleanup.undeleted.length) {
24
console.error('Cleanup needs attention:', cleanup);
25
process.exitCode = 1;
26
}
27
}
```
```sh
1
node agent.mjs
```
Both examples create `hello-from-agent.py` locally. The download checks that the agent actually produced its artifact: an agent process can exit zero without completing the requested task.
For Codex, build with `--agent codex` and match the SDK’s agent selection:
| Language | Create | Prompt |
| ---------- | ----------------------------------------------------------- | ------------------------------- |
| Python | `AgentVm(region, [AgentSpec.codex()])` (import `AgentSpec`) | `vm.prompt_sync("codex", task)` |
| JavaScript | `AgentVm.create(region, [{ agent: 'codex' }])` | `vm.promptSync('codex', task)` |
`prompt_sync` / `promptSync` defaults to a 900-second task deadline. Override with `timeout=...` in Python or `{ timeoutSec: ... }` in JavaScript. The selected agent must be installed in the image. `AgentVm` enables internet egress for Bedrock calls.
## Connect your own agent tools
[Section titled “Connect your own agent tools”](#connect-your-own-agent-tools)
Keep your orchestrator in your application and point its command and file tools at a session. For example, use these calls inside the `try` block after obtaining `session`:
| Task | Python | JavaScript |
| ------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------ |
| Start a command without waiting | `session.run(["python3", "job.py"])` | `await session.run(['python3', 'job.py'])` |
| Upload an input | `session.upload_file("/workspace/input.txt", b"hello")` | `await session.uploadFile('/workspace/input.txt', Buffer.from('hello'))` |
| Download a result | `session.download_file("/workspace/result.txt")` | `await session.downloadFile('/workspace/result.txt')` |
| Upload a project tar archive | `session.upload_tar("/workspace", tar_bytes)` | `await session.uploadTar('/workspace', tarBytes)` |
`run()` returns an `ExecHandle` for polling, streaming, or reattaching by exec ID. Use `wait_and_ack()` / `waitAndAck()` when collecting its final output. JavaScript also offers `spawn()` with readable byte streams. Pass `shell=True` / `{ shell: true }` for shell-script strings; argv arrays need no shell option.
For arbitrary agent-generated commands, set the exec’s `user` and `group` to a non-root UID/GID and make its workspace writable by that user. The built-in `AgentVm` prompt methods already select UID/GID 1000. Keep the VM execution role limited to the task’s needs.
VM isolation does not imply blocked outbound networking. Omitting `egress` does not block traffic, and `deny_egress` / `denyEgress` only sets advisory proxy variables. For a network boundary, use a customer-managed VPC connector and a VPC without internet or NAT gateways; see [Networking](/microvms-agentd/learn/operations/configure-networking/).
## Rust
[Section titled “Rust”](#rust)
```sh
1
cargo add microvms-core
```
`microvms-core` is the Rust SDK behind the CLI and bindings. Use `Sandbox::new(region).await`, `RunRequest` for launch, `Session::run_sync` for command results, and `Sandbox::terminate` for cleanup. The async API also includes `agents::AgentVm`, file transfer, streaming, suspend/resume, and cost estimates.
[Run the complete Rust example](https://github.com/laithalsaadoon/microvms-agentd/tree/main/microvms-core#run-your-first-command) to launch the same image and execute a command. [docs.rs](https://docs.rs/microvms-core) documents the Rust methods and request types. [Public API](/microvms-agentd/reference/public-api/) maps the available surfaces. The [Python package guide](https://github.com/laithalsaadoon/microvms-agentd/tree/main/microvms-py) and [Node package guide](https://github.com/laithalsaadoon/microvms-agentd/tree/main/microvms-js) cover supported hosts.
## Build and operate beyond the quickstart
[Section titled “Build and operate beyond the quickstart”](#build-and-operate-beyond-the-quickstart)
The SDKs expose `build_artifact` / `buildArtifact` and `build_image` / `buildImage` for custom images. Upload the artifact bytes to S3 before calling the build method; the SDK does not perform that upload. Build once and reuse the resulting image ARN across fresh sandboxes.
* [Run coding agents on Bedrock](/microvms-agentd/learn/operations/run-coding-agents-on-bedrock/) covers model access and repeated agent tasks.
* [Read the cost report](/microvms-agentd/learn/operations/read-the-cost-report/) covers estimates, measured durations, and retained resources.
* [Embedding](/microvms-agentd/internals/embedding/) describes the daemon contract for a custom harness or transport.
# Install the CLI or an SDK
> Get the prebuilt microvm CLI or install the Python, Node.js, and Rust SDKs.
The `microvm` CLI runs agents and commands in sandboxed AWS Lambda MicroVMs. Install it first to prepare an image, run a coding agent, or try the SDKs.
## CLI: use a prebuilt binary
[Section titled “CLI: use a prebuilt binary”](#cli-use-a-prebuilt-binary)
Download the archive for your machine from the [latest release](https://github.com/laithalsaadoon/microvms-agentd/releases/latest), extract it, and put `microvm` (`microvm.exe` on Windows) on your `PATH`. Releases cover Linux x64/ARM64, macOS Intel/Apple silicon, and Windows x64. No Rust compiler is required.
If [cargo-binstall](https://github.com/cargo-bins/cargo-binstall) is already installed, it selects and installs the release for you:
```bash
1
cargo binstall microvms-cli --no-confirm
2
microvm --version
```
With a Rust toolchain, compiling from crates.io is another option:
```bash
1
cargo install microvms-cli --locked
```
`microvm --version` works without AWS credentials. To see the installed command surface, use `microvm --help`; `microvm manifest` emits it as JSON.
Next: [configure AWS and start an agent](/microvms-agentd/learn/tutorial/first-run/).
## SDKs
[Section titled “SDKs”](#sdks)
Choose your language:
| Language | Install | Runtime |
| -------------------- | ------------------------------------- | -------------- |
| Python | `pip install microvms` | Python 3.9+ |
| Node.js / TypeScript | `npm install @theagenticguy/microvms` | Node.js 22.13+ |
| Rust | `cargo add microvms-core` | Rust toolchain |
[Run sandboxed tools from your application](/microvms-agentd/learn/tutorial/from-code/) has complete examples, including cleanup. The SDKs connect to the same AWS service as the CLI.
## The guest daemon is automatic
[Section titled “The guest daemon is automatic”](#the-guest-daemon-is-automatic)
`agent-up`, `run`, `build`, and `quickstart` download the matching `agentd` release binary when needed and cache it locally. Keep `gh` or `curl` on your `PATH`. A successful `gh` download uses `gh attestation verify` for provenance; the `curl` fallback verifies the release’s SHA256 checksum. The guest daemon is a static ARM64 Linux binary on every host platform.
For a daemon you build or manage yourself, set `MICROVM_AGENTD` to its path, or pass the path as the positional argument to `agent-up`, `run`, or `build`. `microvm doctor --binary ./agentd` checks its architecture.
## Build the CLI from this repository
[Section titled “Build the CLI from this repository”](#build-the-cli-from-this-repository)
From a clone with Rust installed:
```bash
1
cargo install --path microvms-cli --locked
```
This builds the host CLI. It can still download the guest daemon automatically. Repository contributors can use `mise install` and `mise run install` to install the pinned tools and Git hooks.
# Keep a VM running and work inside it
> Launch a VM with --keep and a local name, run commands in it with exec, move files with cp, freeze it with suspend, thaw it with resume, and release it with terminate.
`microvm run` tears the VM down when the command finishes. This tutorial keeps one, gives it a local name, and works inside it with the attached commands.
At the end of this page you will have run commands in a named VM, streamed and detached one, copied files in and out, suspended and resumed it, and terminated it, and you will know what `microvm ls` reports if any of that is interrupted.
You need a working first run, so [complete that tutorial](/microvms-agentd/learn/tutorial/first-run/) first.
## 1. Launch and name it
[Section titled “1. Launch and name it”](#1-launch-and-name-it)
```bash
1
microvm run --keep --vm-name dev
```
`--keep` leaves the VM and image running; you are then paying for them. `--vm-name` registers a local name for the kept VM, so later commands can say `--name dev` instead of pasting the endpoint, agent token, and MicroVM id triple.
The name is a purely local fact. The registry is one file per name in the CLI’s state directory (`~/.microvm/runs/names/.json`, or `$MICROVM_STATE_DIR`), written owner-only because the record carries the agent token, and resolving a name costs zero AWS calls. Names take ASCII letters, digits, `-`, and `_`, up to 128 bytes, and never the `microvm-` prefix the service uses for ids, which is what lets every identifier-taking command tell a name from an id. A name already registered to a live VM is refused locally with `ERR_NAME_TAKEN` (exit 14) before any billable call.
The explicit `--endpoint`, `--agent-token`, and `--microvm-id` flags still work everywhere, for a VM some other machine launched. The `run` envelope reports all three, plus `kept: true` and `vmName`.
## 2. Run commands in it
[Section titled “2. Run commands in it”](#2-run-commands-in-it)
```bash
1
microvm exec --name dev "python3 -V"
```
`exec` runs one command in a MicroVM that is already running. `--cwd` sets the working directory, `--env KEY=VALUE` sets one variable for the command and is repeatable, `--user` and `--group` run it as a numeric uid and gid, and `--timeout` (default 300 seconds) bounds the wait. The daemon spawns execs with a minimal environment, so a command that needs `PATH` or `HOME` gets them from `--env`, or for every exec in the VM from `run --launch-env KEY=VALUE`.
`--stream` streams output as it arrives rather than waiting for the whole thing. Under `--json` this is the one invocation that writes more than one object to stdout: NDJSON events, then the envelope last, with type `microvm.exec.stream`. `--from-offset` resumes a stream at a byte offset.
`--stdin` gives the command a stdin pipe, feeds it this process’s stdin, and then closes it:
```bash
1
echo "hello" | microvm exec --name dev --stdin "cat"
```
`--detach` starts the command and returns immediately, without waiting and without acking. `--poll ` reads an existing exec’s status and output instead of starting anything, and `ack ` releases a finished exec’s buffered output, which starts its collection clock:
```bash
1
microvm exec --name dev --detach --json "make test" # the envelope carries execId
2
microvm exec --name dev --poll
3
microvm ack --name dev
```
Polling is terminal-only: a running exec reports `phase: running` with no partial stdout, so stream it or have it write to a file if you want to watch. Output lives in the daemon until it is acked, so nothing a slow reader has not seen is destroyed, and a detached exec outlives the hourly rotation of the endpoint’s proxy token by design. `--exec-id` supplies your own id, so a retry of the same start returns the original exec instead of spawning a second child.
## 3. Move files in and out
[Section titled “3. Move files in and out”](#3-move-files-in-and-out)
```bash
1
microvm cp ./data.csv vm:/tmp/data.csv --name dev
2
microvm cp vm:/tmp/result.json ./result.json --name dev
3
microvm cp --tar ./project.tar vm:/workspace --name dev
4
microvm cp --tar vm:/workspace ./workspace-backup.tar --name dev
```
`vm:/path` names the VM side; anything else is a local path. `--tar` moves a whole directory tree as an uncompressed tar archive, and extraction in the guest is confined so a hostile archive cannot write outside its target. `--mode` sets the permissions of an uploaded file, octal as a string (`644`, `0755`). The daemon runs as root, so an uploaded file is root-owned; a workload you run demoted with `--user` needs a `chown` exec before it can read one.
## 4. Freeze and thaw it
[Section titled “4. Freeze and thaw it”](#4-freeze-and-thaw-it)
```bash
1
microvm suspend dev
2
microvm resume dev
```
`suspend`, `resume`, and `terminate` take the MicroVM id as their positional argument, and a registered name stands in for it: a bare name is resolved through the local registry with zero AWS calls, and anything shaped like an id passes through.
Suspend is a freeze. Memory, the filesystem, the agent token, and the endpoint survive, and a running process resumes mid-flight. A suspended VM pays snapshot storage only, and each suspend/resume cycle pays a snapshot write plus a read, so a long suspension is cheap and constant cycling is the habit to avoid.
The platform also suspends on its own. `--max-idle-sec` (default 600) suspends the VM after that much inbound-traffic idleness, `--suspended-sec` (default 600) terminates it after that long suspended, and `--auto-resume` lets the platform resume a suspended VM on an incoming request. A resume past the suspended window cannot work: the CLI refuses it with `ERR_WINDOW_CLOSED` (exit 8), and no call extends the window once the VM is launched. `--max-duration-sec` (default 3600) is the hard ceiling on the VM’s life and is refused above 28800 (eight hours) before any call.
Idleness is measured by inbound traffic through the endpoint proxy, which terminates outside the VM. So a keepalive has to come from outside too: `microvm health --name dev` polls `/v1/health`, which is unauthenticated and resets the idle timer, and reports `busy` and `execs` so the poll can stop once the VM is drained. A guest process cannot keep its own VM alive, and a multi-hour exec with no outside traffic is frozen at the idle window with its process intact.
## 5. A shell and a browser
[Section titled “5. A shell and a browser”](#5-a-shell-and-a-browser)
Launch with `--shell` and `microvm shell --name dev` opens an interactive root shell in the VM, a real PTY with job control, signals, and resize. `microvm port-forward 8080:8080 --name dev` serves a guest port on localhost so a browser here reaches a server in the VM, and `microvm tunnel 5432 --name dev` does the same for arbitrary TCP so `psql` or `ssh` here reaches a server there. Both bind `127.0.0.1` by default, deliberately. [Remote dev with code-server](/microvms-agentd/learn/operations/remote-dev-with-code-server/) builds a dev box on these.
## 6. Release it
[Section titled “6. Release it”](#6-release-it)
```bash
1
microvm terminate dev
```
The name is released when the terminate is accepted. `--wait` waits for `TERMINATED` rather than returning as soon as the call is accepted. `--delete-image` also deletes the image, read off the run record `run --keep` left (pass `--image-identifier ` and `--image-name ` only for a VM this state directory did not launch), and names its build log group, which the service created and Terraform never owns. Deleting the image early saves nothing, because its snapshot has a one-week minimum retention.
## 7. If something is interrupted
[Section titled “7. If something is interrupted”](#7-if-something-is-interrupted)
`microvm ls` lists what this CLI created and could not confirm it deleted, from the local ledger, with zero AWS calls. `microvm history dev` prints what was asked of one MicroVM and what the platform reported back, and the record survives terminate on purpose. [Recover a leaked VM](/microvms-agentd/learn/operations/recover-a-leaked-vm/) is the runbook.
## 8. From another machine
[Section titled “8. From another machine”](#8-from-another-machine)
The registry record is the export format. `attach` registers a name for a running MicroVM this state directory did not launch, from a record file or the explicit triple:
```bash
1
ssh other cat ~/.microvm/runs/names/dev.json | microvm attach --from -
```
**For an agent.** Capture `execId` from a `--detach --json` start, poll it with `--poll`, and `ack` it when you have read the output. Pass your own `--exec-id` on a start you may have to retry, so a retry returns the original exec rather than spawning a second child. Read `phase` on a poll before treating empty stdout as the command’s answer.
Next: [run a project through a VM](/microvms-agentd/learn/tutorial/run-a-project/).
# Run a project through a VM
> Point run at a directory to pack it, upload it to /workspace, run a command there, and bring artifacts back; then bake the project's dependencies into the image with build --project.
When the positional argument to `run` is a directory rather than a binary, `run` becomes a pack-run-collect round trip against an existing image. This tutorial runs a project’s test command that way, brings its reports back, and then builds an image that already carries the project’s dependencies.
At the end of this page your project’s command will have run inside a VM with the tree at `/workspace`, its artifacts will be back on your disk, and the image will carry a dependency layer so a launch skips installation.
You need an image to launch, so [build and keep one](/microvms-agentd/learn/tutorial/first-run/) first.
## 1. The round trip
[Section titled “1. The round trip”](#1-the-round-trip)
```bash
1
microvm run . --image ci-image --exec "make test"
```
A positional that names a directory switches `run` into sync mode. Sync mode launches an existing image, named by `--image` or by `image` in `microvm.toml`; a directory with nothing supplying an image is refused with `ERR_PRECONDITION` before any call. A positional that names a file is a daemon binary you manage yourself, and the two readings cannot collide, because a path is a directory or it is not.
## 2. What is packed and what stays home
[Section titled “2. What is packed and what stays home”](#2-what-is-packed-and-what-stays-home)
The tree is packed locally in deterministic member order. `.git`, `target`, `node_modules`, and `.venv` are skipped whole; sockets, fifos, and devices are skipped individually; symlinks are preserved as links and never followed, so a link out of the project does not pull the rest of your disk into the archive.
The pack is budgeted against the daemon’s own caps, 512 MiB of file bytes and its member limit, during the walk. An over-budget tree is `ERR_SYNC` (exit 16) naming the offending subtree, before any archive bytes are allocated and before any AWS call is made.
## 3. In the guest
[Section titled “3. In the guest”](#3-in-the-guest)
The archive is uploaded to `/workspace` in the guest, and the exec runs with `/workspace` as its working directory. The daemon spawns execs with a minimal environment, so a command that expects `PATH` gets it from the `[env]` table in `microvm.toml` or from `--launch-env KEY=VALUE`.
The `run` envelope’s `sync` key reports the workdir, the uploaded bytes, and the member count.
## 4. Bring artifacts back
[Section titled “4. Bring artifacts back”](#4-bring-artifacts-back)
Which members come back is declared in `microvm.toml` as `artifacts` globs. There is deliberately no flag spelling for a list this shape:
```toml
1
image = "ci-image"
2
exec = "make test"
3
artifacts = ["dist/**", "*.log"]
```
With that file beside the invocation, `microvm run .` needs no flags at all. A typed flag still wins over the file, and the file wins over the built-in default; the envelope’s `resolvedConfig` names which source won for each knob. [Configure the project file](/microvms-agentd/learn/operations/configure-the-project-file/) lists every key.
Afterwards, members matching the globs are written into the local directory, including when the command failed, because a failing run’s report is the artifact CI most wants. The command’s own exit code stands: `ERR_EXEC_FAILED` (exit 13) says the sandbox worked and the command in it exited non-zero, which is a different sentence from “we never got a VM”.
Only glob-matched regular-file members land, and never under `.git`. Symlinks, hardlinks, specials, unmatched members, anything attempting traversal outside the directory, and any `.git` path are skipped, because the returned archive is the VM’s word, the VM is where untrusted work runs, and a workload-written `.git/hooks/pre-commit` would execute on your machine at your next commit. With no `artifacts` globs configured the workdir is not downloaded at all, and `sync.note` says so. A download failure does not fail the run; the error lands in `sync.error` and the exec’s exit code stands.
## 5. Bake the dependencies into the image
[Section titled “5. Bake the dependencies into the image”](#5-bake-the-dependencies-into-the-image)
Every launch of a plain image pays dependency installation inside the guest, and it needs `--egress` to do it. `build --project` moves that work into the image:
```bash
1
microvm build --project . --reuse --name ci --json
```
Exactly one ecosystem’s manifest and lockfile pair must be present in the directory: `pyproject.toml` with `uv.lock`, `package.json` with `package-lock.json`, or `Cargo.toml` with `Cargo.lock`. Only that pair is zipped into the build context beside the Dockerfile; nothing else in the directory enters the shared image snapshot. The derived Dockerfile copies the pair into `/project` and installs from the lockfile (`uv sync --locked`, `npm ci`, or `cargo fetch --locked`), each of which refuses a lockfile that disagrees with its manifest.
With `--reuse`, the pair joins the content hash. The image name becomes `-`, an unchanged project reuses its image in well under a second with `reused: true`, and a lockfile-only edit builds a fresh image under a new name. The lockfile is the identity, and the layer follows it: [Platform](/microvms-agentd/internals/platform/) measured a launch-to-first-import delta of about twenty seconds for the smallest project `--project` accepts, paid for by a build about fifteen seconds longer.
Take the image identifier off the build envelope and launch the project against it:
```bash
1
IMAGE=$(microvm build --project . --reuse --name ci --json \
2
| python3 -c "import json,sys; print(json.load(sys.stdin)['data']['imageIdentifier'])")
3
microvm run . --image "$IMAGE" --exec "make test"
```
The exec environment carries no `PATH` and no `HOME`, so a Python workload should call `/project/.venv/bin/python` directly or receive a `PATH` through `--launch-env`.
## 6. Keep syncing while you work
[Section titled “6. Keep syncing while you work”](#6-keep-syncing-while-you-work)
For a kept VM, `microvm sync . --name dev` syncs a project directory into the running VM’s `/workspace`, uploading only what changed, with the same skip list and budgets as `run `. `--watch` keeps syncing on filesystem changes until Ctrl-C, and `--full` uploads the whole tree even when the guest manifest claims members are unchanged.
**For an agent.** Tell the exits apart before retrying anything. Exit 13 (`ERR_EXEC_FAILED`) is the workload’s own failure, and the artifacts are already back. Exit 16 (`ERR_SYNC`) is a pack or extraction failure on this machine’s filesystem, and the message names the subtree. Exit 12 (`ERR_PRECONDITION`) with a directory positional usually means nothing supplied an image. Read `data.sync.error` and `data.sync.note` before concluding artifacts are missing.
Next: [drive it from code](/microvms-agentd/learn/tutorial/from-code/).
# Reference
> The microvm command surface and the daemon's wire schema, generated from the files the binaries themselves emit.
## 1. What this tier is
[Section titled “1. What this tier is”](#1-what-this-tier-is)
This tier is generated. Its pages are built from two files the binaries emit about themselves: `docs/manifest.json`, the output of `microvm manifest` at version `0.8.0`, and `docs/schema.json`, the daemon’s JSON schema. Where a page states a count, the number is the length of an array in one of those files, so a page here cannot name a command or a code the binary does not have.
| Page | Holds | Members |
| ------------------------------------------------------------ | ------------------------------------------ | ------- |
| [Commands](/microvms-agentd/reference/#2-the-commands) | one page per subcommand | 28 |
| [Exit codes](/microvms-agentd/reference/exit-codes/) | process statuses and their stable codes | 17 |
| [The envelope](/microvms-agentd/reference/envelope/) | fields across the success and error shapes | 12 |
| [Response types](/microvms-agentd/reference/response-types/) | distinct payload shapes | 26 |
| [Wire schema](/microvms-agentd/reference/wire-schema/) | daemon types under `$defs` | 21 |
## 2. The commands
[Section titled “2. The commands”](#2-the-commands)
`microvm` accepts 28 commands, each with its own page. The order is the manifest’s, which is also the sidebar’s.
| Command | Summary | Response type |
| --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------- |
| [`microvm run`](/microvms-agentd/reference/commands/run/) | Build an image, launch a VM, run a command, report the cost, tear it down | `microvm.run` |
| [`microvm quickstart`](/microvms-agentd/reference/commands/quickstart/) | Zero to a live exec: provision the daemon, build, launch, run, report, tear down | `microvm.run` |
| [`microvm build`](/microvms-agentd/reference/commands/build/) | Build a MicroVM image and wait for it to be usable | `microvm.image` |
| [`microvm agent-up`](/microvms-agentd/reference/commands/agent-up/) | Bring up a VM with a coding agent in it: image, launch, model credentials, non-root user | `microvm.agent` |
| [`microvm agent-prompt`](/microvms-agentd/reference/commands/agent-prompt/) | Hand a coding agent in a running agent VM one task, headless, as the non-root user | `microvm.agent.prompt` |
| [`microvm exec`](/microvms-agentd/reference/commands/exec/) | Run one command in a MicroVM that is already running | `microvm.exec` |
| [`microvm health`](/microvms-agentd/reference/commands/health/) | Ask a running MicroVM’s daemon whether it is up, and what its identity repair did | `microvm.health` |
| [`microvm ack`](/microvms-agentd/reference/commands/ack/) | Release a finished exec’s buffered output, which starts its collection clock | `microvm.exec` |
| [`microvm kill`](/microvms-agentd/reference/commands/kill/) | Stop a running exec: SIGTERM its whole process group, SIGKILL after the daemon’s grace | `microvm.kill` |
| [`microvm ps`](/microvms-agentd/reference/commands/ps/) | List what is running in a MicroVM: every exec’s process group and its live pids | `microvm.procs` |
| [`microvm stdin`](/microvms-agentd/reference/commands/stdin/) | Write to a running exec’s stdin, and optionally close it | `microvm.stdin` |
| [`microvm cp`](/microvms-agentd/reference/commands/cp/) | Copy a file or a tar archive between here and a running MicroVM | `microvm.copy` |
| [`microvm sync`](/microvms-agentd/reference/commands/sync/) | Sync a project directory into a running MicroVM’s /workspace, uploading only what changed | `microvm.sync` |
| [`microvm attach`](/microvms-agentd/reference/commands/attach/) | Register a name for a running MicroVM this state directory did not launch | `microvm.attach` |
| [`microvm tunnel`](/microvms-agentd/reference/commands/tunnel/) | Tunnel arbitrary TCP to a guest port, so `psql` or `ssh` here reaches a server in the VM | `microvm.tunnel` |
| [`microvm port-forward`](/microvms-agentd/reference/commands/port-forward/) | Serve a guest port on localhost, so a browser here reaches a server in the VM | `microvm.port-forward` |
| [`microvm shell`](/microvms-agentd/reference/commands/shell/) | Open an interactive root shell in a running MicroVM — a real PTY, with job control, signals, and resize | `microvm.shell` |
| [`microvm suspend`](/microvms-agentd/reference/commands/suspend/) | Freeze a MicroVM. It keeps its memory, filesystem, token, and endpoint | `microvm.state` |
| [`microvm resume`](/microvms-agentd/reference/commands/resume/) | Thaw a suspended MicroVM and report its endpoint | `microvm.state` |
| [`microvm terminate`](/microvms-agentd/reference/commands/terminate/) | Tear down a MicroVM, and optionally its image and build log group | `microvm.teardown` |
| [`microvm ls`](/microvms-agentd/reference/commands/ls/) | List what this CLI created and could not confirm it deleted; `--remote` asks the account too | `microvm.runs` |
| [`microvm history`](/microvms-agentd/reference/commands/history/) | Print what was asked of one MicroVM and what the platform reported back | `microvm.history` |
| [`microvm logs`](/microvms-agentd/reference/commands/logs/) | Name an image’s build log group and print the `aws logs tail` command that reads it | `microvm.logs` |
| [`microvm cost`](/microvms-agentd/reference/commands/cost/) | What a run cost, or what a plan will cost. Every figure labelled | `microvm.cost` |
| [`microvm doctor`](/microvms-agentd/reference/commands/doctor/) | Check every prerequisite and say which one is wrong | `microvm.doctor` |
| [`microvm manifest`](/microvms-agentd/reference/commands/manifest/) | Emit the whole command surface, its exit codes, and its envelope schema | `microvm.manifest` |
| [`microvm constants`](/microvms-agentd/reference/commands/constants/) | Emit every service constraint this client believes, for the drift gate | `microvm.constants` |
| [`microvm dockerfile`](/microvms-agentd/reference/commands/dockerfile/) | Print the Dockerfile stanza that wraps any base image with agentd | `microvm.dockerfile` |
## 3. Global flags
[Section titled “3. Global flags”](#3-global-flags)
The manifest publishes 3 command-wide flags under `globalFlags`. Each parses on either side of the subcommand, so `microvm --json ls` and `microvm ls --json` are the same invocation, and none appears in any command’s own parameter table.
| Flag | Type | Default | Help |
| --------- | --------- | ------- | ------------------------------------------------------------------------------------------------------------ |
| `--json` | `boolean` | none | Emit the typed JSON envelope on stdout instead of human output |
| `--dense` | `boolean` | none | Token-lean output, for a consumer paying per token: tab-separated alone, compact one-line JSON with `--json` |
| `--quiet` | `boolean` | none | Suppress progress on stderr. Warnings still print |
The manifest marks each command with `supportsJson`, and all 28 set it. So `--json` is accepted by every command, selects the envelope described on [The envelope](/microvms-agentd/reference/envelope/), and is left out of every per-command parameter table rather than repeated 28 times.
The manifest names no command-wide flag beyond these 3, so a flag neither this table nor a command page lists is not part of that command’s surface as the manifest states it.
## 4. The annotated pages
[Section titled “4. The annotated pages”](#4-the-annotated-pages)
3 further pages share this directory and are not generated from the manifest. They were produced by a per-file documentation pass over the source tree, and every factual claim in them carries a `path:line` citation:
* [`cli`](/microvms-agentd/reference/cli/)
* [`public-api`](/microvms-agentd/reference/public-api/)
* [`rpc-tools`](/microvms-agentd/reference/rpc-tools/)
## 5. Provenance
[Section titled “5. Provenance”](#5-provenance)
This page is generated from `docs/manifest.json`, the output of `microvm manifest`, together with `docs/schema.json` for the wire-schema page. `site/scripts/gen-reference.mjs` writes it into the site’s content directory on every `pnpm run sync`, so an edit made here is overwritten by the next run.
To change the page, change the source. Regenerate the source with `mise run manifest` from the repository root; `mise run manifest:check` fails when the committed file no longer matches what the binary emits.
# CLI
> The microvm binary has twenty-eight subcommands, declared as one clap Subcommand enum in microvms-cli/src/cli.rs and built from microvms-cli/Cargo.toml:20-22.
The `microvm` binary has twenty-eight subcommands, declared as one `clap` `Subcommand` enum in `microvms-cli/src/cli.rs` and built from [`microvms-cli/Cargo.toml:20-22`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/Cargo.toml#L20-L22).
## Global flags
[Section titled “Global flags”](#global-flags)
These three are `global = true`, so they parse on either side of the subcommand. `microvm --json ls` and `microvm ls --json` are the same invocation. [`microvms-cli/src/cli.rs:63-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L63-L80).
Flags:
* `--json` — emit the typed JSON envelope on stdout instead of human output; wins over every other format, including an interactive terminal. [`microvms-cli/src/cli.rs:67-72`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L67-L72).
* `--dense` — token-lean output, for a consumer paying per token: tab-separated alone, compact one-line JSON with `--json`. [`microvms-cli/src/cli.rs:74-76`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L74-L76).
* `--quiet` — suppress progress on stderr; warnings still print. [`microvms-cli/src/cli.rs:78-80`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L78-L80).
The output format depends only on the two flags and on whether stdout is a terminal. `--json` is checked first, then `--dense`; after that, a terminal gets a ratatui surface and a pipe gets plain text. [`microvms-cli/src/envelope.rs:298-305`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/envelope.rs#L298-L305).
The manifest publishes the three as `globalFlags` (#131), in the same parameter shape as a command’s own, read off the root command’s `global = true` arguments without building the tree — building would propagate each global into every subcommand’s parameter list. [`microvms-cli/src/manifest.rs:91-95`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/manifest.rs#L91-L95). The generated Reference overview renders that array as its “Global flags” table.
## Shared flag groups
[Section titled “Shared flag groups”](#shared-flag-groups)
Four flattened `Args` structs supply the flags that repeat across commands, so a relationship like the region conflict is declared once rather than per command. `RegionFlags`, `AttachFlags` and `InfraFlags` are described here; `ConfigFlags` (`--config` / `--no-config`, flattened into `run` and `doctor`) is described under `run`. [`microvms-cli/src/cli.rs:460-461`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L460-L461), [`microvms-cli/src/cli.rs:498-499`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L498-L499), [`microvms-cli/src/cli.rs:544-545`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L544-L545), [`microvms-cli/src/cli.rs:595-596`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L595-L596).
`RegionFlags` — flattened into every command that talks to AWS. [`microvms-cli/src/cli.rs:460-472`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L460-L472).
Flags:
* `--region ` — AWS region; defaults to `$AWS_REGION`, then `$AWS_DEFAULT_REGION`, then `us-east-1`. Closed set: `us-east-1`, `us-east-2`, `us-west-2`, `eu-west-1`, `ap-northeast-1`. [`microvms-cli/src/cli.rs:462-464`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L462-L464), domain at [`microvms-cli/src/cli.rs:412-424`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L412-L424).
* `--unlisted-region ` — use a region this client has not seen carry MicroVMs; conflicts with `--region`. An unsupported region answers `AccessDeniedException` with a null message, which looks like an IAM denial, so the caller loses the real diagnostic. [`microvms-cli/src/cli.rs:466-471`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L466-L471).
`AttachFlags` — the identifiers that address a VM this invocation did not launch: the explicit triple, or a registered name standing in for it. Carried by `exec`, `health`, `ack`, `stdin`, `cp`, and `sync`. `microvms-cli/src/cli.rs`, `AttachFlags`.
Flags:
* `--endpoint ` — the VM’s endpoint, as reported by `run`. Required unless `--name` is given.
* `--agent-token ` — the agent token delivered to the VM at launch. Required unless `--name` is given.
* `--microvm-id ` — the MicroVM id, needed to mint the endpoint proxy token. Required unless `--name` is given.
* `--name ` — the name `run --keep --vm-name` registered, standing in for the whole triple. Resolved through the local name registry with zero AWS calls: the record carries the endpoint, the agent token, the MicroVM id, and the launch region (used as the region default when no `--region` flag is given). Conflicts with the explicit triple. A name this state directory never registered fails locally with `ERR_PRECONDITION`.
* `--port ` — the daemon’s port inside the guest.
* `--state-dir ` — where the local state lives: the name registry, and exec’s per-VM history. Defaults to `$MICROVM_STATE_DIR` or `~/.microvm/runs`.
`InfraFlags` — the three account-specific values the AWS commands need. Carried by `run`, `build`, and `doctor`. [`microvms-cli/src/cli.rs:594-608`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L594-L608).
Flags:
* `--bucket ` — S3 bucket for the build artifact; defaults to `$MICROVM_BUCKET`. [`microvms-cli/src/cli.rs:597-599`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L597-L599).
* `--build-role-arn ` — build role ARN; defaults to `$MICROVM_BUILD_ROLE_ARN`. [`microvms-cli/src/cli.rs:601-603`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L601-L603).
* `--execution-role-arn ` — execution role ARN; defaults to `$MICROVM_EXECUTION_ROLE_ARN`. [`microvms-cli/src/cli.rs:605-607`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L605-L607).
## run
[Section titled “run”](#run)
```plaintext
1
microvm run [OPTIONS] [BINARY]
```
Builds an image, launches a VM, runs a command, reports the cost, and tears the VM down. Teardown is the default so that a closed laptop does not leave a billable VM.
[`microvms-cli/src/commands/lifecycle.rs:119`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L119)
Flags:
* `[BINARY_OR_DIR]` — the aarch64 agentd binary to bake in as the image CMD (ignored when `--image` names an image to launch instead), or a directory to sync. The two readings cannot collide: a path is a directory or it is not. See “Sync mode” below. `microvms-cli/src/cli.rs`, `RunArgs::binary`.
* `--image ` — launch this existing image instead of building one. Takes an ARN or a bare image name: a name is resolved to its ARN through the account’s image listing (exact match, every page read) before the launch, with a progress line naming the resolved ARN. An identifier already shaped like an ARN passes through with zero extra calls. The envelope’s `imageName` reports the launched image’s own name (the ARN’s last colon segment), never the per-invocation default a build would have used. A name that resolves to nothing fails locally with `ERR_PRECONDITION` naming the name and suggesting `microvm build` — the service’s own answer to a bare name is HTTP 400 “Malformed ARN”, which says nothing about names. [`microvms-cli/src/commands/lifecycle.rs:965-979`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L965-L979), resolution in [`microvms-core/src/control/image.rs:411-475`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-core/src/control/image.rs#L411-L475).
* `--image-version ` — launch this exact image version instead of the image’s latest active one. Omitted takes whatever `latestActiveImageVersion` is at the moment the call lands, which is right for the ordinary case and wrong for the two that matter: a canary wants the version it just built rather than whatever became latest while it was starting, and a rollback wants the known-good version, which “latest” cannot name once a bad version is the latest one. A version the control plane has set `INACTIVE` refuses to launch when named here — measured, the answer is HTTP 404 `No active version found for MicroVM image and version `, which is what makes a retire real rather than advisory. Free text rather than a closed set, because a version’s legal values are an account fact only `ListManagedMicrovmImageVersions` can answer; the constraint that *is* knowable is checked before any call, so an empty version, one over 2048 characters, or one containing whitespace anywhere fails locally with `ERR_INVALID_ARG` and the reason. A version pasted from a terminal carries a trailing newline, which is that case. `microvms-core/src/control/mod.rs`’s `require_valid_version`, wiring in `microvms-cli/src/commands/lifecycle.rs`.
* `--artifact-uri ` — where the build artifact already is; `microvms-core` builds the artifact bytes and takes the URI but does not upload. [`microvms-cli/src/cli.rs:671-672`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L671-L672).
* `--exec ` — a shell command to run in the VM. When it is omitted, the run only launches and tears down, which is how you check that an image boots. [`microvms-cli/src/cli.rs:677-678`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L677-L678).
* `--name ` — image name; defaults to a per-invocation name, because reusing a name can trigger a `clientToken` replay that wedges the image. [`microvms-cli/src/cli.rs:682-683`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L682-L683).
* `--vm-name ` — register a local name for the kept VM, so later commands can say `--name ` (attached commands) or use the name as the positional (suspend, resume, terminate, history) instead of pasting identifiers. Requires `--keep`. The name is a purely local fact in the state directory’s registry (`/names/.json`, written owner-only because the record carries the agent token), costs zero AWS calls, and is released when a terminate is accepted. Names take ASCII letters, digits, `-` and `_`, at most 128 bytes, and never a MicroVM id prefix (`microvm-` is the service’s real prefix, measured live; `mvm-` is the test fixtures’) — that exclusion is what lets every identifier-taking command tell a name from a MicroVM id. A name registered to a live VM is refused locally with `ERR_NAME_TAKEN` (exit 14) before any billable call. `microvms-cli/src/cli.rs`, `RunArgs::vm_name`; registry in `microvms-cli/src/ledger.rs`, `Names`.
* `--memory ` — baseline MiB, selecting a documented size class; default `2048`. Closed set: `512`, `1024`, `2048`, `4096`, `8192`. [`microvms-cli/src/cli.rs:690-691`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L690-L691).
* `--dockerfile ` — a Dockerfile to use instead of the library’s default; its `FROM` must match the base. [`microvms-cli/src/cli.rs:694-695`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L694-L695).
* `--log-group ` / `--log-stream ` — build-log destination, applied when this invocation builds; same semantics as `build`’s flags (the stream is a prefix the client suffixes with `/<16 hex>` per build). Both are also `microvm.toml` keys (`log-group`, `log-stream`), with the flag winning per knob; a stream with no group from either layer is refused locally. `microvms-cli/src/cli.rs`, merge in `microvms-cli/src/commands/lifecycle.rs`’s `merge_config`.
* `--repair-identity` — request identity repair and additional OS capabilities. Inspect health for `identity_degraded`; requested capabilities do not guarantee every repair succeeds.
* `--egress` — request the managed `INTERNET_EGRESS` connector. Omission does not disable internet access. Also the `egress` config key. See [Networking](../internals/networking.md).
* `--egress-network-connector ` — attach an existing custom VPC connector; repeatable. Conflicts with `--egress`. Config key: `egress-network-connectors`; explicit flags replace the configured list. Internet isolation requires a VPC without an IGW or NAT gateway and no alternative internet path.
* `--deny-egress` — set proxy variables pointing to an unreachable local proxy. Reports `best-effort`; workloads can bypass it. This is not network enforcement and conflicts with `--egress`. Also the `deny-egress` config key.
* `--launch-env ` — set one launch-environment variable for every exec in the VM; repeatable. Delivered in the same `runHookPayload` as the agent token, at launch, so it never touches the shared image snapshot and never touches disk. The daemon applies it as the *base* environment of every exec, with `exec --env` on the same key winning. Same parser as `exec --env`, so the first `=` splits, an empty VALUE is legal, and a missing `=` or empty KEY is refused at parse time. The whole payload shares a 4096-byte ceiling with the token, checked locally before the launch: an over-budget env fails with the byte count and the env’s share of it, rather than as an AWS `ValidationException` after the call. The total must fit after serialization; large credential sets belong on `microvm cp` after bootstrap or on a role the workload assumes. `microvms-cli/src/cli.rs`, wiring in `microvms-cli/src/commands/lifecycle.rs`.
* `--keep` — retain the VM and image; the VM continues to incur charges until stopped, suspended, or terminated according to its lifecycle. `--keep` is also the only case in which the envelope’s `agentToken` carries a value: a run that tears its VM down emits the key as `null`, because stdout outlives the process and the token would name a VM that no longer exists (#161). [`microvms-cli/src/cli.rs:800-801`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L800-L801), `microvms-cli/src/render.rs`, `RunOutcome::to_data`.
* `--timeout ` — how long to wait for the exec, in seconds; default `300`. [`microvms-cli/src/cli.rs:838-839`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L838-L839).
* `--max-idle-sec ` — suspend the VM after this much inbound-traffic idleness; default `600`. [`microvms-cli/src/cli.rs:842-843`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L842-L843).
* `--suspended-sec ` — terminate the VM after this long suspended; a resume attempted after this window fails because the VM no longer exists. Default `600`. [`microvms-cli/src/cli.rs:846-847`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L846-L847).
* `--auto-resume` — let the platform resume a suspended VM on an incoming request, instead of requiring an explicit `microvm resume`; omitted by default. Sets `idlePolicy.autoResumeEnabled` on the launch. Also a `microvm.toml` key (`auto-resume`). `microvms-cli/src/cli.rs`, merge in `merge_config`.
* `--max-duration-sec ` — hard ceiling on the VM’s life; refused above 28800 before any call. Default `3600`. [`microvms-cli/src/cli.rs:854-855`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L854-L855).
* `--port ` — the daemon’s port inside the guest. [`microvms-cli/src/cli.rs:858-859`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L858-L859).
* `--state-dir ` — where the run ledger is written; defaults to `$MICROVM_STATE_DIR` or `~/.microvm/runs`. [`microvms-cli/src/cli.rs:862-863`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L862-L863).
* `--config ` — read this project config file instead of `./microvm.toml`. Naming a file that does not exist is refused with `ERR_CONFIG`: a typed path that is wrong must not silently become “no config”. Conflicts with `--no-config`. `microvms-cli/src/cli.rs`, `ConfigFlags`.
* `--no-config` — ignore any `microvm.toml`, even a malformed one; flags and built-in defaults apply. `microvms-cli/src/cli.rs`, `ConfigFlags`.
* Plus `RegionFlags` and `InfraFlags`. [`microvms-cli/src/cli.rs:875-879`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L875-L879).
Precedence, when a config file is in play: a typed flag beats the file, and the file beats the built-in default. “Typed” is read off the parse (`clap`’s `value_source`), not off the value, so `--memory 2048` overrides a file that says `4096` even though 2048 is also the default. The merge happens in exactly one place (`merge_config` in `microvms-cli/src/commands/lifecycle.rs`, per-knob precedence in `config::pick`) and its outcome is reported in the success envelope’s `resolvedConfig` key — each knob’s winning value and the source it came from (`flag`, `config`, `env`, or `default`; `env` appears only on the region, the one knob whose chain continues past the file into `$AWS_REGION`/`$AWS_DEFAULT_REGION`) — so a caller never has to re-derive which source won. `egress` and `denyEgress` are two of those knobs; the conservative assessment of internet isolation is a separate top-level envelope key, `egressPosture` (`open` | `unsealed` | `best-effort` | `sealed`), because a request flag is not an answer — see docs/TRUST.md, **Egress**. One deliberate pairing rule: a typed `BINARY` positional with no typed `--image` suppresses the file’s `image`, because `run` builds exactly when the merged image is absent, and a file that silently won that pair would run the caller’s tests against a stale pinned image. A positional that names a directory does not suppress it — sync mode launches, so the file’s pinned image is exactly what `run .` wants. See “Project config” below for the file itself.
Sync mode (issue #72): a positional that names a directory switches `run` into a pack-run-collect round trip — `microvm run . --image ci-image --exec "make test"` is the headline spelling. The tree is packed locally (deterministic member order; `.git`, `target`, `node_modules`, and `.venv` skipped whole; sockets, fifos, and devices skipped individually; symlinks preserved as links, never followed), uploaded to `/workspace` in the guest, and the exec runs with `/workspace` as its working directory. The pack is budgeted against the daemon’s own caps — 512 MiB of file bytes, 100 000 members — during the walk, so an over-budget tree is `ERR_SYNC` naming the offending subtree before any archive bytes are allocated or any AWS call is made.
Afterwards — including when the exec exited non-zero, because a failing run’s report is the artifact CI most wants — the workdir comes back and the members matching the config file’s `artifacts` globs are written into the directory. Only glob-matched regular-file members land, and never under `.git`: symlinks, hardlinks, specials, unmatched members, anything attempting traversal outside the directory, and any `.git` path are skipped, because the returned archive is the VM’s word, the VM is where untrusted work runs, and a workload-written `.git/hooks/pre-commit` would execute on the host at the caller’s next commit. With no `artifacts` globs configured the workdir is not downloaded at all, and the `sync` key says so in a `note`. A download failure does not fail the run — by then the exec’s result is already in hand, and discarding a green test run because `make clean` removed the workdir would be the report lying — the error lands in `sync.error` and the exec’s own exit code stands.
Sync mode launches an existing image (`--image`, or `image` pinned in microvm.toml); a directory with nothing supplying an image is refused with `ERR_PRECONDITION` before any call. A local pack or extraction failure is `ERR_SYNC` (exit 16). The envelope’s `sync` key reports the workdir, uploaded bytes and member count, and each artifact brought back with its size — or `error` / `note` for the two no-artifact shapes. `microvms-cli/src/sync.rs`; wiring in `microvms-cli/src/commands/lifecycle.rs`.
## build
[Section titled “build”](#build)
```plaintext
1
microvm build [OPTIONS] [BINARY]
```
Builds a MicroVM image and waits for it to be usable. Nothing is torn down afterward. The image is the durable artifact, and because its snapshot has a one-week minimum retention, deleting it early saves nothing.
[`microvms-cli/src/commands/lifecycle.rs:1296`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L1296)
Flags:
* `[BINARY]` — the aarch64 agentd binary to bake in as the image CMD. Omitted, the CLI provisions its own version’s release asset and caches it under the state directory; `$MICROVM_AGENTD` names a binary without touching the command line. [`microvms-cli/src/cli.rs:918-919`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L918-L919).
* `--artifact-uri ` — where the build artifact already is, as an `s3://` URI. [`microvms-cli/src/cli.rs:928-929`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L928-L929).
* `--name ` — image name; defaults to a per-invocation name. [`microvms-cli/src/cli.rs:932-933`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L932-L933).
* `--memory ` — baseline MiB, selecting a documented size class; default `2048`. [`microvms-cli/src/cli.rs:936-937`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L936-L937).
* `--dockerfile ` — a Dockerfile to use instead of the library’s default. [`microvms-cli/src/cli.rs:940-941`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L940-L941).
* `--project ` — bake an environment layer from the directory’s dependency files (#74). Exactly one ecosystem’s manifest+lockfile pair must be present — `pyproject.toml`+`uv.lock`, `package.json`+`package-lock.json`, or `Cargo.toml`+`Cargo.lock` — and only that pair enters the shared image snapshot, under names fixed by the ecosystem (nothing else in the directory can enter, which is what keeps a `.env` out of a snapshot every VM shares). The derived Dockerfile copies the pair into the working directory (`/project` when none is named) and installs from the lockfile with the lockfile-faithful spelling: `uv sync --locked`, `npm ci`, or `cargo fetch --locked`, each refusing a lockfile that disagrees with its manifest rather than quietly re-resolving. Launches from the image then start with dependencies already installed — the 31–48% env-init share of launch time `docs/STRATEGY.md` measures is paid once at build. A caller `--dockerfile` that never mentions the lockfile is refused before the upload, because it would bake no layer while building cleanly. Missing lockfile, missing manifest, and two-ecosystem directories are each refused naming their remedy. Measured launch delta, and why in-guest code should call `/project/.venv/bin/python` directly rather than `uv` (an exec sees no `PATH`): `docs/PLATFORM.md`, “A baked environment layer removes the guest’s env init”. `microvms-cli/src/commands/lifecycle.rs`’s `read_project_files`, the entries in `microvms-core/src/control/artifact.rs`.
* `--base-image-version ` — pin the managed base image to one version instead of taking the service’s default. Without this a build floats: the managed base’s version list is not static — `al2023-1` carried one version in June and two by July — so two builds of identical inputs weeks apart can sit on different bases and neither recorded which. The build succeeds either way; the difference shows up in the guest. The legal values come from `ListManagedMicrovmImageVersions`, which `microvm doctor` prints as its `base-image-versions` check, and they are bare integers for a managed base (`0`, `1`) where a custom image’s versions are `1.0`. A bogus pin is refused by the service before anything is created (HTTP 400 `No managed MicroVM Image with arn and version 999 is available`), but it costs the artifact upload first, so the `Version` shape’s own constraints — non-empty, at most 2048 characters, no whitespace anywhere — are checked locally before the upload. Note that the value comes back **normalised**: a build pinned with `1` reads back `baseImageVersion: "1.0"` from `GetMicrovmImageVersion`, so the echoed value cannot be fed back into a request. `microvms-cli/src/commands/lifecycle.rs`’s `BuildSpec`, guard in `microvms-core/src/control/image.rs`’s `create_image`.
* `--log-group ` — CloudWatch log group for the build’s logs, instead of the service-created `/aws/lambda-microvms/`. Letters, digits, and `_ - / . #` only, up to 512 characters, validated locally before the artifact upload. The build role must be able to write to whatever this names (`logs:CreateLogGroup`/`CreateLogStream`/`PutLogEvents`); a group outside a granted prefix builds with **no logs at all**, the same silent outcome as the wrong-prefix policy in `docs/PLATFORM.md`.
* `--log-stream ` — log stream name **prefix** inside `--log-group`; requires it. The platform’s `logging.logStream` member is an exact stream name — prefixes are unsupported — and one image build is three VMs writing three streams (docker build, Graviton 3 snapshot, Graviton 4 snapshot), so a fixed configured name would collapse every build’s logs into one indistinguishable stream. The client therefore appends `/<16 hex>` of fresh randomness per build attempt, and the envelope reports the resolved exact name as `logStream` — the only place it exists. No `:` or `*` (the shape’s pattern is `[^:*]*`); up to 495 characters (the platform’s 512 minus the suffix’s 17). See `docs/PLATFORM.md`, “An image build is three VMs and three log streams”.
* `--repair-identity` — request identity repair and additional OS capabilities. Inspect health for `identity_degraded`; requested capabilities do not guarantee every repair succeeds.
* `--reuse` — reuse an existing image whose build inputs match, instead of building. Computes a sha256 over the build inputs (the daemon binary’s bytes, the Dockerfile, and — with `--project` — the manifest and lockfile, names and bytes both), derives the image name `-` — where the prefix is `--name` or the stable stem `microvm-cli` — and checks the listing for that exact name. A hit skips the build entirely and reports the existing image with `reused: true` in the envelope; a miss builds under the derived name, so the next invocation with the same inputs hits. The hash is in the name because recreating an image under a previously-used fixed name can serve a stale snapshot (measured; the same hazard class as the clientToken replay in `docs/PLATFORM.md`) — content-keying gives both properties at once: unchanged inputs reuse their image, changed inputs get a fresh name and a fresh build. With `--project` that is #74’s promise: two projects with identical dependency files share a layer, and a lockfile edit builds a fresh one. `--memory` is not part of the identity, so a reused image keeps the size class it was created with; the envelope’s `size` is the requested class and the text says so. [`microvms-cli/src/commands/lifecycle.rs:1370-1376`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/lifecycle.rs#L1370-L1376), the hash at `microvms-core/src/control/artifact.rs`’s `artifact_content_hash`.
* `--port ` — the daemon’s port inside the guest. [`microvms-cli/src/cli.rs:1010-1011`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1010-L1011).
* Plus `RegionFlags` and `InfraFlags`. [`microvms-cli/src/cli.rs:1013-1017`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1013-L1017).
The success envelope always carries `reused` (`false` for a plain build) and `logStream` (`null` when no `--log-stream` was configured; a reused image also reports `null`, because no build ran and no stream was resolved), so a consumer never guards for either key. `microvms-cli/src/commands/mod.rs`.
## agent-up
[Section titled “agent-up”](#agent-up)
```plaintext
1
microvm agent-up [OPTIONS] --vm-name [BINARY]
```
Brings up a VM with a coding agent in it: image, launch, model credentials, non-root user. The L3 helper (`docs/AGENT-VMS.md`) composes `build --reuse`, `run --keep --egress --vm-name`, a Bedrock bearer token minted from the caller’s own AWS credentials, and the file uploads and `chown` the coding-agents example performed by hand. Two paths, decided by one local registry read: a name this state directory has not registered means build (or reuse), launch, provision, register; a name it has registered means attach, mint a fresh token, re-install the same files, and report `vmReused: true`, with no build and no launch. That refresh path is how a 12-hour token is renewed on a long-lived VM. Teardown is `microvm terminate `.
`microvms-cli/src/commands/agent.rs`
Flags:
* `[BINARY]`: the aarch64 agentd binary to bake in. Omitted provisions this CLI’s own release asset, the same chain `run` and `build` use; the envelope’s `agentd` key reports the path and source. `microvms-cli/src/cli.rs`, `AgentUpArgs::binary`.
* `--vm-name `: the local name to register for the VM, and the handle every later command uses. Required, because an agent VM is kept by definition. Same grammar as `run --vm-name`. A name already registered to a live VM is not a collision here: it selects the refresh path. A name registered to a torn record (empty MicroVM id, a process died mid-register) is refused with `ERR_NAME_TAKEN` and the record’s path. `microvms-cli/src/cli.rs`, `AgentUpArgs::vm_name`.
* `--agent `: which agent to install. Closed set: `claude-code`, `codex`. Repeatable; omitted means `claude-code`; a repeated value collapses to one row. On the refresh path, omitted keeps the agents and models the guest marker names, and a typed `--agent` is the caller changing them. `microvms-cli/src/cli.rs`, `AgentArg`.
* `--claude-model `: the Bedrock inference-profile id Claude Code uses; default `global.anthropic.claude-opus-5`, from the profile table. `microvms-core/src/agents/profile.rs`, `CLAUDE_CODE`.
* `--codex-model `: the Bedrock model id Codex uses; default `global.openai.gpt-5.6-sol`, from the profile table. `microvms-core/src/agents/profile.rs`, `CODEX`.
* `--claude-version ` / `--codex-version `: pin the agent’s npm package to this version (`@anthropic-ai/claude-code@`, `@openai/codex@`). Unpinned, the image carries the registry’s latest at build time. A pin changes the Dockerfile text and therefore the image’s reuse hash. `microvms-cli/src/cli.rs`, `AgentUpArgs::claude_version`.
* `--project `: a local directory to upload into `/workspace` after launch, packed the way `run ` packs: same skip list (`.git`, `target`, `node_modules`, `.venv`), same budgets, packed before any AWS call so an unreadable tree costs nothing. The upload lands before the install’s `chown`, so the agent owns the tree. Bring results back with `microvm cp --tar vm:/workspace --name `. Also honored on the refresh path. `microvms-cli/src/cli.rs`, `AgentUpArgs::project`.
* `--memory `: baseline MiB; default `1024`, not `run`’s `2048`: a 4 GiB always-present ceiling at half the floor cost, which fits peaky agent sessions. Closed set as for `run`. `microvms-cli/src/cli.rs`, `AgentUpArgs::memory`; `microvms-core/src/agents/mod.rs`, `DEFAULT_SIZE`.
* `--token-ttl-hours `: how long the Bedrock bearer token lives; default and ceiling `12`. Core refuses a lifetime outside `(0, 12 h]` with `ERR_INVALID_ARG`; the service additionally caps validity at the signing credentials’ own expiry. `microvms-core/src/agents/bedrock.rs`, `MAX_LIFETIME`.
* `--max-idle-sec `: suspend the VM after this much inbound-traffic idleness; default `600`.
* `--suspended-sec `: terminate the VM after this long suspended; default `600`.
* `--auto-resume`: let the platform resume a suspended VM on an incoming request; omitted by default.
* `--max-duration-sec `: hard ceiling on the VM’s life; refused above 28800 before any call. Default `3600`.
* `--port `: the daemon’s port inside the guest.
* `--state-dir `: where the run ledger and name registry live; defaults to `$MICROVM_STATE_DIR` or `~/.microvm/runs`.
* Plus `RegionFlags` and `InfraFlags`. The fresh path requires the bucket, the build role, and the execution role (`ERR_PRECONDITION` naming the missing one); the refresh path makes no control-plane call before the attach, and takes the region from the flag when typed, else from the name record.
Egress is always requested, because neither agent reaches Bedrock without it. The image is named `agent-vm--`, the hash over the same inputs `build --reuse` covers, so an unchanged daemon and agent set reuse their image and a version pin builds a fresh one. Provisioning writes `/workspace/.agent-env` (mode `0600`; `HOME`, `PATH`, `AWS_REGION`, and each agent’s credential and model lines), `/workspace/.codex/config.toml` when Codex is present, and the marker `/workspace/.agent-vm.json` (mode `0644`) naming the installed agents and models, then runs one root `chown -R 1000:1000 /workspace`. The token travels only as a file over the authenticated channel; it is never an argv element and never in the launch payload. `microvms-core/src/agents/mod.rs`, `provisioning_files` and `install_access`.
A failure after the launch and before the registration (token mint, project upload, credential install, or an interrupt) tears the VM down, because a VM with no name and no credentials is one nobody can use; the failure envelope carries `microvmId`, `imageIdentifier`, `leaked`, and `terminateAccepted`. The image stays, as the durable artifact. The name is registered last, over a VM every step succeeded on; a registry write failure is `ERR_PRECONDITION` with the identifier triple in `data`. No new exit row: every failure maps onto the existing table.
The success envelope’s type is `microvm.agent`; its keys are `vmName`, `microvmId`, `endpoint`, `agentToken`, `imageIdentifier`, `imageName`, `imageReused`, `vmReused`, `agents` (`[{agent, model, cliVersion, headlessCommand}]`), `credentialExpiresAt` (epoch seconds), `workdir`, `project` (`{workdir, uploadedBytes, uploadedMembers}` or `null`), and `agentd`. On the refresh path the two image keys and `imageReused` are `null`, because no image was built or looked up. `agents[].headlessCommand` is the exact template `agent-prompt` runs with `` where the quoted task goes, so a caller who wants `exec --stream --user 1000 --group 1000` over it can spell the command without knowing the profile. `microvms-cli/src/commands/mod.rs`, the `agent-up` row.
## agent-prompt
[Section titled “agent-prompt”](#agent-prompt)
```plaintext
1
microvm agent-prompt [OPTIONS]
```
Hands a coding agent in a running agent VM one task, headless, as the non-root user. Runs the agent’s headless command (`claude -p --allowedTools Bash,Read,Edit,Write,Grep,Glob` or `codex exec --skip-git-repo-check -s workspace-write `) as uid 1000, gid 1000, in `/workspace`, with `. /workspace/.agent-env &&` in front so the credentials `agent-up` installed are the environment; the request’s own `env` map is empty, so the token never rides in a request body. By default it waits and acks, exactly as `exec` does; `--detach` starts and returns the exec id.
`microvms-cli/src/commands/agent.rs`
Flags:
* ``: the task, as prose. Passed to the headless command single-quoted for `sh`. Required. An empty or whitespace-only task is refused locally with `ERR_INVALID_ARG` before the attach, so it costs zero calls. `microvms-cli/src/cli.rs`, `AgentPromptArgs::task`.
* `--agent `: which installed agent to prompt; closed set `claude-code`, `codex`. Omitted reads the guest marker `/workspace/.agent-vm.json` and takes the one agent it names; a marker naming two agents is refused with `ERR_PRECONDITION` listing both and suggesting `--agent`. A typed `--agent` still reads the marker when it can, so the prompt uses the model the VM was provisioned with; a VM with no marker and a typed agent runs the profile’s defaults; a VM with no marker and no flag is `ERR_PRECONDITION` naming `agent-up`. `microvms-cli/src/commands/agent.rs`, `choose_spec`; `microvms-core/src/agents/mod.rs`, `installed_agents`.
* `--timeout `: how long this process waits for the agent, in seconds; default `900`, because agent tasks run minutes, not seconds. A client-side deadline: the exec on the daemon’s side is started with no wall-clock budget of its own. `microvms-cli/src/cli.rs`, `AgentPromptArgs::timeout`.
* `--detach`: start the agent and return immediately, without waiting and without acking; the envelope reports `phase: running` and a `null` exit code. `microvm exec --poll --name ` reads it back and `microvm ack` releases it, as for `exec --detach`. `microvms-cli/src/cli.rs`, `AgentPromptArgs::detach`.
* `--exec-id `: use this exec id instead of a fresh one, making a retry idempotent. `microvms-cli/src/cli.rs`, `AgentPromptArgs::exec_id`.
* Plus `AttachFlags` and `RegionFlags`; `--name ` is the usual spelling, the name `agent-up --vm-name` registered.
The success envelope’s type is `microvm.agent.prompt`, under its own discriminant because a consumer that learned `microvm.exec` is reading a command it chose and this one ran a template it did not. Its keys are `exec`’s plus which agent ran: `execId`, `agent`, `model`, `phase`, `exitCode`, `stdout`, `stderr`, `truncated`. A non-zero agent exit earns `ERR_EXEC_FAILED` on a success envelope, as `exec` does. `microvms-cli/src/commands/mod.rs`, the `agent-prompt` row; `microvms-cli/src/commands/attached.rs`, `render_exec_as`.
## exec
[Section titled “exec”](#exec)
```plaintext
1
microvm exec [OPTIONS] --endpoint --agent-token --microvm-id [COMMAND]
```
Runs one command in a MicroVM that is already running. The single subcommand covers four uses. It can start a command and wait for it, start one and stream its output (`--stream`), start one and feed its stdin (`--stdin`), or read an existing exec (`--poll`).
[`microvms-cli/src/commands/attached.rs:178`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L178)
Flags:
* `[COMMAND]` — a shell command to run in the VM; omitted only with `--poll`. [`microvms-cli/src/cli.rs:1023-1024`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1023-L1024).
* `--timeout ` — how long to wait for the command, in seconds; default `300`. [`microvms-cli/src/cli.rs:1027-1028`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1027-L1028).
* `--cwd ` — working directory. When omitted, the command inherits the image WORKDIR, which is not the same as passing `/`. [`microvms-cli/src/cli.rs:1034-1035`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1034-L1035).
* `--env ` — set one environment variable for the command; repeatable. These flags are the child’s whole environment: the daemon starts every exec from an empty one and applies exactly this map, so there is no inherited PATH to append to. Split at the first `=`, so a value may itself contain `=`; an empty VALUE is legal (`--env EMPTY=`), and a missing `=` or an empty KEY is refused at parse time. [`microvms-cli/src/cli.rs:1050-1051`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1050-L1051).
* `--user ` — numeric uid to run the command as; omitted runs as the daemon’s own user. Numeric because that is the protocol’s type and the daemon’s mechanism (`Command::uid`, between fork and exec) — a name would need an `/etc/passwd` lookup inside a guest whose base image may not have one. [`microvms-cli/src/cli.rs:1060-1061`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1060-L1061).
* `--group ` — numeric gid to run the command as; omitted keeps the daemon’s own group. [`microvms-cli/src/cli.rs:1064-1065`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1064-L1065).
* `--exec-id ` — use this exec id instead of a fresh one, making a retry idempotent; the daemon returns success for a known id without spawning a second child. [`microvms-cli/src/cli.rs:1084-1085`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1084-L1085).
* `--poll ` — read an existing exec’s status and output instead of starting anything; read-only server-side, does not ack. Conflicts with `--exec-id`, `--stream`, `--stdin`, `--cwd`, `--detach`, `--env`, `--user`, `--group`. [`microvms-cli/src/cli.rs:1093-1094`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1093-L1094).
* `--detach` — start the command and return immediately, without waiting and without acking; prints the exec id and `phase: running`. Conflicts with `--stream` and `--stdin`. [`microvms-cli/src/cli.rs:1113-1114`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1113-L1114).
* `--stream` — stream output as it arrives rather than waiting for the whole thing; under `--json` or into a pipe this writes NDJSON. [`microvms-cli/src/cli.rs:1123-1124`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1123-L1124).
* `--from-offset ` — resume a stream at this byte offset; requires `--stream`. [`microvms-cli/src/cli.rs:1131-1132`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1131-L1132).
* `--stdin` — give the command a stdin pipe, feed it this process’s stdin, then close it. [`microvms-cli/src/cli.rs:1139-1140`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1139-L1140).
* `--reap` — signal the command’s whole process group once its own child exits, so nothing it backgrounded outlives it (`reap_group_on_exit: true` on the wire). Off by default, and the default is a contract: a backgrounded grandchild that inherited the output pipe keeps running and keeps writing. With the flag, `ps` afterwards shows the group empty and the exec’s `writersMayBeAlive` reads false because the linger saw EOF. Conflicts with `--poll`. [`microvms-cli/src/cli.rs:1151-1152`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1151-L1152).
* `--kill-on-timeout` — on `ERR_TIMEOUT`, send one `POST /v1/exec/{id}/kill` and put the daemon’s verdict in the failure envelope’s `data.killed`. A plain `--timeout` is a client-side deadline that abandons the exec and leaves it running in the guest; this turns it into a stop. The exit code stays `ERR_TIMEOUT`, because the deadline is still what ended the wait. Conflicts with `--poll`, `--detach`, and `--stream`. [`microvms-cli/src/cli.rs:1162-1163`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1162-L1163).
* Plus `AttachFlags` and `RegionFlags`. [`microvms-cli/src/cli.rs:1165-1169`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1165-L1169).
A timeout is not a stop. `ERR_TIMEOUT`’s suggestions say so in as many words and name `microvm kill ` as the remedy, beside the older fact that the exec and its output are untouched and re-pollable. [`microvms-cli/src/exit.rs:402`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/exit.rs#L402).
## health
[Section titled “health”](#health)
```plaintext
1
microvm health [OPTIONS] --endpoint --agent-token --microvm-id
```
Asks a running MicroVM’s daemon whether it is up and what its identity repair did. This is the one command that reports `identityDegraded` and `diskUnderPressure`, and either flag is a reason to drain the VM rather than keep scheduling onto it.
It also reports `busy` and `execs`, which is what makes this the command an orchestrator loops on to hold a long-running VM alive. The platform measures idleness by inbound traffic through the endpoint proxy, and that proxy terminates outside the guest, so a request sent from *inside* the VM cannot reset the idle timer — only a poll from outside counts, and this poll is that traffic. `busy` is what makes the loop informed rather than unconditional: it is true only while some exec is actually running, so an exec that exited and is waiting to be acked reads false. `execs` counts every registered entry in any phase, so `busy: false` with a non-zero count is a VM holding unacked output somebody still has to collect before terminating it. See `docs/PROTOCOL.md`, “Idle policy, and why liveness is a field rather than a route”.
And it reports `hooks`, the daemon’s own record of every lifecycle-hook invocation it observed — `[{hook, firedAt}]`, oldest first, with `firedAt` in epoch seconds on the daemon’s clock — beside `hooksDropped`, how many invocations the daemon’s capped log discarded. The platform writes no CloudWatch logs for the validate hook, so this is the only place “did my validate hook even run?” is answerable; `validate` and `ready` fire in the snapshot VM before the snapshot is taken, so a launched VM commonly reports them from the memory image it restored from. Each observation the poll returns is also appended to the VM’s local history as a `hookObserved` event, deduplicated on the (hook, firedAt) pair, so re-polling appends nothing and the record survives the VM. One caveat travels with the field: the daemon’s hook routes are unauthenticated and reachable over loopback from inside the guest, so a hostile workload can forge *additional* observations by posting the hook paths itself — it can never remove or alter real ones, and the capped log keeps the earliest entries, which are the platform’s. Both fields are empty/zero against a daemon that predates them.
[`microvms-cli/src/commands/attached.rs:635`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L635)
Flags:
* `AttachFlags` and `RegionFlags` only; this command has no arguments of its own. [`microvms-cli/src/cli.rs:1172-1179`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1172-L1179).
## ack
[Section titled “ack”](#ack)
```plaintext
1
microvm ack [OPTIONS] --endpoint --agent-token --microvm-id
```
Releases a finished exec’s buffered output, which starts its collection clock. A second ack returns a 409 because the first one already released the output.
[`microvms-cli/src/commands/attached.rs:811`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L811)
Flags:
* `` — the exec whose output to release. Required. [`microvms-cli/src/cli.rs:1346-1347`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1346-L1347).
* Plus `AttachFlags` and `RegionFlags`. [`microvms-cli/src/cli.rs:1349-1353`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1349-L1353).
## kill
[Section titled “kill”](#kill)
```plaintext
1
microvm kill [OPTIONS] --endpoint --agent-token --microvm-id
```
Stops a running exec: SIGTERM to its whole process group, SIGKILL after the daemon’s grace (`POST /v1/exec/{id}/kill`). This is the stop button `exec --timeout` is not — a client-side timeout abandons an exec and leaves it running in the guest (issue #156). The VM is addressed the way every attached command addresses it, by `--name` or the identifier triple, and the exec by its id, which `exec --detach`, `exec --exec-id`, and `ps` all print.
The envelope carries the daemon’s own verdict. `killed: false` with exit 0 means the process group had already exited, which is the outcome a kill was asking for, so `microvm kill x-1 && collect` runs its second half either way. An unknown exec id is the daemon’s 404, arriving as `ERR_PROTOCOL` with `data.kind: NotFound`.
[`microvms-cli/src/commands/attached.rs:852`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L852)
Flags:
* `` — the exec whose process group to signal. Required. [`microvms-cli/src/cli.rs:1359-1360`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1359-L1360).
* Plus `AttachFlags` and `RegionFlags`.
## ps
[Section titled “ps”](#ps)
```plaintext
1
microvm ps [OPTIONS] --endpoint --agent-token --microvm-id
```
Lists every exec’s process group and its live pids (`GET /v1/procs`). The daemon reads `/proc` itself, so this works against the al2023 base image, which ships no `ps` (issue #157); nothing needs installing in the image for this command. The row worth reading is `childExited: true` with a non-empty `pids`: a command that finished while something it backgrounded did not, which `health`’s `busy` cannot show because busy is about the exec’s own child. The `execId` on that row is what `kill` takes.
`data.procs` is one object per registered exec in any phase: `{execId, pgid, startedAt, childExited, reap, pids}`. `pgid` is `null` (never absent) when the daemon captured none; `startedAt` is epoch seconds on the daemon’s clock; `reap` echoes `exec --reap`. Zombies are not live and are not listed. `--dense` prints one TSV row per group — exec id, pgid, childExited, pid count, startedAt — exec id first, so `cut -f1` feeds `kill`.
[`microvms-cli/src/commands/attached.rs:904`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L904)
Flags:
* `AttachFlags` and `RegionFlags` only; this command has no arguments of its own. [`microvms-cli/src/cli.rs:1369-1376`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/cli.rs#L1369-L1376).
## stdin
[Section titled “stdin”](#stdin)
```plaintext
1
microvm stdin [OPTIONS] --endpoint --agent-token --microvm-id
```
Writes to a running exec’s stdin and optionally closes it. It only works on an exec started with `exec --stdin`, and it is the only way to close the pipe.
[`microvms-cli/src/commands/attached.rs:995`](https://github.com/laithalsaadoon/microvms-agentd/blob/741287b92afc4c7dae9e4091166582a759791f76/microvms-cli/src/commands/attached.rs#L995)
Flags:
* `