Wire protocol v1

The lifecycle hook paths are fixed by the platform, so they stay unversioned. Everything this project owns lives under /v1/. Every response carries a microvms-agentd-version header.

Routes

Hooks are served under /aws/lambda-microvms/runtime/v1/, abbreviated HOOKS below. That prefix is fixed by the service, so a daemon serving a bare /run never gets bootstrapped.

Route	Auth	Purpose
POST HOOKS/ready	none (platform hook)	image-build readiness probe
POST HOOKS/validate	none (platform hook)	image-build validation probe
POST HOOKS/run	none (platform hook)	one-shot token bootstrap from runHookPayload, plus the optional launch environment
POST HOOKS/suspend	none (platform hook)	acknowledged and logged
POST HOOKS/resume	none (platform hook)	acknowledged; signals in-memory state loss
POST HOOKS/terminate	none (platform hook)	acknowledged; begins graceful shutdown
POST /v1/exec/start	bearer	start a command under a caller-minted exec_id
GET /v1/exec/{id}	bearer	poll status and output; never mutates
GET /v1/exec/{id}/stream?offset=	bearer	follow output as SSE from a byte offset
POST /v1/exec/{id}/stdin	bearer	write to a child’s stdin, or signal EOF
POST /v1/exec/{id}/ack	bearer	release output, enter TTL collection
POST /v1/exec/{id}/kill	bearer	signal escalation to the process group
GET /v1/procs	bearer	process accounting: every exec’s group and its live pids, read from /proc
GET /v1/tcp?port=&identity=	bearer	WebSocket relay to 127.0.0.1:<port> in the guest; close codes carry the outcome
PUT /v1/fs/tar	bearer	streaming tar upload and confined extraction
GET /v1/fs/tar?path=	bearer	streaming tar download
PUT /v1/fs/file	bearer	write one file
GET /v1/fs/file?path=&start_line=&end_line=	bearer	read one file, or a 1-based inclusive line range of it
GET /v1/health	none	liveness, version, bootstrap state, exec-activity, hook observations
GET /v1/schema	none	this contract as a JSON Schema document: every route, shape, status, and limit

Every hook invocation is recorded and reported on /v1/health. The daemon records each invocation in memory — the hook’s name and the daemon’s clock, in arrival order — before the handler does any work or chooses a status, and GET /v1/health reports the log as hooks ([{hook, fired_at}], oldest first) beside hooks_dropped, a count of invocations past the log’s cap. A run hook that arrives malformed or answers 409 still leaves a run observation: the record is of the invocation, not the verdict. The platform writes no CloudWatch logs for the validate hook, so this record is the only trace that hook leaves anywhere. Build-time hooks (ready, validate) fire in the snapshot VM before the snapshot is taken, so their in-memory records ride the memory image into every VM launched from it and are visible on the first health poll.

One caveat is part of this contract rather than a footnote. The hook routes are unauthenticated and arrive over loopback indistinguishably from an in-VM process (see “Trust boundary” below), so a hostile workload can forge ADDITIONAL observations by posting the hook paths itself. It cannot remove or alter real ones: the daemon records before it responds, and the capped log keeps the earliest entries — the platform’s real lifecycle firings, which arrive before anything runs in the guest — while later spam is dropped and counted in hooks_dropped. Read an observation as “the daemon saw this hook fire”, never as “only the platform could have fired it”. Both fields are #[serde(default)] like busy and execs, and for the same reason: an older daemon omits them, and empty/zero is the honest reading.

Rules that exist because a defect proved them necessary

Each of these rules comes from a real bug found during the Harbor PR #2469 integration. They are part of the protocol contract, so implementations must follow them.

Bootstrap is one-shot, and a replay of the identical token succeeds. A first /run installs the token and returns 200. A later /run carrying the same token returns 200, because the platform may retry its own hook and must not be told the VM is broken. A later /run carrying a different token returns 409 and changes nothing. The model in model/ checks this over every interleaving, including a racing in-VM caller.

Control routes answer 503 before bootstrap. They do not answer 404, and they do not drop the connection.

A missing or malformed body key is 400, never 404. Clients map 404 onto “file not found”. Because of that mapping, returning 404 for a protocol typo makes the client believe an artifact is absent when it is not. One defect went undetected this way.

Authorization is decided before any body byte is read. An unauthenticated caller must not be able to make the daemon allocate. Rejected requests still drain a small body so pooled client connections keep working; larger ones close.

Token comparison happens on bytes, in constant time. Comparing str values raises on non-ASCII input in some languages. Any caller controls the header, so a non-ASCII header value could crash the connection.

No exception on the parse, auth, or routing path may drop a connection. A catch-all returns 500. Raw TLS handshake bytes get a 400 and a debug log, since something in the platform’s path probes the port with TLS first.

cwd is omitted when unset. When the client sends no working directory, the daemon emits no cd prefix, and the child inherits the daemon’s own working directory. Because the daemon is the container CMD, that directory is the image WORKDIR. Forcing / breaks prebuilt-image tasks and defeats any harness that discovers the image workdir with pwd.

Exec is idempotent on a caller-minted exec_id. A retried /exec/start returns success without spawning a second child. Polling is read-only. Output lives until the caller acks, and only acked entries are collected. The Python predecessor unlinked output files at child exit, which destroyed anything a backgrounded grandchild wrote afterward.

A shell wraps the command only when the caller asks for one. An argv array execs directly. shell: true wraps in sh -c with the command as a single argument. The predecessor’s brace-group wrapper turned empty and comment-terminated commands into syntax errors and let an unbalanced } escape the group.

Tar extraction mirrors the CPython data filter contract. In-tree symlinks are preserved, because harnesses legitimately pack them. Absolute link targets are refused. Relative targets must resolve under the root, using normpath semantics rather than realpath. Symlinks resolve relative to their own directory, while hard links resolve against the archive root. Member count and total size are capped. Modes are applied after content lands.

A symlink an archive wrote cannot redirect a later member, and the kernel is what enforces that. The normpath rule above judges a member at the depth its name implies, and that is not the same as the depth the write reaches once a symlink is in the path. So the daemon opens the extraction root once and creates every member relative to that descriptor with openat2, using RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS. A member whose path would traverse a symlink is refused with 400 rather than written somewhere else. openat2 needs Linux 5.6 or newer. On an older kernel the syscall answers ENOSYS and extraction answers 500, which is a refusal rather than a silent fall back to the weaker check.

Bodies stream to disk, and caps are enforced on the wire. The predecessor buffered whole archives in memory on a VM whose baseline can be 512 MiB, where an OOM-killed daemon is unrecoverable. It also measured archive size inside the gzip with block, where the stream is unflushed. There, tell() reported 10 bytes for a 327-byte archive, so the size guard almost never fired.

Output is bounded, and truncation is marked explicitly. A post-exit linger deadline bounds how long the daemon waits on grandchildren still holding the pipe.

A grandchild outlives its exec unless the caller asks otherwise, and /v1/procs is how anyone finds it. Each exec runs in its own process group, and until now only /v1/exec/{id}/kill ever signalled that group: a command that backgrounded something and exited left it running with nothing able to enumerate it (issue #157, measured 2026-09-11 — a ticker still counting six seconds after its exec reported exit 0). GET /v1/procs answers {procs: [{exec_id, pgid, started_at, child_exited, reap, pids}]}, one entry per registered exec in any phase. pids are the live pids whose process group is pgid, read from /proc/<pid>/stat inside the guest, so the route needs no ps in the image; zombies are not live and are not listed, and an entry whose pgid was never captured lists none. child_exited is read from the same write-once marker busy uses, never from the polled result, so an acked exec still reads exited. started_at is epoch seconds on the daemon’s clock, like a hook’s fired_at. The row worth reading is child_exited: true with a non-empty pids: that is the survivor, and its exec_id is what the kill route takes.

reap_group_on_exit: true on /v1/exec/start is the opt-in that closes the gap at the source. When set, the daemon runs the kill route’s SIGTERM-then-SIGKILL escalation against the group as soon as the child’s exit is observed and before the linger begins, so the survivors die, the pipe closes, and writers_may_be_alive reads false because the linger saw EOF rather than its deadline. A timed-out exec is already escalated and is not escalated twice. The escalation’s grace ends as soon as /proc shows no live member of the group — not when killpg(pgid, 0) fails, because the daemon is PID 1 in the guest and a signalled grandchild stays a zombie that still answers that probe. The flag defaults to false and the default is the contract above: a backgrounded grandchild that keeps writing is a feature for the caller who did not set it. reap on a procs entry echoes what the start request asked.

The launch environment

The run-hook payload may carry an env map alongside agent_token. It becomes the base environment of every later exec.

{"runHookPayload": "{\"agent_token\": \"...\", \"env\": {\"KEY\": \"VALUE\"}}"}

The per-request env wins on a key both set. The daemon applies the launch env first and the request’s own map second, so a launch env is a default for the whole VM and a request is the specific thing happening now. A caller who never sends a launch env sees no change: the child’s environment is the request’s map and nothing else, exactly as before.

The agent token never becomes part of it. env_clear() still runs, so the daemon’s own environment reaches no child, and the launch env and the token are separate parameters through the whole install path — there is no field a refactor could forward one into. Proven by a test that spawns /usr/bin/env with a launch env installed and asserts the child’s environment is exactly that map: an extra variable of any name fails it, which is what a leak would look like.

Only the first successful bootstrap sets it. A replay of the identical token answers 200 and leaves the installed env alone, and a conflicting token answers 409 and leaves it alone. Without that, a caller who cannot win the token could still rewrite the environment every later child runs in.

Every value is a string, and a malformed env is 400 naming the problem. A non-object env, or a value that is a number or a nested object, is refused with a body that names the key or the shape. The body never quotes a value, because the payload carries the token.

Unknown payload keys are ignored. A 400 at this hook makes the platform terminate the VM before forwarding any traffic, so a newer client sending a field this daemon has never heard of must still be able to bootstrap it. Forward compatibility here is the difference between an ignored field and a dead launch.

The token and the env share one 4096-byte budget, measured in UTF-8 bytes and inclusive (PLATFORM.md). The daemon cannot enforce what the platform already rejected — an over-ceiling payload never reaches the guest — so the check is on the client side, in microvms-core’s RunHookPayload::for_launch, and it fires before any control-plane call. botocore does not enforce the ceiling either, so without that local check there is no signal at all until AWS answers with a ValidationException on a member the caller did not know they were filling. The refusal names the byte count, the ceiling, and how much of it the env is, because “4142 bytes, ceiling 4096” alone does not say whether to shorten the token or drop a variable. One bearer token fits with room to spare; a set of AWS session credentials does not, and that is what makes this ceiling reachable in practice.

Line-ranged text reads

GET /v1/fs/file takes optional start_line and end_line. The semantics are the AI SDK harness contract’s readTextFile, copied rather than chosen, because this route is what that method is implemented on top of:

Both bounds are 1-based and inclusive. start_line=2&end_line=4 is three lines. start_line absent means 1 and end_line absent means through EOF.

An end_line past the last line reads through EOF without an error. Lines 1..1000 of a twelve-line file is a 200 carrying twelve lines, never a 416. A start_line past the last line is an empty 200, not a 404: the file is there and the window is empty, which is a different fact from the file being absent.

A line owns its terminating newline. Lines 1..2 and lines 3..5 concatenate back into the file rather than losing a separator at the seam. A last line with no trailing newline has none to own and is returned as it is.

start_line=0 and end_line < start_line are 400. Neither is 416: 416 is about a range the file cannot satisfy, and both of these are ranges no file could, so a client sent to look at the file would be looking in the wrong place. A non-integer bound is also 400, and it does not masquerade as the missing-path refusal.

A range still streams. The read is filtered chunk by chunk and stops reading once the window closes, so lines 1..5 of a large file cost the first chunk. Nothing buffers a file to slice it, for the same reason nothing buffers an upload: an OOM-killed daemon in a MicroVM is unrecoverable.

With no range the response is byte-identical to what it always was. The un-ranged read hands back the reader stream untouched, so the path every existing caller uses does not acquire the range feature’s bug surface.

Idle policy, and why liveness is a field rather than a route

The platform measures idleness by inbound traffic through the endpoint proxy (PLATFORM.md, “idlePolicy”). A workload holding an outbound connection, or one simply computing for hours, receives none, so auto-suspend can freeze a VM mid-work. Multi-hour agent runs are the real case.

A guest-side request cannot fix this, and the daemon does not pretend otherwise. The endpoint proxy terminates outside the VM and forwards over loopback — measured, PLATFORM.md, “The platform’s own hook arrives over loopback”. A request an in-VM process sends to the daemon’s own port therefore never reaches the thing counting traffic. A “keep myself alive” route would be a keepalive that keeps nothing alive, and it would be discovered as broken by a multi-hour run auto-suspending mid-work, which is the failure it was added to prevent.

So GET /v1/health carries busy and execs, for an orchestrator outside the VM. The orchestrator’s own poll is the inbound traffic, and busy is what makes that poll informed rather than unconditional. The assertion of liveness is therefore repeated and is explicitly the caller’s, which is the property that rules out the daemon self-keepaliving: a hung process would then bill to the 8-hour maximumDurationInSeconds ceiling with nobody having asked.

busy means producing, not unfinished. An exec whose child has exited and whose result is waiting to be acked is not busy — nothing is running, and holding a VM alive at baseline billing for a command that is over is the mistake this distinction exists to prevent. execs counts every registered entry in any phase, so busy: false with a non-zero count is a VM holding unacked output somebody still has to collect before terminating it.

Both fields default to false and 0 when absent, unlike every other field on the response. The daemon is baked into an image while a client is installed separately, so a current client routinely talks to a daemon from whenever that image was built; a required field would make a health call fail outright against an older daemon, turning a missing signal into an unreachable VM.

Streaming and stdin

Both features serve one consumer, an agent harness running inside the VM. The harness emits output for minutes and may need a prompt written to it. Polling serves neither need well, because it re-sends the whole buffer each time and truncates at the output cap.

The stream is a read-only view of the exec. An exec is a server-side record keyed by its caller-minted exec_id. Attaching, detaching, or dropping a connection must not affect the command. Both views must keep working, so poll returns the buffer, stream follows it, and neither disturbs the other.

Resume is by byte offset. ?offset=N yields exactly the bytes after N, so a client that reconnects can pick up where it left off. For comparison, E2B’s reattach takes no offset, so a reconnecting E2B client loses everything produced during the gap. A reattach past the retained window gets an explicit gap event naming the missing range. Without that event, the client would keep streaming and never learn that bytes were skipped.

SSE is used because it can carry a typed terminal event. A raw chunked byte stream cannot distinguish a finished command from a dropped connection, because the bytes are identical in both cases. The stream therefore emits a typed exit event carrying the status and then ends. Keep-alive comments fill silences, so an agent harness thinking for two minutes does not look like a dead connection.

stdin is opt-in and a separate request. A command that does not ask for stdin gets Stdio::null(), so nothing inherits a surprise descriptor. Writing to a command that did not request stdin returns 409. Writes go to POST /v1/exec/{id}/stdin and are never multiplexed onto the output connection. Because the two connections are separate, a dropped attach cannot corrupt stdin. EOF is an explicit signal rather than inferred, because a child reading stdin cannot exit until the daemon drops its own handle. Child::wait() drops the child’s copy of the handle, not the daemon’s.

Reconnect-at-cursor across a real suspend, measured

The claim above — that resume by byte offset is what distinguishes this from a backend whose reattach loses the gap — was asserted from the daemon’s side only. The cut under unit test is a response body ending without an exit frame, which is the condition the reconnect keys on, but nothing had put a real platform suspend across a live exec.

Measured 2026-08-15, us-east-1, through Session.spawn in microvms-js against a guest ticking once a second and also appending each tick to a file, so the guest’s own record is an oracle independent of the stream under test:

	Observation
Before the suspend	tick-1, tick-2, tick-3 delivered on the handle’s stdout
Suspend / resume	SUSPENDED, held ~20s, RUNNING
The handle already held	Resumed at tick-4 — contiguous, no restart, no duplicate
Gaps recorded	None, on either the held handle or a fresh one
A fresh handle at offset: 0	Replayed 200 of 200 ticks, indices contiguous 1..200
The guest’s own file	Agreed with the stream at every point

So the property holds against the thing it was designed for, and it holds in both shapes: the handle a caller was already reading recovers at its cursor, and a new handle for the same exec_id replays the whole output across the suspension. The exec itself is untouched by the freeze, which is the PLATFORM.md finding this depends on.

The other three ExecProcess properties were measured on the same VM. stdout and stderr arrive as separate streams carrying only their own bytes, from one interleaved SSE channel with a per-frame discriminator. wait() reports the daemon’s real exit code — 0 for a success, 42 for exit 42 — because it reads the exec record rather than inferring from the stream ending. kill() terminates the process group and a second call succeeds rather than 404ing, and wait() afterwards reports signal: 15 with no exit code, so a killed build cannot read as passing.

Credentials for a caller’s own connection

Session::connect_headers(port) and connect_subprotocols(port) hand a caller what it needs to open its own connection to some other port on the same VM — the header pair for HTTPS, the three subprotocols for a WebSocket.

Both mint a token scoped to the port they name, and that is load-bearing rather than an implementation detail. The service scopes a proxy token at mint time, so a token minted for the agent port answers 403 Access to port denied for any other port and close code 1006 with no reason on a WebSocket. An implementation that reused the session’s cached token would therefore return a correct-looking port value behind a credential that does not authorize it — which is what this client did until the live run of 2026-08-15 caught it. See PLATFORM.md, “allowedPorts is a union of three forms”.

The scope is extended rather than replaced: a mint for a new port asks for every port already covered plus that one, so warming the cache for a workload port does not cost the session access to its own. One extra control-plane call per new port, cached thereafter, and proxy_mint_count() is the observable that says so.

A caller debugging a failed WebSocket should retry the same port over HTTPS with connect_headers. Every handshake failure is 1006 and none of them says why; the HTTPS request distinguishes 403 (wrong scope) from 502 (right scope, nothing listening).

Trust boundary

The platform’s /run hook arrives from 127.0.0.1 and is indistinguishable at the socket level from a request sent by a process inside the VM (measured; see PLATFORM.md). Filtering by source address therefore cannot separate the platform from an in-VM process, so it provides no protection here.

The remaining defenses, all checked in model/, are the following:

1. Bootstrap is one-shot, so a losing racer never replaces the winner’s token.
2. A post-bootstrap hijack attempt is refused at the hook with 409 and at the control API with 401.
3. The agent token never enters an exec’d child’s environment.

One risk remains. The design assumes the daemon is the container CMD and that the harness issues its first exec only after readiness. The daemon does not enforce this invariant. A base image that starts its own background process before bootstrap breaks it. model/ includes that configuration and reports the counterexample path, so the consequence of breaking the invariant is a checked result rather than a prediction. Enforcing the invariant is the responsibility of whoever builds the image, not of this daemon.