[chruffins]

Summary

Experimental native macOS image builds, opt-in and off by default. This PR covers building, publishing and pulling. The operator tooling for creating a pinned base is stacked on top in #536, and the unrelated stuck-image-pull fix that was bundled here moved to #535.

POST /builds takes an installed, digest-pinned darwin/arm64 base and a source tarball containing build.sh. It runs the script as root in an isolated VZ guest, strictly stops the guest, exports its closed storage, and publishes an OCI machine image to this server's registry as builds/<id>@sha256:…. Mac hosts create instances from that digest (pull on create). External distribution uses the existing POST /pushes with registry credentials. Linux builds are unchanged.

Existing and new flows

Existing Linux build (unchanged):
  POST /builds -> Linux parser/source staging -> shared queue -> Linux builder -> publish

New macOS build:
  POST /builds (bounded multipart, pinned base, source with build.sh)
  -> shared queue -> isolated VZ guest from base -> /bin/sh build.sh as root
  -> remove source -> strict graceful stop + confirmed VMM exit
  -> export closed storage, validate -> destroy builder
  -> stream and verify OCI blobs -> commit manifest last
  -> wait for the exact published digest -> Ready

Distribution:
  POST /pushes (digest, external repo, credentials) -> external registry
  POST /images (digest, darwin/arm64, borrowed credentials) on each Mac host

Scope changes

• The versioned JSON recipe is replaced by a fixed entrypoint: the source archive must contain build.sh at its root. The strict recipe parser and the recipe_hash provenance field are gone; provenance.source_hash already pins exactly what ran. Failures report the script's exit code.
• Base publication tooling (cmd/publish-macos-base, scripts/macos, its docs and CI job) moved to macOS guests 7/7: base publication tooling #536, stacked on this branch.
• The stuck-image-pull fix moved to Fix stuck image pulls after a failed unpack #535 against main.
• Removed earlier: storage admission and reservations, storage metrics, build progress logs, cross-repository tag promotion and pipeline.py.

Behavior

• HTTP activation requires macos_only: true and macos_builds: true. Linux servers reject the machine selector.
• The base must be Ready, canonical SHA256-pinned, darwin/arm64, and provisioned with the system GuestService. CPU and memory inherit from the base.
• Jobs on the same base serialize. Identity is preserved, so a copy must not run concurrently on any host.
• Networking defaults to isolated. machine_network_mode=egress enables unrestricted VZ NAT, not a domain allowlist.
• Source extraction happens on the host into a private directory: links, devices, duplicates, path escapes and decompression bombs fail before anything reaches the VM. Guest stdout and stderr are discarded; build logs contain no guest output.

Known gaps

• No disk admission. Run machine builds serially on a dedicated host with enough free space for the base copy, the export and the published layer.
• No durable pre-commit digest record. A crash between manifest commit and job completion re-runs the build. It does not reconcile the earlier publication.
• No machine-build metrics.
• Native Apple-silicon validation of this trimmed branch has not been run.

Testing

• go test -race ./lib/builds -count=1 — pass
• go test -race ./cmd/api/api -run 'Machine|MacOS|HostMachine|LinuxCreateDefaults|Capabilit|Build' -count=1 — pass
• go test -race ./lib/images -run 'MacOS|TagImage|Bundle|Delta' -count=1 — pass
• go vet ./lib/builds ./cmd/api/api ./lib/providers ./lib/images ./lib/instances — pass; lib/images also type-checks for darwin/arm64
• Not run: the full lib/images and cmd/api/api suites, which need KVM and network permissions. Not run: darwin-only files in lib/instances (need cgo and the vz-shim binary), native Apple-silicon builds, cross-host pulls, cold boots, and CI.

Change size

65 files, +5,043 / −466 against main:

Category	Files	Added	Removed
Code	38	2,307	70
Tests	22	2,163	4
Docs and config	4	152	1
Generated lib/oapi/oapi.go	1	421	391

Seven of the code files are one-line lib/instances call-site changes: stopInstance and tryGuestShutdown take a strict flag instead of having wrapper functions.

Rollout

Remaining gates before production: approved fresh base with final audit and graceful-stop evidence (tooling in #536); choice of external registry and a 26–28 GB push test; cold-boot validation of the promoted digest on an authorized Mac; Apple licence review for distributing macOS-derived images.

Disk deltas for derivative machine images (commit 2ea7464)

A machine layer can now be a delta: it patches the boot disk unpacked from the layers beneath it. A derivative's manifest lists the base's layer, then one delta layer. Unpacking writes only the changed 64 KiB chunks over the base disk in place. The delta layer keeps the same budget, diff ID and entry checks as other machine layers, and its entries must arrive in order: delta.json, disk.img.patch, aux.img, config.json.

Measured on a QA guest: a derivative that adds a 1 GB file and runs the clean build script changes 5.1% of 64 KiB chunks. Its gzip size is 1.22 GB. The base layer is about 30.8 GB. A 1 MiB chunk size gives 1.57 GB.

Publishing (commit de945c0): a derivative built from a base in the internal registry mounts the base layers by digest and uploads only the delta layer and its config.

• The build token gets pull on that one base repository (builds/...) and push on its own build repository.
• The registry answers a blob mount with 201 when it already stores the blob, and otherwise opens an upload session as before.
• The auth middleware refuses a mount whose source the token cannot pull from.
• The publisher checks the base manifest and config against their digests, and rejects a base that is not darwin/arm64 with matching layers.
• Bases outside the internal registry, and builds without a base, publish standalone as before.
• Standalone publishes now use a repo_access token instead of the legacy repos claim. The registry accepts both.

Known gaps:

• No garbage collection rule yet. Deleting a base that a derivative still references would break a later re-pull of that derivative.
• A derivative still unpacks the full base on the host, so local disk use is unchanged. APFS cloning of the base is the local optimization.
• The blob store is keyed by digest and shared across repositories, so a token that can read a blob's digest can already read it from any repository. The mount check narrows what a mount can link; it does not isolate the store. Fixing that is a separate change.
• Deltas assume the base disk on the build host is the same bytes as the base's disk.img layer. The derivative is not re-hashed against the base on pull.

Tests: go test -race ./lib/images -run 'MacOSDelta|UnpackMachine' and go test -race ./lib/builds -run 'Derived|BlobMount' cover the round trip (changed chunks only, aux and config replaced), a base of the wrong size, a missing base, malformed or out-of-order manifests, incomplete layers, and wrong entry order.

[chruffins]

Actual full-size transport QA passed against lib/registry.New(...).Handler() with an empty filesystem-backed destination: streamed the real 28,150,700,918-byte (26.22GiB) compressed installed macOS machine layer plus config over loopback HTTP using monolithic POST /blobs/uploads/?digest=sha256:.... No disk seeding at destination. Server verified streaming digests, then the test independently rehashed persisted blobs, published the original manifest last and fetched it to verify the original manifest SHA256; tag was404 before publication.

Runner/supervisor exit0,47.96s total, sampled peak process-group RSS17.98MiB, minimum free disk491.61GiB. Bounded by1.5GiB RSS/60GiB free disk/20min deadline, nice10/GOMAXPROCS1/GOMEMLIMIT256MiB; no VM ran during upload. Sampled RSS does not include OS cache or account for physical APFS allocation.

This demonstrates a viable streamed transport for the concrete publisher, not a completed PR5 publisher/build API. Cache namespace avoided image conversion; registry auth middleware, remote credentials/redirect/retry/resume and ambiguous commit reconciliation were not exercised. Default go-containerregistry remote.Write PATCH still buffers upload data in0.20.6 even with a disk BlobPutHandler; it was not fixed or passed here. Separately, the earlier pulled artifact passed real normal-API cold boot/lifecycle/recovery; it was not freshly pulled from this upload destination, so these are not an end-to-end build→upload→pull→boot result.

[chruffins]

Simplification update at ed21906: standalone production MachineBuildRunner replaced by internal MachineBuildBackend executing machine-specific phases through ordinary CreateBuild/runBuild. Normal manager owns queue, persisted request, deadline, private source staging/hash verification, status/log completion, provenance and image-readiness gate. Machine resources inherit base; unsupported Linux builder/cache/secret options fail admission; recovered source is verified before start. Source helper is shared without adding a Linux source-size cap. Normal-manager synthetic tests prove ready/failed status, source consumption and no publication after provision failure. Build race suite passed 3 runs excluding exactly five existing Linux config-volume tests requiring unavailable mkfs.ext4; focused image/machine tests, CGO0 machine/source tests and vet pass. CGO0 VZ OCI round-trip remains unsupported. No concrete VZ/GuestService driver, streamed publisher, recipe, secret/log stream or public HTTP machine mode: still a foundation.

All PRs remain draft. This is source-level/synthetic validation, not a new live boot or production build proof. Default Codex independent review was attempted but is still blocked by authentication (HTTP401). Published atomically after checking reviewer heads with explicit per-ref force-with-lease; prior heads preserved locally.

[chruffins]

Published 951e8f68, restacked on #500 (base updated), with explicit lease against prior ed219069. Still draft.

Implemented concrete GuestService/VZ machine driver, bounded tar.gz/versioned JSON recipe admission, instance-locked strict stop/export, normal-job cancellation/quarantine, and scoped-auth streaming publisher with independent persisted blob verification and manifest-last digest verification. Darwin shutdown now schedules power-off after acceptance so vsock teardown cannot race the reply.

Native isolated QA passed: ordinary queued build → root provisioning → source sanitation → acknowledged no-force shutdown + owned VMM exit → matching export → authenticated28.85GB streaming publication → shared image-readiness gate → fresh empty destination/cache HTTP pull → cold boot → root exec marker + source-absence check. Run483.74s, sampled test-process peak36.875MiB, minimum free313.17GiB. First attempt correctly refused export/publication when shutdown acknowledgement was lost. Final cold-boot test cleanup used ordinary short-timeout forced fallback; that stop was not used for export.

Evidence limits: activation remains internal Go CreateBuild, not HTTP POST /builds/SDK. Base was a manually installed isolated development template, not an automatically provisioned or credential-sanitized production image. Identity exclusivity preserved; no rekey/fork/parallel claim. Common safe recipe logs, clean-base policy, storage quotas, native repeat/cancel/recovery and Linux runtime regressions remain gates.

Repeated focused race regressions and supported full build suite pass; exactly five existing mkfs.ext4-dependent tests excluded on Darwin. Selected CGO0 synthetic tests and vet pass. Default independent autoreview remains401 authentication-blocked, not a completed clean review. All QA VMMs stopped/storage closed; original API and identity slot preserved. See updated docs/macos-builds.md for contracts and remaining gates.

[chruffins]

Published 0983495e, explicit lease against 951e8f68, still draft/base#500.

Default-off macos_builds now wires the concrete backend into the actual server provider. Requires macos_only and an explicit host registry URL. Existing authenticated POST /builds accepts machine_base_image (installed SHA256-pinned base), source, timeout_seconds, memory_mb, cpus and tags. Source64MiB/fields64KiB/full HTTP body65MiB (before OpenAPI validation); duplicate/unknown/Linux/cache/secret fields and malformed numbers rejected. Linux source parser remains unchanged; Linux servers explicitly reject the machine selector. OpenAPI and generated Go multipart request types updated; external language SDK release pipelines were not operated.

**Real HTTP/provider QA passed:**401 missing/invalid auth,403 read-only scope,202 POST /builds, ordinary queue → real guest provisioning/sanitation → acknowledged no-force export → verified scoped streaming publication → shared readiness → Ready. POST /instances cold-booted the returned image_ref; normal authenticated exec WebSocket verified UID0, the new marker and source absence. New output cold boot used converted cache; previous native run separately proved fresh pull. Private development template remains unsuitable for external distribution without credential sanitation. All QA storage closed/API stopped/slot released; original instance preserved.

Repeated focused API/config/provider/scopes race tests, full config/provider/scopes race tests and vet pass. Default independent autoreview remains401-blocked; no clean review claim. Still gates: external SDK releases, safe recipe logs, clean-base policy, storage quotas, native cancellation/recovery/repeat and Linux runtime validation. Updated docs/macos-builds.md includes opt-in configuration and curl example.

[github-actions]

-->

✱ stlc build

✅ go code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

116 files generated at 9d705f1 (pushed)

go get github.com/kernel/hypeman-go-staging@9d705f1c5208c176d90f7b123d2bada0784f8364

✅ python code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

232 files generated at bffdb28 (pushed)

✅ typescript code · compare

Your SDK build was successful.

generate ✅ → bootstrap ✅ → format ✅

138 files generated at 664289b (pushed)

Diagnostics: ❗ 0 new / 1 total error, 💡 0 new / 5 total note

Level	Code	Message	Targets

Build metadata

Build	bd_76dyxKGK-compact-fish
Timestamp	2026-10-29T20:11:20.023Z
stlc	8413509
Spec hash	871e1d46cee2
Config hash	659c3687c3f0

---

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-10-29 20:11:58 UTC