Allow a container hotplugged into an already-running LinuxPod VM
(addContainer after create()) to have a virtiofs (directory-share)
rootfs, not just an ext4 virtio-blk image, on the cloud-hypervisor
backend.
Host side:
- Generalize CHHotplugProvider.hotplug to branch on the rootfs
runtimeOptions (.virtioblk / .virtiofs / unsupported), reusing the
per-tag virtiofsd/vm.add-fs machinery factored into a new
ensureVirtiofsDevice helper. Make registerMounts the sole writer of the
per-VM mount registry so the rootfs is always first and mounts are not
double-registered (also fixes a latent ordering bug on the block path).
Exclude the rootfs from the /run/virtiofs holding-dir loop in
addContainer.
Guest side (fixes a pre-existing, feature-independent bug):
- The addContainer-after-create (runtime hotplug) path had never been
exercised by any test. cloud-hypervisor adds hot-plugged
virtio-blk/virtio-fs devices to the guest PCI bus but the guest does not
auto-probe them, so a mount by /dev/vdX or virtio-fs tag fails (ENOENT /
tag-not-found) for ALL CH hotplug.
- vminitd's mount RPC now, on a mount failure for a hot-plug-candidate
source (virtiofs type or /dev/vd* source), forces /sys/bus/pci/rescan
and retries with a bounded wait. Scoped so boot-time and non-device
mounts are unaffected.
Tests / build:
- First coverage of LinuxPod.addContainer after create() (CH-only; VZ
has no runtime hotplug): pod hotplug virtiofs rootfs and pod hotplug
block rootfs, registered in a Linux-only test list, plus a
directory-rootfs unpack helper.
- make linux-integration gains an optional FILTER passthrough (make
linux-integration FILTER="pod hotplug").
Signed-off-by: michael_crosby <michael_crosby@apple.com>
apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
Michael Crosby