apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
Move the MiB rounding from LinuxContainer into
`VZVirtualMachineInstance`, so all callers are covered in one place and
the alignment constraint stays with the VMM layer that owns it.
This will also ensure LinuxPod can get memory alignment for free.
- Closes#51. Again.
- No real notable API changes. The largest change code-wise is in
Vminitd. The connection loop is set up immediately in the new API by
calling runConnections() on the client, and this blocks until either the
task it's running on is cancelled or beginGracefulShutdown is called.
Because of this, either we'd have to modify our API to have
VirtualMachineAgent implementations call some run() method, or we just
put runConnections in an internal task on Vminitd. This change uses the
latter approach as it's simpler.
- Updated dependencies to pick up latest protobuf and grpc-swift-2
fixes.
- ClientBootstrap.withConnectedSocket(fd).wait() registers the vsock fd
with epoll/kqueue immediately. vminitd sends its HTTP/2 SETTINGS frame
right away upon accepting the connection, so those bytes can arrive on
the fd — and be read by NIO — before the gRPC pipeline is installed.
With a bare pipeline, NIO has nowhere to send the bytes and discards
them.
- Buffer until ClientConnectionHandler is in the pipeline so it can
process the SETTINGS frame and fire .ready, then schedule delivery of
the buffered data with assumeIsolatedUnsafeUnchecked().execute.
- To help produce more accurate merged logs, defines a
`StderrLogHandler` in the vminitd Application.swift that writes
timestamps to the boot log with millisecond precision and format
identical to that of the log files created using the `--log-root` arg on
`container`.
- Updates hawkeye to latest version (v6.5.1).
- In vminitd, use the Server event loop group for the gRPC server.
Co-authored-by: Danny Canter <danny_canter@apple.com>
- No real notable API changes. The largest change code-wise is in
Vminitd. The connection loop is set up immediately in the new API by
calling runConnections() on the client, and this blocks until either the
task it's running on is cancelled or beginGracefulShutdown is called.
Because of this, either we'd have to modify our API to have
VirtualMachineAgent implementations call some run() method, or we just
put runConnections in an internal task on Vminitd. This change uses the
latter approach as it's simpler.
- Updated dependencies to pick up latest protobuf and grpc-swift-2
fixes.
- ClientBootstrap.withConnectedSocket(fd).wait() registers the vsock fd
with epoll/kqueue immediately. vminitd sends its HTTP/2 SETTINGS frame
right away upon accepting the connection, so those bytes can arrive on
the fd — and be read by NIO — before the gRPC pipeline is installed.
With a bare pipeline, NIO has nowhere to send the bytes and discards
them.
- Buffer until ClientConnectionHandler is in the pipeline so it can
process the SETTINGS frame and fire .ready, then schedule delivery of
the buffered data with assumeIsolatedUnsafeUnchecked().execute.
- To help produce more accurate merged logs, defines a
`StderrLogHandler` in the vminitd Application.swift that writes
timestamps to the boot log with millisecond precision and format
identical to that of the log files created using the `--log-root` arg on
`container`.
- Updates hawkeye to latest version (v6.5.1).
---------
Co-authored-by: Danny Canter <danny_canter@apple.com>
This adds support to LinuxContainer to be able to provide an optional
writable layer. This is useful as today the rootfs size is determined by
the size of the image block itself which is set in stone at unpack time
of the image. This leaves a lot to be desired because a user might want
a larger or smaller rootfs size, but it's not configurable on a per
container basis because of this. To support this, we can pass an
additional writable block device that we can overlayfs with the image
contents in the guest. All writes will go to this writable layer, and
the rootfs size in the container is now whatever size this writable
layer is.
The block devices (via our ext4 package, but you could use whatever) are
cheap enough to generate on the fly that you can choose whatever size
you want easily.
If we have the same source we should dedupe these. Previously
virt.framework would yell at us for a duplicate tag because we derive
the tag from hashing the mount source.
- Simplifies processing and validation of MAC addresses for
apple/container#1005, where we need to create MAC addresses and
corresponding link local IPv6 addresses.
- Adds Codable for IPAddress and CIDR for apple/container#1006, so we
can parse and relay IP addresses regardless of address family for port
forwards.
Taking in a filehandle gives the user quite a bit more freedom on how to
handle boot log output. They can set up a kqueue watch on it and
redirect output somewhere else etc etc. The implementation for this has
us take in a new BootLog type that has two options:
1. .file, which is analogous to what we had prior. Just provide a URL
and a true by default append field.
2. .fileHandle which is the new addition. Can pass any fd that is
writable, and the VMM should write serial console output to it.
Everytime we grab a vsock connection we dup the conn and close the
original, otherwise we'd need to carry around the vsock connection type
everywhere as it closes the fd in its destructor. We weren't checking
the return value of dup however, so if it did fail we'd have a useless
filehandle with an fd of -1.
Today this protocols create method is just odd. Our only implementation
of it immediately casts to LinuxContainer, failing if it cannot do so. I
think what might make more sense is to pass in a configuration itself
with core parameters that we expect every vmm to be able to support, and
then in a specific implementation they can continue to cast this type to
a specific one to possibly extract some extra configuration values
(rosetta for VZ for example).
This rework will also make it simpler to support a Pod type, as the vm
setup is identical and simple.
While this only supports AS, xcode's default will build all arches, so
this presents a bit nicer default for Archiving an xcode application
that utilizes containerization.
Fixes#179
Signed-off-by: crosbymichael <michael_crosby@apple.com>
Because we were installing rosetta directly in the constructor for
VZVirtualMachineInstance, and we'd prefer to not have the constructor
async as it pollutes so much more, we had devised this gnarly callback
approach for the install that is a bit of an eyesore. This changes focus
is on moving the install flow to a method that is already async so we
can piggyback off of it, and removing the install logic from the config
-> VZConfig conversion. Now the install will occur during start() if
rosetta is not installed.
Fixes#85
The virtualization bool on LinuxContainer mostly just forwards to
VZVirtualMachineInstance which today would silently take your boolean
and do nothing if the underlying platform doesn't have support for it.
This is (to me) arguably worse than erroring, as it gives the client a
false security that the setting is on, and they should have virt
capabilities in the container/guest now.
This change makes it so that we throw a ContainerizationError for this
case, with a code of .unsupported so it's checkable by a user if they
want more information on the "why".
This change adds documentation to quite a few existing public types that
didn't have a blurb before.
Additionally, this fixes a couple things that I think either didn't make
sense when going to document them:
- Rename ConnectionStream to VsockConnectionStream. This type only
functions for vsock connections.
- Deletes NsLock+Closure. This was not used anywhere.
- Rename ContainerizationOCI/Config.swift to ImageConfig.swift.
Signed-off-by: Danny Canter <danny_canter@apple.com>
Michael Crosby