apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
The `PodVolume` type in `LinuxPod` only defined the `nbd` enum value -
however, disk based images are also supported and the pattern is
essentially the same
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
Extends the network plumbing to support per-interface IPv6 address
configuration.
The `Interface` protocol supports `ipv6Address` and `ipv6Gateway`.
The agent's networking RPCs carry per-family fields via new
`InterfaceAddress`, `LinkRoute`, and `DefaultRoute` types in
`ContainerizationExtras`.
`NetlinkSession` adds IPv6 methods for address and route operations.
---------
Co-authored-by: michael_crosby <michael_crosby@apple.com>
Co-authored-by: Michael Crosby <crosbymichael@gmail.com>
Today we craft the proxied sockets we create at a path that contains the
container ID. This is normally completely fine, except if the ID is
enormous, as it can cause the UDS path to be longer than the allowed
length.
I think realistically we should have a limit on this. Technically today
the ID length restriction is whatever our grpc buffer length is..
I think 64 is long enough so that folks could use an entire hex based
sha256, any uuid, and also sethostname has a length restriction of 64
anyways.
This change adds support for attaching network block device (NBD) to
both LinuxContainer and LinuxPod.
For LinuxContainer, whether to use the underlying
`VZNetworkBlockDeviceStorageDeviceAttachment` is determined by the URL
of the container Mount source.
For LinuxPod, adds additional API to support pod-level volumes that can
be mounted into multiple containers. The PodVolume type provides enum to
support multiple types of volume source. LinuxContainer can reference
the pod level volume using the `Mount.sharedMount()` constructor with
the name referencing the name of the pod volume. This will allow the NBD
to be attached to the pod at the VM level and then bind-mounted into the
container.
For integration tests, added a lightweight NBD server implementation in
swift that speaks the NBD protocol to ensure there is sufficient
coverage for the changes introduced.
Apple Virtualization NBD support documentation:
https://developer.apple.com/documentation/virtualization/vznetworkblockdevicestoragedeviceattachment
Replace the old platform-gated Signals helper in ContainerizationOS with
a Signal struct in Containerization. Signal provides named constants for
common signals (.kill, .term, etc.), platform-specific namespaces
(Signal.Linux for rtmin/stkflt/pwr, Signal.Darwin for emt/info), string
parsing, and ExpressibleByIntegerLiteral support.
LinuxContainer, LinuxProcess, and LinuxPod kill APIs now take Signal
instead of raw Int32.
Related to https://github.com/apple/container/issues/1251
Change to sharing in the parent directory, and then bind mounting in the
file into the container. This has the unfortunate reality of being less
secure, but the current approach is burdended by a couple things,
namely:
1. You can't share in files that are on a different volume
2. There is a Virtualization bug that causes spurious errors when trying
to open the file.
I've added a doc to go over the approach we've taken, and some
workarounds if the approach is not satisfactory.
Closes#256
If the target of the uds proxy goes through a symlink (/var/run) we had
the same bug we had with mounts previously where we'd follow this and if
the resolved path existed in the root of the VM we'd end up mounting
there. Fix this by plopping the sockets in a holding spot and then bind
mounting them in (which has the correct resolution logic).
This change fixes an issue in linux pod where the exec process
does not inherit the container process config which means any
environment variables, working directory, user, rlimits,
capabilities would be different for the exec process.
This change ensures that exec process would inherit
environment variables, working directory, user, rlimits,
and capabilities from the container process config but
resets arguments, terminal, and stdio as those should not
be inherited.
This change also adds an integration test to exercise this.
Just like we have for hosts and dns configurations, we can add a
hostname field to make it easier to setup if every container is going to
be using the same hostname.
There are some workloads that benefit from having a "true" init process
that reaps zombies. This change implements a minimal init as part of
vminitd and exposes an API to be able to use this init process in our
containers. signals will be forwarded, any child procs will be reaped
and the exit code of the actual workload will be propagated. The
LinuxContainer/LinuxPod API is a very simple bool to ask to use this as
our init process, and the init simply spawns whatever command you want
as a child underneath this.
```
➜ ./bin/cctl run --kernel bin/kernel.arm64 --init
/ # ps aux
PID USER TIME COMMAND
1 root 0:00 /.cz-init -- /bin/sh
2 root 0:00 /bin/sh
3 root 0:00 ps aux
```
"clean up" is a verb, and "cleanup" is a noun. There are several
examples of the incorrect usage of "cleanup" as a verb in this
repository, including one in the public API (`cleanupOrphanedBlobs`).
For this public method, I've added a deprecation notice to the name with
the incorrect usage.
Upon further thought, we should probably support this at the pod level
and allow per container overrides. I've changed my mind on how tedious
it is to specify it on a per-container basis.
Fixes#79
This adds support for single file (virtiofs based) mounts to the two
main container types. This is transparent, and doesn't need to be setup
by a user, so there is no visible API change. Virtiofs does not support
single file mounts today, and it would be less than ideal from a
security standpoint to just expose the parent directory of the file to
the vm and then bind mount in the file, so we chose the following:
1. Create a temp directory the container types will manage.
2. Hardlink in the file we want.
3. Share that tempdir into the vm.
4. Finally bind mount in the file into the container.
The main goal I wanted for this is to leave the logic out of the
`VirtualMachineInstance`. I really want this to just be a little dance
the container types do, and leave the vm out of it. Because of that,
most of the logic is shoved in a new `FileMountContext` type that does
the dirty tricks of rewriting what the user asked for to be a bind mount
from a temporary holding spot we'll use in the guest for these temp
directories.
One of the downsides to this is today the tempdirs are solely on the
main volume, so cross volume will need to copy the file to the tempdir,
so writes won't get written back.
Co-authored-by: Dean Coulstock <d_coulstock@apple.com>
Co-authored-by: Jaewon Hur <hurjaewon@snu.ac.kr>
When we set a gateway outside a subnet in Linux, this fails with:
```
ip-route-add-default: netlink response indicates error, rc = -101
```
To resolve the error, we should first add a route to the gateway outside
the subnet, which this PR does.
It's possible a user doesn't want the full stats list, and only wants
cpu/mem etc. This plumbs through the ability to filter to only what is
requested. This, while we're already here, adds in memory.event output
to the stats list. For that specifically, I think eventually we may want
a streaming variant of this so you can get alerted of changes in the
file immediately instead of polling/one off reads, but this is useful
for now.
In a prior change I'd added a way for vminitd to double as a simple
pause container. This change wires this up by adding a new bool to the
pod config to ask for pid ns sharing.
Taking in a filehandle gives the user quite a bit more freedom on how to
handle boot log output. They can set up a kqueue watch on it and
redirect output somewhere else etc etc. The implementation for this has
us take in a new BootLog type that has two options:
1. .file, which is analogous to what we had prior. Just provide a URL
and a true by default append field.
2. .fileHandle which is the new addition. Can pass any fd that is
writable, and the VMM should write serial console output to it.
This was a funny oversight. We have two port spaces, one for listening
sockets on the host, and one for vsock proxies in the guest, but only
the host "allocator" was being used. This didn't really matter as the
ports in the guest would still be unique, but would still be good to
fix.
Closes#319
Introduces a new type capable of running > 1 container in the guest. The
API mostly follows LinuxContainer, and each individual container can be
addressed via any of the methods that require you to pass the
containerID as the first param. Today there's no support for namespace
sharing, but that shouldn't be terrible to support.
Michael Crosby