- Fixes#776
- Release 2.9.0 of `grpc-swift-nio-transport` fixes
the HTTP/2 initialization race that required a
`HTTP2ConnectBufferingHandler` for the vminitd
API. This upgrades the dependency and eliminates
the workaround.
apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
The `PodVolume` type in `LinuxPod` only defined the `nbd` enum value -
however, disk based images are also supported and the pattern is
essentially the same
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
Extends the network plumbing to support per-interface IPv6 address
configuration.
The `Interface` protocol supports `ipv6Address` and `ipv6Gateway`.
The agent's networking RPCs carry per-family fields via new
`InterfaceAddress`, `LinkRoute`, and `DefaultRoute` types in
`ContainerizationExtras`.
`NetlinkSession` adds IPv6 methods for address and route operations.
---------
Co-authored-by: michael_crosby <michael_crosby@apple.com>
Co-authored-by: Michael Crosby <crosbymichael@gmail.com>
This pull request enhances the handling of file and directory copy
operations in Linux containers, particularly by improving destination
path resolution and error handling for the `copyIn` operation with the
`Stat RPC`
This PR is needed for
[container#1190](https://github.com/apple/container/pull/1190)
Today we craft the proxied sockets we create at a path that contains the
container ID. This is normally completely fine, except if the ID is
enormous, as it can cause the UDS path to be longer than the allowed
length.
Move the MiB rounding from LinuxContainer into
`VZVirtualMachineInstance`, so all callers are covered in one place and
the alignment constraint stays with the VMM layer that owns it.
This will also ensure LinuxPod can get memory alignment for free.
If someone wanted to use a custom guest agent/guest image in general the
overhead we chose is kind of a nuisance.. This allows the values to be
configurable, and raises the default to 128, and vminitds cgroup
monitoring to 80.
This change adds an additional unit test to test
attaching a raw block device to a container to ensure raw block works
without API changes. We just need to set format to none and bind mount
the raw block device to the container path
This change adds support for attaching network block device (NBD) to
both LinuxContainer and LinuxPod.
For LinuxContainer, whether to use the underlying
`VZNetworkBlockDeviceStorageDeviceAttachment` is determined by the URL
of the container Mount source.
For LinuxPod, adds additional API to support pod-level volumes that can
be mounted into multiple containers. The PodVolume type provides enum to
support multiple types of volume source. LinuxContainer can reference
the pod level volume using the `Mount.sharedMount()` constructor with
the name referencing the name of the pod volume. This will allow the NBD
to be attached to the pod at the VM level and then bind-mounted into the
container.
For integration tests, added a lightweight NBD server implementation in
swift that speaks the NBD protocol to ensure there is sufficient
coverage for the changes introduced.
Apple Virtualization NBD support documentation:
https://developer.apple.com/documentation/virtualization/vznetworkblockdevicestoragedeviceattachment
Replace the old platform-gated Signals helper in ContainerizationOS with
a Signal struct in Containerization. Signal provides named constants for
common signals (.kill, .term, etc.), platform-specific namespaces
(Signal.Linux for rtmin/stkflt/pwr, Signal.Darwin for emt/info), string
parsing, and ExpressibleByIntegerLiteral support.
LinuxContainer, LinuxProcess, and LinuxPod kill APIs now take Signal
instead of raw Int32.
Add `Stat` RPC to vminitd.
`Stat` RPC is used to stat file system objects in the VM. It accepts the
path to the object (resolved against VM root) and returns the stat
result.
- Closes#671.
- Adds optional journal parameter to `EXT4.Formatter.init()`, with nil
default specifying the current no-journal filesystem configuration.
Otherwise the parameter value contains the journal size and mode.
- `minDiskSize` parameter for formatter init specifies the minimum
usable capacity of the resulting filesystem. The resulting disk image
grows past this value to accommodate the filesystem on-disk structures,
including the journal if specified.
Related to https://github.com/apple/container/issues/1251
Change to sharing in the parent directory, and then bind mounting in the
file into the container. This has the unfortunate reality of being less
secure, but the current approach is burdended by a couple things,
namely:
1. You can't share in files that are on a different volume
2. There is a Virtualization bug that causes spurious errors when trying
to open the file.
I've added a doc to go over the approach we've taken, and some
workarounds if the approach is not satisfactory.
To be able to test vminitd/vmexec/linux specific packages on ci it'd be
a heck of a lot easier if `make` just worked. This should be the last
bit needed. The default goal currently compiles just fine after the
linux specific `make deps` is ran. Next in line would be adding decent
unit tests/actually getting ci setup for the linux bits.
The network protocol and VMNetNetwork implementation currently are
housed on ContainerManager even though they are generally useful types
even outside of this easy to use helper type. This change moves them to
not be nested types anymore, as well as exposes a new param on
VMNetNetworks constructor so we can pass the type of network.
Now that the static linux SDK we use has libarchive linked against it,
we can finally live our dream (/s) of supporting copying directories
in/out a little easier. This was.. kind of annoying. I think the very
simple route of writing archive to guest/host -> streaming over grpc is
simplest, but on the guest end we have a couple problems:
1. The VMs rootfs is read only today always, which is a good thing to me
and I don't want to change if we don't have to.
2. Writing the archive to the containers rootfs temporarily could work,
but it's a bit weird, and the user can make the containers rootfs
readonly which would screw that plan.
3. We could write it to /run or /tmp, but they're tmpfs and dealing with
the headache of the user possibly tarring an enormous dir is one I don't
want to care about.
So, that leaves us with the (truthfully better to me) approach of trying
to write the tar data directly to the host and skipping grpc which kinda
forces us to have a temp spot. Because of that, I made it such that we
pass a port number from host<->guest, and transfer the actual data
(either the single file or tarred dir contents) over the vsock port
instead. The stream is kinda clunky, but it just serves as a means to
exchange metadata and a "we're done" signifier.
Closes#256
If the target of the uds proxy goes through a symlink (/var/run) we had
the same bug we had with mounts previously where we'd follow this and if
the resolved path existed in the root of the VM we'd end up mounting
there. Fix this by plopping the sockets in a holding spot and then bind
mounting them in (which has the correct resolution logic).
This change fixes an issue in linux pod where the exec process
does not inherit the container process config which means any
environment variables, working directory, user, rlimits,
capabilities would be different for the exec process.
This change ensures that exec process would inherit
environment variables, working directory, user, rlimits,
and capabilities from the container process config but
resets arguments, terminal, and stdio as those should not
be inherited.
This change also adds an integration test to exercise this.
Just like we have for hosts and dns configurations, we can add a
hostname field to make it easier to setup if every container is going to
be using the same hostname.
This PR adds a `networking: Bool = true` parameter to
`ContainerManager.create()` so callers can opt out of network interface
creation on a per-container basis.
## Motivation
Currently, `ContainerManager.create()` unconditionally allocates a vmnet
network interface for every container (when the manager has a network
configured). Some use cases don't need network access and benefit from
having it disabled to reduce attack surface.
There's no way to achieve this today without either:
- Initializing the `ContainerManager` without a network (which disables
networking for *all* containers)
- Clearing `config.interfaces` in the configuration closure (which
wastes an IP allocation from the vmnet pool since `network.create(id)`
has already been called)
## Changes
- Add `networking: Bool = true` to all three `create()` overloads on
`ContainerManager`
- When `false`, `self.network?.create(id)` is skipped entirely. No
interface is allocated, and no DNS is configured.
- `releaseNetwork`/`delete` remain safe to call regardless, since
`Allocator.release` silently ignores unknown IDs.
- Add unit test `testNetworkingFalseSkipsInterfaceCreation` using the
existing `NilGatewayNetwork` fixture
- Add integration tests `testNetworkingDisabled` and
`testNetworkingEnabled` that create containers through a network-enabled
`ContainerManager` and verify the presence/absence of `eth0` via
`/sys/class/net/`
There are some workloads that benefit from having a "true" init process
that reaps zombies. This change implements a minimal init as part of
vminitd and exposes an API to be able to use this init process in our
containers. signals will be forwarded, any child procs will be reaped
and the exit code of the actual workload will be propagated. The
LinuxContainer/LinuxPod API is a very simple bool to ask to use this as
our init process, and the init simply spawns whatever command you want
as a child underneath this.
```
➜ ./bin/cctl run --kernel bin/kernel.arm64 --init
/ # ps aux
PID USER TIME COMMAND
1 root 0:00 /.cz-init -- /bin/sh
2 root 0:00 /bin/sh
3 root 0:00 ps aux
```
This adds support to LinuxContainer to be able to provide an optional
writable layer. This is useful as today the rootfs size is determined by
the size of the image block itself which is set in stone at unpack time
of the image. This leaves a lot to be desired because a user might want
a larger or smaller rootfs size, but it's not configurable on a per
container basis because of this. To support this, we can pass an
additional writable block device that we can overlayfs with the image
contents in the guest. All writes will go to this writable layer, and
the rootfs size in the container is now whatever size this writable
layer is.
The block devices (via our ext4 package, but you could use whatever) are
cheap enough to generate on the fly that you can choose whatever size
you want easily.
If we have the same source we should dedupe these. Previously
virt.framework would yell at us for a duplicate tag because we derive
the tag from hashing the mount source.
I was not a huge fan of how rlimits were specified using the OCI type as
it just takes in strings for the type which isn't super ergonomic. This
adds a new type for rlimits that we can convert to oci type like we
already do for capabilities.
Upon further thought, we should probably support this at the pod level
and allow per container overrides. I've changed my mind on how tedious
it is to specify it on a per-container basis.
Fixes#79
This adds support for single file (virtiofs based) mounts to the two
main container types. This is transparent, and doesn't need to be setup
by a user, so there is no visible API change. Virtiofs does not support
single file mounts today, and it would be less than ideal from a
security standpoint to just expose the parent directory of the file to
the vm and then bind mount in the file, so we chose the following:
1. Create a temp directory the container types will manage.
2. Hardlink in the file we want.
3. Share that tempdir into the vm.
4. Finally bind mount in the file into the container.
The main goal I wanted for this is to leave the logic out of the
`VirtualMachineInstance`. I really want this to just be a little dance
the container types do, and leave the vm out of it. Because of that,
most of the logic is shoved in a new `FileMountContext` type that does
the dirty tricks of rewriting what the user asked for to be a bind mount
from a temporary holding spot we'll use in the guest for these temp
directories.
One of the downsides to this is today the tempdirs are solely on the
main volume, so cross volume will need to copy the file to the tempdir,
so writes won't get written back.
Co-authored-by: Dean Coulstock <d_coulstock@apple.com>
Co-authored-by: Jaewon Hur <hurjaewon@snu.ac.kr>
Fixes#477
Because we start piping stdin before process launch we can fill up the
guest pipe buffer before the process even starts. We'd need some
backpressure mechanism to handle this and slow consumers (register the
other end with epoll and buffer some data etc.), but we should probably
just start piping after the process is up and running. This change does
exactly that, as well as stops holding the process mutex while draining
stdin in the guest for `CloseStdin()`.
This fixes issues where we try and write > pipe_buf bytes through stdin.
Today this hangs.
It's possible a user doesn't want the full stats list, and only wants
cpu/mem etc. This plumbs through the ability to filter to only what is
requested. This, while we're already here, adds in memory.event output
to the stats list. For that specifically, I think eventually we may want
a streaming variant of this so you can get alerted of changes in the
file immediately instead of polling/one off reads, but this is useful
for now.
Add a streaming rpc to copy single files in and out. This can easily be
extended to support copying directories, but we need the guest agent to
be able to tar/untar, and we'll need to figure out what that model is.
In a prior change I'd added a way for vminitd to double as a simple
pause container. This change wires this up by adding a new bool to the
pod config to ask for pid ns sharing.
Michael Crosby