This expands `cctl run` to utilize entrypoint and cmd when present in
the OCI image, instead of always defaulting to `/bin/sh`. The user can
still provide explicit commands if necessary to override these values.
- `ContainerManager` now accepts & stores an optional `Logger`
and forwards it to `VZVirtualMachineManager` and `LinuxContainer`.
- Existing callers should work unchanged.
Closes#896.
libarchive permits path traversal components (`.` and `..`) in entry
pathnames during entry iteration. The library will only extract (in
`archive_write_disk`) an entry if the normalized pathname resolves under
the extraction root.
Our EXT4 unarchiver currently prepends '/' to a pathname — an entry's
own path, or a hardlink entry's link target, both of which are resolved
through the same normalization function — and then attempts to create
the corresponding directory entry or link regardless of whether the
original relative path would escape the root, yielding a filesystem
image that is incorrect at least and corrupt at worst.
Valid OCI archive-type content blobs should never contain escaping
pathnames. The unarchiver should treat such pathnames as garbage input
and throw an exception on the first such pathname, whether it originates
from an entry path or a hardlink target.
- Closes#829
- Updated the filesystem operation to take in the `containerID`, so that
operations are always executed against the intended container’s mount
namespace.
This PR fixes a number of issues:
- validate descriptor digests before filesystem use
- validate container and exec ids before building paths in
ManagedContainer
- Check for attribute length when consuming extended attributes in
ContainerizationEXT4
- Check directory name length when formatting an ext4 fs
- Prevent OCI image layout load from reading host files through a
symlink
- Validate registry auth realm before exchanging credentials
---------
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
Co-authored-by: John Logan <john_logan@apple.com>
Co-authored-by: Raj Aryan Singh <rajaryan_singh@apple.com>
Co-authored-by: Raj <realrajaryan@gmail.com>
`ExecCommand.execInNamespaces` entered a hardcoded `CLONE_NEWCGROUP |
CLONE_NEWPID | CLONE_NEWUTS | CLONE_NEWNS`, while the init process gets
its namespaces from the OCI spec via `RunCommand.setupNamespaces` —
which includes `CLONE_NEWIPC` whenever the spec lists an `ipc`
namespace. `LinuxContainer` (and `LinuxPod`) always declare one, so the
init process runs in a private IPC namespace and every exec'd process
stays in the guest's root one.
The effect is that an exec cannot see SysV IPC segments or POSIX message
queues created by the workload — and anything the exec creates leaks
into the guest's root namespace, where it outlives the container.
IPC-namespaced sysctls (`kernel.shm*`, `kernel.msg*`, `kernel.sem`,
`fs.mqueue.*`) also read back as the guest default, because they resolve
against the reading process's IPC namespace. That last one is what fails
critest's "should support safe sysctls" and "should support unsafe
sysctls", which set such a sysctl and read it back over ExecSync.
Add `CLONE_NEWIPC` to the mask. `setns(2)` into a namespace the caller
is already in is a no-op, so the mask is now a superset of everything
`setupNamespaces` can unshare rather than a set that has to be kept in
sync by hand. `CLONE_NEWUSER` stays out deliberately: nothing puts a
container in a user namespace today, and entering one carries
constraints that deserve their own change.
The integration test asserts the general invariant rather than the
specific sysctl: inside an exec, `/proc/self/ns/*` must equal
`/proc/1/ns/*` for `ipc`, `uts`, `mnt`, `pid`, `cgroup`, and `net`.
`kernel.shm_rmid_forced` is checked alongside it as the consumer-visible
symptom.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
- Omit the redundant `v8` variant for arm64 when rendering
`description`, so equal arm64 platforms always describe as
`linux/arm64` — matching how Docker and containerd display
the platform. Other variants (`arm/v7`) and architectures
(`amd64`) are unaffected.
- Only the rendered `description` changes. The stored `variant`
and the `Codable` encoding are untouched, so OCI content
digests remain stable.
- Closesapple/container#1542 (normalization-consistency aspect).
This fixes a latent bug in the `Platform` types equality operator where
two platforms with differing OS's would be treated as the same if they
both had variant set to 'v8' or nil
Fixes#518.
## What
vminitd logs the full OCI spec and exec process at debug level in
`ManagedContainer` ("created bundle with spec …", "creating exec process
with …"), which puts every `NAME=value` environment entry into the boot
log. Environment variables routinely carry secrets, so `container logs
--boot web | grep PASSWORD` reproduces the leak exactly as described in
#518.
Rather than redacting at the call sites, this makes the redacted form
the *default* rendering of the types that own an environment: `Process`
and `Hook` conform to `CustomStringConvertible` with values masked and
names kept. `Spec` and `Hooks` inherit it, because Swift's
reflection-based description renders a nested value through that value's
own `description`.
The effect is that any `\(spec)` or `\(process)` is safe without the
author knowing this file exists, which is what stops a log line added
later from reintroducing the leak. The two existing log sites are
unchanged, so this no longer touches vminitd at all.
Two details worth calling out:
- **`Codable` is untouched.** `description` governs text rendering only,
so an encoded spec still carries the real values and nothing changes
about what is written to disk or sent to the guest. The unredacted
environment also remains available to callers through `process.env`.
- **`description` renders through a mirror** rather than a hand-written
field list. `Process` has 13 fields; listing them by hand would drop the
rest from the log line and would rot as fields are added.
## Verification
- New `SpecRedactionTests` (9 tests) cover: a whole `Spec` interpolated
into a log line never renders the values; `String(describing:)` and
`String(reflecting:)` are redacted too; variable names survive;
`NAME`-only inherit entries pass through; `NAME=` and values containing
further `=` are masked whole; encoding round-trips with the real values;
rendering does not mutate; and the other fields are still rendered.
- Negative control: with the redaction disabled the suite fails with 13
issues, and the output shows the secret in the clear, reproducing #518.
- Full `ContainerizationOCITests` passes, 58 tests in 9 suites.
- `swift format lint --strict --configuration .swift-format-nolint` is
clean, and `swift format` leaves both files unchanged.
Every line here is one I can explain and justify; the reasoning above is
the complete rationale for each change.
This PR masks the high mode bits when setting file attributes for
consistency with mode at file creation.
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
- `FileTree.lookup` resolved each path component by linearly scanning the
node's `children` array. This changes the node's child storage to an
`OrderedDictionary<String, Ptr<FileTreeNode>>` (from swift-collections,
which is already a package dependency) keyed by name, so `lookup`
resolves each component in O(1) while iteration keeps the existing
insertion order.
- Little or no difference in unpack time for images with ~10k files, significant
improvement for images with ~100k files or more.
Allow a container hotplugged into an already-running LinuxPod VM
(addContainer after create()) to have a virtiofs (directory-share)
rootfs, not just an ext4 virtio-blk image, on the cloud-hypervisor
backend.
Host side:
- Generalize CHHotplugProvider.hotplug to branch on the rootfs
runtimeOptions (.virtioblk / .virtiofs / unsupported), reusing the
per-tag virtiofsd/vm.add-fs machinery factored into a new
ensureVirtiofsDevice helper. Make registerMounts the sole writer of the
per-VM mount registry so the rootfs is always first and mounts are not
double-registered (also fixes a latent ordering bug on the block path).
Exclude the rootfs from the /run/virtiofs holding-dir loop in
addContainer.
Guest side (fixes a pre-existing, feature-independent bug):
- The addContainer-after-create (runtime hotplug) path had never been
exercised by any test. cloud-hypervisor adds hot-plugged
virtio-blk/virtio-fs devices to the guest PCI bus but the guest does not
auto-probe them, so a mount by /dev/vdX or virtio-fs tag fails (ENOENT /
tag-not-found) for ALL CH hotplug.
- vminitd's mount RPC now, on a mount failure for a hot-plug-candidate
source (virtiofs type or /dev/vd* source), forces /sys/bus/pci/rescan
and retries with a bounded wait. Scoped so boot-time and non-device
mounts are unaffected.
Tests / build:
- First coverage of LinuxPod.addContainer after create() (CH-only; VZ
has no runtime hotplug): pod hotplug virtiofs rootfs and pod hotplug
block rootfs, registered in a Linux-only test list, plus a
directory-rootfs unpack helper.
- make linux-integration gains an optional FILTER passthrough (make
linux-integration FILTER="pod hotplug").
Signed-off-by: michael_crosby <michael_crosby@apple.com>
- Closes#807.
- Ensure everything builds and passes with latest conforming package
versions.
- Use swift-collections 1.5.1 instead of 1.6.0 due to Swift 6.3
limitations.
- Closes#787.
- The `blockSizeInBytes` parameter on `EXT4Unpacker`
was misleading — it does not represent an ext4 filesystem
block size.
- This change renames `blockSizeInBytes` to
`capacityInBytes`, updates the doc comment to accurately
describe the parameter and updates the name in the call
sites.
- Fixes#776
- Release 2.9.0 of `grpc-swift-nio-transport` fixes
the HTTP/2 initialization race that required a
`HTTP2ConnectBufferingHandler` for the vminitd
API. This upgrades the dependency and eliminates
the workaround.
Changes the default Linux capability set for container processes from
`.allCapabilities` to `.defaultOCICapabilities`, making the library
secure-by-default. Callers that genuinely need elevated capabilities
must now opt in explicitly.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
The `PodVolume` type in `LinuxPod` only defined the `nbd` enum value -
however, disk based images are also supported and the pattern is
essentially the same
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
When unpacking an OCI/tar layer, create() already creates missing parent
directories recursively, so regular files and symlinks with absent
parent entries unpack correctly. link() did not, so a hardlink whose
parent directory had no explicit archive entry failed with "<path> not
found" (e.g. images produced by Bazel rules_img). Mirror create()'s
implicit parent creation in link() so such layers unpack, matching
Docker/containerd.
Adds a direct link() unit test and an end-to-end unpack regression test
covering a hardlink, regular file, and symlink with no explicit parent.
Fixes https://github.com/apple/container/issues/1797
- Closes#774.
- Adds a default journal configuration parameter to the EXT4Unpacker
init function, defaulting to nil (no journal) to preserve source
compatibility.
The `--log-level` option when running the agent sub-command for vminitd
was being silently ignored cause of the way the agent is being run. As a
workaround we need to read `/proc/self/cmdline` to get the right args
Seems like a typo.
The change only affected a comment and does not disturb the code at all
(backward compatible).
Co-authored-by: J Logan <john_logan@apple.com>
## Summary
`Platform.==` treats `arm64` with `nil` variant as equal to `arm64/v8`,
but `hash(into:)` used `description` which serializes them differently
(`linux/arm64` vs `linux/arm64/v8`). This violates the `Hashable`
contract — equal values must produce the same hash.
### Root cause
```swift
// == returns true for these two
let a = Platform(arch: "arm64", os: "linux", variant: nil)
let b = Platform(arch: "arm64", os: "linux", variant: "v8")
a == b // true ✓
// but hash was different — broken
a.hashValue == b.hashValue // false ✗ (before this fix)
```
This mismatch caused `Set<Platform>` and `Dictionary<Platform, ...>`
lookups to silently miss entries when one platform was decoded from JSON
(no `variant` field in the manifest) and another was created via
`Platform(from:)` or `Platform.current` (which both set `variant =
"v8"`).
### Practical consequence
In `apple/container`, this manifests as inconsistent platform-string
normalization across stages of a single `container build` — some stages
log `linux/arm64`, others `linux/arm64/v8` — which can cause `COPY
--from=<stage>` to fail to resolve the source stage under concurrent
builds. See apple/container#1542.
### Fix
`hash(into:)` now normalizes `arm64` with `nil` variant to `"v8"` before
hashing, matching the existing `==` behavior.
Extends the network plumbing to support per-interface IPv6 address
configuration.
The `Interface` protocol supports `ipv6Address` and `ipv6Gateway`.
The agent's networking RPCs carry per-family fields via new
`InterfaceAddress`, `LinkRoute`, and `DefaultRoute` types in
`ContainerizationExtras`.
`NetlinkSession` adds IPv6 methods for address and route operations.
---------
Co-authored-by: michael_crosby <michael_crosby@apple.com>
Co-authored-by: Michael Crosby <crosbymichael@gmail.com>
`.totalFileAllocatedSizeKey` returns nil for directories on Darwin but
on Linux it returns `st_blocks * st_blksize` (4 KB each) in Foundation.
The empty-store test summed three directory inodes on Linux and failed
with `#expect(size == 0)`. This change adds filter on the enumerator to
regular files only so the totals are content-only and will work for both
Darwin and Linux.
This PR adds `totalAllocatedSize()` to the `ContentStore` protocol so it
can be used to get the on-disk footprint without reaching past the
abstraction. `LocalContentStore` implements it by walking its base path,
covering both committed blobs and active ingest sessions.
- Closes#712.
- Replace synchronous `write()` calls in
`BidirectionalRelay` with non-blocking I/O
and `DispatchSourceWrite` backpressure
handling. Under concurrent vsock proxy load,
a single blocked write on the shared serial
dispatch queue would freeze all relay
connections permanently, including unrelated
new connections.
- Set relay file descriptors to `O_NONBLOCK`
and handle `EAGAIN` by suspending reads and
installing a write source to drain pending data.
- Give each `BidirectionalRelay` its own serial queue
instead of sharing one from `UnixSocketRelayManager`,
eliminating cross-connection blocking.
- Resume suspended read sources before cancelling
in `stop()` — GCD does not deliver cancel handlers
on suspended dispatch sources, which caused file
descriptor and memory leaks on teardown under
backpressure.
- Guard-unwrap `buf.baseAddress` in
`drainPendingWrite`.
- Closes#744
- Adds the initial `FilePathOps` utility type
- Adds the `absolutePath` implementation
- Adds the `FilePathOpsTests` file and initial test cases
The `IPv4Address(_ bytes: [UInt8])` initializer in
ContainerizationExtras shifts the third octet by 16 bits instead of 8:
```swift
self.value =
(UInt32(bytes[0]) << 24)
| (UInt32(bytes[1]) << 16)
| (UInt32(bytes[2]) << 16) // should be << 8
| UInt32(bytes[3])
```
Because `bytes[2]` lands in the same bit range as `bytes[1]`, the second
octet gets corrupted by the OR, the third octet is dropped, and bits 8
through 15 are always left zero. Concretely, decoding `[192, 168, 1, 1]`
yields `192.169.0.1` instead of `192.168.1.1`, and `[18, 52, 86, 120]`
yields `18.118.0.120` instead of `18.52.86.120`.
This went unnoticed because the `bytes` computed property getter uses
the correct `>> 8` for the third octet, but there was no test exercising
the byte-array initializer, so the encode and decode paths were never
checked against each other. The sibling `IPv6Address(_ bytes:)`
initializer uses the correct descending shifts (`<< 120, << 112, ... <<
8, << 0`), which is what the IPv4 version should mirror.
The fix changes the third octet shift to 8 bits so the initializer is
the exact inverse of the `bytes` property. I also added two tests to the
initializer suite: a valid-input test that asserts both the resulting
`value` and that `init(bytes).bytes == bytes` round-trips, and an
invalid-length test. The round-trip test fails on the current code and
passes with the fix.
Verification: `swift test --filter ContainerizationExtrasTests` passes
221 tests in 26 suites (the IPv4Address suite goes from 23 to 25 tests).
The new round-trip test fails before the one-line change and passes
after.
Signed-off-by: Aditya Singh <adisin650@gmail.com>
- Closes#749.
- ExportOperation hardcoded the pushed index descriptor's
mediaType to the OCI image index type. RegistryClient.push
uses that descriptor's mediaType as the HTTP Content-Type
header. When the source index was in
Docker manifest.list.v2+json format (the common case for
images pulled from Docker Hub and other public registries),
the body's embedded mediaType field disagreed with the
header, and OCI registries rejected the index PUT with
HTTP 400 MANIFEST_INVALID.
- Use the source index's mediaType for the pushed descriptor
so the header always matches the body. Per-architecture
child manifests are unaffected because they were already
pushed with their actual mediaType.
- Add a parameterized unit test for ExportOperation.export
covering both Docker manifest.list
Noah Thornton