This expands `cctl run` to utilize entrypoint and cmd when present in
the OCI image, instead of always defaulting to `/bin/sh`. The user can
still provide explicit commands if necessary to override these values.
- `ContainerManager` now accepts & stores an optional `Logger`
and forwards it to `VZVirtualMachineManager` and `LinuxContainer`.
- Existing callers should work unchanged.
Closes#896.
libarchive permits path traversal components (`.` and `..`) in entry
pathnames during entry iteration. The library will only extract (in
`archive_write_disk`) an entry if the normalized pathname resolves under
the extraction root.
Our EXT4 unarchiver currently prepends '/' to a pathname — an entry's
own path, or a hardlink entry's link target, both of which are resolved
through the same normalization function — and then attempts to create
the corresponding directory entry or link regardless of whether the
original relative path would escape the root, yielding a filesystem
image that is incorrect at least and corrupt at worst.
Valid OCI archive-type content blobs should never contain escaping
pathnames. The unarchiver should treat such pathnames as garbage input
and throw an exception on the first such pathname, whether it originates
from an entry path or a hardlink target.
- Closes#829
- Updated the filesystem operation to take in the `containerID`, so that
operations are always executed against the intended container’s mount
namespace.
This PR fixes a number of issues:
- validate descriptor digests before filesystem use
- validate container and exec ids before building paths in
ManagedContainer
- Check for attribute length when consuming extended attributes in
ContainerizationEXT4
- Check directory name length when formatting an ext4 fs
- Prevent OCI image layout load from reading host files through a
symlink
- Validate registry auth realm before exchanging credentials
---------
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
Co-authored-by: John Logan <john_logan@apple.com>
Co-authored-by: Raj Aryan Singh <rajaryan_singh@apple.com>
Co-authored-by: Raj <realrajaryan@gmail.com>
`ExecCommand.execInNamespaces` entered a hardcoded `CLONE_NEWCGROUP |
CLONE_NEWPID | CLONE_NEWUTS | CLONE_NEWNS`, while the init process gets
its namespaces from the OCI spec via `RunCommand.setupNamespaces` —
which includes `CLONE_NEWIPC` whenever the spec lists an `ipc`
namespace. `LinuxContainer` (and `LinuxPod`) always declare one, so the
init process runs in a private IPC namespace and every exec'd process
stays in the guest's root one.
The effect is that an exec cannot see SysV IPC segments or POSIX message
queues created by the workload — and anything the exec creates leaks
into the guest's root namespace, where it outlives the container.
IPC-namespaced sysctls (`kernel.shm*`, `kernel.msg*`, `kernel.sem`,
`fs.mqueue.*`) also read back as the guest default, because they resolve
against the reading process's IPC namespace. That last one is what fails
critest's "should support safe sysctls" and "should support unsafe
sysctls", which set such a sysctl and read it back over ExecSync.
Add `CLONE_NEWIPC` to the mask. `setns(2)` into a namespace the caller
is already in is a no-op, so the mask is now a superset of everything
`setupNamespaces` can unshare rather than a set that has to be kept in
sync by hand. `CLONE_NEWUSER` stays out deliberately: nothing puts a
container in a user namespace today, and entering one carries
constraints that deserve their own change.
The integration test asserts the general invariant rather than the
specific sysctl: inside an exec, `/proc/self/ns/*` must equal
`/proc/1/ns/*` for `ipc`, `uts`, `mnt`, `pid`, `cgroup`, and `net`.
`kernel.shm_rmid_forced` is checked alongside it as the consumer-visible
symptom.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
- Omit the redundant `v8` variant for arm64 when rendering
`description`, so equal arm64 platforms always describe as
`linux/arm64` — matching how Docker and containerd display
the platform. Other variants (`arm/v7`) and architectures
(`amd64`) are unaffected.
- Only the rendered `description` changes. The stored `variant`
and the `Codable` encoding are untouched, so OCI content
digests remain stable.
- Closesapple/container#1542 (normalization-consistency aspect).
This fixes a latent bug in the `Platform` types equality operator where
two platforms with differing OS's would be treated as the same if they
both had variant set to 'v8' or nil
Fixes#518.
## What
vminitd logs the full OCI spec and exec process at debug level in
`ManagedContainer` ("created bundle with spec …", "creating exec process
with …"), which puts every `NAME=value` environment entry into the boot
log. Environment variables routinely carry secrets, so `container logs
--boot web | grep PASSWORD` reproduces the leak exactly as described in
#518.
Rather than redacting at the call sites, this makes the redacted form
the *default* rendering of the types that own an environment: `Process`
and `Hook` conform to `CustomStringConvertible` with values masked and
names kept. `Spec` and `Hooks` inherit it, because Swift's
reflection-based description renders a nested value through that value's
own `description`.
The effect is that any `\(spec)` or `\(process)` is safe without the
author knowing this file exists, which is what stops a log line added
later from reintroducing the leak. The two existing log sites are
unchanged, so this no longer touches vminitd at all.
Two details worth calling out:
- **`Codable` is untouched.** `description` governs text rendering only,
so an encoded spec still carries the real values and nothing changes
about what is written to disk or sent to the guest. The unredacted
environment also remains available to callers through `process.env`.
- **`description` renders through a mirror** rather than a hand-written
field list. `Process` has 13 fields; listing them by hand would drop the
rest from the log line and would rot as fields are added.
## Verification
- New `SpecRedactionTests` (9 tests) cover: a whole `Spec` interpolated
into a log line never renders the values; `String(describing:)` and
`String(reflecting:)` are redacted too; variable names survive;
`NAME`-only inherit entries pass through; `NAME=` and values containing
further `=` are masked whole; encoding round-trips with the real values;
rendering does not mutate; and the other fields are still rendered.
- Negative control: with the redaction disabled the suite fails with 13
issues, and the output shows the secret in the clear, reproducing #518.
- Full `ContainerizationOCITests` passes, 58 tests in 9 suites.
- `swift format lint --strict --configuration .swift-format-nolint` is
clean, and `swift format` leaves both files unchanged.
Every line here is one I can explain and justify; the reasoning above is
the complete rationale for each change.
This PR allows callers to use `linux_run` in the makefile without
relying on nested virtualization by default. This allows us to build a
subset of makefile targets on machines where nested virtualization is
not supported.
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
This PR masks the high mode bits when setting file attributes for
consistency with mode at file creation.
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
- `FileTree.lookup` resolved each path component by linearly scanning the
node's `children` array. This changes the node's child storage to an
`OrderedDictionary<String, Ptr<FileTreeNode>>` (from swift-collections,
which is already a package dependency) keyed by name, so `lookup`
resolves each component in O(1) while iteration keeps the existing
insertion order.
- Little or no difference in unpack time for images with ~10k files, significant
improvement for images with ~100k files or more.
Allow a container hotplugged into an already-running LinuxPod VM
(addContainer after create()) to have a virtiofs (directory-share)
rootfs, not just an ext4 virtio-blk image, on the cloud-hypervisor
backend.
Host side:
- Generalize CHHotplugProvider.hotplug to branch on the rootfs
runtimeOptions (.virtioblk / .virtiofs / unsupported), reusing the
per-tag virtiofsd/vm.add-fs machinery factored into a new
ensureVirtiofsDevice helper. Make registerMounts the sole writer of the
per-VM mount registry so the rootfs is always first and mounts are not
double-registered (also fixes a latent ordering bug on the block path).
Exclude the rootfs from the /run/virtiofs holding-dir loop in
addContainer.
Guest side (fixes a pre-existing, feature-independent bug):
- The addContainer-after-create (runtime hotplug) path had never been
exercised by any test. cloud-hypervisor adds hot-plugged
virtio-blk/virtio-fs devices to the guest PCI bus but the guest does not
auto-probe them, so a mount by /dev/vdX or virtio-fs tag fails (ENOENT /
tag-not-found) for ALL CH hotplug.
- vminitd's mount RPC now, on a mount failure for a hot-plug-candidate
source (virtiofs type or /dev/vd* source), forces /sys/bus/pci/rescan
and retries with a bounded wait. Scoped so boot-time and non-device
mounts are unaffected.
Tests / build:
- First coverage of LinuxPod.addContainer after create() (CH-only; VZ
has no runtime hotplug): pod hotplug virtiofs rootfs and pod hotplug
block rootfs, registered in a Linux-only test list, plus a
directory-rootfs unpack helper.
- make linux-integration gains an optional FILTER passthrough (make
linux-integration FILTER="pod hotplug").
Signed-off-by: michael_crosby <michael_crosby@apple.com>
- Closes#807.
- Ensure everything builds and passes with latest conforming package
versions.
- Use swift-collections 1.5.1 instead of 1.6.0 due to Swift 6.3
limitations.
- Closes#787.
- The `blockSizeInBytes` parameter on `EXT4Unpacker`
was misleading — it does not represent an ext4 filesystem
block size.
- This change renames `blockSizeInBytes` to
`capacityInBytes`, updates the doc comment to accurately
describe the parameter and updates the name in the call
sites.
## Summary
Exclude `Sources/CloudHypervisor/README.md` from the `CloudHypervisor`
SwiftPM target. The README remains tracked and available as source
documentation, while package evaluation and builds no longer emit an
unhandled-file warning.
## Motivation
[apple/containerization#782](https://github.com/apple/containerization/pull/782)
added a standalone `CloudHypervisor` target with a README inside its
source directory. SwiftPM requires non-source files in target
directories to be declared as resources or excluded. This README is
documentation rather than a runtime resource, so exclusion is the
smallest accurate manifest change.
## Changes
- Add `README.md` to the `CloudHypervisor` target's `exclude` list.
- Preserve the existing README content and location.
- Leave runtime code, products, resources, and package APIs unchanged.
## Validation
```sh
swift package describe >/dev/null
swift test --disable-automatic-resolution --filter CloudHypervisorTests
make check
git diff --check upstream/main...HEAD
```
- Package evaluation completes without the
`Sources/CloudHypervisor/README.md` warning.
- All 41 focused `CloudHypervisorTests` pass.
- Swift formatting and Hawkeye license checks pass.
- The committed diff has no whitespace errors.
## Compatibility And Risk
This manifest-only change does not alter compiled sources, runtime
resources, public APIs, or supported platforms. The README is already
non-executable documentation, so excluding it accurately records
SwiftPM's intended treatment of the file.
## Related
- Follow-up to
[apple/containerization#782](https://github.com/apple/containerization/pull/782).
## Release Note Highlight
- Stops SwiftPM builds from warning about the `CloudHypervisor` README
added by
[apple/containerization#782](https://github.com/apple/containerization/pull/782).
- Fixes#776
- Release 2.9.0 of `grpc-swift-nio-transport` fixes
the HTTP/2 initialization race that required a
`HTTP2ConnectBufferingHandler` for the vminitd
API. This upgrades the dependency and eliminates
the workaround.
Changes the default Linux capability set for container processes from
`.allCapabilities` to `.defaultOCICapabilities`, making the library
secure-by-default. Callers that genuinely need elevated capabilities
must now opt in explicitly.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
apple/containerization currently runs containers in per-container VMs on
macOS hosts via Virtualization.framework. This adds a second VMM backend
so the same Swift orchestration layer (LinuxContainer / LinuxPod /
Vminitd gRPC contract) runs on Linux hosts via cloud-hypervisor + KVM.
**CloudHypervisor Swift package** (`Sources/CloudHypervisor/`) — a thin
client for cloud-hypervisor's REST-over-UDS API, layered on
AsyncHTTPClient. Endpoints cover VMM / VM lifecycle / hotplug (disk, fs,
net, vsock, remove-device). Cross-platform (compiles on macOS for unit
tests; consumed at runtime only by the Linux side of Containerization).
**CH backend in Containerization** — one cloud-hypervisor subprocess per
VM, gated behind `#if os(Linux)`. CHVirtualMachineManager /
CHVirtualMachineInstance mirror the VZ shape behind the existing
VirtualMachineManager / VirtualMachineInstance protocol. CHProcess and
VirtiofsdProcess manage the binaries; CHHotplugProvider handles
virtio-blk and virtio-fs runtime hotplug (with one virtiofsd per unique
source-hash tag, refcounted across containers).
**Linux host networking** — BridgeManager brings up a Linux bridge with
an IPv4 subnet and (opt-in via `--enable-nat`) iptables MASQUERADE +
scoped FORWARD rules. LinuxBridgedNetwork enslaves a fresh TAP per
container to the bridge. State is recorded under `/run/containerization`
so `cctl bridge delete` reverses exactly what create did. Bridge
teardown verifies the link kind via sysfs to refuse deleting non-bridge
interfaces.
**cctl run / bridge** — end-to-end Linux container run path (image pull,
ext4 rootfs assembly, VM boot, container exec) plus `cctl bridge
create|delete` for the host network plumbing.
**Build & dist** — `make linux-build` / `make linux-integration` build
and exercise the host side inside an apple/container `--virtualization`
dev container. `make dist-x86_64` produces a deployment tarball (cctl +
cloud-hypervisor + virtiofsd + initfs + kernel) cross-compiled from the
aarch64 dev container; pipeline documented in `docs/x86_64-build.md`.
Static-musl C deps and the Zig cross compiler are pinned by SHA256.
The host orchestrator runs as root. Per-VM runtime state lives under
`/run/containerization/ch/<UUID>` with mode 0700; UDS sockets inside are
bound with mode 0600. Vminitd's gRPC channel inherits that trust
boundary — socket-file perms are the auth.
Sandbox flags are upstream-secure by default. Two per-component opt-outs
exist for the apple/container dev-container case (where the host seccomp
profile SIGSYS-kills CH and virtiofsd):
- `CONTAINERIZATION_NO_CH_SECCOMP=1` — `cloud-hypervisor --seccomp
false`.
- `CONTAINERIZATION_NO_VIRTIOFSD_SANDBOX=1` — `virtiofsd --sandbox
none`. Each emits a one-shot `logger.warning` at process start. Legacy
alias `CONTAINERIZATION_RELAXED_SANDBOX=1` flips both. cctl spawns both
binaries with `setsid` and a minimal env allowlist (PATH / HOME /
RUST_LOG / RUST_BACKTRACE) so the parent's secrets don't leak to
children.
`make linux-integration` runs the cross-platform integration suite
against a real cloud-hypervisor VM inside the dev container. Linux runs
the cross-platform subset (`process true`/`false`/`echo hi`, virtiofs
round-trip, hotplug); the macOS suite is unchanged.
Signed-off-by: michael_crosby <michael_crosby@apple.com>
The `PodVolume` type in `LinuxPod` only defined the `nbd` enum value -
however, disk based images are also supported and the pattern is
essentially the same
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
When unpacking an OCI/tar layer, create() already creates missing parent
directories recursively, so regular files and symlinks with absent
parent entries unpack correctly. link() did not, so a hardlink whose
parent directory had no explicit archive entry failed with "<path> not
found" (e.g. images produced by Bazel rules_img). Mirror create()'s
implicit parent creation in link() so such layers unpack, matching
Docker/containerd.
Adds a direct link() unit test and an end-to-end unpack regression test
covering a hardlink, regular file, and symlink with no explicit parent.
Fixes https://github.com/apple/container/issues/1797
- Closes#774.
- Adds a default journal configuration parameter to the EXT4Unpacker
init function, defaulting to nil (no journal) to preserve source
compatibility.
The `--log-level` option when running the agent sub-command for vminitd
was being silently ignored cause of the way the agent is being run. As a
workaround we need to read `/proc/self/cmdline` to get the right args
Seems like a typo.
The change only affected a comment and does not disturb the code at all
(backward compatible).
Co-authored-by: J Logan <john_logan@apple.com>
- CI runners moved to the Xcode developer beta as the default, but macOS
builds are failing with a conflicting options error for
`-warnings-as-errors` and `-suppress-warnings`.
## Summary
`Platform.==` treats `arm64` with `nil` variant as equal to `arm64/v8`,
but `hash(into:)` used `description` which serializes them differently
(`linux/arm64` vs `linux/arm64/v8`). This violates the `Hashable`
contract — equal values must produce the same hash.
### Root cause
```swift
// == returns true for these two
let a = Platform(arch: "arm64", os: "linux", variant: nil)
let b = Platform(arch: "arm64", os: "linux", variant: "v8")
a == b // true ✓
// but hash was different — broken
a.hashValue == b.hashValue // false ✗ (before this fix)
```
This mismatch caused `Set<Platform>` and `Dictionary<Platform, ...>`
lookups to silently miss entries when one platform was decoded from JSON
(no `variant` field in the manifest) and another was created via
`Platform(from:)` or `Platform.current` (which both set `variant =
"v8"`).
### Practical consequence
In `apple/container`, this manifests as inconsistent platform-string
normalization across stages of a single `container build` — some stages
log `linux/arm64`, others `linux/arm64/v8` — which can cause `COPY
--from=<stage>` to fail to resolve the source stage under concurrent
builds. See apple/container#1542.
### Fix
`hash(into:)` now normalizes `arm64` with `nil` variant to `"v8"` before
hashing, matching the existing `==` behavior.
Noah Thornton