This issue doesn't affect any of our existing products. This is a
preemptive fix for downstream consumers of EXT4.format where, in some
platforms, leading `//` in the path could get resolved into a FileTree
that looks like this
```sh
/
└── /
└── usr
```
If you're requesting on `/v2/` with basic auth, AWS ECR will return 400
Bad Request and won't provide `www-authenticate` information.
Retry the request after removing the Authorization header.
Continue PR #429Fixedapple/container#847
Signed-off-by: ChengHao Yang <17496418+tico88612@users.noreply.github.com>
If someone wanted to use a custom guest agent/guest image in general the
overhead we chose is kind of a nuisance.. This allows the values to be
configurable, and raises the default to 128, and vminitds cgroup
monitoring to 80.
This change adds an additional unit test to test
attaching a raw block device to a container to ensure raw block works
without API changes. We just need to set format to none and bind mount
the raw block device to the container path
This change adds support for attaching network block device (NBD) to
both LinuxContainer and LinuxPod.
For LinuxContainer, whether to use the underlying
`VZNetworkBlockDeviceStorageDeviceAttachment` is determined by the URL
of the container Mount source.
For LinuxPod, adds additional API to support pod-level volumes that can
be mounted into multiple containers. The PodVolume type provides enum to
support multiple types of volume source. LinuxContainer can reference
the pod level volume using the `Mount.sharedMount()` constructor with
the name referencing the name of the pod volume. This will allow the NBD
to be attached to the pod at the VM level and then bind-mounted into the
container.
For integration tests, added a lightweight NBD server implementation in
swift that speaks the NBD protocol to ensure there is sufficient
coverage for the changes introduced.
Apple Virtualization NBD support documentation:
https://developer.apple.com/documentation/virtualization/vznetworkblockdevicestoragedeviceattachment
Replace the old platform-gated Signals helper in ContainerizationOS with
a Signal struct in Containerization. Signal provides named constants for
common signals (.kill, .term, etc.), platform-specific namespaces
(Signal.Linux for rtmin/stkflt/pwr, Signal.Darwin for emt/info), string
parsing, and ExpressibleByIntegerLiteral support.
LinuxContainer, LinuxProcess, and LinuxPod kill APIs now take Signal
instead of raw Int32.
Add `Stat` RPC to vminitd.
`Stat` RPC is used to stat file system objects in the VM. It accepts the
path to the object (resolved against VM root) and returns the stat
result.
Make the `HTTP2ConnectBufferingHandler` class public temporarily so that
consumers of the containerization package may reuse it for other grpc
clients.
- Closes#671.
- Adds optional journal parameter to `EXT4.Formatter.init()`, with nil
default specifying the current no-journal filesystem configuration.
Otherwise the parameter value contains the journal size and mode.
- `minDiskSize` parameter for formatter init specifies the minimum
usable capacity of the resulting filesystem. The resulting disk image
grows past this value to accommodate the filesystem on-disk structures,
including the journal if specified.
- Setting 6.3 in the .swift-version is pulling in 6.3.1 but we can't go
there until there is a docker image for it, or Linux build breaks.
Alternatively we'd need to have different Swift SDKs for Linux musl
build vs macOS.
Now that we can run this project on linux, lets add some unit tests for
the surfaces that don't..
This additionally changes delete to throw in most cases. I don't really
see how masking these errors is ideal.
Closes#606
This lets the unit tests be runnable on linux. The change:
- Adds a linux-test makefile target so we can run the unit tests locally
- Fixes up some test code to work on Linux (mostly ifdefs)
- Runs the unit tests in CI now
We had a couple spots we were using this logic, probably better to shove
it in a function. The other thing this fixes is today .current was only
checking for arm64 which on a linux host uname with return aarch64, so
we'll fatalError trying to use it today..
Sets `memory.low` for `vminitd` cgroup to make kernel avoid reclaiming
its memory under high memory pressure. Even if the container process
runs in memory cgroup, consuming lots of page cache can lead to
reclaiming `vminitd`'s page cache memory , which hangs the `vminitd`.
Setting `memory.low` makes kernel avoid reclaiming `vminitd`'s page so
that we can guarantee the proper `vminitd` operation under high memory
pressure.
Closes: #678
There's no reason to have an idle timeout per connection as users may
use this library in many different ways. The grpc lib defaults to a 30
min idle timeout which could be trivially reached if instead of waiting
on the init process, you were waiting on a bunch of execs. NIO under the
hood asserts on ebadfs (which is good, but bad for this scenario..), so
lets turn off the timeout altogether.
- Closes#51. Again.
- No real notable API changes. The largest change code-wise is in
Vminitd. The connection loop is set up immediately in the new API by
calling runConnections() on the client, and this blocks until either the
task it's running on is cancelled or beginGracefulShutdown is called.
Because of this, either we'd have to modify our API to have
VirtualMachineAgent implementations call some run() method, or we just
put runConnections in an internal task on Vminitd. This change uses the
latter approach as it's simpler.
- Updated dependencies to pick up latest protobuf and grpc-swift-2
fixes.
- ClientBootstrap.withConnectedSocket(fd).wait() registers the vsock fd
with epoll/kqueue immediately. vminitd sends its HTTP/2 SETTINGS frame
right away upon accepting the connection, so those bytes can arrive on
the fd — and be read by NIO — before the gRPC pipeline is installed.
With a bare pipeline, NIO has nowhere to send the bytes and discards
them.
- Buffer until ClientConnectionHandler is in the pipeline so it can
process the SETTINGS frame and fire .ready, then schedule delivery of
the buffered data with assumeIsolatedUnsafeUnchecked().execute.
- To help produce more accurate merged logs, defines a
`StderrLogHandler` in the vminitd Application.swift that writes
timestamps to the boot log with millisecond precision and format
identical to that of the log files created using the `--log-root` arg on
`container`.
- Updates hawkeye to latest version (v6.5.1).
- In vminitd, use the Server event loop group for the gRPC server.
Co-authored-by: Danny Canter <danny_canter@apple.com>
This helps on binary size if we can successfully get all Foundation
imports out of our dep chain. It seems Foundation brings in a 30MiB ICU
blob, which bloats vmexec and vminitd.
Restores a configurable block size in the EXT4 formatter.
https://github.com/apple/containerization/pull/662 resolved a conflict
between the hardcoded log block size and a user-provided block size, but
removed the later. This PR restores the configurable block size without
creating the conflict.
- Fixes a hardlink count decrement threshold
- Fixes unlinking not freeing the first inode
Resolves the failing added tests:
```
✘ Test hardlinkLinksCount() recorded an issue at TestEXT4Format+Link.swift:43:9: Expectation failed: try EXT4.EXT4Reader(blockDevice: afterUnlink).stat("/original").inode.linksCount == 1
✘ Test hardlinkLinksCount() failed after 0.016 seconds with 1 issue.
```
```
✘ Test unlinkFirstInodeFreesInode() recorded an issue at TestEXT4Format+Link.swift:58:9: Expectation failed: try EXT4.EXT4Reader(blockDevice: path).superBlock.freeInodesCount == EXT4.EXT4Reader(blockDevice: emptyPath).superBlock.freeInodesCount
✘ Test unlinkFirstInodeFreesInode() failed after 0.014 seconds with 1 issue.
```
Today our CI builds all of the linux code we have with musl and libc to
ensure everything is silky smooth. We should have a way to do the same
locally. This adds a new macOS only goal that spins up a container
(requires `container` to be installed) that builds an image with our
deps (libarchive and compression libs) and then builds the project. It
supports supplying whatever libc (musl or glibc) to verify they both
work.
Fixes the `xattr` read loop bounds. Resolves the failing added tests:
> ✘ Test lastXattrNotDroppedAtBufferBoundary() recorded an issue at
TestEXT4ExtendedAttributes.swift:72:13: Expectation failed: (attrs.count
→ 0) == 1
> ✘ Test lastXattrNotDroppedAtBufferBoundary() failed after 0.001
seconds with 1 issue.
> Swift/Array.swift:430: Fatal error: Array index is out of range
Removes the incorrect check for visited inodes. Fixes the added failing
tests:
> ✘ Test sameAbsoluteSymlinkFollowedTwice() recorded an issue at
TestEXT4Reader+IO.swift:502:6: Caught error: symlink loop while
resolving: target/../symlink/file.txt
> ✘ Test sameAbsoluteSymlinkFollowedTwice() failed after 0.009 seconds
with 1 issue.
> ✘ Test sameRelativeSymlinkFollowedTwice() recorded an issue at
TestEXT4Reader+IO.swift:516:6: Caught error: symlink loop while
resolving: ../target/../symlink/file.txt
> ✘ Test sameRelativeSymlinkFollowedTwice() failed after 0.010 seconds
with 1 issue.
Fixes the construction of the path and removed unnecessary code. Fixes
the added failing test:
> ✘ Test fileTreeNodePathWithAbsoluteRoot() recorded an issue at
TestEXT4Reader+IO.swift:600:9: Expectation failed: (dirPtr.pointee.path
→ /) == (FilePath("/dir") → /dir)
> ✘ Test fileTreeNodePathWithAbsoluteRoot() recorded an issue at
TestEXT4Reader+IO.swift:601:9: Expectation failed: (filePtr.pointee.path
→ /) == (FilePath("/dir/file") → /dir/file)
> ✘ Test fileTreeNodePathWithAbsoluteRoot() failed after 0.001 seconds
with 2 issues.
Related to https://github.com/apple/container/issues/1251
Change to sharing in the parent directory, and then bind mounting in the
file into the container. This has the unfortunate reality of being less
secure, but the current approach is burdended by a couple things,
namely:
1. You can't share in files that are on a different volume
2. There is a Virtualization bug that causes spurious errors when trying
to open the file.
I've added a doc to go over the approach we've taken, and some
workarounds if the approach is not satisfactory.
Related to https://github.com/apple/containerization/issues/606
To make sure the project (and many linux specific libraries) builds on
linux lets add a linux step to build for both musl and glibc. Eventually
this will run unit tests as well. This change needs a bit more libc
agnostic-ifying work to make sure things work for both libcs.
Sidhartha Mani