Add `Stat` RPC to vminitd.
`Stat` RPC is used to stat file system objects in the VM. It accepts the
path to the object (resolved against VM root) and returns the stat
result.
Make the `HTTP2ConnectBufferingHandler` class public temporarily so that
consumers of the containerization package may reuse it for other grpc
clients.
- Closes#671.
- Adds optional journal parameter to `EXT4.Formatter.init()`, with nil
default specifying the current no-journal filesystem configuration.
Otherwise the parameter value contains the journal size and mode.
- `minDiskSize` parameter for formatter init specifies the minimum
usable capacity of the resulting filesystem. The resulting disk image
grows past this value to accommodate the filesystem on-disk structures,
including the journal if specified.
- Setting 6.3 in the .swift-version is pulling in 6.3.1 but we can't go
there until there is a docker image for it, or Linux build breaks.
Alternatively we'd need to have different Swift SDKs for Linux musl
build vs macOS.
Now that we can run this project on linux, lets add some unit tests for
the surfaces that don't..
This additionally changes delete to throw in most cases. I don't really
see how masking these errors is ideal.
Closes#606
This lets the unit tests be runnable on linux. The change:
- Adds a linux-test makefile target so we can run the unit tests locally
- Fixes up some test code to work on Linux (mostly ifdefs)
- Runs the unit tests in CI now
We had a couple spots we were using this logic, probably better to shove
it in a function. The other thing this fixes is today .current was only
checking for arm64 which on a linux host uname with return aarch64, so
we'll fatalError trying to use it today..
Sets `memory.low` for `vminitd` cgroup to make kernel avoid reclaiming
its memory under high memory pressure. Even if the container process
runs in memory cgroup, consuming lots of page cache can lead to
reclaiming `vminitd`'s page cache memory , which hangs the `vminitd`.
Setting `memory.low` makes kernel avoid reclaiming `vminitd`'s page so
that we can guarantee the proper `vminitd` operation under high memory
pressure.
Closes: #678
There's no reason to have an idle timeout per connection as users may
use this library in many different ways. The grpc lib defaults to a 30
min idle timeout which could be trivially reached if instead of waiting
on the init process, you were waiting on a bunch of execs. NIO under the
hood asserts on ebadfs (which is good, but bad for this scenario..), so
lets turn off the timeout altogether.
- Closes#51. Again.
- No real notable API changes. The largest change code-wise is in
Vminitd. The connection loop is set up immediately in the new API by
calling runConnections() on the client, and this blocks until either the
task it's running on is cancelled or beginGracefulShutdown is called.
Because of this, either we'd have to modify our API to have
VirtualMachineAgent implementations call some run() method, or we just
put runConnections in an internal task on Vminitd. This change uses the
latter approach as it's simpler.
- Updated dependencies to pick up latest protobuf and grpc-swift-2
fixes.
- ClientBootstrap.withConnectedSocket(fd).wait() registers the vsock fd
with epoll/kqueue immediately. vminitd sends its HTTP/2 SETTINGS frame
right away upon accepting the connection, so those bytes can arrive on
the fd — and be read by NIO — before the gRPC pipeline is installed.
With a bare pipeline, NIO has nowhere to send the bytes and discards
them.
- Buffer until ClientConnectionHandler is in the pipeline so it can
process the SETTINGS frame and fire .ready, then schedule delivery of
the buffered data with assumeIsolatedUnsafeUnchecked().execute.
- To help produce more accurate merged logs, defines a
`StderrLogHandler` in the vminitd Application.swift that writes
timestamps to the boot log with millisecond precision and format
identical to that of the log files created using the `--log-root` arg on
`container`.
- Updates hawkeye to latest version (v6.5.1).
- In vminitd, use the Server event loop group for the gRPC server.
Co-authored-by: Danny Canter <danny_canter@apple.com>
This helps on binary size if we can successfully get all Foundation
imports out of our dep chain. It seems Foundation brings in a 30MiB ICU
blob, which bloats vmexec and vminitd.
Restores a configurable block size in the EXT4 formatter.
https://github.com/apple/containerization/pull/662 resolved a conflict
between the hardcoded log block size and a user-provided block size, but
removed the later. This PR restores the configurable block size without
creating the conflict.
- Fixes a hardlink count decrement threshold
- Fixes unlinking not freeing the first inode
Resolves the failing added tests:
```
✘ Test hardlinkLinksCount() recorded an issue at TestEXT4Format+Link.swift:43:9: Expectation failed: try EXT4.EXT4Reader(blockDevice: afterUnlink).stat("/original").inode.linksCount == 1
✘ Test hardlinkLinksCount() failed after 0.016 seconds with 1 issue.
```
```
✘ Test unlinkFirstInodeFreesInode() recorded an issue at TestEXT4Format+Link.swift:58:9: Expectation failed: try EXT4.EXT4Reader(blockDevice: path).superBlock.freeInodesCount == EXT4.EXT4Reader(blockDevice: emptyPath).superBlock.freeInodesCount
✘ Test unlinkFirstInodeFreesInode() failed after 0.014 seconds with 1 issue.
```
Today our CI builds all of the linux code we have with musl and libc to
ensure everything is silky smooth. We should have a way to do the same
locally. This adds a new macOS only goal that spins up a container
(requires `container` to be installed) that builds an image with our
deps (libarchive and compression libs) and then builds the project. It
supports supplying whatever libc (musl or glibc) to verify they both
work.
Fixes the `xattr` read loop bounds. Resolves the failing added tests:
> ✘ Test lastXattrNotDroppedAtBufferBoundary() recorded an issue at
TestEXT4ExtendedAttributes.swift:72:13: Expectation failed: (attrs.count
→ 0) == 1
> ✘ Test lastXattrNotDroppedAtBufferBoundary() failed after 0.001
seconds with 1 issue.
> Swift/Array.swift:430: Fatal error: Array index is out of range
Removes the incorrect check for visited inodes. Fixes the added failing
tests:
> ✘ Test sameAbsoluteSymlinkFollowedTwice() recorded an issue at
TestEXT4Reader+IO.swift:502:6: Caught error: symlink loop while
resolving: target/../symlink/file.txt
> ✘ Test sameAbsoluteSymlinkFollowedTwice() failed after 0.009 seconds
with 1 issue.
> ✘ Test sameRelativeSymlinkFollowedTwice() recorded an issue at
TestEXT4Reader+IO.swift:516:6: Caught error: symlink loop while
resolving: ../target/../symlink/file.txt
> ✘ Test sameRelativeSymlinkFollowedTwice() failed after 0.010 seconds
with 1 issue.
Fixes the construction of the path and removed unnecessary code. Fixes
the added failing test:
> ✘ Test fileTreeNodePathWithAbsoluteRoot() recorded an issue at
TestEXT4Reader+IO.swift:600:9: Expectation failed: (dirPtr.pointee.path
→ /) == (FilePath("/dir") → /dir)
> ✘ Test fileTreeNodePathWithAbsoluteRoot() recorded an issue at
TestEXT4Reader+IO.swift:601:9: Expectation failed: (filePtr.pointee.path
→ /) == (FilePath("/dir/file") → /dir/file)
> ✘ Test fileTreeNodePathWithAbsoluteRoot() failed after 0.001 seconds
with 2 issues.
Related to https://github.com/apple/container/issues/1251
Change to sharing in the parent directory, and then bind mounting in the
file into the container. This has the unfortunate reality of being less
secure, but the current approach is burdended by a couple things,
namely:
1. You can't share in files that are on a different volume
2. There is a Virtualization bug that causes spurious errors when trying
to open the file.
I've added a doc to go over the approach we've taken, and some
workarounds if the approach is not satisfactory.
Related to https://github.com/apple/containerization/issues/606
To make sure the project (and many linux specific libraries) builds on
linux lets add a linux step to build for both musl and glibc. Eventually
this will run unit tests as well. This change needs a bit more libc
agnostic-ifying work to make sure things work for both libcs.
To be able to test vminitd/vmexec/linux specific packages on ci it'd be
a heck of a lot easier if `make` just worked. This should be the last
bit needed. The default goal currently compiles just fine after the
linux specific `make deps` is ran. Next in line would be adding decent
unit tests/actually getting ci setup for the linux bits.
Fixes a force-unwrap crash on the last leaf of an extent tree:
> ContainerizationEXT4/EXT4+Formatter.swift:1148: Fatal error:
Unexpectedly found nil while unwrapping an Optional value
Adds a guard against an empty range in the EXT4 formatter to prevent a
possible crash:
> Swift/arm64e-apple-macos.swiftinterface:6314: Fatal error: Range
requires lowerBound <= upperBound
Removes not used `FilePath.init?(Data)` which had a buffer overread. It
used `String(cString:)` which reads until `\0`, but `Data` is not
null-terminated.
jwhur