This PR masks the high mode bits when setting file attributes for
consistency with mode at file creation.
Signed-off-by: Kathryn Baldauf <k_baldauf@apple.com>
Seems like a typo.
The change only affected a comment and does not disturb the code at all
(backward compatible).
Co-authored-by: J Logan <john_logan@apple.com>
- Closes#745.
- Facilitates TOCTOU-safe recursion over directory contents.
- Replace FileDescriptor extensions with a static utility type to
prevent potential namespacing issues as this project and Swift evolve.
This PR refactors `ArchiveWriter` to add an API `archive(_ paths:
base:)`. This API is used to archive the contents at each path
independently, similar to doing `tar -cvf archive.tar foo.bin
/bar/baz.txt`.
I've added an optional progress handler for rootfs unpacking so that
consumers can show progress. The Ubuntu image takes about 8 seconds to
unpack on my machine, and I'm developing an app where it would be useful
to show progress in the UI.
Total size is determined in an optional first pass that scans archive
headers. Bytes written are then reported during unpacking. The optional
first pass adds 15 (Alpine) to 115 (Ubuntu) ms to unpacking duration on
my machine, depending on image size.
finishEncoding() called archive_free() and then passed the freed pointer
to wrap(), which would call archive_error_string() on it if the return
code was not ARCHIVE_OK. Fix by nil-ing the pointer before freeing and
throwing directly without attempting to read the (already freed) error
string.
Fix a bug in ArchiveWriter where large files would only partly be
written into an archive.
This is done so by writing the data in chunks.
---------
Signed-off-by: Aditya Ramani <a_ramani@apple.com>
ContainerizationArchive internally used a `FileArchiveWriterDelegate`
type that defined callback methods which were handed to
`archive_write_open2` from libarchive
```
int archive_write_open2(struct archive *, void *,
archive_open_callback *, archive_write_callback *,
archive_close_callback *, archive_free_callback *);
```
Turns out those methods were at the very least partly broken, when used
alongside gzip compression.
Utilities like gnutar and gunzip couldn't entirely process the archives
produced by them, leading to errors like
```
➜ gtar -tvzf rootfs.tar.gz
...
drwxr-xr-x 0/0 0 2026-03-07 11:24 sbin/
-rwxr-xr-x 0/0 122451504 2026-03-07 11:24 sbin/vminitd
-rwxr-xr-x 0/0 121138856 2026-03-07 11:24 sbin/vmexec
gzip: (stdin): trailing garbage ignored
..
gtar: Child returned status 2
gtar: Error is not recoverable: exiting now
```
libarchive provides alternate APIs, `archive_write_open_filename` and
`archive_write_open_fd` where these callbacks are defined within the
library.
Lets use those instead and let the library handle this / future edge
cases (in this particular case it was adding some padding at the end of
the gzipped compressed file)
Now that the static linux SDK we use has libarchive linked against it,
we can finally live our dream (/s) of supporting copying directories
in/out a little easier. This was.. kind of annoying. I think the very
simple route of writing archive to guest/host -> streaming over grpc is
simplest, but on the guest end we have a couple problems:
1. The VMs rootfs is read only today always, which is a good thing to me
and I don't want to change if we don't have to.
2. Writing the archive to the containers rootfs temporarily could work,
but it's a bit weird, and the user can make the containers rootfs
readonly which would screw that plan.
3. We could write it to /run or /tmp, but they're tmpfs and dealing with
the headache of the user possibly tarring an enormous dir is one I don't
want to care about.
So, that leaves us with the (truthfully better to me) approach of trying
to write the tar data directly to the host and skipping grpc which kinda
forces us to have a temp spot. Because of that, I made it such that we
pass a port number from host<->guest, and transfer the actual data
(either the single file or tarred dir contents) over the vsock port
instead. The stream is kinda clunky, but it just serves as a means to
exchange metadata and a "we're done" signifier.
- Closes#532.
- Relocate all zstd calls down into CArchive, and export a function
zstd_decompress_fd(src_fd, dst_fd) that ContainerizationArchive can use.
This eliminates problems that can occur when treating Swift warnings as
errors in Xcode projects that use ContainerizationArchive.
- InputSource(url:) rejects URLs with nil scheme, which was what was
passed when running the `container system kernel set` integration test
with a local kernel archive path.
- Closesapple/container#988.
- macOS libarchive is not built with zstd support, so the workaround in
ArchiveReader is to attempt to decompress every archive as zstd. If
decompression fails, we pass the original archive to libarchive. If it
succeeds, we pass the uncompressed archive.
- Adds blob media type recognition for zstd to EXT4Unpacker. Tested zstd
blob unpack using `image pull tonistiigi/hello-world:zstd-docker`.
Co-authored-by: Aditya Ramani <a_ramani@apple.com>
* Admit only valid OCI layout files when loading image.
- Adds `AdmissionMapper` protocol for validating archive
member paths and normalizing them to relative paths
under the extraction root directory.
- Renames `Reader.swift` to `ArchiveReader.swift` to
match type name.
- Rework `TempDir` to address `NSString.utf8String`
deprecation warning in Swift 6.2.3.
- Rework `ArchiveReader.extractContent()` to use an
`AdmissionMapper` to validate and remap archive
members before extracting.
- Adds `IdentityAdmissionWrapper` for naive extraction.
- Adds `NoSymlinkAdmissionWrapper` that only extracts
regular files and directories under the extraction root.
- Adds `OCIImageAdmissionWrapper` that only extracts
valid OCI image layout paths.
- Use `OCIImageAdmissionWrapper` for `cctl image load`
and print rejected paths.
* PR feedback.
* Adds public init() for TrustedAdmissionMapper.
* Replace AdmissionMapper with more secure extraction.
- Adds FileDescriptor.mkdirSecure() to prevent
root escapes on member pathnames, and to prevent
symlink traversal.
- Adds FileDescriptor.unlinkRecursive() to
facilitate overwrites when there are multiple
archive entries with the same member path.
- Adds FileDescriptor.validateSymlinkTargetInRoot()
to validate that extracted symlink targets do
not escape the root.
- Rewrite ArchiveReader.extractContents() to use
secure path functions.
* Remove unneeded symlink check, rename files.
* Simplify the lexical normalizer workaround.
* Reject member paths containing parent traversal components.
* Remove unused lexical normalization workaround.
* Fix leaking fds, extract absolute members as relative.
Optimize unpack a little by trying to reduce allocations in the hot
path. Today for every file we read the entire file into memory and then
pass the data blob to the ext4 writer to eventually be written to the
sparse file. Before being written to the sparse file the data is copied
*again* to a temp buffer before finally hitting write(2) in FileHandle.
This change moves things around such that we can pass an optional buffer
to the ext4 create() (so we can reuse a buffer for file writes), as well
as stops reading entire files into memory by passing the archive entry
itself (wrapped in a ReaderStream object albeit) down to the writer.
Testing with unpacking every platform for
`docker.io/jenkins/jenkins:lts` on an M1 Max:
Old Avg (5 runs): 7.43s
New Avg (5 runs): 5.31s
Continue the allocations journey for anything that is in the codepaths
for pulling images. This time there's a couple spots in archive and ext4
we can get rid of some copies.
Kathryn Baldauf