oci: Add fast path for oci: transport using ocidir crate#233
Open
cgwalters wants to merge 5 commits intocomposefs:mainfrom
Open
oci: Add fast path for oci: transport using ocidir crate#233cgwalters wants to merge 5 commits intocomposefs:mainfrom
cgwalters wants to merge 5 commits intocomposefs:mainfrom
Conversation
cgwalters
force-pushed
the
fastpath-oci-import
branch
2 times, most recently
from
168de92 to
256bd97
Compare
Collaborator
Author
|
OK this now depends on bootc-dev/ocidir-rs#57 which cleans this up a lot. |
cgwalters
force-pushed
the
fastpath-oci-import
branch
from
256bd97 to
33304b6
Compare
cgwalters
force-pushed
the
fastpath-oci-import
branch
from
33304b6 to
cec8563
Compare
cgwalters
force-pushed
the
fastpath-oci-import
branch
from
f1be28a to
cc11746
Compare
Collaborator
|
Looks like this needs rebased after #248 landed |
cgwalters
force-pushed
the
fastpath-oci-import
branch
from
cc11746 to
2af7a01
Compare
Collaborator
Author
Done |
Collaborator
|
Conflicts again, and while I was looking at doing the rebase it's not totally clear to me what needs to happen since this intersects with the recent tar changes |
Collaborator
|
More concretely, I think we're losing the parts that added |
Extract decompression, tar import, blob storage, and media type checking from skopeo.rs and oci_image.rs into a reusable layer module. This prepares for adding a direct OCI layout import path that needs the same functionality without going through the skopeo proxy. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>
ocidir 0.7.1 adds open_image_this_platform() for resolving manifest lists, which is needed for the upcoming OCI layout fast path. Move both ocidir and cap-std-ext to workspace dependencies so they stay in sync across composefs-oci and integration-tests. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>
Use containers_image_proxy::ImageReference to parse the image reference once in pull_image() and pass it through to ImageOp::new(), which now takes &ImageReference instead of re-parsing the transport from the raw string. This also lets us use open_image_ref() instead of open_image(). This prepares for transport-based dispatch (e.g. fast-pathing oci: references) without manual string prefix matching. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>
For local OCI layout directories (oci: transport), read the layout directly using the ocidir crate instead of going through the containers-image-proxy / skopeo subprocess. This avoids subprocess spawning, IPC overhead, and proxy protocol parsing for local imports. The new oci_layout module handles manifest list resolution for the current platform via ocidir's open_image_this_platform(), imports layers in parallel using the shared layer module, and produces identical splitstream output to the proxy path. Assisted-by: OpenCode (Claude claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>
Ensure all import paths add named stream refs in the order that layers appear in the OCI image config (diff_ids), rather than in whatever order the import happens to process them (e.g. sorted by size for parallel fetching, or non-deterministic HashMap iteration). The skopeo, oci_layout, and write_config paths now iterate the config diff_ids array and look up layer verities by key, returning an error if any layer verity is missing. The write_manifest signature changes from HashMap to an ordered slice so callers control the order structurally. Assisted-by: OpenCode (claude-opus-4-6) Signed-off-by: Colin Walters <walters@verbum.org>
cgwalters
force-pushed
the
fastpath-oci-import
branch
from
2af7a01 to
bf7932e
Compare
Collaborator
Author
I think with this new resolution it's right - we're using the bufread changes in decompression (and when doing OCI artifact storage) but the tar parsing code is using the Bytes abstraction. |
Collaborator
Author
|
While not blocking this PR directly it'd be helpful to land bootc-dev/ocidir-rs#64 since I think as we head towards #224 we will need good support for referrers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When importing from local OCI layout directories (oci: transport), read
the layout directly instead of going through skopeo; this path
is faster.
But another motivation here is that right now the skopeo proxy
always peels manifest lists, but I want to add proper support
for storing those. Fetching from local
ocihelps us writeunit/integration tests for that without blocking on patching
skopeo.