fix(s3): adapt CopyObject to multistore's copy-source path mapping - #198
Merged
Conversation
Patch the multistore crates to the git `main` branch so CI exercises the CopyObject fixes queued for the 0.7.1 release (developmentseed/multistore#128, #129) ahead of publication. This is a temporary patch: the version requirements in `[dependencies]` already accept 0.7.1, so the `[patch.crates-io]` block is dropped once the release lands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
Claude finished @alukach's task in 3m 50s —— View job Review
✅ No blocking issues — safe to merge.
I wasn't able to run |
|
🚀 Latest commit deployed to https://source-data-proxy-pr-198.source-coop.workers.dev
|
`CopyObject` names its source in a header rather than the URL, in client coordinates (`/account/product/key`). This deployment is path-mapped, so the registry only knows mapped bucket names — an unmapped source resolves to a bucket it has never heard of and every copy fails with 404 NoSuchBucket. multistore 0.7.1 adds `RequestInfo::with_copy_source` for exactly this (developmentseed/multistore#128), but it is opt-in: the host app has to supply the mapped value. Wire it up via `PathMapping::rewrite_copy_source`. The client signed the original header, so it is left untouched and signature verification still uses it as sent; only the out-of-band mapped value differs. Adds two integration tests: an anonymous copy is denied at the authz gate, and a credentialed copy must reach the federation seam rather than dying as NoSuchBucket — which is what distinguishes a mapped source from an unmapped one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This reverts commit a389871.
The integration test added alongside the mapping fix claimed to pin it but could not: federation is attempted against the destination before the copy source is resolved, so a copy fails identically whether or not `with_copy_source` is wired up. Verified by deleting the wiring and pushing — CI stayed green. Move the check to a layer that can fail. The header-to-mapped-source logic is extracted from `fetch` into `object_path::mapped_copy_source` (that module is deliberately wasm-free so it can be unit-tested natively, the lib being cdylib with `test = false`), and covered there: the account/product fold, leading-slash normalization, nested keys, `versionId` and percent-encoding preservation, and the two None cases. These fail without the mapping. The integration test keeps its narrower real value — a copy is parsed and dispatched as a copy and fails closed at federation — and its docstring now states what it does not prove. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0.7.1 is released, so drop the temporary `[patch.crates-io]` block that pointed the multistore crates at git `main` and pin the released versions instead. `Cargo.lock` now resolves them from crates.io with checksums rather than a moving branch ref. This is what unblocks the copy-source mapping in this branch: `RequestInfo::with_copy_source` is new in 0.7.1 and absent from 0.7.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…suite A copy is authorized in two halves — the destination as a write, the source as a synthetic GetObject. Nothing asserted the source half, so a caller holding write on some product could in principle have used CopyObject to read a product they aren't entitled to. multistore covers this upstream (#128's `copy_source_read_enforces_prefix_via_get_object`); this pins it through this repo's subject-scoped registry wiring. It goes in the deployed-environment suite rather than the CI integration tier because CI could not fail it: federation is attempted against the destination before the source is resolved, so on CI's throwaway signing key every copy dies at federation first — and `AccessDenied` is itself a federation error code, so the assertion would pass without proving anything. The staging suite has a deployment where the destination write genuinely succeeds, leaving the source half as the only thing that can deny the copy. Dormant like the rest of that suite: it needs FEDERATION_WRITE_PRODUCT (a product the caller may write) plus a caller identity the *deployed* worker accepts — whose audience is not CI's `source-data-proxy-ci`, so the token is minted only when FEDERATION_TEST_AUDIENCE names the deployment's AUTH_AUDIENCE. Unset, the test skips and the rest of the suite is unaffected. staging.yml also gains boto3, which the SigV4 client needs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint step added with the copy-source authz test could never have produced a usable caller. That job talks to the *deployed* worker, whose AUTH_ISSUER is Ory (`https://auth.staging.source.coop`, see wrangler.toml [env.staging.vars]) — a GitHub Actions OIDC token fails issuer verification before its audience is ever examined, so no FEDERATION_TEST_AUDIENCE value could have made it authenticate. The variable was a knob on a door that doesn't open; it is removed rather than documented, along with the `id-token: write` permission it needed. The token now comes from a secret, FEDERATION_CALLER_TOKEN, which must be issued by the deployment's own issuer with an `aud` among its AUTH_AUDIENCE client_ids and a subject holding write on FEDERATION_WRITE_PRODUCT. The test's gate is unchanged, so it still skips cleanly until that is provisioned. Not added to the wrangler files: this is a property of the CI caller, while AUTH_AUDIENCE there is what the worker accepts. Duplicating it would be a second source of truth able to drift from the value it must match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit replaced the unusable GitHub-OIDC mint with `secrets.FEDERATION_CALLER_TOKEN`, which is no better: an Ory ID token is short-lived, so a token parked in a repo secret goes stale within the hour. Setting it would buy a brief window of green followed by failures that read as an authorization regression rather than an expired credential — worse than the test plainly skipping. Supplying a caller needs a step that mints a token per run from a durable credential (a service-account client_id/secret, or a refresh token) against the deployment's own issuer. That is a provisioning decision this PR has no basis to make, so no token is wired and the copy-source authz test skips, with the requirement recorded where someone acting on it will look. The unambiguous parts of the earlier wiring stay: boto3, which the SigV4 client needs, and the FEDERATION_WRITE_PRODUCT passthrough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restores the per-run GitHub OIDC mint for the copy-source authz test, ahead of registering GitHub as a valid IdP with Source so products can accept writes from GitHub Actions. This is the right shape for the credential: minted per run and short-lived, with nothing durable stored. The earlier attempt at a repo secret was worse — an ID token parked there goes stale within the hour, and the resulting failures read as an authorization regression rather than an expired credential. Still dormant, now waiting on the IdP registration rather than on a design decision. Two things must land before it can run: the deployment's AUTH_ISSUER must accept GitHub's issuer, which is a code change and not only config — src/config.rs reads AUTH_ISSUER as a single String, unlike the comma-separated AUTH_AUDIENCE — and FEDERATION_TEST_AUDIENCE must name the audience Source expects. Until both, no token is minted and the test skips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alukach
pushed a commit
that referenced
this pull request
Jul 28, 2026
🤖 I have created a release *beep* *boop* --- ## [2.3.1](v2.3.0...v2.3.1) (2026-07-28) ### Bug Fixes * **s3:** adapt CopyObject to multistore's copy-source path mapping ([#198](#198)) ([40acbcb](40acbcb)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: source-release-bot[bot] <265100246+source-release-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps multistore to 0.7.1 and adapts this proxy to the copy-source handling it introduces.
Why
CopyObjectnames its source in thex-amz-copy-sourceheader rather than the URL, in client coordinates (/account/product/key). This deployment is path-mapped, so the registry only knows mapped bucket names (account:product) — an unmapped source resolves to a bucket it has never heard of and the copy fails with 404 NoSuchBucket. That affects the server-side CopyObject support landed in 97d5121.multistore 0.7.1 provides the mechanism for this (#128), but it is opt-in:
RequestInfo::with_copy_sourcetakes a mapped value that the host app must supply. This wires it up.The bump also picks up #129, which authorizes a versioned copy-source against its version.
Changes
Cargo.toml/Cargo.lock— multistore crates to 0.7.1, from crates.io with checksums.object_path::mapped_copy_source— maps the header throughPathMapping::rewrite_copy_source, returningNonefor a non-copy request or an unmappable value. Lives inobject_pathbecause that module is deliberately wasm-free and so can be unit-tested natively, the lib beingcdylibwithtest = false.src/lib.rs— passes the mapped value viawith_copy_source. The client signed the original header, so it is left untouched and signature verification still uses it as sent; only the value passed alongside differs.Tests
The mapping is pinned natively in
tests/object_path.rs: the account/product fold, leading-slash normalization, nested keys,versionIdand percent-encoding preservation, and bothNonecases.Natively rather than through an integration test, because an integration test cannot detect this regression: federation is attempted against the destination before the copy source is resolved, so in CI — where the throwaway signing key means federation never succeeds — a copy fails identically whether or not the mapping is wired up. This was verified, not assumed.
Copy-source authorization — a caller holding write on the destination must not be able to read a product they aren't entitled to via
CopyObject— is asserted intests/test_federation.py, the deployed-environment suite. Same reason: in CI the destination write never succeeds, andAccessDeniedis itself a federation error code, so the assertion would pass without proving anything. It is dormant and skips until bothFEDERATION_WRITE_PRODUCTand a caller token are supplied, and skips cleanly otherwise;staging.ymlgains boto3, theFEDERATION_WRITE_PRODUCTpassthrough, and a step minting the caller as a GitHub Actions OIDC token — short-lived and per-run, with nothing stored. This anticipates registering GitHub as a valid IdP with Source so products can accept writes from GitHub Actions.It stays dormant until two things land: the deployment's
AUTH_ISSUERmust accept GitHub's issuer, which is a code change and not only config —src/config.rs:80readsAUTH_ISSUERas a singleString, unlike the comma-separatedAUTH_AUDIENCEatsrc/config.rs:87— andFEDERATION_TEST_AUDIENCEmust name the audience Source expects. Until both, no token is minted and the test skips.Two integration tests in
tests/test_writes.pycover what that layer can genuinely show, CopyObject having had no coverage there at all:test_anonymous_copy_is_denied(a copy is a write, so it dies at the authz gate) andtest_copy_reaches_the_federation_seam(a copy is parsed and dispatched as a copy, and fails closed).Verification
cargo test— 73 pass across 8 suites (was 69)cargo check --lib --target wasm32-unknown-unknown— cleanwrangler dev; credentialed tests need a GitHub OIDC token and run in CI🤖 Generated with Claude Code