Skip to content

fix(s3): adapt CopyObject to multistore's copy-source path mapping - #198

Merged
alukach merged 10 commits into
mainfrom
test/multistore-main
Jul 28, 2026
Merged

fix(s3): adapt CopyObject to multistore's copy-source path mapping#198
alukach merged 10 commits into
mainfrom
test/multistore-main

Conversation

@alukach

@alukach alukach commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Bumps multistore to 0.7.1 and adapts this proxy to the copy-source handling it introduces.

Why

CopyObject names its source in the x-amz-copy-source header rather than the URL, in client coordinates (/account/product/key). This deployment is path-mapped, so the registry only knows mapped bucket names (account:product) — an unmapped source resolves to a bucket it has never heard of and the copy fails with 404 NoSuchBucket. That affects the server-side CopyObject support landed in 97d5121.

multistore 0.7.1 provides the mechanism for this (#128), but it is opt-in: RequestInfo::with_copy_source takes a mapped value that the host app must supply. This wires it up.

The bump also picks up #129, which authorizes a versioned copy-source against its version.

Changes

  • Cargo.toml / Cargo.lock — multistore crates to 0.7.1, from crates.io with checksums.
  • object_path::mapped_copy_source — maps the header through PathMapping::rewrite_copy_source, returning None for a non-copy request or an unmappable value. Lives in object_path because that module is deliberately wasm-free and so can be unit-tested natively, the lib being cdylib with test = false.
  • src/lib.rs — passes the mapped value via with_copy_source. The client signed the original header, so it is left untouched and signature verification still uses it as sent; only the value passed alongside differs.

Tests

The mapping is pinned natively in tests/object_path.rs: the account/product fold, leading-slash normalization, nested keys, versionId and percent-encoding preservation, and both None cases.

Natively rather than through an integration test, because an integration test cannot detect this regression: federation is attempted against the destination before the copy source is resolved, so in CI — where the throwaway signing key means federation never succeeds — a copy fails identically whether or not the mapping is wired up. This was verified, not assumed.

Copy-source authorization — a caller holding write on the destination must not be able to read a product they aren't entitled to via CopyObject — is asserted in tests/test_federation.py, the deployed-environment suite. Same reason: in CI the destination write never succeeds, and AccessDenied is itself a federation error code, so the assertion would pass without proving anything. It is dormant and skips until both FEDERATION_WRITE_PRODUCT and a caller token are supplied, and skips cleanly otherwise; staging.yml gains boto3, the FEDERATION_WRITE_PRODUCT passthrough, and a step minting the caller as a GitHub Actions OIDC token — short-lived and per-run, with nothing stored. This anticipates registering GitHub as a valid IdP with Source so products can accept writes from GitHub Actions.

It stays dormant until two things land: the deployment's AUTH_ISSUER must accept GitHub's issuer, which is a code change and not only configsrc/config.rs:80 reads AUTH_ISSUER as a single String, unlike the comma-separated AUTH_AUDIENCE at src/config.rs:87 — and FEDERATION_TEST_AUDIENCE must name the audience Source expects. Until both, no token is minted and the test skips.

Two integration tests in tests/test_writes.py cover what that layer can genuinely show, CopyObject having had no coverage there at all: test_anonymous_copy_is_denied (a copy is a write, so it dies at the authz gate) and test_copy_reaches_the_federation_seam (a copy is parsed and dispatched as a copy, and fails closed).

Verification

  • cargo test — 73 pass across 8 suites (was 69)
  • cargo check --lib --target wasm32-unknown-unknown — clean
  • Local run against stub API + wrangler dev; credentialed tests need a GitHub OIDC token and run in CI

🤖 Generated with Claude Code

Patch the multistore crates to the git `main` branch so CI exercises the
CopyObject fixes queued for the 0.7.1 release
(developmentseed/multistore#128, #129) ahead of publication.

This is a temporary patch: the version requirements in `[dependencies]`
already accept 0.7.1, so the `[patch.crates-io]` block is dropped once the
release lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Claude finished @alukach's task in 3m 50s —— View job


Review

  • Gather context (diff, changed files)
  • Review src/object_path.rs mapping logic
  • Review src/lib.rs wiring
  • Review tests and CI workflow changes
  • Post final review

✅ No blocking issues — safe to merge.

  • src/object_path.rs:42-50mapped_copy_source correctly returns None both for absent headers and unmappable values, and the docstring correctly identifies why that fallback is safe (an unmappable source fails on its own merits rather than resolving to an invented bucket). src/lib.rs:322,334 reuses the same mapping instance already built for MappedRegistry, so the copy-source fold uses the identical account/product rule as the rest of the request — no drift between the two mappings.
  • src/lib.rs:322-334 — the client-signed x-amz-copy-source header itself is left untouched in parts.headers; only the mapped value is threaded through with_copy_source, so SigV4 verification still runs against what the client actually signed. This matches the stated intent and is the crux of the fix.
  • tests/object_path.rs — good coverage: bucket folding, optional leading slash, nested keys, versionId/percent-encoding preservation, and both None cases (absent header, too few segments). This is the one test tier that can actually catch a regression here, per the PR's own reasoning about the federation seam masking it in integration tests — confirmed by reading tests/test_writes.py:420-431, which documents that it was verified by deleting the wiring and watching the integration test still pass.
  • .github/workflows/staging.yml:71${{ vars.FEDERATION_TEST_AUDIENCE }} is interpolated directly into the run: shell block rather than passed through env:. Normally this pattern is flagged as a script-injection risk, but this workflow only triggers on push to main (no pull_request/fork input), and vars.* are maintainer-controlled, not attacker-controlled — so this is a low-risk style nit, not a blocking issue. Optional hardening if you want to preempt a future linter flag:
    env:
      AUDIENCE: ${{ vars.FEDERATION_TEST_AUDIENCE }}
    run: |
      set -euo pipefail
      token=$(curl -sSf -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
        "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$AUDIENCE" \
        | jq -r '.value')
  • The new id-token: write permission is scoped to the federation-smoke job only (least privilege), the minted token is masked before being written to GITHUB_ENV, and the mint step is correctly gated behind FEDERATION_TEST_AUDIENCE so it stays fully inert until that variable is set.
  • tests/test_federation.py's new test_copy_source_authorization_is_enforced correctly accepts 401/403/404 as success (a source-side auth failure could plausibly surface as any of the three depending on whether the check happens before or after existence), and its skip condition doesn't require FEDERATION_TEST_KEY, falling back to a literal "any-key" — reasonable since the assertion is about denial, not object content.
  • tests/test_writes.py's new test_anonymous_copy_is_denied expecting 403 is consistent with the existing convention used by other anonymous-write-denied tests in the same file (e.g. line 128).

I wasn't able to run cargo test/cargo check in this sandbox (network access for crate resolution requires approval that isn't available in this context), so I relied on static reading plus the PR's own stated verification (73 tests passing, cargo check --lib --target wasm32-unknown-unknown clean). Nothing in the diff contradicts those claims.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown

🚀 Latest commit deployed to https://source-data-proxy-pr-198.source-coop.workers.dev

  • Date: 2026-07-28T00:24:49Z
  • Commit: 923e3e0

`CopyObject` names its source in a header rather than the URL, in client
coordinates (`/account/product/key`). This deployment is path-mapped, so the
registry only knows mapped bucket names — an unmapped source resolves to a
bucket it has never heard of and every copy fails with 404 NoSuchBucket.

multistore 0.7.1 adds `RequestInfo::with_copy_source` for exactly this
(developmentseed/multistore#128), but it is opt-in: the host app has to supply
the mapped value. Wire it up via `PathMapping::rewrite_copy_source`. The client
signed the original header, so it is left untouched and signature verification
still uses it as sent; only the out-of-band mapped value differs.

Adds two integration tests: an anonymous copy is denied at the authz gate, and
a credentialed copy must reach the federation seam rather than dying as
NoSuchBucket — which is what distinguishes a mapped source from an unmapped one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alukach alukach changed the title chore(deps): test against multistore main (unreleased 0.7.1) fix(copy): map x-amz-copy-source on this path-mapped deployment (multistore main / 0.7.1) Jul 27, 2026
@source-release-bot source-release-bot Bot added fix and removed chore labels Jul 27, 2026
@alukach alukach changed the title fix(copy): map x-amz-copy-source on this path-mapped deployment (multistore main / 0.7.1) fix(s3): adapt CopyObject to multistore's copy-source path mapping Jul 27, 2026
The integration test added alongside the mapping fix claimed to pin it but
could not: federation is attempted against the destination before the copy
source is resolved, so a copy fails identically whether or not
`with_copy_source` is wired up. Verified by deleting the wiring and pushing —
CI stayed green.

Move the check to a layer that can fail. The header-to-mapped-source logic is
extracted from `fetch` into `object_path::mapped_copy_source` (that module is
deliberately wasm-free so it can be unit-tested natively, the lib being cdylib
with `test = false`), and covered there: the account/product fold, leading-slash
normalization, nested keys, `versionId` and percent-encoding preservation, and
the two None cases. These fail without the mapping.

The integration test keeps its narrower real value — a copy is parsed and
dispatched as a copy and fails closed at federation — and its docstring now
states what it does not prove.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alukach
alukach marked this pull request as ready for review July 27, 2026 22:25
0.7.1 is released, so drop the temporary `[patch.crates-io]` block that pointed
the multistore crates at git `main` and pin the released versions instead.
`Cargo.lock` now resolves them from crates.io with checksums rather than a
moving branch ref.

This is what unblocks the copy-source mapping in this branch:
`RequestInfo::with_copy_source` is new in 0.7.1 and absent from 0.7.0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…suite

A copy is authorized in two halves — the destination as a write, the source as
a synthetic GetObject. Nothing asserted the source half, so a caller holding
write on some product could in principle have used CopyObject to read a
product they aren't entitled to. multistore covers this upstream (#128's
`copy_source_read_enforces_prefix_via_get_object`); this pins it through this
repo's subject-scoped registry wiring.

It goes in the deployed-environment suite rather than the CI integration tier
because CI could not fail it: federation is attempted against the destination
before the source is resolved, so on CI's throwaway signing key every copy dies
at federation first — and `AccessDenied` is itself a federation error code, so
the assertion would pass without proving anything. The staging suite has a
deployment where the destination write genuinely succeeds, leaving the source
half as the only thing that can deny the copy.

Dormant like the rest of that suite: it needs FEDERATION_WRITE_PRODUCT (a
product the caller may write) plus a caller identity the *deployed* worker
accepts — whose audience is not CI's `source-data-proxy-ci`, so the token is
minted only when FEDERATION_TEST_AUDIENCE names the deployment's AUTH_AUDIENCE.
Unset, the test skips and the rest of the suite is unaffected. staging.yml also
gains boto3, which the SigV4 client needs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mint step added with the copy-source authz test could never have produced a
usable caller. That job talks to the *deployed* worker, whose AUTH_ISSUER is Ory
(`https://auth.staging.source.coop`, see wrangler.toml [env.staging.vars]) — a
GitHub Actions OIDC token fails issuer verification before its audience is ever
examined, so no FEDERATION_TEST_AUDIENCE value could have made it authenticate.
The variable was a knob on a door that doesn't open; it is removed rather than
documented, along with the `id-token: write` permission it needed.

The token now comes from a secret, FEDERATION_CALLER_TOKEN, which must be issued
by the deployment's own issuer with an `aud` among its AUTH_AUDIENCE client_ids
and a subject holding write on FEDERATION_WRITE_PRODUCT. The test's gate is
unchanged, so it still skips cleanly until that is provisioned.

Not added to the wrangler files: this is a property of the CI caller, while
AUTH_AUDIENCE there is what the worker accepts. Duplicating it would be a second
source of truth able to drift from the value it must match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous commit replaced the unusable GitHub-OIDC mint with
`secrets.FEDERATION_CALLER_TOKEN`, which is no better: an Ory ID token is
short-lived, so a token parked in a repo secret goes stale within the hour.
Setting it would buy a brief window of green followed by failures that read as
an authorization regression rather than an expired credential — worse than the
test plainly skipping.

Supplying a caller needs a step that mints a token per run from a durable
credential (a service-account client_id/secret, or a refresh token) against the
deployment's own issuer. That is a provisioning decision this PR has no basis to
make, so no token is wired and the copy-source authz test skips, with the
requirement recorded where someone acting on it will look.

The unambiguous parts of the earlier wiring stay: boto3, which the SigV4 client
needs, and the FEDERATION_WRITE_PRODUCT passthrough.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restores the per-run GitHub OIDC mint for the copy-source authz test, ahead of
registering GitHub as a valid IdP with Source so products can accept writes from
GitHub Actions.

This is the right shape for the credential: minted per run and short-lived, with
nothing durable stored. The earlier attempt at a repo secret was worse — an ID
token parked there goes stale within the hour, and the resulting failures read as
an authorization regression rather than an expired credential.

Still dormant, now waiting on the IdP registration rather than on a design
decision. Two things must land before it can run: the deployment's AUTH_ISSUER
must accept GitHub's issuer, which is a code change and not only config —
src/config.rs reads AUTH_ISSUER as a single String, unlike the comma-separated
AUTH_AUDIENCE — and FEDERATION_TEST_AUDIENCE must name the audience Source
expects. Until both, no token is minted and the test skips.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@alukach
alukach merged commit 40acbcb into main Jul 28, 2026
14 checks passed
@alukach
alukach deleted the test/multistore-main branch July 28, 2026 01:18
alukach pushed a commit that referenced this pull request Jul 28, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.3.1](v2.3.0...v2.3.1)
(2026-07-28)


### Bug Fixes

* **s3:** adapt CopyObject to multistore's copy-source path mapping
([#198](#198))
([40acbcb](40acbcb))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: source-release-bot[bot] <265100246+source-release-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant