Skip to content

feat: enforce conditional-write preconditions on PutObject (#120) - #122

Merged
alukach merged 6 commits into
mainfrom
worktree-put-conditional-writes
Jul 22, 2026
Merged

feat: enforce conditional-write preconditions on PutObject (#120)#122
alukach merged 6 commits into
mainfrom
worktree-put-conditional-writes

Conversation

@alukach

@alukach alukach commented Jul 21, 2026

Copy link
Copy Markdown
Member

Closes #120.

What I'm changing

Native Zarr/Icechunk writers use conditional writes (If-Match / If-None-Match) as the compare-and-swap that protects refs from concurrent commits. Today a write with a deliberately wrong ETag succeeds where S3 returns 412 Precondition Failed — the precondition is silently dropped and concurrent commits can clobber each other.

The precondition was dropped on all three write paths that produce an object, each with its own forward whitelist:

  1. PutObject, presigned path (build_forward) — plain bodies. if-match/if-none-match were absent. These are standard HTTP headers (not x-amz-*), so S3 honors them on a host-only presigned URL unsigned — the same way the GET/HEAD read paths already do.
  2. PutObject, aws-chunked streaming path (build_streaming_forward) — what AWS SDKs and the CLI send by default (STREAMING-UNSIGNED-PAYLOAD-TRAILER). This path re-signs the seed. The precondition was missing from AWS_CHUNKED_FORWARD_HEADERS, so an SDK-based writer's If-Match was still silently dropped — the exact reported case. Fixing only the presigned path would have missed the most common real-world client.
  3. CompleteMultipartUpload (execute_multipart) — the request that materializes a multipart object. Its whitelist forwarded only entity + checksum headers, so a large multipart write silently ignored the precondition too (flagged in review). This raw path signs every header, so the fix is on the safe signed path.

The x-amz-copy-source-if-* copy preconditions remain out of scope — server-side copy is still rejected (#119), so there is nothing to hang them on yet.

How I did it

  • crates/core/src/proxy.rs
    • Presigned path: added "if-match" / "if-none-match" to the PutObject signed-forward set, mirroring GET/HEAD.
    • Streaming path: added the same two to AWS_CHUNKED_FORWARD_HEADERS (sign_s3_request signs the whole map, so S3 enforces them).
    • Multipart path: added the same two to the execute_multipart whitelist. No-op on Create/UploadPart/Abort (clients only send them on completion).
    • Every path returns the backend's 412 via the Forward/raw response unchanged.
  • crates/core/tests/conditional_writes.rs — Rust integration tests through ProxyGateway::handle_request against a mock backend emulating S3's compare-and-swap on both forward (presigned + streaming PutObject) and send_raw (multipart completion). Regression guard for all three whitelists (each verified failing when its fix is reverted).
  • tests/integration/test_integration.py (TestConditionalWrites) — real end-to-end tests: boto3 → proxy → MinIO. Covers PutObject (failing precondition/412 + object unchanged, create-if-absent, successful If-Match swap) and CompleteMultipartUpload (412 on If-None-Match: * and wrong If-Match).
  • crates/core/src/proxy.rs (unit test)put_forward_preserves_conditional_headers asserts the presigned Forward carries both headers.
  • docs/reference/operations.md — documented the honored preconditions and 412 compare-and-swap semantics across all three write paths.

Test plan

  • cargo test -p multistore — 128 lib + 10 Rust integration tests pass
  • cargo fmt
  • cargo clippy --fix — clean
  • cargo check
  • cargo check -p multistore-cf-workers --target wasm32-unknown-unknown
  • scripts/integration-test.sh -k ConditionalWrites — 6 passed end-to-end (boto3 → wrangler dev → MinIO); confirms the proxy signs the precondition and MinIO enforces it (412 PreconditionFailed) on both PutObject and CompleteMultipartUpload

🤖 Generated with Claude Code

Add `if-match`/`if-none-match` to the PutObject signed-forward whitelist so
the backend evaluates the precondition and returns 412 Precondition Failed
on a stale/wrong ETag instead of silently clobbering. These are standard
HTTP headers (not `x-amz-*`), so S3 applies them on a host-only presigned
URL exactly like the GET/HEAD read paths already do.

Native Zarr/Icechunk writers use conditional writes as the compare-and-swap
that protects refs from concurrent commits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 21, 2026

Copy link
Copy Markdown

Claude finished @alukach's task in 2m 1s —— View job


✅ No blocking issues — safe to merge.

Reviewed the diff (crates/core/src/proxy.rs, crates/core/tests/conditional_writes.rs, tests/integration/test_integration.py, docs/reference/operations.md) against origin/main.

  • Traced PendingRequest.original_headers (crates/core/src/route_handler.rs:123) to confirm it's populated fresh per inbound HTTP request (crates/core/src/proxy.rs:891-897), not carried over from CreateMultipartUpload — the doc comment's claim that if-match/if-none-match are None on Create/UploadPart/Abort and only populated on the CompleteMultipartUpload request itself holds up.
  • Confirmed AWS_CHUNKED_FORWARD_HEADERS (crates/core/src/proxy.rs:104) is shared between the PutObject and UploadPart aws-chunked streaming callers (proxy.rs:946, proxy.rs:1023) — adding if-match/if-none-match there is a no-op for UploadPart since S3 doesn't evaluate those on part uploads and boto3 never sets them there. This is explicitly locked in by test_upload_part_tolerates_leaked_precondition in test_integration.py, which hand-injects a signed If-Match on UploadPart and asserts the upload still completes against real MinIO — good regression coverage for the shared-list assumption rather than a blind claim.
  • The three whitelist changes (presigned build_forward, AWS_CHUNKED_FORWARD_HEADERS, and the execute_multipart whitelist) each mirror an already-established pattern in the codebase (GET/HEAD forwarding, checksum-header forwarding), so this isn't introducing a new trust boundary — just extending existing unsigned/signed forwarding to two more standard HTTP headers.
  • Rust integration tests (conditional_writes.rs) and Python end-to-end tests (TestConditionalWrites) both cover all three write paths (presigned PUT, streaming PUT, CompleteMultipartUpload) with wrong-ETag/412, If-None-Match: */exists, and matching-ETag/success cases — solid coverage, nothing missing that stood out.
  • Docs update in operations.md accurately reflects the new behavior and correctly notes x-amz-copy-source-if-* remains out of scope pending Support S3 CopyObject (server-side copy via x-amz-copy-source) #119.

No correctness, security, or best-practice issues found. Nothing to flag under Simplify (ponytail) either — the diff is a minimal, well-scoped whitelist extension with no new abstractions or dependencies.


💰 Estimated review cost: $0.84 · 2m01s · 25 turns

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

📖 Docs preview deployed to https://multistore-docs-pr-122.development-seed.workers.dev

  • Date: 2026-07-22T04:50:33Z
  • Commit: ebfdee0

@github-actions

github-actions Bot commented Jul 21, 2026

Copy link
Copy Markdown

🚀 Latest commit deployed to https://multistore-proxy-pr-122.development-seed.workers.dev

  • Date: 2026-07-22T04:50:33Z
  • Commit: ebfdee0

alukach and others added 2 commits July 21, 2026 16:49
The initial fix only forwarded `if-match`/`if-none-match` on the presigned
PutObject path. But AWS SDKs and the CLI send PutObject bodies as `aws-chunked`
streaming uploads, which take the header-signed `build_streaming_forward` path
instead — so an SDK-based Icechunk writer's precondition was still silently
dropped, exactly the reported case.

Add the two preconditions to `AWS_CHUNKED_FORWARD_HEADERS`. Unlike the presigned
path these are signed (`sign_s3_request` signs the whole header map), so S3
evaluates them and returns 412 on a mismatch. They don't apply to `UploadPart`,
but clients never send them there, so sharing the list is a no-op for parts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drive the full `handle_request` pipeline against a backend that emulates S3's
compare-and-swap, asserting the client sees 412 on a failed precondition and
200 otherwise — on both the presigned and the aws-chunked streaming path.
Regression guard for both forward whitelists.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Exercise If-Match / If-None-Match on PutObject end-to-end through boto3 →
proxy → MinIO. boto3 streams PutObject bodies as aws-chunked, so these hit
the header-signed streaming path (the one SDKs use) against a real backend
that enforces the precondition and returns 412 — the compare-and-swap
Icechunk relies on. Covers the failing precondition (412, object unchanged),
the create-if-absent case, and the successful If-Match swap.

Verified end-to-end locally (scripts/integration-test.sh -k ConditionalWrites):
4 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alukach
alukach marked this pull request as ready for review July 22, 2026 04:13
Third and final write path: a large object written via multipart is
materialized by CompleteMultipartUpload, whose forward whitelist
(`execute_multipart`) dropped `if-match`/`if-none-match` — so a multipart
write silently ignored the precondition, the same failure mode as the
PutObject bug. Add both to the whitelist; the raw multipart path signs every
header (`sign_s3_request`), so they are enforced (S3 and MinIO return 412 on
a mismatch). They are a no-op on Create/UploadPart/Abort (clients only send
them on completion).

Tests: extend the mock CasBackend to enforce the precondition on `send_raw`
and add CompleteMultipartUpload cases (Rust integration + regression guard),
plus boto3 → proxy → MinIO end-to-end multipart cases. Verified end-to-end
(scripts/integration-test.sh -k ConditionalWrites): 6 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two review follow-ups on the multipart conditional-write tests:

- Hygiene: a failed CompleteMultipartUpload leaves the upload open, so
  repeated CI runs accumulated stray incomplete uploads. `_complete_multipart`
  now aborts the upload before propagating the error.

- Lock in the "clients never send If-Match on UploadPart" assumption that
  makes sharing AWS_CHUNKED_FORWARD_HEADERS with the part path safe: botocore
  has no IfMatch member on UploadPart, but a hand-injected signed precondition
  must still ride through the proxy's sign-and-forward without breaking the
  upload (the backend ignores it). New test injects one and asserts the
  multipart upload completes intact.

Verified end-to-end (scripts/integration-test.sh -k ConditionalWrites): 7 passed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@alukach
alukach merged commit f5ff9a8 into main Jul 22, 2026
16 checks passed
@alukach
alukach deleted the worktree-put-conditional-writes branch July 22, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce conditional-write preconditions (If-Match / If-None-Match) on PutObject

1 participant