fix(deps): bump multistore to 0.7.2 to stop federated reads hanging

[alukach]

What this does

Bumps multistore 0.7.1 → 0.7.2, which fixes production requests being cancelled with The Workers runtime canceled this request because it detected that your Worker's code had hung and would never generate a response.

That is the whole diff. The fix itself is developmentseed/multistore#133; this PR started as a local reimplementation of it and was reduced to a dependency bump once that merged and released.

The bug

multistore-oidc-provider's CredentialCache single-flighted concurrent misses on a role ARN behind a futures::lock::Mutex. On Cloudflare Workers, awaiting that lock is fatal rather than slow: per Cloudflare's docs, the runtime cancels a request once "all the code associated with the request has executed and no events are left in the event loop", and a future parked on an in-memory waker has no pending I/O of its own. It is killed immediately, not when the lock holder finishes.

The fast path took the same lock, so requests whose credentials were already cached died too. Nearly all of data.source.coop's open data sits behind one connection — one role ARN, therefore one lock — so a single isolate renewing credentials took out every concurrent read on it.

A five-minute wrangler tail --status error on production caught 167 cancellations, all 167 on that one connection (tge-labs/aef 163, protomaps/openstreetmap 3, ftw/global-data 1). The timing split as the mechanism predicts: 149 at 5–8ms wall / 1–2ms CPU (warm requests, killed instantly with nothing pending) and 19 at 30–66ms / 21–26ms CPU (cold isolates — the lock holders' siblings).

Why it looked random

The failures need a credential fetch in flight, so they cluster at isolate cold start and at renewal, and vanish once an isolate is warm. That is the "product won't load, works on reload" shape.

Verification

Two live Cloudflare deployments, same staging API, same product, same federated connection (aws-opendata-eu-central-1) — so the dependency version is the only variable. Control is data.staging.source.coop, running main on multistore 0.7.1. Four rounds of 80 concurrent GETs of /alukach/euro-test/README.md each, cache-busted:

round	main (0.7.1)	this PR (0.7.2)
1 (cold)	59× 200, 21× 500	80× 200
2	80× 200	80× 200
3	79× 200, 1× 500	80× 200
4	80× 200	80× 200

wrangler tail over that window: staging logged 22 hung and would never generate a response cancellations, all on this product, at the same 5–8ms / 1–2ms signature. The preview logged zero events with exceptions across all 320 requests.

Also verified locally against the stub-API harness (50 concurrent federated reads: 49/50 cancelled on main, 0/50 on this branch), and 73 native tests, cargo clippy --target wasm32-unknown-unknown -D warnings, and the integration suite all pass.

What changed upstream

For anyone reading this later, 0.7.2 replaces the async lock with a cache that never blocks a caller:

• Fresh credentials are served from a plain map under a sync lock that is never held across an .await.
• Renewals are single-flighted without anyone waiting: inside the refresh lead the credential is still valid, so one caller claims the exchange and the rest keep serving what they have. That works precisely because latecomers never need the claimant's result — the thing a lock cannot give you here.
• Two guards: a 5s floor so a credential that could expire in transit is never served (an expired one comes back from S3 as ExpiredToken and reaches the client as a misleading AccessDenied), and a 30s claim timeout so a claimant killed mid-exchange cannot pin the key as renewing until expiry.
• Cache entries are now keyed by role ARN and subject. Keying on the ARN alone was unsound when two connections share a role: the trust policy conditions on the assertion's sub, so a connection that policy would reject could be served another's cached credentials.

A cold key is deliberately not single-flighted — there is nothing valid to serve, so collapsing that burst would mean waiting. Worth knowing that a deploy or mass eviction cools every isolate at once, so those bursts coincide fleet-wide; the mitigation is an L2 tier (#175), not a larger in-memory cache, since that tier is exactly what a deploy discards.

Relationship to #175

#175 adds an L2 Cloudflare-Cache-API tier for the STS response. It does not fix this on its own: it wraps FetchHttpExchange::post_form, which ran inside the old lock's critical section, so concurrent arrivals were still cancelled. It shortens the held window a lot, but a cold colo — L2 miss, highest concurrency — is exactly when it doesn't help.

The two compose: 0.7.2 is L1, #175 is L2, and #175 keeps independent value by making the cold-key duplication mostly network-free. Its central constraint ("the only seam we control is post_form") is worth revisiting now that upstream owns the cache.

[claude]

Claude finished @alukach's task in 1m 21s —— View job

---

Review

• Gather context and read diff
• Review Cargo.toml / Cargo.lock changes
• Post findings

✅ No blocking issues — safe to merge.

The diff vs origin/main is exactly the dependency bump described in the PR body's "Re-verified against the upstream code" section — nothing else changed:

• Cargo.toml: multistore, multistore-oidc-provider, multistore-path-mapping, multistore-sts, multistore-cf-workers bumped 0.7.1 → 0.7.2 (plus a trailing blank line removed at EOF, cosmetic).
• Cargo.lock: matching version/checksum bumps for all five multistore* crates, consistent with each other (verified no stale 0.7.1 multistore entries remain — grep for 0\.7\.1 in Cargo.lock only matches unrelated packages). multistore-oidc-provider's futures dependency is dropped, consistent with the PR's premise that the blocking futures::lock::Mutex cache was removed upstream in multistore#133.

This means the local federation.rs fork (the actual fix logic) is not part of this diff — it was superseded once the fix landed upstream and released as multistore 0.7.2, per the PR's "Follow-up" section. The PR description's live A/B tests (staging vs. preview, 320 requests each) validate that the released 0.7.2 behaves identically to the local fork it replaces (0 hangs vs. 22-23 on main/0.7.1).

CI is green (CI and both Ensure conventional commits checks passed; Preview still in progress at time of review).

[github-actions]

🚀 Latest commit deployed to https://source-data-proxy-pr-197.source-coop.workers.dev

• Date: 2026-07-28T06:31:54Z
• Commit: 7333615