feat(s3-datastore, gcs-datastore): detect SSO/credential errors and prepend remediation hint

[stack72]

Summary

• Refs swamp-club#226. Detect AWS SDK and GCP ADC credential failures in the S3 and GCS datastore extension error wrappers, and prepend a swamp-flavoured "Datastore session expired" / "Datastore credentials rejected" summary line that names the cause and points at the right remediation command (aws sso login / gcloud auth application-default login). The SDK detail and cause chain remain underneath; error.name is unchanged so existing call-site checks keep working.
• Filed swamp-club#228 as out-of-scope follow-up to apply the same treatment to @swamp/aws-sm-vault.

Why

When AWS SSO sessions expire or GCP ADC refresh tokens are revoked mid-execution, the resulting S3OperationError [CredentialsProviderError] or GcsOperationError surfaces with the remediation hint (aws sso login) buried inside the SDK stack — users initially attribute the failure to lock contention or a backend issue. This change front-loads the swamp-flavoured summary so the cause and next action are visible first.

What

S3 (datastore/s3/extensions/datastores/_lib/s3_client.ts):

• New exported pure helpers: classifyAwsCredentialError, deriveAwsErrorCode, formatAwsCredentialHint, AwsCredentialErrorKind.
• wrapError calls them; cause-chain walk handles minified SDK builds where the real class name appears as _CredentialsProviderError on .cause (per the issue's stack trace).
• Detects CredentialsProviderError + ExpiredTokenException (session-expired) and InvalidAccessKeyId + SignatureDoesNotMatch + 403 AccessDenied (credentials-rejected). AWS_PROFILE is read for the hint; falls back to generic phrasing when unset.
• Existing 401/403 generic hint stays as fallback for non-credential auth failures (e.g. BucketRegionMismatch).

GCS (datastore/gcs/extensions/datastores/_lib/gcs_client.ts):

• New exported pure helpers: classifyGcpCredentialError, formatGcpCredentialHint, tokenRefreshError, GcpCredentialErrorKind.
• tokenFromUserCredentials and tokenFromServiceAccount now throw a typed GcsOperationError with name === "CredentialsProviderError" when the response body indicates invalid_grant — the GCP equivalent of expired SSO.
• send() prepends the credentials-rejected hint on 401/403; suppresses the old generic hint when the credential-specific hint fires.

Manifests: Both datastore/s3 and datastore/gcs bumped from 2026.05.04.1 to 2026.05.04.2 (CalVer same-day .2).

Test plan

• S3 unit tests for deriveAwsErrorCode — 7 cases covering e.Code precedence, e.name fallback, generic Error ignored, single + multiple leading underscore strip on cause, non-Error cause ignored, no-signal undefined.
• S3 unit tests for classifyAwsCredentialError — 8 cases covering each error code branch and status guards.
• S3 unit tests for formatAwsCredentialHint — 5 cases covering profile set/unset variants for both kinds + other returns undefined.
• S3 integration tests via withMockServer — 4 cases for server-side errors (InvalidAccessKeyId, AccessDenied, SignatureDoesNotMatch, BucketRegionMismatch regression) + 1 case for 500 regression.
• S3 client-side CredentialsProviderError integration test — sets AWS_PROFILE to a non-existent profile, unsets all access-key env vars, points config files at empty temp file, disables IMDS, calls headBucket(). Asserts the wrapped message starts with "Datastore session expired", includes the configured profile in the aws sso login --profile hint, and S3OperationError.name reflects the underlying SDK error name. This is the issue's primary failure path, exercised end-to-end against a real SDK chain failure.
• GCS unit tests for classifyGcpCredentialError / formatGcpCredentialHint — 8 cases.
• GCS unit tests for tokenRefreshError — 4 cases covering invalid_grant (session-expired framing + name = CredentialsProviderError), non-invalid_grant 4xx (generic name), 5xx (no credential framing), and a composition test that proves the resulting error flows through classifyGcpCredentialError to session-expired.
• GCS integration tests via Deno.serve({ port: 0 }) — 401, 500 regression, 404/412 narrow-type regression guards. Existing 403 and gcs_cache_sync_test.ts 403 tests updated to assert the new hint.
• assertDatastoreExportConformance continues to pass for both extensions.
• deno check / lint / fmt --check / test / install --frozen clean for both extensions.

Out of scope

• @swamp/aws-sm-vault has the same problem but no wrapError pattern to extend — introducing one is a meaningful separate lift. Filed as swamp-club#228.
• The swamp-core summarizeSyncError reference path lives outside this repo.

🤖 Generated with Claude Code

[github-actions]

Code Review

Blocking Issues

None.

Suggestions

1. Double docblock on tokenRefreshError (gcs_client.ts lines ~278–291): Two consecutive JSDoc blocks appear before the function — likely a merge artifact. TypeScript associates only the last one with the function; the first is dead documentation. The first block should be removed.

2. classifyGcpCredentialError's causeName parameter is always undefined in production send() calls: send() always calls classifyGcpCredentialError(undefined, response.status), so the causeName === "CredentialsProviderError" branch in that function is never reached from send(). The session-expired path for GCS flows through tokenRefreshError() directly (which embeds the hint before throwing). The function is correctly exercised in the composition unit test (err.name is passed in), but a brief note in the JSDoc that causeName is intended for callers re-classifying a caught error — not for send() itself — would clarify the design intent.

3. Stale section comment in gcs_client_test.ts (line ~467): The block comment preceding the wrapper integration tests says "writing a temporary authorized_user credentials file, pointing the OAuth token endpoint at our mock server, and intercepting the refresh request" but the actual tests call tokenRefreshError directly as pure unit tests. The description matches the test design, not what was implemented.

---

The implementation is well-structured: pure helper functions for testability, proper cause-chain walking for minified SDK builds, regression guards on non-credential 403s, env vars restored in finally blocks throughout, and sanitizeResources: false following the existing file-level comment pattern. Test coverage is thorough across both providers.

[github-actions]

Adversarial Review

Critical / High

None found.

Medium

1. GCS error type change in token-refresh functions (gcs_client.ts:310–319, gcs_client.ts:336–340).
   tokenFromServiceAccount and tokenFromUserCredentials previously threw plain Error; they now throw GcsOperationError via tokenRefreshError(). This is a behavioral change: any catch block that dispatches on instanceof GcsOperationError and inspects httpStatusCode will now match these token-endpoint errors when it didn't before.

   Breaking example: A caller does catch (err) { if (err instanceof GcsOperationError && err.httpStatusCode === 400) { /* "GCS said bad request" */ } }. Before this PR, a 400 from the OAuth2 token endpoint (e.g. invalid_grant) would be a plain Error and skip this block. Now it enters the block with a status code whose semantic origin is the token endpoint, not GCS.

   Verified safe in existing code: isRetryableError in gcs_cache_sync.ts:182–195 handles the new type correctly — 400 → not retried (correct), 5xx → retried (arguably an improvement over the old behavior where 5xx token-endpoint failures were non-retryable plain Errors). The name field ("CredentialsProviderError" / "TokenRefreshError") disambiguates if needed. No action required unless there are call sites outside this repo that match instanceof GcsOperationError.

Low

1. Duplicate JSDoc on tokenRefreshError (gcs_client.ts:278–293). Two consecutive /** */ blocks precede the function. The first (lines 278–284) is a stale draft; only the second (lines 285–293) is attached to the function by the parser. Harmless, but messy — delete the first block.

2. Unquoted AWS_PROFILE in remediation hint (s3_client.ts:192). formatAwsCredentialHint interpolates the profile name bare: `aws sso login --profile ${awsProfile}`. If AWS_PROFILE contains spaces (uncommon but not prohibited by AWS config format), the copy-paste command would be broken. Consider wrapping: `aws sso login --profile '${awsProfile}'`.

3. tokenFromMetadataServer still throws plain Error (gcs_client.ts:376). The other two token functions now use tokenRefreshError() → GcsOperationError, but the metadata-server path still throws new Error(...). This means metadata-server token failures won't get the swamp-flavoured hint. The remediation for metadata-server failures is different (check instance SA, not run gcloud auth), so this may be intentional — but it's worth a comment to make that explicit.

Verdict

PASS. The new helpers are pure, well-tested, and correctly integrated into both providers' error paths. Classification logic handles all documented error codes with appropriate regression guards. The medium finding is a real behavioral change but is handled correctly by the existing retry classifier. Solid work.