objstore: support GCS S3-compatible signing (#69721)

[ti-chi-bot]

This is an automated cherry-pick of #69721

What problem does this PR solve?

Issue Number: close #69718

Problem Summary:

IMPORT INTO can fail when reading source files from GCS through the S3-compatible endpoint after the S3 implementation moved to AWS SDK for Go v2.

With gcs s3 compatible endpoint, SDK v2 can sign accept-encoding into the SigV4 canonical request. GCS S3 interoperability can reject these requests with SignatureDoesNotMatch, and region discovery can fail on HeadBucket before the import starts.

What changed and how does it work?

This PR adds a GCS S3-compatible mode for the S3 object store:

• Detect GCS S3-compatible usage by provider=gcs and endpoint=https://storage.googleapis.com, or virtual-hosted GCS endpoints.
• Do AWS bucket-region discovery only for aws s3 endpoints.
• Use a SigV4 signer wrapper that excludes accept-encoding from the canonical request while still sending the header on the HTTP request.

The signer keeps the AWS SDK v2 client path. Live GCS S3-compatible testing showed that amz-sdk-invocation-id and amz-sdk-request can remain signed; only accept-encoding needs to be excluded.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No need to test

  • I checked and no code files have been changed.

Manual live test against the real GCS S3-compatible endpoint:

• Setup next-gen cluster locally, and import data from gcs.
• IMPORT INTO succeeded, which validates both source discovery (ListObjectsV2) and the following object reads (GetObject).
• A retryable request failure was manually injected in the request path; the SDK retried and the import still completed.
• The imported data was verified by row-count checks and ADMIN CHECK TABLE.
• The test did not hit the previous GCS failures: SignatureDoesNotMatch, Forbidden, or failed to walk dir.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Fix IMPORT INTO failures when reading GCS through the S3-compatible endpoint with AWS SDK for Go v2.

Summary by CodeRabbit

• New Features

  • Added support for Google Cloud Storage S3-compatible endpoints.
  • Improved request signing compatibility by excluding Accept-Encoding from signed headers.
  • Added endpoint detection and handling for GCS-compatible configurations.

• Tests

  • Added coverage for GCS endpoint detection and compatible request signing across supported request types.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign benjamin2037 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/objstore/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: daa0277a-0956-4e52-a8d3-2cee27697883

📥 Commits

Reviewing files that changed from the base of the PR and between 02d7646 and 0ba69d5.

📒 Files selected for processing (4)

• pkg/objstore/s3store/BUILD.bazel
• pkg/objstore/s3store/gcs_s3_signer.go
• pkg/objstore/s3store/gcs_s3_test.go
• pkg/objstore/s3store/store.go

---

📝 Walkthrough

Walkthrough

Adds GCS S3-compatible endpoint detection and configures a custom AWS SigV4 signer that excludes Accept-Encoding from canonical signing. AWS bucket-region discovery is skipped for these endpoints, with tests covering detection and signed HEAD/list requests.

Changes

GCS S3 compatibility

Layer / File(s)	Summary
Endpoint detection and storage wiring pkg/objstore/s3store/store.go	Detects GCS providers and endpoint hostnames, enables the compatible signer, and skips AWS bucket-region discovery.
Compatible SigV4 signer pkg/objstore/s3store/gcs_s3_signer.go, pkg/objstore/s3store/BUILD.bazel	Temporarily removes Accept-Encoding while AWS V4 signing runs, then restores the request header.
Compatibility request tests pkg/objstore/s3store/gcs_s3_test.go, pkg/objstore/s3store/BUILD.bazel	Tests endpoint detection and verifies signed HEAD and list requests exclude accept-encoding.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant S3Store
  participant GCSCompatibility
  participant V4Signer
  participant GCS S3 Endpoint
  S3Store->>GCSCompatibility: detect provider or endpoint
  GCSCompatibility-->>S3Store: enable compatible signer
  S3Store->>V4Signer: sign HEAD or list request
  V4Signer-->>S3Store: return signed request
  S3Store->>GCS S3 Endpoint: send request with restored Accept-Encoding

Loading

Possibly related PRs

• pingcap/tidb#69721: Adds the same GCS-compatible signer, endpoint handling, and request tests.

Poem

I’m a rabbit with a signed little note,
Through GCS clouds my requests now float.
Accept-Encoding hops aside,
While SigV4 guards the payload ride.
HEAD and list both reach the store—
Hare-brained headers trouble no more!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: GCS S3-compatible signing support.
Description check	✅ Passed	The description includes the required issue reference, problem summary, implementation details, tests, side effects, and release note.
Linked Issues check	✅ Passed	The changes address the reported GCS S3-compatible failures by detecting GCS endpoints, skipping AWS region discovery, and adjusting SigV4 signing.
Out of Scope Changes check	✅ Passed	The PR stays focused on GCS S3-compatible signing and related tests, with no obvious unrelated code changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

level=error msg="Running error: context loading failed: failed to load packages: failed to load packages: failed to load with go/packages: context deadline exceeded"
level=error msg="Timeout exceeded: try increasing it by passing --timeout option"

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

@ti-chi-bot: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-integration-realcluster-test-next-gen	0ba69d5	link	true	/test pull-integration-realcluster-test-next-gen

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.