Support for S3 AWS IAM auth

[mish-elle]

closes #8211

Background

Self-hosters running Hive on AWS with AWS S3 currently have no way to use IAM-based authentication for S3 connections, which forces them to manage and rotate static access keys S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY.

This PR adds opt-in AWS IAM authentication support for S3 connections. When IAM auth is enabled, services authenticate to S3 using the AWS SDK default credential chain (IRSA, EKS Pod Identity, instance profile, etc.) instead of static credentials.

This PR is part of the following issue. We will have separate PRs for each IAM support to help decrease the scope per PR.

• IAM authentication support for Redis, Kafka, S3 and PG #7780

Description

Since S3 connections are HTTP requests, there is no need for a token refresher. Instead, tokens are generated per request via a credential provider.

Additionally, since the console already has a custom AWS S3 client (aws4fetch) which already supports a session token, we'll build on top of that and use a credential provider interface to avoid deeply modifying the underlying pre-existing S3 client.

We created a new createDefaultCredentialProvider() function in api/src/modules/cdn/providers/aws.ts which creates an AwsCredentialProvider that:

• When IAM auth is enabled, uses fromNodeProviderChain() from @aws-sdk/credential-providers
  (the SDK handles credential caching and automatic refresh before expiry)
• When IAM auth is disabled, wraps the static credentials in an async provider

The AwsClient class is updated to accept a credentialProvider instead of raw credential strings. On every sign() call, it resolves fresh credentials from the provider.

Since the cdn-worker package targets Cloudflare Workers and cannot import from the AWS SDK. To maintain compatibility, the AwsCredentialProvider interface is duplicated in cdn-worker/src/aws.ts. So this supports S3 IAM Auth on the CDN when the server service has CDN_API=1.

Deploying the CDN to AWS Lambda does not support passwordless connections to S3 at this time.

This PR also fixes a bug where the audit log S3 client was incorrectly using the primary S3 bucket's endpoint and bucket name instead of its own. The AuditLogS3Config now correctly uses s3AuditLogs.endpoint and s3AuditLogs.bucketName. Additionally, the audit log CSV upload now includes aws: { signQuery: true } to ensure the S3 PUT request is signed correctly.

New environment variables

Variable	Required	Description
S3_AWS_IAM_AUTH_ENABLED	No	Set to 1 to enable IAM authentication for the primary S3 bucket.
S3_MIRROR_AWS_IAM_AUTH_ENABLED	No	Set to 1 to enable IAM authentication for the S3 mirror bucket.
S3_AUDIT_LOG_AWS_IAM_AUTH_ENABLED	No	Set to 1 to enable IAM authentication for the S3 audit log bucket.

Environment Variable Validation

When IAM auth is not enabled for a given S3 bucket, the environment validation enforces that static credentials (S3*_ACCESS_KEY_ID / S3*_SECRET_ACCESS_KEY) are provided. Previously these were always required; now they are only required when IAM is disabled.

When IAM auth is enabled:

• S3_ENDPOINT should be set to the AWS S3 regional endpoint (e.g.
  https://s3.us-east-1.amazonaws.com)
• S3_BUCKET_NAME must be set to the AWS S3 bucket name
• The pod/instance must have AWS credentials available (e.g. IRSA, EKS Pod Identity, instance profile)

Pnpm-lock file generation

Like MSK, the CI pipelines will fail because of the pnpm-lock file. We're downgrading dependencies and not building a few packages, due to constraints in our environment. Is it possible for a code maintainer to help us generate the pnpm-lock file?

Checklist

• Input validation
• Output encoding
• Authentication management
• Session management
• Access control
• Cryptographic practices
• Error handling and logging
• Data protection
• Communication security
• System configuration
• Database security
• File management
• Memory management
• Testing

[gemini-code-assist]

Code Review

This pull request introduces opt-in AWS IAM authentication for S3 connections across services, allowing dynamic credential resolution (such as EKS Pod Identity) via the AWS SDK credential chain alongside static fallbacks. It adds new environment variables, updates environment validation schemas, refactors AwsClient to use an AwsCredentialProvider, and introduces comprehensive unit tests. A TypeScript compilation issue was identified in aws.ts where the sdkProvider type definition lacks the expiration property, which is subsequently accessed when resolving credentials.

[dotansimha]

We did not have it before - any reason to add it now? Is it something only AWS supports? or is it supported also with other vendors providing S3 services?

[mish-elle]

Hey! Without this addition we were unable to export audit logs to S3. We noticed that this aws sign happens for all S3 calls in the rest of the repo, but was missing in audit logs.

https://github.com/graphql-hive/console/blob/main/packages/services/api/src/modules/schema/providers/artifact-storage-writer.ts#L115-L118
https://github.com/graphql-hive/console/blob/main/packages/services/api/src/modules/app-deployments/providers/app-deployments.ts#L480-L484

[dotansimha]

Thanks @mish-elle , please see my comments above. Also some rebasing is needed.

[dotansimha]

Hi @mish-elle , just wanted to check if you had the chance to evaluate my review feedback ^

thanks

[mish-elle]

Hi @mish-elle , just wanted to check if you had the chance to evaluate my review feedback ^

thanks

Hey @dotansimha, sorry about the delay! Thanks for the feedback! I believe I got everything.

[n1ru4l]

hey @mish-elle, I pushed the branch to our origin and updated the lockfile for CI tests to run. Some unit tests are currently failing. Can you please have a look at these and fix the implementation/tests?

[mish-elle]

Updated! Thanks, @n1ru4l