Improve error message when ECS credentials endpoint returns HTTP 429 (rate limited)

[fearofbug]

Describe the feature

When the ECS task metadata credentials endpoint returns HTTP 429 (rate limited), the Ruby SDK raises Aws::Errors::MissingCredentialsError. in my case the complete error was:

Error Class: Aws::Errors::MissingCredentialsError
Message: unable to sign request without credentials set

FULL STACK TRACE:

0: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:145:in `rescue in sign'
1: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:136:in `sign'
2: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:48:in `call'
3: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/transfer_encoding.rb:27:in `call'
4: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in `call'
5: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `block in call'
6: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/user_agent.rb:92:in `metric'
7: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:385:in `with_metric'
8: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in `call'
9: /usr/local/bundle/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/http_checksum.rb:20:in `call'

This is misleading because credentials are not missing—they are being rate-limited by the ECS agent. misleading my troubleshooting efforts

The Python SDK (boto3/botocore) provides a much clearer error message that includes the HTTP status code and the actual response from the endpoint.

Python SDK (for comparison):

botocore.exceptions.CredentialRetrievalError: Error when retrieving credentials from container-role: Error retrieving metadata: Received non 200 response 429 from container metadata: You have reached maximum request limit.

The ECS agent logs do not help either as there were no rate limiting errors in the agent logs. I had to run a tcp dump command below to troubleshoot the issue:

sudo tcpdump -i any -A port 80 and host 169.254.170.2 2>/dev/null | grep -E "HTTP/1\.[01] (200|429|503)"

Use Case

When running applications on Amazon ECS with IAM task roles, the ECS agent enforces rate limiting on the credentials endpoint (http://169.254.170.2/v2/credentials/) via the ECS_TASK_METADATA_RPS_LIMIT configuration (default: 40 steady, 60 burst requests per second).

When this rate limit is exceeded, the ECS agent returns HTTP 429 with the body: You have reached maximum request limit.

The current Ruby SDK error message leads developers to debug credential provider chain configuration burning hours investigating the wrong problem When the actual issue is simply rate limiting, which can be resolved by Caching/reusing SDK clients instead of creating new ones per request or adjusting ECS_TASK_METADATA_RPS_LIMIT on the ECS agent.

References
[1] https://github.com/aws/amazon-ecs-agent/blob/master/README.md#:~:text=ECS_TASK_METADATA_RPS_LIMIT

Proposed Solution

Include HTTP status code and response body in the error message

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

3.241.4

Environment details (OS name and version, etc.)

amazonlinux 2023

[richardwang1124]

Hey, thanks for opening an issue and I apologize for the misleading error. I'll look into improving the error message.

[richardwang1124]

Just for my understanding, were you suppressing stderr outputs? When reproducing locally I see

Error retrieving ECS Credentials: Aws::ECSCredentials::Non200Response
Error Class: Aws::Errors::MissingCredentialsError
Message: unable to sign request without credentials set

FULL STACK TRACE:
0: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:145:in 'Aws::Plugins::Sign::SignatureV4#sign'
1: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/sign.rb:48:in 'Aws::Plugins::Sign::Handler#call'
2: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/transfer_encoding.rb:27:in 'Aws::Plugins::TransferEncoding::Handler#call'
3: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:12:in 'Aws::Plugins::HelpfulSocketErrors::Handler#call'
4: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in 'block in Aws::Plugins::RetryErrors::LegacyHandler#call'
5: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/user_agent.rb:92:in 'Aws::Plugins::UserAgent.metric'
6: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:385:in 'Aws::Plugins::RetryErrors::LegacyHandler#with_metric'
7: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/retry_errors.rb:365:in 'Aws::Plugins::RetryErrors::LegacyHandler#call'
8: /Users/wangrch/.gem/ruby/3.4.1/gems/aws-sdk-core-3.241.4/lib/aws-sdk-core/plugins/http_checksum.rb:20:in 'Aws::Plugins::HttpChecksum::Handler#call'

which still isn't the best message but it does give me more information about the underlying issue.

[fearofbug]

Hello @richardwang1124 and thank you for having a look. We were not suppressing the stderr outputs and we did get the stack which led us into looking at the ECS_TASK_METADATA_RPS_LIMIT but after spending sometime thinking the issue was credential not being available initially.

[richardwang1124]

Sorry, my previous comment was a little vague; I meant to ask if you were able to see the Error retrieving ECS Credentials: Aws::ECSCredentials::Non200Response warning since I didn't see it in your original post.

I've opened a PR to help clarify the warning message and it now looks like

Error retrieving ECS Credentials: HTTP 429: You have reached maximum request limit.
Error: Aws::Errors::MissingCredentialsError - unable to sign request without credentials set

I don't think I'll be able to change the actual error message surfaced (unable to sign request without credentials set) but the team is exploring ways to create a log of credentials providers that weren't able to be resolved and why and surfacing that with the error. In the meantime let me know if the updated warning message helps!

[fearofbug]

Hello @richardwang1124 Yes, the message is clear and provides the guidance needed regarding reaching the limit. thanks for your help!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.