Skip to content

Support for resolving S3-backed payloads in SNS non-raw messages (Fixes #175)#177

Open
danielferraz-git wants to merge 1 commit intoawslabs:masterfrom
danielferraz-git:feature/support-for-sns-non-raw-messages
Open

Support for resolving S3-backed payloads in SNS non-raw messages (Fixes #175)#177
danielferraz-git wants to merge 1 commit intoawslabs:masterfrom
danielferraz-git:feature/support-for-sns-non-raw-messages

Conversation

@danielferraz-git
Copy link

Description

When SNS is configured to deliver messages to SQS without RawMessageDelivery enabled, it wraps the message body in a JSON envelope. This currently prevents the SQS Extended Client from resolving large payloads, as it expects the SQS message body itself to be the S3 pointer.

This PR introduces an opt-in feature to support SNS-wrapped messages. It allows the client to identify S3 pointers within the SNS "Message" field, resolve the actual payload from S3, and update the JSON envelope in-place. This ensures that all other SNS metadata (e.g., Subject, Type, MessageId, Timestamp) is preserved for the consumer.

Key Changes

  • New Configuration: Added payloadSupportFromSnsEnabled to both ExtendedClientConfiguration and ExtendedAsyncClientConfiguration. This feature is disabled by default to maintain backward compatibility.
  • Robust JSON Resolution: Utilized Jackson (ObjectMapper) to perform structural updates to the SNS JSON body. This ensures proper character escaping and structural integrity when replacing the S3 pointer with the resolved content.
  • Preservation of Metadata: Unlike simple unwrapping, this implementation preserves the entire SNS envelope, updating only the "Message" field while keeping other attributes like Subject intact.
  • Sync & Async Support: Enhanced AmazonSQSExtendedClient and AmazonSQSExtendedAsyncClient to handle recursive payload resolution within JSON-formatted message bodies.
  • Testing: Added a comprehensive test suite (AmazonSQSExtendedClientSnsTest and AmazonSQSExtendedClientUtilTest) covering:
    • Successful resolution of large payloads inside SNS envelopes.
    • Preservation of non-message fields in the envelope.
    • Standard SQS message compatibility when the SNS flag is enabled.
    • Graceful handling of malformed or non-SNS JSON bodies.

Related Issue
Fixes #175

…sages

  This change introduces the ability for sync and async clients to resolve
  large message payloads stored in S3 when the SQS message body is wrapped
  in an SNS JSON envelope.

  Key changes:
   - Added payloadSupportFromSnsEnabled configuration option to opt-in
    to SNS envelope processing.
   - Implemented robust JSON manipulation using Jackson to resolve S3
    pointers within the SNS "Message" field while preserving other
    envelope attributes (e.g., Subject, Type, MessageId).
   - Enhanced both AmazonSQSExtendedClient and
    AmazonSQSExtendedAsyncClient to handle recursive payload resolution
    within JSON-formatted message bodies.
   - Added comprehensive unit and integration tests covering SNS envelope
    preservation, pointer resolution, and standard SQS compatibility.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SNS to SQS messages only work with raw message delivery.

1 participant