The Analytics account offers two deployment options. Following V1.xx customer feedback, we introduced an EC2 & SSM agent receiver option for processing large datasets that exceed Lambda's 15-minute execution limit.
Below are comprehensive details for both options:
| # | Item | Description |
|---|---|---|
| 1 | Repository Link | https://github.com/aws-samples/sample-aws-multi-account-observability |
| 2 | Download | Download sample-aws-multi-account-observability the folder as a zip from AWS Samples repository |
| 3 | Extract | Extract all and place it in a folder of your choice |
| 4 | Verify | Check that you have all the files |
| 5 | QuickSuite Account | MUST DO - Please make sure to have an Amazon Quick Suite account created. • Login to AWS Console > Search for Amazon Quick Suite • Create Account & Log in to Amazon Quick Suite and ensure its working. |
Using a Serverless approach with only Lambda functions handling the transactions.
|
|||
| Step | Component | Action | Description |
|---|---|---|---|
| 1 | EventBridge (Daily Schedule) | Triggers | Daily trigger at 01:00 AM activates sender function |
| 2 | Sender Function (Lambda) | Executes | Collects data from AWS services (10GB/15min capacity) |
| 3 | KMS Key | Encrypts | Provides encryption for data security |
| 4 | S3 Data Bucket | Receives | Stores encrypted JSON data from sender accounts |
| 5 | Receiver Function (Lambda) | Processes | Triggered by S3 PUT events, processes JSON data (10GB/15min) |
| 6 | Aurora Primary | Stores | Primary database instance (Serverless v2 - Postgres) |
| 7 | QuickSight / QuickSuite | Visualizes | Connects via VPC endpoint for dashboard access |
|
|||
|
|||
|
|||
| Step | Component | Action | Description |
|---|---|---|---|
| 1 | S3 Event (ObjectCreated) | Triggers | New JSON file uploaded to S3 data/ folder |
| 2 | Lambda Function (a360-receiver) | Receives | S3 event notification triggers receiver function |
| 3 | Lambda Function (a360-receiver) | Fetches | Downloads receiver.py script from S3 bucket |
| 4 | Lambda Function (a360-receiver) | Processes | Reads and processes JSON data from uploaded file |
| 5 | Lambda Function (a360-receiver) | Updates/Inserts | Writes processed data to Aurora PostgreSQL database |
| 6 | Lambda Function (a360-receiver) | Moves | Transfers processed JSON file to loaded/ folder in S3 |
| 7 | S3 Bucket (Analytics Account) | Stores | Maintains processed files in loaded/ACCOUNT/REGION/YYYY-MM-DD_DAILY.json structure |
Using a EC2 & SSM agent in the Receiver with a Lambda Function to invoke based on events.
|
|||
| Step | Component | Action | Description |
|---|---|---|---|
| 1 | EventBridge (Daily Schedule) | Triggers | Daily trigger at 01:00 AM activates sender function |
| 2 | Sender Function (Lambda) | Executes | Collects data from AWS services (10GB/15min capacity) |
| 3 | KMS Key | Encrypts | Provides encryption for data security |
| 4 | S3 Data Bucket | Receives | Stores encrypted JSON data from sender accounts |
| 5 | Receiver Function (Lambda) | Processes | Triggered by S3 PUT events, processes JSON data (10GB/15min) |
| 6 | SSM Endpoint & EC2 Receiver | Connects & Processes | Downloads receiver.py script from S3 bucket, Reads and processes JSON data from uploaded file |
| 7 | RDS Data Endpoint | Connects | Enables secure database connectivity |
| 8 | Aurora Primary | Stores | Primary database instance (Serverless v2 - Postgres) |
| 9 | QuickSight | Visualizes | Connects via VPC endpoint for dashboard access |
|
|||
|
|||
|
|||
| Step | Component | Action | Description |
|---|---|---|---|
| 1 | S3 Event (ObjectCreated) | Triggers | New JSON file uploaded to S3 data/ folder |
| 2 | Lambda Function (a360-ec2-trigger) | Receives | S3 event notification triggers EC2-trigger function |
| 3 | Lambda Function (a360-ec2-trigger) | Invokes | Calls SSM agent to execute receiver function |
| 4 | SSM (a360-ssm) | Triggers | SSM agent triggers EC2 instance to run receiver |
| 5 | EC2 Instance (a360-ec2) | Fetches | Downloads receiver.py script from S3 bucket |
| 6 | EC2 Instance (a360-ec2) | Processes | Reads and processes JSON data from uploaded file |
| 7 | EC2 Instance (a360-ec2) | Updates/Inserts | Writes processed data to Aurora PostgreSQL database |
| 8 | EC2 Instance (a360-ec2) | Moves | Transfers processed JSON file to loaded/ folder in S3 |
| 9 | S3 Bucket (Analytics Account) | Stores | Maintains processed files in loaded/ACCOUNT/REGION/YYYY-MM-DD_DAILY.json structure |