feat: add LocalStack-based local testing environment - #724
Draft
redhat-chai-bot wants to merge 13 commits into
Draft
feat: add LocalStack-based local testing environment#724redhat-chai-bot wants to merge 13 commits into
redhat-chai-bot wants to merge 13 commits into
Conversation
Add a LocalStack development environment that emulates the multi-account AWS infrastructure locally, allowing developers to test the deployment pipeline without real AWS accounts or clusters. New files: - docker-compose.localstack.yaml: LocalStack service configuration - scripts/localstack/init-aws.sh: Bootstrap mock AWS resources - scripts/localstack/localstack-env.sh: Lifecycle management CLI - config/localstack/: Environment config (defaults + us-east-1) - docs/localstack-testing.md: Setup, usage, and limitations guide - deploy/localstack/: Rendered pipeline inputs and ArgoCD manifests The init script creates mock IAM roles (OrganizationAccountAccessRole), SSM parameters, Route53 zones, S3 state buckets, VPC infrastructure, DynamoDB tables, KMS keys, and CodePipeline/CodeBuild projects matching the patterns used by the real ephemeral provisioning flow. Makefile targets: localstack-up, localstack-provision, localstack-teardown, localstack-shell, localstack-status, localstack-reset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Switch to localstack/localstack-pro image for full EKS emulation, Lambda container image support, and IAM policy enforcement. Add LOCALSTACK_AUTH_TOKEN pass-through and set ENFORCE_IAM=1 by default so IAM policies are tested against real permission checks. - docker-compose: use Pro image, add auth token and ENFORCE_IAM=1 - init-aws.sh: create IAM users with access keys and scoped policies - localstack-env.sh: preflight check for LOCALSTACK_AUTH_TOKEN - defaults.yaml: reflect Pro capabilities in config header - docs: document Pro requirements, auth token setup, IAM enforcement Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adopt Podman as the container engine for the LocalStack development environment, following the patterns established in rosa-log-router. docker-compose.localstack.yaml: - Mount podman socket via DOCKER_SOCK env var with fallback - Add :Z SELinux labels on all volume mounts - Run as user "0" with privileged: true for socket access - Use named network with explicit bridge driver - Switch LAMBDA_EXECUTOR to docker-reuse for faster cold starts localstack-env.sh: - Add podman socket activation (systemctl --user enable --now) - Simplify compose detection to docker compose only - Replace manual health-check loop with timeout-based approach (120s) - Remove emoji characters for cleaner terminal output Makefile: - Add podman socket activation in localstack-up target docs/localstack-testing.md: - Update prerequisites to recommend Podman with docker compose - Add Podman socket setup section - Add Podman socket troubleshooting section Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove DEFAULT_REGION (deprecated since LocalStack 0.12.7) and LAMBDA_EXECUTOR (deprecated since LocalStack 2.0.0) from compose file - Add wait_for_service helper and service readiness checks before each AWS service section in init-aws.sh to prevent race conditions - Add retry wrapper around SSM put-parameter calls with || true to handle transient failures that caused exit code 255 - Change default Docker socket path from /run/user/1000/podman/podman.sock to /var/run/docker.sock (works for Docker and rootful Podman); document DOCKER_SOCK override for Podman rootless users Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Root cause: create_iam_user() stores access key credentials in SSM
(put-parameter), but wait_for_service "ssm" only ran later in the
standalone SSM Parameters section. When the init script ran as a
ready.d hook during container startup, the SSM service wasn't ready
yet — awslocal returned exit code 255 (AWS CLI catchall), and
set -euo pipefail aborted the script.
Fix:
- Add wait_for_service "ssm" before the IAM Users section (first SSM
usage in the script, inside create_iam_user)
- Add || true to all EC2 VPC/subnet/security-group variable assignments
that were also missing error handling (same class of bug)
- Use ${VAR:-<failed>} in echo output for visibility when commands fail
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The init-aws.sh script was crashing with exit code 255 during the SSM Parameters section, but `set -euo pipefail` caused silent failures with no indication of which command failed or why. Changes: - Add `set -x` (bash debug trace) to print every command before execution - Add ERR trap that prints exact source file, line number, and exit code - Replace `set -e` with explicit per-command error handling via run_aws() and run_aws_capture() wrappers that capture stderr and log the full command line, exit code, and error output on failure - Add log() helper with timestamps for every operation - Add `set -E` to propagate ERR trap into functions and subshells - Remove all `2>/dev/null` from awslocal calls so errors are visible - Add `|| true` safety on arithmetic expressions to prevent edge cases - Quote all $AWSLOCAL expansions for correctness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…uce debug noise - Remove all --no-cli-pager flags from init-aws.sh (unsupported by AWS CLI v1 bundled in LocalStack container). Set AWS_PAGER="" at the top of the script instead, which works on both v1 and v2. - Trim SERVICES list in docker-compose.localstack.yaml to only the 18 services actually exercised by init-aws.sh, cutting startup time. - Fix false "FAILED [exit 0]" reporting in run_aws/run_aws_capture: capture exit code via `cmd || rc=$?` instead of relying on `$?` after an `if` block (which always yields 0 in bash). - Remove per-attempt retry logging and wait_for_service progress messages to reduce noise. Error and milestone messages retained. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Expand SERVICES in docker-compose.localstack.yaml to all 29 AWS services the project uses (excluding cloudtrail and codestar-connections) - Add EKS cluster creation (rc-cluster, mc-cluster) to init-aws.sh using LocalStack Pro k3s emulation, with ACTIVE polling and timeout - Add assume-role subcommand to localstack-env.sh + Makefile target that maps account names (central/rc/mc/customer) to IDs and drops into a subshell with temporary STS credentials - Add eks-kubeconfig subcommand + Makefile target to update kubeconfig for LocalStack EKS clusters via awslocal - Add trigger-pipeline subcommand + Makefile target supporting both CodeBuild (source upload to S3) and CodePipeline, with optional REPO= and COMMIT= parameters for custom source - Update docs/localstack-testing.md with documentation for all new tools Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ORCE_IAM Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…for root bootstrap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
LocalStack's ENFORCE_IAM does not evaluate inline or managed policies attached to the root user. After 4 attempts (AdministratorAccess, PassRole inline, wildcard inline), cross-account operations still fail with AccessDeniedException during bootstrap. Disable ENFORCE_IAM by default (opt-in via LOCALSTACK_ENFORCE_IAM=1). The 4 scoped IAM users are still created for realistic IAM testing after init completes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a LocalStack-based local development environment that emulates the multi-account AWS setup used by the ROSA HyperFleet deployment pipeline. This allows developers to test configuration rendering, pipeline bootstrapping, and environment provisioning workflows locally without needing real AWS accounts or deploying actual clusters.
Motivation
The existing
make ephemeral-provisionworkflow provisions real AWS infrastructure across 5 accounts via CodePipeline, which is slow (~1 hour) and requires AWS credentials. A LocalStack-based alternative enables rapid iteration on infrastructure changes, config rendering, and pipeline logic without incurring AWS costs or waiting for real provisioning.What's Included
Docker Compose (
docker-compose.localstack.yaml)Init Script (
scripts/localstack/init-aws.sh)000000000001) — pipeline infrastructure, SSM parameters, terraform state000000000002) — regional cluster account000000000003) — management cluster account000000000004) — customer workload accountOrganizationAccountAccessRolepattern for cross-account accessrender.pyexpects (/infra/localstack/<region>/account_id)Lifecycle Manager (
scripts/localstack/localstack-env.sh)scripts/dev/ephemeral-env.shup,provision,teardown,status,reset,shellConfig Preset (
config/localstack/)localstack.rosa.local)Makefile Targets
localstack-up— Start LocalStack serviceslocalstack-provision— Bootstrap the local AWS environmentlocalstack-teardown— Stop and clean uplocalstack-shell— Interactive AWS CLI shell against LocalStacklocalstack-status— Health checklocalstack-reset— Full destroy + recreateDocumentation (
docs/localstack-testing.md)Validation
shellcheckpasses on both shell scriptsrender.pysuccessfully rendersconfig/localstack/alongside existing environmentsprettierformatting applied to documentationcheck-docsannotation check passes.gitignoreupdated for LocalStack state directoriesKnown Limitations
@psav requested in Slack thread