diff --git a/.chai-bot/rosa_hyperfleet_ci_daily_health_report.md b/.chai-bot/rosa_hyperfleet_ci_daily_health_report.md index db4b22b98..fedded58f 100644 --- a/.chai-bot/rosa_hyperfleet_ci_daily_health_report.md +++ b/.chai-bot/rosa_hyperfleet_ci_daily_health_report.md @@ -132,7 +132,7 @@ For each job whose **latest run failed**, produce a **separate threaded reply** - Fetch scope based on Prow analysis: RC-only, MC + RC, or both if unclear - If S3 logs are inaccessible, report the specific error — classification ceiling becomes Unclear -3. **Git commit correlation** (Step 5c in ci-troubleshooter) — **MANDATORY.** Identify the last passing run, find all commits between last-good and current-bad, and examine commits touching the failing component. Also check `rosa-hyperfleet-api` for API/CLM failures. +3. **Git commit correlation** (Step 5c in ci-troubleshooter) — **MANDATORY.** Identify the last passing run, find all commits between last-good and current-bad, and examine commits touching the failing component. Also check `rosa-hyperfleet-api` for API/hyperfleet-operator failures. **S3 log handling:** Always extract tar.gz locally for full analysis. Clean up downloaded files immediately after analysis is complete — never leave S3 logs on disk between runs. See Step 5b in `.claude/agents/ci-troubleshooter.md` for the full procedure. @@ -247,10 +247,10 @@ integration: ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ❌ 🔧 *Genuine* — E2E test `TestClusterCreation` timed out waiting for hosted cluster to become ready. Evidence: Prow ✅ | S3 Logs ✅ | Git History ✅ | Trend ✅ -Root cause: MC maestro-agent pod in CrashLoopBackOff due to MQTT connection failure — incorrect broker endpoint in ArgoCD values. -S3 Log Evidence: `maestro-agent/pods/agent-xyz/agent/logs/current.log` — 47x `CONNACK refused: not authorized`; pod status: CrashLoopBackOff -Suspect Commits: `a1b2c3d` — `feat(argocd): update maestro broker endpoint` — touches `argocd/config/management-cluster/maestro/` -Consecutive failures (2 days): same root cause as Jun 29 — maestro CONNACK failure with identical error signature. +Root cause: MC kube-applier pod in CrashLoopBackOff due to DynamoDB connectivity failure — incorrect table name in ArgoCD values. +S3 Log Evidence: `kube-applier/pods/kube-applier-xyz/kube-applier/logs/current.log` — 47x `ResourceNotFoundException: Requested resource not found`; pod status: CrashLoopBackOff +Suspect Commits: `a1b2c3d` — `feat(argocd): update kube-applier DynamoDB config` — touches `argocd/config/management-cluster/kube-applier/` +Consecutive failures (2 days): same root cause as Jun 29 — kube-applier DynamoDB failure with identical error signature. Most recent failure: (Jun 30) Failing since: Jun 29 (2 consecutive days) diff --git a/.claude/agents/architect.md b/.claude/agents/architect.md index 7f47dd1eb..1363bfcaf 100644 --- a/.claude/agents/architect.md +++ b/.claude/agents/architect.md @@ -80,8 +80,8 @@ Issues: 3. Database access violates separation of concerns Recommendations: -1. Subscribe to cluster-events topic for event-driven processing -2. Implement status reporting via REST API (see clm-gcp-environment-validation example) +1. Use controller-runtime watches for event-driven processing +2. Implement status reporting via CR status updates (see hyperfleet-operator controller pattern) 3. Use GET /api/v1/clusters/{id} to fetch data instead of direct DB access Consider documenting this controller pattern in a new design decision. diff --git a/.claude/agents/ci-troubleshooter.md b/.claude/agents/ci-troubleshooter.md index e031a6cb8..74b8d24bb 100644 --- a/.claude/agents/ci-troubleshooter.md +++ b/.claude/agents/ci-troubleshooter.md @@ -121,16 +121,16 @@ Single `e2e-tests` step — fetch and analyze `/e2e-tests/build-l Single step matching job name — fetch `//build-log.txt`. -## Step 5b: Pull Cluster Logs from S3 (MANDATORY for cluster-backed jobs) +## Step 5b: Pull Environment Dumps from S3 (MANDATORY for cluster-backed jobs) -When e2e tests fail, the CI job collects pod logs from the RC and MC clusters and uploads them to S3. These logs are **not** included in the public Prow artifacts (they may contain secrets), but the S3 URIs are printed in the e2e build log. +When e2e tests fail, the CI job dumps environment state from the RC and MC clusters and uploads it to S3. RC dumps include both Kubernetes logs and a PostgreSQL database snapshot; MC dumps contain Kubernetes logs only. These dumps are **not** included in the public Prow artifacts (they may contain secrets), but the S3 URIs are printed in the e2e build log. **Applies to:** `on-demand-e2e`, `nightly-ephemeral`, `nightly-integration` — jobs that provision clusters and produce S3 log archives. **Does not apply to** validation jobs (`terraform-validate`, `helm-lint`, `check-rendered-files`, `check-docs`) which have no cluster logs — those jobs are classified using Prow build logs and git history only. **S3 log analysis is mandatory for all cluster-backed job failure classifications.** You MUST download, extract, and analyze S3 logs before classifying any failure from these jobs. A classification of Genuine or Flake is not valid without S3 log evidence. Use the Prow build logs from Step 5 to determine which clusters to fetch logs for: -- **RC-only failure** (e.g., provision failure, API error, ArgoCD sync issue on RC, maestro-server error): fetch **only RC logs** from S3. -- **MC failure or RC↔MC interaction** (e.g., maestro-agent errors, HyperShift issues, hosted cluster failures, connectivity between RC and MC): fetch **both RC and MC logs** from S3 — MC failures often have an RC-side root cause. +- **RC-only failure** (e.g., provision failure, API error, ArgoCD sync issue on RC, hyperfleet-operator error): fetch **only RC logs** from S3. +- **MC failure or RC↔MC interaction** (e.g., kube-applier errors, HyperShift issues, hosted cluster failures, connectivity between RC and MC): fetch **both RC and MC logs** from S3 — MC failures often have an RC-side root cause. - **Unclear scope**: fetch **both RC and MC logs**. If S3 logs are inaccessible for any reason (credentials, expired logs, network issues), you MUST still attempt the access and report the specific error. When S3 logs cannot be obtained, the classification ceiling is **⚠️ Unclear** — you cannot claim Genuine or Flake without S3 evidence. @@ -161,7 +161,7 @@ There will be one URI per cluster (RC + each MC). The bucket names follow the pa - RC: `bastion-log-collection---an` - MC: `bastion-log-collection---an` -### Fetching the logs +### Fetching the dumps **Always extract tar.gz archives locally for full analysis.** Download to a temp directory, extract, perform broad grep-based analysis across all namespaces, and clean up after: @@ -173,9 +173,9 @@ trap 'rm -rf "$LOGDIR"' EXIT # Use separate subdirectories for RC and MC to avoid archive name collisions mkdir -p "$LOGDIR/rc" "$LOGDIR/mc" -aws s3 cp s3://bastion-log-collection---an/collect-logs-.tar.gz \ +aws s3 cp s3://bastion-log-collection---an/dump-env-.tar.gz \ "$LOGDIR/rc/" --profile && \ - tar xzf "$LOGDIR/rc"/collect-logs-*.tar.gz -C "$LOGDIR/rc" + tar xzf "$LOGDIR/rc"/dump-env-*.tar.gz -C "$LOGDIR/rc" # Perform broad analysis: grep across ALL namespaces, not just suspected ones grep -rli "error\|fail\|crash\|panic\|fatal\|timeout\|refused\|denied" "$LOGDIR/rc"/inspect-logs/namespaces/ 2>/dev/null @@ -189,12 +189,12 @@ Fetch logs based on the failure scope determined from Prow artifacts. Use the ap LOGDIR=$(mktemp -d /tmp/ci-logs-XXXXXX) trap 'rm -rf "$LOGDIR"' EXIT mkdir -p "$LOGDIR/rc" "$LOGDIR/mc" -aws s3 cp s3://bastion-log-collection-720644165472-us-east-1-an/collect-logs-.tar.gz \ +aws s3 cp s3://bastion-log-collection-720644165472-us-east-1-an/dump-env-.tar.gz \ "$LOGDIR/rc/" --profile chai-rc-ci && \ - tar xzf "$LOGDIR/rc"/collect-logs-*.tar.gz -C "$LOGDIR/rc" -aws s3 cp s3://bastion-log-collection-129678139271-us-east-1-an/collect-logs-.tar.gz \ + tar xzf "$LOGDIR/rc"/dump-env-*.tar.gz -C "$LOGDIR/rc" +aws s3 cp s3://bastion-log-collection-129678139271-us-east-1-an/dump-env-.tar.gz \ "$LOGDIR/mc/" --profile chai-mc-ci && \ - tar xzf "$LOGDIR/mc"/collect-logs-*.tar.gz -C "$LOGDIR/mc" + tar xzf "$LOGDIR/mc"/dump-env-*.tar.gz -C "$LOGDIR/mc" # Analyze $LOGDIR/rc/inspect-logs/ and $LOGDIR/mc/inspect-logs/ ``` @@ -222,38 +222,51 @@ Classification ceiling is Unclear — cannot claim Genuine or Flake without S3 e Do **not** stop the investigation — proceed with whatever information is available from the Prow artifacts and git history. However, **without successfully analyzed S3 log evidence, the maximum classification confidence is ⚠️ Unclear.** You cannot classify as Genuine or Flake without having analyzed S3 logs. -### Analyzing the logs +### Analyzing the dumps -Once extracted, the logs are organized as: +Once extracted, the dump is organized as: -``` +```text inspect-logs/ namespaces// - .yaml # Resource definitions + .yaml # Resource definitions (pods, services, etc.) pods///logs/ current.log # Current container log previous.log # Previous container log (if restarted) + cluster-scoped-resources/ # Cluster-scoped CRs (nodes, etc.) + //.yaml + / # CRD instances collected by oc adm inspect + .yaml # e.g., hostedclusters, nodepools, applications + db-state/ # RC only — hyperfleet-db state dump + resource-summary.txt # Tabular listing of all kubernetes_resources rows + resources//.json # Individual resource objects (spec, status, metadata) ``` Key namespaces and what to look for: -| Cluster | Namespace | What to check | -| ------- | ---------------- | ----------------------------------------------------------- | -| RC | `maestro-server` | Server MQTT connectivity, resource bundle creation | -| RC | `platform-api` | API errors, registration failures | -| RC | `argocd` | Sync failures, application health | -| MC | `maestro-agent` | Agent MQTT connectivity (CONNACK errors), work agent status | -| MC | `argocd` | Sync failures on MC applications | -| MC | `hypershift` | HyperShift operator errors | +| Cluster | Namespace | What to check | +| ------- | -------------- | ------------------------------------------------------------ | +| RC | `hyperfleet` | Operator reconciliation, Manifest CR and hyperfleet-db state | +| RC | `platform-api` | API errors, registration failures | +| RC | `argocd` | Sync failures, application health | +| MC | `kube-applier` | DynamoDB Streams connectivity, resource apply status | +| MC | `argocd` | Sync failures on MC applications | +| MC | `hypershift` | HyperShift operator errors | + +**Other dump components** (not Kubernetes namespaces): + +| Cluster | Directory | What to check | +| ------- | ----------- | ------------------------------------------------------------------------------------- | +| RC | `db-state/` | Hyperfleet DB contents — resource summary and individual JSON objects (RC dumps only) | -For maestro connectivity issues specifically, check: +For resource distribution issues specifically, check: ```bash -# Agent connection errors -grep -i "connack\|connect\|error\|fail" /tmp/-mc01-logs/inspect-logs/namespaces/maestro-agent/pods/*/agent/agent/logs/current.log +# kube-applier errors on MC +grep -i "error\|fail\|dynamo" /tmp/-mc01-logs/inspect-logs/namespaces/kube-applier/pods/*/kube-applier/logs/current.log -# Server-side issues -grep -i "error\|fail\|connect" /tmp/-regional-logs/inspect-logs/namespaces/maestro-server/pods/*/service/service/logs/current.log +# Operator errors on RC +grep -i "error\|fail\|reconcil" /tmp/-regional-logs/inspect-logs/namespaces/hyperfleet/pods/*/manager/logs/current.log ``` ### S3 log retention @@ -271,11 +284,12 @@ The key question is: **did anything change between the last passing and current # Provision failure → terraform/, scripts/buildspec/, ci/ephemeral-provider/ # E2E test failure → ci/e2e-tests.sh, ci/e2e-platform-api-test.sh # ArgoCD sync failure → argocd/ -# Maestro failure → argocd/config/*/maestro* +# Hyperfleet operator failure → argocd/config/regional-cluster/hyperfleet* +# kube-applier failure → argocd/config/management-cluster/kube-applier* # Platform API failure → (check rosa-hyperfleet-api repo) ``` -**Cross-repo:** the git commands above cover `rosa-hyperfleet` only. For API/CLM failures, also check recent `rosa-hyperfleet-api` commits via `gh api`. Only check `rosa-hyperfleet-cli` if e2e tests invoke CLI commands. +**Cross-repo:** the git commands above cover `rosa-hyperfleet` only. For API/hyperfleet-operator failures, also check recent `rosa-hyperfleet-api` commits via `gh api`. Only check `rosa-hyperfleet-cli` if e2e tests invoke CLI commands. If a commit strongly correlates with the failure, this is strong evidence for a Genuine classification even on first occurrence. @@ -356,7 +370,7 @@ When today's failure is part of a **consecutive failure streak** (2+ days in a r 1. **Collect failure artifacts from each consecutive failing run** — use the job history to identify the streak, then fetch Prow artifacts and S3 logs (selectively, per Step 5b) for at least the current and previous failing runs. 2. **Compare error signatures** — are the failures the same root cause, or did the root cause shift? - - **Same root cause across streak**: reinforce the diagnosis with the additional evidence. Note the streak length (e.g., "failing for 3 consecutive days with the same maestro-agent CONNACK error"). + - **Same root cause across streak**: reinforce the diagnosis with the additional evidence. Note the streak length (e.g., "failing for 3 consecutive days with the same kube-applier DynamoDB connectivity error"). - **Root cause shifted**: clearly state that the root cause changed. Identify when it changed and what the new root cause is. This affects PR management (see Step 9). 3. **Aggregate the signal** — a 3-day streak of the same error is much stronger signal than a single failure. Reflect this confidence in the classification (almost certainly Genuine, not Flake). @@ -388,16 +402,16 @@ Present findings in this format: **S3 Log Evidence:** -- `inspect-logs/namespaces/maestro-agent/pods/agent-xyz/agent/logs/current.log`: 47 occurrences of `CONNACK refused: not authorized` +- `inspect-logs/namespaces/kube-applier/pods/kube-applier-xyz/kube-applier/logs/current.log`: 47 occurrences of `DynamoDB stream error` - `inspect-logs/namespaces/hypershift/pods/operator-abc/manager/logs/current.log`: `OOMKilled` at 03:42 UTC -- Pod health scan: 2 pods in CrashLoopBackOff (`maestro-agent`, `work-agent`) +- Pod health scan: 2 pods in CrashLoopBackOff (`kube-applier`, `hyperfleet-operator`) **Suspect Commits:** - `a1b2c3d` — `fix(terraform): update NAT gateway config` — touches `terraform/modules/eks-cluster/` (relevant: provision failure) -- `e4f5g6h` — `feat(argocd): add maestro-agent resource limits` — touches `argocd/config/management-cluster/maestro/` (relevant: maestro-agent OOMKilled) +- `e4f5g6h` — `feat(argocd): add kube-applier resource limits` — touches `argocd/config/management-cluster/kube-applier/` (relevant: kube-applier OOMKilled) ) and current run () touch the failing component's paths."> **Cross-Day Analysis** (if consecutive failures): @@ -426,9 +440,9 @@ Share the root cause and raise a fix PR immediately: 1. **Identify the target repo**: - `rosa-hyperfleet` — Terraform modules, ArgoCD configs, CI scripts, buildspecs - - `rosa-hyperfleet-api` — Platform API, CLM service code + - `rosa-hyperfleet-api` — Platform API, hyperfleet-operator service code - `rosa-hyperfleet-cli` — CLI tooling -2. **Create a fix branch** — branch from `main`: `chai-bot/fix--` (e.g., `chai-bot/fix-ephemeral-maestro-mqtt-config`). +2. **Create a fix branch** — branch from `main`: `chai-bot/fix--` (e.g., `chai-bot/fix-ephemeral-kube-applier-config`). 3. **Implement the fix** — make the minimal change needed to address the root cause. Follow the project's development guidelines (run `make pre-push` before committing). 4. **Raise the PR** — use `gh pr create` with: - Title: `fix(): ` diff --git a/.claude/skills/add-pre-merge/SKILL.md b/.claude/skills/add-pre-merge/SKILL.md index f0e81c6d8..8f4ad1176 100644 --- a/.claude/skills/add-pre-merge/SKILL.md +++ b/.claude/skills/add-pre-merge/SKILL.md @@ -10,7 +10,7 @@ You are helping the user onboard a new component repository for cross-component If not provided via `$ARGUMENTS`, ask the user for: -1. **Component name** — the directory name under `argocd/config/regional-cluster/` or `argocd/config/management-cluster/` in this repo (e.g., `platform-api`, `maestro-server`). Validate it exists. +1. **Component name** — the directory name under `argocd/config/regional-cluster/` or `argocd/config/management-cluster/` in this repo (e.g., `platform-api`, `hyperfleet`). Validate it exists. 2. **Org and repo** — the GitHub org/repo for the component (e.g., `openshift-online/rosa-hyperfleet-api`). This determines the CI config path in openshift/release. 3. **Branch** — the branch to configure (default: `main`). 4. **Dockerfile path** — path to the Dockerfile in the component repo (default: `Dockerfile`). diff --git a/.spec/002-spec-to-pr-agent/context-requirements.md b/.spec/002-spec-to-pr-agent/context-requirements.md index 26358a5af..4e746c195 100644 --- a/.spec/002-spec-to-pr-agent/context-requirements.md +++ b/.spec/002-spec-to-pr-agent/context-requirements.md @@ -7,9 +7,9 @@ An agentic workflow that takes a feature specification (from a JIRA ticket or de ## Codebase Research Findings - **Existing agents**: adversary, architect, ci-troubleshooter, code-reviewer, documentation-updater, scope-creep-craig, tech-spec-beck -- **Ephemeral env targets**: `ephemeral-{provision,teardown,resync,swap-branch,list,shell,bastion-rc,bastion-mc,port-forward-*,e2e,collect-logs}` +- **Ephemeral env targets**: `ephemeral-{provision,teardown,resync,swap-branch,list,shell,bastion-rc,bastion-mc,port-forward-*,e2e,dump-env}` - **E2E testing**: Tests live in `rosa-hyperfleet-api` repo, run via `ci/e2e-tests.sh`, use `make ephemeral-e2e ID=` -- **Component repos**: platform-api, maestro-agent, maestro-server, hyperfleet-adapter, hyperfleet-api, hyperfleet-sentinel +- **Component repos**: platform-api, hyperfleet-operator, hyperfleet-db, kube-applier - **CLI proxy**: Credential-isolating sidecar for `gh` CLI with deny list for destructive commands - **Config rendering**: `uv run scripts/render.py` for region configs @@ -69,7 +69,7 @@ An agentic workflow that takes a feature specification (from a JIRA ticket or de ### Ephemeral Environment Script (`scripts/dev/ephemeral-env.sh`) -- Commands: provision, teardown, resync, swap-branch, shell, bastion, port-forward, e2e, collect-logs, list +- Commands: provision, teardown, resync, swap-branch, shell, bastion, port-forward, e2e, dump-env, list - State tracking: `.ephemeral-envs` file with KEY=VALUE pairs (ID, REPO, BRANCH, STATE, REGION, API_URL, CI_BRANCH, CREATED) - Credentials: Fetched from Vault via OIDC, never persisted to disk - Container-based execution with AWS credentials and API URL injection diff --git a/.spec/002-spec-to-pr-agent/implementation-plan.md b/.spec/002-spec-to-pr-agent/implementation-plan.md index 2a0e95e4f..befcaa07c 100644 --- a/.spec/002-spec-to-pr-agent/implementation-plan.md +++ b/.spec/002-spec-to-pr-agent/implementation-plan.md @@ -293,7 +293,7 @@ Parse the user's request from: $ARGUMENTS - resync: `make ephemeral-resync ID=` - list: `make ephemeral-list` - e2e: `make ephemeral-e2e ID=` -- collect-logs: `make ephemeral-collect-logs ID=` +- dump-env: `make ephemeral-dump-env ID=` - shell: `make ephemeral-shell ID=` - swap-branch: `make ephemeral-swap-branch ID= BRANCH=` ... diff --git a/.spec/002-spec-to-pr-agent/persona-schema.yaml b/.spec/002-spec-to-pr-agent/persona-schema.yaml index e3db0d405..63c01f289 100644 --- a/.spec/002-spec-to-pr-agent/persona-schema.yaml +++ b/.spec/002-spec-to-pr-agent/persona-schema.yaml @@ -147,7 +147,7 @@ example_personas: - Implement feature code following existing patterns and conventions - Write and refine E2E tests - Inject new component versions into ArgoCD configurations - - Implement CLM adapters where required + - Implement hyperfleet-operator controllers where required - Self-validate (compile, lint, unit tests pass) before signaling ready approach: > Read the spec and implementation plan thoroughly. Study existing diff --git a/.spec/002-spec-to-pr-agent/requirements.md b/.spec/002-spec-to-pr-agent/requirements.md index aa0eddc1d..501d7ee57 100644 --- a/.spec/002-spec-to-pr-agent/requirements.md +++ b/.spec/002-spec-to-pr-agent/requirements.md @@ -32,7 +32,7 @@ Build a **Spec-to-PR Agent** — a Python-based orchestrator using the Claude Ag - The agent MUST be able to implement E2E tests for the feature being developed - The agent MUST be able to implement the feature itself, including: - Injecting new versions of components into ArgoCD configurations - - Implementing new CLM adapters where required + - Implementing new hyperfleet-operator controllers where required - Making changes across multiple component repositories (e.g., hypershift, platform-api, CLI) - The agent MUST refine E2E tests and implementation based on test feedback @@ -54,10 +54,10 @@ Build a **Spec-to-PR Agent** — a Python-based orchestrator using the Claude Ag - **swap-branch**: Switch an environment to a different branch/repo - **list**: Display all tracked environments with status - **e2e**: Run end-to-end tests against an environment - - **collect-logs**: Gather Kubernetes logs from clusters + - **dump-env**: Dump environment state (Kubernetes logs and DB state) from clusters - **shell**: Open an interactive shell with credentials - **bastion**: Connect to RC/MC cluster bastions - - **port-forward**: Tunnel Kubernetes services (Maestro, ArgoCD, Prometheus, Grafana) + - **port-forward**: Tunnel Kubernetes services (ArgoCD, Prometheus, Grafana) - The skill MUST support managing multiple concurrent environments - The skill MUST wrap the existing `scripts/dev/ephemeral-env.sh` operations diff --git a/CLAUDE.md b/CLAUDE.md index 33be4b14d..47e1c6dbf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,8 +17,8 @@ The **ROSA HyperFleet** is a strategic redesign of Red Hat OpenShift Service on 1. **Regional Cluster (RC)** - EKS-based cluster running core services: - Platform API (customer-facing with AWS IAM auth) - - CLM (Cluster Lifecycle Manager) - single source of truth - - Maestro - MQTT-based configuration distribution + - Hyperfleet Operator - postgres-based cluster lifecycle controller + - kube-applier - DynamoDB-backed resource distribution to MCs - ArgoCD - GitOps deployment - Tekton - infrastructure provisioning pipelines @@ -33,11 +33,11 @@ The **ROSA HyperFleet** is a strategic redesign of Red Hat OpenShift Service on - **Compute**: Amazon EKS (Regional + Management Clusters) - **Networking**: VPC, API Gateway (regional), VPC Link v2, ALBs -- **Storage**: Amazon RDS (CLM state), EBS volumes +- **Storage**: Amazon RDS (hyperfleet-db), Amazon ElastiCache Valkey (rate limiting), EBS volumes - **Identity**: AWS IAM for authentication and authorization - **Infrastructure**: Terraform modules with GitOps patterns - **CI/CD**: ArgoCD (apps), Tekton (infrastructure pipelines) -- **Messaging**: Maestro (MQTT-based resource distribution) +- **Resource Distribution**: kube-applier (DynamoDB-backed controller applying resources to MCs) - **Languages**: Go (primary backend), Shell scripting - **Container Orchestration**: Kubernetes via EKS @@ -71,8 +71,8 @@ When creating ROSAENG issues: - **GitOps First**: ArgoCD for cluster configuration management, infrastructure via Terraform - **Private-by-Default**: EKS clusters use fully private architecture with ECS bootstrap -- **Declarative State**: CLM maintains single source of truth for all cluster state -- **Event-Driven**: Maestro handles CLM ↔ MC communication for configuration distribution +- **Declarative State**: Hyperfleet Operator (backed by hyperfleet-db) maintains single source of truth for all cluster state +- **Event-Driven**: kube-applier reads desire documents from DynamoDB (written by hyperfleet-operator) and applies them to MCs via DynamoDB Streams - **Regional Isolation**: Each region operates independently with minimal cross-region dependencies - **Explicit Feature Flags**: Optional or environment-specific infrastructure (e.g., CloudTrail, PagerDuty, resources with per-account limits) should be gated behind `enable_*` configuration flags. Avoid patterns like checking against the environment's name to change behavior or functionality. - Feature flags should default to what keeps the best developer experience — focus on the lowest barrier to getting a new region started. We'd rather have verbose production configs than require developers to understand every flag just to get going. @@ -82,6 +82,7 @@ When creating ROSAENG issues: - **Bootstrap Strategy**: Use ECS Fargate for private EKS cluster bootstrap (see `docs/design/fully-private-eks-bootstrap.md`) - **No Public APIs**: All EKS clusters are fully private with VPC-only access - **ArgoCD Self-Management**: Clusters manage their own ArgoCD installations via GitOps +- **Rate Limiting**: Per-account rate limiting at the Platform API layer using GCRA algorithm with ElastiCache Valkey as shared counter store (see `docs/design/rate-limiting-architecture.md`) ### Repository Structure @@ -176,6 +177,31 @@ See [`docs/development-environment.md`](docs/development-environment.md) for ful Platform alerting and recording rules are defined as PrometheusRule CRs in the `alerting-rules` chart (`argocd/config/regional-cluster/alerting-rules/templates/`). Rules are evaluated by Thanos Ruler against Thanos Query. See [docs/adding-alerting-rules.md](docs/adding-alerting-rules.md) for a developer guide on adding new rules, including the error budget burn rate pattern used for SLA alerts. +### Rate Limiting + +The Platform API implements per-account rate limiting using the GCRA (Generic Cell Rate Algorithm) via `go-redis/redis_rate`. Rate limit counters are stored in ElastiCache Valkey 9.1, chosen over Redis OSS for 20% lower cost, open-source licensing (BSD 3-Clause), and performance improvements. + +**Architecture**: API Gateway (global throttle) → Platform API middleware (per-account GCRA) → ElastiCache Valkey (shared counters). Rate limiting runs after identity extraction but before authorization, so over-limit requests are rejected cheaply. The system fails open — if Valkey is unavailable, requests pass through. + +**Key files (rosa-hyperfleet)**: + +| File | Purpose | +| ---------------------------------------------------------- | --------------------------------------------------------------------------- | +| `terraform/modules/elasticache-valkey/main.tf` | ElastiCache Valkey replication group, KMS, security groups, parameter group | +| `terraform/modules/elasticache-valkey/variables.tf` | `cluster_id`, `vpc_id`, `node_type`, `engine_version` | +| `terraform/modules/elasticache-valkey/outputs.tf` | Valkey endpoint and port outputs | +| `scripts/bootstrap-argocd.sh` | Threads Valkey endpoint from Terraform outputs to ECS bootstrap env vars | +| `terraform/modules/ecs-bootstrap/main.tf` | Writes `redis_endpoint` annotation on the ArgoCD cluster secret | +| `config/templates/argocd-bootstrap/applicationset.yaml.j2` | Reads `redis_endpoint` annotation into Helm valuesObject | +| `argocd/config/regional-cluster/platform-api/values.yaml` | Rate limit config (routes, rates, burst), Valkey endpoint | +| `argocd/config/regional-cluster/platform-api/templates/` | Deployment, ratelimit-configmap, servicemonitor templates | +| `argocd/config/regional-cluster/alerting-rules/templates/` | PrometheusRule CRs for rate limit alerts | +| `docs/design/rate-limiting-architecture.md` | ADR: rate limiting design decisions | + +**Key files (rosa-hyperfleet-api)**: Rate limiting Go implementation lives in the API repo — `pkg/ratelimit/` (GCRA limiter, config loading), `pkg/middleware/ratelimit.go` (HTTP middleware), `cmd/rosa-regional-platform-api/main.go` (wiring). + +**Data flow for Valkey endpoint**: Terraform output → `bootstrap-argocd.sh` (combines host:port) → ECS task env var → cluster secret annotation → ApplicationSet valuesObject → Helm values → `REDIS_ENDPOINT` env var on platform-api pods. + ### Chai Bot Scheduled Tasks Scheduled CI/documentation tasks run via Chai Bot. Schedules are defined in `.chai-bot/`. diff --git a/Makefile b/Makefile index aeb0f8029..d769e6614 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: help terraform-fmt terraform-init terraform-validate terraform-upgrade terraform-output-management terraform-output-regional helm-lint check-rendered-files promtool-test ephemeral-provision ephemeral-teardown ephemeral-resync ephemeral-list ephemeral-shell ephemeral-bastion-rc ephemeral-bastion-mc ephemeral-port-forward-rc ephemeral-port-forward-mc ephemeral-port-forward-rc-all ephemeral-port-forward-mc-all ephemeral-sre-ui ephemeral-e2e ephemeral-collect-logs int-shell int-bastion-rc int-bastion-mc int-port-forward-rc int-port-forward-mc int-port-forward-rc-all int-port-forward-mc-all int-e2e int-collect-logs check-docs check-default-tags pre-push render +.PHONY: help terraform-fmt terraform-init terraform-validate terraform-upgrade terraform-output-management terraform-output-regional helm-lint check-rendered-files promtool-test ephemeral-provision ephemeral-teardown ephemeral-resync ephemeral-list ephemeral-shell ephemeral-bastion-rc ephemeral-bastion-mc ephemeral-port-forward-rc ephemeral-port-forward-mc ephemeral-port-forward-rc-all ephemeral-port-forward-mc-all ephemeral-sre-ui ephemeral-e2e ephemeral-dump-env int-shell int-bastion-rc int-bastion-mc int-port-forward-rc int-port-forward-mc int-port-forward-rc-all int-port-forward-mc-all int-e2e int-dump-env check-docs check-default-tags pre-push render # ============================================================================= # Local tool management @@ -230,8 +230,8 @@ ephemeral-sre-ui: ## Tunnel SRE UI tools (Grafana, ArgoCD, Prometheus, Thanos, L ephemeral-e2e: ## Run e2e tests against an ephemeral env @ID="$(ID)" E2E_REF="$(or $(E2E_REF),main)" E2E_REPO="$(E2E_REPO)" ./scripts/dev/ephemeral-env.sh e2e -ephemeral-collect-logs: ## Collect logs from an ephemeral env (CLUSTER=rc|mc) - @ID="$(ID)" ./scripts/dev/ephemeral-env.sh collect-logs $(CLUSTER) +ephemeral-dump-env: ## Dump EKS must-gather and DB state from an ephemeral env (CLUSTER=rc|mc) + @ID="$(ID)" ./scripts/dev/ephemeral-env.sh dump-env $(CLUSTER) # ============================================================================= # Integration Environment @@ -263,8 +263,8 @@ int-port-forward-mc-all: ## Port-forward all MC services in int env int-e2e: ## Run e2e tests against int env @E2E_REF="$(or $(E2E_REF),main)" E2E_REPO="$(E2E_REPO)" ./scripts/dev/int-env.sh e2e -int-collect-logs: ## Collect logs from int env (CLUSTER=rc|mc) - @./scripts/dev/int-env.sh collect-logs $(CLUSTER) +int-dump-env: ## Dump EKS must-gather and DB state from int env (CLUSTER=rc|mc) + @./scripts/dev/int-env.sh dump-env $(CLUSTER) render: ## Render config templates @uv run scripts/render.py diff --git a/argocd/config/regional-cluster/.gitkeep b/argocd/config/regional-cluster/.gitkeep index 16d443b63..76dcfc544 100644 --- a/argocd/config/regional-cluster/.gitkeep +++ b/argocd/config/regional-cluster/.gitkeep @@ -1,2 +1,2 @@ # This directory holds regional-cluster helm chart configurations -# Add helm charts here as needed (e.g., maestro/, platform-api/, etc.) +# Add helm charts here as needed (e.g., platform-api/, hyperfleet/, etc.) diff --git a/argocd/config/regional-cluster/alerting-rules/templates/ratelimit.yaml b/argocd/config/regional-cluster/alerting-rules/templates/ratelimit.yaml new file mode 100644 index 000000000..ad4ad0c9f --- /dev/null +++ b/argocd/config/regional-cluster/alerting-rules/templates/ratelimit.yaml @@ -0,0 +1,56 @@ +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: ratelimit + namespace: thanos + labels: + app.kubernetes.io/name: alerting-rules + app.kubernetes.io/managed-by: Helm + operator.thanos.io/prometheus-rule: "true" +spec: + groups: + - name: ratelimit + rules: + - alert: RateLimitFailOpenActive + expr: | + ( + sum(rate(ratelimit_requests_total{result="failure_mode_allowed"}[5m])) + / sum(rate(ratelimit_requests_total[5m])) + ) > 0.20 + and sum(rate(ratelimit_requests_total[5m])) > 0 + for: 5m + labels: + severity: warning + annotations: + summary: "Rate limit enforcement bypassed (>20% fail-open)" + description: "{{ "{{" }} $value | humanizePercentage {{ "}}" }} of rate-limited traffic passing without enforcement for >5m. Check Valkey connectivity." + + - alert: RateLimitHighDenialRatio + expr: | + ( + sum by(method, path) (rate(ratelimit_requests_total{result="over_limit"}[5m])) + / + sum by(method, path) (rate(ratelimit_requests_total[5m])) + ) > 0.5 + and sum by(method, path) (rate(ratelimit_requests_total[5m])) > 0 + for: 10m + labels: + severity: warning + annotations: + summary: "Over 50% of requests are being rate limited" + description: "More than 50% of rate-limited requests on method={{ "{{" }} $labels.method {{ "}}" }}, path={{ "{{" }} $labels.path {{ "}}" }} have been denied for over 10 minutes. This may indicate a misbehaving client or rate limits that need tuning." + + - alert: RateLimitPodHighDenialRatio + expr: | + ( + sum by(method, path, pod) (rate(ratelimit_requests_total{result="over_limit"}[5m])) + / + sum by(method, path, pod) (rate(ratelimit_requests_total[5m])) + ) > 0.7 + and sum by(method, path, pod) (rate(ratelimit_requests_total[5m])) > 0 + for: 5m + labels: + severity: warning + annotations: + summary: "Single pod denying over 70% of rate-limited requests" + description: "Pod {{ "{{" }} $labels.pod {{ "}}" }} is denying over 70% of requests on method={{ "{{" }} $labels.method {{ "}}" }}, path={{ "{{" }} $labels.path {{ "}}" }} for over 5 minutes. This may indicate a pod-level issue rather than a global rate limit problem." diff --git a/argocd/config/regional-cluster/grafana/dashboards/infrastructure/clm.json b/argocd/config/regional-cluster/grafana/dashboards/infrastructure/clm.json deleted file mode 100644 index 5ca131cf5..000000000 --- a/argocd/config/regional-cluster/grafana/dashboards/infrastructure/clm.json +++ /dev/null @@ -1,2134 +0,0 @@ -{ - "title": "CLM Observability", - "uid": "clm-observability", - "description": "CLM (Cluster Lifecycle Manager) observability dashboard for the Regional Cluster deployment. Monitors Sentinel, API, HC Adapter, and end-to-end flow metrics in the hyperfleet namespace.", - "tags": ["clm", "hyperfleet", "sentinel", "infrastructure"], - "timezone": "browser", - "refresh": "30s", - "schemaVersion": 38, - "graphTooltip": 1, - "time": { "from": "now-3h", "to": "now" }, - "templating": { - "list": [ - { - "name": "datasource", - "type": "datasource", - "label": "Datasource", - "query": "prometheus", - "current": { "text": "Thanos", "value": "Thanos" }, - "hide": 0, - "refresh": 1 - }, - { - "current": { - "text": "All", - "value": ["$__all"] - }, - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "definition": "label_values(hyperfleet_sentinel_pending_resources, resource_type)", - "includeAll": true, - "label": "Resource Type", - "multi": true, - "name": "resource_type", - "options": [], - "query": { - "query": "label_values(hyperfleet_sentinel_pending_resources, resource_type)", - "refId": "PrometheusVariableQueryEditor-VariableQuery" - }, - "refresh": 2, - "sort": 1, - "type": "query" - } - ] - }, - "panels": [ - { - "collapsed": true, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 0 - }, - "id": 1, - "panels": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 1 - }, - "id": 2, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "hyperfleet_sentinel_pending_resources{resource_type=~\"$resource_type\"}", - "legendFormat": "{{resource_type}} - {{resource_selector}}", - "refId": "A" - } - ], - "title": "Pending Resources", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "yellow", - "value": 10 - }, - { - "color": "red", - "value": 50 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 1 - }, - "id": 3, - "options": { - "minVizHeight": 75, - "minVizWidth": 75, - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showThresholdLabels": false, - "showThresholdMarkers": true, - "sizing": "auto" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(hyperfleet_sentinel_pending_resources{resource_type=~\"$resource_type\"})", - "legendFormat": "total", - "refId": "A" - } - ], - "title": "Total Pending Resources", - "type": "gauge" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 9 - }, - "id": 4, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(hyperfleet_sentinel_resources_skipped_total{resource_type=~\"$resource_type\"}[${__rate_interval}])", - "legendFormat": "{{resource_type}} - {{reason}}", - "refId": "A" - } - ], - "title": "Resources Skipped Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 9 - }, - "id": 5, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(hyperfleet_sentinel_events_published_total{resource_type=~\"$resource_type\"}[${__rate_interval}])", - "legendFormat": "{{resource_type}} - {{reason}}", - "refId": "A" - } - ], - "title": "Events Published Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "align": "auto", - "cellOptions": { - "type": "auto" - }, - "inspect": false - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 9 - }, - "id": 6, - "options": { - "cellHeight": "sm", - "footer": { - "countRows": false, - "fields": "", - "reducer": ["sum"], - "show": false - }, - "showHeader": true - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum by (resource_type, resource_selector) (rate(hyperfleet_sentinel_poll_duration_seconds_count{resource_type=~\"$resource_type\"}[${__rate_interval}]))", - "format": "table", - "instant": true, - "refId": "A" - } - ], - "title": "Poll Rate by Resource Type", - "transformations": [ - { - "id": "organize", - "options": { - "excludeByName": { - "Time": true - }, - "indexByName": {}, - "renameByName": { - "Value": "Polls/sec", - "resource_selector": "Resource Selector", - "resource_type": "Resource Type" - } - } - } - ], - "type": "table" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "normal" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 1 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 17 - }, - "id": 7, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(hyperfleet_sentinel_broker_errors_total{resource_type=~\"$resource_type\"}[${__rate_interval}])", - "legendFormat": "{{resource_type}} - {{error_type}}", - "refId": "A" - } - ], - "title": "Broker Error Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 17 - }, - "id": 8, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum by (le) (rate(hyperfleet_sentinel_poll_duration_seconds_bucket{resource_type=~\"$resource_type\"}[${__rate_interval}])))", - "legendFormat": "p50", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum by (le) (rate(hyperfleet_sentinel_poll_duration_seconds_bucket{resource_type=~\"$resource_type\"}[${__rate_interval}])))", - "legendFormat": "p95", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum by (le) (rate(hyperfleet_sentinel_poll_duration_seconds_bucket{resource_type=~\"$resource_type\"}[${__rate_interval}])))", - "legendFormat": "p99", - "refId": "C" - } - ], - "title": "Poll Duration (p50 / p95 / p99)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "normal" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 1 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 17 - }, - "id": 9, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "rate(hyperfleet_sentinel_api_errors_total{resource_type=~\"$resource_type\"}[${__rate_interval}])", - "legendFormat": "{{resource_type}} - {{error_type}}", - "refId": "A" - } - ], - "title": "API Error Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 24, - "x": 0, - "y": 25 - }, - "id": 10, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "time() - sum(hyperfleet_sentinel_last_successful_poll_timestamp_seconds)", - "legendFormat": "time since last poll", - "refId": "A" - } - ], - "title": "Time Since Last Successful Poll (Deadman's Switch)", - "type": "timeseries" - } - ], - "title": "Sentinel", - "type": "row" - }, - { - "collapsed": true, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 1 - }, - "id": 11, - "panels": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 2 - }, - "id": 12, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum by (method, path, code) (rate(hyperfleet_api_requests_total[${__rate_interval}]))", - "legendFormat": "{{method}} {{path}} {{code}}", - "refId": "A" - } - ], - "title": "Request Rate (by path / method / code)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "reqps" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 2 - }, - "id": 13, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_api_requests_total{code=~\"4..\"}[${__rate_interval}]))", - "legendFormat": "4xx", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_api_requests_total{code=~\"5..\"}[${__rate_interval}]))", - "legendFormat": "5xx", - "refId": "B" - } - ], - "title": "Error Rate (4xx / 5xx)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 10 - }, - "id": 14, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum by (le) (rate(hyperfleet_api_request_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p50", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum by (le) (rate(hyperfleet_api_request_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p95", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum by (le) (rate(hyperfleet_api_request_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p99", - "refId": "C" - } - ], - "title": "Request Latency (p50 / p95 / p99)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 10 - }, - "id": 15, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum by (le, table, operation) (rate(hyperfleet_db_query_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p50 {{table}}/{{operation}}", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum by (le, table, operation) (rate(hyperfleet_db_query_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p95 {{table}}/{{operation}}", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum by (le, table, operation) (rate(hyperfleet_db_query_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p99 {{table}}/{{operation}}", - "refId": "C" - } - ], - "title": "DB Query Latency (p50 / p95 / p99)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 10 - }, - "id": 16, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(hyperfleet_db_connections_open)", - "legendFormat": "open", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(hyperfleet_db_connections_in_use)", - "legendFormat": "in-use", - "refId": "B" - } - ], - "title": "Connection Pool (open vs in-use)", - "type": "timeseries" - } - ], - "title": "API", - "type": "row" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 2 - }, - "id": 17, - "panels": [], - "title": "HC Adapter", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 0, - "y": 3 - }, - "id": 18, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum by (status) (rate(hyperfleet_adapter_events_processed_total[${__rate_interval}]))", - "legendFormat": "{{status}}", - "refId": "A" - } - ], - "title": "Events Processed Rate (by status)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 8, - "y": 3 - }, - "id": 19, - "options": { - "legend": { - "calcs": ["last", "sum"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum by (phase) (rate(hyperfleet_adapter_errors_total[${__rate_interval}]))", - "legendFormat": "{{phase}}", - "refId": "A" - } - ], - "title": "Errors by Phase", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "s" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 8, - "x": 16, - "y": 3 - }, - "id": 20, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.50, sum by (le) (rate(hyperfleet_adapter_event_processing_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p50", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.95, sum by (le) (rate(hyperfleet_adapter_event_processing_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p95", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "histogram_quantile(0.99, sum by (le) (rate(hyperfleet_adapter_event_processing_duration_seconds_bucket[${__rate_interval}])))", - "legendFormat": "p99", - "refId": "C" - } - ], - "title": "Processing Duration (p50 / p95 / p99)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 11 - }, - "id": 21, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_broker_messages_consumed_total[${__rate_interval}]))", - "legendFormat": "consumed/s", - "refId": "A" - } - ], - "title": "Broker Consume Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "mappings": [], - "max": 1, - "min": 0, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": 0 - }, - { - "color": "green", - "value": 1 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 11 - }, - "id": 22, - "options": { - "minVizHeight": 75, - "minVizWidth": 75, - "orientation": "auto", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showThresholdLabels": false, - "showThresholdMarkers": true, - "sizing": "auto" - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(hyperfleet_adapter_up)", - "legendFormat": "adapter up", - "refId": "A" - } - ], - "title": "Adapter Status (up/down)", - "type": "gauge" - }, - { - "collapsed": false, - "gridPos": { - "h": 1, - "w": 24, - "x": 0, - "y": 19 - }, - "id": 23, - "panels": [], - "title": "End-to-End Flow", - "type": "row" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 0, - "y": 20 - }, - "id": 24, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_broker_messages_published_total[${__rate_interval}]))", - "legendFormat": "published (sentinel)", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_broker_messages_consumed_total[${__rate_interval}]))", - "legendFormat": "consumed (adapter)", - "refId": "B" - } - ], - "title": "Broker Messages: Published vs Consumed", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 5, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "never", - "spanNulls": false, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "mappings": [], - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": 0 - }, - { - "color": "red", - "value": 80 - } - ] - }, - "unit": "short" - }, - "overrides": [] - }, - "gridPos": { - "h": 8, - "w": 12, - "x": 12, - "y": 20 - }, - "id": 25, - "options": { - "legend": { - "calcs": ["last", "max"], - "displayMode": "table", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "mode": "multi", - "sort": "desc" - } - }, - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_sentinel_events_published_total[${__rate_interval}]))", - "legendFormat": "sentinel published", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${datasource}" - }, - "expr": "sum(rate(hyperfleet_adapter_events_processed_total[${__rate_interval}]))", - "legendFormat": "adapter processed", - "refId": "B" - } - ], - "title": "Event Correlation: Sentinel Published vs Adapter Processed", - "type": "timeseries" - } - ] -} diff --git a/argocd/config/regional-cluster/grafana/dashboards/infrastructure/rds.json b/argocd/config/regional-cluster/grafana/dashboards/infrastructure/rds.json index ebc4fdc42..70bc0d6cf 100644 --- a/argocd/config/regional-cluster/grafana/dashboards/infrastructure/rds.json +++ b/argocd/config/regional-cluster/grafana/dashboards/infrastructure/rds.json @@ -1,7 +1,7 @@ { "title": "RDS", "uid": "rds-overview", - "description": "Per-instance RDS metrics for Maestro and HyperFleet PostgreSQL databases. BurstBalance is critical for t-type instances — sustained 0% means CPU is throttled.", + "description": "Per-instance RDS metrics for HyperFleet PostgreSQL databases. BurstBalance is critical for t-type instances — sustained 0% means CPU is throttled.", "tags": ["aws", "cloudwatch", "rds", "postgresql", "sre"], "timezone": "browser", "refresh": "1m", diff --git a/argocd/config/regional-cluster/grafana/dashboards/rc/rc-health.json b/argocd/config/regional-cluster/grafana/dashboards/rc/rc-health.json index c0a6af809..f860169b9 100644 --- a/argocd/config/regional-cluster/grafana/dashboards/rc/rc-health.json +++ b/argocd/config/regional-cluster/grafana/dashboards/rc/rc-health.json @@ -23,10 +23,10 @@ "name": "hyperfleet_namespaces", "type": "custom", "label": "Hyperfleet Namespaces", - "query": "clm|argocd|tekton-pipelines|hyperfleet", + "query": "argocd|tekton-pipelines|hyperfleet", "current": { - "text": "clm|argocd|tekton-pipelines|hyperfleet", - "value": "clm|argocd|tekton-pipelines|hyperfleet" + "text": "argocd|tekton-pipelines|hyperfleet", + "value": "argocd|tekton-pipelines|hyperfleet" }, "hide": 2, "includeAll": false, diff --git a/argocd/config/regional-cluster/grafana/templates/dashboards/dashboard-clm.yaml b/argocd/config/regional-cluster/grafana/templates/dashboards/dashboard-clm.yaml deleted file mode 100644 index 97dc34028..000000000 --- a/argocd/config/regional-cluster/grafana/templates/dashboards/dashboard-clm.yaml +++ /dev/null @@ -1,11 +0,0 @@ -apiVersion: v1 -kind: ConfigMap -metadata: - name: grafana-dashboard-clm - namespace: {{ .Release.Namespace }} - labels: - grafana_dashboard: "1" - annotations: - k8s-sidecar-target-directory: "Infrastructure" -data: - clm.json: {{ .Files.Get "dashboards/infrastructure/clm.json" | quote }} diff --git a/argocd/config/regional-cluster/grafana/values.yaml b/argocd/config/regional-cluster/grafana/values.yaml index 0b10049cf..f30fca5b9 100644 --- a/argocd/config/regional-cluster/grafana/values.yaml +++ b/argocd/config/regional-cluster/grafana/values.yaml @@ -61,6 +61,7 @@ grafana: users: allow_sign_up: false auto_assign_org_role: Admin + viewers_can_edit: true analytics: reporting_enabled: false diff --git a/argocd/config/regional-cluster/hyperfleet/values.yaml b/argocd/config/regional-cluster/hyperfleet/values.yaml index 964c1da97..dfef0f4a3 100644 --- a/argocd/config/regional-cluster/hyperfleet/values.yaml +++ b/argocd/config/regional-cluster/hyperfleet/values.yaml @@ -8,9 +8,9 @@ hyperfleet: project: default source: - repoURL: https://github.com/typeid/hyperfleet-operator.git - targetRevision: main - path: charts/hyperfleet-operator + repoURL: https://github.com/openshift-online/rosa-hyperfleet-api.git + targetRevision: f7ec7d59ed8523bc66277cce3f4814e0dee5fa83 + path: hyperfleet-operator/charts syncPolicy: automated: @@ -19,8 +19,8 @@ hyperfleet: helmValues: image: - repository: quay.io/cbusse_openshift/hyperfleet-operator - tag: "pgruntimev2" + repository: quay.io/redhat-user-workloads/rosa-tenant/hyperfleet-operator + tag: "f7ec7d59ed8523bc66277cce3f4814e0dee5fa83" pullPolicy: Always replicaCount: 2 diff --git a/argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml b/argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml index 6645daf82..331a5a534 100644 --- a/argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml +++ b/argocd/config/regional-cluster/loki/templates/targetgroupbinding.yaml @@ -33,21 +33,3 @@ spec: targetGroupARN: {{ .Values.platform.queryFrontendTargetGroup.arn | quote }} targetType: {{ .Values.platform.queryFrontendTargetGroup.targetType | default "ip" }} {{- end }} ---- -{{- if .Values.platform.sre.targetGroup.arn }} -apiVersion: eks.amazonaws.com/v1 -kind: TargetGroupBinding -metadata: - name: loki-query-frontend-sre - namespace: {{ include "loki.namespace" . }} - labels: - {{- include "loki.labels" . | nindent 4 }} - annotations: - {{- include "loki.annotations" . | nindent 4 }} -spec: - serviceRef: - name: loki-query-frontend - port: 3100 - targetGroupARN: {{ .Values.platform.sre.targetGroup.arn | quote }} - targetType: ip -{{- end }} diff --git a/argocd/config/regional-cluster/loki/values.yaml b/argocd/config/regional-cluster/loki/values.yaml index a95b2c34f..481f4655b 100644 --- a/argocd/config/regional-cluster/loki/values.yaml +++ b/argocd/config/regional-cluster/loki/values.yaml @@ -23,10 +23,6 @@ platform: serviceName: loki-query-frontend port: 3100 targetType: ip - # SRE UI ALB — injected at runtime by ApplicationSet; empty when ALB is not deployed - sre: - targetGroup: - arn: "" serviceAccount: name: loki diff --git a/argocd/config/regional-cluster/platform-api/README.md b/argocd/config/regional-cluster/platform-api/README.md index 1e7d495d0..d4eb1d3dd 100644 --- a/argocd/config/regional-cluster/platform-api/README.md +++ b/argocd/config/regional-cluster/platform-api/README.md @@ -32,7 +32,6 @@ platformApi: tag: nodb args: allowedAccounts: "123456789012" # Comma-separated AWS account IDs - maestroUrl: http://maestro:8000 envoy: enabled: true @@ -69,7 +68,6 @@ platformApi: tag: "v1.2.3" args: allowedAccounts: "111111111111,222222222222" - maestroUrl: http://maestro.maestro.svc.cluster.local:8000 logLevel: debug targetGroup: @@ -126,7 +124,6 @@ kubectl delete namespace platform-api | `platformApi.app.image.repository` | Container image repository | `quay.io/cdoan0/rosa-regional-platform-api` | | `platformApi.app.image.tag` | Container image tag | `nodb` | | `platformApi.app.args.allowedAccounts` | Comma-separated AWS account IDs | `"123456789012"` | -| `platformApi.app.args.maestroUrl` | Maestro service URL | `http://maestro:8000` | | `platformApi.app.args.logLevel` | Log level (debug, info, warn, error) | `info` | | `platformApi.deployment.replicas` | Number of replicas | `1` | diff --git a/argocd/config/regional-cluster/platform-api/templates/deployment.yaml b/argocd/config/regional-cluster/platform-api/templates/deployment.yaml index 83ecb44f3..7eccd8dfa 100644 --- a/argocd/config/regional-cluster/platform-api/templates/deployment.yaml +++ b/argocd/config/regional-cluster/platform-api/templates/deployment.yaml @@ -17,6 +17,9 @@ spec: annotations: checksum/zoa-job-config: {{ include (print $.Template.BasePath "/zoa-job-config-configmap.yaml") . | sha256sum }} checksum/zoa-ta-templates: {{ include (print $.Template.BasePath "/zoa-templates-configmap.yaml") . | sha256sum }} + {{- if .Values.platformApi.rateLimit.enabled }} + checksum/rate-limits: {{ include (print $.Template.BasePath "/ratelimit-configmap.yaml") . | sha256sum }} + {{- end }} spec: serviceAccountName: {{ .Values.platformApi.serviceAccount.name }} topologySpreadConstraints: @@ -69,8 +72,17 @@ spec: - name: ZOA_JOB_CONFIG_DIR value: /etc/zoa/job-config {{- end }} - {{- if .Values.platformApi.zoa.enabled }} + {{- if .Values.platformApi.rateLimit.enabled }} + - name: RATE_LIMIT_ENABLED + value: "true" + - name: RATE_LIMIT_CONFIG_FILE + value: /etc/platform-api/rate-limits/limits.yaml + - name: REDIS_ENDPOINT + value: {{ .Values.platformApi.rateLimit.redisEndpoint | quote }} + {{- end }} + {{- if or .Values.platformApi.zoa.enabled .Values.platformApi.rateLimit.enabled }} volumeMounts: + {{- if .Values.platformApi.zoa.enabled }} - name: zoa-templates mountPath: /etc/zoa/templates readOnly: true @@ -78,6 +90,12 @@ spec: mountPath: /etc/zoa/job-config readOnly: true {{- end }} + {{- if .Values.platformApi.rateLimit.enabled }} + - name: rate-limits + mountPath: /etc/platform-api/rate-limits + readOnly: true + {{- end }} + {{- end }} ports: - name: api containerPort: {{ .Values.platformApi.app.args.apiPort }} @@ -158,3 +176,8 @@ spec: configMap: name: zoa-job-config {{- end }} + {{- if .Values.platformApi.rateLimit.enabled }} + - name: rate-limits + configMap: + name: rate-limits + {{- end }} diff --git a/argocd/config/regional-cluster/platform-api/templates/ratelimit-configmap.yaml b/argocd/config/regional-cluster/platform-api/templates/ratelimit-configmap.yaml new file mode 100644 index 000000000..eca66a3f4 --- /dev/null +++ b/argocd/config/regional-cluster/platform-api/templates/ratelimit-configmap.yaml @@ -0,0 +1,10 @@ +{{- if .Values.platformApi.rateLimit.enabled }} +apiVersion: v1 +kind: ConfigMap +metadata: + name: rate-limits + namespace: {{ .Values.platformApi.namespace }} +data: + limits.yaml: | +{{ .Values.platformApi.rateLimit.config | indent 4 }} +{{- end }} diff --git a/argocd/config/regional-cluster/platform-api/templates/servicemonitor.yaml b/argocd/config/regional-cluster/platform-api/templates/servicemonitor.yaml new file mode 100644 index 000000000..229ef66cc --- /dev/null +++ b/argocd/config/regional-cluster/platform-api/templates/servicemonitor.yaml @@ -0,0 +1,24 @@ +{{- if .Values.platformApi.serviceMonitor.enabled }} +--- +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ .Values.platformApi.app.name }} + namespace: {{ .Values.platformApi.namespace }} + labels: + app: {{ .Values.platformApi.app.name }} + {{- with .Values.platformApi.serviceMonitor.labels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + endpoints: + - port: metrics + interval: {{ .Values.platformApi.serviceMonitor.interval }} + scrapeTimeout: {{ .Values.platformApi.serviceMonitor.scrapeTimeout }} + namespaceSelector: + matchNames: + - {{ .Values.platformApi.namespace }} + selector: + matchLabels: + app: {{ .Values.platformApi.app.name }} +{{- end }} diff --git a/argocd/config/regional-cluster/platform-api/values.yaml b/argocd/config/regional-cluster/platform-api/values.yaml index 2aee5b3c9..5eebbc130 100644 --- a/argocd/config/regional-cluster/platform-api/values.yaml +++ b/argocd/config/regional-cluster/platform-api/values.yaml @@ -17,8 +17,8 @@ platformApi: app: name: platform-api image: - repository: quay.io/cbusse_openshift/rosa-regional-platform-api - tag: "pgruntime" + repository: quay.io/redhat-user-workloads/rosa-tenant/platform-api + tag: "f7ec7d59ed8523bc66277cce3f4814e0dee5fa83" pullPolicy: Always # Application arguments @@ -100,6 +100,66 @@ platformApi: cpu: "250m" memory: "256Mi" + # ServiceMonitor for Prometheus scraping + serviceMonitor: + enabled: true + interval: 30s + scrapeTimeout: 10s + labels: {} + + # Rate Limiting configuration + rateLimit: + enabled: true + redisEndpoint: "" + config: | + enabled: true + redisTimeout: 20 # milliseconds — fail open faster than the 50ms code default if Valkey is degraded + exemptAccounts: [] + default: + rate: 20 + burst: 40 + routes: + - path: "/api/v0/clusters" + method: POST + rate: 5 + burst: 10 + - path: "/api/v0/clusters" + method: GET + rate: 20 + burst: 40 + - path: "/api/v0/clusters/*" + method: GET + rate: 30 + burst: 60 + - path: "/api/v0/clusters/*" + method: PATCH + rate: 10 + burst: 20 + - path: "/api/v0/clusters/*" + method: DELETE + rate: 5 + burst: 10 + - path: "/api/v0/nodepools" + method: POST + rate: 10 + burst: 20 + - path: "/api/v0/nodepools" + method: GET + rate: 20 + burst: 40 + - path: "/api/v0/nodepools/*" + method: GET + rate: 30 + burst: 60 + - path: "/api/v0/nodepools/*" + method: PATCH + rate: 10 + burst: 20 + - path: "/api/v0/trusted-actions/*/run" + method: POST + rate: 20 + burst: 30 + # Health probe configuration probes: liveness: diff --git a/ci/Containerfile b/ci/Containerfile index b2a8028e1..da7f72ca2 100644 --- a/ci/Containerfile +++ b/ci/Containerfile @@ -14,7 +14,7 @@ RUN if [ -n "${PROXY_CA_CERT}" ]; then \ # Tool versions ARG TERRAFORM_VERSION=1.14.3 -ARG HELM_VERSION=v3.16.0 +ARG HELM_VERSION=v3.21.3 ARG UV_VERSION=0.6.0 ARG AWS_CLI_VERSION=2.34.19 ARG PRETTIER_VERSION=3.8.1 @@ -79,12 +79,25 @@ RUN arch=$(uname -m) && \ rm -f "${TF_PACKAGE}" "terraform_${TERRAFORM_VERSION}_SHA256SUMS" \ "terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig" -# Helm 3 (pinned release tag; get-helm-3 verifies checksums by default) +# Helm 3 (SHA-256 verified with retry logic) ARG CURL_RETRY ARG CURL_RETRY_DELAY ARG CURL_MAX_TIME -RUN curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -fsSL "https://raw.githubusercontent.com/helm/helm/${HELM_VERSION}/scripts/get-helm-3" \ - | DESIRED_VERSION="${HELM_VERSION}" VERIFY_CHECKSUM=true bash +RUN arch=$(uname -m) && \ + case "${arch}" in \ + x86_64) arch="amd64" ;; \ + aarch64) arch="arm64" ;; \ + esac && \ + HELM_PACKAGE="helm-${HELM_VERSION}-linux-${arch}.tar.gz" && \ + HELM_BASE_URL="https://get.helm.sh" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -fsSLO "${HELM_BASE_URL}/${HELM_PACKAGE}" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -fsSLO "${HELM_BASE_URL}/${HELM_PACKAGE}.sha256sum" && \ + sha256sum -c "${HELM_PACKAGE}.sha256sum" && \ + tar -xzf "${HELM_PACKAGE}" -C /tmp && \ + mv /tmp/linux-${arch}/helm /usr/local/bin/helm && \ + chmod +x /usr/local/bin/helm && \ + rm -rf /tmp/linux-${arch} "${HELM_PACKAGE}" "${HELM_PACKAGE}.sha256sum" && \ + helm version # uv (Python package manager - pinned version, installer verifies binary checksum) ARG CURL_RETRY diff --git a/ci/e2e-tests.sh b/ci/e2e-tests.sh index 922e79716..eac23afa6 100755 --- a/ci/e2e-tests.sh +++ b/ci/e2e-tests.sh @@ -165,7 +165,7 @@ if [[ "$_have_customer_creds" == "true" ]]; then # Collect cluster logs before HCP cleanup so the HCP namespace is captured. if [[ -n "${CLUSTER_PREFIX+set}" ]]; then - export PRE_CLEANUP_HOOK="S3_ONLY=true ${REPO_ROOT}/scripts/dev/collect-cluster-logs.sh" + export PRE_CLEANUP_HOOK="S3_ONLY=true ${REPO_ROOT}/scripts/dev/dump-env.sh" fi export GINKGO_NO_COLOR=TRUE @@ -194,7 +194,7 @@ if [[ $platform_rc -ne 0 ]] || [[ $zoa_rc -ne 0 ]] || [[ $monitoring_rc -ne 0 ]] # The S3 URIs are printed below for manual retrieval. if [[ -n "${CLUSTER_PREFIX+set}" ]]; then S3_ONLY=true \ - "${REPO_ROOT}/scripts/dev/collect-cluster-logs.sh" || true + "${REPO_ROOT}/scripts/dev/dump-env.sh" || true fi fi diff --git a/ci/ephemeral-provider/git.py b/ci/ephemeral-provider/git.py index 78bb6d7e4..4389400ab 100644 --- a/ci/ephemeral-provider/git.py +++ b/ci/ephemeral-provider/git.py @@ -15,6 +15,7 @@ GIT_TIMEOUT = 120 # seconds; clone/push can be slow on large repos HTTP_TIMEOUT = 30 # seconds; GitHub API calls RENDER_TIMEOUT = 300 # seconds; render.py may run terraform/heavy scripts +UPSTREAM_REPO_NAME = "rosa-hyperfleet" class GitManager: @@ -126,8 +127,7 @@ def create_eph_branch(self, eph_prefix: str): # Add the token owner's fork as the push remote fork_owner = self._resolve_fork_owner(token) - repo_name = self.source_repo.split("/")[-1] - self.fork_repo = f"{fork_owner}/{repo_name}" + self.fork_repo = f"{fork_owner}/{UPSTREAM_REPO_NAME}" fork_url = f"https://github.com/{self.fork_repo}.git" self._run_git("remote", "add", "ci", fork_url) log.info("Push remote: %s (fork of %s)", self.fork_repo, self.source_repo) @@ -173,8 +173,7 @@ def checkout_eph_branch(self, eph_prefix: str): # Resolve fork fork_owner = self._resolve_fork_owner(token) - repo_name = self.source_repo.split("/")[-1] - self.fork_repo = f"{fork_owner}/{repo_name}" + self.fork_repo = f"{fork_owner}/{UPSTREAM_REPO_NAME}" fork_url = f"https://github.com/{self.fork_repo}.git" tmpdir = tempfile.mkdtemp(prefix="ephemeral-") diff --git a/ci/ephemeral-provider/orchestrator.py b/ci/ephemeral-provider/orchestrator.py index d82045128..62d13a128 100644 --- a/ci/ephemeral-provider/orchestrator.py +++ b/ci/ephemeral-provider/orchestrator.py @@ -145,11 +145,6 @@ def teardown(self, fire_and_forget: bool = False): self._setup_aws() - # Collect CodeBuild logs before teardown destroys infrastructure. - # In Prow, teardown runs as a separate step — this captures logs - # from the provisioning phase that would otherwise be lost. - self.collect_codebuild_logs() - # Purge clusters and resource bundles before infrastructure teardown. # Deleting bundles triggers ManifestWork removal on the MC, which is # what actually tears down the HostedClusters. Without this, terraform @@ -217,9 +212,11 @@ def _setup_aws(self): def _inject_ephemeral_config(self, git: GitManager): """Inject the ephemeral environment config into the cloned repo. - If an override directory (.ephemeral-env/) is provided, it replaces the - config/ephemeral/ directory entirely. Otherwise the repo's default - config/ephemeral/ is used as-is. + If an override directory (.ephemeral-env/) is provided, region YAML + files in config/ephemeral/ are replaced with the override's, while + defaults.yaml is deep-merged with the override (not replaced) to + preserve existing environment settings the override omits. Otherwise + the repo's default config/ephemeral/ is used as-is. In both cases, AWS account IDs are injected into the region config from the runtime credentials (never from config files). @@ -229,12 +226,19 @@ def _inject_ephemeral_config(self, git: GitManager): # Replace config with overrides if provided if self.override_dir and self.override_dir.exists(): log.info("Applying environment overrides from %s", self.override_dir) - # Ensure target directory exists and clear existing config env_config_dir.mkdir(parents=True, exist_ok=True) + # Delete only region files (not defaults.yaml) to purge stale configs. + # defaults.yaml is merged rather than replaced to preserve env-level + # settings (e.g. dns.domain) that the override file omits. for existing in env_config_dir.glob("*.yaml"): - existing.unlink() + if existing.name != "defaults.yaml": + existing.unlink() for override_file in self.override_dir.glob("*.yaml"): - shutil.copy2(override_file, env_config_dir / override_file.name) + target = env_config_dir / override_file.name + if override_file.name == "defaults.yaml" and target.exists(): + load_and_merge(target, override_file) + else: + shutil.copy2(override_file, target) # Validate: exactly 1 region file must exist (enforced by discover_region # at startup, but re-check after override replacement) @@ -408,7 +412,10 @@ def _wait_for_provision(self): failed.append(pipeline_name) if failed: - self.collect_codebuild_logs() + try: + self.collect_codebuild_logs() + except Exception: + log.exception("Failed to collect CodeBuild logs") raise RuntimeError( f"{len(failed)} pipeline(s) failed during provisioning: {', '.join(failed)}" ) @@ -722,6 +729,7 @@ def set_delete_flag(region_config): ] # Monitor all teardown pipelines concurrently + failed = [] if teardown_pipelines: with ThreadPoolExecutor(max_workers=len(teardown_pipelines)) as executor: future_to_pipeline = { @@ -735,7 +743,16 @@ def set_delete_flag(region_config): future.result() except (RuntimeError, TimeoutError) as e: log.error("Teardown pipeline '%s' failed: %s", pipeline_name, e) - # Continue with teardown even if infrastructure destroy fails + failed.append(pipeline_name) + + if failed: + try: + self.collect_codebuild_logs() + except Exception: + log.exception("Failed to collect teardown CodeBuild logs") + raise RuntimeError( + f"{len(failed)} pipeline(s) failed during teardown: {', '.join(failed)}" + ) # Phase 2: Pipeline teardown log.info("") diff --git a/ci/promtool-test/ratelimit-rules_test.yaml b/ci/promtool-test/ratelimit-rules_test.yaml new file mode 100644 index 000000000..1eeb13486 --- /dev/null +++ b/ci/promtool-test/ratelimit-rules_test.yaml @@ -0,0 +1,171 @@ +evaluation_interval: 1m + +rule_files: + - rules.yaml + +tests: + # RateLimitFailOpenActive fires when >20% of requests are fail-open for >5m + # 100% fail-open: all requests are failure_mode_allowed, none are ok + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="failure_mode_allowed", method="GET", path="/api/v0/clusters"}' + values: "0+10x15" + alert_rule_test: + - eval_time: 6m + alertname: RateLimitFailOpenActive + exp_alerts: + - exp_labels: + severity: warning + exp_annotations: + summary: "Rate limit enforcement bypassed (>20% fail-open)" + description: "100% of rate-limited traffic passing without enforcement for >5m. Check Valkey connectivity." + + # RateLimitFailOpenActive does NOT fire when fail-open ratio is below 20% + # 10% fail-open: 1 failure_mode_allowed per 9 ok + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="failure_mode_allowed", method="GET", path="/api/v0/clusters"}' + values: "0+1x15" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters"}' + values: "0+9x15" + alert_rule_test: + - eval_time: 10m + alertname: RateLimitFailOpenActive + exp_alerts: [] + + # RateLimitFailOpenActive does NOT fire when there are no failure_mode_allowed requests + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters"}' + values: "0+10x15" + alert_rule_test: + - eval_time: 10m + alertname: RateLimitFailOpenActive + exp_alerts: [] + + # RateLimitFailOpenActive does NOT fire before 5m even at 100% fail-open + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="failure_mode_allowed", method="GET", path="/api/v0/clusters"}' + values: "0+10x4" + alert_rule_test: + - eval_time: 4m + alertname: RateLimitFailOpenActive + exp_alerts: [] + + # RateLimitHighDenialRatio fires when > 50% of requests are over_limit for > 10m + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters"}' + values: "0+6x20" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters"}' + values: "0+4x20" + alert_rule_test: + - eval_time: 11m + alertname: RateLimitHighDenialRatio + exp_alerts: + - exp_labels: + severity: warning + method: GET + path: /api/v0/clusters + exp_annotations: + summary: "Over 50% of requests are being rate limited" + description: "More than 50% of rate-limited requests on method=GET, path=/api/v0/clusters have been denied for over 10 minutes. This may indicate a misbehaving client or rate limits that need tuning." + + # RateLimitHighDenialRatio fires when combined ratio across pods exceeds 50% + # Pod A: 3 denied + 7 ok = 30% per pod, Pod B: 4 denied + 2 ok = 67% per pod + # Combined: 7 denied / 16 total = 43.75% — does NOT fire (under 50%) + # But with higher combined: Pod A: 5 denied + 3 ok, Pod B: 4 denied + 2 ok + # Combined: 9 denied / 14 total = 64% — fires + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+5x20" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+3x20" + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters", pod="api-1"}' + values: "0+4x20" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters", pod="api-1"}' + values: "0+2x20" + alert_rule_test: + - eval_time: 11m + alertname: RateLimitHighDenialRatio + exp_alerts: + - exp_labels: + severity: warning + method: GET + path: /api/v0/clusters + exp_annotations: + summary: "Over 50% of requests are being rate limited" + description: "More than 50% of rate-limited requests on method=GET, path=/api/v0/clusters have been denied for over 10 minutes. This may indicate a misbehaving client or rate limits that need tuning." + + # RateLimitHighDenialRatio does NOT fire when denial ratio is below 50% + # over_limit is 10% of total traffic (1 denied per 9 ok) + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters"}' + values: "0+1x20" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters"}' + values: "0+9x20" + alert_rule_test: + - eval_time: 15m + alertname: RateLimitHighDenialRatio + exp_alerts: [] + + # RateLimitHighDenialRatio does NOT fire before 10m even with high denial ratio + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters"}' + values: "0+9x8" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters"}' + values: "0+1x8" + alert_rule_test: + - eval_time: 8m + alertname: RateLimitHighDenialRatio + exp_alerts: [] + + # RateLimitPodHighDenialRatio fires when a single pod exceeds 70% denial for > 5m + # Pod api-0: 8 denied + 2 ok = 80% denial ratio + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+8x15" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+2x15" + alert_rule_test: + - eval_time: 6m + alertname: RateLimitPodHighDenialRatio + exp_alerts: + - exp_labels: + severity: warning + method: GET + path: /api/v0/clusters + pod: api-0 + exp_annotations: + summary: "Single pod denying over 70% of rate-limited requests" + description: "Pod api-0 is denying over 70% of requests on method=GET, path=/api/v0/clusters for over 5 minutes. This may indicate a pod-level issue rather than a global rate limit problem." + + # RateLimitPodHighDenialRatio does NOT fire when pod denial ratio is below 70% + # Pod api-0: 6 denied + 4 ok = 60% denial ratio + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+6x15" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+4x15" + alert_rule_test: + - eval_time: 10m + alertname: RateLimitPodHighDenialRatio + exp_alerts: [] + + # RateLimitPodHighDenialRatio does NOT fire before 5m even with high denial ratio + - interval: 1m + input_series: + - series: 'ratelimit_requests_total{result="over_limit", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+9x4" + - series: 'ratelimit_requests_total{result="ok", method="GET", path="/api/v0/clusters", pod="api-0"}' + values: "0+1x4" + alert_rule_test: + - eval_time: 4m + alertname: RateLimitPodHighDenialRatio + exp_alerts: [] diff --git a/config/README.md b/config/README.md index 300f90a08..8b27a86bf 100644 --- a/config/README.md +++ b/config/README.md @@ -147,8 +147,8 @@ terraform_common: applications: regional-cluster: - maestro: - mqttEndpoint: "xxx.iot.{{ aws_region }}.amazonaws.com" + pagerduty: + integrationKey: "xxx" ``` ### integration/defaults.yaml diff --git a/config/defaults.yaml b/config/defaults.yaml index 8f1f18b8f..997a8823b 100644 --- a/config/defaults.yaml +++ b/config/defaults.yaml @@ -68,6 +68,9 @@ regional_cluster: # @doc regional_cluster.enable_api_custom_domain Enable API Gateway custom domain and ACM certificate. Off by default; adds ~20 minutes for certificate validation. # @used-by regional_cluster.enable_api_custom_domain pipeline-regional-cluster-inputs/terraform.json.j2 enable_api_custom_domain: false + # @doc regional_cluster.enable_rate_limit_redis Enable ElastiCache Redis for Platform API rate limiting. + # @used-by regional_cluster.enable_rate_limit_redis pipeline-regional-cluster-inputs/terraform.json.j2 + enable_rate_limit_redis: true # @doc regional_cluster.enable_write_sre_tools Enable write/admin access to SRE tools (Grafana, ArgoCD). Off by default. Can only be set to true for ephemeral environments — render.py enforces this. # @used-by regional_cluster.enable_write_sre_tools _context enable_write_sre_tools: false @@ -120,9 +123,6 @@ regional_cluster: # @doc regional_cluster.sre_thanos_oidc_client_id OIDC client ID for Thanos. Required when enable_sre_oidc_auth is true. # @used-by regional_cluster.sre_thanos_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 sre_thanos_oidc_client_id: "" - # @doc regional_cluster.sre_loki_oidc_client_id OIDC client ID for Loki. Required when enable_sre_oidc_auth is true. - # @used-by regional_cluster.sre_loki_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 - sre_loki_oidc_client_id: "" management_cluster_defaults: # @doc management_cluster_defaults.enable_bastion Default enable bastion for management clusters. diff --git a/config/integration/defaults.yaml b/config/integration/defaults.yaml index b26a0c1ba..300887778 100644 --- a/config/integration/defaults.yaml +++ b/config/integration/defaults.yaml @@ -16,7 +16,6 @@ regional_cluster: sre_argocd_oidc_client_id: "rrp-argocd-sre-int-us-east-1" sre_prometheus_oidc_client_id: "rrp-prometheus-sre-int-us-east-1" sre_thanos_oidc_client_id: "rrp-thanos-sre-int-us-east-1" - sre_loki_oidc_client_id: "rrp-loki-sre-int-us-east-1" management_cluster_defaults: enable_bastion: true diff --git a/config/templates/argocd-bootstrap/applicationset.yaml.j2 b/config/templates/argocd-bootstrap/applicationset.yaml.j2 index 5628ff7a6..3644f07e4 100644 --- a/config/templates/argocd-bootstrap/applicationset.yaml.j2 +++ b/config/templates/argocd-bootstrap/applicationset.yaml.j2 @@ -46,6 +46,7 @@ spec: selfHeal: true retry: limit: -1 + refresh: true backoff: duration: 5s factor: 2 @@ -84,6 +85,8 @@ spec: tableName: '{{ '{{ .metadata.annotations.zoa_table_name }}' }}' auditTableName: '{{ '{{ .metadata.annotations.zoa_audit_table_name }}' }}' bucketName: '{{ '{{ .metadata.annotations.zoa_bucket_name }}' }}' + rateLimit: + redisEndpoint: '{{ '{{ .metadata.annotations.redis_endpoint }}' }}' # Thanos Operator values (keys must match chart's values.yaml structure) thanos: kmsKeyArn: '{{ '{{ .metadata.annotations.thanos_kms_key_arn }}' }}' @@ -102,10 +105,6 @@ spec: arn: '{{ '{{ .metadata.annotations.loki_distributor_target_group_arn }}' }}' queryFrontendTargetGroup: arn: '{{ '{{ .metadata.annotations.loki_query_frontend_target_group_arn }}' }}' - # SRE UI ALB target group ARNs (empty when enable_sre_tools_gateway = false) - sre: - targetGroup: - arn: '{{ '{{ .metadata.annotations.sre_loki_target_group_arn }}' }}' # Loki subchart runtime values (bucket names, region, SSE-KMS) loki: loki: diff --git a/config/templates/pipeline-regional-cluster-inputs/terraform.json.j2 b/config/templates/pipeline-regional-cluster-inputs/terraform.json.j2 index ab8e7df77..bb198bad8 100644 --- a/config/templates/pipeline-regional-cluster-inputs/terraform.json.j2 +++ b/config/templates/pipeline-regional-cluster-inputs/terraform.json.j2 @@ -12,6 +12,7 @@ "enable_cloudtrail": {{ regional_cluster.enable_cloudtrail | default(true) | tojson }}, "enable_pagerduty": {{ observability.pagerduty.enabled | default(false) | tojson }}, "pagerduty_escalation_policy_id": {{ observability.pagerduty.escalation_policy_id | default('') | tojson }}, + "enable_rate_limit_redis": {{ regional_cluster.enable_rate_limit_redis | default(false) | tojson }}, "enable_api_custom_domain": {{ regional_cluster.enable_api_custom_domain | default(false) | tojson }}, "zone_shard_count": {{ regional_cluster.zone_shard_count | default(1) | tojson }}, "enable_sns_alerting": {{ observability.sns_alerting.enabled | default(false) | tojson }}, @@ -24,7 +25,6 @@ "sre_argocd_oidc_client_id": {{ regional_cluster.sre_argocd_oidc_client_id | default("") | tojson }}, "sre_prometheus_oidc_client_id": {{ regional_cluster.sre_prometheus_oidc_client_id | default("") | tojson }}, "sre_thanos_oidc_client_id": {{ regional_cluster.sre_thanos_oidc_client_id | default("") | tojson }}, - "sre_loki_oidc_client_id": {{ regional_cluster.sre_loki_oidc_client_id | default("") | tojson }}, "environment_domain": {{ dns.domain | default(None) | tojson }}, "regional_id": "{% if eph_prefix %}{{ eph_prefix }}-regional{% else %}regional{% endif %}", "eph_prefix": "{{ eph_prefix | default('') }}", diff --git a/dashboard/fetch-data.sh b/dashboard/fetch-data.sh index c619e8989..f3b0b6026 100755 --- a/dashboard/fetch-data.sh +++ b/dashboard/fetch-data.sh @@ -57,6 +57,17 @@ echo "Fetching needs-ok-to-test PRs (bot authors only)..." fetch_label "needs-ok-to-test" | jq --arg bots "$BOT_AUTHORS" \ '[.[] | select(.author.login | test($bots))]' > /tmp/okt.json +echo "Fetching all open bot PRs (regardless of label)..." +for repo in "${REPOS[@]}"; do + gh pr list --repo "$repo" --author redhat-chai-bot --state open \ + --limit 100 --json "$JSON_FIELDS" 2>/dev/null | \ + jq --arg name "${repo#*/}" '[.[] | . + {repository: {name: $name}}]' +done | jq -s 'add // []' > /tmp/bot_all.json + +# Merge bot PRs into okt, deduplicating by URL +jq -s '(.[0] + .[1]) | unique_by(.url)' /tmp/okt.json /tmp/bot_all.json > /tmp/okt_merged.json +mv /tmp/okt_merged.json /tmp/okt.json + jq -n \ --arg updated "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --slurpfile rr /tmp/rr.json \ diff --git a/deploy/ephemeral/us-east-1/_merged_config.yaml b/deploy/ephemeral/us-east-1/_merged_config.yaml index 190ca248c..f2df6a62b 100644 --- a/deploy/ephemeral/us-east-1/_merged_config.yaml +++ b/deploy/ephemeral/us-east-1/_merged_config.yaml @@ -71,6 +71,9 @@ regional_cluster: # @doc regional_cluster.enable_api_custom_domain Enable API Gateway custom domain and ACM certificate. Off by default; adds ~20 minutes for certificate validation. # @used-by regional_cluster.enable_api_custom_domain pipeline-regional-cluster-inputs/terraform.json.j2 enable_api_custom_domain: false + # @doc regional_cluster.enable_rate_limit_redis Enable ElastiCache Redis for Platform API rate limiting. + # @used-by regional_cluster.enable_rate_limit_redis pipeline-regional-cluster-inputs/terraform.json.j2 + enable_rate_limit_redis: true # @doc regional_cluster.enable_write_sre_tools Enable write/admin access to SRE tools (Grafana, ArgoCD). Off by default. Can only be set to true for ephemeral environments — render.py enforces this. # @used-by regional_cluster.enable_write_sre_tools _context enable_write_sre_tools: true @@ -123,9 +126,6 @@ regional_cluster: # @doc regional_cluster.sre_thanos_oidc_client_id OIDC client ID for Thanos. Required when enable_sre_oidc_auth is true. # @used-by regional_cluster.sre_thanos_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 sre_thanos_oidc_client_id: "" - # @doc regional_cluster.sre_loki_oidc_client_id OIDC client ID for Loki. Required when enable_sre_oidc_auth is true. - # @used-by regional_cluster.sre_loki_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 - sre_loki_oidc_client_id: "" management_cluster_defaults: # @doc management_cluster_defaults.enable_bastion Default enable bastion for management clusters. diff --git a/deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml b/deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml index fbbba4e1e..404b249c5 100644 --- a/deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml +++ b/deploy/ephemeral/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml @@ -45,6 +45,7 @@ spec: selfHeal: true retry: limit: -1 + refresh: true backoff: duration: 5s factor: 2 @@ -83,6 +84,8 @@ spec: tableName: '{{ .metadata.annotations.zoa_table_name }}' auditTableName: '{{ .metadata.annotations.zoa_audit_table_name }}' bucketName: '{{ .metadata.annotations.zoa_bucket_name }}' + rateLimit: + redisEndpoint: '{{ .metadata.annotations.redis_endpoint }}' # Thanos Operator values (keys must match chart's values.yaml structure) thanos: kmsKeyArn: '{{ .metadata.annotations.thanos_kms_key_arn }}' @@ -101,10 +104,6 @@ spec: arn: '{{ .metadata.annotations.loki_distributor_target_group_arn }}' queryFrontendTargetGroup: arn: '{{ .metadata.annotations.loki_query_frontend_target_group_arn }}' - # SRE UI ALB target group ARNs (empty when enable_sre_tools_gateway = false) - sre: - targetGroup: - arn: '{{ .metadata.annotations.sre_loki_target_group_arn }}' # Loki subchart runtime values (bucket names, region, SSE-KMS) loki: loki: diff --git a/deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml b/deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml index 9b352dc50..fdff0e336 100644 --- a/deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml +++ b/deploy/ephemeral/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml @@ -45,6 +45,7 @@ spec: selfHeal: true retry: limit: -1 + refresh: true backoff: duration: 5s factor: 2 @@ -81,6 +82,8 @@ spec: tableName: '{{ .metadata.annotations.zoa_table_name }}' auditTableName: '{{ .metadata.annotations.zoa_audit_table_name }}' bucketName: '{{ .metadata.annotations.zoa_bucket_name }}' + rateLimit: + redisEndpoint: '{{ .metadata.annotations.redis_endpoint }}' # Thanos Operator values (keys must match chart's values.yaml structure) thanos: kmsKeyArn: '{{ .metadata.annotations.thanos_kms_key_arn }}' @@ -99,10 +102,6 @@ spec: arn: '{{ .metadata.annotations.loki_distributor_target_group_arn }}' queryFrontendTargetGroup: arn: '{{ .metadata.annotations.loki_query_frontend_target_group_arn }}' - # SRE UI ALB target group ARNs (empty when enable_sre_tools_gateway = false) - sre: - targetGroup: - arn: '{{ .metadata.annotations.sre_loki_target_group_arn }}' # Loki subchart runtime values (bucket names, region, SSE-KMS) loki: loki: diff --git a/deploy/ephemeral/us-east-1/pipeline-regional-cluster-inputs/terraform.json b/deploy/ephemeral/us-east-1/pipeline-regional-cluster-inputs/terraform.json index f754fb407..36dc7866b 100644 --- a/deploy/ephemeral/us-east-1/pipeline-regional-cluster-inputs/terraform.json +++ b/deploy/ephemeral/us-east-1/pipeline-regional-cluster-inputs/terraform.json @@ -12,6 +12,7 @@ "enable_cloudtrail": false, "enable_pagerduty": false, "pagerduty_escalation_policy_id": "P5KE444", + "enable_rate_limit_redis": true, "enable_api_custom_domain": false, "zone_shard_count": 1, "enable_sns_alerting": true, @@ -24,7 +25,6 @@ "sre_argocd_oidc_client_id": "", "sre_prometheus_oidc_client_id": "", "sre_thanos_oidc_client_id": "", - "sre_loki_oidc_client_id": "", "environment_domain": "dev0.rosa.devshift.net", "regional_id": "regional", "eph_prefix": "", diff --git a/deploy/integration/us-east-1/_merged_config.yaml b/deploy/integration/us-east-1/_merged_config.yaml index 1c546f358..3eb051f7c 100644 --- a/deploy/integration/us-east-1/_merged_config.yaml +++ b/deploy/integration/us-east-1/_merged_config.yaml @@ -71,6 +71,9 @@ regional_cluster: # @doc regional_cluster.enable_api_custom_domain Enable API Gateway custom domain and ACM certificate. Off by default; adds ~20 minutes for certificate validation. # @used-by regional_cluster.enable_api_custom_domain pipeline-regional-cluster-inputs/terraform.json.j2 enable_api_custom_domain: true + # @doc regional_cluster.enable_rate_limit_redis Enable ElastiCache Redis for Platform API rate limiting. + # @used-by regional_cluster.enable_rate_limit_redis pipeline-regional-cluster-inputs/terraform.json.j2 + enable_rate_limit_redis: true # @doc regional_cluster.enable_write_sre_tools Enable write/admin access to SRE tools (Grafana, ArgoCD). Off by default. Can only be set to true for ephemeral environments — render.py enforces this. # @used-by regional_cluster.enable_write_sre_tools _context enable_write_sre_tools: false @@ -123,9 +126,6 @@ regional_cluster: # @doc regional_cluster.sre_thanos_oidc_client_id OIDC client ID for Thanos. Required when enable_sre_oidc_auth is true. # @used-by regional_cluster.sre_thanos_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 sre_thanos_oidc_client_id: "rrp-thanos-sre-int-us-east-1" - # @doc regional_cluster.sre_loki_oidc_client_id OIDC client ID for Loki. Required when enable_sre_oidc_auth is true. - # @used-by regional_cluster.sre_loki_oidc_client_id pipeline-regional-cluster-inputs/terraform.json.j2 - sre_loki_oidc_client_id: "rrp-loki-sre-int-us-east-1" management_cluster_defaults: # @doc management_cluster_defaults.enable_bastion Default enable bastion for management clusters. diff --git a/deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml b/deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml index f646b78e4..13d2a9eb3 100644 --- a/deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml +++ b/deploy/integration/us-east-1/argocd-bootstrap-management-cluster/applicationset.yaml @@ -45,6 +45,7 @@ spec: selfHeal: true retry: limit: -1 + refresh: true backoff: duration: 5s factor: 2 @@ -83,6 +84,8 @@ spec: tableName: '{{ .metadata.annotations.zoa_table_name }}' auditTableName: '{{ .metadata.annotations.zoa_audit_table_name }}' bucketName: '{{ .metadata.annotations.zoa_bucket_name }}' + rateLimit: + redisEndpoint: '{{ .metadata.annotations.redis_endpoint }}' # Thanos Operator values (keys must match chart's values.yaml structure) thanos: kmsKeyArn: '{{ .metadata.annotations.thanos_kms_key_arn }}' @@ -101,10 +104,6 @@ spec: arn: '{{ .metadata.annotations.loki_distributor_target_group_arn }}' queryFrontendTargetGroup: arn: '{{ .metadata.annotations.loki_query_frontend_target_group_arn }}' - # SRE UI ALB target group ARNs (empty when enable_sre_tools_gateway = false) - sre: - targetGroup: - arn: '{{ .metadata.annotations.sre_loki_target_group_arn }}' # Loki subchart runtime values (bucket names, region, SSE-KMS) loki: loki: diff --git a/deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml b/deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml index abf3eff23..e39904232 100644 --- a/deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml +++ b/deploy/integration/us-east-1/argocd-bootstrap-regional-cluster/applicationset.yaml @@ -45,6 +45,7 @@ spec: selfHeal: true retry: limit: -1 + refresh: true backoff: duration: 5s factor: 2 @@ -81,6 +82,8 @@ spec: tableName: '{{ .metadata.annotations.zoa_table_name }}' auditTableName: '{{ .metadata.annotations.zoa_audit_table_name }}' bucketName: '{{ .metadata.annotations.zoa_bucket_name }}' + rateLimit: + redisEndpoint: '{{ .metadata.annotations.redis_endpoint }}' # Thanos Operator values (keys must match chart's values.yaml structure) thanos: kmsKeyArn: '{{ .metadata.annotations.thanos_kms_key_arn }}' @@ -99,10 +102,6 @@ spec: arn: '{{ .metadata.annotations.loki_distributor_target_group_arn }}' queryFrontendTargetGroup: arn: '{{ .metadata.annotations.loki_query_frontend_target_group_arn }}' - # SRE UI ALB target group ARNs (empty when enable_sre_tools_gateway = false) - sre: - targetGroup: - arn: '{{ .metadata.annotations.sre_loki_target_group_arn }}' # Loki subchart runtime values (bucket names, region, SSE-KMS) loki: loki: diff --git a/deploy/integration/us-east-1/pipeline-regional-cluster-inputs/terraform.json b/deploy/integration/us-east-1/pipeline-regional-cluster-inputs/terraform.json index 4d299b4fb..b622d9630 100644 --- a/deploy/integration/us-east-1/pipeline-regional-cluster-inputs/terraform.json +++ b/deploy/integration/us-east-1/pipeline-regional-cluster-inputs/terraform.json @@ -12,6 +12,7 @@ "enable_cloudtrail": true, "enable_pagerduty": true, "pagerduty_escalation_policy_id": "P5KE444", + "enable_rate_limit_redis": true, "enable_api_custom_domain": true, "zone_shard_count": 1, "enable_sns_alerting": true, @@ -24,7 +25,6 @@ "sre_argocd_oidc_client_id": "rrp-argocd-sre-int-us-east-1", "sre_prometheus_oidc_client_id": "rrp-prometheus-sre-int-us-east-1", "sre_thanos_oidc_client_id": "rrp-thanos-sre-int-us-east-1", - "sre_loki_oidc_client_id": "rrp-loki-sre-int-us-east-1", "environment_domain": "int0.rosa.devshift.net", "regional_id": "regional", "eph_prefix": "", diff --git a/docs/FAQ.md b/docs/FAQ.md index 6de58fabf..ab7bb1f84 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -4,7 +4,7 @@ The regional architecture is designed exclusively for **ROSA HCP** (Hosted Contr ### What is the Regional Cluster and what services run on it? -The Regional Cluster (RC) is an EKS-based cluster running core regional services (Platform API, CLM, Maestro, ArgoCD, Tekton). For the complete three-layer architecture and component details, see [Architecture at a Glance](README.md#architecture-at-a-glance). +The Regional Cluster (RC) is an EKS-based cluster running core regional services (Platform API, hyperfleet-operator, kube-applier, ArgoCD, Tekton). For the complete three-layer architecture and component details, see [Architecture at a Glance](README.md#architecture-at-a-glance). ### What is the difference between the Regional Cluster and the Regional-Access Cluster? @@ -12,7 +12,7 @@ In this implementation we will **not** use Regional-Access Clusters. Instead, we ### What is the Management Cluster Reconciler (MCR)? -- MCR is a component within CLM that orchestrates Management Cluster lifecycle +- MCR is a component within the hyperfleet-operator that orchestrates Management Cluster lifecycle - It enables scalable management of multiple Management Clusters (MCs) per region, as opposed to having a statically defined list of MCs per region. - This component will be developed by the Hyperfleet team. @@ -48,13 +48,13 @@ In this implementation we will **not** use Regional-Access Clusters. Instead, we ### What is the path to recovery after a disaster? -- **Source of truth**: CLM is the single declarative source of truth for cluster state. Its data is persisted in a dedicated RDS database, with regular cross-region backups. +- **Source of truth**: hyperfleet-db (Aurora PostgreSQL) is the single declarative source of truth for cluster state, with regular cross-region backups. - **etcd state of MCs**: Critical for hosted cluster data; etcd snapshots will be continuously backed up to a dedicated DR AWS account (per region) -- **Maestro cache**: Can be rebuilt from CLM; Maestro caches state for performance but CLM is authoritative. Loss of Maestro cache does not impact recovery. +- **kube-applier DynamoDB tables**: Can be rebuilt from hyperfleet-db; kube-applier tables cache desire/status state for resource distribution but hyperfleet-db is authoritative. Loss of these tables does not prevent recovery, though observability and reconciliation continuity may be temporarily affected until state is rehydrated. - **Recovery path**: - Management Cluster recovery: Restore from etcd backups in the DR account - Hosted Cluster recovery: etcd snapshots allow restoration of customer control planes - - CLM state: Persisted in a dedicated RDS database + - hyperfleet-db state: Persisted in Aurora PostgreSQL - **Break-glass access**: On-demand break-glass access for emergency access when normal management flows are unavailable ### What are the key SLOs to maintain during an outage? @@ -62,20 +62,20 @@ In this implementation we will **not** use Regional-Access Clusters. Instead, we This list is not complete, but some key ones are: - Customer cluster API access (HCP control planes) and CUJs -- CLM for cluster lifecycle operations +- Hyperfleet-operator for cluster lifecycle operations - MC Reconciler for dynamic scaling of MCs - Management Cluster availability (hosting control planes) ### What happens when the Kubernetes API on a Management Cluster goes down? - Management Clusters are EKS clusters managed. We would open a support case with AWS to restore the API. -- If the Management Cluster is unrecoverable, we will have to provision a new MC, and restore all the HCPs from etcd backups, as well as update the single source of truth (CLM). +- If the Management Cluster is unrecoverable, we will have to provision a new MC, and restore all the HCPs from etcd backups, as well as update the single source of truth (hyperfleet-db). -### Where does the Maestro client run and how does it handle API unavailability? +### How does kube-applier distribute resources to Management Clusters? -A Maestro agent runs on each Management Cluster, subscribing to MQTT topics and applying received resources to the local Kubernetes API. If the MC API is non-responsive, observability alerts notify SREs. +A kube-applier controller runs on each Management Cluster, reading desire documents from DynamoDB tables (written by the hyperfleet-operator in the RC account) via DynamoDB Streams and applying them to the local Kubernetes API. If the MC API is non-responsive, observability alerts notify SREs. -For full architecture details, see [Maestro MQTT Resource Distribution](design/maestro-mqtt-resource-distribution.md). +For the current architecture, see [HyperFleet Architecture](design/hyperfleet-architecture.md). ### How are new regions deployed? @@ -114,9 +114,9 @@ An AWS feature that enables private connectivity between API Gateway and VPC res ### Is OCM/CS deployed to each region? -**No** — OCM, CS, and AMS are replaced by **CLM** (Cluster Lifecycle Manager), developed as part of the HyperFleet project. One CLM instance runs in each Regional Cluster as the single source of truth for cluster state. +**No** — OCM, CS, and AMS are replaced by the **hyperfleet-operator**, developed as part of the HyperFleet project. One hyperfleet-operator instance runs in each Regional Cluster, backed by hyperfleet-db (Aurora PostgreSQL) as the single source of truth for cluster state. -For CLM component details (hyperfleet-api, hyperfleet-sentinel, hyperfleet-adapter), see the [HyperFleet Adapter1 chart](../argocd/config/regional-cluster/hyperfleet-adapter1-chart/README.md) and [HyperFleet Infrastructure module](../terraform/modules/hyperfleet-infrastructure/README.md). +For architecture details, see [HyperFleet Architecture](design/hyperfleet-architecture.md). ### Is this design without App-Interface in favor of ArgoCD? diff --git a/docs/README.md b/docs/README.md index e99deab1c..a2dc9c733 100644 --- a/docs/README.md +++ b/docs/README.md @@ -10,7 +10,7 @@ The goal is to improve reliability, reduce dependencies on global services, and The architecture consists of three layers within each region: -1. **Regional Cluster (RC)** - EKS-based cluster running core services (Platform API, CLM, Maestro, ArgoCD, Tekton) +1. **Regional Cluster (RC)** - EKS-based cluster running core services (Platform API, hyperfleet-operator, kube-applier, ArgoCD, Tekton) 2. **Management Clusters (MC)** - EKS clusters hosting customer Hosted Control Planes via HyperShift 3. **Customer Hosted Clusters** - ROSA HCP clusters with control planes in MCs and workers in customer accounts @@ -20,31 +20,32 @@ The architecture consists of three layers within each region: Detailed architecture and rationale for key technical decisions: -| Document | Topic | -| ---------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| [Alerting Architecture](design/alerting-architecture.md) | Fan-out alert routing (AlertManager, PagerDuty, SNS) | -| [AWS IAM Hosted Cluster Auth](design/aws-iam-hosted-cluster-authentication.md) | AWS IAM authentication for hosted clusters (experimental) | -| [DNS Architecture](design/dns-architecture.md) | Hierarchical DNS with zone shards, `deployment_name`, DNSSEC chain | -| [ECS Fargate Bootstrap](design/fully-private-eks-bootstrap.md) | How fully private EKS clusters are bootstrapped via ECS | -| [FIPS-Only EKS Compute](design/fips-eks-compute.md) | FIPS NodeClass/NodePool strategy for FedRAMP workload nodes | -| [GitOps Cluster Configuration](design/gitops-cluster-configuration.md) | ApplicationSet pattern, progressive deployment, config modes | -| [Infrastructure Logging](design/infrastructure-logging.md) | AWS CloudWatch log groups, KMS encryption, Grafana access | -| [Logging Platform](design/logging-platform.md) | Application-level log collection (Vector + Loki) | -| [Maestro MQTT Resource Distribution](design/maestro-mqtt-resource-distribution.md) | RC-to-MC communication via AWS IoT Core MQTT | -| [MC Metrics Remote Write](design/mc-metrics-remote-write.md) | MC-to-RC metrics forwarding via RHOBS API Gateway | -| [Monitoring Platform](design/monitoring-platform.md) | Metrics pipeline (Prometheus + Thanos) | -| [Pipeline-Based Lifecycle](design/pipeline-based-lifecycle.md) | CodePipeline hierarchy for cluster provisioning | -| [Rate Limiting](design/rate-limiting-architecture.md) | Per-account rate limiting for Platform API | -| [Regional Account Minting](design/regional-account-minting.md) | AWS account structure and minting pipelines | -| [Regional OIDC Ownership](design/regional-oidc-ownership.md) | Shared OIDC bucket per region, cross-account MC writes | -| [Spec-to-PR Agent](design/spec-to-pr-agent.md) | AI agent workflow for spec-driven implementation | -| [SRE UI Access](design/sre-ui-access.md) | ALB + OIDC access to SRE UIs replacing SSM port-forward | -| [Terraform Resource Adoption](design/terraform-resource-adoption.md) | Idempotent import of auto-created AWS resources into Terraform | -| [Testing Strategy](design/testing-strategy.md) | Ephemeral and long-lived test environments | -| [Thanos Metrics Infrastructure](design/thanos-metrics-infrastructure.md) | Thanos S3 storage, operator, and Pod Identity setup | -| [ZOA Architecture](design/zoa-architecture.md) | Zero Operator Access — system components, flows, infrastructure | -| [ZOA Trusted Actions](design/zoa-trusted-actions.md) | TA template format, API design, CLI, dispatch flow | -| [ZOA Security Model](design/zoa-security-model.md) | SA isolation, RBAC, audit trail, threat model, FIPS | +| Document | Topic | +| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------ | +| [Alerting Architecture](design/alerting-architecture.md) | Fan-out alert routing (AlertManager, PagerDuty, SNS) | +| [AWS IAM Hosted Cluster Auth](design/aws-iam-hosted-cluster-authentication.md) | AWS IAM authentication for hosted clusters (experimental) | +| [DNS Architecture](design/dns-architecture.md) | Hierarchical DNS with zone shards, `deployment_name`, DNSSEC chain | +| [ECS Fargate Bootstrap](design/fully-private-eks-bootstrap.md) | How fully private EKS clusters are bootstrapped via ECS | +| [FIPS-Only EKS Compute](design/fips-eks-compute.md) | FIPS NodeClass/NodePool strategy for FedRAMP workload nodes | +| [GitOps Cluster Configuration](design/gitops-cluster-configuration.md) | ApplicationSet pattern, progressive deployment, config modes | +| [HyperFleet Architecture](design/hyperfleet-architecture.md) | Operator + kube-applier architecture, component replacement map | +| [Infrastructure Logging](design/infrastructure-logging.md) | AWS CloudWatch log groups, KMS encryption, Grafana access | +| [Logging Platform](design/logging-platform.md) | Application-level log collection (Vector + Loki) | +| [MC Metrics Remote Write](design/mc-metrics-remote-write.md) | MC-to-RC metrics forwarding via RHOBS API Gateway | +| [Monitoring Platform](design/monitoring-platform.md) | Metrics pipeline (Prometheus + Thanos) | +| [Pipeline-Based Lifecycle](design/pipeline-based-lifecycle.md) | CodePipeline hierarchy for cluster provisioning | +| [Rate Limiting](design/rate-limiting-architecture.md) | Per-account rate limiting for Platform API | +| [Regional Account Minting](design/regional-account-minting.md) | AWS account structure and minting pipelines | +| [Regional Control Plane Architecture](design/regional-control-plane-architecture.md) | Operator + PostgreSQL control plane (hyperfleet-operator, hyperfleet-db) | +| [Regional OIDC Ownership](design/regional-oidc-ownership.md) | Shared OIDC bucket per region, cross-account MC writes | +| [Spec-to-PR Agent](design/spec-to-pr-agent.md) | AI agent workflow for spec-driven implementation | +| [SRE UI Access](design/sre-ui-access.md) | ALB + OIDC access to SRE UIs replacing SSM port-forward | +| [Terraform Resource Adoption](design/terraform-resource-adoption.md) | Idempotent import of auto-created AWS resources into Terraform | +| [Testing Strategy](design/testing-strategy.md) | Ephemeral and long-lived test environments | +| [Thanos Metrics Infrastructure](design/thanos-metrics-infrastructure.md) | Thanos S3 storage, operator, and Pod Identity setup | +| [ZOA Architecture](design/zoa-architecture.md) | Zero Operator Access — system components, flows, infrastructure | +| [ZOA Trusted Actions](design/zoa-trusted-actions.md) | TA template format, API design, CLI, dispatch flow | +| [ZOA Security Model](design/zoa-security-model.md) | SA isolation, RBAC, audit trail, threat model, FIPS | ### How-To Guides @@ -74,16 +75,14 @@ Each module has its own README with usage, inputs, outputs, and architecture: - [`api-gateway`](../terraform/modules/api-gateway/README.md) - API Gateway with VPC Link to internal ALB - [`authz`](../terraform/modules/authz/README.md) - Cedar/AVP authorization (DynamoDB, IAM) - [`bastion`](../terraform/modules/bastion/README.md) - Ephemeral bastion for private cluster access -- [`maestro-infrastructure`](../terraform/modules/maestro-infrastructure/README.md) - IoT Core, RDS, Secrets Manager for Maestro Server -- [`maestro-agent`](../terraform/modules/maestro-agent/README.md) - IAM and Pod Identity for Maestro Agent +- [`kube-applier`](../terraform/modules/kube-applier/README.md) - IAM and Pod Identity for the kube-applier controller on MCs +- [`kube-applier-dynamodb`](../terraform/modules/kube-applier-dynamodb/README.md) - DynamoDB tables and backend IAM role for kube-applier (RC account) +- [`hyperfleet-db`](../terraform/modules/hyperfleet-db/) - Aurora PostgreSQL for hyperfleet-operator cluster/nodepool state - [`grafana-cloudwatch-logs`](../terraform/modules/grafana-cloudwatch-logs/) - IAM + Pod Identity for Grafana CloudWatch Logs datasources (RC primary + MC reader) -- [`hyperfleet-infrastructure`](../terraform/modules/hyperfleet-infrastructure/README.md) - RDS, Amazon MQ, IAM for HyperFleet (CLM) ### ArgoCD Helm Chart Documentation -- [`hyperfleet-api-chart`](../argocd/config/regional-cluster/hyperfleet-api-chart/) - HyperFleet API (CLM) -- [`hyperfleet-sentinel-chart`](../argocd/config/regional-cluster/hyperfleet-sentinel-chart/) - HyperFleet Sentinel -- [`hyperfleet-adapter1-chart`](../argocd/config/regional-cluster/hyperfleet-adapter1-chart/README.md) - HyperFleet Adapter (cluster status reporting) +- [`hyperfleet`](../argocd/config/regional-cluster/hyperfleet/) - Hyperfleet Operator (postgres-based cluster lifecycle controller) - [`platform-api`](../argocd/config/regional-cluster/platform-api/README.md) - Platform API with Envoy sidecar - [`thanos`](../argocd/config/regional-cluster/thanos/) - Thanos platform resources (CRs, S3 secret, Pod Identity SA, ALB TargetGroupBinding) plus app-of-apps Application that installs the upstream operator - [`thanos-operator`](../argocd/config/regional-cluster/thanos-operator/) - Thin wrapper chart that delivers the Thanos operator via OCI-packaged Helm subchart diff --git a/docs/design/aws-iam-hosted-cluster-authentication.md b/docs/design/aws-iam-hosted-cluster-authentication.md index de073d367..4ce8a4797 100644 --- a/docs/design/aws-iam-hosted-cluster-authentication.md +++ b/docs/design/aws-iam-hosted-cluster-authentication.md @@ -54,9 +54,9 @@ sequenceDiagram flowchart TB subgraph RC["Regional Cluster"] API["Platform API"] - CLM["CLM"] - Adapter["Adapter"] - Maestro["Maestro"] + HFO["hyperfleet-operator"] + DDB["DynamoDB"] + KAA["kube-applier"] end subgraph MC["Management Cluster"] @@ -74,10 +74,10 @@ flowchart TB HSO["HC Controller"] end - API -->|"creatorARN in spec"| CLM - CLM --> Adapter - Adapter -->|ManifestWork| Maestro - Maestro --> HC_NS + API -->|"creatorARN in spec"| HFO + HFO -->|"desire document"| DDB + KAA -->|"reads desires"| DDB + KAA --> HC_NS HSO -->|"sync ConfigMap"| CM_HCP CM_HCP -.->|"volume mount"| SIDECAR KAS -->|"webhook :21362"| SIDECAR @@ -89,7 +89,7 @@ flowchart TB 1. **Platform API** captures the cluster creator's IAM ARN from the SigV4 request context (`X-Amz-Caller-Arn` header from API Gateway) and stores it in the cluster spec as `creatorARN`. -2. **Adapter** reads `creatorARN` via CEL expression and templates it into the ManifestWork, which delivers to the HC namespace: +2. **Hyperfleet operator** reads `creatorARN` from the cluster spec and includes it in the Manifest CR, which kube-applier delivers to the HC namespace: - A `HostedCluster` with annotation `hypershift.openshift.io/aws-iam-authenticator: "true"` - An `aws-iam-auth-config` ConfigMap mapping the creator ARN to `system:masters` @@ -105,15 +105,14 @@ If `creatorARN` is not set (e.g. API change not deployed), the ConfigMap is stil ### Changes by Repository -| Repository | Files | Change | -| --------------------- | ----------------------------------------- | ----------------------------------------------------------- | -| `rosa-hyperfleet` | `manifestwork.yaml` | `aws-iam-auth-config` ConfigMap, HC annotation | -| `rosa-hyperfleet` | `adapter-task-config.yaml` | `creatorARN` CEL capture | -| `rosa-hyperfleet-api` | `pkg/handlers/cluster.go` | Inject `creatorARN` from SigV4 caller identity | -| `rosa-hyperfleet-cli` | `internal/commands/cluster/kubeconfig.go` | `rosactl cluster kubeconfig` command | -| `hypershift` | `hostedcluster_controller.go` | ConfigMap sync HC->HCP, annotation in `mirroredAnnotations` | -| `hypershift` | `kas/deployment.go` | `aws-iam-authenticator` sidecar injection | -| `hypershift` | `kas/oauth.go` | Webhook redirect to localhost:21362 | +| Repository | Files | Change | +| --------------------- | ---------------------------------------------------------------------------------- | --------------------------------------------------------------------- | +| `rosa-hyperfleet-api` | `hyperfleet-operator/internal/controller/`, `hyperfleet-operator/internal/render/` | `aws-iam-auth-config` ConfigMap, HC annotation, `creatorARN` handling | +| `rosa-hyperfleet-api` | `pkg/handlers/cluster.go` | Inject `creatorARN` from SigV4 caller identity | +| `rosa-hyperfleet-cli` | `internal/commands/cluster/kubeconfig.go` | `rosactl cluster kubeconfig` command | +| `hypershift` | `hostedcluster_controller.go` | ConfigMap sync HC->HCP, annotation in `mirroredAnnotations` | +| `hypershift` | `kas/deployment.go` | `aws-iam-authenticator` sidecar injection | +| `hypershift` | `kas/oauth.go` | Webhook redirect to localhost:21362 | ### Key Configuration @@ -144,4 +143,4 @@ users: ## Related Documentation - [aws-iam-authenticator](https://github.com/kubernetes-sigs/aws-iam-authenticator) -- [Maestro MQTT Resource Distribution](maestro-mqtt-resource-distribution.md) +- [kube-applier Resource Distribution](kube-applier-resource-distribution.md) diff --git a/docs/design/dns-architecture.md b/docs/design/dns-architecture.md index 4fce7110d..bc060807e 100644 --- a/docs/design/dns-architecture.md +++ b/docs/design/dns-architecture.md @@ -87,20 +87,20 @@ Zones created by HyperShift CPO in the customer account (not delegated from shar ### Zone Ownership -| # | Zone / Record | Owner | Notes | -| :-- | :------------------------------------------------------ | :----------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | `openshiftapps.com` | app-interface | Top-level, exists in Cloudflare | -| 2 | `rosa.openshiftapps.com` | Control-account terraform | Commons zone, NS via app-interface | -| 3 | `{deployment_name}.rosa.openshiftapps.com` | Control-account terraform | Regional zone; creates NS record in the commons zone (2) | -| 4 | `api.{deployment_name}.rosa.openshiftapps.com` | Regional pipeline | Platform API record | -| 5 | `{zone_shard}.{deployment_name}.rosa.openshiftapps.com` | Regional pipeline | Zone shard; creates NS record in the regional zone (3). Grants permissions to external-dns and cert-manager from each MC. Informs CLM of all zone shards. | -| 6–8 | Cluster API, OAuth, ACME records | MC external-dns / cert-manager | Created in the zone shard (5) | -| 9 | NS delegation for `in.{...}` in shard | HyperShift CPO | DNSEndpoint CR picked up by external-dns on MC; delegates to the public ingress zone (11) | -| 10 | Private ingress zone + records | HyperShift CPO | VPC-associated (not NS-delegated); created and reconciled in the customer account | -| 11 | Public ingress zone + records | HyperShift CPO | NS-delegated from the shard (5) via (9); includes ACME CNAME delegation for cert-manager | -| 12 | `{cluster_alias}.hypershift.local` | HyperShift CPO | Private zone, VPC-associated, in customer account | - -**CLM responsibilities:** +| # | Zone / Record | Owner | Notes | +| :-- | :------------------------------------------------------ | :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 1 | `openshiftapps.com` | app-interface | Top-level, exists in Cloudflare | +| 2 | `rosa.openshiftapps.com` | Control-account terraform | Commons zone, NS via app-interface | +| 3 | `{deployment_name}.rosa.openshiftapps.com` | Control-account terraform | Regional zone; creates NS record in the commons zone (2) | +| 4 | `api.{deployment_name}.rosa.openshiftapps.com` | Regional pipeline | Platform API record | +| 5 | `{zone_shard}.{deployment_name}.rosa.openshiftapps.com` | Regional pipeline | Zone shard; creates NS record in the regional zone (3). Grants permissions to external-dns and cert-manager from each MC. Informs hyperfleet-operator of all zone shards. | +| 6–8 | Cluster API, OAuth, ACME records | MC external-dns / cert-manager | Created in the zone shard (5) | +| 9 | NS delegation for `in.{...}` in shard | HyperShift CPO | DNSEndpoint CR picked up by external-dns on MC; delegates to the public ingress zone (11) | +| 10 | Private ingress zone + records | HyperShift CPO | VPC-associated (not NS-delegated); created and reconciled in the customer account | +| 11 | Public ingress zone + records | HyperShift CPO | NS-delegated from the shard (5) via (9); includes ACME CNAME delegation for cert-manager | +| 12 | `{cluster_alias}.hypershift.local` | HyperShift CPO | Private zone, VPC-associated, in customer account | + +**Hyperfleet-operator responsibilities:** - Monitor capacity and manage zone shard allocation (zone placement decision) - Propagate the selected zone shard to HyperShift Operator via the HostedCluster CR spec @@ -121,7 +121,7 @@ Zones created by HyperShift CPO in the customer account (not delegated from shar | :---------------- | :-------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------ | :----------------- | :------------------------------------------------------------------------- | | `deployment_name` | 1–25 characters | Subdomain for a regional deployment. Defaults to the AWS region name; suffixed when multiple deployments share a region or for per-run CI environments. | Service | `us-east-1` (len 9), `us-east-1-2` (len 12), `us-east-1-eph-a1b2` (len 19) | | `zone_shard` | 1–3 characters | Subdomain for a regional deployment's HostedZone shard (capped at 100) | Service | `0` (len 1), `99` (len 2) | -| `hash4` | 4 characters | Slug to allow duplicate `cluster_alias` | Service | `1fb9` (len 4) | +| `hash4` | 4 characters | Unique slug per `cluster_alias` within a zone shard; derived from cluster UUID, uniqueness enforced by platform-api at creation time | Service | `1fb9` (len 4) | | `cluster_alias` | 1–15 characters | Alias for the cluster: user-provided (`domain_prefix`) or service-generated hash | Service / Customer | `typeidhcp` (len 10), `4354c27df47cf4e` (len 15) | All identifiers must be DNS-subdomain compatible: lowercase alphanumeric characters or `-`, starting and ending with an alphanumeric character. @@ -177,8 +177,8 @@ Phase 1 covers DNS hierarchy levels 1–8 (all records in the service provider's 1. **DNS delegation setup** — establish the Cloudflare → environment zone → regional zone delegation chain for int, ci, and dev environments 2. **Zone shard creation** — create initial zone shard(s) per regional zone 3. **Cross-account IAM** — create the RC-side IAM role with OU-based trust and MC-side Pod Identity roles for external-dns and cert-manager -4. **CLM zone shard awareness** — CLM tracks zone shard capacity and propagates the selected shard to the HostedCluster CR -5. **Adapter update** — update HostedCluster creation in the adapter to include the DNS domain in the HostedCluster spec +4. **Operator zone shard awareness** — hyperfleet-operator tracks zone shard capacity and propagates the selected shard to the HostedCluster CR +5. **Operator update** — update HostedCluster manifest rendering in the operator to include the DNS domain in the HostedCluster spec ### Phase 2 — Customer Ingress DNS (pending HyperShift RFE) @@ -215,7 +215,7 @@ Phase 2 covers DNS hierarchy levels 9–12 (customer account zones) and depends ### Negative -- Zone shard management adds operational complexity (CLM must track capacity and placement) +- Zone shard management adds operational complexity (hyperfleet-operator must track capacity and placement) - DNSSEC key rotation requires coordination across account tiers - Multi-deployment scenarios require explicit `deployment_name` and `aws_region` overrides in config @@ -241,7 +241,7 @@ Phase 2 covers DNS hierarchy levels 9–12 (customer account zones) and depends ### Operability - Zone shards are created by the regional pipeline — no manual DNS management -- CLM automates shard allocation and capacity monitoring +- Hyperfleet-operator automates shard allocation and capacity monitoring - `deployment_name` defaults to `aws_region` — no separate configuration step needed for the common case ## Testing Strategy diff --git a/docs/design/gitops-cluster-configuration.md b/docs/design/gitops-cluster-configuration.md index 9440b01ce..a600d7691 100644 --- a/docs/design/gitops-cluster-configuration.md +++ b/docs/design/gitops-cluster-configuration.md @@ -53,7 +53,7 @@ sequenceDiagram AppSet->>Secret: Read cluster labels/annotations for identity AppSet->>Git: Discover charts based on cluster_type AppSet->>Git: Use cluster context for value overrides - AppSet->>Apps: Generate Applications (maestro, hypershift, etc.) + AppSet->>Apps: Generate Applications (kube-applier, hypershift, etc.) ``` **Core Process**: ECS bootstrap creates the cluster identity secret and initial Application. ArgoCD then takes over, using the ApplicationSet to dynamically discover and deploy applications based on the cluster's identity. @@ -160,7 +160,7 @@ spec: 4. **GitHub Workaround**: The `replace "github.com" "github.com:443"` in the values source avoids ArgoCD issues with duplicate repository references. -5. **Matrix Result**: If Git discovers 3 directories (hypershift, maestro, argocd), the matrix creates 3 Applications for that cluster, each using the appropriate chart version and latest rendered values. +5. **Matrix Result**: If Git discovers 3 directories (hypershift, kube-applier, argocd), the matrix creates 3 Applications for that cluster, each using the appropriate chart version and latest rendered values. ### Configuration Hierarchy @@ -307,7 +307,7 @@ sequenceDiagram ECS->>ArgoCD: Create Application pointing to static path ArgoCD->>Git: Pull from argocd/charts/management-cluster Git->>ArgoCD: Return static charts + values - ArgoCD->>Apps: Deploy applications (maestro, hypershift, etc.) + ArgoCD->>Apps: Deploy applications (kube-applier, hypershift, etc.) ``` ```yaml @@ -354,14 +354,14 @@ sequenceDiagram ECS->>ArgoCD: Create Application pointing to rendered path ArgoCD->>Git: Pull from rendered/management-cluster/integration/us-west-1 Git->>ArgoCD: Return region-specific rendered charts - ArgoCD->>Apps: Deploy applications (maestro, hypershift, etc.) + ArgoCD->>Apps: Deploy applications (kube-applier, hypershift, etc.) ``` **Repository Structure**: -``` +```text config/management-cluster/ -├── maestro/ +├── kube-applier/ │ ├── Chart.yaml │ ├── values.yaml (defaults) │ └── integration/us-west-1/ # Override yamls/applications (values set in rendered/) @@ -371,7 +371,7 @@ config/management-cluster/ └── integration/us-west-1/ # Override yamls/applications (values set in rendered/) rendered/management-cluster/integration/us-west-1/ # Generated by render script -├── maestro/ +├── kube-applier/ │ ├── Chart.yaml (copied) │ └── values.yaml (merged with overrides) └── hypershift/ @@ -435,7 +435,7 @@ sequenceDiagram AppSet->>Secret: Read cluster labels/annotations for identity AppSet->>Git: Discover charts from config/management-cluster/* AppSet->>Git: Use rendered values for cluster region/env - AppSet->>Apps: Generate Applications (maestro, hypershift, etc.) + AppSet->>Apps: Generate Applications (kube-applier, hypershift, etc.) ``` ```yaml diff --git a/docs/design/infrastructure-logging.md b/docs/design/infrastructure-logging.md index cd91cb07d..a623e956c 100644 --- a/docs/design/infrastructure-logging.md +++ b/docs/design/infrastructure-logging.md @@ -8,7 +8,7 @@ All AWS services deployed in Regional Cluster (RC) and Management Cluster (MC) a ## Scope -This document covers **AWS infrastructure-level logs** — logs produced by AWS services themselves (EKS control plane, RDS, AmazonMQ, IoT Core, API Gateway, ECS). It does NOT cover application-level logs collected by Vector and stored in Loki (see [Logging Platform](logging-platform.md)). +This document covers **AWS infrastructure-level logs** — logs produced by AWS services themselves (EKS control plane, RDS, API Gateway, ECS). It does NOT cover application-level logs collected by Vector and stored in Loki (see [Logging Platform](logging-platform.md)). ## FedRAMP Moderate Controls @@ -20,21 +20,16 @@ This document covers **AWS infrastructure-level logs** — logs produced by AWS ## Regional Cluster Account -| AWS Service | Log Group Name | Terraform Module | Log Level | Retention | KMS | -| -------------------------------- | -------------------------------------------------------- | --------------------------- | ------------------------------------------------------- | --------- | ------------------------------ | -| EKS Control Plane | `/aws/eks/${cluster_id}/cluster` | `eks-cluster` | api, audit, authenticator, controllerManager, scheduler | 365 days | `aws_kms_key.cloudwatch_logs` | -| ECS Bootstrap | `/ecs/${cluster_id}/bootstrap` | `ecs-bootstrap` | Container stdout/stderr | 365 days | `aws_kms_key.bootstrap_logs` | -| ECS Bastion | `/ecs/${cluster_id}/bastion` | `bastion` | Container stdout/stderr + ECS Exec | 365 days | `aws_kms_key.bastion_logs` | -| Maestro RDS (postgresql) | `/aws/rds/instance/${regional_id}-maestro/postgresql` | `maestro-infrastructure` | postgresql | 365 days | `aws_kms_key.rds_logs` | -| Maestro RDS (upgrade) | `/aws/rds/instance/${regional_id}-maestro/upgrade` | `maestro-infrastructure` | upgrade | 365 days | `aws_kms_key.rds_logs` | -| HyperFleet RDS (postgresql) | `/aws/rds/instance/${regional_id}-hyperfleet/postgresql` | `hyperfleet-infrastructure` | postgresql | 365 days | `aws_kms_key.rds_logs` | -| HyperFleet RDS (upgrade) | `/aws/rds/instance/${regional_id}-hyperfleet/upgrade` | `hyperfleet-infrastructure` | upgrade | 365 days | `aws_kms_key.rds_logs` | -| HyperFleet AmazonMQ (general) | `/aws/amazonmq/broker/${broker_id}/general` | `hyperfleet-infrastructure` | general | 365 days | `aws_kms_key.mq_logs` | -| HyperFleet AmazonMQ (connection) | `/aws/amazonmq/broker/${broker_id}/connection` | `hyperfleet-infrastructure` | connection | 365 days | `aws_kms_key.mq_logs` | -| IoT Core | `AWSIotLogsV2` | `maestro-infrastructure` | INFO | 365 days | `aws_kms_key.iot_logs` | -| Platform API Gateway (access) | `/aws/api-gateway/${regional_id}/${stage}/access` | `api-gateway` | Structured JSON (requestId, caller, status, latency) | 365 days | `aws_kms_key.api_gateway_logs` | -| Platform API Gateway (execution) | `API-Gateway-Execution-Logs_${api_id}/${stage}` | `api-gateway` | ERROR | 365 days | `aws_kms_key.api_gateway_logs` | -| RHOBS API Gateway (access) | `/aws/api-gateway/${regional_id}-rhobs/${stage}/access` | `rhobs-api-gateway` | Structured JSON (requestId, caller, status, latency) | 365 days | `aws_kms_key.api_gateway_logs` | +| AWS Service | Log Group Name | Terraform Module | Log Level | Retention | KMS | +| -------------------------------- | -------------------------------------------------------- | ------------------- | ------------------------------------------------------- | --------- | ------------------------------ | +| EKS Control Plane | `/aws/eks/${cluster_id}/cluster` | `eks-cluster` | api, audit, authenticator, controllerManager, scheduler | 365 days | `aws_kms_key.cloudwatch_logs` | +| ECS Bootstrap | `/ecs/${cluster_id}/bootstrap` | `ecs-bootstrap` | Container stdout/stderr | 365 days | `aws_kms_key.bootstrap_logs` | +| ECS Bastion | `/ecs/${cluster_id}/bastion` | `bastion` | Container stdout/stderr + ECS Exec | 365 days | `aws_kms_key.bastion_logs` | +| HyperFleet DB RDS (postgresql) | `/aws/rds/instance/${regional_id}-hyperfleet/postgresql` | `hyperfleet-db` | postgresql | 365 days | `aws_kms_key.rds_logs` | +| HyperFleet DB RDS (upgrade) | `/aws/rds/instance/${regional_id}-hyperfleet/upgrade` | `hyperfleet-db` | upgrade | 365 days | `aws_kms_key.rds_logs` | +| Platform API Gateway (access) | `/aws/api-gateway/${regional_id}/${stage}/access` | `api-gateway` | Structured JSON (requestId, caller, status, latency) | 365 days | `aws_kms_key.api_gateway_logs` | +| Platform API Gateway (execution) | `API-Gateway-Execution-Logs_${api_id}/${stage}` | `api-gateway` | ERROR | 365 days | `aws_kms_key.api_gateway_logs` | +| RHOBS API Gateway (access) | `/aws/api-gateway/${regional_id}-rhobs/${stage}/access` | `rhobs-api-gateway` | Structured JSON (requestId, caller, status, latency) | 365 days | `aws_kms_key.api_gateway_logs` | ## Management Cluster Account diff --git a/docs/design/maestro-mqtt-resource-distribution.md b/docs/design/maestro-mqtt-resource-distribution.md deleted file mode 100644 index fecdd33b7..000000000 --- a/docs/design/maestro-mqtt-resource-distribution.md +++ /dev/null @@ -1,999 +0,0 @@ -# Design Decision 002: MQTT-Based Resource Distribution via Maestro - -## Status - -**Implemented** - -## Table of Contents - -1. [Scope](#scope) -2. [Context](#context) -3. [Alternatives Explored](#alternatives-explored) - - [Alternative 1: Direct REST API Push](#alternative-1-direct-rest-api-push) - - [Alternative 2: Pull-Based with Polling](#alternative-2-pull-based-with-polling) - - [Alternative 3: Message Queue with AWS Services](#alternative-3-message-queue-with-aws-services) -4. [Decision: Maestro with AWS IoT Core MQTT](#decision-maestro-with-aws-iot-core-mqtt) -5. [High-Level Architecture](#high-level-architecture) -6. [Complete Message Flow](#complete-message-flow) -7. [MQTT Topic Structure](#mqtt-topic-structure) -8. [Implementation Design](#implementation-design) - - [Maestro Server](#maestro-server-regional-cluster) - - [Maestro Agent](#maestro-agent-management-cluster) - - [Authentication & Security](#authentication--security) - - [IAM Roles and Cross-Account Setup](#iam-roles-and-cross-account-setup) - - [IAM Trust Relationships](#iam-trust-relationships) - - [Secret Flow Architecture](#secret-flow-architecture) - - [Key IAM Components](#key-iam-components) - - [State Management](#state-management) -9. [Deployment Workflow](#deployment-workflow) -10. [Certificate Transfer Process](#certificate-transfer-process) -11. [Network Topology](#network-topology) -12. [Benefits](#benefits) -13. [Operational Considerations](#operational-considerations) - - [Monitoring & Alerting](#monitoring--alerting) - - [Troubleshooting Guide](#troubleshooting-guide) - - [Performance Tuning](#performance-tuning) - - [Cost Optimization](#cost-optimization) -14. [Related Documentation](#related-documentation) - ---- - -## Scope - -This design decision addresses how the Regional Cluster distributes cluster configuration and resources to Management Clusters without direct network connectivity between their Kubernetes APIs. - -The solution must work in an environment where Management Clusters have fully private Kubernetes APIs with no network path to the Regional Cluster, enabling maximum security isolation while maintaining operational capability. This document provides comprehensive architecture diagrams, detailed implementation guidance, and operational procedures for the Maestro MQTT-based orchestration system in the ROSA HyperFleet. - -## Context - -The rosa-hyperfleet requires a mechanism to distribute HostedCluster and NodePool resources from the Regional Cluster's CLM (Cluster Lifecycle Manager) to multiple Management Clusters across potentially different AWS accounts. - -**Critical Constraint**: Management Clusters MUST have no network path to the Regional Cluster Kubernetes API, and vice versa. This eliminates traditional push mechanisms that rely on direct API access. - -This constraint arises from fundamental security requirements: - -- **Network Isolation**: No VPC peering, Transit Gateway, or VPN connections between Regional and Management VPCs -- **Account Separation**: Management Clusters may reside in different AWS accounts with independent governance -- **Zero Trust Architecture**: No implicit trust relationships based on network topology -- **Attack Surface Minimization**: Eliminating direct API exposure reduces potential security vulnerabilities - -## Alternatives Explored - -### Alternative 1: Direct REST API Push - -**Approach**: Regional Cluster makes HTTPS requests to Management Cluster API endpoints - -- Regional Cluster CLM directly calls Management Cluster Kubernetes APIs -- VPC Peering or Transit Gateway for network connectivity -- API Gateway or PrivateLink for secure exposure - -**Assessment**: Violates fundamental security requirement of network isolation. Creates operational coupling and increases attack surface. Requires complex network topology management. - -### Alternative 2: Pull-Based with Polling - -**Approach**: Management Clusters poll Regional Cluster for updates - -- Management Clusters periodically query Regional API for resource changes -- Time-based reconciliation loop (e.g., every 30 seconds) -- Regional API exposed via PrivateLink or VPC Peering - -**Assessment**: Still requires network connectivity. Introduces latency (bounded by poll interval), increases API load, and creates inefficient resource utilization. Not event-driven, leading to delayed propagation. - -### Alternative 3: Message Queue with AWS Services - -**Implementation Options**: - -- **Amazon SQS/SNS**: Queue-based message delivery with topic subscriptions -- **Amazon EventBridge**: Event bus for cross-account event routing -- **AWS IoT Core MQTT**: Publish-subscribe messaging with certificate-based authentication - -**Assessment**: Viable approach that satisfies network isolation requirement. Each has different trade-offs in terms of message delivery semantics, authentication models, and operational complexity. - -## Decision: Maestro with AWS IoT Core MQTT - -**Chosen Approach**: Alternative 3 (Message Queue) implemented via **Maestro** orchestration system using **AWS IoT Core MQTT** as the transport layer. - -**Implementation Rationale**: - -- **Network Isolation**: AWS IoT Core is an internet-accessible service, eliminating need for VPC connectivity between Regional and Management clusters -- **Cross-Account Support**: IoT Core supports cross-account authentication via IAM policies, enabling Management Clusters in different AWS accounts to connect to Regional account IoT endpoint -- **Message Delivery Guarantees**: MQTT QoS 1 (at least once delivery) ensures reliable resource distribution -- **Event-Driven Architecture**: Immediate message delivery upon publication, enabling low-latency cluster operations -- **Proven Technology**: Maestro is used in production by ARO-HCP (Azure Red Hat OpenShift - Hosted Control Planes) - -**Trade-offs**: Introduces dependency on AWS IoT Core availability and requires MQTT expertise for troubleshooting. Certificate management adds operational complexity (manual transfer process). - -## High-Level Architecture - -The following diagram shows all AWS components and their relationships across Regional and Management clusters, providing a complete view of the distributed system topology. - -```mermaid -graph TB - subgraph Regional["Regional AWS Account (123456789012)"] - subgraph RegVPC["VPC: 10.0.0.0/16"] - subgraph RegEKS["EKS Cluster: regional"] - Server["Maestro Server
───────────
Replicas: 2
Ports: 8080(HTTP), 8090(gRPC)
ServiceAccount: maestro-server"] - ASCP_S["AWS Secrets Store
CSI Driver
Mount: /mnt/secrets-store"] - - Server -->|Mounts| ASCP_S - end - - subgraph DBSubnet["Private Subnets (Multi-AZ)"] - RDS["RDS PostgreSQL 16.4
───────────
Instance: db.t4g.micro
Storage: 20GB (encrypted)
Backups: 7 days"] - end - - SG_DB["Security Group
Port 5432
Source: EKS cluster only"] - end - - IoT["AWS IoT Core
───────────
Endpoint: *.iot.us-east-1.amazonaws.com:8883
Auth: X.509 Certificates
Protocol: MQTT TLS"] - - subgraph SM_Reg["AWS Secrets Manager"] - SM_Server["regional-maestro-server-cert
(cert + key)"] - SM_DB["regional-maestro-db-credentials
(host, port, user, pass)"] - SM_Consumers["regional-maestro-consumers
(pre-provisioned metadata)"] - end - - IAM_Server["IAM Role
regional-maestro-server
───────────
Permissions:
• IoT: Connect, Publish, Subscribe
• RDS: Connect
• Secrets: GetSecretValue"] - - Server -->|Pod Identity| IAM_Server - IAM_Server -->|Read| SM_Server - IAM_Server -->|Read| SM_DB - IAM_Server -->|Read| SM_Consumers - ASCP_S -.->|Mounts as files| SM_Server - ASCP_S -.->|Mounts as files| SM_DB - Server -->|Port 5432
SSL/TLS| SG_DB - SG_DB -->|Access| RDS - Server -->|MQTT
Port 8883| IoT - end - - subgraph Mgmt1["Management AWS Account (987654321098)"] - subgraph MgmtVPC1["VPC: 10.1.0.0/16"] - subgraph MgmtEKS1["EKS Cluster: mc01"] - Agent1["Maestro Agent
───────────
Replicas: 1
Consumer: mc01
ServiceAccount: maestro-agent"] - ASCP_A1["AWS Secrets Store
CSI Driver
Mount: /mnt/secrets-store"] - - Agent1 -->|Mounts| ASCP_A1 - end - end - - SM_Agent1["AWS Secrets Manager
mc01-maestro-agent-cert
(manually created)"] - - IAM_Agent1["IAM Role
mc01-maestro-agent
───────────
Permissions:
• IoT: Connect to Regional IoT (cross-account)
• Secrets: GetSecretValue (local)
Trust: pods.eks.amazonaws.com (same account)"] - - Agent1 -->|Pod Identity
Same Account| IAM_Agent1 - IAM_Agent1 -->|Read
Local Secret| SM_Agent1 - ASCP_A1 -.->|Mounts as files| SM_Agent1 - Agent1 -.->|MQTT
Port 8883
Cross-Account IAM| IoT - end - - subgraph Mgmt2["Management AWS Account (234567890123)"] - subgraph MgmtVPC2["VPC: 10.2.0.0/16"] - subgraph MgmtEKS2["EKS Cluster: mc02"] - Agent2["Maestro Agent
───────────
Replicas: 1
Consumer: mc02
ServiceAccount: maestro-agent"] - ASCP_A2["AWS Secrets Store
CSI Driver"] - - Agent2 -->|Mounts| ASCP_A2 - end - end - - SM_Agent2["AWS Secrets Manager
mc02-maestro-agent-cert
(manually created)"] - - IAM_Agent2["IAM Role
mc02-maestro-agent
───────────
Trust: pods.eks.amazonaws.com (same account)"] - - Agent2 -->|Pod Identity
Same Account| IAM_Agent2 - IAM_Agent2 -->|Read
Local Secret| SM_Agent2 - ASCP_A2 -.->|Mounts as files| SM_Agent2 - Agent2 -.->|MQTT
Port 8883
Cross-Account IAM| IoT - end - - style Server fill:#e1f5ff,stroke:#0066cc,stroke-width:2px - style Agent1 fill:#ffe1e1,stroke:#cc0000,stroke-width:2px - style Agent2 fill:#ffe1e1,stroke:#cc0000,stroke-width:2px - style IoT fill:#d4edda,stroke:#28a745,stroke-width:3px - style RDS fill:#e8e8ff,stroke:#6666ff,stroke-width:2px - style SM_Server fill:#fff3cd,stroke:#ffc107,stroke-width:2px - style SM_DB fill:#fff3cd,stroke:#ffc107,stroke-width:2px - style IAM_Server fill:#cce5ff - style IAM_Agent1 fill:#ffcccc - style IAM_Agent2 fill:#ffcccc -``` - -**Key Architectural Points:** - -- **Regional Cluster**: Centralized Maestro Server with RDS state database -- **Management Clusters**: Distributed agents in separate AWS accounts -- **AWS IoT Core**: MQTT broker enabling pub/sub communication (cross-account via IAM permissions) -- **Same-Account Pod Identity**: Each cluster uses Pod Identity with roles in their own account -- **Local Secrets**: Each cluster reads MQTT certificates from its own Secrets Manager -- **Network Isolation**: No direct network path between regional and management clusters - -## Complete Message Flow - -The following sequence diagram illustrates the end-to-end flow showing how ManifestWork resources are distributed from Regional to Management clusters, including initialization, resource creation, and status reporting. - -```mermaid -sequenceDiagram - participant User as Platform Operator - participant API as Regional Cluster
Maestro HTTP API - participant Server as Maestro Server
(Regional) - participant DB as RDS PostgreSQL
(Regional) - participant SM as Secrets Manager
(Regional) - participant IoT as AWS IoT Core
MQTT Broker - participant Agent as Maestro Agent
(Management) - participant K8s as Management Cluster
Kubernetes API - - Note over User,K8s: Initialization Phase - Server->>SM: Read server MQTT cert via ASCP (local) - Server->>DB: Initialize schema, load consumers - Server->>IoT: Connect with X.509 cert - Note over Agent: Agent reads from Management account Secrets Manager - Agent->>Agent: Read agent MQTT cert via ASCP (local) - Agent->>IoT: Connect with X.509 cert (cross-account via IAM) - Agent->>IoT: Subscribe to topic:
sources/{regional_id}/consumers/mc01/sourceevents - - Note over User,K8s: ManifestWork Creation - User->>API: POST /api/maestro/v1/resources
ManifestWork manifest - API->>Server: Create ManifestWork - Server->>DB: Store ManifestWork (status: pending) - DB-->>Server: Stored (ID: abc123) - Server->>Server: Wrap in CloudEvent envelope - Server->>IoT: Publish to topic:
sources/{regional_id}/consumers/mc01/sourceevents - - Note over User,K8s: Message Delivery - IoT->>Agent: Deliver MQTT message - Agent->>Agent: Parse CloudEvent - Agent->>Agent: Extract ManifestWork payload - Agent->>K8s: Apply Kubernetes resources
(Deployment, Service, etc.) - K8s-->>Agent: Resources created - Agent->>Agent: Create AppliedManifestWork
(status: Applied) - - Note over User,K8s: Status Reporting - Agent->>Agent: Wrap status in CloudEvent - Agent->>IoT: Publish to topic:
sources/{regional_id}/consumers/mc01/agentevents - IoT->>Server: Deliver status message - Server->>Server: Parse status CloudEvent - Server->>DB: Update ManifestWork
(status: Applied) - - Note over User,K8s: Status Query - User->>API: GET /api/maestro/v1/resources/abc123 - API->>Server: Get ManifestWork status - Server->>DB: Query status - DB-->>Server: status: Applied - Server-->>API: Return status - API-->>User: HTTP 200 OK
status: Applied - - Note over User,K8s: Health Monitoring - Agent->>K8s: Watch applied resources - K8s-->>Agent: Resource health events - Agent->>IoT: Publish status updates
(periodic heartbeat) - IoT->>Server: Deliver updates - Server->>DB: Update resource status -``` - -**Message Flow Steps:** - -1. **Initialization**: Server and agents connect to IoT Core with X.509 certificates -2. **Subscription**: Agents subscribe to their consumer-specific topics -3. **Publication**: Server publishes ManifestWork wrapped in CloudEvent to agent topic -4. **Application**: Agent receives, parses, and applies Kubernetes resources -5. **Status Update**: Agent reports status back through separate MQTT topic -6. **Persistence**: Server stores all state in RDS for API queries - -## MQTT Topic Structure - -The hierarchical topic organization provides consumer isolation and message routing. Each Management Cluster has dedicated topics that enforce strict authorization boundaries. - -The topic root is scoped by `{regional_id}` — the identifier of the Regional Cluster. This ensures topic namespaces are fully isolated across environments (e.g., ephemeral, integration, production), so multiple Regional Clusters sharing the same AWS IoT Core endpoint cannot interfere with each other's message flows. - -```mermaid -graph TB - Root["MQTT Topic Root
sources/{regional_id}/consumers"] - - Root --> C1["/{consumer-name-1}"] - Root --> C2["/{consumer-name-2}"] - Root --> CN["/{consumer-name-N}"] - - C1 --> SE1["sourceevents
───────────
Direction: Server → Agent
Publisher: Maestro Server
Subscriber: Maestro Agent
QoS: 1 (at least once)
Payload: CloudEvent + ManifestWork"] - - C1 --> AE1["agentevents
───────────
Direction: Agent → Server
Publisher: Maestro Agent
Subscriber: Maestro Server
QoS: 1 (at least once)
Payload: CloudEvent + Status"] - - C2 --> SE2["sourceevents"] - C2 --> AE2["agentevents"] - - CN --> SEN["sourceevents"] - CN --> AEN["agentevents"] - - style Root fill:#d4edda,stroke:#28a745,stroke-width:3px - style SE1 fill:#e1f5ff,stroke:#0066cc,stroke-width:2px - style AE1 fill:#ffe1e1,stroke:#cc0000,stroke-width:2px - style SE2 fill:#e1f5ff,stroke:#0066cc,stroke-width:2px - style AE2 fill:#ffe1e1,stroke:#cc0000,stroke-width:2px - style SEN fill:#e1f5ff,stroke:#0066cc,stroke-width:2px - style AEN fill:#ffe1e1,stroke:#cc0000,stroke-width:2px -``` - -**Topic Examples:** - -- **Server publishes to**: `sources/{regional_id}/consumers/mc01/sourceevents` -- **Agent subscribes to**: `sources/{regional_id}/consumers/mc01/sourceevents` -- **Agent publishes to**: `sources/{regional_id}/consumers/mc01/agentevents` -- **Server subscribes to**: `sources/{regional_id}/consumers/+/agentevents` (wildcard) - -**Topic Security Model:** - -- Each agent has IoT Policy allowing subscribe/receive ONLY on its consumer-specific topic -- Server has IoT Policy allowing publish to all `sourceevents` topics within its `{regional_id}` namespace -- Topic isolation ensures multi-tenant security — agents cannot intercept messages for other clusters -- Wildcard subscriptions (`+`) are restricted to the Maestro Server role only -- Environment isolation — the `{regional_id}` prefix prevents cross-environment message leakage when multiple environments (e.g., ephemeral, integration) share the same AWS IoT Core endpoint -- Client IDs for the Maestro Server are set to the pod name, preventing connection collisions across replicas - -## Implementation Design - -### Maestro Server (Regional Cluster) - -- Runs in Regional Cluster with HTTP (8080) and gRPC (8090) APIs -- Connects to AWS IoT Core using X.509 certificate authentication; each pod uses its pod name as the MQTT client ID to avoid connection collisions across replicas -- Stores resource state in dedicated RDS PostgreSQL database -- Publishes ManifestWork resources to consumer-specific MQTT topics scoped under `sources/{regional_id}/consumers/` -- Subscribes to status update topics from all agents - -### Maestro Agent (Management Cluster) - -- Runs in each Management Cluster (single replica per cluster) -- Connects to Regional AWS account IoT Core via cross-account IAM permissions -- Subscribes to consumer-specific topic: `sources/{regional_id}/consumers/{cluster-id}/sourceevents` -- Applies received Kubernetes resources to local Management Cluster API -- Reports status back via: `sources/{regional_id}/consumers/{cluster-id}/agentevents` -- See [MQTT Topic Structure](#mqtt-topic-structure) section for detailed topic organization - -### Authentication & Security - -The authentication model leverages AWS-native mechanisms for both same-account secret access and cross-account IoT connectivity. - -#### IAM Roles and Cross-Account Setup - -The following diagram shows detailed IAM role configuration, Pod Identity associations, and cross-account authentication flows. - -```mermaid -graph TB - subgraph "Regional AWS Account (123456789012)" - subgraph "Regional EKS Cluster (regional)" - MS[Maestro Server
ServiceAccount] - ASCP_RC[AWS Secrets Store
CSI Driver] - - MS -->|Pod Identity| MSRole[IAM Role:
regional-maestro-server] - MS -->|Volume Mount| ASCP_RC - end - - subgraph "AWS Secrets Manager (Regional Account)" - SecretDB[(Secret:
regional-maestro-db-credentials)] - SecretMQTTServer[(Secret:
regional-maestro-server-cert)] - end - - subgraph "AWS IoT Core (Regional Account)" - IoTThing1[IoT Thing:
regional-maestro-server] - IoTThing2[IoT Thing:
mc01-maestro-agent] - IoTBroker[MQTT Broker
Port 8883] - - IoTThing1 -->|publishes to| IoTBroker - IoTThing2 -->|subscribes to| IoTBroker - end - - RDS[(RDS PostgreSQL
Maestro State)] - - MSRole -->|GetSecretValue| SecretDB - MSRole -->|GetSecretValue| SecretMQTTServer - ASCP_RC -->|Mounts via
Pod Identity| SecretDB - ASCP_RC -->|Mounts via
Pod Identity| SecretMQTTServer - MSRole -->|Connect| IoTBroker - MSRole -->|Read/Write| RDS - end - - subgraph "Management AWS Account (987654321098)" - subgraph "Management EKS Cluster (mc01)" - MA[Maestro Agent
ServiceAccount] - ASCP_MC[AWS Secrets Store
CSI Driver] - - MA -->|Pod Identity| MARole[IAM Role:
mc01-maestro-agent
SAME ACCOUNT] - MA -->|Volume Mount| ASCP_MC - end - - subgraph "AWS Secrets Manager (Management Account)" - SecretMQTTAgentLocal[(Secret:
mc01-maestro-agent-cert
Manually Created)] - end - - MARole -->|GetSecretValue
Same Account| SecretMQTTAgentLocal - ASCP_MC -->|Mounts via
Pod Identity| SecretMQTTAgentLocal - MA -.->|Connect via MQTT Certificate
Cross-Account IAM Permissions| IoTBroker - end - - style MSRole fill:#e1f5ff - style MARole fill:#ffe1e1 - style SecretMQTTAgentLocal fill:#fff3cd - style IoTBroker fill:#d4edda - style ASCP_RC fill:#e8f4f8 - style ASCP_MC fill:#e8f4f8 -``` - -#### IAM Trust Relationships - -This detailed flow shows how Pod Identity enables same-account secret access and cross-account IoT authentication. - -```mermaid -sequenceDiagram - participant MC as Management Cluster Pod
(Account 987654321098) - participant ASCP as ASCP CSI Driver - participant STS as AWS STS - participant SM as Secrets Manager
(Account 987654321098) - participant IoT as IoT Core
(Account 123456789012) - - Note over MC,SM: Pod Identity Same-Account Flow - - MC->>ASCP: Mount secret volume - ASCP->>STS: AssumeRole(mc01-maestro-agent)
Source Account: 987654321098 - - STS->>STS: Verify Pod Identity Token - - Note over STS: Trust Policy allows:
- Service: pods.eks.amazonaws.com
- Same Account Only - - STS-->>ASCP: Temporary credentials for role - - ASCP->>SM: GetSecretValue(mc01-maestro-agent-cert)
Same Account - - Note over SM: No resource policy needed:
Same-account IAM permissions apply - - SM-->>ASCP: Return secret (MQTT certificate + key) - ASCP-->>MC: Mount secret as files - - MC->>IoT: Connect to Regional IoT Core
using IAM permissions and MQTT certificate - IoT-->>MC: Authenticated MQTT connection (cross-account via IAM) -``` - -#### Secret Flow Architecture - -This diagram shows how secrets flow from Terraform creation through manual transfer to pod consumption. - -```mermaid -graph LR - subgraph "Regional Cluster (Regional Account)" - TF1[Terraform] -->|Creates| IoTCerts[IoT Certificates] - IoTCerts -->|Stores in| SM1[Secrets Manager
Regional Account
server-mqtt-cert] - - ASCP_RC1[ASCP CSI Driver] -->|Mounts from
Same Account| SM1 - ASCP_RC1 -->|Mounts as| FilesRC[Files in Pod] - - MS1[Maestro Server] -->|Reads| FilesRC - MS1 -->|Publishes to| MQTT[IoT Core MQTT
Regional Account] - end - - subgraph "Management Cluster (Management Account)" - SM2[Secrets Manager
Management Account
agent-mqtt-cert
Manually Created] - - ASCP_MC1[ASCP CSI Driver] -->|Mounts from
Same Account| SM2 - ASCP_MC1 -->|Mounts as| FilesMC[Files in Pod] - - MA1[Maestro Agent] -->|Reads| FilesMC - MA1 -.->|Subscribes to
Cross-Account IAM| MQTT - end - - TF1 -.->|Manual Transfer
Certificate Data| SM2 - - style SM1 fill:#fff3cd - style SM2 fill:#fff3cd - style ASCP_MC1 fill:#ffe1e1 - style ASCP_RC1 fill:#e8f4f8 - style MQTT fill:#d4edda -``` - -#### Key IAM Components - -**Regional Account (123456789012)** - -**IAM Role:** `regional-maestro-server` - -- Access to IoT Core (connect, publish, subscribe) -- Access to RDS (connect, read, write) -- Access to Secrets Manager (GetSecretValue) -- Mounted via ASCP CSI Driver - -**Resources:** - -- AWS IoT Core Things, Certificates, and Policies (for server + all agents) -- AWS Secrets Manager secrets (server cert, DB credentials, consumer registrations) -- RDS PostgreSQL database -- EKS cluster running Maestro Server - -**Trust Policy (Same-Account):** - -```json -{ - "Statement": [ - { - "Principal": { "Service": "pods.eks.amazonaws.com" }, - "Action": ["sts:AssumeRole", "sts:TagSession"] - } - ] -} -``` - -**Note:** Agent certificates are created in Regional IoT Core but stored in Management account Secrets Manager via manual transfer. - -**Management Account (987654321098)** - -**IAM Role:** `mc01-maestro-agent` - -- Created in **Management Account** (same account as cluster) -- Accesses local Secrets Manager (same-account) -- Has cross-account IoT permissions to Regional IoT Core - -**Resources:** - -- EKS cluster running Maestro Agent -- AWS Secrets Manager secret (manually created with transferred certificate data) -- Pod Identity association (same-account role) - -**Pod Identity Association:** - -```hcl -# In Management Cluster Terraform -resource "aws_eks_pod_identity_association" "maestro_agent" { - cluster_name = "mc01" - namespace = "maestro" - service_account = "maestro-agent" - role_arn = "arn:aws:iam::987654321098:role/mc01-maestro-agent" - # ↑ Role is in SAME account as management cluster -} -``` - -**Agent IAM Permissions:** - -```json -{ - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "secretsmanager:GetSecretValue", - "secretsmanager:DescribeSecret" - ], - "Resource": "arn:aws:secretsmanager:*:987654321098:secret:mc01-maestro-agent-cert*" - }, - { - "Effect": "Allow", - "Action": ["iot:Connect", "iot:Subscribe", "iot:Receive", "iot:Publish"], - "Resource": [ - "arn:aws:iot:us-east-1:123456789012:client/*", - "arn:aws:iot:us-east-1:123456789012:topic/sources/{regional_id}/consumers/mc01/*", - "arn:aws:iot:us-east-1:123456789012:topicfilter/sources/{regional_id}/consumers/mc01/*" - ] - } - ] -} -``` - -**Note:** The agent reads secrets from its own account, but has IAM permissions to access IoT Core in the Regional account. - -#### Authentication Flow Summary - -1. **Regional Cluster (Same Account)** - - Maestro Server uses Pod Identity → assumes regional account role (same account) - - ASCP CSI Driver mounts secrets from regional account Secrets Manager (same account) - - Maestro Server reads mounted files → connects to IoT Core (same account) - -2. **Management Cluster (Same Account for Secrets, Cross-Account for IoT)** - - Maestro Agent uses Pod Identity → assumes management account role (same account) - - ASCP CSI Driver mounts secrets from management account Secrets Manager (same account) - - Agent reads mounted certificate files → connects to Regional IoT Core (cross-account via IAM permissions) - -#### Why This Design? - -**Centralized Certificate Creation:** - -- All IoT certificates created in one place (regional account IoT Core) -- Certificate data manually transferred to management clusters (not automated) - -**Security Benefits:** - -- Explicit IAM permissions for cross-account IoT access -- Secrets never in Terraform state (manual transfer process) -- Secrets never transmitted over network (mounted via CSI driver from local account) -- Least privilege access (each role has minimal permissions) -- Account sovereignty (each cluster owns its own secrets) - -**Operational Simplicity:** - -- No cross-account secret access policies needed -- No cross-account IAM trust policies needed -- Each cluster uses standard same-account Pod Identity -- Simple IAM permissions for IoT access (resource-based authorization) -- Clear operational boundaries between regional and management teams - -### State Management - -- **CLM as Source of Truth**: All cluster state authoritative in CLM RDS database -- **Maestro as Distribution Cache**: Maestro RDS database caches published resources for performance -- **Rebuild Capability**: Maestro cache can be fully reconstructed from CLM if data loss occurs - -## Deployment Workflow - -The complete deployment sequence shows how Regional and Management infrastructure is provisioned, including the manual certificate transfer process between accounts. - -```mermaid -sequenceDiagram - participant RegOp as Regional
Operator - participant RegTF as Regional
Terraform - participant AWS_R as Regional
AWS Account - participant Transfer as Secure
Transfer Channel - participant MgmtOp as Management
Operator - participant MgmtCLI as AWS CLI
(Management) - participant MgmtTF as Management
Terraform - participant AWS_M as Management
AWS Account - - Note over RegOp,AWS_M: Phase 1: Regional Infrastructure - RegOp->>RegTF: terraform apply
maestro-infrastructure - RegTF->>AWS_R: Create IoT Things + Certs - RegTF->>AWS_R: Create RDS PostgreSQL - RegTF->>AWS_R: Create Secrets Manager secrets - RegTF->>AWS_R: Create Server IAM role + Pod Identity - AWS_R-->>RegTF: Resources created - RegTF-->>RegOp: Apply complete - - Note over RegOp,AWS_M: Phase 2: Certificate Extraction - RegOp->>RegTF: terraform output -json
maestro_agent_certificates - RegTF-->>RegOp: {"mc01": {
"certificateArn": "...",
"certificatePem": "...",
"privateKey": "...",
"endpoint": "..."}} - RegOp->>RegOp: jq '["mc01"]' > cert.json - RegOp->>RegOp: Encrypt cert.json - - Note over RegOp,AWS_M: Phase 3: Secure Transfer - RegOp->>Transfer: Transfer encrypted cert.json
(GPG / AWS Secrets Manager /
HashiCorp Vault / etc.) - Transfer->>MgmtOp: Receive encrypted file - MgmtOp->>MgmtOp: Decrypt cert.json - - Note over RegOp,AWS_M: Phase 4: Management Cluster Secret - MgmtOp->>MgmtCLI: aws secretsmanager create-secret
--name mc01-maestro-agent-cert
--secret-string file://cert.json - MgmtCLI->>AWS_M: Create secret in Secrets Manager - AWS_M-->>MgmtCLI: Secret created - MgmtOp->>MgmtOp: shred -u cert.json - - Note over RegOp,AWS_M: Phase 5: Management Cluster IAM - MgmtOp->>MgmtTF: terraform apply
maestro-agent - MgmtTF->>AWS_M: Create Agent IAM role (same account) - MgmtTF->>AWS_M: Add IoT permissions (to Regional IoT) - MgmtTF->>AWS_M: Create Pod Identity association - AWS_M-->>MgmtTF: Resources created - MgmtTF-->>MgmtOp: Apply complete - - Note over RegOp,AWS_M: Phase 6: Helm Deployments - RegOp->>RegTF: terraform output
maestro_configuration_summary - RegTF-->>RegOp: Helm values (role ARN, secrets, endpoint) - RegOp->>AWS_R: helm install maestro-server
--set aws.podIdentity.roleArn=...
--set ascp.mqttCertSecretName=... - AWS_R-->>RegOp: Server deployed - - MgmtOp->>MgmtTF: terraform output helm_values - MgmtTF-->>MgmtOp: Helm values - MgmtOp->>AWS_M: helm install maestro-agent
--set maestro.consumerName=mc01
--set broker.endpoint=... - AWS_M-->>MgmtOp: Agent deployed - - Note over RegOp,AWS_M: Phase 7: Verification - RegOp->>AWS_R: kubectl logs maestro-server - AWS_R-->>RegOp: Connected to IoT Core ✓
DB connection established ✓ - MgmtOp->>AWS_M: kubectl logs maestro-agent - AWS_M-->>MgmtOp: Connected to IoT Core ✓
Subscribed to topic ✓ -``` - -**Why Manual Transfer?** - -- Keeps sensitive certificate data OUT of Terraform state -- No automated secrets distribution needed between accounts -- Explicit, auditable security process -- Follows principle of least privilege -- Simplifies secret rotation workflow -- Each cluster maintains sovereignty over its own secrets - -## Certificate Transfer Process - -The following diagram provides a detailed view of secure certificate transfer between Regional and Management operators. - -```mermaid -graph LR - subgraph Regional["Regional AWS Account
(123456789012)"] - TF["Terraform Apply
maestro-infrastructure"] - IoT["AWS IoT Core
Create Certificate"] - Out["Terraform Output
maestro_agent_certificates
(SENSITIVE)"] - - TF -->|Creates| IoT - IoT -->|Certificate Data| Out - end - - subgraph Extract["Regional Operator Actions"] - Cmd1["terraform output -json
maestro_agent_certificates"] - JQ["jq '.["mc01"]'
> cert.json"] - Encrypt["gpg --encrypt
--recipient management-op
cert.json"] - - Cmd1 --> JQ - JQ --> Encrypt - end - - subgraph Transfer["Secure Transfer"] - Channel["Encrypted Channel
───────────
Options:
• GPG-encrypted email
• AWS Secrets Manager cross-account
• HashiCorp Vault transit
• Secure file share (Box, OneDrive)
• Encrypted S3 bucket"] - end - - subgraph Receive["Management Operator Actions"] - Decrypt["gpg --decrypt
cert.json.gpg
> cert.json"] - Verify["jq . cert.json
(validate JSON)"] - CLI["aws secretsmanager
create-secret
--secret-string file://cert.json"] - Shred["shred -u cert.json
(secure delete)"] - - Decrypt --> Verify - Verify --> CLI - CLI --> Shred - end - - subgraph Management["Management AWS Account
(987654321098)"] - SM["AWS Secrets Manager
mc01-maestro-agent-cert"] - TF2["Terraform Apply
maestro-agent
(references secret)"] - - SM --> TF2 - end - - Out --> Cmd1 - Encrypt --> Channel - Channel --> Decrypt - CLI --> SM - - style Out fill:#fff3cd,stroke:#ffc107,stroke-width:2px - style Channel fill:#ffe1e1,stroke:#cc0000,stroke-width:3px - style SM fill:#d4edda,stroke:#28a745,stroke-width:2px - style Encrypt fill:#ffcccc - style Decrypt fill:#ccffcc -``` - -**Certificate Content Structure:** - -```json -{ - "certificateArn": "arn:aws:iot:us-east-1:123456789012:cert/abc...", - "certificatePem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----", - "privateKey": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----", - "endpoint": "abc123.iot.us-east-1.amazonaws.com", - "port": 8883, - "consumerName": "mc01" -} -``` - -This structure contains all necessary information for the Management Cluster agent to authenticate with Regional IoT Core. - -## Network Topology - -The following diagram shows the physical network layout, VPC isolation, and connectivity patterns across Regional and Management AWS accounts. - -```mermaid -graph TB - subgraph Internet["Internet / AWS Global Services"] - IoT_Global["AWS IoT Core
Global Service
Endpoint: *.iot.{region}.amazonaws.com:8883"] - end - - subgraph Regional_Account["Regional AWS Account (123456789012)
Region: us-east-1"] - subgraph Regional_VPC["VPC: 10.0.0.0/16"] - subgraph AZ1_R["Availability Zone 1"] - Pub1_R["Public Subnet
10.0.192.0/22
NAT Gateway"] - Priv1_R["Private Subnet
10.0.0.0/18
EKS Nodes"] - DB1_R["RDS Subnet
10.0.201.0/24"] - end - - subgraph AZ2_R["Availability Zone 2"] - Pub2_R["Public Subnet
10.0.196.0/22
NAT Gateway"] - Priv2_R["Private Subnet
10.0.64.0/18
EKS Nodes"] - DB2_R["RDS Subnet
10.0.202.0/24"] - end - - IGW_R["Internet Gateway"] - ALB_R["Application Load Balancer
(Optional - Admin Access)"] - - Priv1_R -->|Route 0.0.0.0/0| Pub1_R - Priv2_R -->|Route 0.0.0.0/0| Pub2_R - Pub1_R --> IGW_R - Pub2_R --> IGW_R - - RDS_R["RDS PostgreSQL
Multi-AZ
Security Group:
Port 5432 from EKS only"] - - DB1_R -.->|Primary| RDS_R - DB2_R -.->|Standby| RDS_R - - Priv1_R -->|Private connection| RDS_R - Priv2_R -->|Private connection| RDS_R - end - end - - subgraph Mgmt_Account["Management AWS Account (987654321098)
Region: us-east-1"] - subgraph Mgmt_VPC["VPC: 10.1.0.0/16"] - subgraph AZ1_M["Availability Zone 1"] - Pub1_M["Public Subnet
10.1.101.0/24
NAT Gateway"] - Priv1_M["Private Subnet
10.1.1.0/24
EKS Nodes
Maestro Agent"] - end - - subgraph AZ2_M["Availability Zone 2"] - Pub2_M["Public Subnet
10.1.102.0/24
NAT Gateway"] - Priv2_M["Private Subnet
10.1.2.0/24
EKS Nodes"] - end - - IGW_M["Internet Gateway"] - - Priv1_M -->|Route 0.0.0.0/0| Pub1_M - Priv2_M -->|Route 0.0.0.0/0| Pub2_M - Pub1_M --> IGW_M - Pub2_M --> IGW_M - end - end - - IGW_R -->|HTTPS/TLS
Port 8883| IoT_Global - IGW_M -->|HTTPS/TLS
Port 8883| IoT_Global - - NoPath["❌ NO DIRECT NETWORK PATH
between Regional VPC and Management VPC"] - - Regional_VPC -.->|No peering
No transit gateway
No VPN| NoPath - NoPath -.->|No peering
No transit gateway
No VPN| Mgmt_VPC - - style IoT_Global fill:#d4edda,stroke:#28a745,stroke-width:3px - style RDS_R fill:#e8e8ff,stroke:#6666ff,stroke-width:2px - style NoPath fill:#ffe1e1,stroke:#cc0000,stroke-width:3px - style Priv1_R fill:#e1f5ff - style Priv2_R fill:#e1f5ff - style Priv1_M fill:#ffe1e1 - style Priv2_M fill:#ffe1e1 -``` - -**Key Network Characteristics:** - -- **Complete VPC Isolation**: No VPC peering, no Transit Gateway, no VPN between Regional and Management VPCs -- **Internet Gateway Only**: All clusters access IoT Core through NAT Gateway → Internet Gateway → AWS IoT endpoint -- **Private EKS Clusters**: Control planes have private endpoints only (no public access) -- **RDS Isolation**: Database accessible only from Regional EKS cluster security group -- **Multi-AZ Deployment**: High availability across multiple availability zones -- **Security**: All traffic encrypted in transit (TLS 1.2+) -- **No Direct Paths**: Regional and Management clusters communicate ONLY through AWS IoT Core -- **NAT Gateway Redundancy**: Each availability zone has its own NAT Gateway for resilience - -This network topology demonstrates the fundamental isolation principle: Regional and Management clusters have no direct network connectivity, relying entirely on AWS IoT Core as the message broker. - -## Benefits - -**Network Security & Isolation**: - -- Complete network isolation between Regional and Management Clusters (no VPC peering, Transit Gateway, or VPN) -- Management Clusters in separate AWS accounts maintain full autonomy -- Reduced attack surface - no direct API exposure between clusters - -**Operational Flexibility**: - -- Management Clusters can be provisioned dynamically in any AWS account -- No network topology changes required when adding new Management Clusters -- Simplified disaster recovery - Maestro state rebuilds from CLM - -**Event-Driven Performance**: - -- Immediate resource propagation (milliseconds vs. seconds with polling) -- Efficient resource utilization - no continuous polling overhead -- Reliable delivery with MQTT QoS 1 guarantees - -**Strategic Alignment**: - -- Leverages proven Maestro technology from ARO-HCP production deployments -- Aligns with AWS-native architecture (IoT Core, IAM, Secrets Manager) -- Foundation for future event-driven workflows beyond resource distribution -- Compatible with multi-region and multi-cloud expansion strategies - -**Scalability & Performance**: - -- **Horizontal Scaling**: Maestro Server runs with multiple replicas (2+) for high availability -- **Connection Pooling**: Each agent maintains a single persistent MQTT connection (no polling overhead) -- **Message Batching**: CloudEvent envelope supports batch operations for efficiency -- **Topic-Based Routing**: AWS IoT Core handles message routing at the broker level -- **QoS Guarantees**: MQTT QoS 1 ensures at-least-once delivery with minimal latency - -**Operational Advantages**: - -- **Observability**: AWS IoT Core provides CloudWatch metrics for connection health, message throughput, and error rates -- **Auditability**: All MQTT connections and message deliveries logged in CloudTrail -- **Certificate Rotation**: Manual transfer process enables controlled, auditable certificate lifecycle management -- **Disaster Recovery**: Regional Maestro RDS can be rebuilt from CLM source of truth -- **Testing & Validation**: Easy to test with mock MQTT clients for integration testing - -## Operational Considerations - -### Monitoring & Alerting - -**CloudWatch Metrics**: - -- `AWS/IoT/Connect.Success` - Monitor successful MQTT connections -- `AWS/IoT/PublishIn.Success` - Track message publication rates -- `AWS/IoT/Subscribe.Success` - Verify agent subscriptions -- Custom metrics from Maestro Server/Agent for ManifestWork processing - -**Health Checks**: - -- Maestro Server: HTTP `/healthz` endpoint on port 8080 -- Maestro Agent: Kubernetes liveness/readiness probes -- RDS: Automated CloudWatch alarms for connection count, CPU, and storage - -### Troubleshooting Guide - -**Agent Cannot Connect to IoT Core**: - -1. Verify IAM role has cross-account IoT permissions -2. Check certificate validity: `openssl x509 -in cert.pem -text -noout` -3. Validate IoT endpoint: `nslookup .iot..amazonaws.com` -4. Review agent logs for authentication errors -5. Confirm NAT Gateway and Internet Gateway routing - -**Messages Not Delivered**: - -1. Check MQTT topic subscriptions match publication topics -2. Verify IoT Policy allows subscribe/publish on correct topics -3. Review CloudWatch Logs for IoT Core rule errors -4. Confirm QoS level matches (should be QoS 1) -5. Check Maestro Server database connection to RDS - -**Certificate Rotation**: - -1. Create new certificate in Regional IoT Core -2. Extract certificate using Terraform output -3. Transfer encrypted certificate to Management operator -4. Update Management Secrets Manager secret -5. Restart Maestro Agent pods to reload secret -6. Deactivate old certificate in IoT Core (after 24-hour grace period) -7. Delete old certificate - -### Performance Tuning - -- **RDS Instance Sizing**: Start with `db.t4g.micro`, scale to `db.t4g.medium` for 50+ Management Clusters -- **Connection Limits**: AWS IoT Core supports 100,000 concurrent connections per account (adjust quotas if needed) -- **Message Throughput**: Default IoT Core message throughput is sufficient for 1000s of ManifestWork operations/minute -- **Network Bandwidth**: NAT Gateway bandwidth auto-scales; monitor CloudWatch metrics for saturation - -### Cost Optimization - -- **IoT Core Pricing**: Pay per message published/delivered (~$1 per million messages) -- **RDS Costs**: Use reserved instances for production Regional clusters (40-60% savings) -- **Secrets Manager**: $0.40/secret/month - negligible for typical deployments -- **Data Transfer**: NAT Gateway data transfer charges apply ($0.045/GB outbound) - -## Related Documentation - -### Terraform Infrastructure Modules - -- **[maestro-infrastructure](../../terraform/modules/maestro-infrastructure/)** - Regional cluster Maestro server infrastructure - - IoT Core provisioning (Things, Certificates, Policies) - - RDS PostgreSQL database - - Secrets Manager configuration - - Server IAM roles and Pod Identity associations - -- **[maestro-agent](../../terraform/modules/maestro-agent/)** - Management cluster Maestro agent infrastructure - - Agent IAM roles with cross-account IoT permissions - - Pod Identity associations (same-account) - - Helm chart value generation - -### Design Decisions - -- **[001-fully-private-eks-bootstrap.md](./001-fully-private-eks-bootstrap.md)** - ECS-based bootstrap strategy for fully private EKS clusters -- **[maestro-agent-iot-provisioning.md](./maestro-agent-iot-provisioning.md)** - Detailed IoT Core provisioning and certificate management - -### External References - -- **[Maestro Project](https://github.com/openshift-online/maestro)** - Upstream Maestro orchestration system -- **[ARO-HCP](https://github.com/Azure/ARO-HCP)** - Azure Red Hat OpenShift HCP implementation using Maestro -- **[AWS IoT Core MQTT](https://docs.aws.amazon.com/iot/latest/developerguide/mqtt.html)** - AWS IoT Core MQTT protocol documentation -- **[AWS Secrets Store CSI Driver](https://docs.aws.amazon.com/secretsmanager/latest/userguide/integrating_csi_driver.html)** - ASCP integration with EKS -- **[EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html)** - AWS EKS Pod Identity documentation - -### Configuration Files - -- **[argocd/config/regional-cluster/maestro/](../../argocd/config/regional-cluster/maestro/)** - Maestro Server Helm chart configurations -- **[argocd/config/management-cluster/maestro/](../../argocd/config/management-cluster/maestro/)** - Maestro Agent Helm chart configurations - ---- - -**Decision Date**: January 30, 2026 -**Decision Maker**: RRP Team -**Review Date**: June 30, 2026 (6-month review cycle) diff --git a/docs/design/monitoring-platform.md b/docs/design/monitoring-platform.md index 3fc70d20c..eb53676ee 100644 --- a/docs/design/monitoring-platform.md +++ b/docs/design/monitoring-platform.md @@ -36,7 +36,7 @@ graph TB subgraph RC["Regional Cluster"] RC_KSM["kube-state-metrics"] RC_NE["node-exporter"] - RC_YACE["YACE
EKS, IoT, RDS, ALB,
API GW, DynamoDB,
AmazonMQ, ACM"] + RC_YACE["YACE
EKS, RDS, ALB,
API GW, DynamoDB, ACM"] RC_SM["ServiceMonitors
app metrics"] RC_PROM["Prometheus (HA)"] RECEIVE["Thanos Receive
router + ingesters"] @@ -109,16 +109,15 @@ YACE polls AWS CloudWatch APIs and exposes metrics in Prometheus format. Both cl **Regional Cluster** scrapes: -| AWS Namespace | Metrics | Purpose | -| ------------------------ | -------------------------------------------------------------------------------------------------- | ----------------------------- | -| `AWS/EKS` | apiserver_storage_size_bytes, scheduler_pending_pods, scheduler_schedule_attempts_total | EKS control plane health | -| `AWS/IoT` | Connect.Success, Connect.AuthError, PublishIn.Success, PublishOut.Success | Maestro MQTT broker | -| `AWS/ApiGateway` | Count, Latency, 4XX/5XXError, IntegrationLatency | Platform + RHOBS API Gateways | -| `AWS/RDS` | CPUUtilization, FreeableMemory, ReadLatency, WriteLatency, BurstBalance, DatabaseConnections, IOPS | CLM + Maestro databases | -| `AWS/ApplicationELB` | RequestCount, TargetResponseTime, HealthyHostCount, HTTPCode counts | API load balancer | -| `AWS/DynamoDB` | ConsumedRead/WriteCapacityUnits, UserErrors, ThrottledRequests, SuccessfulRequestLatency | Authorization tables | -| `AWS/AmazonMQ` | MessageCount, MessageUnacknowledgedCount, ConsumerCount, QueueCount, NetworkIn/Out | HyperFleet message broker | -| `AWS/CertificateManager` | DaysToExpiry | API certificate lifecycle | +| AWS Namespace | Metrics | Purpose | +| ----------------------------- | -------------------------------------------------------------------------------------------------- | ----------------------------- | +| `AWS/EKS` | apiserver_storage_size_bytes, scheduler_pending_pods, scheduler_schedule_attempts_total | EKS control plane health | +| `AWS/DynamoDB` (kube-applier) | ConsumedRead/WriteCapacityUnits, ReturnedRecordsCount, ThrottledRequests | kube-applier desire tables | +| `AWS/ApiGateway` | Count, Latency, 4XX/5XXError, IntegrationLatency | Platform + RHOBS API Gateways | +| `AWS/RDS` | CPUUtilization, FreeableMemory, ReadLatency, WriteLatency, BurstBalance, DatabaseConnections, IOPS | hyperfleet-db | +| `AWS/ApplicationELB` | RequestCount, TargetResponseTime, HealthyHostCount, HTTPCode counts | API load balancer | +| `AWS/DynamoDB` | ConsumedRead/WriteCapacityUnits, UserErrors, ThrottledRequests, SuccessfulRequestLatency | Authorization tables | +| `AWS/CertificateManager` | DaysToExpiry | API certificate lifecycle | **Management Cluster** scrapes: @@ -158,7 +157,7 @@ Grafana on the RC queries Thanos Query Frontend for a unified view of all cluste | RDS | RC | CPU, burst balance, connections, IOPS, storage (CW) | | ALB | RC | Request count, response time, healthy hosts (CW) | | DynamoDB | RC | Read/write capacity, latency, throttled requests (CW) | -| Platform Services | RC | IoT/MQTT, AmazonMQ, ACM certificate expiry (CW) | +| Platform Services | RC | ACM certificate expiry (CW) | | HCP Health | RC + MC | Hosted control plane status | | ArgoCD Application Overview | RC | Application sync status, health | | ArgoCD Notifications Overview | RC | Notification delivery status | diff --git a/docs/design/rate-limiting-architecture.md b/docs/design/rate-limiting-architecture.md index f3187c357..0ce144490 100644 --- a/docs/design/rate-limiting-architecture.md +++ b/docs/design/rate-limiting-architecture.md @@ -672,6 +672,8 @@ When the ConfigMap changes, the hash annotation changes, triggering a rolling re - SigV4 identity headers (`X-Caller-Account`, `X-Caller-Arn`) are injected by API Gateway after authentication and cannot be forged by customers - Fail-open behavior on Redis failure is a deliberate choice — matches ROSA HCP v1's Limitador fail-open pattern - API GW throttle fail-closes by default (returns 429) +- **Encryption in transit**: TLS enabled on ElastiCache Valkey (`transit_encryption_enabled = true`). The Go client connects with `tls.Config{MinVersion: tls.VersionTLS12}`. TLS overhead is negligible (<1ms per request) relative to the 50ms fail-open timeout. Required by FedRAMP SC-8 for all network communications. +- **Encryption at rest**: Enabled with a dedicated customer-managed KMS key (`at_rest_encryption_enabled = true`, `aws_kms_key.elasticache`), consistent with RDS and Amazon MQ patterns in this repo. Zero performance cost. Required by FedRAMP SC-13. The data stored is purely ephemeral GCRA counters (e.g., `rl:account:method:path → count`) with auto-TTL — no PII, credentials, or customer data — but encryption is applied uniformly across all data stores as a compliance baseline. ### Performance diff --git a/docs/design/regional-account-minting.md b/docs/design/regional-account-minting.md index 3b30071a2..2a6e1cb10 100644 --- a/docs/design/regional-account-minting.md +++ b/docs/design/regional-account-minting.md @@ -161,7 +161,7 @@ sequenceDiagram ### Performance -- Care should be taken that any pipelines that can run in parallel during new-region creation can be. Our goal is to create a new region within an hour. For example, we can parallelize RC and MC cluster creation using a fan-out pattern, and then come back after those jobs are completed to register the MCs as consumers within the RC's Maestro instance. +- Care should be taken that any pipelines that can run in parallel during new-region creation can be. Our goal is to create a new region within an hour. For example, we can parallelize RC and MC cluster creation using a fan-out pattern, and then come back after those jobs are completed to register the MCs with the hyperfleet-operator on the RC. ### Cost diff --git a/docs/design/regional-control-plane-architecture.md b/docs/design/regional-control-plane-architecture.md new file mode 100644 index 000000000..3d7f0ac11 --- /dev/null +++ b/docs/design/regional-control-plane-architecture.md @@ -0,0 +1,112 @@ +# Regional Control Plane Architecture + +**Last Updated Date**: 2026-07-31 + +## Summary + +The regional control plane manages ROSA HCP cluster lifecycles using Kubernetes-style reconciliation loops (controller-runtime) backed by PostgreSQL instead of etcd. This gives the team the reconciliation model it has deep expertise in while gaining the PITR, scalability, and queryability of a real database. The implementation consists of `hyperfleet-operator` (the controllers) and `hyperfleet-db` (a PostgreSQL-backed controller-runtime library). + +## Context + +- **Problem Statement**: The platform needs cluster lifecycle management that scales beyond etcd's ~8 GB hard ceiling and supports querying fleet state (e.g. "all clusters in degraded state"). The previous architecture (CLM) depended on another team's framework for lifecycle primitives, which constrained development velocity. +- **Constraints**: + - Team must own the full lifecycle stack to move at its own pace + - Must integrate with the existing Regional Cluster (EKS) and Management Cluster topology + - Must support the platform-api as a stateless REST frontend +- **Assumptions**: + - PostgreSQL (via RDS/Aurora) is available in all target AWS regions + - controller-runtime's interfaces (`Manager`, `Client`, `Cache`) are stable and sufficient for the reconciliation model + +## Design + +### Components + +```mermaid +graph TD + Customer["Customer"] -->|SigV4| APIGW["API Gateway"] + APIGW --> PlatformAPI["platform-api\n(stateless REST)"] + PlatformAPI -->|hyperfleet-db client| PG["PostgreSQL\n(RDS/Aurora)"] + + Operator["hyperfleet-operator\n(controller-runtime)"] -->|reconcile loop| PG + Operator -->|writes desires| DynamoDB["DynamoDB\n(→ kube-applier → MCs)"] + Compactor["compactor"] -->|tombstone GC| PG + + DynamoDB ~~~ Compactor + + style PG fill:#f0f0f0,stroke:#333 +``` + +**hyperfleet-operator** is a controller-runtime operator running on the Regional Cluster. It reconciles custom resources that model the cluster lifecycle. + +The operator communicates with Management Clusters via DynamoDB desire documents. A desire is a declarative spec for a Kubernetes resource that should exist on an MC. The operator writes desires to DynamoDB; **kube-applier**, running on each MC, watches for desires and applies them to the local Kubernetes API server. Status flows back the same way — kube-applier writes observed state to DynamoDB status tables, and the operator reads it to update its own resource status. + +**hyperfleet-db** is a Go library that implements controller-runtime's `Manager`, `Client`, and `Cache` interfaces against PostgreSQL. It stores all Kubernetes resources in a single `kubernetes_resources` table. The operator and platform-api both use it: + +- **Operator**: uses the full `Manager` (client + cache + watch) for reconciliation +- **platform-api**: uses `Client` directly (no cache needed for stateless request/response) + +This means the operator's CRDs are not stored in etcd — PostgreSQL is the sole state store. + +**compactor** is a separate process that periodically deletes soft-deleted tombstones from PostgreSQL. It runs alongside the operator and advances a compaction horizon to prevent watchers from seeing gaps in the event stream. + +### Why PostgreSQL + +- **Scales beyond etcd**: No 8 GB ceiling. +- **Point-in-time recovery**: RDS/Aurora PITR provides disaster recovery without custom backup tooling. +- **Fleet querying**: SQL queries over cluster state (e.g. degraded clusters, clusters by region, placement utilization) without building a separate reporting layer. +- **Multi-AZ**: RDS synchronous standby provides zero acknowledged-write loss on failover. + +For detailed internals (schema, invariants, watch mechanism, race catalog), see [hyperfleet-db DESIGN.md](../../../rosa-hyperfleet-api/hyperfleet-db/docs/DESIGN.md). + +## Alternatives Considered + +1. **CLM (Cluster Lifecycle Manager)**: A REST-based lifecycle service with adapters, sentinels, and CloudEvents. The CLM pattern used a stateless API server with GORM, a polling sentinel for change detection, CloudEvents for notification, and adapters for reconciliation. This was rejected because: + - **Velocity**: The framework was owned by another team, creating a dependency that constrained the platform team's development pace. + - **Component count**: 5+ components in the reconcile loop (API server, sentinel, message broker, adapters, status reporters) vs. 2 (operator + PostgreSQL). + - **Operational overhead**: More services to deploy, monitor, and debug during incidents. + + For a detailed comparison of reliability and performance characteristics, see [Architecture Comparison](../../../rosa-hyperfleet-api/hyperfleet-db/docs/ARCHITECTURE_COMPARISON.md). + +2. **Standard controller-runtime with etcd**: Using controller-runtime with its default etcd backend. This was rejected because: + - **8 GB hard ceiling**: etcd's storage limit would require sharding to scale beyond a few thousand clusters, adding significant operational complexity. + - **No fleet querying**: etcd supports key-prefix listing but not the rich queries needed for fleet management. + - **No PITR**: etcd snapshots are coarse-grained; RDS PITR provides second-granularity recovery. + +## Design Rationale + +- **Justification**: The team has deep expertise in Kubernetes-style reconciliation (watch, reconcile, requeue). By implementing controller-runtime's storage interfaces against PostgreSQL, the operator retains that programming model while gaining a real database's PITR, scalability beyond etcd's 8 GB ceiling, and SQL queryability over fleet state. The team owns the full stack. +- **Evidence**: Measured write latency of p50=6.3ms / p99=29ms and throughput of 6,132 writes/s with realistic 15-20KB payloads (Aurora I/O Optimized, db.r6g.8xlarge). See [Architecture Comparison](../../../rosa-hyperfleet-api/hyperfleet-db/docs/ARCHITECTURE_COMPARISON.md) for full benchmarks. +- **Comparison**: CLM's advantages (standard REST API, operational familiarity, existing ecosystem) are real but secondary to the team velocity and component reduction goals that motivated the change. + +## Consequences + +### Positive + +- Team owns the full cluster lifecycle stack with no external framework dependencies +- Fewer components to deploy, monitor, and debug (2 vs. 5+) +- Point-in-time recovery via RDS without custom backup infrastructure +- SQL-based fleet querying without a separate reporting layer + +### Negative + +- No direct `kubectl` access to cluster state — state lives in PostgreSQL, not the Kubernetes API server +- hyperfleet-db is a custom library that must be maintained alongside upstream controller-runtime changes + +## Cross-Cutting Concerns + +### Reliability + +- **Resiliency**: RDS Multi-AZ synchronous standby provides zero acknowledged-write loss on failover. A continuous production verifier checks correctness invariants on live data. See [hyperfleet-db DESIGN.md](../../../rosa-hyperfleet-api/hyperfleet-db/docs/DESIGN.md) for the invariant catalog. +- **Observability**: The operator exposes standard controller-runtime metrics. The compactor logs tombstone deletion counts and compaction horizon advances. + +### Performance + +- This is a low write-rate system — cluster lifecycle events (create, update, delete) and underlying status updates are infrequent relative to the throughput ceiling +- Write latency: p50=6.3ms, p99=29ms with realistic 15-20KB payloads (Aurora I/O Optimized, db.r6g.8xlarge) +- Throughput ceiling: 6,132 writes/s with realistic payloads — orders of magnitude above expected load +- No-op suppression: content-equal writes consume no sequence, version bump, or watch event + +### Cost + +- RDS/Aurora instance per region (sized by cluster count — db.r6g.large at 5,000 clusters, db.r6g.2xlarge at 50,000) +- Eliminates CLM API server, sentinel, and adapter compute costs diff --git a/docs/design/spec-to-pr-agent.md b/docs/design/spec-to-pr-agent.md index 5460cb678..332bbaf5b 100644 --- a/docs/design/spec-to-pr-agent.md +++ b/docs/design/spec-to-pr-agent.md @@ -5,7 +5,7 @@ flowchart TD subgraph ImplementLoop["Implementation Iteration (circular)"] E2E["Implement E2E /\nRefine E2E"] Feature["Implement Feature"] - Inject["Inject new versions of\ncomponents /\nImplement new CLM adapters"] + Inject["Inject new versions of\ncomponents /\nImplement new hyperfleet-operator controllers"] end E2E --> Feature diff --git a/docs/design/terraform-resource-adoption.md b/docs/design/terraform-resource-adoption.md index 78896a262..539cfb8e6 100644 --- a/docs/design/terraform-resource-adoption.md +++ b/docs/design/terraform-resource-adoption.md @@ -284,37 +284,21 @@ resource "aws_kms_key" "rds_logs" { } resource "aws_kms_alias" "rds_logs" { - name = "alias/${var.regional_id}-maestro-rds-logs" + name = "alias/${var.regional_id}-hyperfleet-rds-logs" target_key_id = aws_kms_key.rds_logs.key_id } # Log group with retention + KMS (FedRAMP AU-09, AU-11) resource "aws_cloudwatch_log_group" "rds_postgresql" { - name = "/aws/rds/instance/${var.regional_id}-maestro/postgresql" + name = "/aws/rds/instance/${var.regional_id}-hyperfleet/postgresql" retention_in_days = 365 kms_key_id = aws_kms_key.rds_logs.arn # Ensure the DB instance exists before we claim its log group - depends_on = [aws_db_instance.maestro] + depends_on = [aws_db_instance.hyperfleet] tags = merge(local.common_tags, { - Name = "${var.regional_id}-maestro-rds-postgresql-logs" - }) -} -``` - -For resources with server-assigned IDs (e.g., AmazonMQ), the log group name references the parent resource directly: - -```hcl -resource "aws_cloudwatch_log_group" "mq_general" { - name = "/aws/amazonmq/broker/${aws_mq_broker.hyperfleet.id}/general" - retention_in_days = 365 - kms_key_id = aws_kms_key.mq_logs.arn - - depends_on = [aws_mq_broker.hyperfleet] - - tags = merge(local.common_tags, { - Name = "${var.regional_id}-hyperfleet-mq-general-logs" + Name = "${var.regional_id}-hyperfleet-rds-postgresql-logs" }) } ``` @@ -326,31 +310,16 @@ resource "aws_cloudwatch_log_group" "mq_general" { RDS creates log groups with predictable names based on the DB instance identifier: ```bash -# RDS instance identifier is: ${regional_id}-maestro +# RDS instance identifier is: ${regional_id}-hyperfleet # RDS creates: /aws/rds/instance// import_if_needed \ - 'module.maestro_infrastructure.aws_cloudwatch_log_group.rds_postgresql' \ - "/aws/rds/instance/${TF_VAR_regional_id}-maestro/postgresql" + 'module.hyperfleet_db.aws_cloudwatch_log_group.rds_postgresql' \ + "/aws/rds/instance/${TF_VAR_regional_id}-hyperfleet/postgresql" ``` #### Dynamic imports (server-assigned IDs) -AmazonMQ and API Gateway use UUIDs in their log group names. These are only known after the parent resource is created: - -```bash -# Broker ID is a UUID assigned by AWS — look it up from state -BROKER_ID=$(tf_state_value \ - 'module.hyperfleet_infrastructure.aws_mq_broker.hyperfleet' '.values.id') -if [ -n "$BROKER_ID" ]; then - import_if_needed \ - 'module.hyperfleet_infrastructure.aws_cloudwatch_log_group.mq_general' \ - "/aws/amazonmq/broker/${BROKER_ID}/general" -else - # First deploy: broker hasn't been created yet, so no log group exists. - # Terraform will create both the broker and the log group. - echo " [skip] AmazonMQ log groups — broker not yet provisioned" -fi -``` +API Gateway uses UUIDs in its log group names. These are only known after the parent resource is created: ### Pipeline output across environment states @@ -360,7 +329,6 @@ fi --- Importing resources --- [not-found] ...rds_postgresql — resource does not exist in AWS (expected on fresh env) [not-found] ...rds_upgrade — resource does not exist in AWS (expected on fresh env) - [skip] AmazonMQ log groups — broker not yet provisioned [skip] API GW execution log group — API not yet provisioned === Import summary === @@ -375,14 +343,12 @@ fi ```text --- Importing resources --- - [imported] ...rds_postgresql <- /aws/rds/instance/int-regional-maestro/postgresql - [imported] ...rds_upgrade <- /aws/rds/instance/int-regional-maestro/upgrade - [imported] ...mq_general <- /aws/amazonmq/broker/b-abc123/general - [imported] ...mq_connection <- /aws/amazonmq/broker/b-abc123/connection + [imported] ...rds_postgresql <- /aws/rds/instance/int-regional-hyperfleet/postgresql + [imported] ...rds_upgrade <- /aws/rds/instance/int-regional-hyperfleet/upgrade [imported] ...api_gateway_execution <- API-Gateway-Execution-Logs_xyz789/prod === Import summary === - Imported: 5 + Imported: 3 Already in state: 0 Not found (fresh): 0 FAILED: 0 @@ -395,13 +361,11 @@ fi --- Importing resources --- [skip] ...rds_postgresql — already in state [skip] ...rds_upgrade — already in state - [skip] ...mq_general — already in state - [skip] ...mq_connection — already in state [skip] ...api_gateway_execution — already in state === Import summary === Imported: 0 - Already in state: 5 + Already in state: 3 Not found (fresh): 0 FAILED: 0 ====================== diff --git a/docs/design/testing-strategy.md b/docs/design/testing-strategy.md index d973ad908..f02501f4b 100644 --- a/docs/design/testing-strategy.md +++ b/docs/design/testing-strategy.md @@ -27,7 +27,7 @@ The nightly pipeline includes k6-based load tests that stress the Platform API u Two load test scripts target different concerns: - **Platform API load** (`ci/load-test/scripts/platform-api-load.js`): Ramps to 50 concurrent virtual users over 2 minutes, holds for 10 minutes, then ramps down. Exercises health endpoints, management cluster CRUD, resource bundle listing, and ManifestWork posting. Thresholds: p99 latency < 5s, error rate < 1%. -- **HCP lifecycle load** (`ci/load-test/scripts/hcp-lifecycle-load.js`): Creates multiple HostedClusters concurrently via the Platform API, polls for visibility, and posts ManifestWork to each. Validates that Maestro MQTT distribution and HyperShift operator scaling handle parallel cluster creation. +- **HCP lifecycle load** (`ci/load-test/scripts/hcp-lifecycle-load.js`): Creates multiple HostedClusters concurrently via the Platform API, polls for visibility, and posts ManifestWork to each. Validates that kube-applier DynamoDB-backed resource distribution and HyperShift operator scaling handle parallel cluster creation. Results are saved as JSON to Prow artifacts (`${ARTIFACT_DIR}/load-test-results/`). A baseline comparison script (`ci/load-test/compare-baseline.py`) checks for performance regressions against a baseline stored in S3, failing if any metric regresses beyond a configurable threshold (default 20%). diff --git a/docs/design/zoa-architecture.md b/docs/design/zoa-architecture.md index ec7cb0341..515632db3 100644 --- a/docs/design/zoa-architecture.md +++ b/docs/design/zoa-architecture.md @@ -1,6 +1,6 @@ # Zero Operator Access (ZOA) — Architecture -**Last Updated Date**: 2026-06-14 +**Last Updated Date**: 2026-07-29 ## Summary @@ -50,7 +50,9 @@ graph TB direction TB DynamoExec[DynamoDB
executions]:::storage DynamoAudit[DynamoDB
audit log]:::storage + DynamoApply[DynamoDB
ApplyDesire]:::storage S3[S3 Bucket
artifacts]:::storage + PG[PostgreSQL
hyperfleet-db]:::storage subgraph RC ["Regional Cluster (RC)"] direction TB @@ -63,11 +65,13 @@ graph TB APIGW --> PAPI - MaestroServer[Maestro Server
gRPC + MQTT]:::component - MaestroAgentRC[Maestro Agent
RC-targeted TAs]:::component + HFOperator[hyperfleet-operator
ManifestReconciler]:::component + KubeApplierRC[kube-applier
RC-targeted TAs]:::component - PAPI --> MaestroServer - MaestroServer <--> MaestroAgentRC + PAPI --> PG + HFOperator -->|watches Manifest CRs| PG + HFOperator -->|writes ApplyDesire| DynamoApply + DynamoApply -->|DynamoDB Streams| KubeApplierRC end PAPI -.-> DynamoExec @@ -79,7 +83,7 @@ graph TB subgraph MC ["Management Cluster (MC)"] direction TB - MaestroAgentMC[Maestro Agent
applies MW]:::component + KubeApplierMC[kube-applier
applies manifests]:::component subgraph NS ["Namespace: zoa-jobs"] direction TB @@ -97,12 +101,12 @@ graph TB Uploader -->|reads after Runner exits| CMOutput end - MaestroAgentMC -->|applies manifests| NS + KubeApplierMC -->|applies manifests| NS end end end - MaestroServer -->|"MQTT (no direct network)"| MaestroAgentMC + DynamoApply -->|"DynamoDB Streams (no direct network)"| KubeApplierMC Uploader -->|S3 output upload| S3 class rosa cluster @@ -114,17 +118,19 @@ graph TB ### Component Responsibilities -| Component | Location | Role | -| ------------------------- | ----------------- | ------------------------------------------------------- | -| **API Gateway** | AWS (regional) | SigV4 authentication, request routing | -| **Platform API** | RC (EKS pod) | TA validation, job generation, dispatch, reconciliation | -| **Maestro Server** | RC (EKS pod) | ManifestWork storage, MQTT distribution | -| **Maestro Agent** | RC + MC (EKS pod) | Applies ManifestWorks, reports status via MQTT | -| **DynamoDB (executions)** | AWS (regional) | Execution metadata, status tracking | -| **DynamoDB (audit)** | AWS (regional) | API call audit trail | -| **S3** | AWS (regional) | Artifact storage (output.json, execution.log) | -| **KMS** | AWS (regional) | Encryption at rest for DynamoDB and S3 | -| **zoa-jobs namespace** | RC + MC | Execution environment (Jobs, RBAC, ConfigMaps) | +| Component | Location | Role | +| ------------------------------ | ----------------- | ------------------------------------------------------------------------------ | +| **API Gateway** | AWS (regional) | SigV4 authentication, request routing | +| **Platform API** | RC (EKS pod) | TA validation, job generation, Manifest CR creation, reconciliation | +| **hyperfleet-operator** | RC (EKS pod) | ManifestReconciler watches Manifest CRs, writes ApplyDesire to DynamoDB | +| **kube-applier** | RC + MC (EKS pod) | Reads ApplyDesire from DynamoDB Streams, applies manifests, writes status back | +| **PostgreSQL (hyperfleet-db)** | AWS (regional) | Manifest CR storage (single source of truth) | +| **DynamoDB (ApplyDesire)** | AWS (regional) | Resource distribution layer between RC and target clusters | +| **DynamoDB (executions)** | AWS (regional) | Execution metadata, status tracking | +| **DynamoDB (audit)** | AWS (regional) | API call audit trail | +| **S3** | AWS (regional) | Artifact storage (output.json, execution.log) | +| **KMS** | AWS (regional) | Encryption at rest for DynamoDB, PostgreSQL, and S3 | +| **zoa-jobs namespace** | RC + MC | Execution environment (Jobs, RBAC, ConfigMaps) | ## Request Flow — Sequence Diagram @@ -135,10 +141,11 @@ sequenceDiagram participant Op as Operator (zoa CLI) participant GW as API Gateway participant API as Platform API - participant DB as DynamoDB - participant MS as Maestro Server - participant MQTT as MQTT Broker - participant MA as Maestro Agent + participant PG as PostgreSQL (hyperfleet-db) + participant DB as DynamoDB (executions) + participant HFO as hyperfleet-operator + participant DDB as DynamoDB (ApplyDesire) + participant KA as kube-applier participant MC as Target Cluster K8s API participant Runner as Runner Job participant CM as ConfigMap @@ -149,17 +156,19 @@ sequenceDiagram Op->>GW: POST /trusted-actions/get_pods/run (SigV4) GW->>GW: Validate SigV4, extract caller identity GW->>API: Forward request + X-Amz headers - API->>API: Validate params, build ManifestWork + API->>API: Validate params, build manifest payload API->>DB: Create execution record (status=pending, jira, ttl) - API->>MS: gRPC CreateManifestWork + API->>PG: Create Manifest CR in PostgreSQL API-->>Op: 202 {id, status: "pending"} - Note over Op,S3: 2. Dispatch (MQTT, no direct network) - MS->>MQTT: Publish ManifestWork to target cluster topic - MQTT->>MA: Deliver ManifestWork - MA->>MC: Apply manifests (SA, RBAC, CMs, Jobs) on target cluster - MA->>MQTT: Report "Applied" status - MQTT->>MS: Status feedback + Note over Op,S3: 2. Dispatch (DynamoDB, no direct network) + HFO->>PG: ManifestReconciler watches Manifest CR + HFO->>DDB: Write ApplyDesire to DynamoDB + DDB->>KA: DynamoDB Streams delivers ApplyDesire + KA->>MC: Apply manifests (SA, RBAC, CMs, Jobs) on target cluster + KA->>DDB: Write "Applied" status back to DynamoDB + HFO->>DDB: Read status update + HFO->>PG: Update Manifest CR status Note over Op,S3: 3. Execution (Two-Job model on target cluster) MC->>Runner: Start runner Job (per-exec SA) @@ -174,10 +183,11 @@ sequenceDiagram Uploader->>Uploader: Exit Note over Op,S3: 4. Reconciliation (5s loop) - API->>MS: gRPC GetManifestWork (poll feedback) - MS-->>API: feedbackRules: succeeded/failed + Job timestamps - API->>MS: gRPC DeleteManifestWork (cleanup) - MA->>MC: Delete all ZOA resources from target cluster + API->>PG: Read Manifest CR status (poll for updates) + PG-->>API: Status: succeeded/failed + Job timestamps + API->>PG: Delete Manifest CR (cleanup) + HFO->>DDB: Remove ApplyDesire from DynamoDB + KA->>MC: Delete all ZOA resources from target cluster API->>DB: Update: status, runner_seconds, upload_seconds, duration_seconds, output_status Note over Op,S3: 5. Retrieval @@ -190,31 +200,31 @@ sequenceDiagram ### Per-Endpoint Data Flow Summary -| Endpoint | Components Touched | -| -------------------------- | -------------------------------------------------------------------------------------------- | -| `POST /{action}/run` | API Gateway → Platform API → DynamoDB (executions) → Maestro → MQTT → Agent → Target (RC/MC) | -| `GET /runs/{id}` | API Gateway → Platform API → DynamoDB (executions) + S3 | -| `GET /runs` | API Gateway → Platform API → DynamoDB (executions) | -| `GET /` (catalog) | API Gateway → Platform API (in-memory registry) | -| `GET /{action}` (describe) | API Gateway → Platform API (in-memory registry) | -| `GET /audit` | API Gateway → Platform API → DynamoDB (audit table) | +| Endpoint | Components Touched | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `POST /{action}/run` | API Gateway → Platform API → DynamoDB (executions) → PostgreSQL (Manifest CR) → hyperfleet-operator → DynamoDB (ApplyDesire) → kube-applier → Target (RC/MC) | +| `GET /runs/{id}` | API Gateway → Platform API → DynamoDB (executions) + S3 | +| `GET /runs` | API Gateway → Platform API → DynamoDB (executions) | +| `GET /` (catalog) | API Gateway → Platform API (in-memory registry) | +| `GET /{action}` (describe) | API Gateway → Platform API (in-memory registry) | +| `GET /audit` | API Gateway → Platform API → DynamoDB (audit table) | ## Network Architecture ### Key Constraint: No Direct Network Path from RC to MC -The Regional Cluster cannot reach the Management Cluster's Kubernetes API directly. All communication to MCs flows through Maestro's MQTT-based protocol: +The Regional Cluster cannot reach the Management Cluster's Kubernetes API directly. All communication to MCs flows through DynamoDB as an intermediary: -``` -RC → Maestro Server (gRPC) → MQTT Broker → Maestro Agent (target) → Target Kubernetes API +```text +RC → Platform API (creates Manifest CR in PostgreSQL) → hyperfleet-operator (writes ApplyDesire to DynamoDB) → kube-applier (reads from DynamoDB Streams on target) → Target Kubernetes API ``` -For MC-targeted TAs, the MQTT path crosses the network boundary. For RC-targeted TAs, the Maestro Agent on the RC applies the ManifestWork locally. +For MC-targeted TAs, the DynamoDB path crosses the network boundary. For RC-targeted TAs, kube-applier on the RC reads the ApplyDesire and applies it locally. This means: - Platform API cannot kubectl into remote clusters -- Status feedback flows back the same path: Target → MQTT → Maestro Server → Platform API (gRPC) +- Status feedback flows back through DynamoDB: Target kube-applier → DynamoDB (status) → hyperfleet-operator → Manifest CR status update → Platform API - Output must be uploaded to S3 directly from the target cluster (via the uploader Job) ### Authentication Flow @@ -242,12 +252,12 @@ Platform API │ Records in DynamoDB: full caller identity with every execution │ ▼ -Maestro (gRPC CreateManifestWork) +PostgreSQL (creates Manifest CR) │ - │ No additional auth — internal service call within RC + │ No additional auth — internal database within RC │ ▼ -MQTT → Maestro Agent → Job on target cluster +hyperfleet-operator → DynamoDB (ApplyDesire) → kube-applier → Job on target cluster ``` ### S3 Output Pipeline (Two-Job Architecture) @@ -287,7 +297,7 @@ Operator (via GET /runs/{id}?include=output) ### 1. Submission ``` -Operator: zoa run get_pods -t mc-useast1-1 -n maestro +Operator: zoa run get_pods -t mc-useast1-1 -n hyperfleet │ ▼ Platform API receives POST /api/v0/trusted-actions/get_pods/run @@ -299,22 +309,22 @@ Platform API receives POST /api/v0/trusted-actions/get_pods/run - Derives runner SA from scope + type (kube-api → per-exec SA) - Generates execution UUID - Creates DynamoDB record (status: pending, output_status: pending, jira, ttl=365d) - - Builds ManifestWork (SA, RBAC, output CM, scripts CM, uploader RBAC, runner Job, upload Job) - - Dispatches via Maestro gRPC CreateManifestWork + - Builds manifest payload (SA, RBAC, output CM, scripts CM, uploader RBAC, runner Job, upload Job) + - Creates Manifest CR in PostgreSQL (hyperfleet-db) - Returns {id, status: "pending"} to caller ``` ### 2. Dispatch -``` -Maestro Server - - Stores ResourceBundle in database - - Publishes to MQTT topic for target cluster consumer (RC or MC) +```text +hyperfleet-operator (ManifestReconciler on RC) + - Watches Manifest CRs in PostgreSQL (hyperfleet-db) + - Writes ApplyDesire to DynamoDB for the target cluster (RC or MC) │ - ▼ MQTT + ▼ DynamoDB Streams │ -Maestro Agent (on target cluster — RC or MC) - - Receives ManifestWork via MQTT subscription +kube-applier (on target cluster — RC or MC) + - Receives ApplyDesire via DynamoDB Streams - Applies all manifests to target cluster Kubernetes API: 1. ServiceAccount: zoa-runner- (per-execution) 2. ClusterRole/Role (per-execution RBAC) @@ -325,7 +335,7 @@ Maestro Agent (on target cluster — RC or MC) 7. Role/RoleBinding: zoa-uploader- (dynamic, scoped to output CM + runner Job) 8. Runner Job: zoa- (executes TA, writes to output CM) 9. Uploader Job: zoa--upload (reads CM, uploads to S3) - - Reports status back via MQTT (Applied, Available) + - Writes status back to DynamoDB (Applied, Available) ``` ### 3. Execution (Two-Job Model) @@ -340,7 +350,7 @@ Runner Job (zoa-): │ ├── Logs metadata: [zoa] execution_id=... action=... target=... ├── Executes /zoa/run.sh (the TA script) - │ └── kubectl get pods -n maestro -o json > /artifacts/output.json + │ └── kubectl get pods -n hyperfleet -o json > /artifacts/output.json ├── Captures exit code ├── Patches ConfigMap zoa-output- with: │ - data.output.json (if exists) @@ -371,9 +381,9 @@ Platform API Reconciler (5-second loop on RC) │ ├── For each pending/running execution: │ │ - │ ├── Calls Maestro gRPC GetManifestWork + │ ├── Reads Manifest CR status from PostgreSQL │ │ - │ ├── Parses feedbackRules from BOTH Jobs: + │ ├── Parses status feedback from BOTH Jobs: │ │ Runner: .status.succeeded, .status.failed, .status.startTime, .status.completionTime │ │ Uploader: .status.succeeded, .status.failed, .status.completionTime │ │ @@ -385,13 +395,13 @@ Platform API Reconciler (5-second loop on RC) │ │ │ runner_seconds = runner.completionTime - runner.startTime │ │ │ upload_seconds = uploader.completionTime - runner.completionTime │ │ │ duration_seconds = now - created_at (total wall-clock) - │ │ ├── Delete ResourceBundle from Maestro (gRPC) - │ │ │ └── Cascades: Agent removes ManifestWork → all resources on target cluster + │ │ ├── Delete Manifest CR from PostgreSQL + │ │ │ └── Cascades: operator removes ApplyDesire → kube-applier removes all resources on target cluster │ │ └── Update DynamoDB: status, completed_at, updated_at, runner_seconds, │ │ upload_seconds, duration_seconds, output_status (uploaded|failed) │ │ │ └── On timeout (exceeded execution_timeout + upload_timeout + 120s dispatch buffer): - │ ├── Delete ResourceBundle from Maestro (cleanup first) + │ ├── Delete Manifest CR from PostgreSQL (cleanup first) │ └── Update DynamoDB: status=timed_out, duration_seconds │ └── Sleep 5s → repeat @@ -502,7 +512,7 @@ terraform/modules/zoa/ Static ZOA infrastructure is deployed via the `zoa-jobs` Helm chart at `argocd/config/shared/zoa-jobs/`. The root ArgoCD ApplicationSet discovers charts under `argocd/config/shared/*` and deploys them to both Regional and Management clusters with `CreateNamespace=true`, which creates the `zoa-jobs` namespace automatically. -The chart provisions static ServiceAccounts (`zoa-uploader`, `zoa-aws-read`, `zoa-aws-write`, plus breakglass SAs). Pod Identity associations for AWS-scoped SAs are wired via Terraform (`terraform/modules/zoa/` and `terraform/modules/zoa-job-pod-identity/`). Per-execution resources (runner SA, RBAC, Jobs, ConfigMaps) are created dynamically by each ManifestWork on the target cluster. +The chart provisions static ServiceAccounts (`zoa-uploader`, `zoa-aws-read`, `zoa-aws-write`, plus breakglass SAs). Pod Identity associations for AWS-scoped SAs are wired via Terraform (`terraform/modules/zoa/` and `terraform/modules/zoa-job-pod-identity/`). Per-execution resources (runner SA, RBAC, Jobs, ConfigMaps) are created dynamically by kube-applier when it processes each ApplyDesire on the target cluster. ## TA Template System @@ -526,10 +536,10 @@ ConfigMap: zoa-ta-templates (mounted into Platform API pod at /templates/) TemplateRegistry (in-memory map of action_name → TATemplate struct) │ ▼ (On each execution request) -BuildManifestWork(template, renderContext) → ManifestWork with all K8s manifests +BuildManifestPayload(template, renderContext) → Manifest CR with all K8s manifests ``` -### Template → ManifestWork Generation +### Template → Manifest CR Generation What the TA author writes (~15 lines): @@ -544,7 +554,7 @@ script: | kubectl get pods ... ``` -What Platform API generates (full ManifestWork with ~200 lines of K8s manifests): +What Platform API generates (Manifest CR with ~200 lines of K8s manifests): - ServiceAccount (per-execution `zoa-runner-`) - Role/ClusterRole (from `rbac.rules`) @@ -556,7 +566,7 @@ What Platform API generates (full ManifestWork with ~200 lines of K8s manifests) - Runner Job (executes TA script, writes output to ConfigMap) - Uploader Job (reads ConfigMap, uploads to S3) - Job (image, volumes, env vars, resources, labels, TTL) -- ManifestWork feedbackRules (extract Job status) +- Status tracking fields (extract Job status via DynamoDB feedback) ### Job Boilerplate (Centrally Managed) @@ -584,12 +594,13 @@ Changing any of these updates ALL future TA executions — no per-TA changes nee ### Normal Cleanup (Reconciler-Driven) -``` -1. Reconciler detects Job terminal status (succeeded/failed) via ManifestWork feedback -2. Reconciler deletes ResourceBundle from Maestro (gRPC) -3. Maestro Agent removes ManifestWork from its local state -4. Agent cascades deletion: Job, Pod, ConfigMap, Role, RoleBinding — all removed from target cluster -5. Reconciler updates DynamoDB with terminal status and duration +```text +1. Reconciler detects Job terminal status (succeeded/failed) via Manifest CR status feedback +2. Reconciler deletes Manifest CR from PostgreSQL +3. hyperfleet-operator sets delete flag on ApplyDesire in DynamoDB +4. kube-applier processes deletion: Job, Pod, ConfigMap, Role, RoleBinding — all removed from target cluster +5. After deletion propagates, hyperfleet-operator removes ApplyDesire from DynamoDB +6. Reconciler updates DynamoDB with terminal status and duration ``` ### Timeout Model @@ -603,11 +614,11 @@ Formula: execution_timeout + upload_timeout + 120s (dispatch buffer) Default: 1800 + 120 + 120 = 2040s (~34 min) ``` -The dispatch buffer (hardcoded 120s) accounts for Maestro MQTT delivery, pod scheduling, and image pull before the uploader poll loop starts. The reconciler polls DynamoDB every 5s and checks `created_at` against this budget. +The dispatch buffer (hardcoded 120s) accounts for DynamoDB Streams delivery, pod scheduling, and image pull before the uploader poll loop starts. The reconciler polls DynamoDB every 5s and checks `created_at` against this budget. When exceeded: -1. Delete ResourceBundle from Maestro (stops all Jobs via cascade) +1. Delete Manifest CR from PostgreSQL (stops all Jobs via cascade through operator and kube-applier) 2. Update DynamoDB: status=timed_out, duration_seconds Fires when: Normal operation. Every timed-out execution goes through this path. @@ -622,13 +633,13 @@ Set on BOTH runner and uploader Job specs. When exceeded: Kubernetes forcibly terminates the pod and marks the Job as Failed with reason=DeadlineExceeded. -Fires when the reconciler FAILED to delete the ResourceBundle: +Fires when the reconciler FAILED to delete the Manifest CR: - Platform API pod crashed or restarted (reconciler loop stopped) -- Maestro gRPC is unreachable (DeleteManifestWork fails repeatedly) +- PostgreSQL is unreachable (Manifest CR deletion fails repeatedly) - DynamoDB query failed (reconciler never found this execution) -In these cases, the ManifestWork stays on the target cluster with Jobs still running. `activeDeadlineSeconds` ensures K8s itself kills the pods after ~35 min, preventing infinite resource consumption. The reconciler will eventually recover and clean up the ResourceBundle on its next successful poll — by then the Jobs are already dead. +In these cases, the ApplyDesire stays in DynamoDB with Jobs still running on the target cluster. `activeDeadlineSeconds` ensures K8s itself kills the pods after ~35 min, preventing infinite resource consumption. The reconciler will eventually recover and clean up the Manifest CR on its next successful poll — by then the Jobs are already dead. **Layer 3 — ttlSecondsAfterFinished (garbage collection)** @@ -639,17 +650,17 @@ Set on BOTH runner and uploader Job specs. This is a native K8s Job controller feature — it deletes the **Job object** (not the pod) from the cluster after the specified duration post-completion. -Fires when a Job already reached terminal state (Complete or Failed) but the ManifestWork was never deleted: +Fires when a Job already reached terminal state (Complete or Failed) but the applied resources were never cleaned up: -- Reconciler deleted the ResourceBundle, but Maestro Agent failed to cascade the ManifestWork deletion (Agent bug, CRD issue) -- `activeDeadlineSeconds` killed the pod (Layer 2), Job became Failed, but the ResourceBundle/ManifestWork still exist on cluster +- Reconciler deleted the Manifest CR, but kube-applier failed to cascade the resource deletion (applier bug, connectivity issue) +- `activeDeadlineSeconds` killed the pod (Layer 2), Job became Failed, but the ApplyDesire and applied resources still exist on cluster What it does NOT cover: Jobs stuck in a running state that never finish — those are handled by Layer 2. **Summary** -``` -Happy path: Reconciler detects completion/timeout → deletes RB → done (~seconds) +```text +Happy path: Reconciler detects completion/timeout → deletes Manifest CR → done (~seconds) Reconciler down: activeDeadlineSeconds kills pods (~2100s) → TTL cleans Job objects (+3600s) Both fail: Jobs run until activeDeadlineSeconds, then GC after TTL ``` @@ -675,7 +686,7 @@ Every execution produces audit data at multiple layers: | Kubernetes (labels on all resources) | execution-id, operator, action, scope, type, revision, target | `kubectl get jobs -l zoa.rosa.io/operator=slopezma` | | Platform API (DynamoDB audit table) | Every audited API call: method, path (full URI), action, target, execution_id, jira, approval_state, operator, status_code, timestamp | `zoa audit` CLI | | AWS CloudTrail | SigV4 caller identity on API Gateway invocation | CloudTrail console | -| Maestro (MQTT events) | ManifestWork create/delete events with metadata | Maestro server logs | +| DynamoDB (ApplyDesire) | Manifest CR create/delete events, kube-applier status updates | DynamoDB console or CloudWatch | ### Correlation @@ -719,4 +730,3 @@ When enabled, write TAs with structured approval policies will require peer appr - [ZOA Trusted Actions — Implementation Details](./zoa-trusted-actions.md) — TA template format, CLI design, API endpoints - [ZOA Security Model](./zoa-security-model.md) — SA isolation strategies, RBAC model, audit -- [Maestro MQTT Resource Distribution](./maestro-mqtt-resource-distribution.md) — ManifestWork dispatch mechanism diff --git a/docs/design/zoa-lambda-architecture.md b/docs/design/zoa-lambda-architecture.md index 260b8187d..4a424410b 100644 --- a/docs/design/zoa-lambda-architecture.md +++ b/docs/design/zoa-lambda-architecture.md @@ -86,16 +86,16 @@ No RC EKS in path. No ALB. No Platform API pods. No Operator. No kube-applier. N ### What changes (summary) -| Concern | Before | After | -| --------------------- | ----------------------------------- | --------------------------------------------------------------------------- | -| TA scheduling + API | Platform API pods (K8s) | ZOA Lambda per-VPC (AWS-managed) | -| Transport to clusters | Maestro chain OR kube-applier chain | Direct kubectl from local Lambda (same VPC) | -| Break-glass | Platform API (chicken-and-egg) | ZOA Lambda per-VPC (independent) | -| Placement | N/A | ZOA Access Lambda (RC, no VPC) | -| Approval | N/A | ZOA Access Lambda + DynamoDB | -| Reconciliation | Platform API goroutine (15s) | EventBridge Scheduler → local ZOA Lambda (reconciler 30s, GC 5m, reaper 5m) | -| State | DynamoDB | DynamoDB (same, centralized in RC) | -| Artifacts | S3 | S3 (same) | +| Concern | Before | After | +| --------------------- | ------------------------------ | --------------------------------------------------------------------------- | +| TA scheduling + API | Platform API pods (K8s) | ZOA Lambda per-VPC (AWS-managed) | +| Transport to clusters | kube-applier chain (DynamoDB) | Direct kubectl from local Lambda (same VPC) | +| Break-glass | Platform API (chicken-and-egg) | ZOA Lambda per-VPC (independent) | +| Placement | N/A | ZOA Access Lambda (RC, no VPC) | +| Approval | N/A | ZOA Access Lambda + DynamoDB | +| Reconciliation | Platform API goroutine (15s) | EventBridge Scheduler → local ZOA Lambda (reconciler 30s, GC 5m, reaper 5m) | +| State | DynamoDB | DynamoDB (same, centralized in RC) | +| Artifacts | S3 | S3 (same) | > **Scope**: Lambda replaces ONLY the ZOA/TA/break-glass path. Cluster lifecycle (create/delete/patch) remains on Hyperfleet Operator + kube-applier. Both coexist. @@ -631,7 +631,7 @@ Per-VPC Lambda reconciler detects: approved AWS break-glass for this target Break-glass uses the same ZOA Lambda as TAs (different code path, same binary). Dependencies: - Lambda (99.95%), DynamoDB (99.999%), Function URL (same as Lambda) -- None of these are Platform API, Maestro, or any custom K8s workload. +- None of these are Platform API, hyperfleet-operator, or any custom K8s workload. - ZOA Access Lambda (for creating rosa-boundary) is NOT VPC-attached, so it works even if EKS is completely down. ### 3.4 Approval Workflow diff --git a/docs/design/zoa-security-model.md b/docs/design/zoa-security-model.md index 54fc55f2b..6776bfc53 100644 --- a/docs/design/zoa-security-model.md +++ b/docs/design/zoa-security-model.md @@ -35,14 +35,14 @@ This document details the security architecture for ZOA Trusted Actions: how pri ┌─────────────────────────────────────────┐ │ Trust Zone B: Regional Cluster (RC) │ │ - Platform API (validates, dispatches) │ -│ - Maestro Server (stores, distributes) │ -│ - DynamoDB + S3 (persists) │ +│ - hyperfleet-operator (ManifestReconciler) │ +│ - hyperfleet-db + DynamoDB + S3 │ └────────────────────┬────────────────────┘ - │ MQTT (encrypted) + │ DynamoDB + DynamoDB Streams ▼ ┌─────────────────────────────────────────┐ │ Trust Zone C: Management Cluster (MC) │ -│ - Maestro Agent (applies manifests) │ +│ - kube-applier (applies resources) │ │ - zoa-jobs namespace (executes TAs) │ │ - Control plane namespaces (HCPs) │ └─────────────────────────────────────────┘ @@ -96,7 +96,7 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: zoa- - namespace: maestro + namespace: hyperfleet labels: zoa.rosa.io/execution-id: "fa65418c-..." zoa.rosa.io/action: "get_pods" @@ -110,7 +110,7 @@ apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: zoa- - namespace: maestro + namespace: hyperfleet subjects: - kind: ServiceAccount name: zoa-runner- @@ -124,7 +124,7 @@ roleRef: - RBAC resources are scoped to exactly what the TA declares — nothing more - RoleBindings bind the per-execution runner SA -- All RBAC resources are deleted when the reconciler cleans up the ResourceBundle +- All RBAC resources are deleted when the reconciler cleans up the Manifest CR - Namespace-scoped Roles for namespace-specific TAs, ClusterRoles for cluster-wide TAs ### ServiceAccount Model (Two-Job Architecture) @@ -140,7 +140,7 @@ ZOA uses a split SA model for privilege separation: **Key design decisions:** -1. **Per-execution SA for kube TAs**: `zoa-runner-` is created dynamically as part of the ManifestWork. It has no Pod Identity (no AWS IAM role). This gives perfect K8s audit-log attribution. +1. **Per-execution SA for kube TAs**: `zoa-runner-` is created dynamically as part of the Manifest CR. It has no Pod Identity (no AWS IAM role). This gives perfect K8s audit-log attribution. 2. **Static SAs for AWS TAs**: `zoa-aws-read` and `zoa-aws-write` require pre-provisioned Pod Identity associations. These are static but still have **no access to the ZOA S3 bucket**. 3. **Dedicated uploader SA**: Only `zoa-uploader` can write to S3. Kubernetes RBAC for the uploader is generated dynamically per execution (not a static Helm template), scoped with `resourceNames` to the specific output ConfigMap and runner Job. 4. **No SA has both**: No single SA has both operational permissions AND S3 write access. @@ -239,7 +239,7 @@ x-amz-meta-target: mc-useast1-1 | All API calls (POST + GET) | DynamoDB (audit table) | 365 days (TTL) | `zoa audit` | | API Gateway access | CloudTrail | 90 days (configurable) | AWS Console | | Kubernetes API calls from Job | Target cluster audit log | Cluster-dependent | kubectl audit | -| ResourceBundle lifecycle | Maestro server logs | Log retention | kubectl logs | +| Desire document lifecycle | DynamoDB desire writes | 365 days (TTL) | DynamoDB query | **Audit table design**: Every audited call (`POST /run`, `GET /runs`, `GET /runs/{id}`, `GET /audit`) is recorded with consistent fields: `id`, `account_id`, `caller_arn`, `operator`, `method`, `path` (full URI), `action`, `target_cluster`, `execution_id`, `jira`, `approval_state`, `status_code`, `timestamp`. Fields not applicable to a call type are empty strings. The sort key uses nanosecond-precision timestamps (`2006-01-02T15:04:05.000000000Z`) for uniqueness. Catalog/describe endpoints are not audited (public metadata, high frequency). @@ -275,7 +275,7 @@ The two-job architecture enforces privilege separation: ### Architecture ``` -ManifestWork contains: +Manifest CR contains: ├── ServiceAccount: zoa-runner- (per-execution, no AWS) ├── Role/ClusterRole (per-execution RBAC for TA) ├── RoleBinding → zoa-runner- @@ -317,7 +317,7 @@ ManifestWork contains: - Shared PVC between Jobs (eliminates size limit but adds provisioning) - Runner direct S3 upload for specific large-output TAs (breaks isolation but pragmatic) -**Uploader RBAC is dynamic per execution**: Platform API generates a `Role`/`RoleBinding` pair (`zoa-uploader-`) in each ManifestWork, scoped with `resourceNames` to: +**Uploader RBAC is dynamic per execution**: Platform API generates a `Role`/`RoleBinding` pair (`zoa-uploader-`) in each Manifest CR, scoped with `resourceNames` to: - The specific output ConfigMap (`zoa-output-`) - The specific runner Job (`zoa-`) @@ -338,7 +338,7 @@ The `zoa-uploader` ServiceAccount is static (required for Pod Identity), but its | TLS | FIPS-validated TLS libraries in RHEL UBI9 base image | | AWS CLI in job | Uses FIPS endpoints when `AWS_USE_FIPS_ENDPOINT=true` | | DynamoDB | FIPS endpoint via VPC Gateway Endpoint | -| MQTT (Maestro) | TLS 1.2+ with FIPS-validated cipher suites | +| hyperfleet-db | TLS 1.2+ enforced by Aurora PostgreSQL | ## Network Security (Planned) @@ -396,7 +396,7 @@ spec: | AU-12 (Audit Generation) | Automatic — all audited API calls recorded; rejections (400/429) also captured | | CM-7 (Least Functionality) | No shell access, no arbitrary commands — only pre-approved TAs | | IA-2 (Identification and Authentication) | SigV4 + STS, caller ARN extracted per request | -| SC-8 (Transmission Confidentiality) | TLS 1.2+ on all channels (API, MQTT, S3) | +| SC-8 (Transmission Confidentiality) | TLS 1.2+ on all channels (API, DynamoDB, hyperfleet-db, S3) | | SC-13 (Cryptographic Protection) | FIPS-validated KMS, SSE-KMS at rest | | SC-28 (Protection of Information at Rest) | SSE-KMS for S3 and DynamoDB | | SI-4 (Information System Monitoring) | Reconciler loop monitors execution status continuously | diff --git a/docs/design/zoa-trusted-actions.md b/docs/design/zoa-trusted-actions.md index 2ebb707cc..abcba6bac 100644 --- a/docs/design/zoa-trusted-actions.md +++ b/docs/design/zoa-trusted-actions.md @@ -4,19 +4,19 @@ ## Summary -Zero Operator Access (ZOA) Trusted Actions provide a mediated, auditable mechanism for executing predefined operational tasks on ROSA HCP v2 regional infrastructure without granting operators direct cluster access. All actions are dispatched via Maestro as ManifestWorks, executed as ephemeral Kubernetes Jobs, and produce artifacts stored in S3 with full audit trails in DynamoDB. +Zero Operator Access (ZOA) Trusted Actions provide a mediated, auditable mechanism for executing predefined operational tasks on ROSA HCP v2 regional infrastructure without granting operators direct cluster access. All actions are dispatched via Platform API as Manifest CRs in PostgreSQL (hyperfleet-db), distributed to target clusters by the hyperfleet-operator and kube-applier pipeline, executed as ephemeral Kubernetes Jobs, and produce artifacts stored in S3 with full audit trails in DynamoDB. ## Context - **Problem Statement**: Operators currently require direct kubectl/AWS CLI access to diagnose and remediate cluster issues. This violates Zero Operator Access principles by creating persistent, unaudited access paths. We need a system that allows operational tasks to be executed exclusively through predefined, auditable channels. - **Constraints**: - EKS Pod Identity allows only one IAM role per ServiceAccount per namespace - - Maestro ManifestWork is the transport mechanism to target clusters (no direct network path from RC to MC) - - ManifestWork `feedbackRules` status values are size-limited (~1KB per field, 128KB total via MQTT) - - All output must be stored in S3 (not in ManifestWork status) + - Manifest CR → DynamoDB ApplyDesire → kube-applier is the transport mechanism to target clusters (no direct network path from RC to MC) + - DynamoDB status table values are size-limited per item (400KB max) + - All output must be stored in S3 (not in Manifest CR status) - Must be FIPS-compliant for FedRAMP - **Assumptions**: - - Maestro Agent runs on both RC and MC clusters + - kube-applier runs on MC clusters - Platform API is the single entry point for TA execution - ArgoCD manages infrastructure provisioning on both cluster types - TAs may move to their own repository in the future @@ -29,7 +29,7 @@ Zero Operator Access (ZOA) Trusted Actions provide a mediated, auditable mechani | ------------------------------------------------------- | ------------------- | -------------------------------------------------------------------- | | Script logic + RBAC rules | TA author | `argocd/config/regional-cluster/platform-api/ta-templates/` | | Job boilerplate (image, volumes, entrypoint, resources) | Platform/infra team | `zoa-job-config` ConfigMap in platform repo | -| Job generation logic | Platform API code | Go code reads template + config, builds ManifestWork | +| Job generation logic | Platform API code | Go code reads template + config, builds Manifest CR | | Infrastructure (namespace, SAs, Pod Identity) | Platform/infra team | `zoa-jobs` Helm chart (`argocd/config/shared/zoa-jobs/`) + Terraform | ### TA Template Format (What Authors Write) @@ -135,8 +135,8 @@ AWS-scoped TAs use static ServiceAccounts (`zoa-aws-read`, `zoa-aws-write`) with "affected_resources": [ { "kind": "Pod", - "namespace": "maestro", - "name": "maestro-xyz", + "namespace": "hyperfleet", + "name": "hyperfleet-operator-xyz", "action": "deleted" } ], @@ -159,7 +159,7 @@ fi ### What Platform API Generates (Per Execution) -From a minimal TA template, Platform API dynamically creates a ManifestWork containing: +From a minimal TA template, Platform API dynamically creates a Manifest CR containing: 1. **ServiceAccount** — per-execution `zoa-runner-` 2. **Role/ClusterRole** — from `rbac.rules` section @@ -219,7 +219,7 @@ The `zoa-job-config` ConfigMap serves as the centralized source of truth for all - **Wrapper scripts (`entrypoint.sh`, `upload_entrypoint.sh`)**: Embedded in the ConfigMap rather than baked into the container image. This allows hotfixing execution behavior (e.g., output capture, logging format) without rebuilding the `zoa-tools` image. - **Base64 encoding for inter-job transfer**: The runner Job writes `execution.log` and `output.json` to the output ConfigMap as `binaryData` (base64-encoded). This avoids YAML escaping issues with arbitrary script output while staying within Kubernetes API limits (~10-15k lines of output). -- **Two-job parallel dispatch**: Both runner and uploader Jobs are created simultaneously in the same ManifestWork to avoid time overhead. The uploader starts immediately and polls the runner Job status every 1s (checking for `Complete` or `Failed` conditions). This detects runner failure in ~1s with no wasted wait time. Once the runner finishes, the uploader reads the output ConfigMap and uploads artifacts to S3. This parallel creation eliminates sequential dispatch latency — the uploader is already scheduled and waiting by the time the runner finishes. Each Job uses its own ServiceAccount, which also avoids shared permission leakage between execution and upload concerns. +- **Two-job parallel dispatch**: Both runner and uploader Jobs are created simultaneously in the same Manifest CR to avoid time overhead. The uploader starts immediately and polls the runner Job status every 1s (checking for `Complete` or `Failed` conditions). This detects runner failure in ~1s with no wasted wait time. Once the runner finishes, the uploader reads the output ConfigMap and uploads artifacts to S3. This parallel creation eliminates sequential dispatch latency — the uploader is already scheduled and waiting by the time the runner finishes. Each Job uses its own ServiceAccount, which also avoids shared permission leakage between execution and upload concerns. - **Exit code preservation**: The runner captures `PIPESTATUS[0]` from the TA script and propagates it both to the ConfigMap (for the uploader/reconciler) and as the container exit code (for Kubernetes Job status). Crucially, the ConfigMap patch happens _before_ the runner exits — so even when the TA script fails, the output and logs are still written to the ConfigMap and subsequently uploaded to S3, making debugging of failed TAs straightforward. - **Stdout + stderr capture**: All script output is captured via `tee` to `/artifacts/execution.log`, ensuring the full execution trace is available in S3 even if the runner Pod is garbage-collected. - **ConfigMap checksum annotation**: The Platform API Deployment uses a checksum of the `zoa-job-config` ConfigMap content as a pod annotation. When the ConfigMap changes (e.g., new image version, updated entrypoint), ArgoCD detects the annotation change and triggers a rolling update of the API pods, which then hot-reload the new config on startup. @@ -240,12 +240,12 @@ script: | Cleanup is **reconciler-driven**, not purely TTL-based: -1. **On terminal status (succeeded, failed, timed_out)**: The Platform API reconciler deletes the ResourceBundle from Maestro via gRPC. Maestro Agent cascades deletion to all resources on the target cluster (Job, Pod, ConfigMap, RBAC). -2. **Race-safe ordering**: ResourceBundle is deleted BEFORE DynamoDB status is updated. If RB deletion fails, status stays `pending`/`running` and the reconciler retries on the next tick. +1. **On terminal status (succeeded, failed, timed_out)**: The hyperfleet-operator (ManifestReconciler) deletes the Manifest CR from PostgreSQL and removes the corresponding ApplyDesire specs from DynamoDB. kube-applier cascades deletion to all resources on the target cluster (Job, Pod, ConfigMap, RBAC). +2. **Race-safe ordering**: Manifest CR is deleted BEFORE DynamoDB status is updated. If deletion fails, status stays `pending`/`running` and the reconciler retries on the next tick. 3. **TTL as safety net**: Jobs have `ttlSecondsAfterFinished: 3600` (1h) as backup GC in case reconciler fails to clean up. 4. **Logs survive cleanup**: The uploader Job uploads `execution.log` to S3 before resources are deleted, so troubleshooting data is available via the API even after the Pod/Job is garbage-collected. -Static ServiceAccounts (`zoa-uploader`, `zoa-aws-read`, `zoa-aws-write`) are infrastructure managed by the `zoa-jobs` chart and are never deleted. Per-execution runner SAs and all other ManifestWork resources are removed on completion. +Static ServiceAccounts (`zoa-uploader`, `zoa-aws-read`, `zoa-aws-write`) are infrastructure managed by the `zoa-jobs` chart and are never deleted. Per-execution runner SAs and all other Manifest CR resources are removed on completion. ### Service Account Strategy — Two-Job Split @@ -260,9 +260,9 @@ ZOA uses a split SA model separating operational permissions from output transpo **Key design decisions:** -1. **Per-execution SA for kube TAs**: `zoa-runner-` is created dynamically in the ManifestWork. No Pod Identity — perfect K8s audit attribution. +1. **Per-execution SA for kube TAs**: `zoa-runner-` is created dynamically in the Manifest CR. No Pod Identity — perfect K8s audit attribution. 2. **Static SAs for AWS TAs**: `zoa-aws-read` and `zoa-aws-write` require pre-provisioned Pod Identity. They have **no access to the ZOA S3 bucket**. -3. **Dedicated uploader SA**: Only `zoa-uploader` can write to S3. Uploader Kubernetes RBAC is generated dynamically per execution in the ManifestWork, scoped with `resourceNames` to the specific output ConfigMap and runner Job. +3. **Dedicated uploader SA**: Only `zoa-uploader` can write to S3. Uploader Kubernetes RBAC is generated dynamically per execution in the Manifest CR, scoped with `resourceNames` to the specific output ConfigMap and runner Job. 4. **No SA has both**: No single SA has both operational permissions AND S3 write access. **Audit chain:** @@ -271,7 +271,7 @@ ZOA uses a split SA model separating operational permissions from output transpo | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | | Platform API (DynamoDB executions) | `execution_id`, `operator`, `jira`, `action`, `target`, `params`, `revision`, `updated_at`, `dry_run`, `force`, timestamps | Who requested what, when, why (Jira), and how (dry-run/forced) | | Platform API (DynamoDB audit table) | `method`, `path` (full URI), `action`, `target_cluster`, `execution_id`, `jira`, `operator`, `status_code`, `timestamp` | Every API call (including reads and rejections) | -| ManifestWork + all resources | Labels: `zoa.rosa.io/execution-id`, `zoa.rosa.io/operator`, `zoa.rosa.io/action`, `zoa.rosa.io/revision` | Full traceability on every K8s resource | +| Manifest CR + all resources | Labels: `zoa.rosa.io/execution-id`, `zoa.rosa.io/operator`, `zoa.rosa.io/action`, `zoa.rosa.io/revision` | Full traceability on every K8s resource | | Kubernetes audit logs | Per-execution SA name (`zoa-runner-`) + pod labels | Perfect execution-level attribution | | S3 object metadata | `x-amz-meta-execution-id`, `x-amz-meta-operator` | Output ownership | @@ -284,7 +284,7 @@ Infrastructure is deployed via the `zoa-jobs` Helm chart at `argocd/config/share | RC | ApplicationSet → `zoa-jobs` chart | Namespace `zoa-jobs`, static SAs | | MC | ApplicationSet → `zoa-jobs` chart | Namespace `zoa-jobs`, static SAs (execution target) | -ManifestWork is used **only** as transport for TA executions (Job + per-execution RBAC + ConfigMap). +The Manifest CR / ApplyDesire pipeline is used **only** as transport for TA executions (Job + per-execution RBAC + ConfigMap). ### Job Image @@ -334,7 +334,10 @@ All `POST /trusted-actions/{action}/run` calls require a `jira` field: { "target_cluster": "mc-useast1-1", "jira": "ROSAENG-1234", - "params": { "namespace": "maestro", "name": "maestro-abc-123" }, + "params": { + "namespace": "hyperfleet", + "name": "hyperfleet-operator-abc-123" + }, "force": false, "dry_run": false } @@ -413,8 +416,8 @@ The API proxies S3 content directly — no presigned URLs exposed to consumers. | Status | Meaning | | ----------- | ---------------------------------------------------------------------- | -| `pending` | Execution created, ManifestWork dispatched but not yet applied | -| `running` | ManifestWork applied, Job running on target cluster | +| `pending` | Execution created, Manifest CR persisted but not yet applied to target | +| `running` | ApplyDesire written, Job running on target cluster | | `succeeded` | Job completed successfully (exit 0) | | `failed` | Job failed (non-zero exit) | | `timed_out` | Execution exceeded per-TA or global timeout — reconciler force-cleaned | @@ -657,26 +660,26 @@ $ zoa run get_nodes -t eph-bc5fee45-mc01 --jira ROSAENG-1234 ] # 3. Fetch a single resource by name -$ zoa run get_pods -t eph-bc5fee45-mc01 -n maestro --name maestro-abc-123 --jira ROSAENG-1234 +$ zoa run get_pods -t eph-bc5fee45-mc01 -n hyperfleet --name hyperfleet-operator-abc-123 --jira ROSAENG-1234 # 4. Pipe to jq for further filtering $ zoa run get_pods -t eph-bc5fee45-mc01 -A --jira ROSAENG-1234 | jq '.[] | select(.restarts > 5)' $ zoa run get_pods -t eph-bc5fee45-mc01 -A --jira ROSAENG-1234 | jq '.[] | select(.status != "Running")' # 5. Filters -$ zoa run get_pods -t eph-bc5fee45-mc01 -n maestro -l app=maestro --jira ROSAENG-1234 +$ zoa run get_pods -t eph-bc5fee45-mc01 -n hyperfleet -l app=hyperfleet-operator --jira ROSAENG-1234 $ zoa run get_pods -t eph-bc5fee45-mc01 -A --jira ROSAENG-1234 $ zoa run get_resource -t eph-bc5fee45-mc01 --resource hostedclusters -A --jira ROSAENG-1234 # 6. Write operations -$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n maestro --name maestro --jira ROSAENG-1234 -$ zoa run delete_pod -t eph-bc5fee45-mc01 -n maestro --name maestro-xyz --jira ROSAENG-1234 +$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n hyperfleet --name hyperfleet-operator --jira ROSAENG-1234 +$ zoa run delete_pod -t eph-bc5fee45-mc01 -n hyperfleet --name hyperfleet-operator-xyz --jira ROSAENG-1234 # 7. Dry-run preview before a write -$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n maestro --name maestro --dry-run --jira ROSAENG-1234 +$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n hyperfleet --name hyperfleet-operator --dry-run --jira ROSAENG-1234 # 8. Force bypass write cooldown -$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n maestro --name maestro --force --jira ROSAENG-1234 +$ zoa run rollout_restart -t eph-bc5fee45-mc01 -n hyperfleet --name hyperfleet-operator --force --jira ROSAENG-1234 # 9. On failure, logs are shown automatically (stderr) $ zoa run get_pods -t eph-bc5fee45-mc01 -n invalid --jira ROSAENG-1234 @@ -782,32 +785,41 @@ The `force: true` flag bypasses both write cooldown and max concurrent checks: ### Dispatch Flow (Two-Job Architecture) -``` -Operator (zoa run) → Platform API → Maestro (gRPC CreateManifestWork) → Maestro Agent → Target Cluster - │ - Applies ManifestWork: - SA, RBAC, ConfigMaps, Jobs - │ - ┌─────────────────┴────────────────────┐ - │ │ - Runner Job Uploader Job - (per-exec SA) (static SA: zoa-uploader) - │ │ - /zoa/entrypoint.sh Poll runner (1s loop) - (tee → execution.log) │ - │ Read output ConfigMap - Patch output ConfigMap Decode base64 → files - (base64: log + output) │ - │ aws s3 cp → S3 bucket - Exit Exit - │ │ - └──────────────────┬───────────────────┘ - │ -Platform API Reconciler (5s loop): │ - ← Maestro (GetManifestWork) ← feedbackRules (succeeded/failed + Job timestamps) ←───────────┘ - → Compute: runner_seconds, upload_seconds, duration_seconds - → Delete ResourceBundle (on terminal status, race-safe → cascades cleanup on target cluster) - → DynamoDB (status, durations, output_status, revision, updated_at) +```text +Operator (zoa run) → Platform API → creates Manifest CR in PostgreSQL (hyperfleet-db) + │ + hyperfleet-operator (ManifestReconciler) + │ + writes ApplyDesire to DynamoDB + │ + kube-applier → Target Cluster + │ + Applies resources: + SA, RBAC, ConfigMaps, Jobs + │ + ┌─────────────────┴────────────────────┐ + │ │ + Runner Job Uploader Job + (per-exec SA) (static SA: zoa-uploader) + │ │ + /zoa/entrypoint.sh Poll runner (1s loop) + (tee → execution.log) │ + │ Read output ConfigMap + Patch output ConfigMap Decode base64 → files + (base64: log + output) │ + │ aws s3 cp → S3 bucket + Exit Exit + │ │ + └──────────────────┬───────────────────┘ + │ + ▼ + Platform API Reconciler (5s loop): + ← Manifest CR status (via DynamoDB Streams → operator) + → Reads: succeeded/failed + Job timestamps + → Compute: runner_seconds, upload_seconds, duration_seconds + → Delete Manifest CR (terminal status → operator removes + ApplyDesire → kube-applier cascades cleanup) + → DynamoDB (status, durations, output_status, revision) ``` ### TA Versioning @@ -818,7 +830,7 @@ Platform API Reconciler (5s loop): ## Alternatives Considered -1. **Per-execution ServiceAccount with dynamic Pod Identity**: Each TA execution creates its own SA and wires Pod Identity dynamically. Rejected because EKS Pod Identity requires Terraform/API calls per SA (cannot be done from within a ManifestWork), adding minutes of latency and significant IAM complexity. +1. **Per-execution ServiceAccount with dynamic Pod Identity**: Each TA execution creates its own SA and wires Pod Identity dynamically. Rejected because EKS Pod Identity requires Terraform/API calls per SA (cannot be done from within a Manifest CR), adding minutes of latency and significant IAM complexity. 2. **Single shared ServiceAccount**: One SA (`zoa-job-runner`) for all TAs. Rejected because Kubernetes audit logs only show SA identity — all TAs would be indistinguishable at the K8s audit level. Additionally, a shared SA bound to N possible Roles means parallel executions share permissions — any running TA would have access to RBAC granted for a different concurrent TA. @@ -826,12 +838,12 @@ Platform API Reconciler (5s loop): 4. **Sidecar container for S3 upload**: A separate container watches `/artifacts` and uploads. Rejected because sidecars add complexity around container ordering and completion detection. Additionally, containers in the same Pod share the same ServiceAccount — the runner would inherit S3 write permissions, breaking the isolation between operational actions and output transport. -5. **Full ManifestWork templates (Job + RBAC defined by TA author)**: TA authors define the entire ManifestWork content including Job spec. Rejected because it couples boilerplate (image, volumes, resources, entrypoint) to each TA, requiring all TAs to be updated when infrastructure changes (e.g., image bump). +5. **Full Manifest CR templates (Job + RBAC defined by TA author)**: TA authors define the entire Manifest CR content including Job spec. Rejected because it couples boilerplate (image, volumes, resources, entrypoint) to each TA, requiring all TAs to be updated when infrastructure changes (e.g., image bump). ## Design Rationale - **Justification**: The split SA model (per-execution runner + static uploader/AWS SAs) balances auditability, operational simplicity, and Pod Identity constraints. Separating TA authoring (script + RBAC) from execution boilerplate (image, wrapper, resources) enables independent evolution of each concern. -- **Evidence**: Maestro is the current transport layer for ManifestWork dispatch across ROSA HCP v2 (hyperfleet), ARO-HCP, and GCP-HCP — a proven mechanism at scale. The `openshift/managed-scripts` project validates the "swiss knife image + script" pattern for OSD/ROSA operations. +- **Evidence**: kube-applier is the proven transport layer for resource distribution across ROSA HCP v2 (hyperfleet), using DynamoDB as the durable desire store and DynamoDB Streams for status feedback. The `openshift/managed-scripts` project validates the "swiss knife image + script" pattern for OSD/ROSA operations. - **Comparison**: Per-execution runner SAs provide execution-level K8s audit attribution. Static AWS and uploader SAs satisfy Pod Identity constraints while keeping IAM association count bounded. Rich labels on all resources enable correlation via kube audit logs. ## Consequences @@ -871,8 +883,8 @@ Platform API Reconciler (5s loop): ### Reliability: - **Scalability**: Stable SAs and ArgoCD-managed infra support thousands of concurrent executions. DynamoDB uses a `status-index` GSI for efficient reconciler queries (no full-table scans) -- **Observability**: DynamoDB provides queryable execution history; S3 stores execution logs and output; ManifestWork status provides real-time job state -- **Resiliency**: Reconciler uses race-safe ordering (delete RB before status update) to prevent stale resources. Per-TA and global timeouts prevent stuck executions. Logs are uploaded unconditionally to S3 before Job exits. +- **Observability**: DynamoDB provides queryable execution history; S3 stores execution logs and output; Manifest CR status provides real-time job state +- **Resiliency**: Reconciler uses race-safe ordering (delete Manifest CR before status update) to prevent stale resources. Per-TA and global timeouts prevent stuck executions. Logs are uploaded unconditionally to S3 before Job exits. - **Timeout handling**: Executions exceeding their timeout are marked `timed_out` (distinct from `failed`), RB is deleted, and the full duration is recorded ### Cost: @@ -895,5 +907,4 @@ Platform API Reconciler (5s loop): ## Related Documentation - [ZOA Framework (Sections 1-9)](https://redhat.atlassian.net/browse/ROSA-672) — Approved layered model and access matrix -- [Maestro MQTT Resource Distribution](./maestro-mqtt-resource-distribution.md) — How ManifestWorks are dispatched - [openshift/managed-scripts](https://github.com/openshift/managed-scripts) — Reference for script execution pattern and job image diff --git a/docs/development-environment.md b/docs/development-environment.md index dd2138b9e..a9186f34d 100644 --- a/docs/development-environment.md +++ b/docs/development-environment.md @@ -226,7 +226,7 @@ The bastion task stays running until explicitly stopped or until the environment ## Port Forwarding -Forward ports from cluster-internal services to your local machine through the bastion, without needing an interactive shell. This is useful for accessing ArgoCD, Prometheus, and Maestro UIs directly in your browser. +Forward ports from cluster-internal services to your local machine through the bastion, without needing an interactive shell. This is useful for accessing ArgoCD and Prometheus UIs directly in your browser. > ⚠️ _Bastion must be enabled in your environment config (`enable_bastion: true` in `defaults.yaml`). The default ephemeral preset already has it enabled._ @@ -255,16 +255,15 @@ make ephemeral-port-forward-mc-all Available services per cluster type: -| Service | RC | MC | Local address | -| ------------ | --- | --- | --------------------------------------------------- | -| Maestro | yes | no | http://localhost:8080 (HTTP), localhost:8090 (gRPC) | -| ArgoCD | yes | yes | https://localhost:8443 | -| Prometheus | yes | yes | http://localhost:9090 | -| Thanos Query | yes | no | http://localhost:10902 | -| Thanos Ruler | yes | no | http://localhost:10903 | -| Loki | yes | no | http://localhost:13100 | -| Alertmanager | yes | no | http://localhost:9093 | -| Grafana | yes | no | http://localhost:3000 | +| Service | RC | MC | Local address | +| ------------ | --- | --- | ---------------------- | +| ArgoCD | yes | yes | https://localhost:8443 | +| Prometheus | yes | yes | http://localhost:9090 | +| Thanos Query | yes | no | http://localhost:10902 | +| Thanos Ruler | yes | no | http://localhost:10903 | +| Loki | yes | no | http://localhost:13100 | +| Alertmanager | yes | no | http://localhost:9093 | +| Grafana | yes | no | http://localhost:3000 | The command fetches the ArgoCD admin password automatically and prints it to the terminal. Port forwards remain active until you press `Ctrl+C`. @@ -300,25 +299,27 @@ make ephemeral-e2e ID=6bd2d3d7 E2E_SKIP_CLEANUP=1 This skips both the cleanup-labeled ginkgo specs and the `DeferCleanup` safety net, so the HCP cluster, VPC, IAM, and OIDC resources survive for investigation. Remember to tear them down manually afterwards with `make ephemeral-teardown` or by re-running without the flag. -## Collect Cluster Logs +## Dump Environment -Collect kubernetes diagnostic logs (`oc adm inspect`) from the RC and/or MC clusters in an ephemeral environment. Logs are gathered by a dedicated log-collector ECS task, uploaded to S3, and downloaded locally. +Collect Kubernetes diagnostic logs (`oc adm inspect`) and PostgreSQL database state from the RC and/or MC clusters in an ephemeral environment. Data is gathered by the log-collector ECS Fargate task, uploaded to S3, and downloaded locally. + +For the RC, this also dumps the `kubernetes_resources` table from Aurora PostgreSQL — producing a tabular summary (`resource-summary.txt`) and per-resource JSON files under `db-state/resources//.json`. ```bash # Collect from both RC and all MCs -make ephemeral-collect-logs +make ephemeral-dump-env # Collect from RC only -make ephemeral-collect-logs CLUSTER=rc +make ephemeral-dump-env CLUSTER=rc # Collect from MCs only -make ephemeral-collect-logs CLUSTER=mc +make ephemeral-dump-env CLUSTER=mc # Explicit environment selection -make ephemeral-collect-logs ID=6bd2d3d7 +make ephemeral-dump-env ID=6bd2d3d7 ``` -Output is written to `/tmp/-logs-/`. In CI, logs are automatically collected on e2e test failure with `S3_ONLY=true` — logs are left in S3 (to avoid publishing sensitive data) and the S3 URIs are printed for manual retrieval. +Output is written to `/tmp/-logs-/`. The RC output includes a `db-state/` subdirectory with the database dump. In CI, data is automatically collected on e2e test failure with `S3_ONLY=true` — results are left in S3 (to avoid publishing sensitive data) and the S3 URIs are printed for manual retrieval. > ⚠️ _Bastion must be enabled in your environment config (`enable_bastion: true` in `defaults.yaml`). The default ephemeral preset already has it enabled._ @@ -369,3 +370,4 @@ make ephemeral-teardown ID=6bd2d3d7 - [Milestone 2 slides](presentations/milestone-2/slides.md) -- ephemeral provider architecture and how environments are provisioned/torn down - [ci/ephemeral-provider/README.md](../ci/ephemeral-provider/README.md) -- ephemeral provider internals +- [SRE UI Access — Integration](sop/sre-ui-access.md) -- accessing Grafana, ArgoCD, Prometheus, Thanos, and Loki in the integration environment diff --git a/docs/environment-provisioning.md b/docs/environment-provisioning.md index 3ea4cd691..3b0b89674 100644 --- a/docs/environment-provisioning.md +++ b/docs/environment-provisioning.md @@ -1,6 +1,6 @@ # Provision a New Environment -Set up a central pipeline that provisions Regional and Management Clusters with ArgoCD and Maestro connectivity. +Set up a central pipeline that provisions Regional and Management Clusters with ArgoCD and kube-applier connectivity. --- @@ -167,10 +167,7 @@ Expected output: ``` NAMESPACE NAME SYNC STATUS HEALTH STATUS argocd argocd Synced Healthy -argocd hyperfleet-adapter1 Synced Healthy -argocd hyperfleet-api Synced Healthy -argocd hyperfleet-sentinel Synced Healthy -argocd maestro-server Synced Healthy +argocd hyperfleet Synced Healthy argocd monitoring Synced Healthy argocd platform-api Synced Healthy argocd root Synced Healthy @@ -190,7 +187,7 @@ NAMESPACE NAME SYNC STATUS HEALTH STATUS argocd argocd Synced Healthy argocd cert-manager Synced Healthy argocd hypershift Synced Healthy -argocd maestro-agent Synced Healthy +argocd kube-applier Synced Healthy argocd monitoring Synced Healthy argocd root Synced Healthy argocd storageclass Synced Healthy @@ -216,17 +213,6 @@ terraform output -raw api_test_command > **Note:** The API Gateway accepts requests from any authenticated AWS principal. Authorization is enforced by the Platform API backend — only accounts registered with the Platform API (starting with the bootstrap account) receive a successful response. -### 4.4 Verify Maestro Connectivity - -From the Regional account, verify IoT certificates are active: - -```bash -export AWS_PROFILE= - -aws iot describe-endpoint --endpoint-type iot:Data-ATS -aws iot list-certificates | jq -r '.certificates[].status' -``` - --- ## Appendix diff --git a/docs/hostedcluster-teardown.md b/docs/hostedcluster-teardown.md index 2e8ba15e5..40fbf580e 100644 --- a/docs/hostedcluster-teardown.md +++ b/docs/hostedcluster-teardown.md @@ -17,7 +17,7 @@ The full teardown flow is: 4. Wait for the HostedCluster and NodePool to be removed from the MC 5. Delete the CloudFormation stacks in the customer AWS account -### Step 1 & 2 — Clean Up the CLM Database (Regional Cluster) +### Step 1 & 2 — Clean Up the hyperfleet-db Database (Regional Cluster) Open a bastion session to the Regional Cluster: @@ -29,13 +29,13 @@ make int-bastion-rc make ephemeral-bastion-rc ``` -Once connected, run the cleanup script below. The script connects to the CLM +Once connected, run the cleanup script below. The script connects to the hyperfleet-db database, lists all clusters, and lets you choose to delete a single cluster or all clusters. It removes records from both the `clusters` and `adapter_statuses` tables.
-cleanup-clm-db.sh +cleanup-hyperfleet-db.sh ```bash #!/bin/bash @@ -146,7 +146,7 @@ ADAPTER_TABLE_EXISTS=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NA "SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = 'public' AND table_name = 'adapter_statuses');") echo "" -print_info "WARNING: This will delete $DELETE_DESC from the CLM database!" +print_info "WARNING: This will delete $DELETE_DESC from the hyperfleet-db database!" if [ "$ADAPTER_TABLE_EXISTS" = "t" ]; then print_info "This will also delete matching records from the adapter_statuses table." fi @@ -228,7 +228,7 @@ done ### Step 4 — Wait for HostedCluster and NodePool Removal (Management Cluster) -After deleting the resource bundles, Maestro will propagate the deletion to the +After deleting the resource bundles, kube-applier will propagate the deletion to the Management Cluster. Wait for the HostedCluster and NodePool resources to be fully removed before proceeding. diff --git a/docs/sop/rebuild-integration.md b/docs/sop/rebuild-integration.md index 3dcf5d8ba..248692a1e 100644 --- a/docs/sop/rebuild-integration.md +++ b/docs/sop/rebuild-integration.md @@ -61,10 +61,10 @@ sequenceDiagram ### Step 3: Clean up leftover resources -After both teardowns complete, three Secrets Manager secrets in the MC account will remain scheduled for deletion with a 30-day recovery window. These block re-provision because Terraform cannot create a secret while another with the same name is pending deletion. Force-delete them: +After both teardowns complete, the HyperShift config secret in the MC account will remain scheduled for deletion with a 30-day recovery window. This blocks re-provision because Terraform cannot create a secret while another with the same name is pending deletion. Force-delete it: ```bash -for secret in mc01-maestro-agent-cert mc01-maestro-agent-config hypershift/mc01-config; do +for secret in hypershift/mc01-config; do aws secretsmanager delete-secret --secret-id "$secret" --force-delete-without-recovery --region us-east-1 done ``` diff --git a/docs/sop/sre-ui-access.md b/docs/sop/sre-ui-access.md new file mode 100644 index 000000000..60ea2a7ef --- /dev/null +++ b/docs/sop/sre-ui-access.md @@ -0,0 +1,82 @@ +# SRE UI Access — Integration Environment + +Browser-based access to Grafana, ArgoCD, Prometheus, and Thanos for the +integration environment via the SRE UI ALB. + +## Prerequisites + +1. **Red Hat VPN** — required for the corporate proxy to function. +2. **Browser proxy configured** — traffic to `*.sre.us-east-1.int0.rosa.devshift.net` + must route through `squid.corp.redhat.com:3128`. See + [Browser proxy configuration](#browser-proxy-configuration) below. +3. **Red Hat employee account** — authentication via Red Hat SSO + (`auth.stage.redhat.com`). + +## URLs + +| Tool | URL | Purpose | +| ---------- | ------------------------------------------------------- | ---------------------------- | +| Grafana | https://grafana.sre.us-east-1.int0.rosa.devshift.net | Metrics dashboards and logs | +| ArgoCD | https://argocd.sre.us-east-1.int0.rosa.devshift.net | GitOps application status | +| Prometheus | https://prometheus.sre.us-east-1.int0.rosa.devshift.net | Raw metric queries (RC) | +| Thanos | https://thanos.sre.us-east-1.int0.rosa.devshift.net | Aggregated metrics (RC + MC) | + +## Authentication + +On first visit the browser redirects to Red Hat SSO. Authenticate with your Red +Hat credentials or via Kerberos SSO. Sessions last 8 hours. + +## Access levels + +| Tool | Access level after login | +| ---------- | ------------------------ | +| Grafana | Read-only (Viewer) | +| ArgoCD | Read-only | +| Prometheus | Read-only | +| Thanos | Read-only | + +## Browser proxy configuration + +The ALB only accepts traffic from Red Hat corporate proxy egress IPs. Configure +your browser to proxy `*.sre.us-east-1.int0.rosa.devshift.net` through +`squid.corp.redhat.com:3128`. The proxy is only reachable from the Red Hat VPN. + +### Chrome — ZeroOmega + +Install [ZeroOmega](https://chromewebstore.google.com/detail/proxy-switchyomega-3-zero/pfnededegaaopdmhkdmcofjmoldfiped) +from the Chrome Web Store. + +1. Open the ZeroOmega options panel. +2. Create a new proxy profile named **hyperfleet-sre**: + - Protocol: `HTTP` + - Server: `squid.corp.redhat.com` + - Port: `3128` +3. In the **Auto Switch** profile add a condition: + - Condition type: `Host wildcard` + - Condition details: `*.sre.us-east-1.int0.rosa.devshift.net` + - Profile: `hyperfleet-sre` +4. Click **Apply changes** and activate the **Auto Switch** profile. + +### Firefox — FoxyProxy + +Install [FoxyProxy](https://addons.mozilla.org/firefox/addon/foxyproxy-standard/). + +1. Open FoxyProxy options → **Proxies** → **Add**. +2. Configure the proxy: + - Title: `hyperfleet-sre` + - Type: `HTTP` + - Hostname: `squid.corp.redhat.com` + - Port: `3128` +3. Under **URL Patterns** add: + - Pattern: `*.sre.us-east-1.int0.rosa.devshift.net` + - Type: `Wildcard` +4. Save and enable FoxyProxy. + +## Troubleshooting + +**500 after SSO redirect** — The AWS Application LB cannot reach the OIDC Provider (auth.stage.redhat.com) to get the SSO token. + +**503 Service Unavailable** — The target group has no healthy targets. Check the cluster TargetGroupBindings and pod health on the RC cluster (make int-bastion-rc). + +**Proxy connection refused** — You are not connected to the Red Hat VPN. +Connect to VPN and retry. diff --git a/scripts/bootstrap-argocd.sh b/scripts/bootstrap-argocd.sh index 5087d9365..259260b27 100755 --- a/scripts/bootstrap-argocd.sh +++ b/scripts/bootstrap-argocd.sh @@ -79,9 +79,15 @@ if [[ "$CLUSTER_TYPE" == "regional-cluster" ]]; then SRE_ARGOCD_TARGET_GROUP_ARN=$(echo "$OUTPUTS" | jq -r '.sre_argocd_target_group_arn.value // ""') SRE_PROMETHEUS_TARGET_GROUP_ARN=$(echo "$OUTPUTS" | jq -r '.sre_prometheus_target_group_arn.value // ""') SRE_THANOS_TARGET_GROUP_ARN=$(echo "$OUTPUTS" | jq -r '.sre_thanos_target_group_arn.value // ""') - SRE_LOKI_TARGET_GROUP_ARN=$(echo "$OUTPUTS" | jq -r '.sre_loki_target_group_arn.value // ""') SRE_ALB_DNS_NAME=$(echo "$OUTPUTS" | jq -r '.sre_alb_dns_name.value // ""') SRE_DOMAIN=$(echo "$OUTPUTS" | jq -r '.sre_domain.value // ""') + _REDIS_HOST=$(echo "$OUTPUTS" | jq -r '.hyperfleet_redis_endpoint.value // ""') + _REDIS_PORT=$(echo "$OUTPUTS" | jq -r '.hyperfleet_redis_port.value // ""') + if [[ -n "$_REDIS_HOST" && -n "$_REDIS_PORT" ]]; then + REDIS_ENDPOINT="${_REDIS_HOST}:${_REDIS_PORT}" + else + REDIS_ENDPOINT="" + fi else API_TARGET_GROUP_ARN="" THANOS_TARGET_GROUP_ARN="" @@ -97,9 +103,9 @@ else SRE_ARGOCD_TARGET_GROUP_ARN="" SRE_PROMETHEUS_TARGET_GROUP_ARN="" SRE_THANOS_TARGET_GROUP_ARN="" - SRE_LOKI_TARGET_GROUP_ARN="" SRE_ALB_DNS_NAME="" SRE_DOMAIN="" + REDIS_ENDPOINT="" fi RHOBS_API_URL="${RHOBS_API_URL:-}" @@ -141,9 +147,9 @@ RUN_TASK_OUTPUT=$(aws ecs run-task \ {\"name\": \"SRE_ARGOCD_TARGET_GROUP_ARN\", \"value\": \"$SRE_ARGOCD_TARGET_GROUP_ARN\"}, {\"name\": \"SRE_PROMETHEUS_TARGET_GROUP_ARN\", \"value\": \"$SRE_PROMETHEUS_TARGET_GROUP_ARN\"}, {\"name\": \"SRE_THANOS_TARGET_GROUP_ARN\", \"value\": \"$SRE_THANOS_TARGET_GROUP_ARN\"}, - {\"name\": \"SRE_LOKI_TARGET_GROUP_ARN\", \"value\": \"$SRE_LOKI_TARGET_GROUP_ARN\"}, {\"name\": \"SRE_ALB_DNS_NAME\", \"value\": \"$SRE_ALB_DNS_NAME\"}, - {\"name\": \"SRE_DOMAIN\", \"value\": \"$SRE_DOMAIN\"} + {\"name\": \"SRE_DOMAIN\", \"value\": \"$SRE_DOMAIN\"}, + {\"name\": \"REDIS_ENDPOINT\", \"value\": \"$REDIS_ENDPOINT\"} ] }] }" 2>&1) diff --git a/scripts/build-platform-image.sh b/scripts/build-platform-image.sh index ca0e92494..62270c7da 100755 --- a/scripts/build-platform-image.sh +++ b/scripts/build-platform-image.sh @@ -118,7 +118,7 @@ echo "" # Build the image echo "Building platform image from ${DOCKERFILE}..." -$CONTAINER_RUNTIME build --platform linux/amd64 -t "${ECR_URL}:${IMAGE_TAG}" "$DOCKERFILE_DIR" +$CONTAINER_RUNTIME build --pull --platform linux/amd64 -t "${ECR_URL}:${IMAGE_TAG}" "$DOCKERFILE_DIR" echo "" # Push the image diff --git a/scripts/buildspec/provision-infra-mc.sh b/scripts/buildspec/provision-infra-mc.sh index 26c025efc..82f8ef456 100755 --- a/scripts/buildspec/provision-infra-mc.sh +++ b/scripts/buildspec/provision-infra-mc.sh @@ -80,11 +80,12 @@ else fi export TF_VAR_oidc_cloudfront_domain TF_VAR_oidc_bucket_name TF_VAR_oidc_bucket_arn TF_VAR_oidc_bucket_region TF_VAR_rhobs_api_url - # ZOA outputs bucket ARN - export TF_VAR_zoa_outputs_bucket_arn=$(cd "$_RC_TF_DIR" && terraform output -raw zoa_bucket_arn 2>/dev/null || echo "") + # ZOA outputs bucket ARN — validate output looks like an ARN to avoid + # capturing terraform warnings as the value (non-ASCII chars break IAM policies) + export TF_VAR_zoa_outputs_bucket_arn=$(cd "$_RC_TF_DIR" && terraform output -raw zoa_bucket_arn 2>/dev/null | grep -E '^arn:' || echo "") # ZOA KMS key ARN (optional — for S3 SSE-KMS cross-account access) - export TF_VAR_zoa_kms_key_arn=$(cd "$_RC_TF_DIR" && terraform output -raw zoa_kms_key_arn 2>/dev/null || echo "") + export TF_VAR_zoa_kms_key_arn=$(cd "$_RC_TF_DIR" && terraform output -raw zoa_kms_key_arn 2>/dev/null | grep -E '^arn:' || echo "") fi # ── Phase 2: Apply/Destroy MC infrastructure ───────────────────────────────── diff --git a/scripts/buildspec/provision-infra-rc.sh b/scripts/buildspec/provision-infra-rc.sh index 49bc4fdcd..f59e7c7f8 100755 --- a/scripts/buildspec/provision-infra-rc.sh +++ b/scripts/buildspec/provision-infra-rc.sh @@ -75,6 +75,7 @@ fi export TF_VAR_container_image="${PLATFORM_IMAGE}" export TF_VAR_enable_bastion="${ENABLE_BASTION}" +export TF_VAR_hyperfleet_db_deletion_protection=$(parseBool '.hyperfleet_db_deletion_protection' true "$DEPLOY_CONFIG_FILE") export TF_VAR_enable_cloudtrail=$(parseBool '.enable_cloudtrail' false "$DEPLOY_CONFIG_FILE") export TF_VAR_enable_api_custom_domain=$(parseBool '.enable_api_custom_domain' false "$DEPLOY_CONFIG_FILE") export TF_VAR_zone_shard_count=$(jq -r '.zone_shard_count // 1' "$DEPLOY_CONFIG_FILE") @@ -112,10 +113,8 @@ if [ "$TF_VAR_enable_sre_oidc_auth" = "true" ]; then export TF_VAR_sre_prometheus_oidc_client_id TF_VAR_sre_thanos_oidc_client_id=$(jq -r '.sre_thanos_oidc_client_id // ""' "$DEPLOY_CONFIG_FILE") export TF_VAR_sre_thanos_oidc_client_id - TF_VAR_sre_loki_oidc_client_id=$(jq -r '.sre_loki_oidc_client_id // ""' "$DEPLOY_CONFIG_FILE") - export TF_VAR_sre_loki_oidc_client_id - for svc in grafana argocd prometheus thanos loki; do + for svc in grafana argocd prometheus thanos; do secret=$(aws secretsmanager get-secret-value \ --secret-id "sre-ui-alb/${svc}/oidc-client-secret" \ --region "${TARGET_REGION}" \ diff --git a/scripts/dev/collect-cluster-logs.sh b/scripts/dev/dump-env.sh similarity index 61% rename from scripts/dev/collect-cluster-logs.sh rename to scripts/dev/dump-env.sh index 274c469c1..21f21dac3 100755 --- a/scripts/dev/collect-cluster-logs.sh +++ b/scripts/dev/dump-env.sh @@ -1,8 +1,10 @@ #!/bin/bash -# Collect RC and MC kubernetes logs via the log-collector ECS task. +# Must-gather for the regional platform: collects Kubernetes logs from RC and +# MC clusters plus the PostgreSQL database state from the RC, all via the +# log-collector ECS Fargate task. # -# This script is the single implementation for log collection, used by both -# the local dev CLI (ephemeral-env.sh, int-env.sh) and CI (ci/e2e-tests.sh). +# This script is the single implementation used by both the local dev CLI +# (ephemeral-env.sh, int-env.sh) and CI (ci/e2e-tests.sh). # # Callers set CLUSTER_PREFIX to control cluster name resolution: # - Ephemeral: CLUSTER_PREFIX="eph-a1b2c3-" → eph-a1b2c3-regional, eph-a1b2c3-mc01 @@ -12,7 +14,7 @@ # ${CLUSTER_PREFIX}mc*-bastion, so mc01, mc02, etc. are all collected. # # Usage: -# collect-cluster-logs.sh [regional|management|all] +# dump-env.sh [regional|management|all] # # Required environment variables: # CLUSTER_PREFIX — Cluster name prefix (e.g. "ci-a1b2c3-" or "" for bare names) @@ -27,6 +29,8 @@ # LEAKTK_GATE — Defaults to "true": abort with non-zero exit when leaktk # detects secrets remaining after redaction. Set to "false" # to log findings as warnings without blocking. +# DB_NAMESPACE — Kubernetes namespace for the DB DSN secret (default: hyperfleet) +# DB_SECRET_NAME — Name of the secret containing the DSN (default: hyperfleet-db-dsn) # # All collection failures are logged but do not cause a non-zero exit, so # this script is safe to call from test failure handlers. @@ -37,6 +41,8 @@ export AWS_REGION="${AWS_REGION:-${AWS_DEFAULT_REGION:-$(aws configure get regio RC_NAMESPACES="all" MC_NAMESPACES="all" +DB_NAMESPACE="${DB_NAMESPACE:-hyperfleet}" +DB_SECRET_NAME="${DB_SECRET_NAME:-hyperfleet-db-dsn}" # --------------------------------------------------------------------------- # Helpers @@ -171,28 +177,190 @@ discover_mc_clusters() { } # --------------------------------------------------------------------------- -# Core: collect logs for one cluster +# Build the ECS command for a cluster dump. +# +# For the RC (include_db=true): overrides the default log-collector command +# with a combined script that does k8s log collection AND DB state dump in +# one ECS task, producing a single tarball. +# +# For MCs (include_db=false): returns empty — the caller uses env-only +# overrides and the task definition's built-in command handles k8s logs. # --------------------------------------------------------------------------- -collect_logs_for_cluster() { +build_dump_command() { + local include_db="$1" + local db_namespace="$2" + local db_secret_name="$3" + + cat </dev/null || true & + batch=\$((batch + 1)) + if [[ \$batch -ge 5 ]]; then + wait + batch=0 + fi +done +wait +EOFCMD + + if [[ "$include_db" == "true" ]]; then + cat < /tmp/inspect-logs/db-state/resource-summary.txt; then + echo "WARNING: resource summary query failed (see above); continuing" + fi + + echo "Dumping individual resources..." + if psql "\$DSN" -At -F \$'\\t' -c " + SELECT split_part(gvk, '/', 3), + name, + jsonb_build_object( + 'apiVersion', split_part(gvk, '/', 1) || '/' || split_part(gvk, '/', 2), + 'kind', split_part(gvk, '/', 3), + 'metadata', jsonb_build_object( + 'name', name, + 'namespace', namespace, + 'uid', uid, + 'resourceVersion', object_version, + 'creationTimestamp', created_at, + 'deletionTimestamp', deletion_timestamp + ) || COALESCE(metadata, '{}'::jsonb), + 'spec', COALESCE(spec, '{}'::jsonb), + 'status', COALESCE(status, '{}'::jsonb) + )::text + FROM kubernetes_resources + ORDER BY gvk, namespace, name; + " | while IFS=\$'\\t' read -r kind rname json; do + mkdir -p "/tmp/inspect-logs/db-state/resources/\$kind" + echo "\$json" | jq '.' > "/tmp/inspect-logs/db-state/resources/\$kind/\$rname.json" + done; then + echo " Dumped \$(find /tmp/inspect-logs/db-state/resources -name '*.json' 2>/dev/null | wc -l) resources" + else + echo "WARNING: individual resource dump failed (see above); continuing" + fi +fi +EOFDB + fi + + cat <<'EOFTAIL' + +# --- Upload --- + +echo "" +echo "Uploading to S3..." +tar czf /tmp/inspect-logs.tar.gz -C /tmp inspect-logs +aws s3 cp /tmp/inspect-logs.tar.gz "s3://$S3_BUCKET/$S3_KEY" + +echo "Done." +EOFTAIL +} + +# --------------------------------------------------------------------------- +# Core: dump a single cluster via the log-collector ECS task +# --------------------------------------------------------------------------- + +dump_cluster() { local cluster_id="$1" local namespaces="$2" local out_dir="$3" + local include_db="${4:-false}" - echo "==> Collecting logs from ${cluster_id}..." + echo "==> Dumping ${cluster_id}..." local ecs_cluster="${cluster_id}-bastion" local task_def="${cluster_id}-log-collector" local account_id region account_id=$(aws sts get-caller-identity --query Account --output text) \ || { echo " Could not determine account ID"; return 1; } - local region="${AWS_REGION}" + region="${AWS_REGION}" local s3_bucket="bastion-log-collection-${account_id}-${region}-an" ensure_logs_bucket "$account_id" "$region" - local s3_key="collect-logs-$(date +%s%N)-$$-${RANDOM}.tar.gz" + local s3_key + s3_key="dump-env-$(date +%s%N)-$$-${RANDOM}.tar.gz" - # Discover network config from the bastion security group local sg_id subnets vpc_id sg_id=$(aws ec2 describe-security-groups \ --filters "Name=group-name,Values=${cluster_id}-bastion" \ @@ -211,28 +379,38 @@ collect_logs_for_cluster() { | tr '\t' ',') \ || { echo " Could not find private subnets for ${cluster_id}"; return 1; } - # Launch the log-collector task with namespace and S3 key overrides - echo " Launching log-collector task..." - local task_arn + local overrides_json + overrides_json=$(jq -n \ + --arg bucket "$s3_bucket" \ + --arg ns "$namespaces" \ + --arg key "$s3_key" \ + '{containerOverrides: [{ + name: "log-collector", + environment: [ + {name: "S3_BUCKET", value: $bucket}, + {name: "INSPECT_NAMESPACES", value: $ns}, + {name: "S3_KEY", value: $key} + ] + }]}') + + if [[ "$include_db" == "true" ]]; then + local ecs_command + ecs_command=$(build_dump_command "true" "$DB_NAMESPACE" "$DB_SECRET_NAME") + overrides_json=$(echo "$overrides_json" | jq \ + --arg cmd "$ecs_command" \ + '.containerOverrides[0].command = [$cmd]') + fi + + echo " Launching dump task..." local run_task_output run_task_output=$(AWS_PAGER="" aws ecs run-task \ --cluster "$ecs_cluster" \ --task-definition "$task_def" \ --launch-type FARGATE \ --network-configuration "awsvpcConfiguration={subnets=[$subnets],securityGroups=[$sg_id],assignPublicIp=DISABLED}" \ - --overrides "{ - \"containerOverrides\": [{ - \"name\": \"log-collector\", - \"environment\": [ - {\"name\": \"S3_BUCKET\", \"value\": \"$s3_bucket\"}, - {\"name\": \"INSPECT_NAMESPACES\", \"value\": \"$namespaces\"}, - {\"name\": \"S3_KEY\", \"value\": \"$s3_key\"} - ] - }] - }") \ - || { echo " Failed to launch log-collector task for ${cluster_id}"; return 1; } - - # Check for placement failures (capacity, etc.) + --overrides "$overrides_json") \ + || { echo " Failed to launch dump task for ${cluster_id}"; return 1; } + local failures failures=$(echo "$run_task_output" | jq -r '.failures[0].reason // empty') if [[ -n "$failures" ]]; then @@ -240,18 +418,17 @@ collect_logs_for_cluster() { return 1 fi + local task_arn task_id task_arn=$(echo "$run_task_output" | jq -r '.tasks[0].taskArn // empty') if [[ -z "$task_arn" ]]; then echo " ECS run-task returned no taskArn for ${cluster_id}" return 1 fi - local task_id task_id=$(echo "$task_arn" | awk -F'/' '{print $NF}') echo " Task started: $task_id" - # Wait for the task to complete - echo " Waiting for log-collector task to finish..." + echo " Waiting for dump task to finish..." if ! aws ecs wait tasks-stopped --cluster "$ecs_cluster" --tasks "$task_id"; then echo " Waiter timed out; polling task status..." local poll_status @@ -268,7 +445,6 @@ collect_logs_for_cluster() { fi fi - # Check exit code local describe_output exit_code describe_output=$(aws ecs describe-tasks \ --cluster "$ecs_cluster" --tasks "$task_id") @@ -283,41 +459,36 @@ collect_logs_for_cluster() { fi if [[ "$exit_code" != "0" ]]; then - echo " Warning: log-collector exited with code $exit_code for ${cluster_id}" + echo " Warning: dump task exited with code $exit_code for ${cluster_id}" echo " Check CloudWatch logs: /ecs/${cluster_id}/bastion (log-collector stream)" return 1 fi - # In S3-only mode, leave logs in the bucket and print the location. - # This is used in CI to avoid publishing sensitive data to public artifacts. if [[ "${S3_ONLY:-}" == "true" ]]; then - echo " Logs uploaded to S3. To download and extract:" + echo " Dump uploaded to S3. To download and extract:" echo "" - echo " mkdir -p /tmp/${cluster_id}-logs && aws s3 cp s3://${s3_bucket}/${s3_key} /tmp/${cluster_id}-logs/${s3_key} && tar xzf /tmp/${cluster_id}-logs/${s3_key} -C /tmp/${cluster_id}-logs" + echo " mkdir -p /tmp/${cluster_id}-dump && aws s3 cp s3://${s3_bucket}/${s3_key} /tmp/${cluster_id}-dump/${s3_key} && tar xzf /tmp/${cluster_id}-dump/${s3_key} -C /tmp/${cluster_id}-dump" echo "" return 0 fi - # Download to a temp file outside the output directory so the unredacted - # tarball never lands in the artifact dir. - echo " Downloading logs from S3..." + echo " Downloading dump from S3..." local tmp_archive - tmp_archive="$(mktemp -t inspect-logs-XXXXXX.tar.gz)" + tmp_archive="$(mktemp -t dump-env-XXXXXX.tar.gz)" aws s3 cp "s3://${s3_bucket}/${s3_key}" "$tmp_archive" --quiet \ - || { echo " Failed to download logs from S3 for ${cluster_id}"; rm -f "$tmp_archive"; return 1; } + || { echo " Failed to download dump from S3 for ${cluster_id}"; rm -f "$tmp_archive"; return 1; } mkdir -p "$out_dir" if ! tar xzf "$tmp_archive" -C "$out_dir" --strip-components=1; then - echo " Failed to extract logs archive for ${cluster_id}; leaving S3 object intact" + echo " Failed to extract dump archive for ${cluster_id}; leaving S3 object intact" rm -f "$tmp_archive" return 1 fi rm -f "$tmp_archive" - # Clean up S3 aws s3 rm "s3://${s3_bucket}/${s3_key}" --quiet || true - echo "==> ${cluster_id} log collection complete: ${out_dir}" + echo "==> ${cluster_id} dump complete: ${out_dir}" } # --------------------------------------------------------------------------- @@ -344,7 +515,7 @@ TIMESTAMP=$(date +%Y%m%d-%H%M%S) OUTPUT_DIR="${LOG_OUTPUT_DIR:-/tmp/${PREFIX:-cluster-}logs-${TIMESTAMP}}" echo "" -echo "Collecting cluster logs..." +echo "Collecting environment state..." failed=0 @@ -352,7 +523,7 @@ failed=0 if [[ "$CLUSTER_SCOPE" == "all" || "$CLUSTER_SCOPE" == "regional" ]]; then echo "" if use_profile "regional"; then - collect_logs_for_cluster "${PREFIX}regional" "$RC_NAMESPACES" "${OUTPUT_DIR}/rc" || failed=1 + dump_cluster "${PREFIX}regional" "$RC_NAMESPACES" "${OUTPUT_DIR}/rc" "true" || failed=1 else failed=1 fi @@ -369,7 +540,7 @@ if [[ "$CLUSTER_SCOPE" == "all" || "$CLUSTER_SCOPE" == "management" ]]; then else while IFS= read -r mc_id; do mc_name="${mc_id#"$PREFIX"}" - collect_logs_for_cluster "$mc_id" "$MC_NAMESPACES" "${OUTPUT_DIR}/${mc_name}" || failed=1 + dump_cluster "$mc_id" "$MC_NAMESPACES" "${OUTPUT_DIR}/${mc_name}" || failed=1 done <<< "$mc_clusters" fi else @@ -389,9 +560,9 @@ fi echo "" if [[ $failed -eq 0 ]]; then - echo "Log collection complete." + echo "Environment dump complete." else - echo "Log collection finished with errors. Check output above for details." + echo "Environment dump finished with errors. Check output above for details." fi exit 0 diff --git a/scripts/dev/ephemeral-env.sh b/scripts/dev/ephemeral-env.sh index 71c41d657..77a3104ae 100755 --- a/scripts/dev/ephemeral-env.sh +++ b/scripts/dev/ephemeral-env.sh @@ -44,7 +44,7 @@ usage() { echo " port-forward Forward ports through RC/MC bastion in an ephemeral env" echo " sre-ui Tunnel SRE UI tools through the internal ALB via bastion" echo " e2e Run e2e tests against an ephemeral env" - echo " collect-logs Collect kubernetes logs from RC/MC in an ephemeral env" + echo " dump-env Dump EKS must-gather and DB state from RC/MC in an ephemeral env" } usage_bastion_interactive() { @@ -194,12 +194,16 @@ setup_override_mount() { fetch_github_token() { if [[ -z "${GITHUB_TOKEN:-}" ]]; then echo "Fetching GitHub token from SSM Parameter Store..." + local ssm_err + ssm_err=$(mktemp) GITHUB_TOKEN=$(aws ssm get-parameter \ --name "$GITHUB_TOKEN_SECRET" \ --with-decryption \ --profile rrp-ephemeral-central \ - --query Parameter.Value --output text 2>/dev/null) \ - || die "Failed to fetch GitHub token from SSM." + --query Parameter.Value --output text 2>"$ssm_err") \ + || die "Failed to fetch GitHub token from SSM. +$(cat "$ssm_err")" + rm -f "$ssm_err" fi export GITHUB_TOKEN } @@ -1037,7 +1041,7 @@ cmd_e2e() { bash ci/e2e-tests.sh } -cmd_collect_logs() { +cmd_dump_env() { local cluster_type="${1:-all}" # Accept short aliases case "$cluster_type" in @@ -1046,7 +1050,7 @@ cmd_collect_logs() { esac # Select environment (ready only) select_env "STATE=ready" \ - "Select environment for log collection:" \ + "Select environment for dump-env:" \ "No ready environments found." \ true @@ -1059,7 +1063,7 @@ cmd_collect_logs() { local eph_prefix eph_prefix="eph-${BUILD_ID}-" - # collect-cluster-logs.sh runs on the host (not in a container) but needs + # dump-env.sh runs on the host (not in a container) but needs # the standardized profile names (rrp-rc, rrp-mc). Point it at the resolved # container config which has those profiles with static credentials. export AWS_CONFIG_FILE="$_CONTAINER_CONFIG" @@ -1070,7 +1074,7 @@ cmd_collect_logs() { export LOG_OUTPUT_DIR="$ARTIFACT_DIR" fi - "${REPO_ROOT}/scripts/dev/collect-cluster-logs.sh" "$cluster_type" + "${REPO_ROOT}/scripts/dev/dump-env.sh" "$cluster_type" } # ============================================================================= @@ -1149,7 +1153,7 @@ cmd_sre_tunnel() { } case "${1:-help}" in - bastion|collect-logs|sre-ui) + bastion|dump-env|sre-ui) for tool in jq uv aws; do command -v "$tool" >/dev/null 2>&1 || die "Missing required tool: $tool" done @@ -1180,7 +1184,7 @@ case "${1:-help}" in port-forward) shift; cmd_bastion_port_forward "$@" ;; sre-ui) cmd_sre_tunnel ;; e2e) cmd_e2e ;; - collect-logs) shift; cmd_collect_logs "$@" ;; + dump-env) shift; cmd_dump_env "$@" ;; help|*) usage ;; diff --git a/scripts/dev/int-env.sh b/scripts/dev/int-env.sh index 07819e767..595af10ba 100755 --- a/scripts/dev/int-env.sh +++ b/scripts/dev/int-env.sh @@ -40,7 +40,7 @@ usage() { echo " bastion Connect to RC/MC bastion" echo " port-forward Forward ports through RC/MC bastion" echo " e2e Run e2e tests" - echo " collect-logs Collect kubernetes logs from RC/MC" + echo " dump-env Dump EKS must-gather and DB state from RC/MC" } usage_bastion() { @@ -477,7 +477,7 @@ cmd_e2e() { bash ci/e2e-tests.sh } -cmd_collect_logs() { +cmd_dump_env() { local cluster_type="${1:-all}" case "$cluster_type" in rc) cluster_type="regional" ;; @@ -487,8 +487,8 @@ cmd_collect_logs() { setup_aws_config write_int_container_config - # collect-cluster-logs.sh runs on the host (not in a container) but needs - # the standardized profile names (rrp-rc, rrp-mc). Point it at the resolved + # dump-env.sh runs on the host (not in a container) but needs the + # standardized profile names (rrp-rc, rrp-mc). Point it at the resolved # container config which has those profiles with static credentials. export AWS_CONFIG_FILE="$_CONTAINER_CONFIG" export AWS_SHARED_CREDENTIALS_FILE=/dev/null @@ -498,7 +498,7 @@ cmd_collect_logs() { export LOG_OUTPUT_DIR="$ARTIFACT_DIR" fi - "${REPO_ROOT}/scripts/dev/collect-cluster-logs.sh" "$cluster_type" + "${REPO_ROOT}/scripts/dev/dump-env.sh" "$cluster_type" } # ============================================================================= @@ -508,7 +508,7 @@ cmd_collect_logs() { REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)" case "${1:-help}" in - bastion|collect-logs) + bastion|dump-env) for tool in jq uv aws; do command -v "$tool" >/dev/null 2>&1 || die "Missing required tool: $tool" done @@ -531,7 +531,7 @@ case "${1:-help}" in bastion) shift; cmd_bastion "$@" ;; port-forward) shift; cmd_port_forward "$@" ;; e2e) cmd_e2e ;; - collect-logs) shift; cmd_collect_logs "$@" ;; + dump-env) shift; cmd_dump_env "$@" ;; help|*) usage ;; diff --git a/scripts/render.py b/scripts/render.py index 3a29c09b5..f8fbc6f7d 100755 --- a/scripts/render.py +++ b/scripts/render.py @@ -361,6 +361,9 @@ def build_mc_list( # -- Documentation ------------------------------------------------------------ # Variables injected by render.py, not from config files. +# Note: Go template keywords (if/else/end) that appear in escaped Go template strings +# (e.g., {{ '{{ if ... }}X{{ else }}Y{{ end }}' }}) are automatically filtered by +# scan_template_variables() and do not need to be listed here. CONTEXT_VARS = { "environment", "aws_region", "account_id", "management_clusters", @@ -382,6 +385,13 @@ def build_mc_list( ] # Matches bracket-string access: {{ var['key'].rest }} → var.key.rest _TPL_BRACKET_PATTERN = re.compile(r"\{\{[\s-]*([a-zA-Z_]\w*)\['([^']+)'\]([\w.]*)") +# Matches escaped Go template strings wrapped in single or double quotes: +# {{ '{{ ... }}' }}, {{ "{{ ... }}" }}, {% '{{ ... }}' %}, {% "{{ ... }}" %} +# These contain Go template keywords (if/else/end) that should not be treated as Jinja2 variables. +_ESCAPED_GO_TEMPLATE_RE = re.compile( + r"""\{\{[\s-]*(?:'[^']*\{\{[^}]*\}\}[^']*'|"[^"]*\{\{[^}]*\}\}[^"]*")[\s-]*\}\}""" + r"""|\{%[\s-]*(?:'[^']*\{\{[^}]*\}\}[^']*'|"[^"]*\{\{[^}]*\}\}[^"]*")[\s-]*%\}""" +) def scan_annotations(content: str) -> dict[str, dict[str, Any]]: @@ -398,14 +408,32 @@ def scan_annotations(content: str) -> dict[str, dict[str, Any]]: return result +def _is_inside_escaped_go_template(pos: int, skip_ranges: list[tuple[int, int]]) -> bool: + """Check if a position falls inside an escaped Go template string.""" + return any(start <= pos < end for start, end in skip_ranges) + + def scan_template_variables(templates_dir: Path) -> dict[str, list[str]]: - """Scan templates for variable references. Returns {var: [template_paths]}.""" + """Scan templates for variable references. Returns {var: [template_paths]}. + + Variables inside escaped Go template strings (e.g., {{ '{{ if ... }}X{{ else }}Y{{ end }}' }}) + are filtered out, as they are Go template keywords, not Jinja2 variables. + """ var_to_templates: dict[str, list[str]] = {} for tpl in sorted(templates_dir.rglob("*.j2")): rel = str(tpl.relative_to(templates_dir)) content = tpl.read_text() + + # Build a set of character ranges to skip (escaped Go template strings) + skip_ranges: list[tuple[int, int]] = [] + for match in _ESCAPED_GO_TEMPLATE_RE.finditer(content): + skip_ranges.append((match.start(), match.end())) + for pattern in _TPL_PATTERNS: for match in pattern.finditer(content): + # Skip variables inside escaped Go template strings + if _is_inside_escaped_go_template(match.start(), skip_ranges): + continue var = match.group(1) if var.startswith("_") or var.split(".")[0] in ("true", "false", "none", "loop"): continue @@ -414,6 +442,9 @@ def scan_template_variables(templates_dir: Path) -> dict[str, list[str]]: if rel not in var_to_templates[var]: var_to_templates[var].append(rel) for match in _TPL_BRACKET_PATTERN.finditer(content): + # Skip variables inside escaped Go template strings + if _is_inside_escaped_go_template(match.start(), skip_ranges): + continue var = match.group(1) + "." + match.group(2) + match.group(3) if var.startswith("_"): continue diff --git a/scripts/test_render.py b/scripts/test_render.py index 5293a4fd4..693dbb197 100644 --- a/scripts/test_render.py +++ b/scripts/test_render.py @@ -2206,6 +2206,30 @@ def test_subdirectory_templates(self, tmp_path): result = scan_template_variables(tpl_dir) assert result["dns.domain"] == ["sub/test.j2"] + def test_ignores_double_quoted_go_template(self, tmp_path): + """Go template keywords inside a double-quoted string literal must not be + reported, but real Jinja variables should still be detected.""" + tpl_dir = tmp_path / "templates" + tpl_dir.mkdir() + (tpl_dir / "test.j2").write_text('{{ real_var }} and {{ "{{ if .foo }}X{{ else }}Y{{ end }}" }}') + result = scan_template_variables(tpl_dir) + assert result == {"real_var": ["test.j2"]} + assert "if" not in result + assert "else" not in result + assert "end" not in result + + def test_ignores_single_quoted_go_template_else_end(self, tmp_path): + """Go template keywords inside a single-quoted string literal must not be + reported, but real Jinja variables should still be detected.""" + tpl_dir = tmp_path / "templates" + tpl_dir.mkdir() + (tpl_dir / "test.j2").write_text("{{ another_var }} and {{ '{{ if .foo }}X{{ else }}Y{{ end }}' }}") + result = scan_template_variables(tpl_dir) + assert result == {"another_var": ["test.j2"]} + assert "if" not in result + assert "else" not in result + assert "end" not in result + class TestCollectLeafPaths: def test_flat_dict(self): diff --git a/terraform/config/README.md b/terraform/config/README.md index 1ac47ca56..30efd70b5 100644 --- a/terraform/config/README.md +++ b/terraform/config/README.md @@ -24,19 +24,8 @@ Three-stage CodePipeline (validate → deploy → bootstrap) for provisioning a ### `regional-cluster/` -Provisions the full regional cluster stack: EKS, VPC, API Gateway, Maestro IoT broker, RDS, authorization (DynamoDB + Pod Identity), ECS bootstrap, optional CloudTrail audit logging (disabled by default; enable with `enable_cloudtrail` for compliance environments), and optional bastion. +Provisions the full regional cluster stack: EKS, VPC, API Gateway, kube-applier DynamoDB tables, RDS (hyperfleet-db), authorization (DynamoDB + Pod Identity), ECS bootstrap, optional CloudTrail audit logging (disabled by default; enable with `enable_cloudtrail` for compliance environments), and optional bastion. ### `management-cluster/` -Provisions a management cluster: private EKS (1–2 nodes), ECS bootstrap, Maestro agent, and optional bastion. Hosts customer control planes. - -### `maestro-agent-iot-provisioning/` - -Standalone wrapper around the `maestro-agent-iot-provisioning` module for pipeline-based IoT provisioning. Provisions AWS IoT Core certificates and policies for Maestro agents in management clusters. - -Usage: - -1. Generate `terraform.tfvars` with cluster-specific values -2. Run `terraform init && terraform apply` -3. Extract certificate data: `terraform output -json certificate_data` -4. Transfer to management account Secrets Manager +Provisions a management cluster: private EKS (1–2 nodes), ECS bootstrap, kube-applier IAM, and optional bastion. Hosts customer control planes. diff --git a/terraform/config/central-account-bootstrap/versions.tf b/terraform/config/central-account-bootstrap/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/config/central-account-bootstrap/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/config/dns-environment-zone/versions.tf b/terraform/config/dns-environment-zone/versions.tf index d250ee47f..760728436 100644 --- a/terraform/config/dns-environment-zone/versions.tf +++ b/terraform/config/dns-environment-zone/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } diff --git a/terraform/config/kube-applier-dynamodb-provisioning/versions.tf b/terraform/config/kube-applier-dynamodb-provisioning/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/config/kube-applier-dynamodb-provisioning/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/config/management-cluster/terraform.tfvars.example b/terraform/config/management-cluster/terraform.tfvars.example index 55cf37b4a..7921e51e7 100644 --- a/terraform/config/management-cluster/terraform.tfvars.example +++ b/terraform/config/management-cluster/terraform.tfvars.example @@ -37,11 +37,10 @@ repository_branch = "main" # enable_bastion = true # ============================================================================= -# Maestro Agent Configuration (Optional) +# Management Cluster Identity # ============================================================================= # Logical cluster ID - MUST match the ID used in regional cluster's management_cluster_ids # Example: "mc01", "mc02", etc. -# This is used as the Maestro consumer name and for looking up pre-provisioned resources cluster_id = "mc01" diff --git a/terraform/config/management-cluster/variables.tf b/terraform/config/management-cluster/variables.tf index e9b0dbc54..7784b12f9 100755 --- a/terraform/config/management-cluster/variables.tf +++ b/terraform/config/management-cluster/variables.tf @@ -59,7 +59,7 @@ variable "enable_bastion" { } # ============================================================================= -# Maestro Configuration Variables +# Management Cluster Configuration Variables # ============================================================================= variable "management_id" { diff --git a/terraform/config/management-cluster/versions.tf b/terraform/config/management-cluster/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/config/management-cluster/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/config/pipeline-management-cluster/versions.tf b/terraform/config/pipeline-management-cluster/versions.tf index 050097854..58a754bda 100644 --- a/terraform/config/pipeline-management-cluster/versions.tf +++ b/terraform/config/pipeline-management-cluster/versions.tf @@ -2,7 +2,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } time = { source = "hashicorp/time" diff --git a/terraform/config/pipeline-regional-cluster/versions.tf b/terraform/config/pipeline-regional-cluster/versions.tf index 050097854..58a754bda 100644 --- a/terraform/config/pipeline-regional-cluster/versions.tf +++ b/terraform/config/pipeline-regional-cluster/versions.tf @@ -2,7 +2,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } time = { source = "hashicorp/time" diff --git a/terraform/config/regional-cluster/backend.tf b/terraform/config/regional-cluster/backend.tf index 5968ed02a..0dcfb78d1 100644 --- a/terraform/config/regional-cluster/backend.tf +++ b/terraform/config/regional-cluster/backend.tf @@ -2,6 +2,10 @@ terraform { backend "s3" {} required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } pagerduty = { source = "PagerDuty/pagerduty" version = ">= 3.0" diff --git a/terraform/config/regional-cluster/main.tf b/terraform/config/regional-cluster/main.tf index 2054c0321..636030b64 100755 --- a/terraform/config/regional-cluster/main.tf +++ b/terraform/config/regional-cluster/main.tf @@ -200,6 +200,9 @@ module "ecs_bootstrap" { loki_kms_key_arn = module.loki_infrastructure.kms_key_arn management_clusters = var.management_clusters + + rc_aws_account_id = var.target_account_id + redis_endpoint = var.enable_rate_limit_redis ? "${module.elasticache_valkey[0].endpoint}:${module.elasticache_valkey[0].port}" : "" } # ============================================================================= @@ -286,7 +289,6 @@ module "sre_ui_alb" { argocd = { client_id = var.sre_argocd_oidc_client_id, client_secret = var.sre_argocd_oidc_client_secret } prometheus = { client_id = var.sre_prometheus_oidc_client_id, client_secret = var.sre_prometheus_oidc_client_secret } thanos = { client_id = var.sre_thanos_oidc_client_id, client_secret = var.sre_thanos_oidc_client_secret } - loki = { client_id = var.sre_loki_oidc_client_id, client_secret = var.sre_loki_oidc_client_secret } } : {} } @@ -350,7 +352,7 @@ resource "aws_route53_record" "regional_delegation" { # # Each shard is a separate Route53 HostedZone under the regional zone, # providing ~10k records per shard. MC operators (external-dns, cert-manager) -# create cluster records in shards. CLM assigns clusters to shards. +# create cluster records in shards. hyperfleet-operator assigns clusters to shards. # ============================================================================= resource "aws_route53_zone" "zone_shard" { @@ -465,6 +467,25 @@ module "hyperfleet_db" { monitoring_interval = var.hyperfleet_db_monitoring_interval } +# ============================================================================= +# ElastiCache Valkey — rate limit counters (GCRA) +# ============================================================================= + +module "elasticache_valkey" { + count = var.enable_rate_limit_redis ? 1 : 0 + source = "../../modules/elasticache-valkey" + + cluster_id = var.regional_id + vpc_id = module.vpc.vpc_id + private_subnet_ids = module.vpc.private_subnet_ids + + eks_cluster_security_group_id = module.vpc.cluster_security_group_id + eks_cluster_primary_security_group_id = module.regional_cluster.node_security_group_id + + node_type = var.valkey_node_type + engine_version = var.valkey_engine_version +} + # ============================================================================= # Hyperfleet Operator IAM (Pod Identity) # @@ -530,8 +551,9 @@ module "cloudwatch_exporter" { module "regional_oidc" { source = "../../modules/regional-oidc" - regional_id = var.regional_id - mc_ou_path = var.mc_ou_path + regional_id = var.regional_id + mc_ou_path = var.mc_ou_path + force_destroy = var.environment == "ephemeral" } # ============================================================================= diff --git a/terraform/config/regional-cluster/outputs.tf b/terraform/config/regional-cluster/outputs.tf index b9c387021..28354a801 100755 --- a/terraform/config/regional-cluster/outputs.tf +++ b/terraform/config/regional-cluster/outputs.tf @@ -288,6 +288,17 @@ output "hyperfleet_db_dsn_secret_name" { value = module.hyperfleet_db.dsn_secret_name } +# ElastiCache Valkey +output "hyperfleet_redis_endpoint" { + description = "ElastiCache Valkey endpoint for rate limiting (null if disabled)" + value = var.enable_rate_limit_redis ? module.elasticache_valkey[0].endpoint : null +} + +output "hyperfleet_redis_port" { + description = "ElastiCache Valkey port (null if disabled)" + value = var.enable_rate_limit_redis ? module.elasticache_valkey[0].port : null +} + # ============================================================================= # CloudWatch Exporter Outputs # ============================================================================= @@ -381,11 +392,6 @@ output "sre_thanos_target_group_arn" { value = try(module.sre_ui_alb[0].thanos_target_group_arn, "") } -output "sre_loki_target_group_arn" { - description = "ARN of the Loki Query Frontend SRE ALB target group" - value = try(module.sre_ui_alb[0].loki_target_group_arn, "") -} - output "sre_alb_dns_name" { description = "DNS name of the SRE UI ALB" value = try(module.sre_ui_alb[0].alb_dns_name, "") diff --git a/terraform/config/regional-cluster/terraform.tfvars.example b/terraform/config/regional-cluster/terraform.tfvars.example index f00d6be5b..4373a33db 100644 --- a/terraform/config/regional-cluster/terraform.tfvars.example +++ b/terraform/config/regional-cluster/terraform.tfvars.example @@ -43,19 +43,3 @@ repository_branch = "main" # AWS region name (required for API Gateway URL construction) region_name = "test" - -# Maestro Infrastructure Configuration - -# ============================================================================= -# Configuration for Maestro MQTT-based orchestration between regional and -# management clusters - -# Database configuration -maestro_db_instance_class = "db.t4g.micro" # Use db.t4g.small or larger for production - -# High availability and protection (recommended for production) -maestro_db_multi_az = false # Set to true for production -maestro_db_deletion_protection = false # Set to true for production - -# MQTT topic prefix for Maestro communication (legacy — topics are now scoped by regional_id) -# maestro_mqtt_topic_prefix = "maestro/consumers" diff --git a/terraform/config/regional-cluster/variables.tf b/terraform/config/regional-cluster/variables.tf index a54356682..2397b1005 100755 --- a/terraform/config/regional-cluster/variables.tf +++ b/terraform/config/regional-cluster/variables.tf @@ -187,23 +187,28 @@ variable "sre_thanos_oidc_client_secret" { sensitive = true } -variable "sre_loki_oidc_client_id" { - description = "OIDC client ID for Loki. Required when enable_sre_oidc_auth = true." - type = string - default = "" +variable "enable_sns_alerting" { + description = "Enable SNS alerting for alert fan-out" + type = bool + default = false } -variable "sre_loki_oidc_client_secret" { - description = "OIDC client secret for Loki. Supply via Secrets Manager — never commit to git." +variable "enable_rate_limit_redis" { + description = "Enable ElastiCache Redis for Platform API rate limiting" + type = bool + default = true +} + +variable "valkey_node_type" { + description = "ElastiCache Valkey node type for rate limiting" type = string - default = "" - sensitive = true + default = "cache.t4g.micro" } -variable "enable_sns_alerting" { - description = "Enable SNS alerting for alert fan-out" - type = bool - default = false +variable "valkey_engine_version" { + description = "ElastiCache Valkey engine version" + type = string + default = "9.1" } # ============================================================================= diff --git a/terraform/modules/api-gateway/main.tf b/terraform/modules/api-gateway/main.tf index 037101758..9405d66fd 100644 --- a/terraform/modules/api-gateway/main.tf +++ b/terraform/modules/api-gateway/main.tf @@ -313,6 +313,13 @@ resource "aws_api_gateway_gateway_response" "default_4xx" { "gatewayresponse.header.Warning" = "'${local.system_use_notification}'" "gatewayresponse.header.X-System-Use-Notification" = "'${local.system_use_notification}'" } + + response_templates = { + "application/json" = jsonencode({ + message = "Request Error" + systemUseNotification = local.system_use_notification + }) + } } # ----------------------------------------------------------------------------- diff --git a/terraform/modules/api-gateway/versions.tf b/terraform/modules/api-gateway/versions.tf index 6a361dd6a..6e9abe15c 100644 --- a/terraform/modules/api-gateway/versions.tf +++ b/terraform/modules/api-gateway/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/authz/iam.tf b/terraform/modules/authz/iam.tf index a7c5ac275..91e6e9194 100644 --- a/terraform/modules/authz/iam.tf +++ b/terraform/modules/authz/iam.tf @@ -116,7 +116,12 @@ resource "aws_iam_role_policy" "frontend_api_avp" { "verifiedpermissions:DeletePolicy", "verifiedpermissions:GetPolicy", "verifiedpermissions:ListPolicies", - "verifiedpermissions:UpdatePolicy" + "verifiedpermissions:UpdatePolicy", + "verifiedpermissions:CreatePolicyTemplate", + "verifiedpermissions:DeletePolicyTemplate", + "verifiedpermissions:GetPolicyTemplate", + "verifiedpermissions:ListPolicyTemplates", + "verifiedpermissions:UpdatePolicyTemplate" ] Resource = "*" }, diff --git a/terraform/modules/authz/versions.tf b/terraform/modules/authz/versions.tf index 69c306948..283b445a6 100644 --- a/terraform/modules/authz/versions.tf +++ b/terraform/modules/authz/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/bastion/README.md b/terraform/modules/bastion/README.md index 8860f46a9..ae313af91 100644 --- a/terraform/modules/bastion/README.md +++ b/terraform/modules/bastion/README.md @@ -74,7 +74,7 @@ make int-port-forward-rc # or: make int-port-forward-mc make ephemeral-port-forward-rc ID= # or: make ephemeral-port-forward-mc ID= ``` -Select a service when prompted (e.g. `argocd`, `maestro`). The script handles the full two-hop chain automatically: +Select a service when prompted (e.g. `argocd`, `platform-api`). The script handles the full two-hop chain automatically: 1. Starts/reuses a bastion ECS task 2. Runs `kubectl port-forward` inside the bastion (bastion -> K8s service) diff --git a/terraform/modules/bastion/versions.tf b/terraform/modules/bastion/versions.tf index cacabebb9..ae8a51567 100644 --- a/terraform/modules/bastion/versions.tf +++ b/terraform/modules/bastion/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } null = { source = "hashicorp/null" diff --git a/terraform/modules/cloudtrail/versions.tf b/terraform/modules/cloudtrail/versions.tf index 3dccf26c7..0b9cbf412 100644 --- a/terraform/modules/cloudtrail/versions.tf +++ b/terraform/modules/cloudtrail/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/cloudwatch-exporter/versions.tf b/terraform/modules/cloudwatch-exporter/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/cloudwatch-exporter/versions.tf +++ b/terraform/modules/cloudwatch-exporter/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/dns-pod-identity/versions.tf b/terraform/modules/dns-pod-identity/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/modules/dns-pod-identity/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/modules/dns-zone-operator/versions.tf b/terraform/modules/dns-zone-operator/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/modules/dns-zone-operator/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/modules/ecs-bootstrap/main.tf b/terraform/modules/ecs-bootstrap/main.tf index cdde99b26..622d2db08 100644 --- a/terraform/modules/ecs-bootstrap/main.tf +++ b/terraform/modules/ecs-bootstrap/main.tf @@ -223,9 +223,9 @@ resource "aws_ecs_task_definition" "bootstrap" { sre_argocd_target_group_arn: "$SRE_ARGOCD_TARGET_GROUP_ARN" sre_prometheus_target_group_arn: "$SRE_PROMETHEUS_TARGET_GROUP_ARN" sre_thanos_target_group_arn: "$SRE_THANOS_TARGET_GROUP_ARN" - sre_loki_target_group_arn: "$SRE_LOKI_TARGET_GROUP_ARN" sre_alb_dns_name: "$SRE_ALB_DNS_NAME" sre_domain: "$SRE_DOMAIN" + redis_endpoint: "$REDIS_ENDPOINT" type: Opaque stringData: name: in-cluster @@ -294,6 +294,10 @@ resource "aws_ecs_task_definition" "bootstrap" { { name = "RC_AWS_ACCOUNT_ID" value = var.rc_aws_account_id + }, + { + name = "REDIS_ENDPOINT" + value = var.redis_endpoint } ] diff --git a/terraform/modules/ecs-bootstrap/variables.tf b/terraform/modules/ecs-bootstrap/variables.tf index 878ca117c..07e002171 100644 --- a/terraform/modules/ecs-bootstrap/variables.tf +++ b/terraform/modules/ecs-bootstrap/variables.tf @@ -76,3 +76,9 @@ variable "rc_aws_account_id" { } } +variable "redis_endpoint" { + description = "ElastiCache Valkey endpoint for Platform API rate limiting" + type = string + default = "" +} + diff --git a/terraform/modules/ecs-bootstrap/versions.tf b/terraform/modules/ecs-bootstrap/versions.tf index 30393d5ae..9d925b7d9 100644 --- a/terraform/modules/ecs-bootstrap/versions.tf +++ b/terraform/modules/ecs-bootstrap/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } \ No newline at end of file diff --git a/terraform/modules/eks-cluster-workerless/versions.tf b/terraform/modules/eks-cluster-workerless/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/modules/eks-cluster-workerless/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/modules/eks-cluster/main.tf b/terraform/modules/eks-cluster/main.tf index 33fe541b1..5734aa118 100644 --- a/terraform/modules/eks-cluster/main.tf +++ b/terraform/modules/eks-cluster/main.tf @@ -175,7 +175,7 @@ resource "aws_eks_addon" "pod_identity" { addon_name = "eks-pod-identity-agent" } -# AWS Secrets Store CSI Driver Provider (e.g. for Maestro agent secret mounting) +# AWS Secrets Store CSI Driver Provider (e.g. for kube-applier or service secret mounting) resource "aws_eks_addon" "aws_secrets_store_csi_driver_provider" { cluster_name = aws_eks_cluster.main.name addon_name = "aws-secrets-store-csi-driver-provider" diff --git a/terraform/modules/eks-cluster/versions.tf b/terraform/modules/eks-cluster/versions.tf index c7e4c3e70..a88a4501d 100644 --- a/terraform/modules/eks-cluster/versions.tf +++ b/terraform/modules/eks-cluster/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } \ No newline at end of file diff --git a/terraform/modules/elasticache-valkey/main.tf b/terraform/modules/elasticache-valkey/main.tf new file mode 100644 index 000000000..7669035cf --- /dev/null +++ b/terraform/modules/elasticache-valkey/main.tf @@ -0,0 +1,154 @@ +# ============================================================================= +# ElastiCache Valkey for Platform API Rate Limiting +# +# Single-node Valkey for shared rate limit counters (GCRA algorithm). +# Valkey is the open-source (BSD 3-Clause) fork of Redis, fully compatible +# with go-redis/v9 and redis_rate. 20% cheaper than Redis OSS on ElastiCache. +# No persistence, no AUTH, no backups — counters are ephemeral by design. +# ============================================================================= + +# Resolve current account ID for the KMS key policy +data "aws_caller_identity" "current" {} + +# KMS key for ElastiCache encryption at rest (FedRAMP SC-13). +# Explicit key policy scopes usage to ElastiCache (CKV2_AWS_64) rather than +# relying on the implicit default that grants kms:* to the account root. +resource "aws_kms_key" "elasticache" { + description = "KMS key for ElastiCache Valkey encryption at rest (FedRAMP SC-13)" + deletion_window_in_days = 30 + enable_key_rotation = true + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "EnableRootAccess" + Effect = "Allow" + Principal = { + AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" + } + Action = "kms:*" + Resource = "*" + }, + { + Sid = "AllowElastiCacheAccess" + Effect = "Allow" + Principal = { + Service = "elasticache.amazonaws.com" + } + Action = [ + "kms:Decrypt", + "kms:DescribeKey", + "kms:Encrypt", + "kms:GenerateDataKey*", + "kms:ReEncrypt*", + "kms:CreateGrant" + ] + Resource = "*" + } + ] + }) + + tags = { + Name = "${var.cluster_id}-elasticache-valkey" + Component = "rate-limiting" + } +} + +resource "aws_kms_alias" "elasticache" { + name = "alias/${var.cluster_id}-elasticache-valkey" + target_key_id = aws_kms_key.elasticache.key_id +} + +# Security Group for ElastiCache Valkey +resource "aws_security_group" "valkey" { + name = "${var.cluster_id}-valkey" + description = "Security group for Platform API rate limiting Valkey" + vpc_id = var.vpc_id + + revoke_rules_on_delete = false + + tags = { + Name = "${var.cluster_id}-valkey-sg" + Component = "rate-limiting" + } +} + +# Ingress rules as standalone resources — these depend on EKS SG IDs but +# do NOT block the ElastiCache cluster from provisioning. + +resource "aws_security_group_rule" "valkey_eks_cluster" { + type = "ingress" + description = "Valkey from EKS cluster additional security group" + from_port = 6379 + to_port = 6379 + protocol = "tcp" + security_group_id = aws_security_group.valkey.id + source_security_group_id = var.eks_cluster_security_group_id +} + +resource "aws_security_group_rule" "valkey_eks_primary" { + type = "ingress" + description = "Valkey from EKS cluster primary security group (Auto Mode)" + from_port = 6379 + to_port = 6379 + protocol = "tcp" + security_group_id = aws_security_group.valkey.id + source_security_group_id = var.eks_cluster_primary_security_group_id +} + +# Subnet Group +resource "aws_elasticache_subnet_group" "valkey" { + name = "${var.cluster_id}-valkey" + subnet_ids = var.private_subnet_ids + + tags = { + Name = "${var.cluster_id}-valkey-subnet-group" + Component = "rate-limiting" + } +} + +# Parameter Group +resource "aws_elasticache_parameter_group" "valkey" { + name = "${var.cluster_id}-valkey" + family = "valkey9" + + parameter { + name = "maxmemory-policy" + value = "volatile-ttl" + } + + tags = { + Name = "${var.cluster_id}-valkey-params" + Component = "rate-limiting" + } +} + +# ElastiCache Valkey Replication Group (single node, no HA, no backups) +# Uses aws_elasticache_replication_group because the AWS CreateCacheCluster +# API (aws_elasticache_cluster) does not support the Valkey engine. +resource "aws_elasticache_replication_group" "valkey" { + replication_group_id = "${var.cluster_id}-hf-rl" + description = "Platform API rate limiting (Valkey)" + engine = "valkey" + engine_version = var.engine_version + node_type = var.node_type + num_node_groups = 1 + replicas_per_node_group = 0 + parameter_group_name = aws_elasticache_parameter_group.valkey.name + subnet_group_name = aws_elasticache_subnet_group.valkey.name + security_group_ids = [aws_security_group.valkey.id] + port = 6379 + maintenance_window = "mon:05:00-mon:06:00" + apply_immediately = true + snapshot_retention_limit = 0 + + transit_encryption_enabled = true + at_rest_encryption_enabled = true + kms_key_id = aws_kms_key.elasticache.arn + + tags = { + Name = "${var.cluster_id}-valkey" + Component = "rate-limiting" + } +} diff --git a/terraform/modules/elasticache-valkey/outputs.tf b/terraform/modules/elasticache-valkey/outputs.tf new file mode 100644 index 000000000..6fa6268af --- /dev/null +++ b/terraform/modules/elasticache-valkey/outputs.tf @@ -0,0 +1,13 @@ +# ============================================================================= +# ElastiCache Valkey Module Outputs +# ============================================================================= + +output "endpoint" { + description = "ElastiCache Valkey primary endpoint address" + value = aws_elasticache_replication_group.valkey.primary_endpoint_address +} + +output "port" { + description = "ElastiCache Valkey port" + value = aws_elasticache_replication_group.valkey.port +} diff --git a/terraform/modules/elasticache-valkey/variables.tf b/terraform/modules/elasticache-valkey/variables.tf new file mode 100644 index 000000000..6b7dbbe71 --- /dev/null +++ b/terraform/modules/elasticache-valkey/variables.tf @@ -0,0 +1,40 @@ +# ============================================================================= +# ElastiCache Valkey Module Variables +# ============================================================================= + +variable "cluster_id" { + description = "Regional cluster identifier for resource naming (e.g. 'regional')" + type = string +} + +variable "vpc_id" { + description = "VPC ID where ElastiCache will be deployed" + type = string +} + +variable "private_subnet_ids" { + description = "Private subnet IDs for ElastiCache subnet group" + type = list(string) +} + +variable "eks_cluster_security_group_id" { + description = "EKS cluster additional security group ID (ingress to Valkey)" + type = string +} + +variable "eks_cluster_primary_security_group_id" { + description = "EKS cluster primary security group ID (Auto Mode ingress to Valkey)" + type = string +} + +variable "node_type" { + description = "ElastiCache node type" + type = string + default = "cache.t4g.micro" +} + +variable "engine_version" { + description = "Valkey engine version" + type = string + default = "9.1" +} diff --git a/terraform/modules/elasticache-valkey/versions.tf b/terraform/modules/elasticache-valkey/versions.tf new file mode 100644 index 000000000..0b9cbf412 --- /dev/null +++ b/terraform/modules/elasticache-valkey/versions.tf @@ -0,0 +1,10 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} diff --git a/terraform/modules/grafana-cloudwatch-logs/versions.tf b/terraform/modules/grafana-cloudwatch-logs/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/grafana-cloudwatch-logs/versions.tf +++ b/terraform/modules/grafana-cloudwatch-logs/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/hyperfleet-db/versions.tf b/terraform/modules/hyperfleet-db/versions.tf index 433c68736..8fb4cbf5d 100644 --- a/terraform/modules/hyperfleet-db/versions.tf +++ b/terraform/modules/hyperfleet-db/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } random = { source = "hashicorp/random" diff --git a/terraform/modules/hypershift-oidc/iam.tf b/terraform/modules/hypershift-oidc/iam.tf index a21e17d6e..9cd75919b 100755 --- a/terraform/modules/hypershift-oidc/iam.tf +++ b/terraform/modules/hypershift-oidc/iam.tf @@ -184,7 +184,7 @@ resource "aws_eks_pod_identity_association" "hypershift_installer" { # # Grants the External Secrets Operator permission to read secrets from SSM # Parameter Store. ESO will sync these to cluster namespaces managed by -# CLM/Maestro. +# hyperfleet-operator/kube-applier. # # The operator runs in the external-secrets namespace and uses Pod Identity # for AWS authentication. diff --git a/terraform/modules/hypershift-oidc/versions.tf b/terraform/modules/hypershift-oidc/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/hypershift-oidc/versions.tf +++ b/terraform/modules/hypershift-oidc/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/kube-applier-dynamodb/versions.tf b/terraform/modules/kube-applier-dynamodb/versions.tf index ddfcb0e05..0ea67afb2 100644 --- a/terraform/modules/kube-applier-dynamodb/versions.tf +++ b/terraform/modules/kube-applier-dynamodb/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 5.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/kube-applier/versions.tf b/terraform/modules/kube-applier/versions.tf index ddfcb0e05..0ea67afb2 100644 --- a/terraform/modules/kube-applier/versions.tf +++ b/terraform/modules/kube-applier/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 5.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/loki-infrastructure/versions.tf b/terraform/modules/loki-infrastructure/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/loki-infrastructure/versions.tf +++ b/terraform/modules/loki-infrastructure/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/loki-log-forwarder/versions.tf b/terraform/modules/loki-log-forwarder/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/loki-log-forwarder/versions.tf +++ b/terraform/modules/loki-log-forwarder/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/pagerduty-service/provider.tf b/terraform/modules/pagerduty-service/provider.tf index f014bf557..e09a034bf 100644 --- a/terraform/modules/pagerduty-service/provider.tf +++ b/terraform/modules/pagerduty-service/provider.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } pagerduty = { source = "PagerDuty/pagerduty" diff --git a/terraform/modules/pipeline-notifications/versions.tf b/terraform/modules/pipeline-notifications/versions.tf index 1cbcc332e..b89f53d3d 100644 --- a/terraform/modules/pipeline-notifications/versions.tf +++ b/terraform/modules/pipeline-notifications/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } archive = { source = "hashicorp/archive" diff --git a/terraform/modules/pipeline-provisioner/versions.tf b/terraform/modules/pipeline-provisioner/versions.tf index 050097854..58a754bda 100644 --- a/terraform/modules/pipeline-provisioner/versions.tf +++ b/terraform/modules/pipeline-provisioner/versions.tf @@ -2,7 +2,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } time = { source = "hashicorp/time" diff --git a/terraform/modules/platform-image/Dockerfile b/terraform/modules/platform-image/Dockerfile index 777762031..8e4482a39 100644 --- a/terraform/modules/platform-image/Dockerfile +++ b/terraform/modules/platform-image/Dockerfile @@ -2,13 +2,18 @@ FROM registry.access.redhat.com/ubi9/ubi-minimal:9.7-1770267347 # Tool versions - update these and rebuild to upgrade ARG KUBECTL_VERSION=v1.31.0 -ARG HELM_VERSION=v3.16.0 +ARG HELM_VERSION=v3.21.3 ARG K9S_VERSION=v0.32.5 ARG STERN_VERSION=1.30.0 ARG YQ_VERSION=v4.44.3 ARG OC_VERSION=4.16.0 ARG TERRAFORM_VERSION=1.14.3 +# Global curl retry and timeout settings for improved download reliability +ARG CURL_RETRY=3 +ARG CURL_RETRY_DELAY=2 +ARG CURL_MAX_TIME=300 + # Install base packages RUN microdnf install -y \ tar \ @@ -33,37 +38,46 @@ RUN microdnf install -y \ rm -rf /var/cache/yum # Install AWS CLI v2 -RUN curl -sL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/tmp/awscliv2.zip" && \ +RUN curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "/tmp/awscliv2.zip" && \ unzip -q /tmp/awscliv2.zip -d /tmp && \ /tmp/aws/install && \ rm -rf /tmp/aws /tmp/awscliv2.zip # Install kubectl -RUN curl -sL "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl && \ +RUN curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" -o /usr/local/bin/kubectl && \ chmod +x /usr/local/bin/kubectl -# Install helm -RUN curl -sL "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" | tar -xz -C /tmp && \ +# Install helm (SHA-256 verified with retry logic) +RUN set -euo pipefail && \ + HELM_PACKAGE="helm-${HELM_VERSION}-linux-amd64.tar.gz" && \ + HELM_BASE_URL="https://get.helm.sh" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfO "${HELM_BASE_URL}/${HELM_PACKAGE}" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfO "${HELM_BASE_URL}/${HELM_PACKAGE}.sha256sum" && \ + sha256sum -c "${HELM_PACKAGE}.sha256sum" && \ + tar -xzf "${HELM_PACKAGE}" -C /tmp && \ mv /tmp/linux-amd64/helm /usr/local/bin/helm && \ chmod +x /usr/local/bin/helm && \ - rm -rf /tmp/linux-amd64 + rm -rf /tmp/linux-amd64 "${HELM_PACKAGE}" "${HELM_PACKAGE}.sha256sum" # Install k9s -RUN curl -sL "https://github.com/derailed/k9s/releases/download/${K9S_VERSION}/k9s_Linux_amd64.tar.gz" | tar -xz -C /tmp && \ +RUN set -euo pipefail && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://github.com/derailed/k9s/releases/download/${K9S_VERSION}/k9s_Linux_amd64.tar.gz" | tar -xz -C /tmp && \ mv /tmp/k9s /usr/local/bin/k9s && \ chmod +x /usr/local/bin/k9s # Install stern -RUN curl -sL "https://github.com/stern/stern/releases/download/v${STERN_VERSION}/stern_${STERN_VERSION}_linux_amd64.tar.gz" | tar -xz -C /tmp && \ +RUN set -euo pipefail && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://github.com/stern/stern/releases/download/v${STERN_VERSION}/stern_${STERN_VERSION}_linux_amd64.tar.gz" | tar -xz -C /tmp && \ mv /tmp/stern /usr/local/bin/stern && \ chmod +x /usr/local/bin/stern # Install yq -RUN curl -sL "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64" -o /usr/local/bin/yq && \ +RUN curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_amd64" -o /usr/local/bin/yq && \ chmod +x /usr/local/bin/yq # Install OpenShift CLI (oc) -RUN curl -sL "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${OC_VERSION}/openshift-client-linux.tar.gz" | tar -xz -C /tmp && \ +RUN set -euo pipefail && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfL "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${OC_VERSION}/openshift-client-linux.tar.gz" | tar -xz -C /tmp && \ mv /tmp/oc /usr/local/bin/oc && \ chmod +x /usr/local/bin/oc && \ rm -f /tmp/kubectl /tmp/README.md @@ -72,9 +86,9 @@ RUN curl -sL "https://mirror.openshift.com/pub/openshift-v4/clients/ocp/${OC_VER RUN set -euo pipefail && \ TF_PACKAGE="terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \ TF_BASE_URL="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}" && \ - curl -sSfO "${TF_BASE_URL}/${TF_PACKAGE}" && \ - curl -sSfO "${TF_BASE_URL}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" && \ - curl -sSfO "${TF_BASE_URL}/terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfO "${TF_BASE_URL}/${TF_PACKAGE}" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfO "${TF_BASE_URL}/terraform_${TERRAFORM_VERSION}_SHA256SUMS" && \ + curl --retry ${CURL_RETRY} --retry-delay ${CURL_RETRY_DELAY} --max-time ${CURL_MAX_TIME} -sSfO "${TF_BASE_URL}/terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig" && \ gpg --batch --keyserver keyserver.ubuntu.com --recv-keys \ C874011F0AB405110D02105534365D9472D7468F || \ gpg --batch --keyserver keys.openpgp.org --recv-keys \ @@ -89,7 +103,14 @@ RUN set -euo pipefail && \ rm -rf "${TF_PACKAGE}" "terraform_${TERRAFORM_VERSION}_SHA256SUMS" \ "terraform_${TERRAFORM_VERSION}_SHA256SUMS.sig" /tmp/tf-bin -# Run as non-root -RUN mkdir -p /opt/platform && chown 65534:65534 /opt/platform -ENV HOME=/opt/platform +# Copy platform scripts (root-owned, non-writable at runtime) +COPY scripts/ /opt/platform/scripts/ +RUN chmod +x /opt/platform/scripts/* && \ + chown -R root:root /opt/platform/scripts && \ + chmod -R a-w /opt/platform/scripts + +# Run as non-root; give UID 65534 its own writable home, leave scripts read-only +RUN mkdir -p /opt/platform/home && chown 65534:65534 /opt/platform/home +ENV HOME=/opt/platform/home +ENV PATH="/opt/platform/scripts:${PATH}" USER 65534 diff --git a/terraform/modules/platform-image/main.tf b/terraform/modules/platform-image/main.tf index ce9253679..9322a1b3d 100644 --- a/terraform/modules/platform-image/main.tf +++ b/terraform/modules/platform-image/main.tf @@ -9,6 +9,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" + version = "~> 6.56.0" configuration_aliases = [aws.us_east_1] } } diff --git a/terraform/modules/platform-image/scripts/refresh-app b/terraform/modules/platform-image/scripts/refresh-app new file mode 100755 index 000000000..83ce29b45 --- /dev/null +++ b/terraform/modules/platform-image/scripts/refresh-app @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# refresh-app — Hard-refresh an ArgoCD application via kubectl annotation. +# +# Usage: refresh-app +# +# Triggers an out-of-band hard refresh on the given ArgoCD Application by +# annotating it with argocd.argoproj.io/refresh=hard. If the target app is +# not the root app, the root app is always refreshed first so that any +# generated child apps are up-to-date before the specific app is re-synced. +# +# Required tools: kubectl (with a valid KUBECONFIG or in-cluster credentials) +set -euo pipefail + +ROOT_APP="root" +ARGOCD_NAMESPACE="argocd" + +# DNS-1123 subdomain: lowercase alphanumeric segments separated by dots/hyphens, +# max 253 characters. Matches the format ArgoCD uses for Application names. +VALID_APP_NAME_RE='^[a-z0-9]([a-z0-9-]*[a-z0-9])?(\.[a-z0-9]([a-z0-9-]*[a-z0-9])?)*$' + +usage() { + echo "Usage: $(basename "$0") " >&2 + echo "" >&2 + echo " Hard-refresh an ArgoCD application via kubectl annotation." >&2 + echo " The '${ROOT_APP}' app is always refreshed first unless it is the target." >&2 + echo " app_name must be a valid DNS-1123 name (max 253 characters)." >&2 + echo "" >&2 + echo "Examples:" >&2 + echo " $(basename "$0") hyperfleet-operator # refreshes root, then hyperfleet-operator" >&2 + echo " $(basename "$0") root # refreshes root only" >&2 + exit 1 +} + +validate_app_name() { + local name="$1" + if [[ ${#name} -gt 253 ]]; then + echo "ERROR: app name exceeds 253 characters" >&2 + exit 1 + fi + if [[ ! "${name}" =~ ${VALID_APP_NAME_RE} ]]; then + echo "ERROR: '${name}' is not a valid DNS-1123 application name" >&2 + exit 1 + fi +} + +refresh_app() { + local app="$1" + echo "=== Refreshing ArgoCD app: ${app} ===" + kubectl annotate app "${app}" \ + -n "${ARGOCD_NAMESPACE}" \ + argocd.argoproj.io/refresh=hard \ + --overwrite +} + +[[ $# -eq 1 ]] || usage + +APP_NAME="$1" +validate_app_name "${APP_NAME}" + +if [[ "${APP_NAME}" != "${ROOT_APP}" ]]; then + refresh_app "${ROOT_APP}" +fi + +refresh_app "${APP_NAME}" + +echo "=== Done ===" diff --git a/terraform/modules/prometheus-remote-write/versions.tf b/terraform/modules/prometheus-remote-write/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/prometheus-remote-write/versions.tf +++ b/terraform/modules/prometheus-remote-write/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/regional-oidc/s3.tf b/terraform/modules/regional-oidc/s3.tf index eb99ad4f2..3f6f8cb7c 100755 --- a/terraform/modules/regional-oidc/s3.tf +++ b/terraform/modules/regional-oidc/s3.tf @@ -1,6 +1,6 @@ resource "aws_s3_bucket" "oidc" { bucket = local.bucket_name - force_destroy = false + force_destroy = var.force_destroy tags = { Name = local.bucket_name diff --git a/terraform/modules/regional-oidc/variables.tf b/terraform/modules/regional-oidc/variables.tf index 462e879e9..1cfb69984 100644 --- a/terraform/modules/regional-oidc/variables.tf +++ b/terraform/modules/regional-oidc/variables.tf @@ -15,4 +15,10 @@ variable "regional_id" { variable "mc_ou_path" { description = "AWS Organizations OU path for Management Cluster accounts (StringLike condition, supports wildcards)" type = string +} + +variable "force_destroy" { + description = "Allow the OIDC S3 bucket to be destroyed even when it contains objects. Enable for ephemeral environments." + type = bool + default = false } \ No newline at end of file diff --git a/terraform/modules/regional-oidc/versions.tf b/terraform/modules/regional-oidc/versions.tf index 914ada1b8..9d925b7d9 100644 --- a/terraform/modules/regional-oidc/versions.tf +++ b/terraform/modules/regional-oidc/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } \ No newline at end of file diff --git a/terraform/modules/rhobs-api-gateway/versions.tf b/terraform/modules/rhobs-api-gateway/versions.tf index 6a361dd6a..6e9abe15c 100644 --- a/terraform/modules/rhobs-api-gateway/versions.tf +++ b/terraform/modules/rhobs-api-gateway/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/sns-alerting/versions.tf b/terraform/modules/sns-alerting/versions.tf index 3dccf26c7..0b9cbf412 100644 --- a/terraform/modules/sns-alerting/versions.tf +++ b/terraform/modules/sns-alerting/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/sre-ui-alb/alb.tf b/terraform/modules/sre-ui-alb/alb.tf index 337724103..87490b54d 100644 --- a/terraform/modules/sre-ui-alb/alb.tf +++ b/terraform/modules/sre-ui-alb/alb.tf @@ -12,7 +12,6 @@ # argocd.sre.{deployment_name}.{domain} -> ArgoCD server :443 # prometheus.sre.{deployment_name}.{domain} -> Prometheus :9090 # thanos.sre.{deployment_name}.{domain} -> Thanos QFE :9090 -# loki.sre.{deployment_name}.{domain} -> Loki QFE :3100 # ============================================================================= locals { @@ -60,13 +59,6 @@ locals { health_path = "/-/ready" priority = 400 } - loki = { - tg_port = 3100 - protocol = "HTTP" - sg_port = 3100 - health_path = "/ready" - priority = 500 - } } # Unique container ports — used to generate security group rules. diff --git a/terraform/modules/sre-ui-alb/logs.tf b/terraform/modules/sre-ui-alb/logs.tf index 7b4a1c515..9934302a8 100644 --- a/terraform/modules/sre-ui-alb/logs.tf +++ b/terraform/modules/sre-ui-alb/logs.tf @@ -19,7 +19,8 @@ data "aws_elb_service_account" "current" {} # ----------------------------------------------------------------------------- resource "aws_s3_bucket" "access_logs" { - bucket = "${var.regional_id}-sre-alb-logs" + bucket = "${var.regional_id}-sre-alb-logs" + force_destroy = true tags = { Name = "${var.regional_id}-sre-alb-logs" diff --git a/terraform/modules/sre-ui-alb/outputs.tf b/terraform/modules/sre-ui-alb/outputs.tf index a2f622172..239120974 100644 --- a/terraform/modules/sre-ui-alb/outputs.tf +++ b/terraform/modules/sre-ui-alb/outputs.tf @@ -22,11 +22,6 @@ output "thanos_target_group_arn" { value = aws_lb_target_group.services["thanos"].arn } -output "loki_target_group_arn" { - description = "ARN of the Loki Query Frontend ALB target group" - value = aws_lb_target_group.services["loki"].arn -} - output "alb_dns_name" { description = "DNS name of the SRE UI ALB" value = aws_lb.sre.dns_name diff --git a/terraform/modules/sre-ui-alb/security-groups.tf b/terraform/modules/sre-ui-alb/security-groups.tf index 7c26c6213..a94b59e02 100644 --- a/terraform/modules/sre-ui-alb/security-groups.tf +++ b/terraform/modules/sre-ui-alb/security-groups.tf @@ -61,6 +61,29 @@ resource "aws_vpc_security_group_ingress_rule" "alb_https_from_cidr" { } +# Egress: HTTPS to OIDC identity provider (required for ALB authenticate-oidc token exchange) +# +# Destination: 0.0.0.0/0 — exception acknowledged. +# The OIDC provider (var.oidc_issuer_url, default: auth.redhat.com) resolves via dynamic +# IPs on Red Hat's CDN/load balancing infrastructure. No stable CIDR block or AWS managed +# prefix list is published for this endpoint, so destination restriction is not possible. +# +# Compensating controls: +# - Rule only exists when var.oidc_enabled = true (opt-in, not default) +# - Restricted to TCP:443 (HTTPS) only — no broad egress +# - ALB ingress is already restricted to allowed_source_cidrs +# - OIDC token exchange is mutually authenticated (client_id + client_secret) +resource "aws_vpc_security_group_egress_rule" "alb_to_oidc" { + count = var.oidc_enabled ? 1 : 0 + + security_group_id = aws_security_group.alb.id + description = "Allow HTTPS to OIDC IdP for token exchange (0.0.0.0/0 - dynamic IdP IPs)" + ip_protocol = "tcp" + from_port = 443 + to_port = 443 + cidr_ipv4 = "0.0.0.0/0" +} + # Egress: one rule per unique container port derived from local.services resource "aws_vpc_security_group_egress_rule" "alb_to_pods" { for_each = local.unique_sg_ports diff --git a/terraform/modules/sre-ui-alb/variables.tf b/terraform/modules/sre-ui-alb/variables.tf index 83f8b378d..15b517039 100644 --- a/terraform/modules/sre-ui-alb/variables.tf +++ b/terraform/modules/sre-ui-alb/variables.tf @@ -117,7 +117,7 @@ variable "oidc_issuer_url" { } variable "oidc_clients" { - description = "Per-service OIDC client credentials. Map key must match a service name in local.services (grafana, argocd, prometheus, thanos, loki). Required when oidc_enabled = true." + description = "Per-service OIDC client credentials. Map key must match a service name in local.services (grafana, argocd, prometheus, thanos). Required when oidc_enabled = true." type = map(object({ client_id = string client_secret = string diff --git a/terraform/modules/sre-ui-alb/versions.tf b/terraform/modules/sre-ui-alb/versions.tf index 6a361dd6a..6e9abe15c 100644 --- a/terraform/modules/sre-ui-alb/versions.tf +++ b/terraform/modules/sre-ui-alb/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/thanos-infrastructure/versions.tf b/terraform/modules/thanos-infrastructure/versions.tf index babd258e1..0b9cbf412 100644 --- a/terraform/modules/thanos-infrastructure/versions.tf +++ b/terraform/modules/thanos-infrastructure/versions.tf @@ -4,7 +4,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/vpc/versions.tf b/terraform/modules/vpc/versions.tf index 6a361dd6a..6e9abe15c 100644 --- a/terraform/modules/vpc/versions.tf +++ b/terraform/modules/vpc/versions.tf @@ -8,7 +8,7 @@ terraform { required_providers { aws = { source = "hashicorp/aws" - version = ">= 6.0" + version = "~> 6.56.0" } } } diff --git a/terraform/modules/zoa-job-pod-identity/versions.tf b/terraform/modules/zoa-job-pod-identity/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/modules/zoa-job-pod-identity/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} + diff --git a/terraform/modules/zoa/versions.tf b/terraform/modules/zoa/versions.tf new file mode 100644 index 000000000..381655a2e --- /dev/null +++ b/terraform/modules/zoa/versions.tf @@ -0,0 +1,11 @@ +terraform { + required_version = ">= 1.14.3" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.56.0" + } + } +} +