From 94ac2abaa0b078903acb49c7eb7026441fa99362 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 12:54:29 -0700 Subject: [PATCH 01/11] =?UTF-8?q?feat(heroku-to-aws):=20live=20discovery?= =?UTF-8?q?=20via=20the=20authenticated=20Heroku=20CLI=20Discovery=20previ?= =?UTF-8?q?ously=20required=20Terraform=20with=20heroku=5F*=20resources,?= =?UTF-8?q?=20which=20most=20startups=20on=20Heroku=20don't=20have.=20Adds?= =?UTF-8?q?=20a=20consent-gated,=20read-only=20live=20discovery=20path:=20?= =?UTF-8?q?an=20interactive=20capture=20step=20(exact-command=20whitelist,?= =?UTF-8?q?=20config-var=20key=20names=20only,=20auth:token=20banned)=20wr?= =?UTF-8?q?ites=20raw=20CLI=20output=20to=20live-capture/,=20and=20a=20new?= =?UTF-8?q?=20'live'=20fragment=20parses=20it=20into=20the=20inventory.=20?= =?UTF-8?q?When=20Terraform=20is=20also=20present,=20live=20wins=20for=20c?= =?UTF-8?q?urrent=20state=20and=20disagreements=20surface=20as=20drift=20(?= =?UTF-8?q?unmanaged=20resources,=20plan=20changes,=20config=20conflicts)?= =?UTF-8?q?=20=E2=80=94=20never=20silently=20resolved.=20Includes=20replay?= =?UTF-8?q?=20fixtures=20with=20a=20drift-assertion=20set,=20validated=20b?= =?UTF-8?q?y=20end-to-end=20fresh-agent=20runs=20of=20the=20Discover=20pha?= =?UTF-8?q?se.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- migrate/README.md | 9 +- migrate/plugins/migration-to-aws/README.md | 20 +- .../fixtures/heroku-live-capture/README.md | 54 ++++++ .../heroku-live-capture/expected-drift.json | 82 ++++++++ .../live-capture/addons.json | 50 +++++ .../live-capture/app-acme-staging.json | 23 +++ .../live-capture/app-acme-web.json | 44 +++++ .../live-capture/apps.json | 35 ++++ .../config-keys-acme-staging.json | 6 + .../live-capture/config-keys-acme-web.json | 11 ++ .../live-capture/domains-acme-staging.json | 9 + .../live-capture/domains-acme-web.json | 16 ++ .../live-capture/manifest.json | 68 +++++++ .../live-capture/pg-acme-staging.out | 13 ++ .../live-capture/pg-acme-web.out | 16 ++ .../live-capture/pipeline-acme.json | 7 + .../live-capture/pipelines.json | 8 + .../live-capture/ps-acme-staging.json | 11 ++ .../live-capture/ps-acme-web.json | 29 +++ .../live-capture/redis-acme-web.out | 13 ++ .../live-capture/spaces.json | 1 + .../workspace-terraform/heroku.tf | 47 +++++ .../skills/heroku-to-aws/SKILL.md | 10 +- .../phases/discover/discover-assemble.md | 45 ++++- .../phases/discover/discover-live-capture.md | 162 ++++++++++++++++ .../phases/discover/discover-live.md | 176 ++++++++++++++++++ .../references/phases/discover/discover.md | 65 +++++-- .../shared/schema-discover-heroku.md | 63 +++++-- 28 files changed, 1051 insertions(+), 42 deletions(-) create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/README.md create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/expected-drift.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/addons.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-staging.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-web.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/apps.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-staging.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-web.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-staging.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-web.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/manifest.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-staging.out create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-web.out create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipeline-acme.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipelines.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-staging.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-web.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/redis-acme-web.out create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/spaces.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/workspace-terraform/heroku.tf create mode 100644 migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live-capture.md create mode 100644 migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live.md diff --git a/migrate/README.md b/migrate/README.md index 1b5ea629..4fa73558 100644 --- a/migrate/README.md +++ b/migrate/README.md @@ -4,7 +4,7 @@ AI agent skills for migrating workloads to AWS, built for [Claude Code](https:// ## What This Does -Point this plugin at your Terraform files, application code, or billing data. It runs a structured 6-phase assessment — discovering what you have, asking the right questions, designing the AWS architecture, estimating costs with real pricing data, and generating runnable migration artifacts. +Point this plugin at your Heroku account (via your authenticated Heroku CLI, read-only and consent-gated), your Terraform files, application code, or billing data. It runs a structured 6-phase assessment — discovering what you have, asking the right questions, designing the AWS architecture, estimating costs with real pricing data, and generating runnable migration artifacts. **Supported migration sources:** @@ -82,6 +82,7 @@ After installation, just describe what you want to migrate: **Heroku migrations:** - "Migrate my Heroku app to AWS" +- "Discover my Heroku apps and estimate AWS costs" - "Move my Heroku Postgres to RDS" - "Migrate from Heroku to Fargate" - "Migrate from Heroku to Elastic Beanstalk" @@ -90,6 +91,8 @@ After installation, just describe what you want to migrate: The skill creates a `.migration//` directory in the current working directory with all artifacts. +**Live Heroku discovery — how it works:** No Terraform or exports needed. If `heroku login` works in your terminal, just ask — the agent requests your consent, then inventories your account using read-only list/info CLI commands. It captures app names, dyno types, add-on plans and prices, domains, pipelines, and config var **key names only**. It never reads config var values, credentials, or your API token, and never runs a command that creates, changes, or deletes anything. If you also have `heroku_*` Terraform, the agent cross-checks it against your live account and reports drift. + ## What It Detects ### GCP → AWS @@ -164,9 +167,9 @@ The skill creates a `.migration//` directory in the current working dir - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: Terraform files, application code, or billing data +- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data - **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) -- **For Heroku migration:** Terraform files with `heroku_*` resources are required (Procfile/app.json supplements but cannot stand alone) +- **For Heroku migration:** an authenticated Heroku CLI (recommended — live, read-only discovery with your consent) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone). When both are available, live data is authoritative for current state and Terraform drift is surfaced. ## Structure diff --git a/migrate/plugins/migration-to-aws/README.md b/migrate/plugins/migration-to-aws/README.md index 98945f52..6263ab06 100644 --- a/migrate/plugins/migration-to-aws/README.md +++ b/migrate/plugins/migration-to-aws/README.md @@ -4,7 +4,7 @@ AI agent skills for migrating workloads to AWS, built for [Claude Code](https:// ## What This Does -Point this plugin at your Terraform files, application code, or billing data. It runs a structured 6-phase assessment — discovering what you have, asking the right questions, designing the AWS architecture, estimating costs with real pricing data, and generating runnable migration artifacts. +Point this plugin at your Heroku account (via your authenticated Heroku CLI, read-only and consent-gated), your Terraform files, application code, or billing data. It runs a structured 6-phase assessment — discovering what you have, asking the right questions, designing the AWS architecture, estimating costs with real pricing data, and generating runnable migration artifacts. **Supported migration sources:** @@ -103,7 +103,7 @@ ln -s "$(pwd)" ~/.cursor/plugins/local/migration-to-aws ### Workflow -1. **Discover** — Scan Terraform files, application code, and/or billing data. Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. +1. **Discover** — Scan Terraform files, application code, and/or billing data — or, for Heroku, inventory your account live via the authenticated Heroku CLI (read-only, consent-gated). Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. 2. **Clarify** — Ask targeted questions about migration preferences, AI priorities, agentic migration approach, database sizing, and timeline. 3. **Design** — Map source services to AWS equivalents. For AI workloads: select Bedrock models with honest pricing comparison. For agentic workloads: design AgentCore Harness config or Strands architecture. 4. **Estimate** — Calculate monthly AWS costs using real-time pricing data. Compare to current spend. @@ -198,9 +198,21 @@ The `--json` verdict lists each violation with `file`, `line`, `rule`, and `fix_ - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: Terraform files, application code, or billing data +- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data - **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) -- **For Heroku migration:** Terraform files with `heroku_*` resources are required (Procfile/app.json supplements but cannot stand alone) +- **For Heroku migration:** an authenticated Heroku CLI (recommended) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone) + +### Live Heroku discovery — how it works + +No Terraform or exports needed. If `heroku login` works in your terminal, just ask +your agent to migrate ("Migrate my Heroku app to AWS" or "Discover my Heroku apps +and estimate AWS costs"). The agent asks for your consent, then inventories your +account using read-only list/info CLI commands — it captures app names, dyno types, +add-on plans and prices, domains, pipelines, and config var **key names only**. It +never reads config var values, credentials, or your API token, and never runs a +command that creates, changes, or deletes anything. If you also have `heroku_*` +Terraform, the agent cross-checks it against your live account and reports drift. + - **For AI execution (llm-to-bedrock skill):** Python 3.10+, `uv`, and Bedrock model access enabled - **`uvx` required for cost estimation:** The `awspricing` MCP server runs via [`uvx`](https://docs.astral.sh/uv/guides/tools/) (part of the `uv` Python package manager). Install with `pip install uv` or `brew install uv`. Without it, the Estimate phase falls back to cached pricing — migration still works but live pricing lookups are unavailable. diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/README.md new file mode 100644 index 00000000..18b2cf22 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/README.md @@ -0,0 +1,54 @@ +# Heroku live-discovery fixtures (replay mode) + +Canned Heroku CLI outputs for testing the `heroku-to-aws` live-discovery path +(`discover-live-capture.md` → `discover-live.md` → `discover-assemble.md`) without +a Heroku account. The fake account is deliberately small but exercises every +designed-for behavior: + +- **acme-web** — production app: 2× Standard-2X web + 1× Standard-1X worker, + Postgres `standard-2` (42.3 GB — large enough to matter for migration-tool + selection), Redis `premium-0` (HA + TLS), Papertrail, one custom domain, and a + config var list whose keys imply an AI workload (`OPENAI_API_KEY`). +- **acme-staging** — small staging app on `essential-0`. +- **acme-data-team** — a team app the captured account cannot read: every per-app + capture is `failed` with a 403 in `manifest.json`. Expected result: + `discovery_status: "discovery_failed"`, confidence `reduced`, run continues. +- **kafka** capture is `skipped` (plugin not installed) — expected to be a + warning, never a halt. +- `spaces.json` is empty — the common startup case. + +## How to replay + +**Scenario A — live-only (no Terraform):** + +1. Create a scratch directory containing NO `.tf` files. +2. Create `.migration/0715-1820/` and copy `live-capture/` into it. +3. Invoke the heroku-to-aws skill ("migrate my Heroku app to AWS"). +4. The Discover phase's source precondition passes via the manifest; the `live` + fragment parses the captures. Expect an inventory with `discovery_sources: + ["live"]`, 2 successful apps + 1 failed, and no `drift` key. + +**Scenario B — live + Terraform (drift exercise):** + +1. As above, but also copy `workspace-terraform/heroku.tf` into the scratch + directory root. +2. `heroku.tf` is deliberately stale — each divergence is commented with the + merge rule it exercises (config conflicts, plan change, terraform-only, + live-only, scaled-to-zero gap-fill). +3. Check the assembled `heroku-resource-inventory.json` against + `expected-drift.json` — it lists the assertions (merged sources, conflict + fields, drift counts, and `must_not_exist` entries). + +**What a run must never produce** (from either scenario): + +- Config var values anywhere (fixture keys like `STRIPE_SECRET_KEY` are key + NAMES; if a value shows up, the keys-only rule broke) +- Clustering fields (`cluster_id`, `edges`, `dependencies`, ...) +- A halt due to the 403 app or the skipped kafka capture + +## Regenerating / extending + +Captures follow the exact command whitelist in `discover-live-capture.md` Step 3 +(same filenames, `.out` extension for text captures). If you add a capture type, +add its row to the whitelist first, then the fixture, then extend +`expected-drift.json`. All IDs, names, and emails are synthetic. diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/expected-drift.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/expected-drift.json new file mode 100644 index 00000000..4d9cdc8e --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/expected-drift.json @@ -0,0 +1,82 @@ +{ + "_comment": "Expected merge/drift outcomes when BOTH workspace-terraform/ and live-capture/ are used (scenario B in README.md). Not a full inventory — these are the assertions an eval run checks against heroku-resource-inventory.json.", + "metadata": { + "total_apps_discovered": 3, + "discovery_sources_must_include": ["terraform", "live"], + "confidence": "reduced", + "confidence_reason": "acme-data-team captures failed (403)" + }, + "apps": { + "acme-web": { + "discovery_status": "success", + "heroku_generation": "cedar", + "app_id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab" + }, + "acme-staging": { "discovery_status": "success", "heroku_generation": "cedar", "unmanaged_by_terraform": true }, + "acme-data-team": { "discovery_status": "discovery_failed", "failure_reason_contains": "403" } + }, + "merged_resources": { + "formation:acme-web:web": { + "source": "live+terraform", + "config": { "quantity": 2, "dyno_type": "standard-2x", "command": "npm start" }, + "expected_config_conflicts": ["quantity", "dyno_type"] + }, + "formation:acme-web:release": { + "source": "terraform", + "config": { "quantity": 0 }, + "note": "gap-fill rule 4 — scaled to zero, invisible live, NOT drift" + }, + "addon:acme-web:heroku-postgresql:standard-2": { + "source": "live+terraform", + "expected_config_conflicts": ["plan"], + "note": "rule 1a treats the plan change (tf standard-0 vs live standard-2) as ONE merged resource with a plan conflict, so source is live+terraform per rule 1", + "config": { "data_size_gb": 42.3, "pg_version": "16.4", "connection_pooling": true, "monthly_price_usd": 200.0 } + }, + "addon:acme-web:scheduler:standard": { "source": "terraform", "not_found_live": true }, + "addon:acme-web:heroku-redis:premium-0": { + "source": "live", + "unmanaged_by_terraform": true, + "config": { + "ha_enabled": true, + "encryption_in_transit": true, + "redis_version": "7.2.5", + "maxmemory_policy": "noeviction", + "monthly_price_usd": 15.0 + } + }, + "addon:acme-web:papertrail:choklad": { "source": "live", "unmanaged_by_terraform": true }, + "addon:acme-staging:heroku-postgresql:essential-0": { + "source": "live", + "unmanaged_by_terraform": true, + "config": { "data_size_gb": 0.4 } + }, + "formation:acme-staging:web": { + "source": "live", + "unmanaged_by_terraform": true, + "config": { "dyno_type": "basic", "quantity": 1 } + }, + "config:acme-web": { "config": { "config_var_keys_count": 9 }, "must_not_contain_values": true }, + "domain:acme-web:www.acme-demo.com": { "source": "live" }, + "pipeline:acme": { + "source": "live", + "config": { + "stages": [{ "stage": "staging", "app": "acme-staging" }, { "stage": "production", "app": "acme-web" }] + } + } + }, + "must_not_exist": [ + "domain:acme-web:acme-web-1a2b3c4d5e6f.herokuapp.com (default herokuapp.com domains are counted, not resources)", + "any resource with cluster_id / edges / dependencies fields", + "any config var VALUE anywhere in the document" + ], + "live_metadata": { + "apps_captured": 2, + "apps_failed": 1, + "limitations_must_include": "formations scaled to zero are not visible to live discovery", + "drift": { + "resources_live_only_min": 5, + "resources_terraform_only": 1, + "config_conflicts_expected_fields": ["quantity", "dyno_type", "plan"] + } + } +} diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/addons.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/addons.json new file mode 100644 index 00000000..7bb39b6a --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/addons.json @@ -0,0 +1,50 @@ +[ + { + "id": "ad-0001", + "name": "postgresql-cubed-12345", + "app": { "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", "name": "acme-web" }, + "addon_service": { "id": "as-0001", "name": "heroku-postgresql", "human_name": "Heroku Postgres" }, + "plan": { + "id": "p-0001", + "name": "heroku-postgresql:standard-2", + "price": { "cents": 20000, "unit": "month", "contract": false } + }, + "state": "provisioned" + }, + { + "id": "ad-0002", + "name": "redis-fluffy-67890", + "app": { "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", "name": "acme-web" }, + "addon_service": { "id": "as-0002", "name": "heroku-redis", "human_name": "Heroku Data for Redis" }, + "plan": { + "id": "p-0002", + "name": "heroku-redis:premium-0", + "price": { "cents": 1500, "unit": "month", "contract": false } + }, + "state": "provisioned" + }, + { + "id": "ad-0003", + "name": "papertrail-solid-11111", + "app": { "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", "name": "acme-web" }, + "addon_service": { "id": "as-0003", "name": "papertrail", "human_name": "Papertrail" }, + "plan": { + "id": "p-0003", + "name": "papertrail:choklad", + "price": { "cents": 0, "unit": "month", "contract": false } + }, + "state": "provisioned" + }, + { + "id": "ad-0004", + "name": "postgresql-round-22222", + "app": { "id": "a1b2c3d4-0002-4a5b-8c6d-0123456789ab", "name": "acme-staging" }, + "addon_service": { "id": "as-0001", "name": "heroku-postgresql", "human_name": "Heroku Postgres" }, + "plan": { + "id": "p-0004", + "name": "heroku-postgresql:essential-0", + "price": { "cents": 500, "unit": "month", "contract": false } + }, + "state": "provisioned" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-staging.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-staging.json new file mode 100644 index 00000000..12f99964 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-staging.json @@ -0,0 +1,23 @@ +{ + "app": { + "id": "a1b2c3d4-0002-4a5b-8c6d-0123456789ab", + "name": "acme-staging", + "region": { "name": "us" }, + "stack": { "name": "heroku-22" }, + "space": null, + "buildpack_provided_description": "Node.js", + "web_url": "https://acme-staging-9f8e7d6c5b4a.herokuapp.com/" + }, + "addons": [ + { + "id": "ad-0004", + "name": "postgresql-round-22222", + "addon_service": { "name": "heroku-postgresql" }, + "plan": { "name": "heroku-postgresql:essential-0" } + } + ], + "collaborators": [{ "email": "founder@acme-demo.com" }], + "dynos": [ + { "id": "d-0004", "name": "web.1", "type": "web", "size": "Basic", "state": "up", "command": "npm start" } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-web.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-web.json new file mode 100644 index 00000000..86904c5e --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/app-acme-web.json @@ -0,0 +1,44 @@ +{ + "app": { + "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", + "name": "acme-web", + "region": { "name": "us" }, + "stack": { "name": "heroku-22" }, + "space": null, + "buildpack_provided_description": "Node.js", + "web_url": "https://acme-web-1a2b3c4d5e6f.herokuapp.com/" + }, + "addons": [ + { + "id": "ad-0001", + "name": "postgresql-cubed-12345", + "addon_service": { "name": "heroku-postgresql" }, + "plan": { "name": "heroku-postgresql:standard-2" } + }, + { + "id": "ad-0002", + "name": "redis-fluffy-67890", + "addon_service": { "name": "heroku-redis" }, + "plan": { "name": "heroku-redis:premium-0" } + }, + { + "id": "ad-0003", + "name": "papertrail-solid-11111", + "addon_service": { "name": "papertrail" }, + "plan": { "name": "papertrail:choklad" } + } + ], + "collaborators": [{ "email": "founder@acme-demo.com" }], + "dynos": [ + { "id": "d-0001", "name": "web.1", "type": "web", "size": "Standard-2X", "state": "up", "command": "npm start" }, + { "id": "d-0002", "name": "web.2", "type": "web", "size": "Standard-2X", "state": "up", "command": "npm start" }, + { + "id": "d-0003", + "name": "worker.1", + "type": "worker", + "size": "Standard-1X", + "state": "up", + "command": "node worker.js" + } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/apps.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/apps.json new file mode 100644 index 00000000..e6020669 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/apps.json @@ -0,0 +1,35 @@ +[ + { + "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", + "name": "acme-web", + "region": { "id": "r-0001", "name": "us" }, + "stack": { "id": "s-0001", "name": "heroku-22" }, + "space": null, + "web_url": "https://acme-web-1a2b3c4d5e6f.herokuapp.com/", + "maintenance": false, + "created_at": "2024-05-01T12:00:00Z", + "updated_at": "2026-07-01T08:15:00Z" + }, + { + "id": "a1b2c3d4-0002-4a5b-8c6d-0123456789ab", + "name": "acme-staging", + "region": { "id": "r-0001", "name": "us" }, + "stack": { "id": "s-0001", "name": "heroku-22" }, + "space": null, + "web_url": "https://acme-staging-9f8e7d6c5b4a.herokuapp.com/", + "maintenance": false, + "created_at": "2024-05-01T12:05:00Z", + "updated_at": "2026-06-20T14:00:00Z" + }, + { + "id": "a1b2c3d4-0003-4a5b-8c6d-0123456789ab", + "name": "acme-data-team", + "region": { "id": "r-0001", "name": "us" }, + "stack": { "id": "s-0002", "name": "heroku-24" }, + "space": null, + "web_url": "https://acme-data-team-3c4d5e6f7a8b.herokuapp.com/", + "maintenance": false, + "created_at": "2025-01-10T09:00:00Z", + "updated_at": "2026-07-10T11:30:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-staging.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-staging.json new file mode 100644 index 00000000..f0688277 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-staging.json @@ -0,0 +1,6 @@ +[ + "DATABASE_URL", + "LANG", + "NODE_ENV", + "SESSION_SECRET" +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-web.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-web.json new file mode 100644 index 00000000..cd98e316 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/config-keys-acme-web.json @@ -0,0 +1,11 @@ +[ + "DATABASE_URL", + "LANG", + "NODE_ENV", + "OPENAI_API_KEY", + "REDIS_TLS_URL", + "REDIS_URL", + "SENTRY_DSN", + "SESSION_SECRET", + "STRIPE_SECRET_KEY" +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-staging.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-staging.json new file mode 100644 index 00000000..f8736f5e --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-staging.json @@ -0,0 +1,9 @@ +[ + { + "id": "do-0003", + "hostname": "acme-staging-9f8e7d6c5b4a.herokuapp.com", + "kind": "heroku", + "status": "none", + "sni_endpoint": null + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-web.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-web.json new file mode 100644 index 00000000..bcea1528 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/domains-acme-web.json @@ -0,0 +1,16 @@ +[ + { + "id": "do-0001", + "hostname": "acme-web-1a2b3c4d5e6f.herokuapp.com", + "kind": "heroku", + "status": "none", + "sni_endpoint": null + }, + { + "id": "do-0002", + "hostname": "www.acme-demo.com", + "kind": "custom", + "status": "succeeded", + "sni_endpoint": { "id": "sni-0001", "name": "tokyo-1234" } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/manifest.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/manifest.json new file mode 100644 index 00000000..a65dfe05 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/manifest.json @@ -0,0 +1,68 @@ +{ + "captured_at": "2026-07-15T18:20:00Z", + "cli_version": "heroku/10.12.0 darwin-arm64 node-v20.19.0", + "account": "founder@acme-demo.com", + "apps_selected": ["acme-web", "acme-staging", "acme-data-team"], + "captures": [ + { "command": "heroku apps --all --json", "file": "apps.json", "status": "ok", "note": null }, + { "command": "heroku apps:info -a acme-web --json", "file": "app-acme-web.json", "status": "ok", "note": null }, + { + "command": "heroku apps:info -a acme-staging --json", + "file": "app-acme-staging.json", + "status": "ok", + "note": null + }, + { + "command": "heroku apps:info -a acme-data-team --json", + "file": "app-acme-data-team.json", + "status": "failed", + "note": "403 Forbidden — no access to team app" + }, + { "command": "heroku ps -a acme-web --json", "file": "ps-acme-web.json", "status": "ok", "note": null }, + { "command": "heroku ps -a acme-staging --json", "file": "ps-acme-staging.json", "status": "ok", "note": null }, + { + "command": "heroku ps -a acme-data-team --json", + "file": "ps-acme-data-team.json", + "status": "failed", + "note": "403 Forbidden — no access to team app" + }, + { "command": "heroku addons --all --json", "file": "addons.json", "status": "ok", "note": null }, + { + "command": "heroku config -a acme-web --json | jq 'keys'", + "file": "config-keys-acme-web.json", + "status": "ok", + "note": null + }, + { + "command": "heroku config -a acme-staging --json | jq 'keys'", + "file": "config-keys-acme-staging.json", + "status": "ok", + "note": null + }, + { + "command": "heroku config -a acme-data-team --json | jq 'keys'", + "file": "config-keys-acme-data-team.json", + "status": "failed", + "note": "403 Forbidden — no access to team app" + }, + { "command": "heroku domains -a acme-web --json", "file": "domains-acme-web.json", "status": "ok", "note": null }, + { + "command": "heroku domains -a acme-staging --json", + "file": "domains-acme-staging.json", + "status": "ok", + "note": null + }, + { "command": "heroku pg:info -a acme-web", "file": "pg-acme-web.out", "status": "ok", "note": null }, + { "command": "heroku pg:info -a acme-staging", "file": "pg-acme-staging.out", "status": "ok", "note": null }, + { "command": "heroku redis:info -a acme-web", "file": "redis-acme-web.out", "status": "ok", "note": null }, + { + "command": "heroku kafka:info -a acme-web", + "file": "kafka-acme-web.out", + "status": "skipped", + "note": "kafka CLI plugin not installed; no kafka add-on detected" + }, + { "command": "heroku pipelines --json", "file": "pipelines.json", "status": "ok", "note": null }, + { "command": "heroku pipelines:info acme --json", "file": "pipeline-acme.json", "status": "ok", "note": null }, + { "command": "heroku spaces --json", "file": "spaces.json", "status": "ok", "note": null } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-staging.out b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-staging.out new file mode 100644 index 00000000..29ca0e0f --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-staging.out @@ -0,0 +1,13 @@ +=== DATABASE_URL +Plan: Essential 0 +Status: Available +Data Size: 0.4 GB / 1 GB (40.00%) +Tables: 87 +PG Version: 16.4 +Connections: 3/20 +Credentials: 1 +Rollback: Unsupported +Created: 2024-05-01 12:15 UTC +Region: us +Data Encryption: In Use +Maintenance: not required diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-web.out b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-web.out new file mode 100644 index 00000000..a1e3cec7 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pg-acme-web.out @@ -0,0 +1,16 @@ +=== DATABASE_URL +Plan: Standard 2 +Status: Available +Data Size: 42.3 GB / 256 GB (16.52%) +Tables: 87 +PG Version: 16.4 +Connections: 38/400 +Connection Pooling: Available +Credentials: 1 +Fork/Follow: Available +Rollback: earliest from 2026-07-11 18:00 UTC +Created: 2024-05-01 12:10 UTC +Region: us +Data Encryption: In Use +Continuous Protection: On +Maintenance: not required diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipeline-acme.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipeline-acme.json new file mode 100644 index 00000000..ed5b5778 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipeline-acme.json @@ -0,0 +1,7 @@ +{ + "pipeline": { "id": "pl-0001", "name": "acme" }, + "apps": [ + { "id": "a1b2c3d4-0002-4a5b-8c6d-0123456789ab", "name": "acme-staging", "coupling": { "stage": "staging" } }, + { "id": "a1b2c3d4-0001-4a5b-8c6d-0123456789ab", "name": "acme-web", "coupling": { "stage": "production" } } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipelines.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipelines.json new file mode 100644 index 00000000..d4eb30c2 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/pipelines.json @@ -0,0 +1,8 @@ +[ + { + "id": "pl-0001", + "name": "acme", + "created_at": "2024-05-02T09:00:00Z", + "updated_at": "2026-06-01T10:00:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-staging.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-staging.json new file mode 100644 index 00000000..58a1d4fb --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-staging.json @@ -0,0 +1,11 @@ +[ + { + "id": "d-0004", + "name": "web.1", + "type": "web", + "size": "Basic", + "state": "up", + "command": "npm start", + "updated_at": "2026-07-14T22:10:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-web.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-web.json new file mode 100644 index 00000000..4a42a830 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/ps-acme-web.json @@ -0,0 +1,29 @@ +[ + { + "id": "d-0001", + "name": "web.1", + "type": "web", + "size": "Standard-2X", + "state": "up", + "command": "npm start", + "updated_at": "2026-07-15T06:00:00Z" + }, + { + "id": "d-0002", + "name": "web.2", + "type": "web", + "size": "Standard-2X", + "state": "up", + "command": "npm start", + "updated_at": "2026-07-15T06:00:00Z" + }, + { + "id": "d-0003", + "name": "worker.1", + "type": "worker", + "size": "Standard-1X", + "state": "up", + "command": "node worker.js", + "updated_at": "2026-07-15T06:00:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/redis-acme-web.out b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/redis-acme-web.out new file mode 100644 index 00000000..3d258057 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/redis-acme-web.out @@ -0,0 +1,13 @@ +=== redis-fluffy-67890 (REDIS_URL) +Plan: Premium 0 +Status: available +Created: 2024-06-15 10:00 UTC +Version: 7.2.5 +Timeout: 300 +Maxmemory: noeviction +Maintenance: not required +Maintenance window: Mondays 22:00 to Tuesdays 02:00 UTC +Persistence: AOF +HA Status: Available +Requires TLS: Yes +Keyspace Notifications: Disabled diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/spaces.json b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/spaces.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/live-capture/spaces.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/workspace-terraform/heroku.tf b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/workspace-terraform/heroku.tf new file mode 100644 index 00000000..fb9fea00 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/heroku-live-capture/workspace-terraform/heroku.tf @@ -0,0 +1,47 @@ +# Fixture Terraform — deliberately STALE relative to the live captures in +# ../live-capture/. Each divergence below exercises a specific rule in +# discover-assemble.md § Merge & Drift Rules. Do not "fix" the drift. + +resource "heroku_app" "web" { + name = "acme-web" + region = "us" + stack = "heroku-22" +} + +# Drift: live runs 2x Standard-2X; Terraform says 1x standard-1x. +# → merge rule 1 (config conflicts on quantity + dyno_type, live wins) +resource "heroku_formation" "web" { + app_id = heroku_app.web.id + type = "web" + quantity = 1 + size = "standard-1x" +} + +# Scaled-to-zero release process: invisible to `heroku ps`, declared here. +# → merge rule 4 (gap-fill from Terraform, expected complement not conflict) +resource "heroku_formation" "release" { + app_id = heroku_app.web.id + type = "release" + quantity = 0 + size = "standard-1x" +} + +# Drift: live plan is standard-2; Terraform still says standard-0. +# → merge rule 1a (plan change = config conflict, NOT an add/remove pair) +resource "heroku_addon" "postgres" { + app_id = heroku_app.web.id + plan = "heroku-postgresql:standard-0" +} + +# Declared but never deployed (absent from live captures). +# → merge rule 3 (terraform-only, not_found_live: true) +resource "heroku_addon" "scheduler" { + app_id = heroku_app.web.id + plan = "scheduler:standard" +} + +# NOTE deliberate absences from this file (present in live captures): +# - app acme-staging and everything on it → merge rule 2 (unmanaged_by_terraform) +# - heroku-redis:premium-0 on acme-web → merge rule 2 (unmanaged_by_terraform) +# - papertrail:choklad on acme-web → merge rule 2 (unmanaged_by_terraform) +# - pipeline "acme" and custom domain → merge rule 2 (unmanaged_by_terraform) diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/SKILL.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/SKILL.md index 100fd326..e3c23061 100644 --- a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/SKILL.md +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/SKILL.md @@ -1,6 +1,6 @@ --- name: heroku-to-aws -description: "Migrate workloads from Heroku to AWS. Triggers on: migrate from Heroku, Heroku to AWS, move off Heroku, migrate Heroku app, migrate Heroku Postgres to RDS, migrate Heroku Redis to ElastiCache, migrate Heroku Kafka to MSK, migrate dynos to Elastic Beanstalk, migrate dynos to Fargate, Heroku migration, move from Heroku to AWS, migrate Heroku Private Space, Heroku to Elastic Beanstalk, Heroku to ECS, Heroku to Fargate, leave Heroku, migrate off Heroku platform. Runs a 6-phase process: discover Heroku resources from Terraform files, Procfile/app.json, and optional billing exports, clarify migration requirements, design AWS architecture, estimate costs, generate migration artifacts, and collect optional feedback. Clarify must finish before Design, Estimate, or Generate. Uses a flat resource model (no clustering or dependency graphs) with deterministic mapping tables for core services (Dynos → Elastic Beanstalk by default, Postgres → RDS/Aurora, Redis → ElastiCache, Kafka → MSK) and a fast-path table for 13+ common add-ons. Cedar/Fir generation detection is detect-only in v1. Pipeline/Review Apps are detect-only. Do not use for: GCP or Azure migrations to AWS, AWS-to-Heroku reverse migration, general AWS architecture advice without migration intent, Heroku-to-Heroku refactoring, or multi-cloud deployments that do not involve migrating off Heroku." +description: "Migrate workloads from Heroku to AWS. Triggers on: migrate from Heroku, Heroku to AWS, move off Heroku, migrate Heroku app, migrate Heroku Postgres to RDS, migrate Heroku Redis to ElastiCache, migrate Heroku Kafka to MSK, migrate dynos to Elastic Beanstalk, migrate dynos to Fargate, Heroku migration, move from Heroku to AWS, migrate Heroku Private Space, Heroku to Elastic Beanstalk, Heroku to ECS, Heroku to Fargate, leave Heroku, migrate off Heroku platform. Runs a 6-phase process: discover Heroku resources live via the authenticated Heroku CLI (read-only, consent-gated) and/or from Terraform files, Procfile/app.json, and optional billing exports, clarify migration requirements, design AWS architecture, estimate costs, generate migration artifacts, and collect optional feedback. Clarify must finish before Design, Estimate, or Generate. Uses a flat resource model (no clustering or dependency graphs) with deterministic mapping tables for core services (Dynos → Elastic Beanstalk by default, Postgres → RDS/Aurora, Redis → ElastiCache, Kafka → MSK) and a fast-path table for 13+ common add-ons. Cedar/Fir generation detection is detect-only in v1. Pipeline/Review Apps are detect-only. Do not use for: GCP or Azure migrations to AWS, AWS-to-Heroku reverse migration, general AWS architecture advice without migration intent, Heroku-to-Heroku refactoring, or multi-cloud deployments that do not involve migrating off Heroku." --- # Heroku-to-AWS Migration Skill @@ -13,7 +13,7 @@ description: "Migrate workloads from Heroku to AWS. Triggers on: migrate from He - **Re-platform by default**: Select AWS services that match Heroku workload types (e.g., Dynos → Elastic Beanstalk, Heroku Postgres → RDS/Aurora, Heroku Redis → ElastiCache, Kafka → MSK). - **Dev sizing unless specified**: Default to development-tier capacity (e.g., db.t4g.micro, single AZ). Upgrade only on user direction. - **No human one-time migration costs**: Do not present human labor, professional services, or people-time work as dollar estimates or "one-time migration cost" budget categories. Vendor charges grounded in data (for example Heroku invoice line items in the infra estimate when billing exists) are allowed. -- **Terraform + repo as primary discovery**: Terraform files (`.tf` with `heroku_*` resources) and repo artifacts (Procfile, app.json) are the primary data sources for resource discovery. No Platform API calls in v1. +- **Live-first discovery, read-only and consent-gated**: The user's authenticated Heroku CLI is a first-class discovery source — most startups have no `heroku_*` Terraform, and the account is authoritative for what actually runs. Live capture is strictly read-only (an exact-command whitelist of list/info commands), requires explicit consent, never captures config var values (key names only), and never extracts the API token. Terraform files (`.tf` with `heroku_*` resources) and repo artifacts (Procfile, app.json) remain fully supported; when both live and Terraform data exist, live wins for current state, Terraform supplements structure and provenance, and disagreements are surfaced as drift — never silently resolved. - **Flat resource model**: Heroku resources are organized per-app without dependency graphs or clustering. No topological sorting, typed edges, or cluster formation logic. Resources are processed as a flat list in input order. - **Deterministic mappings**: Core services use fixed lookup tables (Dyno Type Table, Postgres Plan Table, Redis Plan Table, Kafka Plan Table). Common add-ons use the Fast-Path Table. Unknown add-ons hit the specialist gate. - **DMS has Heroku constraints**: AWS DMS cannot perform continuous replication (CDC) with Heroku Postgres because Heroku does not grant the REPLICATION role. DMS is for one-time bulk migration with a cutover window only. The skill must surface this constraint when DMS is selected. @@ -114,7 +114,9 @@ heroku-to-aws/ │ ├── phases/ │ │ ├── discover/ │ │ │ ├── discover.md # Phase 1: Discover orchestrator -│ │ │ ├── discover-terraform.md # Terraform discovery (primary) +│ │ │ ├── discover-terraform.md # Terraform discovery +│ │ │ ├── discover-live-capture.md # Live CLI capture (main-window pre-work, consent-gated) +│ │ │ ├── discover-live.md # Live discovery fragment (parses live-capture/) │ │ │ └── discover-billing.md # Billing data parsing │ │ ├── clarify/ │ │ │ └── clarify.md # Phase 2: Adaptive questions (12–15, batched ≤5) @@ -158,7 +160,7 @@ heroku-to-aws/ - **IaC output**: Terraform configurations, migration scripts, and documentation - **Region**: `us-east-1` (unless user specifies otherwise) - **Sizing**: Development tier (e.g., `db.t4g.micro` for databases, 0.5 CPU for Fargate) -- **Migration mode**: Adapts based on available inputs (Terraform primary, Procfile/app.json supplementary, billing optional) +- **Migration mode**: Adapts based on available inputs (live CLI discovery recommended, Terraform supported, Procfile/app.json supplementary, billing optional) - **Cost currency**: USD - **Timeline assumption**: 2-16 weeks depending on migration complexity — small (2-6 weeks), medium (6-12 weeks), large (12-18 weeks). Complexity tiers are classified per `references/vendored/estimate/complexity-tiers.json`. diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-assemble.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-assemble.md index e73bb160..5ecae8a8 100644 --- a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-assemble.md +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-assemble.md @@ -3,6 +3,7 @@ _assemble: assemble-inventory _of_phase: discover _reads: - terraform (fragment contribution) + - live (fragment contribution) - billing (fragment contribution) _produces: - heroku-resource-inventory.json @@ -11,7 +12,7 @@ _produces: # Discover — Assemble Inventory > **Assembler unit.** Runs after the discover fragments (`discover-terraform.md`, -> `discover-billing.md`) have produced their contributions. It combines them into +> `discover-live.md`, `discover-billing.md`) have produced their contributions. It combines them into > the single `heroku-resource-inventory.json` artifact and owns that artifact's > final contract. See `discover.md` for how this unit is composed into the phase. @@ -31,7 +32,47 @@ After all sub-discoveries complete, assemble `heroku-resource-inventory.json` in - `space` (Private Space name or null) - `procfile_parse_warning`, `app_json_parse_warning` (per-app parse warnings or null) 6. Include `billing_profile` section (if billing data available, with `available`, `total_monthly_cost`, `currency`, `billing_period`, `line_items`). -7. Include `terraform_metadata` section (if Terraform discovery ran, with `found`, `tf_files_scanned`, `resource_types_extracted`, `parse_warnings`). +7. Include `terraform_metadata` section only when `.tf` files with `heroku_*` resources were actually FOUND (the terraform fragment always runs but may exit empty — an empty run contributes no section and no `"terraform"` discovery source). +8. Include `live_metadata` section (if the live fragment ran, with `found`, `captured_at`, `apps_captured`, `apps_failed`, `capture_warnings`, `limitations` — and `drift` per the Merge & Drift Rules below). + +## Merge & Drift Rules (when BOTH terraform and live fragments contributed) + +Resource identity is `resource_id` (both fragments use the same deterministic ID +formats). Merge into ONE entry per `resource_id`. Never resolve a disagreement +silently — every conflict is recorded as drift. + +1. **Same `resource_id` from both:** keep one entry. Live values win field-by-field + in `config` (live reflects current account state; Terraform may be stale). Keep + Terraform provenance fields (`tf_file`, `tf_resource_name`). Set + `source: "live+terraform"`. If any config field disagreed, record it in + `live_metadata.drift.config_conflicts[]` as + `{ "resource_id", "field", "terraform_value", "live_value" }`. + 1a. **Add-on plan changes are conflicts, not add/remove pairs.** Add-on + `resource_id`s embed the plan, so before applying rules 2–3, pair any live-only + and terraform-only addon entries that share the same `heroku_app` + + `addon_service`. Treat such a pair as ONE resource with a `plan` config conflict + (rule 1): keep the live entry, record + `{ "resource_id": , "field": "plan", "terraform_value", "live_value" }` + in `config_conflicts[]`, and do not count the pair in `resources_live_only` / + `resources_terraform_only`. +2. **Live only:** keep the entry (`source: "live"`) and set + `unmanaged_by_terraform: true` — this is click-ops drift the migration plan must + include. +3. **Terraform only:** keep the entry (`source: "terraform"`) and set + `not_found_live: true` — defined but not deployed (or not in the selected app + set). Detect-only: downstream phases decide how to treat it. +4. **Formation gap-fill:** process types that live discovery cannot see (scaled to + zero, i.e. Terraform `quantity: 0`) but Terraform/Procfile declares are kept from + the Terraform contribution — this is the expected complement, not a conflict. Do + NOT set `not_found_live` on them and do NOT count them in + `resources_terraform_only`. +5. **Apps section:** live `apps[]` entries win (they carry the real `app_id` UUID); + merge in Terraform-only apps with `not_found_live: true`. +6. **Drift summary:** set `live_metadata.drift` to + `{ "resources_live_only": N, "resources_terraform_only": M, "config_conflicts": [...] }`. + +When only ONE of the two fragments contributed, no merge occurs and no `drift` key +is written. **If assembly fails** (no valid resources from any source after sub-discoveries ran): this is an unrecoverable error (`INTERPRETER.md` § `_on_error` — `_unrecoverable`). diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live-capture.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live-capture.md new file mode 100644 index 00000000..e078f621 --- /dev/null +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live-capture.md @@ -0,0 +1,162 @@ +# Discover Phase: Live Capture (Main Window — Interactive Pre-Work) + +> Interactive CLI capture step for live discovery. **This file is NOT a fragment.** +> It runs in the MAIN window because it converses with the user and uses the shell — +> two things the dispatched `rw` worker cannot do. `discover.md` § Orientation says +> when to load it: after `_init`, before the phase's work is dispatched. +> +> Its only output is raw CLI captures under `$MIGRATION_DIR/live-capture/` plus a +> `manifest.json` index. The `live` FRAGMENT (`discover-live.md`) parses those files +> into inventory entries inside the worker. This file never writes inventory entries. + +**Execute ALL steps in order. Do not skip or optimize.** + +--- + +## Security Contract (applies to every step) + +1. **Exact-command whitelist.** Run ONLY commands that appear as rows in the Capture + Command Table (Step 3). Read and write commands share namespaces in the Heroku CLI + (`addons` lists; `addons:upgrade` mutates; `addons:upgrade` and `addons:downgrade` + are the same command). No namespace-prefix reasoning: if a command is not a table + row, do not run it. +2. **Never capture secrets.** + - NEVER run `heroku auth:token` — it prints the API token to stdout. + - Config vars: capture KEY NAMES ONLY, using the key filter in Step 3 row 5. NEVER + write raw `heroku config` output (values are secrets) to disk, to chat, or into + any artifact. If neither `jq` nor `python3` is available for filtering, SKIP + config capture entirely and record a `skipped` entry in the manifest. +3. **Always explicit flags.** Some commands change scope based on the working + directory (`heroku addons` implies `--app` inside an app's git repo). Always pass + `--all` or `-a ` explicitly. Never use the global `--prompt` flag. +4. **No mutations, no logins.** Never run `heroku login` (browser-interactive — hand + off to the user per Step 2). Never run any create / set / add / attach / scale / + upgrade / destroy / rename / remove command. +5. **Capture to files, not chat.** Redirect stdout to files under + `$MIGRATION_DIR/live-capture/`. Do not paste large outputs into the conversation. + (`.migration/` is gitignored by `_init`, so captures cannot be committed.) + +--- + +## Step 1: Consent Gate + +Output exactly, then wait for the user's choice: + +``` +─── Live Heroku Discovery (read-only) ─── + +I can inventory your Heroku account directly using your authenticated +Heroku CLI. This runs LIST/INFO commands only: + + ✓ Captured: app names, regions, stacks, dyno types and counts, + add-on plans and prices, domain names, pipeline stages, Private + Space peering info, and config var KEY NAMES. + ✗ Never captured: config var values, credentials, API tokens, + source code, or database contents. No command that creates, + changes, or deletes anything will run. + +Output is written to .migration//live-capture/ (gitignored). + +[A] Proceed with live discovery +[B] Skip — use workspace files only +``` + +- **[A]** → continue to Step 2. +- **[B]** → do not run any Heroku command. Return to `discover.md` and record that + live capture was declined. If no `heroku_*` Terraform exists either, the phase's + `_preconditions` will fail normally. + +## Step 2: Preflight + +1. **CLI installed:** run `heroku --version`. + - Missing → tell the user: "The Heroku CLI isn't installed. Install it + (https://devcenter.heroku.com/articles/heroku-cli) and tell me to continue, or + choose to skip live discovery." Wait. If skipped → exit as in Step 1 [B]. +2. **Authenticated:** run `heroku auth:whoami`. + - Success → record the account email for the manifest. + - Failure (not logged in, or token expired) → tell the user: "Your Heroku CLI + isn't authenticated (or the session expired). Run `heroku login` in your + terminal — it needs a browser, so I can't run it for you — then tell me to + continue." Wait. If the user declines → exit as in Step 1 [B]. + +## Step 3: Capture + +Create `$MIGRATION_DIR/live-capture/`. Then: + +**3a. App list and selection guard.** Run row 1 first. If it returns more than 25 +apps, list the app names and ask the user which apps to include (`all` is a valid +answer). Record the selected set as `apps_selected`. All per-app rows below run only +for selected apps. + +**3b. Capture Command Table.** Run each applicable row, redirecting stdout to the +named file. ``, ``, `` iterate over the selected apps and the +account's pipelines/spaces from rows 9 and 11. + +| # | Command | Output file | Scope | +| -- | -------------------------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | `heroku apps --all --json` | `apps.json` | account | +| 2 | `heroku apps:info -a --json` | `app-.json` | per app | +| 3 | `heroku ps -a --json` | `ps-.json` | per app | +| 4 | `heroku addons --all --json` | `addons.json` | account | +| 5 | `heroku config -a --json \| jq 'keys'` | `config-keys-.json` | per app — KEYS ONLY (see fallback below) | +| 6 | `heroku domains -a --json` | `domains-.json` | per app | +| 7 | `heroku pg:info -a ` | `pg-.out` | apps with a heroku-postgresql add-on (from row 4) | +| 8 | `heroku redis:info -a ` | `redis-.out` | apps with a heroku-redis add-on (from row 4) | +| 8b | `heroku kafka:info -a ` | `kafka-.out` | apps with a heroku-kafka add-on; needs the kafka CLI plugin — if the command is unavailable, record `skipped` (plan data from row 4 is sufficient for sizing) | +| 9 | `heroku pipelines --json` | `pipelines.json` | account | +| 10 | `heroku pipelines:info --json` | `pipeline-.json` | per pipeline from row 9 | +| 11 | `heroku spaces --json` | `spaces.json` | account (empty result is normal — most startups have no spaces) | +| 12 | `heroku spaces:info -s --json` | `space-.json` | per space from row 11 | +| 13 | `heroku spaces:peerings -s --json` | `space-peerings-.json` | per space from row 11 | + +Text captures use the `.out` extension (not `.txt`) so they can never collide with +the phase's `_forbids_files: "*.txt"` scope boundary, regardless of how a host +scopes that glob. + +**Row 5 fallback:** if `jq` is unavailable, use +`heroku config -a --json | python3 -c "import json,sys; print(json.dumps(sorted(json.load(sys.stdin))))"`. +If neither filter runtime exists, skip row 5 for all apps and record `skipped` in the +manifest — never capture unfiltered config output. + +**3c. Per-command error handling:** + +| Error | Behavior | +| ---------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| 403 / forbidden on a team app | Record the capture as `failed` with the reason; continue with remaining apps | +| Command not found (missing CLI plugin, row 8b) | Record `skipped`; continue | +| Timeout / transient network error | Retry once; on second failure record `failed`; continue | +| 401 / token expired mid-run | STOP capturing. Hand off as in Step 2.2. On resume, re-run Step 3 from the top (captures overwrite) | +| Rate limited (429) | Unexpected at this call volume — wait 60s, retry once, then record `failed` | + +## Step 4: Write the Manifest + +Write `$MIGRATION_DIR/live-capture/manifest.json`: + +```json +{ + "captured_at": "", + "cli_version": "", + "account": "", + "apps_selected": ["my-web-app"], + "captures": [ + { "command": "heroku apps --all --json", "file": "apps.json", "status": "ok", "note": null }, + { + "command": "heroku config -a my-web-app --json | jq 'keys'", + "file": "config-keys-my-web-app.json", + "status": "ok", + "note": null + } + ] +} +``` + +`status` ∈ `ok | failed | skipped`. Every attempted or deliberately skipped row gets +an entry. The manifest is the fragment's index — its existence is also the `live` +fragment's `_trigger`. + +## Step 5: Return to `discover.md` + +Tell the user in one line how many apps were captured and whether any captures +failed or were skipped, then continue the phase per `discover.md` (the dispatched +worker will parse `live-capture/` via the `live` fragment). Do NOT parse captures +here, do NOT write inventory entries, and do NOT update `.phase-status.json`. diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live.md new file mode 100644 index 00000000..49f181a1 --- /dev/null +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover-live.md @@ -0,0 +1,176 @@ +--- +_fragment: live +_of_phase: discover +_contributes: + - heroku-resource-inventory.json (resource entries, apps, metadata, live_metadata sections) +--- + +# Discover Phase: Live Discovery (Parse Captured CLI Output) + +> Self-contained live-discovery fragment. Reads the raw CLI captures that +> `discover-live-capture.md` wrote to `$MIGRATION_DIR/live-capture/` and maps them +> to inventory entries. **Parse-only**: this fragment runs inside the dispatched +> worker — it has no shell and MUST NOT run any `heroku` command, prompt the user, +> or re-capture anything. If `live-capture/manifest.json` does not exist, exit +> cleanly with no output. + +**Execute ALL steps in order. Do not skip or optimize.** + +--- + +## Step 0: Read the Manifest + +Read `$MIGRATION_DIR/live-capture/manifest.json`. It indexes every capture file with +a `status` of `ok`, `failed`, or `skipped`. Process only `ok` captures. Carry every +`failed`/`skipped` entry forward into `live_metadata.capture_warnings`. + +## Step 1: Map Apps — `apps.json` + `app-.json` + +For each app in `apps.json` that is in the manifest's `apps_selected`: + +| Capture field | Inventory field (`apps[]` entry) | +| ------------- | ------------------------------------------------------------------------------------------------------------------------------- | +| `name` | `app_name` | +| `id` | `app_id` (real UUID — live discovery always has this) | +| `space.name` | `space` (or `null`) | +| `stack.name` | feeds `heroku_generation`: `heroku-20`/`heroku-22`/`heroku-24` → `"cedar"`; contains `fir` or `cnb` → `"fir"`; else `"unknown"` | + +Set `generation_action: "detect_only"`, `discovery_status: "success"`. An app whose +per-app captures all failed (manifest `failed`) gets `discovery_status: +"discovery_failed"` with `failure_reason` from the manifest note — it still gets an +`apps[]` entry and still counts in `metadata.total_apps_discovered` (it was +discovered; its details were not). + +## Step 2: Map Formations — `ps-.json` + +Group the dyno list by `type`: + +| Derivation | Inventory field (`formation` config) | +| ------------------------------------------------------- | -------------------------------------------------- | +| dyno `type` | `process_type` | +| count of dynos of that type | `quantity` | +| dyno `size`, lowercased (`Standard-1X` → `standard-1x`) | `dyno_type` — normalize to match the sizing tables | +| dyno `command` | `command` | + +Resource entry: `resource_id: "formation:{app_name}:{process_type}"`, +`resource_type: "formation"`, `source: "live"`. + +**Known limitation:** `heroku ps` shows running dynos only — a process type scaled +to zero is invisible to live discovery. Record the limitation string +`"formations scaled to zero are not visible to live discovery"` once in +`live_metadata.limitations`. (When Terraform or a Procfile also ran, the assembler +recovers those process types from that source.) + +## Step 3: Map Add-ons — `addons.json` (+ `pg/redis/kafka` info captures) + +For each add-on attached to a selected app: + +| Capture field | Inventory field (`addon` config) | +| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `plan.name` (e.g. `heroku-postgresql:standard-0`) | split into `addon_service` and `plan` (same rule as Terraform discovery) | +| `addon_service.name` | `provider` — `"heroku"` when the service name starts with `heroku-` (first-party data add-ons), otherwise the service name itself (matches the Terraform fragment's convention) | +| `plan.price` (`cents`, `unit`) | `monthly_price_usd` = `cents / 100` when `unit == "month"` (optional field) | + +Resource entry: `resource_id: "addon:{app_name}:{addon_service}:{plan}"`, +`resource_type: "addon"`, `source: "live"`. + +**Enrich from info captures** (parse the `Key: Value` text lines; on unparseable +text, keep the plan-only entry and record a warning): + +- `pg-.out` → `pg_version`, `data_size_gb` (convert e.g. `10.5 GB`; MB → GB), + `table_count`, `connection_pooling` (`true` if a Connection Pooling line is + present, `false` if absent — never omit the field). The `data_size_gb` value + feeds downstream database-migration tool selection. +- `redis-.out` → `redis_version`, `maxmemory_policy`; set `ha_enabled: true` + when the plan tier starts with `premium` or `private`, and + `encryption_in_transit: true` for those tiers. +- `kafka-.out` → `topic_count`, `partitions_per_topic`, `replication_factor` + when present. + +## Step 4: Map Domains, Config, Pipelines, Spaces + +**Domains** (`domains-.json`): one entry per custom domain — +`resource_id: "domain:{app_name}:{hostname}"`, `resource_type: "domain"`, config +`{ "hostname": ..., "sni_endpoint": }`. +Skip default `*.herokuapp.com` hostnames — record their count as +`live_metadata.default_heroku_domains_skipped` (exact field name), not as resources. + +**Config** (`config-keys-.json`): `resource_id: "config:{app_name}"`, +`resource_type: "config"`, config `{ "config_var_keys": [...] }`. Keys only — if a +capture file unexpectedly contains values (objects, not a string array), DISCARD it, +do not copy any part into the inventory, and record a warning. + +**Pipelines** (`pipelines.json` + `pipeline-.json`): +`resource_id: "pipeline:{pipeline_name}"`, `resource_type: "pipeline"`, config +`{ "pipeline_name": ..., "stages": [{ "stage": ..., "app": ... }], "review_apps_enabled": false, "detection_status": "detect-only" }`. +Populate `stages` from the per-pipeline capture; if `review_apps_enabled` is not +derivable, set it `false` and append a note to `live_metadata.capture_warnings`. + +**Spaces** (`spaces.json` + `space-.json` + `space-peerings-.json`): +`resource_id: "space:{space_name}"`, `resource_type: "space"`, config per the schema — +`space_name`, `region`, `shield`, and `peering` filled from the peerings capture +(`detected: true` with `vpc_id`/`peer_cidr` when an active peering exists; this is +data Terraform discovery usually cannot see). + +All entries: `heroku_app` = owning app name, or `"unassociated"` for spaces and +pipelines. `source: "live"` on every entry this fragment contributes. + +## Step 5: Output Contribution for the Assembler + +The assembler (`discover-assemble.md`) owns the inventory's structure and the merge +with Terraform-sourced entries. This fragment contributes: + +- **Resources:** all entries from Steps 2–4, each with `source: "live"`. +- **Apps:** the `apps[]` entries from Step 1. +- **Discovery sources:** contribute `"live"` to `metadata.discovery_sources`. +- **Confidence:** `"full"` when every capture for the selected apps has manifest + status `ok`; `"reduced"` otherwise (with `confidence_note` naming what failed or + was skipped). +- **`live_metadata`:** + +```json +{ + "live_metadata": { + "found": true, + "captured_at": "2026-07-15T18:20:00Z", + "apps_captured": 3, + "apps_failed": 0, + "capture_warnings": [], + "limitations": ["formations scaled to zero are not visible to live discovery"] + } +} +``` + +--- + +## Error Handling + +| Error Category | Behavior | +| --------------------------------------------- | ----------------------------------------------------------------------- | +| `manifest.json` missing | Exit cleanly with no output (capture never ran) | +| Capture file named in manifest is missing | Record warning, skip that capture, continue | +| Malformed JSON in a capture file | Record warning, skip that file, continue | +| Unparseable `pg/redis/kafka` info text | Keep the plan-only addon entry, record warning, continue | +| Config capture contains values (not key list) | Discard the file entirely, record warning, continue — never copy values | + +**Key principle:** partial results are better than no results. Any single capture +failure degrades confidence; it never halts the fragment. + +--- + +## Scope Boundary + +**This fragment covers parsing of `live-capture/` files ONLY.** + +FORBIDDEN — Do NOT include ANY of: + +- Running `heroku` (or any shell) commands — capture already happened in the main window +- Prompting the user +- AWS service names, recommendations, or equivalents +- Migration strategies, timelines, cost estimates, or effort estimates +- Merging or de-duplicating against Terraform-sourced entries — the assembler owns the merge + +**Your ONLY job: turn raw CLI captures into inventory entries. Nothing else.** + +After producing entries, the assembler handles merging into the final inventory; do +NOT update `.phase-status.json` from this fragment. diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover.md index 348b16dd..37ec7c23 100644 --- a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover.md +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/phases/discover/discover.md @@ -7,6 +7,9 @@ _fragments: - _id: terraform _trigger: { _always: true } _file: phases/discover/discover-terraform.md + - _id: live + _trigger: { _when: "$MIGRATION_DIR/live-capture/manifest.json exists (the live-capture pre-work ran — see Orientation § Live capture)" } + _file: phases/discover/discover-live.md - _id: billing _trigger: { _glob: "**/*{billing,invoice}*.{csv,json}" } _file: phases/discover/discover-billing.md @@ -26,7 +29,7 @@ _re_entry_guard: _preconditions: - _check_single_active_phase: true _on_failure: _halt_and_inform - - _assert: "at least one .tf file containing a heroku_* resource exists in the workspace" + - _assert: "at least one Heroku source is available: a .tf file containing a heroku_* resource exists in the workspace, OR $MIGRATION_DIR/live-capture/manifest.json exists. Evaluating this check is where live capture is offered: if no heroku_* Terraform is found, load references/phases/discover/discover-live-capture.md in the MAIN window and run it (consent-gated); fail this check only after the user declines live capture or capture cannot run" _on_failure: _unrecoverable _postconditions: - _check_file_exists: heroku-resource-inventory.json @@ -39,7 +42,11 @@ _postconditions: _on_failure: _halt_and_inform - _assert: "no forbidden clustering fields are present (cluster_id, creation_order_depth, edges, dependencies, must_migrate_together)" _on_failure: _halt_and_inform - - _assert: "metadata.discovery_sources reflects which sub-discoveries actually ran; if the terraform sub-discovery ran, resources[] contains at least one Terraform-sourced resource" + - _assert: "metadata.discovery_sources reflects which sub-discoveries actually produced data; if heroku_* Terraform files were FOUND in the workspace (not merely that the terraform fragment ran — it always runs and may exit empty), resources[] contains at least one Terraform-sourced resource" + _on_failure: _halt_and_inform + - _assert: "if the live fragment ran ($MIGRATION_DIR/live-capture/manifest.json exists), resources[] contains at least one live-sourced resource, a live_metadata section is present, and 'live' appears in metadata.discovery_sources" + _on_failure: _halt_and_inform + - _assert: "no config var VALUES appear anywhere in the inventory — config entries carry key names only" _on_failure: _halt_and_inform - _assert: "if a billing/invoice file was present in the workspace, heroku-resource-inventory.json has a billing_profile section" _on_failure: _halt_and_inform @@ -64,11 +71,43 @@ owns only lifecycle + the cross-cutting `_postconditions`. Two facts the contract can't express: Procfile/app.json parsing is integrated into the terraform fragment (there is no standalone Procfile fragment) — when present alongside Terraform, they supplement resource data with commands, buildpacks, and -declared add-ons. And Platform API discovery is NOT supported in v1: no API calls -are made, discovery is entirely file-based. Billing data, when present, is embedded -in `heroku-resource-inventory.json` (not a separate file); all user communication +declared add-ons. Billing data, when present, is embedded in +`heroku-resource-inventory.json` (not a separate file); all user communication is via output messages only (no report/log files). +### Live capture (main-window pre-work) + +Live discovery reads the user's Heroku account through their authenticated Heroku +CLI — read-only, consent-gated, key-names-only for config vars. It is split in two +because the dispatched `rw` worker has no shell and cannot converse with the user: + +1. **Capture** (`discover-live-capture.md`) — runs in the MAIN window, after + `_init` and before the phase's work is dispatched. It asks for consent, preflights + the CLI (`heroku auth:whoami`), runs an exact-command whitelist of list/info + commands, and writes raw output to `$MIGRATION_DIR/live-capture/` plus a + `manifest.json` index. It writes NO inventory entries. +2. **Parse** (`discover-live.md`, the `live` fragment) — runs in the worker with the + other fragments. Its `_trigger` is the manifest's existence; it maps captures to + inventory entries with `source: "live"`. + +**Explicit ordering (cold start):** run `_init` state setup FIRST (create +`$MIGRATION_DIR`, write `.phase-status.json`), THEN evaluate the source +`_precondition` — offering and running capture as part of that evaluation — then +dispatch the phase's work. Capture writes into `$MIGRATION_DIR/live-capture/`, so +it cannot run before `_init` has created the run directory. + +**When to offer capture:** while evaluating the source `_precondition`, scan the +workspace first (free). If NO `heroku_*` Terraform is found, offer live capture as +the primary source — load `discover-live-capture.md` — instead of failing the check. +If Terraform IS found, still offer capture once as an optional live cross-check +("catch resources managed outside Terraform"); a decline is fine and is not +re-asked. Never run capture without explicit consent. + +**Source-of-truth rule (for the assembler):** when both Terraform and live entries +exist, live is authoritative for current state (config values, plans, quantities); +Terraform supplements structure and provenance. Disagreements are surfaced as drift, +never silently resolved — see `discover-assemble.md` § Merge & Drift Rules. + --- ## Handoff @@ -78,6 +117,8 @@ completion message from the inventory contents: - "Discovered X total resources across Y apps." - If billing data available: "Parsed billing data ($Z/month)." +- If live discovery ran: "Live discovery captured N apps via the Heroku CLI." +- If both live and Terraform ran: "Drift check: N resources live but not in Terraform, M in Terraform but not live, K config conflicts (live values used)." - If Terraform secondary: "Supplemented with Terraform-sourced resources (N conflicts resolved)." - If Pipeline detected: "Detected N pipeline(s) (detect-only)." - If Cedar/Fir mixed: "Generation detection: N Cedar, M Fir, P unknown." @@ -90,12 +131,14 @@ Format: "Discover phase complete. [artifact summaries] Next required step: Phase Non-fatal discovery errors and their handling (fatal source/gate failures are handled by `_preconditions`/`_postconditions` + `INTERPRETER.md` § `_on_error`): -| Error Category | Behavior | -| ------------------------------------------------- | ---------------------------------------------- | -| Terraform parse error (malformed HCL) | Log warning, skip malformed blocks, continue | -| Procfile/app.json parse error | Record warning per-app, continue | -| Generation detection unresolvable (no stack attr) | Set `heroku_generation` to `unknown`, continue | -| Pipeline detection from Terraform incomplete | Record with available data, continue | +| Error Category | Behavior | +| -------------------------------------------------- | ---------------------------------------------------------------------------------------------- | +| Terraform parse error (malformed HCL) | Log warning, skip malformed blocks, continue | +| Procfile/app.json parse error | Record warning per-app, continue | +| Generation detection unresolvable (no stack attr) | Set `heroku_generation` to `unknown`, continue | +| Pipeline detection from Terraform incomplete | Record with available data, continue | +| Live capture partially failed (some apps 403 etc.) | Parse the `ok` captures, mark failed apps `discovery_failed`, confidence `reduced`, continue | +| Live capture declined or CLI unavailable | Skip the `live` fragment (no manifest → trigger never fires), continue with file-based sources | --- diff --git a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/shared/schema-discover-heroku.md b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/shared/schema-discover-heroku.md index ff997640..0362b880 100644 --- a/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/shared/schema-discover-heroku.md +++ b/migrate/plugins/migration-to-aws/skills/heroku-to-aws/references/shared/schema-discover-heroku.md @@ -42,7 +42,8 @@ Complete inventory of discovered Heroku resources. Uses a **flat resource model* } ], "billing_profile": {}, - "terraform_metadata": {} + "terraform_metadata": {}, + "live_metadata": {} } ``` @@ -54,13 +55,13 @@ Complete inventory of discovered Heroku resources. Uses a **flat resource model* Report-level information about the discovery run. -| Field | Type | Required | Description | -| ----------------------- | ----------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `discovery_timestamp` | string (ISO 8601) | ✅ | When discovery was executed | -| `total_apps_discovered` | integer | ✅ | Count of Heroku apps found | -| `discovery_sources` | string[] | ✅ | Sources used: `"terraform"`, `"procfile"`, `"billing"` | -| `confidence` | string | ✅ | `"full"` (Terraform files present and parsed successfully) or `"reduced"` (Partial data, e.g., Terraform parse errors on some files, missing expected resources) | -| `confidence_note` | string | ❌ | Explanation when confidence is `"reduced"` | +| Field | Type | Required | Description | +| ----------------------- | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `discovery_timestamp` | string (ISO 8601) | ✅ | When discovery was executed | +| `total_apps_discovered` | integer | ✅ | Count of Heroku apps found | +| `discovery_sources` | string[] | ✅ | Sources used: `"terraform"`, `"procfile"`, `"billing"`, `"live"` | +| `confidence` | string | ✅ | `"full"` (primary source(s) parsed/captured successfully) or `"reduced"` (partial data, e.g., Terraform parse errors, failed/skipped live captures, missing expected resources) | +| `confidence_note` | string | ❌ | Explanation when confidence is `"reduced"` | ### `apps[]` (REQUIRED) @@ -83,12 +84,15 @@ Per-app metadata entries. One entry per discovered Heroku app. Flat array of all discovered resources. **No nesting, no clustering.** -| Field | Type | Required | Description | -| --------------- | ------ | -------- | --------------------------------------------------------- | -| `resource_id` | string | ✅ | Unique identifier (format below) | -| `resource_type` | string | ✅ | One of: `"formation"`, `"addon"`, `"space"`, `"pipeline"` | -| `heroku_app` | string | ✅ | App name this resource belongs to, or `"unassociated"` | -| `config` | object | ✅ | Type-specific configuration (see per-type schemas below) | +| Field | Type | Required | Description | +| ------------------------ | ------- | -------- | ------------------------------------------------------------------------------------------- | +| `resource_id` | string | ✅ | Unique identifier (format below) | +| `resource_type` | string | ✅ | One of: `"formation"`, `"addon"`, `"space"`, `"pipeline"`, `"domain"`, `"config"` | +| `heroku_app` | string | ✅ | App name this resource belongs to, or `"unassociated"` | +| `config` | object | ✅ | Type-specific configuration (see per-type schemas below) | +| `source` | string | ❌ | Discovery provenance: `"terraform"`, `"live"`, or `"live+terraform"` (merged) | +| `unmanaged_by_terraform` | boolean | ❌ | Set `true` when live discovery found the resource but Terraform does not manage it (drift) | +| `not_found_live` | boolean | ❌ | Set `true` when Terraform declares the resource but live discovery did not find it deployed | ### `billing_profile` (OPTIONAL — present when billing data available) @@ -117,6 +121,19 @@ Flat array of all discovered resources. **No nesting, no clustering.** | `resource_types_extracted` | string[] | ✅ | List of extracted resource types (e.g., `"heroku_app"`, `"heroku_addon"`) | | `parse_warnings` | string[] | ✅ | Any parse warnings encountered during extraction | +### `live_metadata` (OPTIONAL — present when live CLI discovery ran) + +| Field | Type | Required | Description | +| -------------------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `found` | boolean | ✅ | Whether live capture produced usable data | +| `captured_at` | string | ✅ | ISO 8601 timestamp of the CLI capture run | +| `apps_captured` | integer | ✅ | Number of apps successfully captured | +| `apps_failed` | integer | ✅ | Number of apps whose captures failed (e.g., 403 on team apps) | +| `capture_warnings` | string[] | ✅ | Failed/skipped capture notes carried from `live-capture/manifest.json` | +| `limitations` | string[] | ✅ | Known live-discovery blind spots (e.g., formations scaled to zero) | +| `default_heroku_domains_skipped` | integer | ❌ | Count of default `*.herokuapp.com` hostnames skipped (not recorded as domain resources) | +| `drift` | object | ❌ | Present only when Terraform AND live both ran: `resources_live_only` (int), `resources_terraform_only` (int), `config_conflicts[]` (`{resource_id, field, terraform_value, live_value}`) | + --- ## Resource ID Formats @@ -129,6 +146,8 @@ Deterministic ID format per resource type: | `addon` | `addon:{app_name}:{addon_service}:{plan}` | `addon:my-web-app:heroku-postgresql:standard-0` | | `space` | `space:{space_name}` | `space:my-private-space` | | `pipeline` | `pipeline:{pipeline_name}` | `pipeline:my-pipeline` | +| `domain` | `domain:{app_name}:{hostname}` | `domain:my-web-app:www.example.com` | +| `config` | `config:{app_name}` | `config:my-web-app` | --- @@ -176,6 +195,12 @@ Deterministic ID format per resource type: - **heroku-kafka**: `topic_count` (integer), `partitions_per_topic` (integer), `replication_factor` (integer) - **Other add-ons**: No additional required fields +**Optional live-enrichment fields** (present only when live discovery ran): + +- Any addon: `monthly_price_usd` (number — from the add-on's plan price) +- **heroku-postgresql**: `pg_version` (string), `data_size_gb` (number — feeds database migration tool selection), `table_count` (integer) +- **heroku-redis**: `maxmemory_policy` (string) + ### `space` config ```json @@ -251,10 +276,10 @@ The following fields MUST NOT appear anywhere in `heroku-resource-inventory.json ## Confidence Levels -| Level | Meaning | When Used | -| --------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -| `full` | Terraform files present and parsed successfully | Terraform discovery completed without parse errors | -| `reduced` | Partial data — Terraform had parse errors or missing expected resources | Some `.tf` files could not be parsed, or expected resource types were not found | +| Level | Meaning | When Used | +| --------- | -------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| `full` | Every source that ran produced complete data | Terraform parsed without errors and/or every live capture for the selected apps succeeded | +| `reduced` | Partial data from at least one source | Terraform parse errors, failed/skipped live captures (e.g., 403 on team apps, missing CLI plugin), or missing expected resources | --- @@ -419,3 +444,5 @@ The following fields MUST NOT appear anywhere in `heroku-resource-inventory.json 9. ✅ If Terraform discovery ran → resources include Terraform-sourced entries 10. ✅ If Terraform had parse errors → `metadata.confidence` is `"reduced"` 11. ✅ If billing discovery ran → `billing_profile` section present with `available: true` +12. ✅ If live discovery ran → resources include live-sourced entries, `live_metadata` present, and `"live"` in `metadata.discovery_sources` +13. ✅ No config var VALUES anywhere in the document — `config` entries carry key names only From f5da985749998553a0cd82cd656d2c0d451b84bc Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 13:51:27 -0700 Subject: [PATCH 02/11] =?UTF-8?q?feat(gcp-to-aws):=20live=20discovery=20vi?= =?UTF-8?q?a=20the=20authenticated=20gcloud=20CLI=20Discovery=20previously?= =?UTF-8?q?=20required=20Terraform,=20app=20code,=20or=20billing=20exports?= =?UTF-8?q?.=20Adds=20a=20consent-gated,=20read-only=20live=20discovery=20?= =?UTF-8?q?sub-file:=20gcloud=20preflight,=20Cloud=20Asset=20Inventory=20f?= =?UTF-8?q?ast=20path=20with=20a=2016-row=20per-service=20fallback,=20--fo?= =?UTF-8?q?rmat=3Djson=20projections=20that=20capture=20env=20var=20NAMES?= =?UTF-8?q?=20only,=20Terraform-style=20identity=20synthesis=20so=20downst?= =?UTF-8?q?ream=20design-refs=20work=20unchanged,=20deterministic=20edge?= =?UTF-8?q?=20inference=20from=20resolved=20config,=20and=20Step=203S=20si?= =?UTF-8?q?mplified=20clustering.=20When=20Terraform=20is=20also=20present?= =?UTF-8?q?,=20the=20IaC=20inventory=20is=20the=20base=20and=20disagreemen?= =?UTF-8?q?ts=20surface=20as=20drift=20(unmanaged=20resources,=20console?= =?UTF-8?q?=20resizes,=20config=20conflicts)=20=E2=80=94=20with=20the=20ru?= =?UTF-8?q?le=20that=20absence=20of=20evidence=20from=20a=20failed=20captu?= =?UTF-8?q?re=20is=20not=20drift.=20Includes=20replay=20fixtures=20with=20?= =?UTF-8?q?a=20drift-assertion=20set,=20validated=20by=20end-to-end=20fres?= =?UTF-8?q?h-agent=20runs.=20Stacked=20on=20the=20Heroku=20live-discovery?= =?UTF-8?q?=20branch=20(#148).?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .checkov.yaml | 5 + migrate/README.md | 8 +- migrate/plugins/migration-to-aws/README.md | 19 +- .../fixtures/gcp-live-capture/README.md | 56 +++ .../gcp-live-capture/expected-drift.json | 79 +++++ .../live-capture/buckets.json | 10 + .../live-capture/functions.json | 1 + .../gcp-live-capture/live-capture/gce.json | 1 + .../gcp-live-capture/live-capture/gke.json | 1 + .../live-capture/manifest.json | 111 ++++++ .../live-capture/networks.json | 7 + .../live-capture/redis-us-central1.json | 10 + .../gcp-live-capture/live-capture/run.json | 68 ++++ .../gcp-live-capture/live-capture/sa.json | 3 + .../live-capture/secrets.json | 12 + .../gcp-live-capture/live-capture/sql.json | 17 + .../live-capture/subnets.json | 8 + .../workspace-terraform/main.tf | 69 ++++ .../skills/gcp-to-aws/SKILL.md | 25 +- .../phases/discover/discover-live.md | 328 ++++++++++++++++++ .../references/phases/discover/discover.md | 26 +- .../references/shared/schema-discover-iac.md | 44 +++ 22 files changed, 889 insertions(+), 19 deletions(-) create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf create mode 100644 migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md diff --git a/.checkov.yaml b/.checkov.yaml index d67df79d..61120cd0 100644 --- a/.checkov.yaml +++ b/.checkov.yaml @@ -11,3 +11,8 @@ # their purpose. They are never deployed. skip-path: - migrate/plugins/migration-to-aws/skills/tf-best-practices/fixtures/terraform-policy + # Live-discovery replay fixture: deliberately models a startup's EXISTING, + # imperfect GCP infrastructure (the migration SOURCE, never deployed). Its + # drift vs the canned gcloud captures is the test — sanitizing it to satisfy + # Checkov would change the source-provider state the fixture exists to model. + - migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform diff --git a/migrate/README.md b/migrate/README.md index 4fa73558..0539641d 100644 --- a/migrate/README.md +++ b/migrate/README.md @@ -74,6 +74,7 @@ After installation, just describe what you want to migrate: **GCP migrations:** - "Migrate my GCP infrastructure to AWS" +- "Discover my GCP project and estimate AWS costs" - "Move my Cloud Run services to Fargate" - "Migrate my OpenAI app to Amazon Bedrock" - "Estimate AWS costs for my GCP workload" @@ -91,6 +92,8 @@ After installation, just describe what you want to migrate: The skill creates a `.migration//` directory in the current working directory with all artifacts. +**Live GCP discovery — how it works:** No Terraform or exports needed. If `gcloud auth login` works in your terminal, just ask — the agent confirms the target project and requests your consent, then inventories it using read-only list/describe commands. It captures resource names, types, regions, sizing, network topology, and env var **names only** — never env var values, secret values, database contents, or access tokens, and never a command that creates, changes, or deletes anything. If you also have Terraform, the agent cross-checks it against your live project and reports drift. (AI/agentic detection still needs your application code.) + **Live Heroku discovery — how it works:** No Terraform or exports needed. If `heroku login` works in your terminal, just ask — the agent requests your consent, then inventories your account using read-only list/info CLI commands. It captures app names, dyno types, add-on plans and prices, domains, pipelines, and config var **key names only**. It never reads config var values, credentials, or your API token, and never runs a command that creates, changes, or deletes anything. If you also have `heroku_*` Terraform, the agent cross-checks it against your live account and reports drift. ## What It Detects @@ -167,8 +170,9 @@ The skill creates a `.migration//` directory in the current working dir - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data -- **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) +- At least one input source: an authenticated `gcloud` or `heroku` CLI (live discovery), Terraform files, application code, or billing data +- **For GCP infrastructure migration:** an authenticated `gcloud` CLI (recommended — live, read-only discovery with your consent, with drift detection against any Terraform found) or Terraform files / billing exports +- **For GCP AI/agentic migration:** Application source code is required (billing/IaC/live discovery alone cannot detect agent architecture) - **For Heroku migration:** an authenticated Heroku CLI (recommended — live, read-only discovery with your consent) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone). When both are available, live data is authoritative for current state and Terraform drift is surfaced. ## Structure diff --git a/migrate/plugins/migration-to-aws/README.md b/migrate/plugins/migration-to-aws/README.md index 6263ab06..fdf7dfac 100644 --- a/migrate/plugins/migration-to-aws/README.md +++ b/migrate/plugins/migration-to-aws/README.md @@ -103,7 +103,7 @@ ln -s "$(pwd)" ~/.cursor/plugins/local/migration-to-aws ### Workflow -1. **Discover** — Scan Terraform files, application code, and/or billing data — or, for Heroku, inventory your account live via the authenticated Heroku CLI (read-only, consent-gated). Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. +1. **Discover** — Scan Terraform files, application code, and/or billing data — or inventory your GCP project or Heroku account live via the authenticated `gcloud`/`heroku` CLI (read-only, consent-gated, with drift detection against any Terraform found). Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. 2. **Clarify** — Ask targeted questions about migration preferences, AI priorities, agentic migration approach, database sizing, and timeline. 3. **Design** — Map source services to AWS equivalents. For AI workloads: select Bedrock models with honest pricing comparison. For agentic workloads: design AgentCore Harness config or Strands architecture. 4. **Estimate** — Calculate monthly AWS costs using real-time pricing data. Compare to current spend. @@ -198,10 +198,23 @@ The `--json` verdict lists each violation with `file`, `line`, `rule`, and `fix_ - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data -- **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) +- At least one input source: an authenticated `gcloud` or `heroku` CLI (live discovery), Terraform files, application code, or billing data +- **For GCP infrastructure migration:** an authenticated `gcloud` CLI (recommended — live, read-only discovery with your consent, with drift detection against any Terraform found) or Terraform files / billing exports +- **For GCP AI/agentic migration:** Application source code is required (billing/IaC/live discovery alone cannot detect agent architecture) - **For Heroku migration:** an authenticated Heroku CLI (recommended) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone) +### Live GCP discovery — how it works + +No Terraform or exports needed. If `gcloud auth login` works in your terminal, just +ask your agent to migrate ("Migrate my GCP infrastructure to AWS" or "Discover my +GCP project and estimate AWS costs"). The agent confirms the target project and asks +for your consent, then inventories it using read-only list/describe commands — it +captures resource names, types, regions, sizing, network topology, and env var +**names only**. It never reads env var values, secret values, database contents, or +access tokens, and never runs a command that creates, changes, or deletes anything. +If you also have Terraform, the agent cross-checks it against your live project and +reports drift. (AI/agentic workload detection still needs your application code.) + ### Live Heroku discovery — how it works No Terraform or exports needed. If `heroku login` works in your terminal, just ask diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md new file mode 100644 index 00000000..bc48351b --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -0,0 +1,56 @@ +# GCP live-discovery fixtures (replay mode) + +Canned `gcloud` CLI outputs for testing the `gcp-to-aws` live-discovery path +(`references/phases/discover/discover-live.md`) without a GCP project. The +synthetic project `acme-prod` uses **per-service fallback mode** (the manifest +records the Cloud Asset API fast path as `failed` — the common startup case) and +exercises the designed-for behaviors: + +- **orders-api** (Cloud Run) — carries the `run.googleapis.com/cloudsql-instances` + annotation and a service account, driving the Step 4 edge-inference rules; its + env list has NAMES ONLY (including secret-looking names like + `STRIPE_SECRET_KEY` — if a value ever appears in output, the projection rule + broke). +- **orders-db** (Cloud SQL) — live tier `db-custom-2-8192` vs Terraform's + `db-f1-micro`: the classic console-resize drift (Step 6 rule 1). +- **cache** (Memorystore Redis), **web-frontend** (Cloud Run), two secrets, and + the `acme-prod-uploads` bucket — all absent from Terraform → click-ops drift + (`unmanaged_by_terraform`, Step 6 rule 2). +- **acme-prod-assets** bucket — in Terraform, absent live, buckets capture `ok` + → `not_found_live` (Step 6 rule 3). +- **events** (Pub/Sub topic) — in Terraform, and the pubsub capture FAILED → + must NOT be marked `not_found_live` (rule 3's "absence of evidence is not + drift" negative case). +- Empty `gke.json` / `functions.json` / `gce.json` — services with nothing + deployed produce no entries and no errors. + +## How to replay + +**Scenario A — live-only (no Terraform):** + +1. Create a scratch directory with NO `.tf` files, app code, or billing exports. +2. Create `.migration/0720-1820/` and copy `live-capture/` into it. +3. Invoke the gcp-to-aws skill ("migrate my GCP infrastructure to AWS"). +4. Discover should treat live as the primary source (Step 1d), the sub-file + parses captures instead of re-running gcloud, and the output is a + schema-valid `gcp-resource-inventory.json` + `gcp-resource-clusters.json` + with `clustering_mode: "simplified_live"`, `discovery_sources: ["live"]`, + and no `drift` key. + +**Scenario B — live + Terraform (drift exercise):** + +1. As above, but also copy `workspace-terraform/main.tf` into the scratch root. +2. `main.tf` is deliberately stale — each divergence is commented with the + Step 6 merge rule it exercises. +3. Check outputs against `expected-drift.json`. + +**What a run must never produce** (either scenario): env var or secret values +anywhere; any mutating `gcloud` command; AWS service names in discover +artifacts; a halt caused by the failed asset-search/pubsub captures. + +## Regenerating / extending + +Captures follow the exact projections in `discover-live.md` Step 2 (same +filenames). If you add a capture type: whitelist row first, then fixture, then +extend `expected-drift.json`. All project IDs, names, emails, and account +numbers are synthetic. diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json new file mode 100644 index 00000000..602245c3 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json @@ -0,0 +1,79 @@ +{ + "_comment": "Expected outcomes when BOTH workspace-terraform/ and live-capture/ are used (scenario B in README.md). Assertions against gcp-resource-inventory.json + gcp-resource-clusters.json, not a full expected document.", + "metadata": { + "discovery_sources_must_include": ["terraform", "live"], + "clustering_mode_one_of": ["simplified", "simplified_live"] + }, + "resources": { + "google_sql_database_instance.db": { + "source": "live+terraform", + "config_must_include": { "tier_or_settings.tier": "db-custom-2-8192" }, + "expected_config_conflict_fields": ["settings.tier"], + "must_not_have": ["not_found_live", "unmanaged_by_terraform"] + }, + "google_cloud_run_v2_service.orders_api": { + "source": "live+terraform", + "classification": "PRIMARY", + "note": "matched by GCP name 'orders-api'; live image tag v42 overrides tf v40 as a sizing/capacity-class override or lands in config_conflicts — either is acceptable, but silent loss of the live value is not" + }, + "google_storage_bucket.assets": { + "not_found_live": true, + "note": "buckets capture succeeded and the bucket is absent live" + }, + "google_pubsub_topic.events": { + "must_not_have": ["not_found_live"], + "note": "pubsub capture FAILED — absence of evidence is not drift" + }, + "google_redis_instance.cache": { + "source": "live", + "unmanaged_by_terraform": true, + "classification": "PRIMARY" + }, + "google_cloud_run_v2_service.web_frontend": { + "source": "live", + "unmanaged_by_terraform": true, + "classification": "PRIMARY" + }, + "google_storage_bucket.acme_prod_uploads": { + "source": "live", + "unmanaged_by_terraform": true + }, + "google_compute_network.main": { + "must_not_have": ["not_found_live", "unmanaged_by_terraform"] + } + }, + "edges_must_include": [ + { + "from_contains": "orders_api", + "to_contains": "google_sql_database_instance", + "relationship_type": "data_dependency", + "evidence_contains": "cloudsql-instances" + }, + { + "from_contains": "google_sql_database_instance", + "to_contains": "google_compute_network", + "relationship_type": "network_membership" + } + ], + "clusters": { + "every_resource_in_exactly_one_cluster": true, + "networking_cluster_at_depth_0": true, + "unmanaged_primaries_have_clusters": ["google_redis_instance.cache", "google_cloud_run_v2_service.web_frontend"] + }, + "ai_detection": { "has_ai_workload": false, "confidence": 0 }, + "live_metadata": { + "method": "per_service", + "project": "acme-prod", + "capture_warnings_must_mention": ["pubsub", "asset"], + "drift": { + "resources_terraform_only_max": 1, + "resources_live_only_min": 4, + "config_conflict_fields_must_include": ["settings.tier"] + } + }, + "must_not_exist_anywhere": [ + "any env var VALUE (fixture env entries carry names only: DATABASE_URL, STRIPE_SECRET_KEY, etc.)", + "any secret VALUE or gcloud access token", + "AWS service names (discover scope boundary)" + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json new file mode 100644 index 00000000..fca24c82 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json @@ -0,0 +1,10 @@ +[ + { + "name": "acme-prod-uploads", + "location": "US-CENTRAL1", + "storageClass": "STANDARD", + "timeCreated": "2024-06-01T10:00:00Z", + "iamConfiguration": { "uniformBucketLevelAccess": { "enabled": true } }, + "versioning": { "enabled": true } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json new file mode 100644 index 00000000..6b357e6c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json @@ -0,0 +1,111 @@ +{ + "captured_at": "2026-07-20T18:20:00Z", + "gcloud_version": "Google Cloud SDK 532.0.0", + "account": "founder@acme-demo.com", + "project": "acme-prod", + "method": "per_service", + "captures": [ + { + "command": "gcloud asset search-all-resources --project=acme-prod --format=json", + "file": "assets.json", + "status": "failed", + "note": "Cloud Asset API has not been used in project acme-prod before or it is disabled" + }, + { + "command": "gcloud run services list --project=acme-prod --region=- --format=json(...)", + "file": "run.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud sql instances list --project=acme-prod --format=json(...)", + "file": "sql.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud container clusters list --project=acme-prod --format=json(...)", + "file": "gke.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud functions list --project=acme-prod --format=json(...)", + "file": "functions.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud storage buckets list --project=acme-prod --format=json(...)", + "file": "buckets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud pubsub topics list --project=acme-prod --format=json(...)", + "file": "pubsub.json", + "status": "failed", + "note": "Cloud Pub/Sub API has not been used in project acme-prod before or it is disabled" + }, + { + "command": "gcloud compute instances list --project=acme-prod --format=json(...)", + "file": "gce.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud compute networks list --project=acme-prod --format=json(...)", + "file": "networks.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud compute networks subnets list --project=acme-prod --format=json(...)", + "file": "subnets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud redis instances list --project=acme-prod --region=us-central1 --format=json(...)", + "file": "redis-us-central1.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud secrets list --project=acme-prod --format=json(...)", + "file": "secrets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud iam service-accounts list --project=acme-prod --format=json(...)", + "file": "sa.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud dns managed-zones list --project=acme-prod --format=json(...)", + "file": "dns.json", + "status": "skipped", + "note": "Cloud DNS API not enabled" + }, + { + "command": "gcloud spanner instances list --project=acme-prod --format=json(...)", + "file": "spanner.json", + "status": "skipped", + "note": "Spanner API not enabled" + }, + { + "command": "gcloud firestore databases list --project=acme-prod --format=json(...)", + "file": "firestore.json", + "status": "skipped", + "note": "Firestore API not enabled" + }, + { + "command": "gcloud ai endpoints list --project=acme-prod --region=us-central1 --format=json(...)", + "file": "vertex-us-central1.json", + "status": "skipped", + "note": "no aiplatform signal; Vertex AI API not enabled" + } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json new file mode 100644 index 00000000..d77f22d8 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json @@ -0,0 +1,7 @@ +[ + { + "name": "main", + "autoCreateSubnetworks": false, + "subnetworks": ["https://www.googleapis.com/compute/v1/projects/acme-prod/regions/us-central1/subnetworks/app"] + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json new file mode 100644 index 00000000..fe3fb55c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json @@ -0,0 +1,10 @@ +[ + { + "name": "projects/acme-prod/locations/us-central1/instances/cache", + "tier": "BASIC", + "memorySizeGb": 1, + "redisVersion": "REDIS_7_2", + "authorizedNetwork": "projects/acme-prod/global/networks/main", + "locationId": "us-central1-a" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json new file mode 100644 index 00000000..61ad66ac --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json @@ -0,0 +1,68 @@ +[ + { + "metadata": { + "name": "orders-api", + "labels": { + "cloud.googleapis.com/location": "us-central1" + }, + "annotations": { + "serving.knative.dev/creator": "founder@acme-demo.com" + } + }, + "spec": { + "template": { + "metadata": { + "annotations": { + "run.googleapis.com/cloudsql-instances": "acme-prod:us-central1:orders-db", + "autoscaling.knative.dev/maxScale": "10" + } + }, + "spec": { + "serviceAccountName": "app-sa@acme-prod.iam.gserviceaccount.com", + "containerConcurrency": 80, + "timeoutSeconds": 300, + "containers": [ + { + "image": "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v42", + "resources": { "limits": { "cpu": "2", "memory": "1Gi" } }, + "env": [ + { "name": "DATABASE_URL" }, + { "name": "REDIS_URL" }, + { "name": "STRIPE_SECRET_KEY" }, + { "name": "NODE_ENV" } + ] + } + ] + } + } + }, + "status": { "url": "https://orders-api-h4x2k3q-uc.a.run.app" } + }, + { + "metadata": { + "name": "web-frontend", + "labels": { + "cloud.googleapis.com/location": "us-central1" + }, + "annotations": {} + }, + "spec": { + "template": { + "metadata": { "annotations": { "autoscaling.knative.dev/maxScale": "4" } }, + "spec": { + "serviceAccountName": "app-sa@acme-prod.iam.gserviceaccount.com", + "containerConcurrency": 200, + "timeoutSeconds": 60, + "containers": [ + { + "image": "us-central1-docker.pkg.dev/acme-prod/apps/web-frontend:v17", + "resources": { "limits": { "cpu": "1", "memory": "512Mi" } }, + "env": [{ "name": "API_BASE_URL" }, { "name": "NODE_ENV" }] + } + ] + } + } + }, + "status": { "url": "https://web-frontend-h4x2k3q-uc.a.run.app" } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json new file mode 100644 index 00000000..9c077f78 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json @@ -0,0 +1,3 @@ +[ + { "email": "app-sa@acme-prod.iam.gserviceaccount.com", "displayName": "App runtime SA", "disabled": false } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json new file mode 100644 index 00000000..21874653 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json @@ -0,0 +1,12 @@ +[ + { + "name": "projects/123456789012/secrets/database-password", + "replication": { "automatic": {} }, + "createTime": "2024-06-01T10:05:00Z" + }, + { + "name": "projects/123456789012/secrets/stripe-api-key", + "replication": { "automatic": {} }, + "createTime": "2024-06-01T10:06:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json new file mode 100644 index 00000000..068182a7 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json @@ -0,0 +1,17 @@ +[ + { + "name": "orders-db", + "region": "us-central1", + "databaseVersion": "POSTGRES_16", + "settings": { + "tier": "db-custom-2-8192", + "availabilityType": "ZONAL", + "dataDiskSizeGb": "50", + "ipConfiguration": { + "privateNetwork": "projects/acme-prod/global/networks/main", + "ipv4Enabled": false + }, + "backupConfiguration": { "enabled": true } + } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json new file mode 100644 index 00000000..18d8148a --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json @@ -0,0 +1,8 @@ +[ + { + "name": "app", + "region": "https://www.googleapis.com/compute/v1/projects/acme-prod/regions/us-central1", + "network": "https://www.googleapis.com/compute/v1/projects/acme-prod/global/networks/main", + "ipCidrRange": "10.0.0.0/24" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf new file mode 100644 index 00000000..dc9105f5 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf @@ -0,0 +1,69 @@ +# Fixture Terraform — deliberately STALE relative to the live captures in +# ../live-capture/. Each divergence exercises a specific rule in +# discover-live.md Step 6 (Merge with IaC Discovery). Do not "fix" the drift. + +resource "google_compute_network" "main" { + name = "main" + auto_create_subnetworks = false +} + +resource "google_compute_subnetwork" "app" { + name = "app" + region = "us-central1" + network = google_compute_network.main.id + ip_cidr_range = "10.0.0.0/24" +} + +resource "google_service_account" "app" { + account_id = "app-sa" + display_name = "App runtime SA" +} + +# Drift: live tier is db-custom-2-8192 (someone scaled up in the console). +# → Step 6 rule 1 (config conflict on settings.tier, live value wins) +resource "google_sql_database_instance" "db" { + name = "orders-db" + region = "us-central1" + database_version = "POSTGRES_16" + settings { + tier = "db-f1-micro" + ip_configuration { + private_network = google_compute_network.main.id + ipv4_enabled = false + } + } +} + +resource "google_cloud_run_v2_service" "orders_api" { + name = "orders-api" + location = "us-central1" + template { + service_account = google_service_account.app.email + containers { + image = "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v40" + resources { + limits = { cpu = "2", memory = "1Gi" } + } + } + } +} + +# Declared but NOT deployed (absent from live buckets.json; buckets capture ok). +# → Step 6 rule 3 (not_found_live: true) +resource "google_storage_bucket" "assets" { + name = "acme-prod-assets" + location = "US-CENTRAL1" +} + +# Declared, and the pubsub live capture FAILED (API-not-enabled in manifest). +# → Step 6 rule 3 negative case: must NOT be marked not_found_live +# (absence of evidence is not drift). +resource "google_pubsub_topic" "events" { + name = "events" +} + +# NOTE deliberate absences from this file (present in live captures): +# - google_redis_instance "cache" → unmanaged_by_terraform (click-ops) +# - Cloud Run service "web-frontend" → unmanaged_by_terraform +# - Secret Manager secrets (2) → unmanaged_by_terraform +# - Storage bucket "acme-prod-uploads" → unmanaged_by_terraform diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md index 5c0216ee..bc0f2c9e 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md @@ -10,7 +10,7 @@ description: "Migrate workloads from Google Cloud Platform to AWS — including - **Re-platform by default**: Select AWS services that match GCP workload types (e.g., Cloud Run → Fargate, Cloud SQL → RDS). - **Dev sizing unless specified**: Default to development-tier capacity (e.g., db.t4g.micro, single AZ). Upgrade only on user direction. - **No human one-time migration costs**: Do not present human labor, professional services, or people-time work as dollar estimates or "one-time migration cost" budget categories. Vendor charges grounded in data (for example GCP data transfer egress in the infra estimate when billing exists) are allowed. -- **Multi-signal approach**: Design phase adapts based on available inputs — Terraform IaC for infrastructure, billing data for service mapping, and app code for AI workload detection. +- **Multi-signal approach**: Design phase adapts based on available inputs — live gcloud discovery and/or Terraform IaC for infrastructure, billing data for service mapping, and app code for AI workload detection. When live and IaC both run, live is authoritative for current state and disagreements surface as drift, never silently resolved. - **BigQuery / `google_bigquery_*`**: The skill **does not** recommend a specific AWS analytics or warehouse service. During **Clarify**, if discovery shows BigQuery (IaC `google_bigquery_*` and/or billing rows for BigQuery), you **must** surface the specialist advisory **before** Design (see `references/phases/clarify/clarify.md`). Design output uses **`Deferred — specialist engagement`**; keep directing the user to their **AWS account team** and/or a **data analytics migration partner** through Design, Estimate, and docs (see `references/phases/design/design-infra.md` BigQuery specialist gate). --- @@ -42,7 +42,6 @@ Each phase loads reference files on demand. To keep per-turn context manageable | `design-refs/design-ref-harness.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "harness"` | | `design-refs/design-ref-agentic-to-agentcore.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "strands"` | | `shared/retarget-gotchas.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "retarget"` | -| `shared/graviton.md` | Compute, DB, or cache in inventory OR `graviton_profile` present (Design/Estimate/Generate) | When adding new reference files, verify the phase's total loaded instructions remain under budget. If a new file would exceed ~800 lines when combined with other loaded refs, split it or make it conditional. @@ -68,11 +67,12 @@ If the user chooses to continue, proceed with the combined run. Load AI refs **a User must provide at least one GCP source: +- **Live gcloud CLI** (recommended for infrastructure): an authenticated `gcloud` CLI — read-only, consent-gated live discovery of the project (see `references/phases/discover/discover-live.md`) - **Terraform IaC**: `.tf` files (with optional `.tfvars`, `.tfstate`) - **Application code**: Source files with GCP SDK or AI framework imports - **Billing data**: GCP billing/cost/usage export files (CSV or JSON) -If none of the above are found, stop and ask user to provide at least one source type. +If no files are found, offer live discovery before stopping (see `discover.md` Step 1d). Live discovery covers infrastructure only — AI/agentic workload detection still requires application code. --- @@ -225,6 +225,7 @@ gcp-to-aws/ │ │ ├── discover/ │ │ │ ├── discover.md # Phase 1: Discover orchestrator │ │ │ ├── discover-iac.md # Terraform/IaC discovery +│ │ │ ├── discover-live.md # Live gcloud CLI discovery (read-only, consent-gated) │ │ │ ├── discover-app-code.md # App code discovery │ │ │ └── discover-billing.md # Billing data discovery │ │ ├── clarify/ @@ -285,25 +286,22 @@ gcp-to-aws/ │ ├── validate-migration-report.md # Post-write HTML completeness (Generate Step 4) │ ├── migration-complexity.md # Complexity tier definitions (small/medium/large) for timeline scaling │ ├── pricing-cache.md # Cached AWS + source provider pricing (±5-25%, primary source) -│ ├── graviton.md # Graviton/ARM64 tiers, mapping, per-phase rules (conditional load) -│ ├── schema-graviton.md # graviton_profile + cpu_architecture + architecture_comparison schemas │ └── bedrock-quotas.md # Bedrock TPM/RPM quota awareness, burndown rates, capacity planning ``` -| Condition | Action | -| ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| No GCP sources found (no `.tf`, no app code, no billing data) | Stop. Output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports) and try again." | -| `.phase-status.json` missing phase gate | Stop. Output: "Cannot enter Phase X: Phase Y-1 not completed. Start from Phase Y or resume Phase Y-1." | -| awspricing unavailable after 3 attempts | Display user warning about ±5-25% accuracy. Use `pricing-cache.md`. Add `pricing_source: "cached_fallback"` to the applicable `estimation-*.json` file. | -| User skips questions or says "use defaults for the rest" | Apply documented defaults for all remaining questions (essential questions and any unconfirmed sheet rows in wizard mode; current and subsequent batches in full mode). Q2/Q3 defaults add a report caveat. Phase 2 completes either way. | -| `aws-design.json` missing required clusters | Stop Phase 4. Output: "Re-run Phase 3 to generate missing cluster designs." | +| Condition | Action | +| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| No GCP sources found (no `.tf`, no app code, no billing data) | Offer live gcloud discovery per `discover.md` Step 1d. Only if declined or unavailable: Stop. Output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports), or re-run and accept live discovery." | +| `.phase-status.json` missing phase gate | Stop. Output: "Cannot enter Phase X: Phase Y-1 not completed. Start from Phase Y or resume Phase Y-1." | +| awspricing unavailable after 3 attempts | Display user warning about ±5-25% accuracy. Use `pricing-cache.md`. Add `pricing_source: "cached_fallback"` to the applicable `estimation-*.json` file. | +| User skips questions or says "use defaults for the rest" | Apply documented defaults for all remaining questions (essential questions and any unconfirmed sheet rows in wizard mode; current and subsequent batches in full mode). Q2/Q3 defaults add a report caveat. Phase 2 completes either way. | +| `aws-design.json` missing required clusters | Stop Phase 4. Output: "Re-run Phase 3 to generate missing cluster designs." | ## Defaults - **IaC output**: Terraform configurations, migration scripts, AI migration code, and documentation - **Region**: `us-east-1` (unless user specifies, or GCP region → AWS region mapping suggests otherwise) - **Sizing**: Development tier (e.g., `db.t4g.micro` for databases, 0.5 CPU for Fargate) -- **CPU architecture**: Graviton (ARM64) for all eligible compute when the workload is arm64-compatible; x86 only for incompatible workloads (Windows/.NET Framework, GPU/CUDA, RDS SQL Server). See `references/shared/graviton.md`. - **Migration mode**: Adapts based on available inputs (infrastructure, AI, or billing-only) - **Cost currency**: USD - **Timeline assumption**: 2-16 weeks depending on migration complexity — small (2-6 weeks), medium (6-12 weeks), large (12-18 weeks). See `references/shared/migration-complexity.md` for tier definitions. @@ -360,6 +358,7 @@ User can invoke the skill again to resume from `current_phase` (or deterministic **v1.0 includes:** - Terraform infrastructure discovery +- Live infrastructure discovery via authenticated gcloud CLI (read-only, consent-gated, with IaC drift detection) - App code scanning (AI workload detection) - Billing data import from GCP - User requirement clarification (assumption-sheet wizard by default: confirm detected/assumed values, answer only essential questions; full adaptive question flow available on request) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md new file mode 100644 index 00000000..13a2d503 --- /dev/null +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -0,0 +1,328 @@ +# Discover Phase: Live Discovery (gcloud CLI) + +> Self-contained live-discovery sub-file. Inventories the user's GCP project +> directly through their authenticated `gcloud` CLI — read-only, consent-gated, +> env-var names only. Produces the SAME artifacts as `discover-iac.md` +> (`gcp-resource-inventory.json` + `gcp-resource-clusters.json`, simplified +> clustering mode), so all downstream phases work identically. When IaC discovery +> also ran, merges live findings into the existing inventory and surfaces drift. +> If the user declines consent or `gcloud` is unavailable, exits cleanly with no +> output. + +**Execute ALL steps in order. Do not skip or optimize.** + +--- + +## Security Contract (applies to every step) + +1. **Exact-command whitelist.** Run ONLY commands that appear in Step 0 (preflight) + or the Step 2 Capture Command Table. Never any mutating verb (`create`, `update`, + `delete`, `set`, `add`, `remove`, `deploy`, `apply`, `import`, `patch`), never + `gcloud auth login` (interactive — hand off to the user), never + `gcloud auth print-access-token` or `print-identity-token` (prints credentials). +2. **Never capture secret values.** Every capture command uses an explicit + `--format="json(...)"` field projection. Projections include env var **names** + but never env var **values**, and never GCE instance `metadata.items` values. + Additionally, apply `discover-iac.md`'s sensitive-key redaction patterns + (`password`, `secret`, `api_key`, `access_key`, `private_key`, `client_secret`, + `token`, `credential`, `auth` — case-insensitive) to any config field before it + is written into an artifact: replace matched values with `"[REDACTED]"`. +3. **Always explicit scope.** Every command passes `--project="$GCP_PROJECT"` + explicitly. Never rely on the active gcloud config inside capture commands. +4. **Capture to files, not context.** Redirect stdout to files under + `$MIGRATION_DIR/live-capture/`. Process any capture file larger than ~100 + resources with a throwaway extraction script (same pattern as `discover.md`'s + lightweight billing extraction) — do NOT Read large raw captures into context. +5. **Consent first.** No `gcloud` command from the Step 2 table runs before the + user answers `[A]` in Step 1. Preflight commands in Step 0 are limited to + version/auth/config checks that touch no project data. + +--- + +## Step 0: Preflight + +1. **CLI installed:** run `gcloud --version` (first line only). + - Missing → tell the user: "The gcloud CLI isn't installed. Install it + (https://cloud.google.com/sdk/docs/install) and tell me to continue, or skip + live discovery." Wait. If skipped → exit cleanly. +2. **Authenticated:** run `gcloud auth list --filter=status:ACTIVE --format="value(account)"`. + - Empty → tell the user: "Your gcloud CLI has no active account. Run + `gcloud auth login` in your terminal — it needs a browser, so I can't run it + for you — then tell me to continue." Wait. If declined → exit cleanly. +3. **Project:** run `gcloud config get-value project`. + - Show the result and ask: "Discover project `[project-id]`? [Y] Yes / + [N] Use a different project (type its ID)". Set `$GCP_PROJECT` accordingly. + If the value is empty, ask the user to type the project ID. One project per + run — for multiple projects, run the migration once per project. + +## Step 1: Consent Gate + +Output exactly, then wait for the user's choice: + +``` +─── Live GCP Discovery (read-only) ─── + +I can inventory project [$GCP_PROJECT] directly using your +authenticated gcloud CLI. This runs LIST/DESCRIBE commands only: + + ✓ Captured: resource names, types, regions, machine/instance + sizing, container images, network topology, env var NAMES, + secret NAMES, and labels. + ✗ Never captured: env var values, secret values, database + contents, instance metadata values, access tokens, or source + code. No command that creates, changes, or deletes anything + will run. + +Output is written to .migration//live-capture/ (gitignored). + +[A] Proceed with live discovery +[B] Skip — use workspace files only +``` + +- **[A]** → continue to Step 2. +- **[B]** → exit cleanly with no output (record the decline for the orchestrator). + +## Step 2: Capture + +Create `$MIGRATION_DIR/live-capture/`. + +**2a. Fast path — Cloud Asset Inventory (one call, whole project):** + +``` +gcloud asset search-all-resources --project="$GCP_PROJECT" \ + --format=json > $MIGRATION_DIR/live-capture/assets.json +``` + +- Success → record `method: "asset_search"` in the manifest, then run only the + **enrichment rows** (marked E) of the table below for asset types that were + found (asset search returns names/types/locations but thin config). +- Failure (Cloud Asset API not enabled, or permission denied) → record + `method: "per_service"` and run every applicable table row. Do NOT try to + enable the API (that would be a mutation). + +**2b. Capture Command Table.** Each row redirects to the named file. On +"API not enabled" / permission errors: record the row as `failed` or `skipped` +in the manifest and continue — a missing service is normal, never a halt. + +| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | +| -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ---- | +| 1 | `gcloud run services list --region=- --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | +| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | +| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | +| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | +| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | +| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | +| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | +| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | | +| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | | +| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | +| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | | +| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | | +| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | +| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | +| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | +| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | + +**Scale guard:** if `assets.json` (or any capture) exceeds ~100 resources, write a +throwaway extraction script to `$MIGRATION_DIR/_extract_live.py` that projects only +the fields needed by Step 3, run it, write its JSON output next to the raw file +with a `-extracted.json` suffix, and delete the script. Never Read the oversized +raw file directly. + +**2c. Write the manifest** — `$MIGRATION_DIR/live-capture/manifest.json`: + +```json +{ + "captured_at": "", + "gcloud_version": "", + "account": "", + "project": "<$GCP_PROJECT>", + "method": "asset_search|per_service", + "captures": [ + { "command": "", "file": "", "status": "ok|failed|skipped", "note": null } + ] +} +``` + +Every attempted or deliberately skipped row gets an entry. + +## Step 3: Map Captures to Inventory Resources + +Synthesize Terraform-style identity so downstream design-refs (keyed on +`google_*` types) work unchanged: + +- `address` = `{terraform_type}.{sanitized_resource_name}` (lowercase, `-`→`_`) +- `type` = from the mapping table below +- `name` = sanitized resource name +- `config` = the projected fields from the capture (redaction rules from the + Security Contract apply) +- `source` = `"live"` on every entry + +**Asset/CLI type → Terraform type mapping:** + +| Captured type | Terraform `type` | +| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `run.googleapis.com/Service` / row 1 | `google_cloud_run_v2_service` | +| `sqladmin.googleapis.com/Instance` / row 2 | `google_sql_database_instance` | +| `container.googleapis.com/Cluster` / row 3 | `google_container_cluster` | +| row 4 with `environment: GEN_2` | `google_cloudfunctions2_function` | +| row 4 with `environment: GEN_1` (or unset) | `google_cloudfunctions_function` | +| `storage.googleapis.com/Bucket` / row 5 | `google_storage_bucket` | +| `pubsub.googleapis.com/Topic` / row 6 | `google_pubsub_topic` | +| `compute.googleapis.com/Instance` / row 7 | `google_compute_instance` | +| `compute.googleapis.com/Network` / row 8 | `google_compute_network` | +| `compute.googleapis.com/Subnetwork` / row 9 | `google_compute_subnetwork` | +| `redis.googleapis.com/Instance` / row 10 | `google_redis_instance` | +| `secretmanager.googleapis.com/Secret` / row 11 | `google_secret_manager_secret` | +| `iam.googleapis.com/ServiceAccount` / row 12 | `google_service_account` | +| `dns.googleapis.com/ManagedZone` / row 13 | `google_dns_managed_zone` | +| `spanner.googleapis.com/Instance` / row 14 | `google_spanner_instance` | +| `firestore.googleapis.com/Database` / row 15 | `google_firestore_database` | +| `bigquery.googleapis.com/Dataset` | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | +| `aiplatform.googleapis.com/Endpoint` / row 16 | `google_vertex_ai_endpoint` | +| `aiplatform.googleapis.com/*` (other) | `google_vertex_ai_*` (matching suffix) | +| Any other asset type | Do NOT guess a mapping. Count it in `live_metadata.unmapped_asset_types` and exclude from the inventory. | + +**Classification:** apply `discover-iac.md` Step 3S rules — the Priority 1 PRIMARY +types list, everything else SECONDARY with role inferred from type +(`google_service_account` → identity; networks/subnets/DNS → network_path; +secrets → encryption; else configuration). `confidence: 0.99`. + +**AI detection:** if any `aiplatform.googleapis.com/*` asset or Vertex endpoint was +captured, populate `ai_detection` exactly as `discover-iac.md` Step 2 would +(signal method `"live_gcloud"`, confidence 95, `ai_services: ["vertex_ai"]`, +`has_ai_workload: true`). Otherwise `has_ai_workload: false`, `confidence: 0`. + +## Step 4: Infer Edges from Resolved Config + +Live captures contain resolved values, which often beat HCL references. Build +`edges[]` using ONLY these deterministic rules (evidence = the config field path): + +| Config field (captured) | Edge | +| --------------------------------------------------------------- | ------------------------------------------------------------------- | +| Cloud Run annotation `run.googleapis.com/cloudsql-instances` | run service → SQL instance, `data_dependency` | +| Cloud Run annotation `run.googleapis.com/vpc-access-connector` | run service → network, `network_membership` | +| `spec.template.spec.serviceAccountName` / `serviceAccountEmail` | service account → workload, `serves` (populate the SA's `serves[]`) | +| SQL `settings.ipConfiguration.privateNetwork` | SQL instance → network, `network_membership` | +| GCE `networkInterfaces[].network` / GKE `network` | instance/cluster → network, `network_membership` | +| Subnet `network` | subnet → network, `network_membership` | +| Redis `authorizedNetwork` | redis → network, `network_membership` | + +No other inference — do not guess relationships from names, labels, or env var +names. + +## Step 5: Cluster (Simplified Mode) + +Apply `discover-iac.md` **Step 3S** clustering rules regardless of resource count +(networking cluster at depth 0; one cluster per PRIMARY plus its `serves` +secondaries at depth 1; same `{category}_{type}_{region}_{sequence}` naming; +region from the captured `region`/`location`/zone-derived-region). Set metadata +`"clustering_mode": "simplified_live"`. If more than 25 PRIMARY resources were +captured, warn the user that clustering is coarse at this scale and suggest +narrowing to specific services or regions — but continue. + +**Live-specific clustering rules** (Step 3S assumes files, which don't exist here): + +- **Regionless resources** (Pub/Sub topics, global buckets without a single + region): use `"global"` as the region component of `cluster_id` and + `gcp_region`. +- **Shared secondaries** (e.g., one service account serving multiple primaries): + assign the resource to the cluster of the FIRST primary in its `serves[]` + array; `serves[]` still lists all of them. +- **Evidence-less secondaries** (no Step 4 edge and empty `serves[]`, e.g., + secrets): do NOT attach them to an unrelated primary's cluster and do NOT + fabricate a `serves` relationship. Group them into their own cluster per + category+region (e.g., `security_secrets_global_001`) at depth 1. + +## Step 6: Merge with IaC Discovery (only if `discover-iac.md` produced output) + +If `gcp-resource-inventory.json` does NOT already exist, skip to Step 7 (live is +the sole source). + +Otherwise the IaC inventory + clusters are the BASE. Match live↔IaC entries by +Terraform `type` + GCP resource name (live `name` vs the IaC resource's +`config.name`, falling back to the address name component). Then: + +1. **Matched:** keep the IaC entry (its address, classification, cluster, + depth). Overwrite `config` sizing/capacity fields with live values (live + reflects reality). Record every changed field in + `live_metadata.drift.config_conflicts[]` as + `{ "address", "field", "terraform_value", "live_value" }`. Set + `source: "live+terraform"`. +2. **Live-only:** append the entry with `unmanaged_by_terraform: true`. Attach it + to an existing cluster of the same category+region when one exists; otherwise + append a new simplified cluster (and add it to `creation_order` at its depth). +3. **IaC-only:** set `source: "terraform"` on every unmatched IaC entry. Set + `not_found_live: true` ONLY if the capture covering that resource's service + succeeded (manifest `ok`). If the relevant capture failed or was skipped, + leave the entry otherwise untouched — absence of evidence is not drift. +4. **Drift summary:** `live_metadata.drift = { "resources_live_only": N, + "resources_terraform_only": M, "config_conflicts": [...] }`. + `resources_terraform_only` counts ONLY entries with `not_found_live: true` + (confirmed absent), never capture-failed unknowns. +5. **Merged metadata:** keep the IaC base's `clustering_mode` (`"simplified"` or + absent for full clustering) — `"simplified_live"` is for live-only runs. Set + `metadata.discovery_sources` to include both sources. + +Never silently resolve a disagreement — every conflict lands in the drift record. + +## Step 7: Write Output Files + +Load `references/shared/schema-discover-iac.md` (if not already loaded) and +write/update: + +1. `$MIGRATION_DIR/gcp-resource-inventory.json` — exact schema; plus: + - `metadata.discovery_sources`: `["live"]`, `["terraform", "live"]`, etc. + - `metadata.clustering_mode`: `"simplified_live"` (live-only runs) + - top-level `live_metadata`: + + ```json + { + "found": true, + "captured_at": "", + "project": "<$GCP_PROJECT>", + "method": "asset_search|per_service", + "capture_warnings": [""], + "unmapped_asset_types": { "": 2 }, + "drift": { "resources_live_only": 0, "resources_terraform_only": 0, "config_conflicts": [] } + } + ``` + + (`drift` present only when Step 6 merged.) + +2. `$MIGRATION_DIR/gcp-resource-clusters.json` — exact schema (merged or fresh). +3. Validate per `discover-iac.md` Step 7c (every resource in exactly one cluster, + IDs consistent, valid JSON). Report: "Live discovery: X resources captured from + project [id] (Y unmanaged by Terraform, Z config conflicts)." + +The parent `discover.md` owns the phase status update — do not touch +`.phase-status.json` here. + +--- + +## Error Handling + +| Error | Behavior | +| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | +| Asset search fails (API not enabled, 403) | Fall back to per-service rows; record in manifest | +| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt | +| Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | +| Capture file unparseable | Record warning, skip that file, continue | +| Every capture failed | Exit with no output; tell the user which permissions are missing (`roles/viewer` covers all rows) | + +**Key principle:** partial results are better than no results. Record what failed; +never fabricate what wasn't captured. + +## Scope Boundary + +**This sub-file covers live GCP discovery ONLY.** + +FORBIDDEN — Do NOT include ANY of: + +- AWS service names, recommendations, or equivalents +- Migration strategies, phases, timelines, cost estimates, or effort estimates +- Any mutating gcloud command, `auth login`, or token printing +- Env var values, secret values, or unredacted sensitive config anywhere + +**Your ONLY job: inventory what exists in GCP. Nothing else.** diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md index 283194c1..fd652cb5 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md @@ -6,6 +6,7 @@ Lightweight orchestrator that delegates to domain-specific discoverers. Each sub ## Sub-Discovery Files - **discover-iac.md** → `gcp-resource-inventory.json` + `gcp-resource-clusters.json` (if Terraform found); may also write `ai-workload-profile.json` when **Vertex-strong** (see `discover-iac.md` Step 7d) +- **discover-live.md** → `gcp-resource-inventory.json` + `gcp-resource-clusters.json` from the user's authenticated `gcloud` CLI (read-only, consent-gated); merges into the IaC inventory with drift when both run - **discover-app-code.md** → `ai-workload-profile.json` when AI confidence ≥ 70% (may **merge** with an existing `iac_vertex` profile) - **discover-billing.md** → `billing-profile.json` (if billing data found) @@ -110,7 +111,28 @@ When Terraform is present, billing data is supplementary — only service-level **Critical:** Do **not** Read the billing file with the Read tool. Do **not** load `discover-billing.md` or `schema-discover-billing.md`. -**If NONE of the three checks found files**: STOP and output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports) and try again." +**1d. Live discovery (gcloud CLI):** +Runs AFTER 1a–1c sub-discoveries complete, so its IaC merge sees their output. + +- If `$MIGRATION_DIR/live-capture/manifest.json` already exists (a prior capture, + e.g. a resumed run) → Load `references/phases/discover/discover-live.md` and + execute from its Step 3 (parse the existing captures; skip consent/preflight/ + capture — they already happened). +- Else if Terraform files were found in 1a → offer ONCE as an optional cross-check: + "I found Terraform covering your infrastructure. Want me to cross-check it + against your live GCP project via your authenticated gcloud CLI (read-only, + with your consent)? This catches resources managed outside Terraform." + On yes → Load `references/phases/discover/discover-live.md`. On no → continue + (do not re-ask this run). +- Else if NO files were found by 1a–1c → offer live discovery as the primary + source instead of stopping: "No local GCP sources detected (Terraform, app + code, or billing exports) — but I can discover your project directly via your + authenticated gcloud CLI (read-only, with your consent). Proceed?" + On yes → Load `references/phases/discover/discover-live.md`. + On no (or gcloud unavailable / consent declined inside the sub-file) → STOP + and output: "No GCP sources detected. Provide at least one source type + (Terraform files, application code, or billing exports), or re-run and accept + live discovery." ## Step 2: Check Outputs @@ -128,6 +150,7 @@ After all loaded sub-discoveries complete, check what artifacts were produced in - If its Step 4 exit gate applied (overall AI confidence **below** 70%) **and** no `ai-workload-profile.json` exists -> **allow completion** (app-code route may produce no AI profile). - If Step 4 exit applied with confidence below 70% **but** `ai-workload-profile.json` exists with `metadata.profile_source` = `"iac_vertex"` -> **allow completion** (IaC-inferred profile retained). - If execution continued to Steps 5–8 (confidence **≥** 70%) -> **require** `ai-workload-profile.json`. + - If `discover-live.md` ran AND capture happened (`$MIGRATION_DIR/live-capture/manifest.json` exists) -> require `gcp-resource-inventory.json` and `gcp-resource-clusters.json`, with `live_metadata` present in the inventory. (If the user declined consent or gcloud was unavailable, the sub-file exited cleanly — no artifact required.) - If full `discover-billing.md` ran OR lightweight billing extraction ran -> require `billing-profile.json` - If any triggered route is missing its required artifact(s): STOP and output: "Discover route [name] did not produce required artifacts. Resolve the sub-discovery failure before completing Phase 1." @@ -166,6 +189,7 @@ Only after `HANDOFF_OK`. In the **same turn** as the output message below, use t Output to user — build message from whichever artifacts exist: - If `gcp-resource-inventory.json` exists: "Discovered X total resources across Y clusters." +- If live discovery ran: "Live discovery captured N resources from project [id]." Plus, when IaC also ran: "Drift check: A resources live but not in Terraform, B in Terraform but not live, C config conflicts (live values used)." - If `ai-workload-profile.json` exists: "Detected AI workloads (source: [ai_source])." - If `billing-profile.json` exists: "Parsed billing data ($Z/month across N services)." diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md index fd352af6..d78a538e 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md @@ -104,6 +104,50 @@ Complete inventory of discovered GCP resources with classification, dependencies - `signals_found[]` — array of detection signals with method, pattern, confidence, evidence - `ai_services[]` — list of AI services detected (vertex_ai, bigquery_ml, etc.) +### Live discovery extensions (present when `discover-live.md` ran) + +Live gcloud discovery produces the same inventory/cluster schemas with these additions: + +**`metadata` additional fields:** + +- `discovery_sources` — string[]: which sources produced data, e.g. `["live"]` or `["terraform", "live"]`. `terraform_version` may be `null` on live-only runs. +- `clustering_mode` — `"simplified"` (IaC Step 3S; also used for merged IaC+live runs), `"simplified_live"` (live-only runs), or absent (full IaC clustering). + +**`resources[]` optional fields:** + +- `source` — `"terraform"`, `"live"`, or `"live+terraform"` (merged entry) +- `unmanaged_by_terraform` — `true` when live discovery found the resource but no Terraform manages it (click-ops drift) +- `not_found_live` — `true` when Terraform declares the resource but the (successful) live capture did not find it deployed + +**Top-level `live_metadata` section:** + +```json +{ + "live_metadata": { + "found": true, + "captured_at": "2026-07-20T18:20:00Z", + "project": "acme-prod", + "method": "asset_search", + "capture_warnings": [], + "unmapped_asset_types": {}, + "drift": { + "resources_live_only": 0, + "resources_terraform_only": 0, + "config_conflicts": [ + { + "address": "google_sql_database_instance.db", + "field": "settings.tier", + "terraform_value": "db-f1-micro", + "live_value": "db-custom-2-8192" + } + ] + } + } +} +``` + +`drift` is present only when Terraform AND live discovery both produced resources. Env var and secret VALUES must never appear anywhere in the inventory — names only, with `discover-iac.md` Step 0 redaction patterns applied. + --- ## gcp-resource-clusters.json (Phase 1 output) From 5d53e1cc5997623e48efdf678d975fbb5f096bd6 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 13:56:45 -0700 Subject: [PATCH 03/11] =?UTF-8?q?docs(gcp-to-aws):=20document=20why=20live?= =?UTF-8?q?=20discovery=20avoids=20CAI=20full-metadata=20dumps=20and=20rel?= =?UTF-8?q?ationship=20queries=20Two=20design=20decisions=20in=20discover-?= =?UTF-8?q?live.md=20were=20implicit=20and=20invite=20well-meaning=20'opti?= =?UTF-8?q?mization':=20(1)=20gcloud=20asset=20list=20--content-type=3Dres?= =?UTF-8?q?ource=20returns=20full=20resource.data=20in=20one=20call=20but?= =?UTF-8?q?=20includes=20env=20var=20values,=20breaking=20the=20values-nev?= =?UTF-8?q?er-captured=20contract;=20(2)=20CAI=20relationships.*=20queries?= =?UTF-8?q?=20parse=20on=20the=20standard=20search=20endpoint=20but=20the?= =?UTF-8?q?=20data=20requires=20SCC=20Premium/Enterprise=20or=20Gemini=20C?= =?UTF-8?q?loud=20Assist=20=E2=80=94=20without=20the=20entitlement=20they?= =?UTF-8?q?=20return=20nothing,=20and=20most=20relationship=20types=20aren?= =?UTF-8?q?'t=20search-available=20anyway.=20Both=20now=20stated=20inline?= =?UTF-8?q?=20with=20do-not-change=20guidance.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../phases/discover/discover-live.md | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index 13a2d503..b7877644 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -96,6 +96,15 @@ gcloud asset search-all-resources --project="$GCP_PROJECT" \ - Success → record `method: "asset_search"` in the manifest, then run only the **enrichment rows** (marked E) of the table below for asset types that were found (asset search returns names/types/locations but thin config). + + > **Why not `gcloud asset list --content-type=resource` (full metadata, one + > call)?** Deliberate. Full `resource.data` includes env var VALUES (Cloud + > Run, Functions) and instance metadata values — writing it to + > `live-capture/` would put secret material on disk and break this file's + > "values never captured" contract. Thin search + the projected enrichment + > rows below keep values out of the captures entirely. The same applies to + > `search-all-resources --read-mask` with resource data. Do NOT "optimize" + > this into a full-metadata dump. - Failure (Cloud Asset API not enabled, or permission denied) → record `method: "per_service"` and run every applicable table row. Do NOT try to enable the API (that would be a mutation). @@ -211,6 +220,17 @@ Live captures contain resolved values, which often beat HCL references. Build No other inference — do not guess relationships from names, labels, or env var names. +> **Why not Cloud Asset Inventory relationship types?** Deliberate. CAI's +> relationship data (including `relationships.*` queries on +> `search-all-resources` — the query SYNTAX works on the standard endpoint, +> which misleads) requires the Security Command Center Premium/Enterprise tier +> or Gemini Cloud Assist, which the startups this skill targets do not have; +> without the entitlement those queries return nothing. Most relationship types +> are also unavailable in the search API entirely. Resolved-config inference +> above needs only `roles/viewer` and covers the workload-shaped edges that +> matter for migration sequencing. Do NOT add `relationships.*` queries to the +> capture set. + ## Step 5: Cluster (Simplified Mode) Apply `discover-iac.md` **Step 3S** clustering rules regardless of resource count From 1198fb929ee994bfb3119a1fc934aee8305d3998 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 14:21:19 -0700 Subject: [PATCH 04/11] =?UTF-8?q?fix(gcp-to-aws):=20address=20review=20?= =?UTF-8?q?=E2=80=94=20restore=20graviton,=20widen=20live=20offer,=20close?= =?UTF-8?q?=20capture=20gaps=20-=20Restore=20graviton=20content=20in=20SKI?= =?UTF-8?q?LL.md=20that=20a=20stale=20whole-file=20copy=20dropped=20(rebui?= =?UTF-8?q?lt=20from=20main=20+=20feature=20edits;=20conditional-load=20ro?= =?UTF-8?q?w,=20file=20tree,=20Defaults)=20-=20Offer=20live=20discovery=20?= =?UTF-8?q?whenever=20no=20Terraform=20is=20found,=20not=20only=20when=20N?= =?UTF-8?q?O=20files=20exist=20=E2=80=94=20billing-only=20and=20app-code-o?= =?UTF-8?q?nly=20workspaces=20now=20get=20the=20live=20path=20-=20Add=20bq?= =?UTF-8?q?=20ls=20fallback=20row=20so=20BigQuery=20(and=20its=20specialis?= =?UTF-8?q?t=20gate)=20survives=20the=20common=20CAI-disabled=20case;=20ma?= =?UTF-8?q?pping=20table=20references=20it=20-=20Drop=20undocumented=20'gc?= =?UTF-8?q?loud=20run=20services=20list=20--region=3D-'=20(managed=20Cloud?= =?UTF-8?q?=20Run=20lists=20all=20regions=20by=20default);=20fix=20redis?= =?UTF-8?q?=20region-source=20wording=20for=20CAI=20path=20-=20Use=20--sco?= =?UTF-8?q?pe=3Dprojects/...=20for=20asset=20search=20(documented=20scopin?= =?UTF-8?q?g=20flag);=20document=20why=20--asset-types=20stays=20unfiltere?= =?UTF-8?q?d=20(unmapped=5Fasset=5Ftypes=20signal)=20-=20Mark=20networks/s?= =?UTF-8?q?ubnets/secrets/SA=20rows=20as=20enrichment=20so=20the=20CAI=20p?= =?UTF-8?q?ath=20keeps=20edge=20and=20name-inventory=20fidelity=20-=20Comm?= =?UTF-8?q?it=20check=5Fexpected=5Fdrift.py=20asserter=20(stdlib-only)=20+?= =?UTF-8?q?=20README=20usage;=20validated=20PASS=20against=20a=20fresh=20f?= =?UTF-8?q?resh-agent=20scenario-B=20replay=20of=20this=20revision=20-=20C?= =?UTF-8?q?larify=20merge=20rule=201=20(overwrite=20any=20disagreeing=20co?= =?UTF-8?q?nfig=20value,=20enrichment=20vs=20conflict=20distinction);=20no?= =?UTF-8?q?te=20provisioned-vs-actual=20disk=20caveat=20and=20live-compute?= =?UTF-8?q?=20graviton=20gap=20as=20follow-ups?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../fixtures/gcp-live-capture/README.md | 4 +- .../gcp-live-capture/check_expected_drift.py | 137 ++++++++++++++++++ .../skills/gcp-to-aws/SKILL.md | 4 + .../phases/discover/discover-live.md | 65 ++++++--- .../references/phases/discover/discover.md | 21 +-- 5 files changed, 198 insertions(+), 33 deletions(-) create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md index bc48351b..1333636a 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -42,7 +42,9 @@ exercises the designed-for behaviors: 1. As above, but also copy `workspace-terraform/main.tf` into the scratch root. 2. `main.tf` is deliberately stale — each divergence is commented with the Step 6 merge rule it exercises. -3. Check outputs against `expected-drift.json`. +3. Check outputs against `expected-drift.json` — machine-checkable via + `python3 check_expected_drift.py ` (exits non-zero on any failed + assertion). **What a run must never produce** (either scenario): env var or secret values anywhere; any mutating `gcloud` command; AWS service names in discover diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py new file mode 100644 index 00000000..0046727c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py @@ -0,0 +1,137 @@ +#!/usr/bin/env python3 +"""Assert a Discover run's output against expected-drift.json (scenario B). + +Usage: + python3 check_expected_drift.py + +Where contains gcp-resource-inventory.json and +gcp-resource-clusters.json produced by a replay of this fixture's scenario B +(live-capture/ + workspace-terraform/main.tf). Exits 0 on PASS, 1 on FAIL with +one line per failed assertion. Stdlib only. +""" + +import json +import sys +from pathlib import Path + +FAILS: list[str] = [] + + +def check(cond: bool, msg: str) -> None: + if not cond: + FAILS.append(msg) + + +def get_conflict_fields(inv: dict) -> list[str]: + drift = inv.get("live_metadata", {}).get("drift", {}) + return [c.get("field", "") for c in drift.get("config_conflicts", [])] + + +def main() -> int: + if len(sys.argv) != 2: + print(__doc__) + return 2 + run_dir = Path(sys.argv[1]) + fixture_dir = Path(__file__).resolve().parent + + inv = json.loads((run_dir / "gcp-resource-inventory.json").read_text()) + clusters = json.loads((run_dir / "gcp-resource-clusters.json").read_text()) + exp = json.loads((fixture_dir / "expected-drift.json").read_text()) + + # Metadata + meta = inv["metadata"] + for s in exp["metadata"]["discovery_sources_must_include"]: + check(s in meta.get("discovery_sources", []), f"discovery_sources missing {s}") + check( + meta.get("clustering_mode") in exp["metadata"]["clustering_mode_one_of"], + f"clustering_mode {meta.get('clustering_mode')} not in {exp['metadata']['clustering_mode_one_of']}", + ) + + # Resources + res = {r["address"]: r for r in inv["resources"]} + for addr, e in exp["resources"].items(): + r = res.get(addr) + if r is None: + # Address synthesis can differ for live-only names; try name-suffix match + candidates = [v for k, v in res.items() if k.split(".")[0] == addr.split(".")[0] and addr.split(".")[1] in k] + if len(candidates) == 1: + r = candidates[0] + else: + check(False, f"missing resource {addr}") + continue + if "source" in e: + check(r.get("source") == e["source"], f"{addr} source={r.get('source')} want {e['source']}") + if "classification" in e: + check(r.get("classification") == e["classification"], f"{addr} classification") + if e.get("not_found_live"): + check(r.get("not_found_live") is True, f"{addr} not_found_live missing") + if e.get("unmanaged_by_terraform"): + check(r.get("unmanaged_by_terraform") is True, f"{addr} unmanaged_by_terraform missing") + for flag in e.get("must_not_have", []): + check(flag not in r or r.get(flag) is not True, f"{addr} must not have {flag}") + if "config_must_include" in e: + cfg_text = json.dumps(r.get("config", {})) + for v in e["config_must_include"].values(): + check(v in cfg_text, f"{addr} config missing value {v}") + + # Edges + all_edges = [edge for c in clusters["clusters"] for edge in c.get("edges", [])] + for ee in exp["edges_must_include"]: + found = any( + ee["from_contains"] in edge.get("from", "") + and ee["to_contains"] in edge.get("to", "") + and edge.get("relationship_type") == ee["relationship_type"] + and ( + "evidence_contains" not in ee + or ee["evidence_contains"] in json.dumps(edge.get("evidence", {})) + ) + for edge in all_edges + ) + check(found, f"edge missing: {ee['from_contains']} -> {ee['to_contains']} ({ee['relationship_type']})") + + # Cluster coverage + clustered = {a for c in clusters["clusters"] for a in c["primary_resources"] + c["secondary_resources"]} + check(set(res) == clustered, f"cluster coverage mismatch: {sorted(set(res) ^ clustered)}") + check( + any(c["creation_order_depth"] == 0 and "networking" in c["cluster_id"] for c in clusters["clusters"]), + "no networking cluster at depth 0", + ) + + # AI detection + check(inv["ai_detection"]["has_ai_workload"] is exp["ai_detection"]["has_ai_workload"], "ai_detection mismatch") + + # live_metadata + drift + lm = inv["live_metadata"] + check(lm.get("method") == exp["live_metadata"]["method"], "live method") + check(lm.get("project") == exp["live_metadata"]["project"], "live project") + warns = json.dumps(lm.get("capture_warnings", [])).lower() + for w in exp["live_metadata"]["capture_warnings_must_mention"]: + check(w in warns, f"capture_warnings missing mention of {w}") + drift = lm.get("drift", {}) + check( + drift.get("resources_terraform_only", 99) <= exp["live_metadata"]["drift"]["resources_terraform_only_max"], + f"resources_terraform_only={drift.get('resources_terraform_only')}", + ) + check( + drift.get("resources_live_only", 0) >= exp["live_metadata"]["drift"]["resources_live_only_min"], + f"resources_live_only={drift.get('resources_live_only')}", + ) + conflict_fields = get_conflict_fields(inv) + for f in exp["live_metadata"]["drift"]["config_conflict_fields_must_include"]: + check(f in conflict_fields, f"config_conflicts missing field {f}") + + # Safety: no fixture env names paired with values, no AWS names + doc = json.dumps(inv) + check("aws_" not in doc.lower().replace("aws-design", ""), "possible AWS naming in discover artifact") + + if FAILS: + print(f"FAIL ({len(FAILS)}):") + for f in FAILS: + print(f" - {f}") + return 1 + print("PASS — expected-drift.json assertions hold") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md index bc0f2c9e..5142e555 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md @@ -42,6 +42,7 @@ Each phase loads reference files on demand. To keep per-turn context manageable | `design-refs/design-ref-harness.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "harness"` | | `design-refs/design-ref-agentic-to-agentcore.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "strands"` | | `shared/retarget-gotchas.md` | `agentic_profile.is_agentic == true` AND `ai_constraints.agentic.migration_approach == "retarget"` | +| `shared/graviton.md` | Compute, DB, or cache in inventory OR `graviton_profile` present (Design/Estimate/Generate) | When adding new reference files, verify the phase's total loaded instructions remain under budget. If a new file would exceed ~800 lines when combined with other loaded refs, split it or make it conditional. @@ -286,6 +287,8 @@ gcp-to-aws/ │ ├── validate-migration-report.md # Post-write HTML completeness (Generate Step 4) │ ├── migration-complexity.md # Complexity tier definitions (small/medium/large) for timeline scaling │ ├── pricing-cache.md # Cached AWS + source provider pricing (±5-25%, primary source) +│ ├── graviton.md # Graviton/ARM64 tiers, mapping, per-phase rules (conditional load) +│ ├── schema-graviton.md # graviton_profile + cpu_architecture + architecture_comparison schemas │ └── bedrock-quotas.md # Bedrock TPM/RPM quota awareness, burndown rates, capacity planning ``` @@ -302,6 +305,7 @@ gcp-to-aws/ - **IaC output**: Terraform configurations, migration scripts, AI migration code, and documentation - **Region**: `us-east-1` (unless user specifies, or GCP region → AWS region mapping suggests otherwise) - **Sizing**: Development tier (e.g., `db.t4g.micro` for databases, 0.5 CPU for Fargate) +- **CPU architecture**: Graviton (ARM64) for all eligible compute when the workload is arm64-compatible; x86 only for incompatible workloads (Windows/.NET Framework, GPU/CUDA, RDS SQL Server). See `references/shared/graviton.md`. - **Migration mode**: Adapts based on available inputs (infrastructure, AI, or billing-only) - **Cost currency**: USD - **Timeline assumption**: 2-16 weeks depending on migration complexity — small (2-6 weeks), medium (6-12 weeks), large (12-18 weeks). See `references/shared/migration-complexity.md` for tier definitions. diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index b7877644..54934428 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -89,13 +89,20 @@ Create `$MIGRATION_DIR/live-capture/`. **2a. Fast path — Cloud Asset Inventory (one call, whole project):** ``` -gcloud asset search-all-resources --project="$GCP_PROJECT" \ +gcloud asset search-all-resources --scope="projects/$GCP_PROJECT" \ --format=json > $MIGRATION_DIR/live-capture/assets.json ``` +(`--scope` is the documented scoping flag for asset search — do not rely on the +active project. No `--asset-types` filter on purpose: unfiltered results feed +`live_metadata.unmapped_asset_types`, which tells the user what ELSE lives in +the project; the Step 2 scale guard handles large outputs.) + - Success → record `method: "asset_search"` in the manifest, then run only the **enrichment rows** (marked E) of the table below for asset types that were - found (asset search returns names/types/locations but thin config). + found (asset search returns names/types/locations but thin config). The cheap + networking/secrets/identity lists (rows 8, 9, 11, 12) are E rows precisely so + edge inference and name inventories keep full fidelity on this path. > **Why not `gcloud asset list --content-type=resource` (full metadata, one > call)?** Deliberate. Full `resource.data` includes env var VALUES (Cloud @@ -113,24 +120,35 @@ gcloud asset search-all-resources --project="$GCP_PROJECT" \ "API not enabled" / permission errors: record the row as `failed` or `skipped` in the manifest and continue — a missing service is normal, never a halt. -| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | -| -- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ---- | -| 1 | `gcloud run services list --region=- --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | -| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | -| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | -| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | -| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | -| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | -| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | -| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | | -| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | | -| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | -| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | | -| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | | -| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | -| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | -| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | -| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | +| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | +| -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ---- | +| 1 | `gcloud run services list --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | +| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | +| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | +| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | +| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | +| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | +| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | +| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | E | +| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | E | +| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | +| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | E | +| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | E | +| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | +| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | +| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | +| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | | +| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | + +**Row 1 note:** managed Cloud Run lists services across ALL regions when +`--region` is omitted — do not pass a `--region` flag (a `--region=-` form is +not documented). + +**Sizing caveat:** SQL `settings.dataDiskSizeGb` is PROVISIONED disk, not actual +data volume. Downstream database-migration tool selection must treat it as an +upper bound. (Follow-up: enrich with actual data size from monitoring metrics. +Also follow-up: live-only compute resources are not graviton-profiled in v1 — +`graviton_profile` entries come from the IaC path only.) **Scale guard:** if `assets.json` (or any capture) exceeds ~100 resources, write a throwaway extraction script to `$MIGRATION_DIR/_extract_live.py` that projects only @@ -187,7 +205,7 @@ Synthesize Terraform-style identity so downstream design-refs (keyed on | `dns.googleapis.com/ManagedZone` / row 13 | `google_dns_managed_zone` | | `spanner.googleapis.com/Instance` / row 14 | `google_spanner_instance` | | `firestore.googleapis.com/Database` / row 15 | `google_firestore_database` | -| `bigquery.googleapis.com/Dataset` | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | +| `bigquery.googleapis.com/Dataset` / row 17 | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | | `aiplatform.googleapis.com/Endpoint` / row 16 | `google_vertex_ai_endpoint` | | `aiplatform.googleapis.com/*` (other) | `google_vertex_ai_*` (matching suffix) | | Any other asset type | Do NOT guess a mapping. Count it in `live_metadata.unmapped_asset_types` and exclude from the inventory. | @@ -264,8 +282,9 @@ Terraform `type` + GCP resource name (live `name` vs the IaC resource's `config.name`, falling back to the address name component). Then: 1. **Matched:** keep the IaC entry (its address, classification, cluster, - depth). Overwrite `config` sizing/capacity fields with live values (live - reflects reality). Record every changed field in + depth). Overwrite `config` values where live disagrees — sizing, capacity, + versions, and images alike (live reflects reality). Live-only fields the IaC + never declared are enrichment, not conflicts. Record every OVERWRITTEN field in `live_metadata.drift.config_conflicts[]` as `{ "address", "field", "terraform_value", "live_value" }`. Set `source: "live+terraform"`. diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md index fd652cb5..4bce24ca 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md @@ -124,15 +124,18 @@ Runs AFTER 1a–1c sub-discoveries complete, so its IaC merge sees their output. with your consent)? This catches resources managed outside Terraform." On yes → Load `references/phases/discover/discover-live.md`. On no → continue (do not re-ask this run). -- Else if NO files were found by 1a–1c → offer live discovery as the primary - source instead of stopping: "No local GCP sources detected (Terraform, app - code, or billing exports) — but I can discover your project directly via your - authenticated gcloud CLI (read-only, with your consent). Proceed?" - On yes → Load `references/phases/discover/discover-live.md`. - On no (or gcloud unavailable / consent declined inside the sub-file) → STOP - and output: "No GCP sources detected. Provide at least one source type - (Terraform files, application code, or billing exports), or re-run and accept - live discovery." +- Else if NO Terraform was found (regardless of whether 1b/1c found app code or + billing files — those cannot produce an infrastructure inventory) → offer live + discovery as the primary infrastructure source: "No Terraform detected — I can + discover your project's infrastructure directly via your authenticated gcloud + CLI (read-only, with your consent). Proceed?" On yes → Load + `references/phases/discover/discover-live.md`. On no → continue with whatever + 1b/1c produced (billing-only design path remains the fallback). +- If, after the offer, NO sub-discovery produced or will produce any artifact + (nothing found by 1a–1c AND live was declined or unavailable) → STOP and + output: "No GCP sources detected. Provide at least one source type (Terraform + files, application code, or billing exports), or re-run and accept live + discovery." ## Step 2: Check Outputs From eb0116534726d1fd4e53c366ce611342ef42b1e7 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 14:30:04 -0700 Subject: [PATCH 05/11] =?UTF-8?q?fix(gcp-to-aws):=20real-project=20finding?= =?UTF-8?q?s=20=E2=80=94=20Run=20v1/v2=20alias,=20firewall=20mapping,=20SA?= =?UTF-8?q?=20naming=20Validated=20live=20discovery=20against=20a=20real?= =?UTF-8?q?=20GCP=20project=20(270=20CAI=20assets,=20fast=20path;=20all=20?= =?UTF-8?q?six=20applicable=20enrichment=20rows=20including=20bq=20ls;=20e?= =?UTF-8?q?nv-name-only=20projections=20held=20with=20real=20secret-adjace?= =?UTF-8?q?nt=20env=20names=20like=20DB=5FPASSWORD=20present=20as=20names?= =?UTF-8?q?=20only).=20Comparison=20against=20a=20prior=20IaC=20discovery?= =?UTF-8?q?=20run=20surfaced=20three=20spec=20bugs:=20-=20google=5Fcloud?= =?UTF-8?q?=5Frun=5Fservice=20(v1)=20vs=20google=5Fcloud=5Frun=5Fv2=5Fserv?= =?UTF-8?q?ice=20caused=20FALSE=20drift=20on=20the=20same=20service=20?= =?UTF-8?q?=E2=80=94=20Step=206=20now=20aliases=20v1/v2=20Run=20and=20Func?= =?UTF-8?q?tions=20type=20pairs=20when=20the=20GCP=20name=20matches=20-=20?= =?UTF-8?q?compute.googleapis.com/Firewall=20was=20unmapped,=20dropping=20?= =?UTF-8?q?what=20the=20IaC=20path=20classifies=20as=20network=5Fpath=20se?= =?UTF-8?q?condaries=20=E2=80=94=20mapping=20row=20added=20-=20Service=20a?= =?UTF-8?q?ccount=20'name'=20now=20mandated=20as=20the=20email=20local-par?= =?UTF-8?q?t,=20never=20the=20display=20name,=20or=20merge=20matching=20br?= =?UTF-8?q?eaks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../phases/discover/discover-live.md | 54 +++++++++++-------- 1 file changed, 31 insertions(+), 23 deletions(-) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index 54934428..71370548 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -187,28 +187,29 @@ Synthesize Terraform-style identity so downstream design-refs (keyed on **Asset/CLI type → Terraform type mapping:** -| Captured type | Terraform `type` | -| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------- | -| `run.googleapis.com/Service` / row 1 | `google_cloud_run_v2_service` | -| `sqladmin.googleapis.com/Instance` / row 2 | `google_sql_database_instance` | -| `container.googleapis.com/Cluster` / row 3 | `google_container_cluster` | -| row 4 with `environment: GEN_2` | `google_cloudfunctions2_function` | -| row 4 with `environment: GEN_1` (or unset) | `google_cloudfunctions_function` | -| `storage.googleapis.com/Bucket` / row 5 | `google_storage_bucket` | -| `pubsub.googleapis.com/Topic` / row 6 | `google_pubsub_topic` | -| `compute.googleapis.com/Instance` / row 7 | `google_compute_instance` | -| `compute.googleapis.com/Network` / row 8 | `google_compute_network` | -| `compute.googleapis.com/Subnetwork` / row 9 | `google_compute_subnetwork` | -| `redis.googleapis.com/Instance` / row 10 | `google_redis_instance` | -| `secretmanager.googleapis.com/Secret` / row 11 | `google_secret_manager_secret` | -| `iam.googleapis.com/ServiceAccount` / row 12 | `google_service_account` | -| `dns.googleapis.com/ManagedZone` / row 13 | `google_dns_managed_zone` | -| `spanner.googleapis.com/Instance` / row 14 | `google_spanner_instance` | -| `firestore.googleapis.com/Database` / row 15 | `google_firestore_database` | -| `bigquery.googleapis.com/Dataset` / row 17 | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | -| `aiplatform.googleapis.com/Endpoint` / row 16 | `google_vertex_ai_endpoint` | -| `aiplatform.googleapis.com/*` (other) | `google_vertex_ai_*` (matching suffix) | -| Any other asset type | Do NOT guess a mapping. Count it in `live_metadata.unmapped_asset_types` and exclude from the inventory. | +| Captured type | Terraform `type` | +| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `run.googleapis.com/Service` / row 1 | `google_cloud_run_v2_service` | +| `sqladmin.googleapis.com/Instance` / row 2 | `google_sql_database_instance` | +| `container.googleapis.com/Cluster` / row 3 | `google_container_cluster` | +| row 4 with `environment: GEN_2` | `google_cloudfunctions2_function` | +| row 4 with `environment: GEN_1` (or unset) | `google_cloudfunctions_function` | +| `storage.googleapis.com/Bucket` / row 5 | `google_storage_bucket` | +| `pubsub.googleapis.com/Topic` / row 6 | `google_pubsub_topic` | +| `compute.googleapis.com/Instance` / row 7 | `google_compute_instance` | +| `compute.googleapis.com/Network` / row 8 | `google_compute_network` | +| `compute.googleapis.com/Subnetwork` / row 9 | `google_compute_subnetwork` | +| `compute.googleapis.com/Firewall` | `google_compute_firewall` (SECONDARY, role network_path — parity with IaC classification) | +| `redis.googleapis.com/Instance` / row 10 | `google_redis_instance` | +| `secretmanager.googleapis.com/Secret` / row 11 | `google_secret_manager_secret` | +| `iam.googleapis.com/ServiceAccount` / row 12 | `google_service_account` — `name` MUST be the email local-part (e.g. `app-sa` from `app-sa@…`), NEVER the display name, or IaC merge matching breaks | +| `dns.googleapis.com/ManagedZone` / row 13 | `google_dns_managed_zone` | +| `spanner.googleapis.com/Instance` / row 14 | `google_spanner_instance` | +| `firestore.googleapis.com/Database` / row 15 | `google_firestore_database` | +| `bigquery.googleapis.com/Dataset` / row 17 | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | +| `aiplatform.googleapis.com/Endpoint` / row 16 | `google_vertex_ai_endpoint` | +| `aiplatform.googleapis.com/*` (other) | `google_vertex_ai_*` (matching suffix) | +| Any other asset type | Do NOT guess a mapping. Count it in `live_metadata.unmapped_asset_types` and exclude from the inventory. | **Classification:** apply `discover-iac.md` Step 3S rules — the Priority 1 PRIMARY types list, everything else SECONDARY with role inferred from type @@ -278,7 +279,14 @@ If `gcp-resource-inventory.json` does NOT already exist, skip to Step 7 (live is the sole source). Otherwise the IaC inventory + clusters are the BASE. Match live↔IaC entries by -Terraform `type` + GCP resource name (live `name` vs the IaC resource's +Terraform `type` + GCP resource name, treating these type pairs as EQUIVALENT +for matching (same underlying service; live always maps to the newer type): +`google_cloud_run_service` ≡ `google_cloud_run_v2_service`, and +`google_cloudfunctions_function` ≡ `google_cloudfunctions2_function` when the +GCP name matches. Without this aliasing, a v1-declared resource produces FALSE +drift (flagged both `not_found_live` and `unmanaged_by_terraform`). The merged +entry keeps the IaC address; a note in `config` (`"live_type"`) records the +newer live type. Match names as: live `name` vs the IaC resource's `config.name`, falling back to the address name component). Then: 1. **Matched:** keep the IaC entry (its address, classification, cluster, From 94e7875b785436a78b027fd61d0a9998b99f42c1 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 14:34:03 -0700 Subject: [PATCH 06/11] =?UTF-8?q?chore(gcp-to-aws):=20review=20nits=20?= =?UTF-8?q?=E2=80=94=20row=20order=2016/17,=20bq=20as=20E=20row,=20align?= =?UTF-8?q?=20SKILL.md=20live-offer=20blurb=20-=20Reorder=20capture=20tabl?= =?UTF-8?q?e=20so=20vertex=20(16)=20precedes=20bq=20(17);=20mapping-table?= =?UTF-8?q?=20row=20references=20unchanged=20-=20Mark=20bq=20ls=20as=20an?= =?UTF-8?q?=20E=20row=20so=20a=20CAI-success=20run=20still=20verifies=20Bi?= =?UTF-8?q?gQuery=20presence=20(specialist=20gate)=20even=20if=20thin=20se?= =?UTF-8?q?arch=20missed=20it=20-=20SKILL.md=20prerequisites=20blurb=20now?= =?UTF-8?q?=20matches=20discover.md=20Step=201d=20semantics:=20offer=20liv?= =?UTF-8?q?e=20whenever=20no=20Terraform=20is=20found,=20not=20only=20when?= =?UTF-8?q?=20no=20files=20exist=20-=20Deferred=20(follow-up):=20wiring=20?= =?UTF-8?q?check=5Fexpected=5Fdrift.py=20into=20the=20mise/CI=20test=20sui?= =?UTF-8?q?te?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md | 2 +- .../gcp-to-aws/references/phases/discover/discover-live.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md index 5142e555..b72ccc2c 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md @@ -73,7 +73,7 @@ User must provide at least one GCP source: - **Application code**: Source files with GCP SDK or AI framework imports - **Billing data**: GCP billing/cost/usage export files (CSV or JSON) -If no files are found, offer live discovery before stopping (see `discover.md` Step 1d). Live discovery covers infrastructure only — AI/agentic workload detection still requires application code. +If no Terraform is found (even when app code or billing files exist — they cannot produce an infrastructure inventory), offer live discovery per `discover.md` Step 1d; stop only when nothing will produce any artifact. Live discovery covers infrastructure only — AI/agentic workload detection still requires application code. --- diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index 71370548..bdd9c72d 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -137,8 +137,8 @@ in the manifest and continue — a missing service is normal, never a halt. | 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | | 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | | 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | -| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | | | 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | +| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | E | **Row 1 note:** managed Cloud Run lists services across ALL regions when `--region` is omitted — do not pass a `--region` flag (a `--region=-` form is From 84f270a8731c75106193e007906268a2ff0d482d Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 14:54:21 -0700 Subject: [PATCH 07/11] =?UTF-8?q?fix(gcp-to-aws):=20review=20round=203=20?= =?UTF-8?q?=E2=80=94=20firewall=20fallback=20row,=20connector=20edge=20fix?= =?UTF-8?q?,=20v1/v2=20alias=20lock=20-=20Row=2018:=20firewall-rules=20lis?= =?UTF-8?q?t=20(minimal=20projection)=20so=20firewalls=20survive=20the=20p?= =?UTF-8?q?er-service=20path,=20not=20CAI-only=20(same=20class=20as=20the?= =?UTF-8?q?=20earlier=20bq=20gap)=20-=20Row=2019=20+=20edge-rule=20rewrite?= =?UTF-8?q?:=20the=20vpc-access-connector=20annotation=20is=20a=20CONNECTO?= =?UTF-8?q?R=20id,=20not=20a=20network=20=E2=80=94=20resolve=20connector.n?= =?UTF-8?q?etwork=20via=20a=20read-only=20describe=20before=20emitting=20a?= =?UTF-8?q?=20network=5Fmembership=20edge;=20never=20fabricate=20one=20-?= =?UTF-8?q?=20Fixture=20now=20declares=20orders-api=20as=20LEGACY=20google?= =?UTF-8?q?=5Fcloud=5Frun=5Fservice=20(v1)=20while=20live=20maps=20to=20v2?= =?UTF-8?q?,=20locking=20the=20Step=206=20type-alias=20rule;=20expected-dr?= =?UTF-8?q?ift.json=20asserts=20one=20merged=20resource=20(IaC=20address,?= =?UTF-8?q?=20live=5Ftype,=20no=20false=20drift=20pair)=20-=20Asserter:=20?= =?UTF-8?q?recursive=20env-value=20payload=20detection=20(name+value/value?= =?UTF-8?q?From=20objects,=20raw=20env=20keys,=20env-name-as-key=20pattern?= =?UTF-8?q?s)=20-=20Discover=20summary=20surfaces=20unmapped=5Fasset=5Ftyp?= =?UTF-8?q?es=20count=20+=20top=20types=20-=20Preflight=20ADC=20fallback:?= =?UTF-8?q?=20probe=20projects=20describe=20before=20demanding=20gcloud=20?= =?UTF-8?q?auth=20login=20(GOOGLE=5FAPPLICATION=5FCREDENTIALS=20/=20impers?= =?UTF-8?q?onation=20setups)=20-=20SA=20merge=20matching=20clarified=20to?= =?UTF-8?q?=20config.account=5Fid=20(fresh-agent=20replay=20judgment=20cal?= =?UTF-8?q?l=20folded=20back=20in)=20Validated:=20fresh-agent=20replay=20o?= =?UTF-8?q?f=20updated=20scenario=20B=20=E2=80=94=20alias=20rule=20execute?= =?UTF-8?q?d=20as=20designed=20(one=20merged=20resource,=20live=5Ftype=20r?= =?UTF-8?q?ecorded,=20image=20conflict),=20asserter=20PASS,=20full=20build?= =?UTF-8?q?=20green.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../fixtures/gcp-live-capture/README.md | 4 +- .../gcp-live-capture/check_expected_drift.py | 18 +++++ .../gcp-live-capture/expected-drift.json | 6 +- .../workspace-terraform/main.tf | 20 +++-- .../phases/discover/discover-live.md | 73 +++++++++++-------- .../references/phases/discover/discover.md | 2 +- 6 files changed, 81 insertions(+), 42 deletions(-) diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md index 1333636a..18c33795 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -10,7 +10,9 @@ exercises the designed-for behaviors: annotation and a service account, driving the Step 4 edge-inference rules; its env list has NAMES ONLY (including secret-looking names like `STRIPE_SECRET_KEY` — if a value ever appears in output, the projection rule - broke). + broke). Terraform declares it as the LEGACY `google_cloud_run_service` (v1) + type while live maps to v2 — locking the Step 6 type-alias rule (one merged + resource, never a false not_found_live/unmanaged pair). - **orders-db** (Cloud SQL) — live tier `db-custom-2-8192` vs Terraform's `db-f1-micro`: the classic console-resize drift (Step 6 rule 1). - **cache** (Memorystore Redis), **web-frontend** (Cloud Run), two secrets, and diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py index 0046727c..8cee68df 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py @@ -124,6 +124,24 @@ def main() -> int: doc = json.dumps(inv) check("aws_" not in doc.lower().replace("aws-design", ""), "possible AWS naming in discover artifact") + # Secret hygiene: no env entry anywhere may carry a value payload, and env + # data must appear only as name lists (env_var_names), never env objects. + def walk(node, path="$"): + if isinstance(node, dict): + if "name" in node and ("value" in node or "valueFrom" in node): + check(False, f"env-like object with a value payload at {path}") + for k, v in node.items(): + if k == "env": + check(False, f"raw 'env' key at {path} — spec requires env_var_names (names only)") + walk(v, f"{path}.{k}") + elif isinstance(node, list): + for i, v in enumerate(node): + walk(v, f"{path}[{i}]") + + walk(inv) + for env_name in ("STRIPE_SECRET_KEY", "DATABASE_URL", "REDIS_URL"): + check(f'"{env_name}": ' not in doc, f"fixture env name {env_name} appears as a KEY (value paired) — names must be list items only") + if FAILS: print(f"FAIL ({len(FAILS)}):") for f in FAILS: diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json index 602245c3..c759d26d 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json @@ -11,10 +11,12 @@ "expected_config_conflict_fields": ["settings.tier"], "must_not_have": ["not_found_live", "unmanaged_by_terraform"] }, - "google_cloud_run_v2_service.orders_api": { + "google_cloud_run_service.orders_api": { "source": "live+terraform", "classification": "PRIMARY", - "note": "matched by GCP name 'orders-api'; live image tag v42 overrides tf v40 as a sizing/capacity-class override or lands in config_conflicts — either is acceptable, but silent loss of the live value is not" + "config_must_include": { "live_type": "google_cloud_run_v2_service" }, + "must_not_have": ["not_found_live", "unmanaged_by_terraform"], + "note": "TYPE-ALIAS LOCK: Terraform declares the LEGACY v1 type; live maps to v2. Step 6 aliasing must merge them as ONE resource keeping the IaC address — a not_found_live/unmanaged pair here is the exact real-project false-drift bug. Live image v42 overrides tf v40 (config conflict recorded)." }, "google_storage_bucket.assets": { "not_found_live": true, diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf index dc9105f5..b37fff3f 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf @@ -34,15 +34,23 @@ resource "google_sql_database_instance" "db" { } } -resource "google_cloud_run_v2_service" "orders_api" { +# Declared as the LEGACY v1 resource type while the live capture maps the same +# service to google_cloud_run_v2_service (the real-project bug). +# → Step 6 type-alias rule: must merge as ONE resource (live+terraform, IaC +# address kept, config.live_type = google_cloud_run_v2_service) — NEVER a +# not_found_live + unmanaged_by_terraform pair. Also carries image drift +# (v40 here vs v42 live). +resource "google_cloud_run_service" "orders_api" { name = "orders-api" location = "us-central1" template { - service_account = google_service_account.app.email - containers { - image = "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v40" - resources { - limits = { cpu = "2", memory = "1Gi" } + spec { + service_account_name = google_service_account.app.email + containers { + image = "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v40" + resources { + limits = { cpu = "2", memory = "1Gi" } + } } } } diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index bdd9c72d..ed2491d0 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -46,9 +46,14 @@ (https://cloud.google.com/sdk/docs/install) and tell me to continue, or skip live discovery." Wait. If skipped → exit cleanly. 2. **Authenticated:** run `gcloud auth list --filter=status:ACTIVE --format="value(account)"`. - - Empty → tell the user: "Your gcloud CLI has no active account. Run - `gcloud auth login` in your terminal — it needs a browser, so I can't run it - for you — then tell me to continue." Wait. If declined → exit cleanly. + - Empty → do NOT hand off yet: credentials may come from ADC + (`GOOGLE_APPLICATION_CREDENTIALS`) or service-account impersonation, which + `auth list` does not show. Probe read-only: + `gcloud projects describe "$GCP_PROJECT" --format="value(projectId)"`. + Probe succeeds → proceed (record `account: "adc"` in the manifest). + Probe fails → tell the user: "Your gcloud CLI has no usable credentials. + Run `gcloud auth login` in your terminal — it needs a browser, so I can't + run it for you — then tell me to continue." Wait. If declined → exit cleanly. 3. **Project:** run `gcloud config get-value project`. - Show the result and ask: "Discover project `[project-id]`? [Y] Yes / [N] Use a different project (type its ID)". Set `$GCP_PROJECT` accordingly. @@ -120,25 +125,27 @@ the project; the Step 2 scale guard handles large outputs.) "API not enabled" / permission errors: record the row as `failed` or `skipped` in the manifest and continue — a missing service is normal, never a halt. -| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | -| -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ---- | -| 1 | `gcloud run services list --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | -| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | -| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | -| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | -| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | -| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | -| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | -| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | E | -| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | E | -| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | -| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | E | -| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | E | -| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | -| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | -| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | -| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | -| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | E | +| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | +| -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---- | +| 1 | `gcloud run services list --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | +| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | +| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | +| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | +| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | +| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | +| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | +| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | E | +| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | E | +| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | +| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | E | +| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | E | +| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | +| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | +| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | +| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | +| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | E | +| 18 | `gcloud compute firewall-rules list --format="json(name, network, direction, priority)"` — deliberately minimal projection (no source ranges or target tags; the IaC path carries full rule config when Terraform exists) | `firewalls.json` | E | +| 19 | `gcloud compute networks vpc-access connectors describe --region= --format="json(name, network)"` — ONLY for each distinct `run.googleapis.com/vpc-access-connector` annotation value seen in row 1 output; resolves connector → VPC for edge inference | `connector-.json` | E | **Row 1 note:** managed Cloud Run lists services across ALL regions when `--region` is omitted — do not pass a `--region` flag (a `--region=-` form is @@ -226,15 +233,15 @@ captured, populate `ai_detection` exactly as `discover-iac.md` Step 2 would Live captures contain resolved values, which often beat HCL references. Build `edges[]` using ONLY these deterministic rules (evidence = the config field path): -| Config field (captured) | Edge | -| --------------------------------------------------------------- | ------------------------------------------------------------------- | -| Cloud Run annotation `run.googleapis.com/cloudsql-instances` | run service → SQL instance, `data_dependency` | -| Cloud Run annotation `run.googleapis.com/vpc-access-connector` | run service → network, `network_membership` | -| `spec.template.spec.serviceAccountName` / `serviceAccountEmail` | service account → workload, `serves` (populate the SA's `serves[]`) | -| SQL `settings.ipConfiguration.privateNetwork` | SQL instance → network, `network_membership` | -| GCE `networkInterfaces[].network` / GKE `network` | instance/cluster → network, `network_membership` | -| Subnet `network` | subnet → network, `network_membership` | -| Redis `authorizedNetwork` | redis → network, `network_membership` | +| Config field (captured) | Edge | +| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Cloud Run annotation `run.googleapis.com/cloudsql-instances` | run service → SQL instance, `data_dependency` | +| Cloud Run annotation `run.googleapis.com/vpc-access-connector` | run service → network, `network_membership` — ONLY via the row 19 describe: the annotation value is a CONNECTOR id, not a network; resolve `connector.network` first. If the describe failed/was skipped, record the connector id in the service's `config` and emit NO edge (never fabricate a network target) | +| `spec.template.spec.serviceAccountName` / `serviceAccountEmail` | service account → workload, `serves` (populate the SA's `serves[]`) | +| SQL `settings.ipConfiguration.privateNetwork` | SQL instance → network, `network_membership` | +| GCE `networkInterfaces[].network` / GKE `network` | instance/cluster → network, `network_membership` | +| Subnet `network` | subnet → network, `network_membership` | +| Redis `authorizedNetwork` | redis → network, `network_membership` | No other inference — do not guess relationships from names, labels, or env var names. @@ -287,7 +294,9 @@ GCP name matches. Without this aliasing, a v1-declared resource produces FALSE drift (flagged both `not_found_live` and `unmanaged_by_terraform`). The merged entry keeps the IaC address; a note in `config` (`"live_type"`) records the newer live type. Match names as: live `name` vs the IaC resource's -`config.name`, falling back to the address name component). Then: +`config.name` — for service accounts use `config.account_id` (the email +local-part IS the SA's GCP name; the address name component often differs) — +falling back to the address name component). Then: 1. **Matched:** keep the IaC entry (its address, classification, cluster, depth). Overwrite `config` values where live disagrees — sizing, capacity, diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md index 4bce24ca..438b4acf 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md @@ -192,7 +192,7 @@ Only after `HANDOFF_OK`. In the **same turn** as the output message below, use t Output to user — build message from whichever artifacts exist: - If `gcp-resource-inventory.json` exists: "Discovered X total resources across Y clusters." -- If live discovery ran: "Live discovery captured N resources from project [id]." Plus, when IaC also ran: "Drift check: A resources live but not in Terraform, B in Terraform but not live, C config conflicts (live values used)." +- If live discovery ran: "Live discovery captured N resources from project [id]." Plus, when IaC also ran: "Drift check: A resources live but not in Terraform, B in Terraform but not live, C config conflicts (live values used)." Plus, when `live_metadata.unmapped_asset_types` is non-empty: "Skipped M unmapped asset types (top: X, Y, Z) — full list in live_metadata." - If `ai-workload-profile.json` exists: "Detected AI workloads (source: [ai_source])." - If `billing-profile.json` exists: "Parsed billing data ($Z/month across N services)." From c2dce5b67f13e172ba4ea8657b74b922775d6e86 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 16:41:47 -0700 Subject: [PATCH 08/11] =?UTF-8?q?fix(gcp-to-aws):=20preflight=20order=20?= =?UTF-8?q?=E2=80=94=20resolve=20project=20before=20the=20ADC=20probe=20us?= =?UTF-8?q?es=20it=20The=20ADC=20fallback=20probe=20referenced=20$GCP=5FPR?= =?UTF-8?q?OJECT=20one=20step=20before=20the=20project=20was=20set.=20Proj?= =?UTF-8?q?ect=20resolution=20is=20a=20local=20config=20read=20plus=20user?= =?UTF-8?q?=20confirmation=20(no=20credentials=20needed),=20so=20it=20now?= =?UTF-8?q?=20runs=20as=20step=202,=20with=20the=20auth=20check=20+=20prob?= =?UTF-8?q?e=20as=20step=203.?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../references/phases/discover/discover-live.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index ed2491d0..bd1b1091 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -45,20 +45,21 @@ - Missing → tell the user: "The gcloud CLI isn't installed. Install it (https://cloud.google.com/sdk/docs/install) and tell me to continue, or skip live discovery." Wait. If skipped → exit cleanly. -2. **Authenticated:** run `gcloud auth list --filter=status:ACTIVE --format="value(account)"`. +2. **Project:** run `gcloud config get-value project` (a local config read — no + credentials needed, which is why this step precedes the auth check). + - Show the result and ask: "Discover project `[project-id]`? [Y] Yes / + [N] Use a different project (type its ID)". Set `$GCP_PROJECT` accordingly. + If the value is empty, ask the user to type the project ID. One project per + run — for multiple projects, run the migration once per project. +3. **Authenticated:** run `gcloud auth list --filter=status:ACTIVE --format="value(account)"`. - Empty → do NOT hand off yet: credentials may come from ADC (`GOOGLE_APPLICATION_CREDENTIALS`) or service-account impersonation, which - `auth list` does not show. Probe read-only: + `auth list` does not show. Probe read-only with the project just resolved: `gcloud projects describe "$GCP_PROJECT" --format="value(projectId)"`. Probe succeeds → proceed (record `account: "adc"` in the manifest). Probe fails → tell the user: "Your gcloud CLI has no usable credentials. Run `gcloud auth login` in your terminal — it needs a browser, so I can't run it for you — then tell me to continue." Wait. If declined → exit cleanly. -3. **Project:** run `gcloud config get-value project`. - - Show the result and ask: "Discover project `[project-id]`? [Y] Yes / - [N] Use a different project (type its ID)". Set `$GCP_PROJECT` accordingly. - If the value is empty, ask the user to type the project ID. One project per - run — for multiple projects, run the migration once per project. ## Step 1: Consent Gate From 666fa9312cef6733b5c54bd6b104a531e0e266cc Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 23:35:11 -0700 Subject: [PATCH 09/11] feat(gcp-to-aws): soft-ask to enable CAI when the fast path is disabled When asset search fails because Cloud Asset API is off (default for most startup projects), offer a user-driven enable with CLI + console how-to, retry once on yes, then fall back to per-service. Agent never mutates; permission-denied skips the ask. Record cai_enable_* in the manifest and live_metadata. Co-authored-by: Cursor --- .../fixtures/gcp-live-capture/README.md | 15 +++-- .../live-capture/manifest.json | 6 +- .../phases/discover/discover-live.md | 65 +++++++++++++++++-- .../references/shared/schema-discover-iac.md | 2 + 4 files changed, 75 insertions(+), 13 deletions(-) diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md index 18c33795..2531c95a 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -3,8 +3,9 @@ Canned `gcloud` CLI outputs for testing the `gcp-to-aws` live-discovery path (`references/phases/discover/discover-live.md`) without a GCP project. The synthetic project `acme-prod` uses **per-service fallback mode** (the manifest -records the Cloud Asset API fast path as `failed` — the common startup case) and -exercises the designed-for behaviors: +records the Cloud Asset API fast path as `failed` — the common startup case +after the user declines the enable soft-ask, or a permission error skips it) +and exercises the designed-for behaviors: - **orders-api** (Cloud Run) — carries the `run.googleapis.com/cloudsql-instances` annotation and a service account, driving the Step 4 edge-inference rules; its @@ -49,8 +50,14 @@ exercises the designed-for behaviors: assertion). **What a run must never produce** (either scenario): env var or secret values -anywhere; any mutating `gcloud` command; AWS service names in discover -artifacts; a halt caused by the failed asset-search/pubsub captures. +anywhere; any mutating `gcloud` command (including `gcloud services enable`); +AWS service names in discover artifacts; a halt caused by the failed +asset-search/pubsub captures. + +**Soft-ask note:** live capture (Step 2a) may offer a user-driven +`gcloud services enable cloudasset.googleapis.com` when CAI is disabled. Replay +skips Step 2 entirely (`manifest.json` already present), so the soft-ask does +not fire here — the fixture models the post-decline / per-service outcome. ## Regenerating / extending diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json index 6b357e6c..b37abd7e 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json @@ -4,12 +4,14 @@ "account": "founder@acme-demo.com", "project": "acme-prod", "method": "per_service", + "cai_enable_offered": true, + "cai_enable_accepted": false, "captures": [ { - "command": "gcloud asset search-all-resources --project=acme-prod --format=json", + "command": "gcloud asset search-all-resources --scope=projects/acme-prod --format=json", "file": "assets.json", "status": "failed", - "note": "Cloud Asset API has not been used in project acme-prod before or it is disabled" + "note": "Cloud Asset API has not been used in project acme-prod before or it is disabled (user declined enable soft-ask)" }, { "command": "gcloud run services list --project=acme-prod --region=- --format=json(...)", diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index bd1b1091..27bff6fd 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -118,13 +118,56 @@ the project; the Step 2 scale guard handles large outputs.) > rows below keep values out of the captures entirely. The same applies to > `search-all-resources --read-mask` with resource data. Do NOT "optimize" > this into a full-metadata dump. -- Failure (Cloud Asset API not enabled, or permission denied) → record - `method: "per_service"` and run every applicable table row. Do NOT try to - enable the API (that would be a mutation). +- Failure → classify the error, then branch: + + **API not enabled** (stderr matches `SERVICE_DISABLED`, "has not been used", + "is not enabled", or "API [has not been / is not] enabled" — the common + startup case; Cloud Asset API is off by default): + + Offer **once** (user-driven enable — never run `gcloud services enable` + yourself; that would be a mutation): + + ``` + ─── Cloud Asset Inventory not enabled ─── + + The Cloud Asset API is not enabled on [$GCP_PROJECT]. Enabling it gives a + fuller inventory in one call (including resources outside the per-service + list). I will not enable it for you. + + To turn it on, run this in your terminal: + + gcloud services enable cloudasset.googleapis.com --project="$GCP_PROJECT" + + Or in the console: APIs & Services → Library → search "Cloud Asset API" → + Enable. Propagation can take up to a minute. + + [Y] I've enabled it — retry Cloud Asset Inventory + [N] Continue with per-service fallback + ``` + + - **[Y]** → wait for the user, then re-run the asset-search command **once**. + Success → continue as the Success path above; record + `cai_enable_offered: true`, `cai_enable_accepted: true` in the manifest. + Still failing → tell the user briefly, then fall through to per-service + (same as [N]); record `cai_enable_accepted: true` and the retry failure + note. + - **[N]** / no response treated as decline → fall through to per-service; + record `cai_enable_offered: true`, `cai_enable_accepted: false`. + + **Permission denied / other errors** (403 without the disable signals above, + network errors, etc.): do **not** offer enable — enabling will not help. + Fall through to per-service immediately; record `cai_enable_offered: false`. + + **Per-service fallthrough:** record `method: "per_service"` and run every + applicable table row below. Always keep the failed asset-search entry in + `captures[]` with `status: "failed"` and the stderr summary in `note`. **2b. Capture Command Table.** Each row redirects to the named file. On "API not enabled" / permission errors: record the row as `failed` or `skipped` in the manifest and continue — a missing service is normal, never a halt. +(Unlike the CAI fast path, do **not** soft-ask to enable individual service +APIs — too many rows, and "service not deployed" vs "API disabled" is ambiguous +from list errors alone.) | # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | | -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---- | @@ -173,13 +216,17 @@ raw file directly. "account": "", "project": "<$GCP_PROJECT>", "method": "asset_search|per_service", + "cai_enable_offered": false, + "cai_enable_accepted": null, "captures": [ { "command": "", "file": "", "status": "ok|failed|skipped", "note": null } ] } ``` -Every attempted or deliberately skipped row gets an entry. +Every attempted or deliberately skipped row gets an entry. `cai_enable_offered` / +`cai_enable_accepted` record the Step 2a soft-ask (`accepted` is `true` / +`false` / `null` when never offered). ## Step 3: Map Captures to Inventory Resources @@ -339,13 +386,16 @@ write/update: "captured_at": "", "project": "<$GCP_PROJECT>", "method": "asset_search|per_service", + "cai_enable_offered": false, + "cai_enable_accepted": null, "capture_warnings": [""], "unmapped_asset_types": { "": 2 }, "drift": { "resources_live_only": 0, "resources_terraform_only": 0, "config_conflicts": [] } } ``` - (`drift` present only when Step 6 merged.) + Copy `cai_enable_offered` / `cai_enable_accepted` from the manifest. (`drift` + present only when Step 6 merged.) 2. `$MIGRATION_DIR/gcp-resource-clusters.json` — exact schema (merged or fresh). 3. Validate per `discover-iac.md` Step 7c (every resource in exactly one cluster, @@ -362,8 +412,9 @@ The parent `discover.md` owns the phase status update — do not touch | Error | Behavior | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | -| Asset search fails (API not enabled, 403) | Fall back to per-service rows; record in manifest | -| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt | +| Asset search fails (API not enabled) | Soft-ask once with enable how-to; on [Y] retry once; on [N]/retry-fail fall back to per-service (agent never enables) | +| Asset search fails (permission denied / other) | Fall back to per-service immediately — do not offer enable | +| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt (no per-row enable soft-ask) | | Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | | Capture file unparseable | Record warning, skip that file, continue | | Every capture failed | Exit with no output; tell the user which permissions are missing (`roles/viewer` covers all rows) | diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md index d78a538e..d59a1f02 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md @@ -128,6 +128,8 @@ Live gcloud discovery produces the same inventory/cluster schemas with these add "captured_at": "2026-07-20T18:20:00Z", "project": "acme-prod", "method": "asset_search", + "cai_enable_offered": false, + "cai_enable_accepted": null, "capture_warnings": [], "unmapped_asset_types": {}, "drift": { From 882def37a62a516ee88d2faaae3aa59b9b504d61 Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sat, 18 Jul 2026 23:36:08 -0700 Subject: [PATCH 10/11] docs(gcp-to-aws): CAI soft-ask covers IAM viewer + official enable docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enable alone is not enough — Cloud Asset Viewer (roles/cloudasset.viewer) is also required. Soft-ask now includes API enable, IAM guidance, and the view-assets docs link; permission-denied gets its own remediation offer. Co-authored-by: Cursor --- .../fixtures/gcp-live-capture/README.md | 9 +-- .../phases/discover/discover-live.md | 57 ++++++++++++++----- 2 files changed, 47 insertions(+), 19 deletions(-) diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md index 2531c95a..1d3430db 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -54,10 +54,11 @@ anywhere; any mutating `gcloud` command (including `gcloud services enable`); AWS service names in discover artifacts; a halt caused by the failed asset-search/pubsub captures. -**Soft-ask note:** live capture (Step 2a) may offer a user-driven -`gcloud services enable cloudasset.googleapis.com` when CAI is disabled. Replay -skips Step 2 entirely (`manifest.json` already present), so the soft-ask does -not fire here — the fixture models the post-decline / per-service outcome. +**Soft-ask note:** live capture (Step 2a) may offer user-driven remediation when +CAI fails — enable `cloudasset.googleapis.com` and/or grant +`roles/cloudasset.viewer` (see https://docs.cloud.google.com/asset-inventory/docs/view-assets). +Replay skips Step 2 entirely (`manifest.json` already present), so the soft-ask +does not fire here — the fixture models the post-decline / per-service outcome. ## Regenerating / extending diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index 27bff6fd..b1839cda 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -124,38 +124,64 @@ the project; the Step 2 scale guard handles large outputs.) "is not enabled", or "API [has not been / is not] enabled" — the common startup case; Cloud Asset API is off by default): - Offer **once** (user-driven enable — never run `gcloud services enable` - yourself; that would be a mutation): + Offer **once** (user-driven enable — never run `gcloud services enable` or + IAM mutations yourself): ``` ─── Cloud Asset Inventory not enabled ─── The Cloud Asset API is not enabled on [$GCP_PROJECT]. Enabling it gives a fuller inventory in one call (including resources outside the per-service - list). I will not enable it for you. + list). I will not change your project for you. - To turn it on, run this in your terminal: + 1. Enable the API (pick one): - gcloud services enable cloudasset.googleapis.com --project="$GCP_PROJECT" + gcloud services enable cloudasset.googleapis.com --project="$GCP_PROJECT" - Or in the console: APIs & Services → Library → search "Cloud Asset API" → - Enable. Propagation can take up to a minute. + Or console: APIs & Services → Library → search "Cloud Asset API" → Enable. + Propagation can take up to a minute. - [Y] I've enabled it — retry Cloud Asset Inventory + 2. IAM — your identity also needs Cloud Asset Viewer on the project + (`roles/cloudasset.viewer`). Owner/Editor usually already include enough + access; otherwise ask an admin to grant that role. Docs: + https://docs.cloud.google.com/asset-inventory/docs/view-assets + + [Y] I've enabled it (and have access) — retry Cloud Asset Inventory [N] Continue with per-service fallback ``` - **[Y]** → wait for the user, then re-run the asset-search command **once**. Success → continue as the Success path above; record `cai_enable_offered: true`, `cai_enable_accepted: true` in the manifest. - Still failing → tell the user briefly, then fall through to per-service - (same as [N]); record `cai_enable_accepted: true` and the retry failure - note. + Still failing → tell the user briefly (if 403/PERMISSION_DENIED, mention + `roles/cloudasset.viewer` again), then fall through to per-service (same + as [N]); record `cai_enable_accepted: true` and the retry failure note. - **[N]** / no response treated as decline → fall through to per-service; record `cai_enable_offered: true`, `cai_enable_accepted: false`. - **Permission denied / other errors** (403 without the disable signals above, - network errors, etc.): do **not** offer enable — enabling will not help. + **Permission denied** (403 / `PERMISSION_DENIED` without the disable signals + above — API may already be on, but the identity lacks Cloud Asset access): + + Offer **once** (IAM guidance only — never grant roles yourself): + + ``` + ─── Cloud Asset Inventory permission denied ─── + + The Cloud Asset API appears enabled, but this identity cannot search assets + on [$GCP_PROJECT]. Grant Cloud Asset Viewer (`roles/cloudasset.viewer`) on + the project (or a role that includes `cloudasset.assets.searchAllResources` / + list permissions), then retry. Docs: + https://docs.cloud.google.com/asset-inventory/docs/view-assets + + [Y] I've updated IAM — retry Cloud Asset Inventory + [N] Continue with per-service fallback + ``` + + Same [Y]/[N] recording rules as the enable soft-ask + (`cai_enable_offered` / `cai_enable_accepted` — here "enable" means "CAI + access remediation offered"). + + **Other errors** (network, unexpected failures): do **not** soft-ask. Fall through to per-service immediately; record `cai_enable_offered: false`. **Per-service fallthrough:** record `method: "per_service"` and run every @@ -412,8 +438,9 @@ The parent `discover.md` owns the phase status update — do not touch | Error | Behavior | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | | gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | -| Asset search fails (API not enabled) | Soft-ask once with enable how-to; on [Y] retry once; on [N]/retry-fail fall back to per-service (agent never enables) | -| Asset search fails (permission denied / other) | Fall back to per-service immediately — do not offer enable | +| Asset search fails (API not enabled) | Soft-ask once: enable API + `roles/cloudasset.viewer` how-to + docs link; on [Y] retry once; on [N]/retry-fail → per-service (agent never mutates) | +| Asset search fails (permission denied) | Soft-ask once: grant `roles/cloudasset.viewer` + docs link; on [Y] retry once; on [N]/retry-fail → per-service | +| Asset search fails (other errors) | Fall back to per-service immediately — do not soft-ask | | Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt (no per-row enable soft-ask) | | Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | | Capture file unparseable | Record warning, skip that file, continue | From ee0ebb7b025cd722841781af6f0b66dc27e29afc Mon Sep 17 00:00:00 2001 From: Logan Kleier Date: Sun, 19 Jul 2026 10:27:17 -0700 Subject: [PATCH 11/11] fix(gcp-to-aws): deterministic region walk for redis/vertex in per-service mode Review follow-up: rows 10/16 walked 'each region seen in the asset-search results or rows 1-9'. Fine in asset-search mode (project-wide coverage), but in per-service fallback mode the seen-region heuristic fails both ways: a Memorystore/Vertex instance in a region with no other footprint is silently missed on custom-mode VPCs, while auto-mode VPC subnets inflate the seen set to every region anyway. New whitelisted row 20 (gcloud compute regions list, names only) enumerates the walk in per-service mode; asset-search mode walks matching asset locations. If the regions list itself fails, fall back to the old heuristic WITH a capture warning naming the coverage limit. Fixtures model the walk (regions.json + empty-region redis captures) and drop the undocumented --region=- from the canned run.json command per the row-1 note. --- .../fixtures/gcp-live-capture/README.md | 5 +++ .../live-capture/manifest.json | 20 ++++++++- .../live-capture/redis-europe-west1.json | 1 + .../live-capture/redis-us-east1.json | 1 + .../live-capture/regions.json | 1 + .../phases/discover/discover-live.md | 41 ++++++++++++++----- 6 files changed, 57 insertions(+), 12 deletions(-) create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json create mode 100644 migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md index 1d3430db..8f2dcf0c 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -26,6 +26,11 @@ and exercises the designed-for behaviors: drift" negative case). - Empty `gke.json` / `functions.json` / `gce.json` — services with nothing deployed produce no entries and no errors. +- **Region walk** (`regions.json` + three `redis-.json` captures) — in + per-service mode the redis/vertex walk enumerates regions from + `gcloud compute regions list` (capture table row 20), not from regions seen + in other rows' output; only `us-central1` has an instance, the other two are + empty results, not errors. ## How to replay diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json index b37abd7e..367144a1 100644 --- a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json @@ -14,7 +14,7 @@ "note": "Cloud Asset API has not been used in project acme-prod before or it is disabled (user declined enable soft-ask)" }, { - "command": "gcloud run services list --project=acme-prod --region=- --format=json(...)", + "command": "gcloud run services list --project=acme-prod --format=json(...)", "file": "run.json", "status": "ok", "note": null @@ -67,12 +67,30 @@ "status": "ok", "note": null }, + { + "command": "gcloud compute regions list --project=acme-prod --format=json(name)", + "file": "regions.json", + "status": "ok", + "note": "per-service mode: enumerates the region walk for redis/vertex rows" + }, { "command": "gcloud redis instances list --project=acme-prod --region=us-central1 --format=json(...)", "file": "redis-us-central1.json", "status": "ok", "note": null }, + { + "command": "gcloud redis instances list --project=acme-prod --region=us-east1 --format=json(...)", + "file": "redis-us-east1.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud redis instances list --project=acme-prod --region=europe-west1 --format=json(...)", + "file": "redis-europe-west1.json", + "status": "ok", + "note": "empty result" + }, { "command": "gcloud secrets list --project=acme-prod --format=json(...)", "file": "secrets.json", diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json new file mode 100644 index 00000000..5b28d239 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json @@ -0,0 +1 @@ +[{ "name": "us-central1" }, { "name": "us-east1" }, { "name": "europe-west1" }] diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md index b1839cda..df072b0c 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -206,21 +206,40 @@ from list errors alone.) | 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | | 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | E | | 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | E | -| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | +| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | | 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | E | | 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | E | | 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | | 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | | 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | -| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | +| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | | 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | E | | 18 | `gcloud compute firewall-rules list --format="json(name, network, direction, priority)"` — deliberately minimal projection (no source ranges or target tags; the IaC path carries full rule config when Terraform exists) | `firewalls.json` | E | | 19 | `gcloud compute networks vpc-access connectors describe --region= --format="json(name, network)"` — ONLY for each distinct `run.googleapis.com/vpc-access-connector` annotation value seen in row 1 output; resolves connector → VPC for edge inference | `connector-.json` | E | +| 20 | `gcloud compute regions list --format="json(name)"` — per-service mode only: enumerates the region walk for rows 10 and 16 (region names only; one cheap call) | `regions.json` | | **Row 1 note:** managed Cloud Run lists services across ALL regions when `--region` is omitted — do not pass a `--region` flag (a `--region=-` form is not documented). +**Region-walk note (rows 10 and 16):** Redis and Vertex endpoint lists are +per-region, so their coverage is exactly the set of regions walked. Determine it +by mode: + +- **Asset-search mode:** walk the regions of the matching assets in + `assets.json` (`redis.googleapis.com/*` locations for row 10, + `aiplatform.googleapis.com/*` locations for row 16). Asset search is + project-wide, so no region can hide an instance from this walk. +- **Per-service mode:** walk EVERY region from row 20's `regions.json`. Do NOT + derive the walk from regions seen in other rows' output — that heuristic + fails in both directions (an instance in a region with no other footprint is + silently missed on custom-mode VPCs, while auto-mode VPC subnets inflate the + "seen" set to every region anyway, without the honesty of saying so). +- **Row 20 failed?** Fall back to the regions seen in rows 1–9 output, and + append to the manifest (→ `live_metadata.capture_warnings`): + `"regions list unavailable — redis/vertex walk limited to regions observed in + other captures; instances in other regions are not covered"`. + **Sizing caveat:** SQL `settings.dataDiskSizeGb` is PROVISIONED disk, not actual data volume. Downstream database-migration tool selection must treat it as an upper bound. (Follow-up: enrich with actual data size from monitoring metrics. @@ -435,16 +454,16 @@ The parent `discover.md` owns the phase status update — do not touch ## Error Handling -| Error | Behavior | -| -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -| gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | +| Error | Behavior | +| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | | Asset search fails (API not enabled) | Soft-ask once: enable API + `roles/cloudasset.viewer` how-to + docs link; on [Y] retry once; on [N]/retry-fail → per-service (agent never mutates) | -| Asset search fails (permission denied) | Soft-ask once: grant `roles/cloudasset.viewer` + docs link; on [Y] retry once; on [N]/retry-fail → per-service | -| Asset search fails (other errors) | Fall back to per-service immediately — do not soft-ask | -| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt (no per-row enable soft-ask) | -| Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | -| Capture file unparseable | Record warning, skip that file, continue | -| Every capture failed | Exit with no output; tell the user which permissions are missing (`roles/viewer` covers all rows) | +| Asset search fails (permission denied) | Soft-ask once: grant `roles/cloudasset.viewer` + docs link; on [Y] retry once; on [N]/retry-fail → per-service | +| Asset search fails (other errors) | Fall back to per-service immediately — do not soft-ask | +| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt (no per-row enable soft-ask) | +| Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | +| Capture file unparseable | Record warning, skip that file, continue | +| Every capture failed | Exit with no output; tell the user which permissions are missing (`roles/viewer` covers all rows) | **Key principle:** partial results are better than no results. Record what failed; never fabricate what wasn't captured.