diff --git a/.checkov.yaml b/.checkov.yaml index d67df79d..61120cd0 100644 --- a/.checkov.yaml +++ b/.checkov.yaml @@ -11,3 +11,8 @@ # their purpose. They are never deployed. skip-path: - migrate/plugins/migration-to-aws/skills/tf-best-practices/fixtures/terraform-policy + # Live-discovery replay fixture: deliberately models a startup's EXISTING, + # imperfect GCP infrastructure (the migration SOURCE, never deployed). Its + # drift vs the canned gcloud captures is the test — sanitizing it to satisfy + # Checkov would change the source-provider state the fixture exists to model. + - migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform diff --git a/migrate/README.md b/migrate/README.md index a21736a7..c4b29f97 100644 --- a/migrate/README.md +++ b/migrate/README.md @@ -92,6 +92,7 @@ After installation, just describe what you want to migrate: **GCP migrations:** - "Migrate my GCP infrastructure to AWS" +- "Discover my GCP project and estimate AWS costs" - "Move my Cloud Run services to Fargate" - "Migrate my OpenAI app to Amazon Bedrock" - "Estimate AWS costs for my GCP workload" @@ -127,6 +128,10 @@ GCP/Heroku migrations write a `.migration//` directory; agent-advisor w **Live Heroku discovery — how it works:** No Terraform or exports needed. If `heroku login` works in your terminal, just ask — the agent requests your consent, then inventories your account using read-only list/info CLI commands. It captures app names, dyno types, add-on plans and prices, domains, pipelines, and config var **key names only**. It never reads config var values, credentials, or your API token, and never runs a command that creates, changes, or deletes anything. If you also have `heroku_*` Terraform, the agent cross-checks it against your live account and reports drift. +**Live GCP discovery — how it works:** No Terraform or exports needed. If `gcloud auth login` works in your terminal, just ask — the agent confirms the target project and requests your consent, then inventories it using read-only list/describe commands. It captures resource names, types, regions, sizing, network topology, and env var **names only** — never env var values, secret values, database contents, or access tokens, and never a command that creates, changes, or deletes anything. If you also have Terraform, the agent cross-checks it against your live project and reports drift. (AI/agentic detection still needs your application code.) + +**Live Heroku discovery — how it works:** No Terraform or exports needed. If `heroku login` works in your terminal, just ask — the agent requests your consent, then inventories your account using read-only list/info CLI commands. It captures app names, dyno types, add-on plans and prices, domains, pipelines, and config var **key names only**. It never reads config var values, credentials, or your API token, and never runs a command that creates, changes, or deletes anything. If you also have `heroku_*` Terraform, the agent cross-checks it against your live account and reports drift. + ## What It Detects ### GCP → AWS @@ -216,8 +221,9 @@ GCP/Heroku migrations write a `.migration//` directory; agent-advisor w - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data -- **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) +- At least one input source: an authenticated `gcloud` or `heroku` CLI (live discovery), Terraform files, application code, or billing data +- **For GCP infrastructure migration:** an authenticated `gcloud` CLI (recommended — live, read-only discovery with your consent, with drift detection against any Terraform found) or Terraform files / billing exports +- **For GCP AI/agentic migration:** Application source code is required (billing/IaC/live discovery alone cannot detect agent architecture) - **For Heroku migration:** an authenticated Heroku CLI (recommended — live, read-only discovery with your consent) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone). When both are available, live data is authoritative for current state and Terraform drift is surfaced. - **For Vercel migration:** repo access with a locally-runnable `next build`, plus a Vercel API token, are both required — the assessment does not run on partial Tier 1 inputs. Vercel tokens can't be permission-scoped to read-only, so scope by resource instead (project-scoped when one project is in scope), pick a short expiration, and revoke after the assessment; the skill only ever issues read (GET) requests, enforced by its capture-step endpoint whitelist - **For agent-advisor:** `uv` (for deterministic scoring); application source code when deploying/migrating existing agents (an idea-only run needs no code) diff --git a/migrate/plugins/migration-to-aws/README.md b/migrate/plugins/migration-to-aws/README.md index 69fc281a..f95dd852 100644 --- a/migrate/plugins/migration-to-aws/README.md +++ b/migrate/plugins/migration-to-aws/README.md @@ -104,7 +104,7 @@ ln -s "$(pwd)" ~/.cursor/plugins/local/migration-to-aws ### Workflow -1. **Discover** — Scan Terraform files, application code, and/or billing data — or, for Heroku, inventory your account live via the authenticated Heroku CLI (read-only, consent-gated). Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. +1. **Discover** — Scan Terraform files, application code, and/or billing data — or inventory your GCP project or Heroku account live via the authenticated `gcloud`/`heroku` CLI (read-only, consent-gated, with drift detection against any Terraform found). Detects infrastructure resources, AI models, agentic frameworks, tools, and orchestration patterns. 2. **Clarify** — Ask targeted questions about migration preferences, AI priorities, agentic migration approach, database sizing, and timeline. 3. **Design** — Map source services to AWS equivalents. For AI workloads: select Bedrock models with honest pricing comparison. For agentic workloads: design AgentCore Harness config or Strands architecture. 4. **Estimate** — Calculate monthly AWS costs using real-time pricing data. Compare to current spend. @@ -221,10 +221,23 @@ See [skills/agent-advisor/SKILL.md](skills/agent-advisor/SKILL.md) for the full - Claude Code >=2.1.29, Codex (latest), or [Cursor >= 2.5](https://cursor.com/changelog/2-5) - AWS CLI configured with appropriate credentials -- At least one input source: an authenticated Heroku CLI (Heroku migrations), Terraform files, application code, or billing data -- **For GCP AI/agentic migration:** Application source code is required (billing/IaC alone cannot detect agent architecture) +- At least one input source: an authenticated `gcloud` or `heroku` CLI (live discovery), Terraform files, application code, or billing data +- **For GCP infrastructure migration:** an authenticated `gcloud` CLI (recommended — live, read-only discovery with your consent, with drift detection against any Terraform found) or Terraform files / billing exports +- **For GCP AI/agentic migration:** Application source code is required (billing/IaC/live discovery alone cannot detect agent architecture) - **For Heroku migration:** an authenticated Heroku CLI (recommended) or Terraform files with `heroku_*` resources (Procfile/app.json supplements but cannot stand alone) +### Live GCP discovery — how it works + +No Terraform or exports needed. If `gcloud auth login` works in your terminal, just +ask your agent to migrate ("Migrate my GCP infrastructure to AWS" or "Discover my +GCP project and estimate AWS costs"). The agent confirms the target project and asks +for your consent, then inventories it using read-only list/describe commands — it +captures resource names, types, regions, sizing, network topology, and env var +**names only**. It never reads env var values, secret values, database contents, or +access tokens, and never runs a command that creates, changes, or deletes anything. +If you also have Terraform, the agent cross-checks it against your live project and +reports drift. (AI/agentic workload detection still needs your application code.) + ### Live Heroku discovery — how it works No Terraform or exports needed. If `heroku login` works in your terminal, just ask diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md new file mode 100644 index 00000000..8f2dcf0c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/README.md @@ -0,0 +1,73 @@ +# GCP live-discovery fixtures (replay mode) + +Canned `gcloud` CLI outputs for testing the `gcp-to-aws` live-discovery path +(`references/phases/discover/discover-live.md`) without a GCP project. The +synthetic project `acme-prod` uses **per-service fallback mode** (the manifest +records the Cloud Asset API fast path as `failed` — the common startup case +after the user declines the enable soft-ask, or a permission error skips it) +and exercises the designed-for behaviors: + +- **orders-api** (Cloud Run) — carries the `run.googleapis.com/cloudsql-instances` + annotation and a service account, driving the Step 4 edge-inference rules; its + env list has NAMES ONLY (including secret-looking names like + `STRIPE_SECRET_KEY` — if a value ever appears in output, the projection rule + broke). Terraform declares it as the LEGACY `google_cloud_run_service` (v1) + type while live maps to v2 — locking the Step 6 type-alias rule (one merged + resource, never a false not_found_live/unmanaged pair). +- **orders-db** (Cloud SQL) — live tier `db-custom-2-8192` vs Terraform's + `db-f1-micro`: the classic console-resize drift (Step 6 rule 1). +- **cache** (Memorystore Redis), **web-frontend** (Cloud Run), two secrets, and + the `acme-prod-uploads` bucket — all absent from Terraform → click-ops drift + (`unmanaged_by_terraform`, Step 6 rule 2). +- **acme-prod-assets** bucket — in Terraform, absent live, buckets capture `ok` + → `not_found_live` (Step 6 rule 3). +- **events** (Pub/Sub topic) — in Terraform, and the pubsub capture FAILED → + must NOT be marked `not_found_live` (rule 3's "absence of evidence is not + drift" negative case). +- Empty `gke.json` / `functions.json` / `gce.json` — services with nothing + deployed produce no entries and no errors. +- **Region walk** (`regions.json` + three `redis-.json` captures) — in + per-service mode the redis/vertex walk enumerates regions from + `gcloud compute regions list` (capture table row 20), not from regions seen + in other rows' output; only `us-central1` has an instance, the other two are + empty results, not errors. + +## How to replay + +**Scenario A — live-only (no Terraform):** + +1. Create a scratch directory with NO `.tf` files, app code, or billing exports. +2. Create `.migration/0720-1820/` and copy `live-capture/` into it. +3. Invoke the gcp-to-aws skill ("migrate my GCP infrastructure to AWS"). +4. Discover should treat live as the primary source (Step 1d), the sub-file + parses captures instead of re-running gcloud, and the output is a + schema-valid `gcp-resource-inventory.json` + `gcp-resource-clusters.json` + with `clustering_mode: "simplified_live"`, `discovery_sources: ["live"]`, + and no `drift` key. + +**Scenario B — live + Terraform (drift exercise):** + +1. As above, but also copy `workspace-terraform/main.tf` into the scratch root. +2. `main.tf` is deliberately stale — each divergence is commented with the + Step 6 merge rule it exercises. +3. Check outputs against `expected-drift.json` — machine-checkable via + `python3 check_expected_drift.py ` (exits non-zero on any failed + assertion). + +**What a run must never produce** (either scenario): env var or secret values +anywhere; any mutating `gcloud` command (including `gcloud services enable`); +AWS service names in discover artifacts; a halt caused by the failed +asset-search/pubsub captures. + +**Soft-ask note:** live capture (Step 2a) may offer user-driven remediation when +CAI fails — enable `cloudasset.googleapis.com` and/or grant +`roles/cloudasset.viewer` (see https://docs.cloud.google.com/asset-inventory/docs/view-assets). +Replay skips Step 2 entirely (`manifest.json` already present), so the soft-ask +does not fire here — the fixture models the post-decline / per-service outcome. + +## Regenerating / extending + +Captures follow the exact projections in `discover-live.md` Step 2 (same +filenames). If you add a capture type: whitelist row first, then fixture, then +extend `expected-drift.json`. All project IDs, names, emails, and account +numbers are synthetic. diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py new file mode 100644 index 00000000..8cee68df --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/check_expected_drift.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Assert a Discover run's output against expected-drift.json (scenario B). + +Usage: + python3 check_expected_drift.py + +Where contains gcp-resource-inventory.json and +gcp-resource-clusters.json produced by a replay of this fixture's scenario B +(live-capture/ + workspace-terraform/main.tf). Exits 0 on PASS, 1 on FAIL with +one line per failed assertion. Stdlib only. +""" + +import json +import sys +from pathlib import Path + +FAILS: list[str] = [] + + +def check(cond: bool, msg: str) -> None: + if not cond: + FAILS.append(msg) + + +def get_conflict_fields(inv: dict) -> list[str]: + drift = inv.get("live_metadata", {}).get("drift", {}) + return [c.get("field", "") for c in drift.get("config_conflicts", [])] + + +def main() -> int: + if len(sys.argv) != 2: + print(__doc__) + return 2 + run_dir = Path(sys.argv[1]) + fixture_dir = Path(__file__).resolve().parent + + inv = json.loads((run_dir / "gcp-resource-inventory.json").read_text()) + clusters = json.loads((run_dir / "gcp-resource-clusters.json").read_text()) + exp = json.loads((fixture_dir / "expected-drift.json").read_text()) + + # Metadata + meta = inv["metadata"] + for s in exp["metadata"]["discovery_sources_must_include"]: + check(s in meta.get("discovery_sources", []), f"discovery_sources missing {s}") + check( + meta.get("clustering_mode") in exp["metadata"]["clustering_mode_one_of"], + f"clustering_mode {meta.get('clustering_mode')} not in {exp['metadata']['clustering_mode_one_of']}", + ) + + # Resources + res = {r["address"]: r for r in inv["resources"]} + for addr, e in exp["resources"].items(): + r = res.get(addr) + if r is None: + # Address synthesis can differ for live-only names; try name-suffix match + candidates = [v for k, v in res.items() if k.split(".")[0] == addr.split(".")[0] and addr.split(".")[1] in k] + if len(candidates) == 1: + r = candidates[0] + else: + check(False, f"missing resource {addr}") + continue + if "source" in e: + check(r.get("source") == e["source"], f"{addr} source={r.get('source')} want {e['source']}") + if "classification" in e: + check(r.get("classification") == e["classification"], f"{addr} classification") + if e.get("not_found_live"): + check(r.get("not_found_live") is True, f"{addr} not_found_live missing") + if e.get("unmanaged_by_terraform"): + check(r.get("unmanaged_by_terraform") is True, f"{addr} unmanaged_by_terraform missing") + for flag in e.get("must_not_have", []): + check(flag not in r or r.get(flag) is not True, f"{addr} must not have {flag}") + if "config_must_include" in e: + cfg_text = json.dumps(r.get("config", {})) + for v in e["config_must_include"].values(): + check(v in cfg_text, f"{addr} config missing value {v}") + + # Edges + all_edges = [edge for c in clusters["clusters"] for edge in c.get("edges", [])] + for ee in exp["edges_must_include"]: + found = any( + ee["from_contains"] in edge.get("from", "") + and ee["to_contains"] in edge.get("to", "") + and edge.get("relationship_type") == ee["relationship_type"] + and ( + "evidence_contains" not in ee + or ee["evidence_contains"] in json.dumps(edge.get("evidence", {})) + ) + for edge in all_edges + ) + check(found, f"edge missing: {ee['from_contains']} -> {ee['to_contains']} ({ee['relationship_type']})") + + # Cluster coverage + clustered = {a for c in clusters["clusters"] for a in c["primary_resources"] + c["secondary_resources"]} + check(set(res) == clustered, f"cluster coverage mismatch: {sorted(set(res) ^ clustered)}") + check( + any(c["creation_order_depth"] == 0 and "networking" in c["cluster_id"] for c in clusters["clusters"]), + "no networking cluster at depth 0", + ) + + # AI detection + check(inv["ai_detection"]["has_ai_workload"] is exp["ai_detection"]["has_ai_workload"], "ai_detection mismatch") + + # live_metadata + drift + lm = inv["live_metadata"] + check(lm.get("method") == exp["live_metadata"]["method"], "live method") + check(lm.get("project") == exp["live_metadata"]["project"], "live project") + warns = json.dumps(lm.get("capture_warnings", [])).lower() + for w in exp["live_metadata"]["capture_warnings_must_mention"]: + check(w in warns, f"capture_warnings missing mention of {w}") + drift = lm.get("drift", {}) + check( + drift.get("resources_terraform_only", 99) <= exp["live_metadata"]["drift"]["resources_terraform_only_max"], + f"resources_terraform_only={drift.get('resources_terraform_only')}", + ) + check( + drift.get("resources_live_only", 0) >= exp["live_metadata"]["drift"]["resources_live_only_min"], + f"resources_live_only={drift.get('resources_live_only')}", + ) + conflict_fields = get_conflict_fields(inv) + for f in exp["live_metadata"]["drift"]["config_conflict_fields_must_include"]: + check(f in conflict_fields, f"config_conflicts missing field {f}") + + # Safety: no fixture env names paired with values, no AWS names + doc = json.dumps(inv) + check("aws_" not in doc.lower().replace("aws-design", ""), "possible AWS naming in discover artifact") + + # Secret hygiene: no env entry anywhere may carry a value payload, and env + # data must appear only as name lists (env_var_names), never env objects. + def walk(node, path="$"): + if isinstance(node, dict): + if "name" in node and ("value" in node or "valueFrom" in node): + check(False, f"env-like object with a value payload at {path}") + for k, v in node.items(): + if k == "env": + check(False, f"raw 'env' key at {path} — spec requires env_var_names (names only)") + walk(v, f"{path}.{k}") + elif isinstance(node, list): + for i, v in enumerate(node): + walk(v, f"{path}[{i}]") + + walk(inv) + for env_name in ("STRIPE_SECRET_KEY", "DATABASE_URL", "REDIS_URL"): + check(f'"{env_name}": ' not in doc, f"fixture env name {env_name} appears as a KEY (value paired) — names must be list items only") + + if FAILS: + print(f"FAIL ({len(FAILS)}):") + for f in FAILS: + print(f" - {f}") + return 1 + print("PASS — expected-drift.json assertions hold") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json new file mode 100644 index 00000000..c759d26d --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/expected-drift.json @@ -0,0 +1,81 @@ +{ + "_comment": "Expected outcomes when BOTH workspace-terraform/ and live-capture/ are used (scenario B in README.md). Assertions against gcp-resource-inventory.json + gcp-resource-clusters.json, not a full expected document.", + "metadata": { + "discovery_sources_must_include": ["terraform", "live"], + "clustering_mode_one_of": ["simplified", "simplified_live"] + }, + "resources": { + "google_sql_database_instance.db": { + "source": "live+terraform", + "config_must_include": { "tier_or_settings.tier": "db-custom-2-8192" }, + "expected_config_conflict_fields": ["settings.tier"], + "must_not_have": ["not_found_live", "unmanaged_by_terraform"] + }, + "google_cloud_run_service.orders_api": { + "source": "live+terraform", + "classification": "PRIMARY", + "config_must_include": { "live_type": "google_cloud_run_v2_service" }, + "must_not_have": ["not_found_live", "unmanaged_by_terraform"], + "note": "TYPE-ALIAS LOCK: Terraform declares the LEGACY v1 type; live maps to v2. Step 6 aliasing must merge them as ONE resource keeping the IaC address — a not_found_live/unmanaged pair here is the exact real-project false-drift bug. Live image v42 overrides tf v40 (config conflict recorded)." + }, + "google_storage_bucket.assets": { + "not_found_live": true, + "note": "buckets capture succeeded and the bucket is absent live" + }, + "google_pubsub_topic.events": { + "must_not_have": ["not_found_live"], + "note": "pubsub capture FAILED — absence of evidence is not drift" + }, + "google_redis_instance.cache": { + "source": "live", + "unmanaged_by_terraform": true, + "classification": "PRIMARY" + }, + "google_cloud_run_v2_service.web_frontend": { + "source": "live", + "unmanaged_by_terraform": true, + "classification": "PRIMARY" + }, + "google_storage_bucket.acme_prod_uploads": { + "source": "live", + "unmanaged_by_terraform": true + }, + "google_compute_network.main": { + "must_not_have": ["not_found_live", "unmanaged_by_terraform"] + } + }, + "edges_must_include": [ + { + "from_contains": "orders_api", + "to_contains": "google_sql_database_instance", + "relationship_type": "data_dependency", + "evidence_contains": "cloudsql-instances" + }, + { + "from_contains": "google_sql_database_instance", + "to_contains": "google_compute_network", + "relationship_type": "network_membership" + } + ], + "clusters": { + "every_resource_in_exactly_one_cluster": true, + "networking_cluster_at_depth_0": true, + "unmanaged_primaries_have_clusters": ["google_redis_instance.cache", "google_cloud_run_v2_service.web_frontend"] + }, + "ai_detection": { "has_ai_workload": false, "confidence": 0 }, + "live_metadata": { + "method": "per_service", + "project": "acme-prod", + "capture_warnings_must_mention": ["pubsub", "asset"], + "drift": { + "resources_terraform_only_max": 1, + "resources_live_only_min": 4, + "config_conflict_fields_must_include": ["settings.tier"] + } + }, + "must_not_exist_anywhere": [ + "any env var VALUE (fixture env entries carry names only: DATABASE_URL, STRIPE_SECRET_KEY, etc.)", + "any secret VALUE or gcloud access token", + "AWS service names (discover scope boundary)" + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json new file mode 100644 index 00000000..fca24c82 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/buckets.json @@ -0,0 +1,10 @@ +[ + { + "name": "acme-prod-uploads", + "location": "US-CENTRAL1", + "storageClass": "STANDARD", + "timeCreated": "2024-06-01T10:00:00Z", + "iamConfiguration": { "uniformBucketLevelAccess": { "enabled": true } }, + "versioning": { "enabled": true } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/functions.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gce.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/gke.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json new file mode 100644 index 00000000..367144a1 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/manifest.json @@ -0,0 +1,131 @@ +{ + "captured_at": "2026-07-20T18:20:00Z", + "gcloud_version": "Google Cloud SDK 532.0.0", + "account": "founder@acme-demo.com", + "project": "acme-prod", + "method": "per_service", + "cai_enable_offered": true, + "cai_enable_accepted": false, + "captures": [ + { + "command": "gcloud asset search-all-resources --scope=projects/acme-prod --format=json", + "file": "assets.json", + "status": "failed", + "note": "Cloud Asset API has not been used in project acme-prod before or it is disabled (user declined enable soft-ask)" + }, + { + "command": "gcloud run services list --project=acme-prod --format=json(...)", + "file": "run.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud sql instances list --project=acme-prod --format=json(...)", + "file": "sql.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud container clusters list --project=acme-prod --format=json(...)", + "file": "gke.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud functions list --project=acme-prod --format=json(...)", + "file": "functions.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud storage buckets list --project=acme-prod --format=json(...)", + "file": "buckets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud pubsub topics list --project=acme-prod --format=json(...)", + "file": "pubsub.json", + "status": "failed", + "note": "Cloud Pub/Sub API has not been used in project acme-prod before or it is disabled" + }, + { + "command": "gcloud compute instances list --project=acme-prod --format=json(...)", + "file": "gce.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud compute networks list --project=acme-prod --format=json(...)", + "file": "networks.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud compute networks subnets list --project=acme-prod --format=json(...)", + "file": "subnets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud compute regions list --project=acme-prod --format=json(name)", + "file": "regions.json", + "status": "ok", + "note": "per-service mode: enumerates the region walk for redis/vertex rows" + }, + { + "command": "gcloud redis instances list --project=acme-prod --region=us-central1 --format=json(...)", + "file": "redis-us-central1.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud redis instances list --project=acme-prod --region=us-east1 --format=json(...)", + "file": "redis-us-east1.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud redis instances list --project=acme-prod --region=europe-west1 --format=json(...)", + "file": "redis-europe-west1.json", + "status": "ok", + "note": "empty result" + }, + { + "command": "gcloud secrets list --project=acme-prod --format=json(...)", + "file": "secrets.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud iam service-accounts list --project=acme-prod --format=json(...)", + "file": "sa.json", + "status": "ok", + "note": null + }, + { + "command": "gcloud dns managed-zones list --project=acme-prod --format=json(...)", + "file": "dns.json", + "status": "skipped", + "note": "Cloud DNS API not enabled" + }, + { + "command": "gcloud spanner instances list --project=acme-prod --format=json(...)", + "file": "spanner.json", + "status": "skipped", + "note": "Spanner API not enabled" + }, + { + "command": "gcloud firestore databases list --project=acme-prod --format=json(...)", + "file": "firestore.json", + "status": "skipped", + "note": "Firestore API not enabled" + }, + { + "command": "gcloud ai endpoints list --project=acme-prod --region=us-central1 --format=json(...)", + "file": "vertex-us-central1.json", + "status": "skipped", + "note": "no aiplatform signal; Vertex AI API not enabled" + } + ] +} diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json new file mode 100644 index 00000000..d77f22d8 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/networks.json @@ -0,0 +1,7 @@ +[ + { + "name": "main", + "autoCreateSubnetworks": false, + "subnetworks": ["https://www.googleapis.com/compute/v1/projects/acme-prod/regions/us-central1/subnetworks/app"] + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-europe-west1.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json new file mode 100644 index 00000000..fe3fb55c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-central1.json @@ -0,0 +1,10 @@ +[ + { + "name": "projects/acme-prod/locations/us-central1/instances/cache", + "tier": "BASIC", + "memorySizeGb": 1, + "redisVersion": "REDIS_7_2", + "authorizedNetwork": "projects/acme-prod/global/networks/main", + "locationId": "us-central1-a" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json new file mode 100644 index 00000000..fe51488c --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/redis-us-east1.json @@ -0,0 +1 @@ +[] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json new file mode 100644 index 00000000..5b28d239 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/regions.json @@ -0,0 +1 @@ +[{ "name": "us-central1" }, { "name": "us-east1" }, { "name": "europe-west1" }] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json new file mode 100644 index 00000000..61ad66ac --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/run.json @@ -0,0 +1,68 @@ +[ + { + "metadata": { + "name": "orders-api", + "labels": { + "cloud.googleapis.com/location": "us-central1" + }, + "annotations": { + "serving.knative.dev/creator": "founder@acme-demo.com" + } + }, + "spec": { + "template": { + "metadata": { + "annotations": { + "run.googleapis.com/cloudsql-instances": "acme-prod:us-central1:orders-db", + "autoscaling.knative.dev/maxScale": "10" + } + }, + "spec": { + "serviceAccountName": "app-sa@acme-prod.iam.gserviceaccount.com", + "containerConcurrency": 80, + "timeoutSeconds": 300, + "containers": [ + { + "image": "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v42", + "resources": { "limits": { "cpu": "2", "memory": "1Gi" } }, + "env": [ + { "name": "DATABASE_URL" }, + { "name": "REDIS_URL" }, + { "name": "STRIPE_SECRET_KEY" }, + { "name": "NODE_ENV" } + ] + } + ] + } + } + }, + "status": { "url": "https://orders-api-h4x2k3q-uc.a.run.app" } + }, + { + "metadata": { + "name": "web-frontend", + "labels": { + "cloud.googleapis.com/location": "us-central1" + }, + "annotations": {} + }, + "spec": { + "template": { + "metadata": { "annotations": { "autoscaling.knative.dev/maxScale": "4" } }, + "spec": { + "serviceAccountName": "app-sa@acme-prod.iam.gserviceaccount.com", + "containerConcurrency": 200, + "timeoutSeconds": 60, + "containers": [ + { + "image": "us-central1-docker.pkg.dev/acme-prod/apps/web-frontend:v17", + "resources": { "limits": { "cpu": "1", "memory": "512Mi" } }, + "env": [{ "name": "API_BASE_URL" }, { "name": "NODE_ENV" }] + } + ] + } + } + }, + "status": { "url": "https://web-frontend-h4x2k3q-uc.a.run.app" } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json new file mode 100644 index 00000000..9c077f78 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sa.json @@ -0,0 +1,3 @@ +[ + { "email": "app-sa@acme-prod.iam.gserviceaccount.com", "displayName": "App runtime SA", "disabled": false } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json new file mode 100644 index 00000000..21874653 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/secrets.json @@ -0,0 +1,12 @@ +[ + { + "name": "projects/123456789012/secrets/database-password", + "replication": { "automatic": {} }, + "createTime": "2024-06-01T10:05:00Z" + }, + { + "name": "projects/123456789012/secrets/stripe-api-key", + "replication": { "automatic": {} }, + "createTime": "2024-06-01T10:06:00Z" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json new file mode 100644 index 00000000..068182a7 --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/sql.json @@ -0,0 +1,17 @@ +[ + { + "name": "orders-db", + "region": "us-central1", + "databaseVersion": "POSTGRES_16", + "settings": { + "tier": "db-custom-2-8192", + "availabilityType": "ZONAL", + "dataDiskSizeGb": "50", + "ipConfiguration": { + "privateNetwork": "projects/acme-prod/global/networks/main", + "ipv4Enabled": false + }, + "backupConfiguration": { "enabled": true } + } + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json new file mode 100644 index 00000000..18d8148a --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/live-capture/subnets.json @@ -0,0 +1,8 @@ +[ + { + "name": "app", + "region": "https://www.googleapis.com/compute/v1/projects/acme-prod/regions/us-central1", + "network": "https://www.googleapis.com/compute/v1/projects/acme-prod/global/networks/main", + "ipCidrRange": "10.0.0.0/24" + } +] diff --git a/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf new file mode 100644 index 00000000..b37fff3f --- /dev/null +++ b/migrate/plugins/migration-to-aws/fixtures/gcp-live-capture/workspace-terraform/main.tf @@ -0,0 +1,77 @@ +# Fixture Terraform — deliberately STALE relative to the live captures in +# ../live-capture/. Each divergence exercises a specific rule in +# discover-live.md Step 6 (Merge with IaC Discovery). Do not "fix" the drift. + +resource "google_compute_network" "main" { + name = "main" + auto_create_subnetworks = false +} + +resource "google_compute_subnetwork" "app" { + name = "app" + region = "us-central1" + network = google_compute_network.main.id + ip_cidr_range = "10.0.0.0/24" +} + +resource "google_service_account" "app" { + account_id = "app-sa" + display_name = "App runtime SA" +} + +# Drift: live tier is db-custom-2-8192 (someone scaled up in the console). +# → Step 6 rule 1 (config conflict on settings.tier, live value wins) +resource "google_sql_database_instance" "db" { + name = "orders-db" + region = "us-central1" + database_version = "POSTGRES_16" + settings { + tier = "db-f1-micro" + ip_configuration { + private_network = google_compute_network.main.id + ipv4_enabled = false + } + } +} + +# Declared as the LEGACY v1 resource type while the live capture maps the same +# service to google_cloud_run_v2_service (the real-project bug). +# → Step 6 type-alias rule: must merge as ONE resource (live+terraform, IaC +# address kept, config.live_type = google_cloud_run_v2_service) — NEVER a +# not_found_live + unmanaged_by_terraform pair. Also carries image drift +# (v40 here vs v42 live). +resource "google_cloud_run_service" "orders_api" { + name = "orders-api" + location = "us-central1" + template { + spec { + service_account_name = google_service_account.app.email + containers { + image = "us-central1-docker.pkg.dev/acme-prod/apps/orders-api:v40" + resources { + limits = { cpu = "2", memory = "1Gi" } + } + } + } + } +} + +# Declared but NOT deployed (absent from live buckets.json; buckets capture ok). +# → Step 6 rule 3 (not_found_live: true) +resource "google_storage_bucket" "assets" { + name = "acme-prod-assets" + location = "US-CENTRAL1" +} + +# Declared, and the pubsub live capture FAILED (API-not-enabled in manifest). +# → Step 6 rule 3 negative case: must NOT be marked not_found_live +# (absence of evidence is not drift). +resource "google_pubsub_topic" "events" { + name = "events" +} + +# NOTE deliberate absences from this file (present in live captures): +# - google_redis_instance "cache" → unmanaged_by_terraform (click-ops) +# - Cloud Run service "web-frontend" → unmanaged_by_terraform +# - Secret Manager secrets (2) → unmanaged_by_terraform +# - Storage bucket "acme-prod-uploads" → unmanaged_by_terraform diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md index 67036256..204ab66b 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/SKILL.md @@ -10,7 +10,7 @@ description: "Migrate workloads from Google Cloud Platform to AWS — including - **Re-platform by default**: Select AWS services that match GCP workload types (e.g., Cloud Run → Fargate, Cloud SQL → RDS). - **Dev sizing unless specified**: Default to development-tier capacity (e.g., db.t4g.micro, single AZ). Upgrade only on user direction. - **No human one-time migration costs**: Do not present human labor, professional services, or people-time work as dollar estimates or "one-time migration cost" budget categories. Vendor charges grounded in data (for example GCP data transfer egress in the infra estimate when billing exists) are allowed. -- **Multi-signal approach**: Design phase adapts based on available inputs — Terraform IaC for infrastructure, billing data for service mapping, and app code for AI workload detection. +- **Multi-signal approach**: Design phase adapts based on available inputs — live gcloud discovery and/or Terraform IaC for infrastructure, billing data for service mapping, and app code for AI workload detection. When live and IaC both run, live is authoritative for current state and disagreements surface as drift, never silently resolved. - **BigQuery / `google_bigquery_*`**: The skill **does not** recommend a specific AWS analytics or warehouse service. During **Clarify**, if discovery shows BigQuery (IaC `google_bigquery_*` and/or billing rows for BigQuery), you **must** surface the specialist advisory **before** Design (see `references/phases/clarify/clarify.md`). Design output uses **`Deferred — specialist engagement`**; keep directing the user to their **AWS account team** and/or a **data analytics migration partner** through Design, Estimate, and docs (see `references/phases/design/design-infra.md` BigQuery specialist gate). --- @@ -68,11 +68,12 @@ If the user chooses to continue, proceed with the combined run. Load AI refs **a User must provide at least one GCP source: +- **Live gcloud CLI** (recommended for infrastructure): an authenticated `gcloud` CLI — read-only, consent-gated live discovery of the project (see `references/phases/discover/discover-live.md`) - **Terraform IaC**: `.tf` files (with optional `.tfvars`, `.tfstate`) - **Application code**: Source files with GCP SDK or AI framework imports - **Billing data**: GCP billing/cost/usage export files (CSV or JSON) -If none of the above are found, stop and ask user to provide at least one source type. +If no Terraform is found (even when app code or billing files exist — they cannot produce an infrastructure inventory), offer live discovery per `discover.md` Step 1d; stop only when nothing will produce any artifact. Live discovery covers infrastructure only — AI/agentic workload detection still requires application code. --- @@ -235,6 +236,7 @@ gcp-to-aws/ │ │ ├── discover/ │ │ │ ├── discover.md # Phase 1: Discover orchestrator │ │ │ ├── discover-iac.md # Terraform/IaC discovery +│ │ │ ├── discover-live.md # Live gcloud CLI discovery (read-only, consent-gated) │ │ │ ├── discover-app-code.md # App code discovery │ │ │ └── discover-billing.md # Billing data discovery │ │ ├── clarify/ @@ -307,13 +309,13 @@ gcp-to-aws/ │ └── bedrock-quotas.md # Bedrock TPM/RPM quota awareness, burndown rates, capacity planning ``` -| Condition | Action | -| ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| No GCP sources found (no `.tf`, no app code, no billing data) | Stop. Output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports) and try again." | -| `.phase-status.json` missing phase gate | Stop. Output: "Cannot enter Phase X: Phase Y-1 not completed. Start from Phase Y or resume Phase Y-1." | -| awspricing unavailable after 3 attempts | Display user warning about ±5-25% accuracy. Use `pricing-cache.md`. Add `pricing_source: "cached_fallback"` to the applicable `estimation-*.json` file. | -| User skips questions or says "use defaults for the rest" | Apply documented defaults for all remaining questions (essential questions and any unconfirmed sheet rows in wizard mode; current and subsequent batches in full mode). Q2/Q3 defaults add a report caveat. Phase 2 completes either way. | -| `aws-design.json` missing required clusters | Stop Phase 4. Output: "Re-run Phase 3 to generate missing cluster designs." | +| Condition | Action | +| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| No GCP sources found (no `.tf`, no app code, no billing data) | Offer live gcloud discovery per `discover.md` Step 1d. Only if declined or unavailable: Stop. Output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports), or re-run and accept live discovery." | +| `.phase-status.json` missing phase gate | Stop. Output: "Cannot enter Phase X: Phase Y-1 not completed. Start from Phase Y or resume Phase Y-1." | +| awspricing unavailable after 3 attempts | Display user warning about ±5-25% accuracy. Use `pricing-cache.md`. Add `pricing_source: "cached_fallback"` to the applicable `estimation-*.json` file. | +| User skips questions or says "use defaults for the rest" | Apply documented defaults for all remaining questions (essential questions and any unconfirmed sheet rows in wizard mode; current and subsequent batches in full mode). Q2/Q3 defaults add a report caveat. Phase 2 completes either way. | +| `aws-design.json` missing required clusters | Stop Phase 4. Output: "Re-run Phase 3 to generate missing cluster designs." | ## Defaults @@ -383,6 +385,7 @@ User can invoke the skill again to resume from `current_phase` (or deterministic **v1.0 includes:** - Terraform infrastructure discovery +- Live infrastructure discovery via authenticated gcloud CLI (read-only, consent-gated, with IaC drift detection) - App code scanning (AI workload detection) - Billing data import from GCP - User requirement clarification (assumption-sheet wizard by default: confirm detected/assumed values, answer only essential questions; full adaptive question flow available on request) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md new file mode 100644 index 00000000..df072b0c --- /dev/null +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover-live.md @@ -0,0 +1,482 @@ +# Discover Phase: Live Discovery (gcloud CLI) + +> Self-contained live-discovery sub-file. Inventories the user's GCP project +> directly through their authenticated `gcloud` CLI — read-only, consent-gated, +> env-var names only. Produces the SAME artifacts as `discover-iac.md` +> (`gcp-resource-inventory.json` + `gcp-resource-clusters.json`, simplified +> clustering mode), so all downstream phases work identically. When IaC discovery +> also ran, merges live findings into the existing inventory and surfaces drift. +> If the user declines consent or `gcloud` is unavailable, exits cleanly with no +> output. + +**Execute ALL steps in order. Do not skip or optimize.** + +--- + +## Security Contract (applies to every step) + +1. **Exact-command whitelist.** Run ONLY commands that appear in Step 0 (preflight) + or the Step 2 Capture Command Table. Never any mutating verb (`create`, `update`, + `delete`, `set`, `add`, `remove`, `deploy`, `apply`, `import`, `patch`), never + `gcloud auth login` (interactive — hand off to the user), never + `gcloud auth print-access-token` or `print-identity-token` (prints credentials). +2. **Never capture secret values.** Every capture command uses an explicit + `--format="json(...)"` field projection. Projections include env var **names** + but never env var **values**, and never GCE instance `metadata.items` values. + Additionally, apply `discover-iac.md`'s sensitive-key redaction patterns + (`password`, `secret`, `api_key`, `access_key`, `private_key`, `client_secret`, + `token`, `credential`, `auth` — case-insensitive) to any config field before it + is written into an artifact: replace matched values with `"[REDACTED]"`. +3. **Always explicit scope.** Every command passes `--project="$GCP_PROJECT"` + explicitly. Never rely on the active gcloud config inside capture commands. +4. **Capture to files, not context.** Redirect stdout to files under + `$MIGRATION_DIR/live-capture/`. Process any capture file larger than ~100 + resources with a throwaway extraction script (same pattern as `discover.md`'s + lightweight billing extraction) — do NOT Read large raw captures into context. +5. **Consent first.** No `gcloud` command from the Step 2 table runs before the + user answers `[A]` in Step 1. Preflight commands in Step 0 are limited to + version/auth/config checks that touch no project data. + +--- + +## Step 0: Preflight + +1. **CLI installed:** run `gcloud --version` (first line only). + - Missing → tell the user: "The gcloud CLI isn't installed. Install it + (https://cloud.google.com/sdk/docs/install) and tell me to continue, or skip + live discovery." Wait. If skipped → exit cleanly. +2. **Project:** run `gcloud config get-value project` (a local config read — no + credentials needed, which is why this step precedes the auth check). + - Show the result and ask: "Discover project `[project-id]`? [Y] Yes / + [N] Use a different project (type its ID)". Set `$GCP_PROJECT` accordingly. + If the value is empty, ask the user to type the project ID. One project per + run — for multiple projects, run the migration once per project. +3. **Authenticated:** run `gcloud auth list --filter=status:ACTIVE --format="value(account)"`. + - Empty → do NOT hand off yet: credentials may come from ADC + (`GOOGLE_APPLICATION_CREDENTIALS`) or service-account impersonation, which + `auth list` does not show. Probe read-only with the project just resolved: + `gcloud projects describe "$GCP_PROJECT" --format="value(projectId)"`. + Probe succeeds → proceed (record `account: "adc"` in the manifest). + Probe fails → tell the user: "Your gcloud CLI has no usable credentials. + Run `gcloud auth login` in your terminal — it needs a browser, so I can't + run it for you — then tell me to continue." Wait. If declined → exit cleanly. + +## Step 1: Consent Gate + +Output exactly, then wait for the user's choice: + +``` +─── Live GCP Discovery (read-only) ─── + +I can inventory project [$GCP_PROJECT] directly using your +authenticated gcloud CLI. This runs LIST/DESCRIBE commands only: + + ✓ Captured: resource names, types, regions, machine/instance + sizing, container images, network topology, env var NAMES, + secret NAMES, and labels. + ✗ Never captured: env var values, secret values, database + contents, instance metadata values, access tokens, or source + code. No command that creates, changes, or deletes anything + will run. + +Output is written to .migration//live-capture/ (gitignored). + +[A] Proceed with live discovery +[B] Skip — use workspace files only +``` + +- **[A]** → continue to Step 2. +- **[B]** → exit cleanly with no output (record the decline for the orchestrator). + +## Step 2: Capture + +Create `$MIGRATION_DIR/live-capture/`. + +**2a. Fast path — Cloud Asset Inventory (one call, whole project):** + +``` +gcloud asset search-all-resources --scope="projects/$GCP_PROJECT" \ + --format=json > $MIGRATION_DIR/live-capture/assets.json +``` + +(`--scope` is the documented scoping flag for asset search — do not rely on the +active project. No `--asset-types` filter on purpose: unfiltered results feed +`live_metadata.unmapped_asset_types`, which tells the user what ELSE lives in +the project; the Step 2 scale guard handles large outputs.) + +- Success → record `method: "asset_search"` in the manifest, then run only the + **enrichment rows** (marked E) of the table below for asset types that were + found (asset search returns names/types/locations but thin config). The cheap + networking/secrets/identity lists (rows 8, 9, 11, 12) are E rows precisely so + edge inference and name inventories keep full fidelity on this path. + + > **Why not `gcloud asset list --content-type=resource` (full metadata, one + > call)?** Deliberate. Full `resource.data` includes env var VALUES (Cloud + > Run, Functions) and instance metadata values — writing it to + > `live-capture/` would put secret material on disk and break this file's + > "values never captured" contract. Thin search + the projected enrichment + > rows below keep values out of the captures entirely. The same applies to + > `search-all-resources --read-mask` with resource data. Do NOT "optimize" + > this into a full-metadata dump. +- Failure → classify the error, then branch: + + **API not enabled** (stderr matches `SERVICE_DISABLED`, "has not been used", + "is not enabled", or "API [has not been / is not] enabled" — the common + startup case; Cloud Asset API is off by default): + + Offer **once** (user-driven enable — never run `gcloud services enable` or + IAM mutations yourself): + + ``` + ─── Cloud Asset Inventory not enabled ─── + + The Cloud Asset API is not enabled on [$GCP_PROJECT]. Enabling it gives a + fuller inventory in one call (including resources outside the per-service + list). I will not change your project for you. + + 1. Enable the API (pick one): + + gcloud services enable cloudasset.googleapis.com --project="$GCP_PROJECT" + + Or console: APIs & Services → Library → search "Cloud Asset API" → Enable. + Propagation can take up to a minute. + + 2. IAM — your identity also needs Cloud Asset Viewer on the project + (`roles/cloudasset.viewer`). Owner/Editor usually already include enough + access; otherwise ask an admin to grant that role. Docs: + https://docs.cloud.google.com/asset-inventory/docs/view-assets + + [Y] I've enabled it (and have access) — retry Cloud Asset Inventory + [N] Continue with per-service fallback + ``` + + - **[Y]** → wait for the user, then re-run the asset-search command **once**. + Success → continue as the Success path above; record + `cai_enable_offered: true`, `cai_enable_accepted: true` in the manifest. + Still failing → tell the user briefly (if 403/PERMISSION_DENIED, mention + `roles/cloudasset.viewer` again), then fall through to per-service (same + as [N]); record `cai_enable_accepted: true` and the retry failure note. + - **[N]** / no response treated as decline → fall through to per-service; + record `cai_enable_offered: true`, `cai_enable_accepted: false`. + + **Permission denied** (403 / `PERMISSION_DENIED` without the disable signals + above — API may already be on, but the identity lacks Cloud Asset access): + + Offer **once** (IAM guidance only — never grant roles yourself): + + ``` + ─── Cloud Asset Inventory permission denied ─── + + The Cloud Asset API appears enabled, but this identity cannot search assets + on [$GCP_PROJECT]. Grant Cloud Asset Viewer (`roles/cloudasset.viewer`) on + the project (or a role that includes `cloudasset.assets.searchAllResources` / + list permissions), then retry. Docs: + https://docs.cloud.google.com/asset-inventory/docs/view-assets + + [Y] I've updated IAM — retry Cloud Asset Inventory + [N] Continue with per-service fallback + ``` + + Same [Y]/[N] recording rules as the enable soft-ask + (`cai_enable_offered` / `cai_enable_accepted` — here "enable" means "CAI + access remediation offered"). + + **Other errors** (network, unexpected failures): do **not** soft-ask. + Fall through to per-service immediately; record `cai_enable_offered: false`. + + **Per-service fallthrough:** record `method: "per_service"` and run every + applicable table row below. Always keep the failed asset-search entry in + `captures[]` with `status: "failed"` and the stderr summary in `note`. + +**2b. Capture Command Table.** Each row redirects to the named file. On +"API not enabled" / permission errors: record the row as `failed` or `skipped` +in the manifest and continue — a missing service is normal, never a halt. +(Unlike the CAI fast path, do **not** soft-ask to enable individual service +APIs — too many rows, and "service not deployed" vs "API disabled" is ambiguous +from list errors alone.) + +| # | Command (always with `--project="$GCP_PROJECT"`) | Output file | Mode | +| -- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---- | +| 1 | `gcloud run services list --format="json(metadata.name, metadata.labels, metadata.annotations, spec.template.metadata.annotations, spec.template.spec.serviceAccountName, spec.template.spec.containers[].image, spec.template.spec.containers[].resources.limits, spec.template.spec.containers[].env[].name, spec.template.spec.containerConcurrency, spec.template.spec.timeoutSeconds, status.url)"` | `run.json` | E | +| 2 | `gcloud sql instances list --format="json(name, region, databaseVersion, settings.tier, settings.availabilityType, settings.dataDiskSizeGb, settings.ipConfiguration.privateNetwork, settings.ipConfiguration.ipv4Enabled, settings.backupConfiguration.enabled)"` | `sql.json` | E | +| 3 | `gcloud container clusters list --format="json(name, location, currentNodeCount, currentMasterVersion, network, subnetwork, autopilot.enabled, nodePools[].name, nodePools[].config.machineType, nodePools[].initialNodeCount)"` | `gke.json` | E | +| 4 | `gcloud functions list --format="json(name, environment, runtime, entryPoint, availableMemoryMb, serviceConfig.availableMemory, serviceConfig.runtime, serviceConfig.timeoutSeconds, eventTrigger.eventType, serviceConfig.serviceAccountEmail)"` | `functions.json` | E | +| 5 | `gcloud storage buckets list --format="json(name, location, storageClass, timeCreated, iamConfiguration.uniformBucketLevelAccess.enabled, versioning.enabled)"` | `buckets.json` | E | +| 6 | `gcloud pubsub topics list --format="json(name, labels)"` | `pubsub.json` | | +| 7 | `gcloud compute instances list --format="json(name, zone, machineType, status, networkInterfaces[].network, networkInterfaces[].subnetwork, disks[].diskSizeGb, serviceAccounts[].email, labels)"` | `gce.json` | E | +| 8 | `gcloud compute networks list --format="json(name, autoCreateSubnetworks, subnetworks)"` | `networks.json` | E | +| 9 | `gcloud compute networks subnets list --format="json(name, region, network, ipCidrRange)"` | `subnets.json` | E | +| 10 | `gcloud redis instances list --region= --format="json(name, tier, memorySizeGb, redisVersion, authorizedNetwork, locationId)"` | `redis-.json` | E | +| 11 | `gcloud secrets list --format="json(name, replication, createTime)"` — secret NAMES only, never `versions access` | `secrets.json` | E | +| 12 | `gcloud iam service-accounts list --format="json(email, displayName, disabled)"` | `sa.json` | E | +| 13 | `gcloud dns managed-zones list --format="json(name, dnsName, visibility)"` | `dns.json` | | +| 14 | `gcloud spanner instances list --format="json(name, config, nodeCount, processingUnits)"` | `spanner.json` | | +| 15 | `gcloud firestore databases list --format="json(name, type, locationId)"` | `firestore.json` | | +| 16 | `gcloud ai endpoints list --region= --format="json(name, displayName, deployedModels[].model)"` — only if asset search found `aiplatform.googleapis.com/*` assets or per-service mode | `vertex-.json` | E | +| 17 | `bq ls --project_id="$GCP_PROJECT" --format=json` — dataset names/locations only (the `bq` CLI ships with the Cloud SDK; if unavailable, record `skipped` — BigQuery presence then requires the asset-search path) | `bq.json` | E | +| 18 | `gcloud compute firewall-rules list --format="json(name, network, direction, priority)"` — deliberately minimal projection (no source ranges or target tags; the IaC path carries full rule config when Terraform exists) | `firewalls.json` | E | +| 19 | `gcloud compute networks vpc-access connectors describe --region= --format="json(name, network)"` — ONLY for each distinct `run.googleapis.com/vpc-access-connector` annotation value seen in row 1 output; resolves connector → VPC for edge inference | `connector-.json` | E | +| 20 | `gcloud compute regions list --format="json(name)"` — per-service mode only: enumerates the region walk for rows 10 and 16 (region names only; one cheap call) | `regions.json` | | + +**Row 1 note:** managed Cloud Run lists services across ALL regions when +`--region` is omitted — do not pass a `--region` flag (a `--region=-` form is +not documented). + +**Region-walk note (rows 10 and 16):** Redis and Vertex endpoint lists are +per-region, so their coverage is exactly the set of regions walked. Determine it +by mode: + +- **Asset-search mode:** walk the regions of the matching assets in + `assets.json` (`redis.googleapis.com/*` locations for row 10, + `aiplatform.googleapis.com/*` locations for row 16). Asset search is + project-wide, so no region can hide an instance from this walk. +- **Per-service mode:** walk EVERY region from row 20's `regions.json`. Do NOT + derive the walk from regions seen in other rows' output — that heuristic + fails in both directions (an instance in a region with no other footprint is + silently missed on custom-mode VPCs, while auto-mode VPC subnets inflate the + "seen" set to every region anyway, without the honesty of saying so). +- **Row 20 failed?** Fall back to the regions seen in rows 1–9 output, and + append to the manifest (→ `live_metadata.capture_warnings`): + `"regions list unavailable — redis/vertex walk limited to regions observed in + other captures; instances in other regions are not covered"`. + +**Sizing caveat:** SQL `settings.dataDiskSizeGb` is PROVISIONED disk, not actual +data volume. Downstream database-migration tool selection must treat it as an +upper bound. (Follow-up: enrich with actual data size from monitoring metrics. +Also follow-up: live-only compute resources are not graviton-profiled in v1 — +`graviton_profile` entries come from the IaC path only.) + +**Scale guard:** if `assets.json` (or any capture) exceeds ~100 resources, write a +throwaway extraction script to `$MIGRATION_DIR/_extract_live.py` that projects only +the fields needed by Step 3, run it, write its JSON output next to the raw file +with a `-extracted.json` suffix, and delete the script. Never Read the oversized +raw file directly. + +**2c. Write the manifest** — `$MIGRATION_DIR/live-capture/manifest.json`: + +```json +{ + "captured_at": "", + "gcloud_version": "", + "account": "", + "project": "<$GCP_PROJECT>", + "method": "asset_search|per_service", + "cai_enable_offered": false, + "cai_enable_accepted": null, + "captures": [ + { "command": "", "file": "", "status": "ok|failed|skipped", "note": null } + ] +} +``` + +Every attempted or deliberately skipped row gets an entry. `cai_enable_offered` / +`cai_enable_accepted` record the Step 2a soft-ask (`accepted` is `true` / +`false` / `null` when never offered). + +## Step 3: Map Captures to Inventory Resources + +Synthesize Terraform-style identity so downstream design-refs (keyed on +`google_*` types) work unchanged: + +- `address` = `{terraform_type}.{sanitized_resource_name}` (lowercase, `-`→`_`) +- `type` = from the mapping table below +- `name` = sanitized resource name +- `config` = the projected fields from the capture (redaction rules from the + Security Contract apply) +- `source` = `"live"` on every entry + +**Asset/CLI type → Terraform type mapping:** + +| Captured type | Terraform `type` | +| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | +| `run.googleapis.com/Service` / row 1 | `google_cloud_run_v2_service` | +| `sqladmin.googleapis.com/Instance` / row 2 | `google_sql_database_instance` | +| `container.googleapis.com/Cluster` / row 3 | `google_container_cluster` | +| row 4 with `environment: GEN_2` | `google_cloudfunctions2_function` | +| row 4 with `environment: GEN_1` (or unset) | `google_cloudfunctions_function` | +| `storage.googleapis.com/Bucket` / row 5 | `google_storage_bucket` | +| `pubsub.googleapis.com/Topic` / row 6 | `google_pubsub_topic` | +| `compute.googleapis.com/Instance` / row 7 | `google_compute_instance` | +| `compute.googleapis.com/Network` / row 8 | `google_compute_network` | +| `compute.googleapis.com/Subnetwork` / row 9 | `google_compute_subnetwork` | +| `compute.googleapis.com/Firewall` | `google_compute_firewall` (SECONDARY, role network_path — parity with IaC classification) | +| `redis.googleapis.com/Instance` / row 10 | `google_redis_instance` | +| `secretmanager.googleapis.com/Secret` / row 11 | `google_secret_manager_secret` | +| `iam.googleapis.com/ServiceAccount` / row 12 | `google_service_account` — `name` MUST be the email local-part (e.g. `app-sa` from `app-sa@…`), NEVER the display name, or IaC merge matching breaks | +| `dns.googleapis.com/ManagedZone` / row 13 | `google_dns_managed_zone` | +| `spanner.googleapis.com/Instance` / row 14 | `google_spanner_instance` | +| `firestore.googleapis.com/Database` / row 15 | `google_firestore_database` | +| `bigquery.googleapis.com/Dataset` / row 17 | `google_bigquery_dataset` (triggers the BigQuery specialist gate downstream — include it) | +| `aiplatform.googleapis.com/Endpoint` / row 16 | `google_vertex_ai_endpoint` | +| `aiplatform.googleapis.com/*` (other) | `google_vertex_ai_*` (matching suffix) | +| Any other asset type | Do NOT guess a mapping. Count it in `live_metadata.unmapped_asset_types` and exclude from the inventory. | + +**Classification:** apply `discover-iac.md` Step 3S rules — the Priority 1 PRIMARY +types list, everything else SECONDARY with role inferred from type +(`google_service_account` → identity; networks/subnets/DNS → network_path; +secrets → encryption; else configuration). `confidence: 0.99`. + +**AI detection:** if any `aiplatform.googleapis.com/*` asset or Vertex endpoint was +captured, populate `ai_detection` exactly as `discover-iac.md` Step 2 would +(signal method `"live_gcloud"`, confidence 95, `ai_services: ["vertex_ai"]`, +`has_ai_workload: true`). Otherwise `has_ai_workload: false`, `confidence: 0`. + +## Step 4: Infer Edges from Resolved Config + +Live captures contain resolved values, which often beat HCL references. Build +`edges[]` using ONLY these deterministic rules (evidence = the config field path): + +| Config field (captured) | Edge | +| --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Cloud Run annotation `run.googleapis.com/cloudsql-instances` | run service → SQL instance, `data_dependency` | +| Cloud Run annotation `run.googleapis.com/vpc-access-connector` | run service → network, `network_membership` — ONLY via the row 19 describe: the annotation value is a CONNECTOR id, not a network; resolve `connector.network` first. If the describe failed/was skipped, record the connector id in the service's `config` and emit NO edge (never fabricate a network target) | +| `spec.template.spec.serviceAccountName` / `serviceAccountEmail` | service account → workload, `serves` (populate the SA's `serves[]`) | +| SQL `settings.ipConfiguration.privateNetwork` | SQL instance → network, `network_membership` | +| GCE `networkInterfaces[].network` / GKE `network` | instance/cluster → network, `network_membership` | +| Subnet `network` | subnet → network, `network_membership` | +| Redis `authorizedNetwork` | redis → network, `network_membership` | + +No other inference — do not guess relationships from names, labels, or env var +names. + +> **Why not Cloud Asset Inventory relationship types?** Deliberate. CAI's +> relationship data (including `relationships.*` queries on +> `search-all-resources` — the query SYNTAX works on the standard endpoint, +> which misleads) requires the Security Command Center Premium/Enterprise tier +> or Gemini Cloud Assist, which the startups this skill targets do not have; +> without the entitlement those queries return nothing. Most relationship types +> are also unavailable in the search API entirely. Resolved-config inference +> above needs only `roles/viewer` and covers the workload-shaped edges that +> matter for migration sequencing. Do NOT add `relationships.*` queries to the +> capture set. + +## Step 5: Cluster (Simplified Mode) + +Apply `discover-iac.md` **Step 3S** clustering rules regardless of resource count +(networking cluster at depth 0; one cluster per PRIMARY plus its `serves` +secondaries at depth 1; same `{category}_{type}_{region}_{sequence}` naming; +region from the captured `region`/`location`/zone-derived-region). Set metadata +`"clustering_mode": "simplified_live"`. If more than 25 PRIMARY resources were +captured, warn the user that clustering is coarse at this scale and suggest +narrowing to specific services or regions — but continue. + +**Live-specific clustering rules** (Step 3S assumes files, which don't exist here): + +- **Regionless resources** (Pub/Sub topics, global buckets without a single + region): use `"global"` as the region component of `cluster_id` and + `gcp_region`. +- **Shared secondaries** (e.g., one service account serving multiple primaries): + assign the resource to the cluster of the FIRST primary in its `serves[]` + array; `serves[]` still lists all of them. +- **Evidence-less secondaries** (no Step 4 edge and empty `serves[]`, e.g., + secrets): do NOT attach them to an unrelated primary's cluster and do NOT + fabricate a `serves` relationship. Group them into their own cluster per + category+region (e.g., `security_secrets_global_001`) at depth 1. + +## Step 6: Merge with IaC Discovery (only if `discover-iac.md` produced output) + +If `gcp-resource-inventory.json` does NOT already exist, skip to Step 7 (live is +the sole source). + +Otherwise the IaC inventory + clusters are the BASE. Match live↔IaC entries by +Terraform `type` + GCP resource name, treating these type pairs as EQUIVALENT +for matching (same underlying service; live always maps to the newer type): +`google_cloud_run_service` ≡ `google_cloud_run_v2_service`, and +`google_cloudfunctions_function` ≡ `google_cloudfunctions2_function` when the +GCP name matches. Without this aliasing, a v1-declared resource produces FALSE +drift (flagged both `not_found_live` and `unmanaged_by_terraform`). The merged +entry keeps the IaC address; a note in `config` (`"live_type"`) records the +newer live type. Match names as: live `name` vs the IaC resource's +`config.name` — for service accounts use `config.account_id` (the email +local-part IS the SA's GCP name; the address name component often differs) — +falling back to the address name component). Then: + +1. **Matched:** keep the IaC entry (its address, classification, cluster, + depth). Overwrite `config` values where live disagrees — sizing, capacity, + versions, and images alike (live reflects reality). Live-only fields the IaC + never declared are enrichment, not conflicts. Record every OVERWRITTEN field in + `live_metadata.drift.config_conflicts[]` as + `{ "address", "field", "terraform_value", "live_value" }`. Set + `source: "live+terraform"`. +2. **Live-only:** append the entry with `unmanaged_by_terraform: true`. Attach it + to an existing cluster of the same category+region when one exists; otherwise + append a new simplified cluster (and add it to `creation_order` at its depth). +3. **IaC-only:** set `source: "terraform"` on every unmatched IaC entry. Set + `not_found_live: true` ONLY if the capture covering that resource's service + succeeded (manifest `ok`). If the relevant capture failed or was skipped, + leave the entry otherwise untouched — absence of evidence is not drift. +4. **Drift summary:** `live_metadata.drift = { "resources_live_only": N, + "resources_terraform_only": M, "config_conflicts": [...] }`. + `resources_terraform_only` counts ONLY entries with `not_found_live: true` + (confirmed absent), never capture-failed unknowns. +5. **Merged metadata:** keep the IaC base's `clustering_mode` (`"simplified"` or + absent for full clustering) — `"simplified_live"` is for live-only runs. Set + `metadata.discovery_sources` to include both sources. + +Never silently resolve a disagreement — every conflict lands in the drift record. + +## Step 7: Write Output Files + +Load `references/shared/schema-discover-iac.md` (if not already loaded) and +write/update: + +1. `$MIGRATION_DIR/gcp-resource-inventory.json` — exact schema; plus: + - `metadata.discovery_sources`: `["live"]`, `["terraform", "live"]`, etc. + - `metadata.clustering_mode`: `"simplified_live"` (live-only runs) + - top-level `live_metadata`: + + ```json + { + "found": true, + "captured_at": "", + "project": "<$GCP_PROJECT>", + "method": "asset_search|per_service", + "cai_enable_offered": false, + "cai_enable_accepted": null, + "capture_warnings": [""], + "unmapped_asset_types": { "": 2 }, + "drift": { "resources_live_only": 0, "resources_terraform_only": 0, "config_conflicts": [] } + } + ``` + + Copy `cai_enable_offered` / `cai_enable_accepted` from the manifest. (`drift` + present only when Step 6 merged.) + +2. `$MIGRATION_DIR/gcp-resource-clusters.json` — exact schema (merged or fresh). +3. Validate per `discover-iac.md` Step 7c (every resource in exactly one cluster, + IDs consistent, valid JSON). Report: "Live discovery: X resources captured from + project [id] (Y unmanaged by Terraform, Z config conflicts)." + +The parent `discover.md` owns the phase status update — do not touch +`.phase-status.json` here. + +--- + +## Error Handling + +| Error | Behavior | +| -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| gcloud missing / no active account / user declines | Exit cleanly with no output (orchestrator falls back to file-based sources) | +| Asset search fails (API not enabled) | Soft-ask once: enable API + `roles/cloudasset.viewer` how-to + docs link; on [Y] retry once; on [N]/retry-fail → per-service (agent never mutates) | +| Asset search fails (permission denied) | Soft-ask once: grant `roles/cloudasset.viewer` + docs link; on [Y] retry once; on [N]/retry-fail → per-service | +| Asset search fails (other errors) | Fall back to per-service immediately — do not soft-ask | +| Individual row fails (API not enabled, 403) | Record `failed`/`skipped`, continue — never a halt (no per-row enable soft-ask) | +| Token expired mid-run | Stop capturing; hand off ("run `gcloud auth login`, then tell me to continue"); on resume re-run Step 2 (captures overwrite) | +| Capture file unparseable | Record warning, skip that file, continue | +| Every capture failed | Exit with no output; tell the user which permissions are missing (`roles/viewer` covers all rows) | + +**Key principle:** partial results are better than no results. Record what failed; +never fabricate what wasn't captured. + +## Scope Boundary + +**This sub-file covers live GCP discovery ONLY.** + +FORBIDDEN — Do NOT include ANY of: + +- AWS service names, recommendations, or equivalents +- Migration strategies, phases, timelines, cost estimates, or effort estimates +- Any mutating gcloud command, `auth login`, or token printing +- Env var values, secret values, or unredacted sensitive config anywhere + +**Your ONLY job: inventory what exists in GCP. Nothing else.** diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md index 08cbd7d5..334ecc6f 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/discover/discover.md @@ -6,6 +6,7 @@ Lightweight orchestrator that delegates to domain-specific discoverers. Each sub ## Sub-Discovery Files - **discover-iac.md** → `gcp-resource-inventory.json` + `gcp-resource-clusters.json` (if Terraform found); may also write `ai-workload-profile.json` when **Vertex-strong** (see `discover-iac.md` Step 7d) +- **discover-live.md** → `gcp-resource-inventory.json` + `gcp-resource-clusters.json` from the user's authenticated `gcloud` CLI (read-only, consent-gated); merges into the IaC inventory with drift when both run - **discover-app-code.md** → `ai-workload-profile.json` when AI confidence ≥ 70% (may **merge** with an existing `iac_vertex` profile) - **discover-billing.md** → `billing-profile.json` (if billing data found) @@ -111,7 +112,31 @@ When Terraform is present, billing data is supplementary — only service-level **Critical:** Do **not** Read the billing file with the Read tool. Do **not** load `discover-billing.md` or `schema-discover-billing.md`. -**If NONE of the three checks found files**: STOP and output: "No GCP sources detected. Provide at least one source type (Terraform files, application code, or billing exports) and try again." +**1d. Live discovery (gcloud CLI):** +Runs AFTER 1a–1c sub-discoveries complete, so its IaC merge sees their output. + +- If `$MIGRATION_DIR/live-capture/manifest.json` already exists (a prior capture, + e.g. a resumed run) → Load `references/phases/discover/discover-live.md` and + execute from its Step 3 (parse the existing captures; skip consent/preflight/ + capture — they already happened). +- Else if Terraform files were found in 1a → offer ONCE as an optional cross-check: + "I found Terraform covering your infrastructure. Want me to cross-check it + against your live GCP project via your authenticated gcloud CLI (read-only, + with your consent)? This catches resources managed outside Terraform." + On yes → Load `references/phases/discover/discover-live.md`. On no → continue + (do not re-ask this run). +- Else if NO Terraform was found (regardless of whether 1b/1c found app code or + billing files — those cannot produce an infrastructure inventory) → offer live + discovery as the primary infrastructure source: "No Terraform detected — I can + discover your project's infrastructure directly via your authenticated gcloud + CLI (read-only, with your consent). Proceed?" On yes → Load + `references/phases/discover/discover-live.md`. On no → continue with whatever + 1b/1c produced (billing-only design path remains the fallback). +- If, after the offer, NO sub-discovery produced or will produce any artifact + (nothing found by 1a–1c AND live was declined or unavailable) → STOP and + output: "No GCP sources detected. Provide at least one source type (Terraform + files, application code, or billing exports), or re-run and accept live + discovery." ## Step 2: Check Outputs @@ -129,6 +154,7 @@ After all loaded sub-discoveries complete, check what artifacts were produced in - If its Step 4 exit gate applied (overall AI confidence **below** 70%) **and** no `ai-workload-profile.json` exists -> **allow completion** (app-code route may produce no AI profile). - If Step 4 exit applied with confidence below 70% **but** `ai-workload-profile.json` exists with `metadata.profile_source` = `"iac_vertex"` -> **allow completion** (IaC-inferred profile retained). - If execution continued to Steps 5–8 (confidence **≥** 70%) -> **require** `ai-workload-profile.json`. + - If `discover-live.md` ran AND capture happened (`$MIGRATION_DIR/live-capture/manifest.json` exists) -> require `gcp-resource-inventory.json` and `gcp-resource-clusters.json`, with `live_metadata` present in the inventory. (If the user declined consent or gcloud was unavailable, the sub-file exited cleanly — no artifact required.) - If full `discover-billing.md` ran OR lightweight billing extraction ran -> require `billing-profile.json` - If any triggered route is missing its required artifact(s): STOP and output: "Discover route [name] did not produce required artifacts. Resolve the sub-discovery failure before completing Phase 1." @@ -167,6 +193,7 @@ Only after `HANDOFF_OK`. In the **same turn** as the output message below, use t Output to user — build message from whichever artifacts exist: - If `gcp-resource-inventory.json` exists: "Discovered X total resources across Y clusters." +- If live discovery ran: "Live discovery captured N resources from project [id]." Plus, when IaC also ran: "Drift check: A resources live but not in Terraform, B in Terraform but not live, C config conflicts (live values used)." Plus, when `live_metadata.unmapped_asset_types` is non-empty: "Skipped M unmapped asset types (top: X, Y, Z) — full list in live_metadata." - If `ai-workload-profile.json` exists: "Detected AI workloads (source: [ai_source])." - If `billing-profile.json` exists: "Parsed billing data ($Z/month across N services)." diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md index fd352af6..d59a1f02 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/shared/schema-discover-iac.md @@ -104,6 +104,52 @@ Complete inventory of discovered GCP resources with classification, dependencies - `signals_found[]` — array of detection signals with method, pattern, confidence, evidence - `ai_services[]` — list of AI services detected (vertex_ai, bigquery_ml, etc.) +### Live discovery extensions (present when `discover-live.md` ran) + +Live gcloud discovery produces the same inventory/cluster schemas with these additions: + +**`metadata` additional fields:** + +- `discovery_sources` — string[]: which sources produced data, e.g. `["live"]` or `["terraform", "live"]`. `terraform_version` may be `null` on live-only runs. +- `clustering_mode` — `"simplified"` (IaC Step 3S; also used for merged IaC+live runs), `"simplified_live"` (live-only runs), or absent (full IaC clustering). + +**`resources[]` optional fields:** + +- `source` — `"terraform"`, `"live"`, or `"live+terraform"` (merged entry) +- `unmanaged_by_terraform` — `true` when live discovery found the resource but no Terraform manages it (click-ops drift) +- `not_found_live` — `true` when Terraform declares the resource but the (successful) live capture did not find it deployed + +**Top-level `live_metadata` section:** + +```json +{ + "live_metadata": { + "found": true, + "captured_at": "2026-07-20T18:20:00Z", + "project": "acme-prod", + "method": "asset_search", + "cai_enable_offered": false, + "cai_enable_accepted": null, + "capture_warnings": [], + "unmapped_asset_types": {}, + "drift": { + "resources_live_only": 0, + "resources_terraform_only": 0, + "config_conflicts": [ + { + "address": "google_sql_database_instance.db", + "field": "settings.tier", + "terraform_value": "db-f1-micro", + "live_value": "db-custom-2-8192" + } + ] + } + } +} +``` + +`drift` is present only when Terraform AND live discovery both produced resources. Env var and secret VALUES must never appear anywhere in the inventory — names only, with `discover-iac.md` Step 0 redaction patterns applied. + --- ## gcp-resource-clusters.json (Phase 1 output)