From b6df27b52d107d1fcc26a7545a53158ea96ef508 Mon Sep 17 00:00:00 2001 From: Gen Li Date: Tue, 4 Aug 2026 16:55:28 -0700 Subject: [PATCH 1/4] fix(gcp-to-aws): stop generated scripts from prescribing a BigQuery data lake on S3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design phase's BigQuery specialist gate (design-infra.md Pass 2 step 0) sets `aws_service` to `Deferred — specialist engagement` with `no_automated_aws_target: true` and explicitly forbids naming Athena, Redshift, Glue, EMR, Lake Formation, or a prescribed "data lake on S3" for any `google_bigquery_*` resource. The generate phase then contradicted that: a "BigQuery to S3" block emitting `bq extract --destination_format=PARQUET` plus `aws s3 sync` into a target bucket was gated on `has_databases` — the very predicate the deferral keys on — so a customer told "no automated AWS target" still received the forbidden architecture. It shipped commented out, so nothing executed silently; the defect misguided rather than ran. - Replace the BigQuery export block with a deferral notice that mirrors the `specialist_engagement` wording already used by design-infra.md and generate-artifacts-docs.md (engage AWS account team and/or a data analytics migration partner), so BigQuery users get a next step instead of silence. - Add a `has_bigquery` detection flag (same name/semantics as the existing flag in discover-preview.md) and gate the notice on it. The old block fired for any database resource, so Cloud SQL-only projects were handed a BigQuery data lake plan with no BigQuery present. - Add self-check rule 6 asserting no script prescribes an AWS analytics or warehouse target for BigQuery. `has_databases` semantics are unchanged, so `has_data_migration`, the Cloud SQL and Firestore blocks, and the 05 RDS validation section behave exactly as before. --- .../generate/generate-artifacts-scripts.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md index 6447920c..f29df5c9 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -30,6 +30,8 @@ Set boolean flags for downstream script generation: - **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB", "ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`, `google_bigtable_`, `google_bigquery_`, `google_redis_` +- **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_` + OR `aws_service` = "Deferred — specialist engagement" - **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket` - **has_containers**: true if ANY resource has `aws_service` containing "Fargate", "ECS", "EKS" OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster` @@ -201,13 +203,25 @@ echo "=== Verification ===" echo "TODO: Compare row counts between source and target" ``` -**BigQuery to S3** — include only if `has_databases`: +**BigQuery — specialist-deferred (no automated AWS target)** — include only if `has_bigquery`: + +Design marked every `google_bigquery_*` resource as **`Deferred — specialist engagement`** with +`no_automated_aws_target: true` (see `references/phases/design/design-infra.md` → BigQuery specialist gate). +Therefore **do not** generate BigQuery export, copy, or load steps, and **do not** name an AWS analytics or +warehouse target (no Athena, Redshift, Glue, EMR, Lake Formation, or a prescribed "data lake on S3"). +Emit only the deferral notice below: ```bash -# BigQuery → S3 data export -# TODO: Configure BigQuery dataset and S3 bucket -# bq extract --destination_format=PARQUET 'dataset.table' 'gs://bucket/export/' -# aws s3 sync gs://bucket/export/ s3://target-bucket/import/ +# BigQuery — AWS target deliberately NOT selected +# This plugin does not choose an AWS analytics or warehouse target for BigQuery +# (no Athena/Redshift/Glue/EMR recommendation, and no prescribed data lake on S3). +# Engage your AWS account team and/or a data analytics migration partner before +# data warehouse, lake, SQL analytics, or BI cutover planning — query patterns, +# data volumes, ETL/ELT, and downstream consumers must be assessed by specialists. +# No BigQuery migration steps are generated here by design. +echo "BigQuery: AWS target deferred — specialist engagement required." +echo " Next step: engage your AWS account team and/or a data analytics migration partner." +echo " No BigQuery export or load steps are generated by this skill." ``` **Firestore to DynamoDB** — include only if `has_databases`: @@ -383,6 +397,9 @@ After generating all scripts, verify the following quality rules: 3. All scripts include verification steps 4. All scripts are numbered for execution order 5. All TODO markers are clearly marked with context +6. No script prescribes an AWS analytics or warehouse target for BigQuery (no Athena, Redshift, Glue, + EMR, Lake Formation, or "data lake on S3", and no `bq extract` / export steps) — BigQuery resources + carry only the specialist-engagement deferral notice, consistent with `no_automated_aws_target: true` ## Phase Completion From 7bd8947408b5c87eacc63edb9a2033d1f7a1e46f Mon Sep 17 00:00:00 2001 From: Gen Li Date: Tue, 11 Aug 2026 14:32:42 -0700 Subject: [PATCH 2/4] fix(aws-startup-advisor): apply the BigQuery deferral gate to the advisor's copy of the scripts phase The advisor plugin vendors a byte-identical copy of generate-artifacts-scripts.md. The previous commit fixed only the gcp-to-aws copy, so the advisor path still emitted the bq-extract / S3-sync block the specialist gate exists to prevent. Same three edits: the has_bigquery flag, the deferral notice replacing the export block, and quality rule 6. The two files are byte-identical again. Nothing catches this drift automatically: sync-vendored-shared.ts only syncs skills/shared within a single plugin, so the two plugins can disagree without failing shared:check. --- .../generate/generate-artifacts-scripts.md | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-) diff --git a/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md b/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md index 6447920c..f29df5c9 100644 --- a/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -30,6 +30,8 @@ Set boolean flags for downstream script generation: - **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB", "ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`, `google_bigtable_`, `google_bigquery_`, `google_redis_` +- **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_` + OR `aws_service` = "Deferred — specialist engagement" - **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket` - **has_containers**: true if ANY resource has `aws_service` containing "Fargate", "ECS", "EKS" OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster` @@ -201,13 +203,25 @@ echo "=== Verification ===" echo "TODO: Compare row counts between source and target" ``` -**BigQuery to S3** — include only if `has_databases`: +**BigQuery — specialist-deferred (no automated AWS target)** — include only if `has_bigquery`: + +Design marked every `google_bigquery_*` resource as **`Deferred — specialist engagement`** with +`no_automated_aws_target: true` (see `references/phases/design/design-infra.md` → BigQuery specialist gate). +Therefore **do not** generate BigQuery export, copy, or load steps, and **do not** name an AWS analytics or +warehouse target (no Athena, Redshift, Glue, EMR, Lake Formation, or a prescribed "data lake on S3"). +Emit only the deferral notice below: ```bash -# BigQuery → S3 data export -# TODO: Configure BigQuery dataset and S3 bucket -# bq extract --destination_format=PARQUET 'dataset.table' 'gs://bucket/export/' -# aws s3 sync gs://bucket/export/ s3://target-bucket/import/ +# BigQuery — AWS target deliberately NOT selected +# This plugin does not choose an AWS analytics or warehouse target for BigQuery +# (no Athena/Redshift/Glue/EMR recommendation, and no prescribed data lake on S3). +# Engage your AWS account team and/or a data analytics migration partner before +# data warehouse, lake, SQL analytics, or BI cutover planning — query patterns, +# data volumes, ETL/ELT, and downstream consumers must be assessed by specialists. +# No BigQuery migration steps are generated here by design. +echo "BigQuery: AWS target deferred — specialist engagement required." +echo " Next step: engage your AWS account team and/or a data analytics migration partner." +echo " No BigQuery export or load steps are generated by this skill." ``` **Firestore to DynamoDB** — include only if `has_databases`: @@ -383,6 +397,9 @@ After generating all scripts, verify the following quality rules: 3. All scripts include verification steps 4. All scripts are numbered for execution order 5. All TODO markers are clearly marked with context +6. No script prescribes an AWS analytics or warehouse target for BigQuery (no Athena, Redshift, Glue, + EMR, Lake Formation, or "data lake on S3", and no `bq extract` / export steps) — BigQuery resources + carry only the specialist-engagement deferral notice, consistent with `no_automated_aws_target: true` ## Phase Completion From e1d4d264854670238269baca95f9f37be98c8a85 Mon Sep 17 00:00:00 2001 From: Gen Li Date: Tue, 11 Aug 2026 14:33:18 -0700 Subject: [PATCH 3/4] fix(gcp-to-aws): a BigQuery-only project must not trigger database-migration scripts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit has_databases still listed google_bigquery_, so a BigQuery-only project set has_data_migration and generated the Cloud SQL to RDS and Firestore to DynamoDB blocks with no such resources in the design. BigQuery is specialist-deferred and owns no automated data steps — it now counts only toward has_bigquery. A project with BigQuery plus real databases is unaffected (the other prefixes still set the flag). Applied to both vendored copies; files remain byte-identical. Noted by ayn-builds in #202 review. --- .../references/phases/generate/generate-artifacts-scripts.md | 3 ++- .../references/phases/generate/generate-artifacts-scripts.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md b/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md index f29df5c9..53b23f30 100644 --- a/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/advisor/plugins/aws-startup-advisor/skills/migration-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -29,7 +29,8 @@ Set boolean flags for downstream script generation: - **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB", "ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`, - `google_bigtable_`, `google_bigquery_`, `google_redis_` + `google_bigtable_`, `google_redis_` (NOT `google_bigquery_` — BigQuery is specialist-deferred + and carries no automated data-migration steps; see `has_bigquery`) - **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_` OR `aws_service` = "Deferred — specialist engagement" - **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket` diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md index f29df5c9..53b23f30 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -29,7 +29,8 @@ Set boolean flags for downstream script generation: - **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB", "ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`, - `google_bigtable_`, `google_bigquery_`, `google_redis_` + `google_bigtable_`, `google_redis_` (NOT `google_bigquery_` — BigQuery is specialist-deferred + and carries no automated data-migration steps; see `has_bigquery`) - **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_` OR `aws_service` = "Deferred — specialist engagement" - **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket` From ed03bc9ad5f1a347eb13e00f9f3d25e57e593c42 Mon Sep 17 00:00:00 2001 From: Gen Li Date: Thu, 13 Aug 2026 13:47:18 -0700 Subject: [PATCH 4/4] fix(gcp-to-aws): keep script 02 reachable and well-formed for BigQuery-only projects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fallout gaps from narrowing has_databases, both found in review: 1. has_data_migration was still 'has_databases OR has_storage', so a BigQuery-only project (no other DBs, no GCS) skipped script 02 entirely — the deferral notice never rendered for exactly the customer the specialist gate protects. It now includes has_bigquery. 2. The shebang, set -euo pipefail, and --execute plumbing lived inside the Cloud SQL block (gated on has_databases), so a BigQuery+GCS project generated a script that was bare echo lines — contradicting Script Quality Rule 1. Hoisted into an ungated preamble emitted whenever the script is generated; the Cloud SQL block keeps only its own content. Applied to both plugin copies; byte-identical, drift green. --- .../generate/generate-artifacts-scripts.md | 26 ++++++++++++------- .../generate/generate-artifacts-scripts.md | 26 ++++++++++++------- 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/advisor/plugins/aws-startup-advisor/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md b/advisor/plugins/aws-startup-advisor/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md index 53b23f30..fa3cbcf1 100644 --- a/advisor/plugins/aws-startup-advisor/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/advisor/plugins/aws-startup-advisor/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -38,7 +38,7 @@ Set boolean flags for downstream script generation: OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster` - **has_secrets**: true if ANY resource has `aws_service` containing "Secrets Manager" OR `gcp_type` starting with `google_secret_manager_` -- **has_data_migration**: has_databases OR has_storage (used for script 02) +- **has_data_migration**: has_databases OR has_storage OR has_bigquery (used for script 02) Report detected categories to user: "Resource categories detected: [list active flags]" @@ -82,6 +82,22 @@ Verify all prerequisites before migration: Based on database and storage resources in `aws-design.json`: +**Script preamble** — emit FIRST, whenever this script is generated (any subsection below +active). The shebang, failure mode, and dry-run plumbing are not gated on any single resource +flag: a BigQuery-only project generates only the deferral notice below, and that script still +has to satisfy Script Quality Rule 1 (`set -euo pipefail`): + +```bash +#!/usr/bin/env bash +set -euo pipefail +# Data migration +# Usage: ./02-migrate-data.sh [--execute] + +DRY_RUN=true +[[ "${1:-}" == "--execute" ]] && DRY_RUN=false +echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')" +``` + **Cloud SQL to RDS/Aurora** — include only if `has_databases`: Read `preferences.json` → `design_constraints.db_size.value` to select the migration tool: @@ -94,11 +110,7 @@ Read `preferences.json` → `design_constraints.db_size.value` to select the mig Generate the script with conditional branches based on `db_size`: ```bash -#!/usr/bin/env bash -set -euo pipefail # Cloud SQL → RDS data migration -# Usage: ./02-migrate-data.sh [--execute] -# # Tool selection based on database size (preferences.json design_constraints.db_size.value): # <10GB: pg_dump/pg_restore # 10-500GB: pgcopydb (parallel copy, 3-5x faster than pg_dump) @@ -106,11 +118,7 @@ set -euo pipefail # unknown: pgcopydb (safer default at unknown scale) # TODO: Verify database size before running — wrong tool choice can exceed your maintenance window. -DRY_RUN=true -[[ "${1:-}" == "--execute" ]] && DRY_RUN=false - echo "=== Database Migration: Cloud SQL → RDS ===" -echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')" SOURCE_HOST="" # TODO: Set Cloud SQL IP TARGET_HOST="" # From terraform output database_endpoint diff --git a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md index 53b23f30..fa3cbcf1 100644 --- a/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md +++ b/migrate/plugins/migration-to-aws/skills/gcp-to-aws/references/phases/generate/generate-artifacts-scripts.md @@ -38,7 +38,7 @@ Set boolean flags for downstream script generation: OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster` - **has_secrets**: true if ANY resource has `aws_service` containing "Secrets Manager" OR `gcp_type` starting with `google_secret_manager_` -- **has_data_migration**: has_databases OR has_storage (used for script 02) +- **has_data_migration**: has_databases OR has_storage OR has_bigquery (used for script 02) Report detected categories to user: "Resource categories detected: [list active flags]" @@ -82,6 +82,22 @@ Verify all prerequisites before migration: Based on database and storage resources in `aws-design.json`: +**Script preamble** — emit FIRST, whenever this script is generated (any subsection below +active). The shebang, failure mode, and dry-run plumbing are not gated on any single resource +flag: a BigQuery-only project generates only the deferral notice below, and that script still +has to satisfy Script Quality Rule 1 (`set -euo pipefail`): + +```bash +#!/usr/bin/env bash +set -euo pipefail +# Data migration +# Usage: ./02-migrate-data.sh [--execute] + +DRY_RUN=true +[[ "${1:-}" == "--execute" ]] && DRY_RUN=false +echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')" +``` + **Cloud SQL to RDS/Aurora** — include only if `has_databases`: Read `preferences.json` → `design_constraints.db_size.value` to select the migration tool: @@ -94,11 +110,7 @@ Read `preferences.json` → `design_constraints.db_size.value` to select the mig Generate the script with conditional branches based on `db_size`: ```bash -#!/usr/bin/env bash -set -euo pipefail # Cloud SQL → RDS data migration -# Usage: ./02-migrate-data.sh [--execute] -# # Tool selection based on database size (preferences.json design_constraints.db_size.value): # <10GB: pg_dump/pg_restore # 10-500GB: pgcopydb (parallel copy, 3-5x faster than pg_dump) @@ -106,11 +118,7 @@ set -euo pipefail # unknown: pgcopydb (safer default at unknown scale) # TODO: Verify database size before running — wrong tool choice can exceed your maintenance window. -DRY_RUN=true -[[ "${1:-}" == "--execute" ]] && DRY_RUN=false - echo "=== Database Migration: Cloud SQL → RDS ===" -echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')" SOURCE_HOST="" # TODO: Set Cloud SQL IP TARGET_HOST="" # From terraform output database_endpoint