Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,16 @@ Set boolean flags for downstream script generation:

- **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB",
"ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`,
`google_bigtable_`, `google_bigquery_`, `google_redis_`
`google_bigtable_`, `google_redis_` (NOT `google_bigquery_` — BigQuery is specialist-deferred
and carries no automated data-migration steps; see `has_bigquery`)
- **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_`
OR `aws_service` = "Deferred — specialist engagement"
- **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket`
- **has_containers**: true if ANY resource has `aws_service` containing "Fargate", "ECS", "EKS"
OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster`
- **has_secrets**: true if ANY resource has `aws_service` containing "Secrets Manager"
OR `gcp_type` starting with `google_secret_manager_`
- **has_data_migration**: has_databases OR has_storage (used for script 02)
- **has_data_migration**: has_databases OR has_storage OR has_bigquery (used for script 02)

Report detected categories to user: "Resource categories detected: [list active flags]"

Expand Down Expand Up @@ -79,6 +82,22 @@ Verify all prerequisites before migration:

Based on database and storage resources in `aws-design.json`:

**Script preamble** — emit FIRST, whenever this script is generated (any subsection below
active). The shebang, failure mode, and dry-run plumbing are not gated on any single resource
flag: a BigQuery-only project generates only the deferral notice below, and that script still
has to satisfy Script Quality Rule 1 (`set -euo pipefail`):

```bash
#!/usr/bin/env bash
set -euo pipefail
# Data migration
# Usage: ./02-migrate-data.sh [--execute]

DRY_RUN=true
[[ "${1:-}" == "--execute" ]] && DRY_RUN=false
echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')"
```

**Cloud SQL to RDS/Aurora** — include only if `has_databases`:

Read `preferences.json` → `design_constraints.db_size.value` to select the migration tool:
Expand All @@ -91,23 +110,15 @@ Read `preferences.json` → `design_constraints.db_size.value` to select the mig
Generate the script with conditional branches based on `db_size`:

```bash
#!/usr/bin/env bash
set -euo pipefail
# Cloud SQL → RDS data migration
# Usage: ./02-migrate-data.sh [--execute]
#
# Tool selection based on database size (preferences.json design_constraints.db_size.value):
# <10GB: pg_dump/pg_restore
# 10-500GB: pgcopydb (parallel copy, 3-5x faster than pg_dump)
# >500GB: AWS DMS recommended — see README-DMS.md if generated
# unknown: pgcopydb (safer default at unknown scale)
# TODO: Verify database size before running — wrong tool choice can exceed your maintenance window.

DRY_RUN=true
[[ "${1:-}" == "--execute" ]] && DRY_RUN=false

echo "=== Database Migration: Cloud SQL → RDS ==="
echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')"

SOURCE_HOST="" # TODO: Set Cloud SQL IP
TARGET_HOST="" # From terraform output database_endpoint
Expand Down Expand Up @@ -201,13 +212,25 @@ echo "=== Verification ==="
echo "TODO: Compare row counts between source and target"
```

**BigQuery to S3** — include only if `has_databases`:
**BigQuery — specialist-deferred (no automated AWS target)** — include only if `has_bigquery`:

Design marked every `google_bigquery_*` resource as **`Deferred — specialist engagement`** with
`no_automated_aws_target: true` (see `references/phases/design/design-infra.md` → BigQuery specialist gate).
Therefore **do not** generate BigQuery export, copy, or load steps, and **do not** name an AWS analytics or
warehouse target (no Athena, Redshift, Glue, EMR, Lake Formation, or a prescribed "data lake on S3").
Emit only the deferral notice below:

```bash
# BigQuery → S3 data export
# TODO: Configure BigQuery dataset and S3 bucket
# bq extract --destination_format=PARQUET 'dataset.table' 'gs://bucket/export/'
# aws s3 sync gs://bucket/export/ s3://target-bucket/import/
# BigQuery — AWS target deliberately NOT selected
# This plugin does not choose an AWS analytics or warehouse target for BigQuery
# (no Athena/Redshift/Glue/EMR recommendation, and no prescribed data lake on S3).
# Engage your AWS account team and/or a data analytics migration partner before
# data warehouse, lake, SQL analytics, or BI cutover planning — query patterns,
# data volumes, ETL/ELT, and downstream consumers must be assessed by specialists.
# No BigQuery migration steps are generated here by design.
echo "BigQuery: AWS target deferred — specialist engagement required."
echo " Next step: engage your AWS account team and/or a data analytics migration partner."
echo " No BigQuery export or load steps are generated by this skill."
```

**Firestore to DynamoDB** — include only if `has_databases`:
Expand Down Expand Up @@ -383,6 +406,9 @@ After generating all scripts, verify the following quality rules:
3. All scripts include verification steps
4. All scripts are numbered for execution order
5. All TODO markers are clearly marked with context
6. No script prescribes an AWS analytics or warehouse target for BigQuery (no Athena, Redshift, Glue,
EMR, Lake Formation, or "data lake on S3", and no `bq extract` / export steps) — BigQuery resources
carry only the specialist-engagement deferral notice, consistent with `no_automated_aws_target: true`

## Phase Completion

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,16 @@ Set boolean flags for downstream script generation:

- **has_databases**: true if ANY resource has `aws_service` containing "RDS", "Aurora", "DynamoDB",
"ElastiCache", "Redshift" OR `gcp_type` starting with `google_sql_`, `google_firestore_`,
`google_bigtable_`, `google_bigquery_`, `google_redis_`
`google_bigtable_`, `google_redis_` (NOT `google_bigquery_` — BigQuery is specialist-deferred
and carries no automated data-migration steps; see `has_bigquery`)
- **has_bigquery**: true if ANY resource has `gcp_type` starting with `google_bigquery_`
OR `aws_service` = "Deferred — specialist engagement"
Comment thread
leon1418 marked this conversation as resolved.
- **has_storage**: true if ANY resource has `aws_service` = "S3" OR `gcp_type` = `google_storage_bucket`
- **has_containers**: true if ANY resource has `aws_service` containing "Fargate", "ECS", "EKS"
OR `gcp_type` starting with `google_cloud_run_`, `google_container_cluster`
- **has_secrets**: true if ANY resource has `aws_service` containing "Secrets Manager"
OR `gcp_type` starting with `google_secret_manager_`
- **has_data_migration**: has_databases OR has_storage (used for script 02)
- **has_data_migration**: has_databases OR has_storage OR has_bigquery (used for script 02)

Report detected categories to user: "Resource categories detected: [list active flags]"

Expand Down Expand Up @@ -79,6 +82,22 @@ Verify all prerequisites before migration:

Based on database and storage resources in `aws-design.json`:

**Script preamble** — emit FIRST, whenever this script is generated (any subsection below
active). The shebang, failure mode, and dry-run plumbing are not gated on any single resource
flag: a BigQuery-only project generates only the deferral notice below, and that script still
has to satisfy Script Quality Rule 1 (`set -euo pipefail`):

```bash
#!/usr/bin/env bash
set -euo pipefail
# Data migration
# Usage: ./02-migrate-data.sh [--execute]

DRY_RUN=true
[[ "${1:-}" == "--execute" ]] && DRY_RUN=false
echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')"
```

**Cloud SQL to RDS/Aurora** — include only if `has_databases`:

Read `preferences.json` → `design_constraints.db_size.value` to select the migration tool:
Expand All @@ -91,23 +110,15 @@ Read `preferences.json` → `design_constraints.db_size.value` to select the mig
Generate the script with conditional branches based on `db_size`:

```bash
#!/usr/bin/env bash
set -euo pipefail
# Cloud SQL → RDS data migration
# Usage: ./02-migrate-data.sh [--execute]
#
# Tool selection based on database size (preferences.json design_constraints.db_size.value):
# <10GB: pg_dump/pg_restore
# 10-500GB: pgcopydb (parallel copy, 3-5x faster than pg_dump)
# >500GB: AWS DMS recommended — see README-DMS.md if generated
# unknown: pgcopydb (safer default at unknown scale)
# TODO: Verify database size before running — wrong tool choice can exceed your maintenance window.

DRY_RUN=true
[[ "${1:-}" == "--execute" ]] && DRY_RUN=false

echo "=== Database Migration: Cloud SQL → RDS ==="
echo "Mode: $([ "$DRY_RUN" = true ] && echo 'DRY RUN' || echo 'EXECUTE')"

SOURCE_HOST="" # TODO: Set Cloud SQL IP
TARGET_HOST="" # From terraform output database_endpoint
Expand Down Expand Up @@ -201,13 +212,25 @@ echo "=== Verification ==="
echo "TODO: Compare row counts between source and target"
```

**BigQuery to S3** — include only if `has_databases`:
**BigQuery — specialist-deferred (no automated AWS target)** — include only if `has_bigquery`:

Design marked every `google_bigquery_*` resource as **`Deferred — specialist engagement`** with
`no_automated_aws_target: true` (see `references/phases/design/design-infra.md` → BigQuery specialist gate).
Therefore **do not** generate BigQuery export, copy, or load steps, and **do not** name an AWS analytics or
warehouse target (no Athena, Redshift, Glue, EMR, Lake Formation, or a prescribed "data lake on S3").
Emit only the deferral notice below:

```bash
# BigQuery → S3 data export
# TODO: Configure BigQuery dataset and S3 bucket
# bq extract --destination_format=PARQUET 'dataset.table' 'gs://bucket/export/'
# aws s3 sync gs://bucket/export/ s3://target-bucket/import/
# BigQuery — AWS target deliberately NOT selected
# This plugin does not choose an AWS analytics or warehouse target for BigQuery
# (no Athena/Redshift/Glue/EMR recommendation, and no prescribed data lake on S3).
# Engage your AWS account team and/or a data analytics migration partner before
# data warehouse, lake, SQL analytics, or BI cutover planning — query patterns,
# data volumes, ETL/ELT, and downstream consumers must be assessed by specialists.
# No BigQuery migration steps are generated here by design.
echo "BigQuery: AWS target deferred — specialist engagement required."
Comment thread
leon1418 marked this conversation as resolved.
echo " Next step: engage your AWS account team and/or a data analytics migration partner."
echo " No BigQuery export or load steps are generated by this skill."
```

**Firestore to DynamoDB** — include only if `has_databases`:
Expand Down Expand Up @@ -383,6 +406,9 @@ After generating all scripts, verify the following quality rules:
3. All scripts include verification steps
4. All scripts are numbered for execution order
5. All TODO markers are clearly marked with context
6. No script prescribes an AWS analytics or warehouse target for BigQuery (no Athena, Redshift, Glue,
EMR, Lake Formation, or "data lake on S3", and no `bq extract` / export steps) — BigQuery resources
carry only the specialist-engagement deferral notice, consistent with `no_automated_aws_target: true`

## Phase Completion

Expand Down
Loading