Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,4 +166,4 @@ jobs:
run: go mod download

- name: Run integration tests
run: cd integration && go test -v -tags integration -count=1 ./... -timeout 120s
run: cd integration && go test -v -tags integration -count=1 ./... -timeout 300s
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,9 @@ seedstorm gaps \
- **Enum coverage** — every enum value appears at least `--rows` times, independently per column
- **AI enrichment** — Gemini rewrites faker hints for domain-meaningful data; supply `--prompt` for richer context
- **Gap analysis** — `gaps` shows which tables are empty with row counts and FK context; `--fill` seeds only the empty ones
- **Schema clone for test DBs** — copy schema-only structure from one connected Postgres/MySQL database into another matching local target, then seed it with safe fake data
- **Interactive TUI** — wizard for table selection, global config, self-reference depth, per-table row volumes, and review before seeding
- **Web UI** — `seedstorm serve` exposes an interactive graph workspace with click-to-select tables, self-reference depth, per-table row overrides, live SSE job logs, multi-DB session switcher, and connection presets in `localStorage`
- **Web UI** — `seedstorm serve` exposes an interactive graph workspace with click-to-select tables, self-reference depth, per-table row overrides, truncate-only runs (`Rows = 0` + `truncate`), live SSE job logs, schema clone between connected DBs, multi-DB session switcher, and connection presets in `localStorage`
- **Dry-run** — preview the seed plan and INSERT SQL without touching the database
- **Export** — generate fake data as YAML, JSON, or SQL without a live connection

Expand Down
44 changes: 43 additions & 1 deletion docs/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ Every seedstorm command, with all flags and examples.
- [`gaps`](#gaps) — find and fill empty tables
- [`generate`](#generate) — generate data without a DB connection
- [`export`](#export) — convert data between formats
- [`clone-schema`](#clone-schema) — copy schema structure into another DB
- [`serve`](#serve) — local web UI for every feature
- [`version`](#version) / [`completion`](#completion)

Expand Down Expand Up @@ -271,6 +272,46 @@ seedstorm export --data data.yaml --format csv --out data.csv

---

## `clone-schema`

Copies schema-only table structure from a source database into a target database of the same engine. This is designed for local/test database setup before running `seed`; it recreates the metadata seedstorm understands: tables, columns, nullability, PKs, FKs, single-column UNIQUE constraints, enum values, and simple CHECK constraints.

```bash
seedstorm clone-schema \
--source-db postgres \
--source-dsn "postgres://user:pass@prod.example/app" \
--target-db postgres \
--target-dsn "postgres://seedstorm:seedstorm@localhost:5432/testdb"

# Replace existing target tables
seedstorm clone-schema \
--source-db mysql \
--source-dsn "user:pass@tcp(staging.example:3306)/app" \
--target-db mysql \
--target-dsn "seedstorm:seedstorm@tcp(localhost:3306)/testdb" \
--drop-existing

# Preview generated DDL
seedstorm clone-schema \
--source-db postgres \
--source-dsn "postgres://..." \
--target-db postgres \
--target-dsn "postgres://..." \
--dry-run
```

| Flag | Default | Description |
|------|---------|-------------|
| `--source-db` / `$SEEDSTORM_SOURCE_DB` | `postgres` | Source database type |
| `--source-dsn` / `$SEEDSTORM_SOURCE_DSN` | — | Source connection string (required) |
| `--target-db` / `$SEEDSTORM_TARGET_DB` | `postgres` | Target database type |
| `--target-dsn` / `$SEEDSTORM_TARGET_DSN` | — | Target connection string (required) |
| `--drop-existing` | false | Drop target tables before creating the cloned schema |
| `--dry-run` / `-n` | false | Print generated DDL, do not execute |
| `--interactive` / `-i` | false | Confirm the clone in the terminal UI |

---

## `serve`

Starts a local web UI that exposes every seedstorm feature behind an interactive graph workspace. The UI is bundled into the binary via `go:embed` — no extra files to ship.
Expand All @@ -284,7 +325,8 @@ SEEDSTORM_ADDR=127.0.0.1:9000 seedstorm serve
What the UI gives you:

- **Workspace** — Cytoscape DAG of every table; click to select, non-nullable parents auto-lock as a dependency closure (mirrors the TUI). The selected-table panel lets you override row counts per table for **Seed**, **Fill empty**, and workspace **Generate** runs while `Rows` remains the default. `Self-ref` controls bounded generated depth for self-referential FK chains. Live SSE log stream + status pill.
- **Connection management** — multi-session: hold several DBs open in one browser and switch from a topbar dropdown. Saved connection presets in `localStorage` with optional password (eye-icon reveal, closed by default). Passwords are kept in process memory only on the server.
- **Seed controls** — `Rows = 0` with `truncate` enabled is a truncate-only run for the selected scope, including auto-required parents. No rows are generated afterward.
- **Connection management** — multi-session: hold several DBs open in one browser and switch from a topbar dropdown. Saved connection presets in `localStorage` with optional password (eye-icon reveal, closed by default). Passwords are kept in process memory only on the server. The workspace can clone schema from the active connection into another matching connected database.
- **Standalone tools** — `/generate`, `/enrich`, `/export` mirror the CLI commands as forms.

| Flag | Default | Description |
Expand Down
4 changes: 2 additions & 2 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,9 +118,9 @@ All tests run automatically on every PR via GitHub Actions (`.github/workflows/p
| `validate` | Directory/file structure via structlint |
| `test` | `go test ./...` + `make build` |
| `lint` | `golangci-lint` |
| `integration` | Full 29-table suite on Postgres 15 + MySQL 8 |
| `integration` | Full 29-table suite plus schema-clone smoke tests on the configured Postgres/MySQL pair |

The integration job in CI uses `--timeout 120s`. Use `300s` locally when running both engines back-to-back.
The integration job in CI uses `--timeout 300s` across the database-version matrix. Use the same timeout locally when running both engines back-to-back.

### Supported database versions

Expand Down
176 changes: 176 additions & 0 deletions integration/integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,114 @@ func assertHardSelfRefSeeded(t *testing.T, conn *sql.DB) {
}
}

func filterTables(tables []db.Table, names ...string) []db.Table {
want := make(map[string]bool, len(names))
for _, name := range names {
want[name] = true
}
var out []db.Table
for _, tbl := range tables {
if want[tbl.Name] {
out = append(out, tbl)
}
}
return out
}

func cloneSmokeSchema(t *testing.T, driver string, conn *sql.DB) {
t.Helper()
if driver == postgresDriver {
execSQL(t, conn, `
DROP TABLE IF EXISTS clone_orders;
DROP TABLE IF EXISTS clone_users;
CREATE TABLE clone_users (
id integer PRIMARY KEY,
email varchar(255) NOT NULL UNIQUE,
status varchar(20) NOT NULL CHECK (status IN ('active', 'blocked'))
);
CREATE TABLE clone_orders (
id integer PRIMARY KEY,
user_id integer NOT NULL REFERENCES clone_users(id),
total integer NOT NULL CHECK (total BETWEEN 1 AND 500)
);
`)
return
}
execSQL(t, conn, `
SET FOREIGN_KEY_CHECKS=0;
DROP TABLE IF EXISTS clone_orders;
DROP TABLE IF EXISTS clone_users;
SET FOREIGN_KEY_CHECKS=1;
CREATE TABLE clone_users (
id integer PRIMARY KEY,
email varchar(255) NOT NULL UNIQUE,
status varchar(20) NOT NULL CHECK (status IN ('active', 'blocked'))
);
CREATE TABLE clone_orders (
id integer PRIMARY KEY,
user_id integer NOT NULL,
total integer NOT NULL CHECK (total BETWEEN 1 AND 500),
FOREIGN KEY (user_id) REFERENCES clone_users(id)
);
`)
}

func dropCloneSmokeSchema(t *testing.T, driver string, conn *sql.DB) {
t.Helper()
if driver == postgresDriver {
execSQL(t, conn, `DROP TABLE IF EXISTS clone_orders; DROP TABLE IF EXISTS clone_users;`)
return
}
execSQL(t, conn, `SET FOREIGN_KEY_CHECKS=0; DROP TABLE IF EXISTS clone_orders; DROP TABLE IF EXISTS clone_users; SET FOREIGN_KEY_CHECKS=1;`)
}

func assertCloneSchemaCanSeed(t *testing.T, driver string, conn *sql.DB, tables []db.Table) {
t.Helper()
s := &schema.Schema{Tables: make(map[string]schema.Table, len(tables))}
for _, tbl := range tables {
st := schema.Table{Columns: make(map[string]schema.Column, len(tbl.Columns))}
for _, col := range tbl.Columns {
sc := schema.Column{
Type: col.Type,
PK: col.IsPK,
Nullable: col.IsNullable,
Faker: faker.MapColumnToFaker(driver, col),
}
if col.Name == "email" {
sc.Faker = "email"
}
if col.FK != nil {
sc.FK = fmt.Sprintf("%s.%s", col.FK.TableName, col.FK.ColumnName)
}
st.Columns[col.Name] = sc
}
s.Tables[tbl.Name] = st
}
g := graph.Build(s)
order, err := g.TopologicalSort()
if err != nil {
t.Fatalf("topological sort cloned schema: %v", err)
}
data, err := faker.Generate(s, order, 5, 0, conn, driver)
if err != nil {
t.Fatalf("generate cloned schema data: %v", err)
}
for _, tableName := range order {
for _, row := range data[tableName] {
query, values := db.BuildInsert(tableName, row, driver)
if _, err := conn.ExecContext(context.Background(), query, values...); err != nil {
t.Fatalf("insert cloned table %s: %v", tableName, err)
}
}
}
if got := countRows(t, conn, "clone_users"); got == 0 {
t.Fatal("clone_users was not seeded")
}
if got := countRows(t, conn, "clone_orders"); got == 0 {
t.Fatal("clone_orders was not seeded")
}
}

// buildAndSeed runs the full introspect → build schema → generate → seed pipeline.
// It prints a summary at the end (not per-row during insert).
func buildAndSeed(t *testing.T, label, driver, dsn string, conn *sql.DB) map[string][]map[string]interface{} {
Expand Down Expand Up @@ -1465,6 +1573,40 @@ func TestPostgresIntegration(t *testing.T) {
})
}

func TestPostgresSchemaCloneDDL(t *testing.T) {
dsn := envOrDefault("POSTGRES_DSN", postgresDSN)
conn := openDB(t, postgresDriver, dsn)
defer conn.Close()
dropCloneSmokeSchema(t, postgresDriver, conn)
cloneSmokeSchema(t, postgresDriver, conn)

sourceTables, err := db.Introspect(postgresDriver, dsn)
if err != nil {
t.Fatalf("introspect source: %v", err)
}
sourceTables = filterTables(sourceTables, "clone_users", "clone_orders")
if len(sourceTables) != 2 {
t.Fatalf("source tables = %d, want 2", len(sourceTables))
}
stmts, err := db.BuildSchemaDDL(sourceTables, postgresDriver, true)
if err != nil {
t.Fatalf("BuildSchemaDDL: %v", err)
}
if err := db.ExecSchemaDDL(context.Background(), conn, postgresDriver, stmts); err != nil {
t.Fatalf("ExecSchemaDDL: %v", err)
}
cloned, err := db.Introspect(postgresDriver, dsn)
if err != nil {
t.Fatalf("introspect cloned: %v", err)
}
cloned = filterTables(cloned, "clone_users", "clone_orders")
if len(cloned) != 2 {
t.Fatalf("cloned tables = %d, want 2", len(cloned))
}
assertCloneSchemaCanSeed(t, postgresDriver, conn, cloned)
dropCloneSmokeSchema(t, postgresDriver, conn)
}

// ── MySQL ──────────────────────────────────────────────────────────────────────

func TestMySQLIntegration(t *testing.T) {
Expand Down Expand Up @@ -2667,6 +2809,40 @@ func TestMySQLIntegration(t *testing.T) {
})
}

func TestMySQLSchemaCloneDDL(t *testing.T) {
dsn := envOrDefault("MYSQL_DSN", mysqlDSN)
conn := openDB(t, mysqlDriver, dsn)
defer conn.Close()
dropCloneSmokeSchema(t, mysqlDriver, conn)
cloneSmokeSchema(t, mysqlDriver, conn)

sourceTables, err := db.Introspect(mysqlDriver, dsn)
if err != nil {
t.Fatalf("introspect source: %v", err)
}
sourceTables = filterTables(sourceTables, "clone_users", "clone_orders")
if len(sourceTables) != 2 {
t.Fatalf("source tables = %d, want 2", len(sourceTables))
}
stmts, err := db.BuildSchemaDDL(sourceTables, mysqlDriver, true)
if err != nil {
t.Fatalf("BuildSchemaDDL: %v", err)
}
if err := db.ExecSchemaDDL(context.Background(), conn, mysqlDriver, stmts); err != nil {
t.Fatalf("ExecSchemaDDL: %v", err)
}
cloned, err := db.Introspect(mysqlDriver, dsn)
if err != nil {
t.Fatalf("introspect cloned: %v", err)
}
cloned = filterTables(cloned, "clone_users", "clone_orders")
if len(cloned) != 2 {
t.Fatalf("cloned tables = %d, want 2", len(cloned))
}
assertCloneSchemaCanSeed(t, mysqlDriver, conn, cloned)
dropCloneSmokeSchema(t, mysqlDriver, conn)
}

// ── Gap Analysis ──────────────────────────────────────────────────────────────
//
// These tests exercise the gaps / GenerateFiltered feature end-to-end:
Expand Down
83 changes: 83 additions & 0 deletions internal/cli/clone_schema.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
package cli

import (
"context"
"fmt"
"strings"

"github.com/AxeForging/seedstorm/internal/db"
"github.com/AxeForging/seedstorm/internal/logging"
"github.com/AxeForging/seedstorm/internal/tui"
"github.com/urfave/cli/v3"
)

func cloneSchemaCmd() *cli.Command {
return &cli.Command{
Name: "clone-schema",
Usage: "Copy schema-only structure from one database into another",
Description: `Introspects a source database and creates matching tables in a target database.
This is same-engine schema cloning for local/test databases, not a lossless migration tool.`,
Flags: []cli.Flag{
&cli.StringFlag{
Name: "source-db",
Usage: "Source database type: mysql or postgres",
Value: "postgres",
Sources: cli.EnvVars("SEEDSTORM_SOURCE_DB"),
},
&cli.StringFlag{
Name: "source-dsn",
Usage: "Source data source name",
Required: true,
Sources: cli.EnvVars("SEEDSTORM_SOURCE_DSN"),
},
&cli.StringFlag{
Name: "target-db",
Usage: "Target database type: mysql or postgres",
Value: "postgres",
Sources: cli.EnvVars("SEEDSTORM_TARGET_DB"),
},
&cli.StringFlag{
Name: "target-dsn",
Usage: "Target data source name",
Required: true,
Sources: cli.EnvVars("SEEDSTORM_TARGET_DSN"),
},
&cli.BoolFlag{
Name: "drop-existing",
Usage: "Drop existing target tables before creating the cloned schema",
},
&cli.BoolFlag{
Name: "dry-run",
Aliases: []string{"n"},
Usage: "Print DDL without executing it",
},
&cli.BoolFlag{
Name: "interactive",
Aliases: []string{"i"},
Usage: "Review and confirm the clone in the terminal UI",
},
},
Action: func(ctx context.Context, cmd *cli.Command) error {
log := logging.Log
sourceType := normalizeDBType(cmd.String("source-db"))
targetType := normalizeDBType(cmd.String("target-db"))
opts := db.CloneOptions{
DropExisting: cmd.Bool("drop-existing"),
DryRun: cmd.Bool("dry-run"),
}
if cmd.Bool("interactive") {
return tui.RunClone(ctx, sourceType, cmd.String("source-dsn"), targetType, cmd.String("target-dsn"), opts)
}
result, err := db.CloneSchema(ctx, sourceType, cmd.String("source-dsn"), targetType, cmd.String("target-dsn"), opts)
if err != nil {
return err
}
if opts.DryRun {
fmt.Println(strings.Join(result.Statements, ";\n") + ";")
return nil
}
log.Info().Int("tables", result.Tables).Int("statements", len(result.Statements)).Msg("Schema cloned")
return nil
},
}
}
9 changes: 9 additions & 0 deletions internal/cli/helpers_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,15 @@ func TestNormalizeDBType(t *testing.T) {
}
}

func TestCommandsIncludesCloneSchema(t *testing.T) {
for _, cmd := range Commands() {
if cmd.Name == "clone-schema" {
return
}
}
t.Fatal("clone-schema command is not registered")
}

// --- Tests from batch-inserts branch ---

func TestBuildInsert_Postgres(t *testing.T) {
Expand Down
1 change: 1 addition & 0 deletions internal/cli/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ func Commands() []*cli.Command {
gapsCmd(),
generateCmd(),
exportCmd(),
cloneSchemaCmd(),
serveCmd(),
versionCmd(),
completionCmd(),
Expand Down
Loading
Loading