diff --git a/README.md b/README.md index a422e6f..af25133 100644 --- a/README.md +++ b/README.md @@ -1,140 +1,85 @@ - - # MockSQL -[![Backend CI](https://github.com/skadel/mocksql/actions/workflows/backend-ci.yml/badge.svg)](https://github.com/skadel/mocksql/actions/workflows/backend-ci.yml) -[![Frontend CI](https://github.com/skadel/mocksql/actions/workflows/frontend-ci.yml/badge.svg)](https://github.com/skadel/mocksql/actions/workflows/frontend-ci.yml) [![PyPI version](https://img.shields.io/pypi/v/mocksql)](https://pypi.org/project/mocksql/) [![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE) -**A native unit-testing layer for data engineers.** MockSQL takes a `.sql` file, automatically generates test data via LLM, runs it locally on DuckDB (zero cost on BigQuery), assigns an argued verdict to each test, and suggests the edge cases you haven't covered. - -https://github.com/user-attachments/assets/ce95cacb-c245-432a-8e4b-6ffc76507980 - -

Full flow: pick a .sql model → MockSQL generates the input data, runs the query locally on DuckDB, and returns an argued verdict per test — plus suggestions for the edge cases you haven't covered.

- -MockSQL never hands raw SQL to the LLM. It first parses the query with **SQLGlot** to extract the used columns, filters, and JOINs — then feeds those constraints to the LLM as structured context. The generated data is then executed on **DuckDB**: if a CTE returns 0 rows, MockSQL identifies which one and automatically re-runs generation until it gets non-empty results. Once the tests are generated, a **contextual chat** lets you refine, add, or edit them directly in natural language — anchored to a specific test or to the whole model. - -Existing SQL mocking libraries ask you to **write the test data by hand**. MockSQL takes the opposite approach: - -| | SQL mocking libraries | MockSQL | -|---|---|---| -| Test data | Written manually | **Auto-generated** by LLM | -| Coverage | No detection | **6 axes** (NULL, empty, ties…) + suggestions | -| Test quality | No evaluation | **LLM verdict** (good / weak / incorrect) | -| Interface | Python library | **Dedicated UI** (GenerateView → TestsView) | -| SQL engine | One connector per DB | **Unified DuckDB** — no BigQuery cost | - -MockSQL comes in two modes: -- **CLI** (`mocksql`) — standalone use directly on your local `.sql` files -- **Web Hub** — full interface with history, verdicts, coverage, and collaboration - ---- - -## Quick start - -Whatever your warehouse, MockSQL generates data with an LLM and runs every test locally on **DuckDB** — zero warehouse cost. The base install runs entirely on DuckDB; the source-warehouse connectors are heavy (`pyarrow`, `grpc`, …) and only needed to **profile/import** real tables, so they ship as optional extras (`mocksql[bigquery]`, `mocksql[snowflake]`, `mocksql[all]`). Run an import without the matching extra and MockSQL fails fast with the exact command. - -Data generation always uses an LLM (Gemini via Vertex AI by default), so set `VERTEX_PROJECT` in every setup below. Pick your source: - -### BigQuery - -```bash -pip install mocksql[bigquery] -gcloud auth application-default login # GCP auth for schema import + Gemini -export VERTEX_PROJECT= - -mocksql init # dialect: bigquery (the default) -mocksql generate models/orders.sql -``` - -Full GCP/IAM setup (roles, service accounts, CI) → **[docs/quickstart.md](docs/quickstart.md)** +MockSQL generates SQL unit-test fixtures with an LLM, evaluates them on local +DuckDB, and stores replayable tests. It is released under the **MIT** license. +Generated test data is never executed on BigQuery, Snowflake, or another source +warehouse. -### Snowflake +## Install and initialize ```bash -pip install mocksql[snowflake] -mocksql init # choose dialect: snowflake +pip install mocksql # CLI and local DuckDB execution +pip install mocksql[bigquery] # BigQuery schema import/profiling +mocksql init ``` -Put credentials in a **gitignored** `.env` at your project root: +`mocksql init` supports `--dialect`, `--models-path`, `--llm-provider`, +`--path`, `--force`, and `--non-interactive`. The LLM provider determines its +credentials: ```dotenv -VERTEX_PROJECT= # LLM (Gemini via Vertex AI) -SNOWFLAKE_ACCOUNT= # ORG-ACCOUNT, or .. -SNOWFLAKE_USER= -SNOWFLAKE_PASSWORD= -SNOWFLAKE_WAREHOUSE= -SNOWFLAKE_DATABASE= -# SNOWFLAKE_ROLE= # optional (required on some accounts) -``` +# Vertex AI / Gemini +VERTEX_PROJECT=my-gcp-project +GOOGLE_CLOUD_LOCATION=us-central1 -```bash -mocksql generate models/orders.sql +# Or OpenAI +OPENAI_API_KEY=sk-... ``` -MockSQL imports the table schemas from Snowflake (`INFORMATION_SCHEMA`), generates data, and runs the tests on DuckDB. Snowflake idioms (`IFF`, `TO_TIMESTAMP_NTZ`, `TO_CHAR`, `LISTAGG`, `NUMBER(p,s)`…) are transpiled automatically. - -### dbt - -A dbt model isn't flat SQL (Jinja: `{{ ref }}`, macros). MockSQL reads the **compiled** SQL via a `dbt:` block in `mocksql.yml`, then imports schemas and runs on DuckDB like any other model: +For a BigQuery source also set `BQ_TEST_PROJECT` (or rely on its +`VERTEX_PROJECT` fallback) and authenticate with Application Default Credentials +or `GOOGLE_APPLICATION_CREDENTIALS`. ```bash -pip install mocksql[bigquery] # or [snowflake] — match your dbt target -cd my_dbt_project -dbt compile # Jinja → flat SQL with real table names -dbt run --select +my_mart # (marts only) materialize parent models -mocksql generate models/marts/my_mart.sql --config mocksql.yml +mocksql generate models/orders.sql +mocksql test --model orders ``` -```yaml -# mocksql.yml — set the dbt block + the dialect of your dbt target -dialect: bigquery # bigquery | snowflake | duckdb (must match your dbt target) -models_path: ./models -dbt: - project_dir: . # folder containing dbt_project.yml -llm: - provider: vertexai -``` +See [docs/quickstart.md](docs/quickstart.md) for credentials, cache behavior, +and BigQuery Sandbox/billing details. -Full recipe (compile profile, materializing parents, scratch DuckDB) → **[docs/quickstart-dbt.md](docs/quickstart-dbt.md)** +## Connector status ---- +| Source dialect | CLI generation | Notes | +|---|---|---| +| BigQuery | Supported | Imports missing schemas with `mocksql[bigquery]`; `--profile` issues real BigQuery queries. | +| DuckDB | Cache-only | Local test execution works; prepare `schema_cache` before generation. | +| PostgreSQL | Cache-only | Validation is available, but this generation flow does not import Postgres schemas. | +| Snowflake | Supported with an explicit schema refresh | Validation/transpilation work; run `mocksql refresh-schemas --table database.schema.table` before generation. | +| Trino | Partial | Validation and `refresh-schemas` support exist; generation still requires cached schemas. | -## Project structure +## dbt status -``` -back/ # FastAPI + LangGraph + CLI - cli/ # mocksql CLI (main.py, generate.py) - ui/ # mocksql-ui package (server + React assets) -front/ # React 18 + TypeScript + Redux (Web Hub) -examples/ # Example MockSQL projects -docs/ # Documentation - quickstart.md # Full setup (GCP, IAM, CLI, Web UI) - quickstart-dbt.md # Testing a dbt-DuckDB project - workflow-query-generation.md # Flow frontend → backend → DuckDB -``` +MockSQL resolves dbt models through `manifest.json` and reads their compiled SQL +from `target/compiled/`. It never treats the dbt manifest as a schema source. ---- +- dbt-BigQuery: supported, including BigQuery schema import. +- dbt-DuckDB: supported when `schema_cache` has been prepared. +- dbt-Snowflake: supported after explicitly refreshing the referenced schemas + into `schema_cache`; compiled-SQL resolution and validation work. -## Contributing +Full setup: [docs/quickstart-dbt.md](docs/quickstart-dbt.md). -```bash -make check-all # back (style + tests) + front (vitest) — full validation -``` +## Development -Backend only: +The package metadata is in [back/pyproject.toml](back/pyproject.toml): version +`0.2.1`, Python `>=3.11,<3.14`, and MIT license. ```bash cd back -make style # lint + format check + dead code (vulture) -make format # auto-format and auto-fix -make test # pytest -make check # style + test +poetry run mocksql --help +make check ``` -Pre-commit hook (recommended): +### Snowflake schema import -```bash -pip install pre-commit && pre-commit install -``` +With `dialect: snowflake` and `mocksql[snowflake]`, both `mocksql generate` and +`mocksql refresh-schemas` read schemas from Snowflake `INFORMATION_SCHEMA`. They +require `SNOWFLAKE_ACCOUNT`, `SNOWFLAKE_USER`, `SNOWFLAKE_PASSWORD`, +`SNOWFLAKE_WAREHOUSE`, and `SNOWFLAKE_DATABASE`; they never require or call +BigQuery. Snowflake profiling is not available yet: `generate --profile` reports +that limitation and continues without profiling rather than falling back to +BigQuery. diff --git a/back/build_query/assertion_modifier.py b/back/build_query/assertion_modifier.py index 3f94fab..9a4dc06 100644 --- a/back/build_query/assertion_modifier.py +++ b/back/build_query/assertion_modifier.py @@ -69,7 +69,7 @@ async def modify_assertions(state: QueryState): updated_fields = json.loads(content.strip()) except Exception as exc: if is_vertex_permission_error(exc): - error_msg = format_vertex_permission_message(get_llm_model()) + error_msg = format_vertex_permission_message(get_llm_model(), exc) return { "messages": [ AIMessage( diff --git a/back/build_query/examples_generator.py b/back/build_query/examples_generator.py index 5f3c246..83bdb79 100644 --- a/back/build_query/examples_generator.py +++ b/back/build_query/examples_generator.py @@ -932,7 +932,7 @@ async def generate_examples(state: QueryState): } except Exception as exc: if is_vertex_permission_error(exc): - error_msg = format_vertex_permission_message(get_llm_model()) + error_msg = format_vertex_permission_message(get_llm_model(), exc) return { "messages": [ AIMessage( diff --git a/back/cli/generate.py b/back/cli/generate.py index 320db95..2f5d2c2 100644 --- a/back/cli/generate.py +++ b/back/cli/generate.py @@ -32,6 +32,16 @@ logger = logging.getLogger(__name__) +def cache_miss_message(dialect: str) -> str: + """Explain schema-cache misses without selecting an unrelated connector.""" + if dialect in {"duckdb", "postgres", "postgresql"}: + return ( + f"[ERROR] {dialect} generation is cache-only: missing schemas must be " + "prepared in schema_cache before running `mocksql generate`." + ) + return "[ERROR] No schema importer is available for this dialect." + + # ── Config ──────────────────────────────────────────────────────────────────── @@ -65,6 +75,36 @@ def read_sql( return clean if clean is not None else sql +def resolve_model_sql( + model: Path, config: Path, cfg: dict, dbt_project: Any = None +) -> tuple[str, str, bool]: + """Return the model identifier and SQL chosen for a CLI generation. + + dbt models always use their compiled SQL; ordinary files retain the existing + preprocessor-aware source read path. Keeping this selection isolated makes + the ``--config``/dbt boundary deterministic and directly testable. + """ + dialect = cfg.get("dialect", "bigquery") + models_path_str = cfg.get("models_path", "./models") + models_base = (config.parent / models_path_str).resolve() + try: + model_name = model.resolve().relative_to(models_base).with_suffix("").as_posix() + except ValueError: + model_name = model.stem + + is_dbt_model = bool(dbt_project and dbt_project.is_dbt_model(model_name)) + + if is_dbt_model: + compiled = dbt_project.compiled_sql_for_model(model_name) + return model_name, extract_select_statement(compiled, dialect) or compiled, True + + return ( + model_name, + read_sql(model, cfg.get("preprocessor_fn"), config.parent, dialect), + False, + ) + + # ── State builder ───────────────────────────────────────────────────────────── @@ -688,27 +728,18 @@ async def run_generate( cache_path = str( config.parent / cfg.get("schema_cache", ".mocksql/schema_cache.json") ) - preprocessor_fn = cfg.get("preprocessor_fn") - # dbt connector : si un bloc `dbt:` est configuré, MockSQL lit le SQL **compilé** - # (refs résolus, macros rendues) et infère les schémas amont depuis le manifest — - # sans jamais interroger l'entrepôt. - dbt_project = storage_config.get_dbt_project() - models_path_str = cfg.get("models_path", "./models") - models_base = (config.parent / models_path_str).resolve() - try: - model_name = model.resolve().relative_to(models_base).with_suffix("").as_posix() - except ValueError: - model_name = model.stem + # dbt connector: read compiled SQL (resolved refs, rendered macros). The + # manifest is not a schema source; resolution below uses the schema cache + # and the selected warehouse-import path. + dbt_project = storage_config.get_dbt_project(config) + model_name, sql, is_dbt_model = resolve_model_sql(model, config, cfg, dbt_project) + models_base = (config.parent / cfg.get("models_path", "./models")).resolve() # Step 1 — read SQL (DECLARE/SET preambles are stripped inside read_sql) typer.echo(f"Reading {model}...") - if dbt_project and dbt_project.is_dbt_model(model_name): + if is_dbt_model: typer.echo(f"[dbt] SQL compilé depuis le manifest pour '{model_name}'.") - compiled = dbt_project.compiled_sql_for_model(model_name) - sql = extract_select_statement(compiled, dialect) or compiled - else: - sql = read_sql(model, preprocessor_fn, config.parent, dialect) # Step 1.5 — fail fast if the query requires generating too many rows from build_query.constraint_simplifier import ( @@ -731,7 +762,14 @@ async def run_generate( ref_names = [".".join(p for p in [r.catalog, r.db, r.name] if p) for r in refs] typer.echo(f"Found {len(refs)} source table(s): {ref_names}") - billing_project = os.getenv("BQ_TEST_PROJECT") or os.getenv("VERTEX_PROJECT") + # BigQuery is a source connector, not a default for every SQL dialect. + # Keep its resolution local to the BigQuery branch so a Snowflake run never + # requires nor accidentally uses BQ_TEST_PROJECT. + billing_project = ( + os.getenv("BQ_TEST_PROJECT") or os.getenv("VERTEX_PROJECT") + if dialect == "bigquery" + else None + ) # Step 3 — resolve schemas via cache local + fetch BigQuery des manquants. # En mode dbt, le SQL est déjà compilé (refs résolus en noms réels) ; la résolution @@ -741,24 +779,42 @@ async def run_generate( if missing: typer.echo(f"Fetching schema for: {missing}") - if not billing_project: - typer.echo( - "[ERROR] BQ_TEST_PROJECT not set. Cannot fetch schemas from BigQuery. " - "Set it in your .env or shell environment." - ) - raise typer.Exit(1) + if dialect == "snowflake": + from build_query.schema_fetcher import fetch_tables_schema_snowflake + from models.env_variables import validate_snowflake_env - unqualified = [r for r in missing if not validate_bq_ref(r)] - if unqualified: - typer.echo( - f"[WARN] Unqualified table refs (need project.dataset.table): {unqualified}" - ) + try: + validate_snowflake_env() + except RuntimeError as exc: + typer.echo(f"[ERROR] {exc}", err=True) + raise typer.Exit(1) + schema_rows, failed = await fetch_tables_schema_snowflake(missing) + partitions = {} + elif dialect == "bigquery": + if not billing_project: + typer.echo( + "[ERROR] BQ_TEST_PROJECT not set. Cannot fetch schemas from BigQuery. " + "Set it in your .env or shell environment." + ) + raise typer.Exit(1) - to_fetch = [r for r in missing if validate_bq_ref(r)] - if to_fetch: - schema_rows, failed, partitions = await fetch_tables_schema( - to_fetch, billing_project - ) + unqualified = [r for r in missing if not validate_bq_ref(r)] + if unqualified: + typer.echo( + f"[WARN] Unqualified table refs (need project.dataset.table): {unqualified}" + ) + + to_fetch = [r for r in missing if validate_bq_ref(r)] + schema_rows, failed, partitions = [], [], {} + if to_fetch: + schema_rows, failed, partitions = await fetch_tables_schema( + to_fetch, billing_project + ) + else: + typer.echo(cache_miss_message(dialect), err=True) + raise typer.Exit(1) + + if schema_rows or failed: if failed: typer.echo(f"[WARN] Could not fetch: {[f['table'] for f in failed]}") if schema_rows: @@ -785,21 +841,30 @@ async def run_generate( # Step 3.5 — profile (optional) profile_data: dict | None = None if profile: - if not billing_project: + if dialect == "snowflake": + # Profiling is a separate feature and must never silently fall + # through to BigQuery for a Snowflake source. + typer.echo( + "[WARN] --profile is not yet supported for Snowflake; continuing without profiling." + ) + elif not billing_project: typer.echo( "[ERROR] --profile requires BQ_TEST_PROJECT. " "Set it in your .env or shell environment." ) raise typer.Exit(1) - typer.echo("Profiling tables on BigQuery (this may take a moment)...") - try: - profile_data = _run_profile_bq(schemas, sql, dialect, billing_project) - typer.echo( - f"[OK] Profile complete ({len(profile_data.get('tables', {}))} table(s), " - f"{len(profile_data.get('joins', []))} join(s))." - ) - except Exception as exc: - typer.echo(f"[WARN] Profiling failed: {exc}. Continuing without profile.") + else: + typer.echo("Profiling tables on BigQuery (this may take a moment)...") + try: + profile_data = _run_profile_bq(schemas, sql, dialect, billing_project) + typer.echo( + f"[OK] Profile complete ({len(profile_data.get('tables', {}))} table(s), " + f"{len(profile_data.get('joins', []))} join(s))." + ) + except Exception as exc: + typer.echo( + f"[WARN] Profiling failed: {exc}. Continuing without profile." + ) # Step 4 — build state + inject schemas into in-memory cache # (model_name / models_base déjà calculés en amont pour la résolution dbt) diff --git a/back/cli/main.py b/back/cli/main.py index f48fba9..8c3aee4 100644 --- a/back/cli/main.py +++ b/back/cli/main.py @@ -103,11 +103,36 @@ def init( "-p", help="Directory where mocksql.yml will be created.", ), + dialect: str | None = typer.Option( + None, "--dialect", help="SQL dialect; skips the interactive prompt." + ), + models_path: str | None = typer.Option( + None, "--models-path", help="SQL models folder; skips the interactive prompt." + ), + llm_provider: str | None = typer.Option( + None, "--llm-provider", help="LLM provider; skips the interactive prompt." + ), + test_dataset: str | None = typer.Option( + None, + "--test-dataset", + help="Deprecated and ignored; generated tests always run in local DuckDB.", + ), + langchain_api_key: str | None = typer.Option( + None, "--langchain-api-key", help="Optional LangSmith API key." + ), + force: bool = typer.Option( + False, "--force", help="Overwrite an existing mocksql.yml without prompting." + ), + non_interactive: bool = typer.Option( + False, + "--non-interactive", + help="Use supplied values or defaults and never prompt.", + ), ) -> None: """Initialize a MockSQL project and generate mocksql.yml.""" config_path = path / CONFIG_FILE - if config_path.exists(): + if config_path.exists() and not force: overwrite = typer.confirm( f"{CONFIG_FILE} already exists. Overwrite?", default=False ) @@ -115,9 +140,10 @@ def init( typer.echo("Aborted.") raise typer.Exit() - dialect = typer.prompt( - f"SQL dialect ({'/'.join(DIALECTS)})", - default="bigquery", + dialect = dialect or ( + "bigquery" + if non_interactive + else typer.prompt(f"SQL dialect ({'/'.join(DIALECTS)})", default="bigquery") ) while dialect not in DIALECTS: typer.echo(f"Invalid dialect. Choose from: {', '.join(DIALECTS)}") @@ -125,11 +151,16 @@ def init( f"SQL dialect ({'/'.join(DIALECTS)})", default="bigquery" ) - models_path = _prompt_models_path(path) + models_path = models_path or ( + "./models" if non_interactive else _prompt_models_path(path) + ) - llm_provider = typer.prompt( - f"LLM provider ({'/'.join(LLM_PROVIDERS)})", - default="vertexai", + llm_provider = llm_provider or ( + "vertexai" + if non_interactive + else typer.prompt( + f"LLM provider ({'/'.join(LLM_PROVIDERS)}),", default="vertexai" + ) ) while llm_provider not in LLM_PROVIDERS: typer.echo(f"Invalid provider. Choose from: {', '.join(LLM_PROVIDERS)}") @@ -137,13 +168,15 @@ def init( f"LLM provider ({'/'.join(LLM_PROVIDERS)})", default="vertexai" ) - test_dataset = typer.prompt( - "BigQuery test dataset (where temp tables are created during validation)", - default="test_dataset", - ) + # Kept as a deprecated non-interactive compatibility option. MockSQL runs + # generated tests in local DuckDB and does not create a BigQuery test dataset. + if test_dataset: + typer.echo("[WARN] --test-dataset is ignored; tests run locally on DuckDB.") - langchain_api_key = ( - typer.prompt( + langchain_api_key = langchain_api_key or ( + None + if non_interactive + else typer.prompt( "LangSmith API key (LANGCHAIN_API_KEY, optional — press Enter to skip)", default="", ).strip() @@ -161,7 +194,6 @@ def init( "provider": llm_provider, }, "schema_cache": ".mocksql/schema_cache.json", - "test_dataset": test_dataset, "langchain_tracing": bool(langchain_api_key), } @@ -190,13 +222,23 @@ def init( asyncio.run(init_db_main()) typer.echo(f"[DB] Database ready at {duckdb_path}") - typer.echo( - "\nRequired environment variables (env var or .env file at project root):\n" - " VERTEX_PROJECT= # Vertex AI / LLM\n" - " GOOGLE_CLOUD_LOCATION=us-central1\n" - " BQ_TEST_PROJECT= # optional, defaults to VERTEX_PROJECT\n" - "\nNext step: mocksql generate " - ) + typer.echo("\nRequired environment variables (shell or .env at the project root):") + if llm_provider == "openai": + typer.echo(" OPENAI_API_KEY= # OpenAI LLM") + else: + typer.echo(" VERTEX_PROJECT= # Vertex AI / Gemini") + typer.echo(" GOOGLE_CLOUD_LOCATION=us-central1") + if dialect == "bigquery": + typer.echo( + " BQ_TEST_PROJECT= # BigQuery schema import/jobs" + ) + typer.echo(" GOOGLE_APPLICATION_CREDENTIALS= # or ADC") + else: + typer.echo( + " `mocksql generate` does not import schemas for this dialect; " + "populate schema_cache first." + ) + typer.echo("\nNext step: mocksql generate ") @app.command() @@ -401,7 +443,8 @@ def _print_test_results(model_results: list) -> None: failed += n_fail skipped += n_skip - icon = "✓" if n_fail == 0 else "✗" + # ASCII keeps the summary readable in legacy Windows code pages. + icon = "[OK]" if n_fail == 0 else "[FAIL]" skip_label = f", {n_skip} skipped" if n_skip else "" typer.echo( f"\n {icon} {mr['model']} ({n_pass}/{len(cases)} passed{skip_label})" @@ -417,8 +460,8 @@ def _print_test_results(model_results: list) -> None: # Attestation de parité warehouse (cf. `mocksql parity`) — informatif, # jamais bloquant. `unverified` reste silencieux (pas de bruit). parity_badge = { - "verified": " [parité ✓]", - "stale": " [parité périmée]", + "verified": " [parity verified]", + "stale": " [parity stale]", }.get(c.get("parity", ""), "") typer.echo(f" [{label}] {title}{parity_badge}") # Description complète en sous-ligne quand elle apporte plus que le titre. @@ -932,7 +975,7 @@ def refresh_schemas( [], "--table", "-t", - help="Re-import only these tables (project.dataset.table). Default: all cached BQ tables.", + help="Re-import only these tables. Use project.dataset.table (BigQuery) or database.schema.table (Snowflake).", ), from_tests: bool = typer.Option( False, @@ -941,10 +984,9 @@ def refresh_schemas( "even those not yet cached. Ce que `mocksql test` exige.", ), ) -> None: - """Re-import schemas from BigQuery to pick up partition info on existing tables.""" + """Re-import cached schemas from the warehouse selected by ``dialect``.""" async def _run() -> None: - from models.env_variables import validate_required_env from build_query.schema_fetcher import fetch_tables_schema, validate_bq_ref from cli.generate import ( load_config, @@ -954,16 +996,54 @@ async def _run() -> None: ) from utils.schema_utils import generate_tables_and_columns_from_project_schema - validate_required_env() - cfg = load_config(config) dialect = cfg.get("dialect", "bigquery") + if dialect != "snowflake": + from models.env_variables import validate_required_env + + validate_required_env() cache_path = str( config.parent / cfg.get("schema_cache", ".mocksql/schema_cache.json") ) cached = load_schema_cache(cache_path) - if dialect == "trino": + if dialect == "snowflake": + from build_query.schema_fetcher import fetch_tables_schema_snowflake + from models.env_variables import validate_snowflake_env + + try: + validate_snowflake_env() + except RuntimeError as exc: + typer.echo(f"[ERROR] {exc}", err=True) + raise typer.Exit(1) + + if tables: + refs = list(tables) + elif from_tests: + from cli.test_runner import collect_test_table_refs + + refs = collect_test_table_refs(config.parent / ".mocksql" / "tests") + if not refs: + typer.echo( + "No tables referenced by saved tests in .mocksql/tests/." + ) + raise typer.Exit() + else: + refs = [ + t["table_name"] + for t in cached + if isinstance(t, dict) and t.get("table_name") + ] + if not refs: + typer.echo( + "No Snowflake tables to import. Pass --table database.schema.table " + "(or schema.table), or run `mocksql generate` first." + ) + raise typer.Exit() + typer.echo(f"Re-importing {len(refs)} table(s) from Snowflake...") + schema_rows, failed = await fetch_tables_schema_snowflake(refs) + partitions = {} + elif dialect == "trino": from build_query.schema_fetcher import fetch_tables_schema_trino if tables: @@ -1046,7 +1126,12 @@ async def _run() -> None: if failed: typer.echo(f"[WARN] Could not fetch: {[f['table'] for f in failed]}") if not schema_rows: - typer.echo("[ERROR] No data returned from BigQuery.", err=True) + source_label = ( + "Snowflake" + if dialect == "snowflake" + else ("Trino" if dialect == "trino" else "BigQuery") + ) + typer.echo(f"[ERROR] No data returned from {source_label}.", err=True) raise typer.Exit(1) new_tables = generate_tables_and_columns_from_project_schema( diff --git a/back/models/env_variables.py b/back/models/env_variables.py index c5a03eb..9697768 100644 --- a/back/models/env_variables.py +++ b/back/models/env_variables.py @@ -36,6 +36,10 @@ def validate_required_env() -> None: + # OpenAI does not use Vertex. The OpenAI factory validates OPENAI_API_KEY + # and returns a provider-specific error if it is absent. + if os.getenv("OPENAI_API_KEY") or os.getenv("OPEN_API_KEY"): + return missing = [ f" • {name} — {desc}" for name, desc in _REQUIRED if not os.getenv(name) ] @@ -82,6 +86,36 @@ def validate_required_env() -> None: SNOWFLAKE_SCHEMA_NAME = os.getenv("SNOWFLAKE_SCHEMA", "PUBLIC") SNOWFLAKE_ROLE = os.getenv("SNOWFLAKE_ROLE", "") + +def validate_snowflake_env() -> None: + """Fail early with the exact Snowflake connection settings to provide.""" + required = { + "SNOWFLAKE_ACCOUNT": "identifiant de compte (ex. ORG-ACCOUNT)", + "SNOWFLAKE_USER": "utilisateur Snowflake", + "SNOWFLAKE_PASSWORD": "mot de passe ou secret d'authentification", + "SNOWFLAKE_WAREHOUSE": "warehouse utilisé pour lire INFORMATION_SCHEMA", + "SNOWFLAKE_DATABASE": "base de données source", + } + missing = [ + f" • {name} — {description}" + for name, description in required.items() + if not os.getenv(name) + ] + if missing: + raise RuntimeError( + "Configuration Snowflake incomplète. Définissez dans votre .env ou shell :\n" + + "\n".join(missing) + + "\n\nExemple :\n" + + " SNOWFLAKE_ACCOUNT=org-account\n" + + " SNOWFLAKE_USER=mocksql\n" + + " SNOWFLAKE_PASSWORD=…\n" + + " SNOWFLAKE_WAREHOUSE=COMPUTE_WH\n" + + " SNOWFLAKE_DATABASE=ANALYTICS\n" + + " # SNOWFLAKE_SCHEMA=PUBLIC et SNOWFLAKE_ROLE=… sont optionnels.\n" + + "Installez aussi le connecteur : pip install mocksql[snowflake]" + ) + + # --------------------------------------------------------------------------- # Trino (source optionnelle) # L'exécution reste DuckDB : Trino ne sert qu'à l'import de schéma, la diff --git a/back/storage/config.py b/back/storage/config.py index 6e6f8fc..f35afd5 100644 --- a/back/storage/config.py +++ b/back/storage/config.py @@ -434,7 +434,7 @@ def output_language_directive() -> str: ) -def get_dbt_project(): +def get_dbt_project(config_path: Path | None = None): """Retourne un `DbtProject` si un bloc `dbt:` est configuré dans mocksql.yml, sinon None. Config attendue : @@ -445,12 +445,24 @@ def get_dbt_project(): Quand un projet dbt est configuré, MockSQL lit le SQL **compilé** (refs résolus, macros rendues) et infère les schémas amont depuis le manifest — sans entrepôt. """ - cfg = load_config().get("dbt") + # ``load_config()`` is intentionally CWD/base-dir scoped for the server and + # legacy CLI calls. A CLI command receiving ``--config`` must however not + # silently switch back to that global configuration when resolving dbt. + # Read that exact file here so both the dbt block and relative paths share + # the same source of truth. + if config_path is None: + cfg = load_config().get("dbt") + config_dir = _base_dir() + else: + config_path = config_path.resolve() + with open(config_path, encoding="utf-8") as f: + cfg = (yaml.safe_load(f) or {}).get("dbt") + config_dir = config_path.parent if not cfg or not cfg.get("project_dir"): return None from storage.dbt_manifest import DbtProject - project_dir = (_base_dir() / cfg["project_dir"]).resolve() + project_dir = (config_dir / cfg["project_dir"]).resolve() return DbtProject(project_dir, cfg.get("target_path", "target")) diff --git a/back/tests/test_cli_generate_cache_only.py b/back/tests/test_cli_generate_cache_only.py new file mode 100644 index 0000000..1ef4709 --- /dev/null +++ b/back/tests/test_cli_generate_cache_only.py @@ -0,0 +1,19 @@ +"""Cache-only dialects must never fall through to BigQuery schema import.""" + +import pytest + +from cli.generate import cache_miss_message + + +@pytest.mark.parametrize("dialect", ["duckdb", "postgres", "postgresql"]) +def test_cache_only_dialects_explain_schema_cache_requirement(dialect: str) -> None: + message = cache_miss_message(dialect) + + assert "cache-only" in message + assert "schema_cache" in message + assert "BQ_TEST_PROJECT" not in message + assert "BigQuery" not in message + + +def test_unknown_dialect_does_not_claim_bigquery_import() -> None: + assert "BigQuery" not in cache_miss_message("sqlite") diff --git a/back/tests/test_cli_generate_dbt.py b/back/tests/test_cli_generate_dbt.py new file mode 100644 index 0000000..fdd06bd --- /dev/null +++ b/back/tests/test_cli_generate_dbt.py @@ -0,0 +1,71 @@ +"""Regression tests for dbt resolution in ``mocksql generate``.""" + +import json +from pathlib import Path + +from cli.generate import resolve_model_sql +from storage.config import get_dbt_project + + +def test_external_config_resolves_dbt_and_selects_compiled_sql( + tmp_path: Path, monkeypatch +): + """``--config`` wins over the CWD config when resolving a dbt model.""" + config_root = tmp_path / "external-config" + models = config_root / "models" / "agg" + models.mkdir(parents=True) + model = models / "monthly_agg_reviews.sql" + model.write_text("select {{ ref('raw_reviews') }}", encoding="utf-8") + config = config_root / "mocksql.yml" + + dbt_root = tmp_path / "dbt-project" + target = dbt_root / "target" + compiled = target / "compiled" / "airbnb" / "models" / "agg" + compiled.mkdir(parents=True) + (target / "manifest.json").write_text( + json.dumps( + { + "nodes": { + "model.airbnb.monthly_agg_reviews": { + "resource_type": "model", + "name": "monthly_agg_reviews", + "package_name": "airbnb", + "original_file_path": "models/agg/monthly_agg_reviews.sql", + } + } + } + ), + encoding="utf-8", + ) + (compiled / "monthly_agg_reviews.sql").write_text( + "SELECT review_id FROM raw.reviews", encoding="utf-8" + ) + config.write_text( + "models_path: ./models\n" + "dialect: bigquery\n" + "dbt:\n" + " project_dir: ../dbt-project\n", + encoding="utf-8", + ) + + # A conflicting CWD config models the original bug: it must be ignored. + cwd = tmp_path / "unrelated-cwd" + cwd.mkdir() + (cwd / "mocksql.yml").write_text("dbt: {}\n", encoding="utf-8") + monkeypatch.chdir(cwd) + + project = get_dbt_project(config) + assert project is not None + assert project.is_dbt_model("agg/monthly_agg_reviews") + + model_name, selected_sql, is_dbt_model = resolve_model_sql( + model, + config, + {"models_path": "./models", "dialect": "bigquery"}, + project, + ) + assert model_name == "agg/monthly_agg_reviews" + assert is_dbt_model is True + assert "review_id" in selected_sql + assert "raw.reviews" in selected_sql + assert "{{" not in selected_sql diff --git a/back/tests/test_cli_init_messages.py b/back/tests/test_cli_init_messages.py new file mode 100644 index 0000000..2aa933f --- /dev/null +++ b/back/tests/test_cli_init_messages.py @@ -0,0 +1,67 @@ +"""Small regression checks for provider-aware ``mocksql init`` guidance.""" + +from pathlib import Path + +from typer.testing import CliRunner + +from cli.main import app + + +def test_init_openai_duckdb_does_not_print_vertex_or_bigquery_guidance( + tmp_path: Path, monkeypatch +) -> None: + async def _no_database_setup() -> None: + return None + + import init.init_db + + monkeypatch.setattr(init.init_db, "main", _no_database_setup) + result = CliRunner().invoke( + app, + [ + "init", + "--path", + str(tmp_path), + "--dialect", + "duckdb", + "--llm-provider", + "openai", + "--non-interactive", + ], + ) + + assert result.exit_code == 0, result.output + assert "OPENAI_API_KEY" in result.output + assert "VERTEX_PROJECT" not in result.output + assert "BQ_TEST_PROJECT" not in result.output + assert "schema_cache first" in result.output + assert "test_dataset:" not in (tmp_path / "mocksql.yml").read_text(encoding="utf-8") + + +def test_init_vertex_bigquery_prints_bigquery_guidance( + tmp_path: Path, monkeypatch +) -> None: + async def _no_database_setup() -> None: + return None + + import init.init_db + + monkeypatch.setattr(init.init_db, "main", _no_database_setup) + result = CliRunner().invoke( + app, + [ + "init", + "--path", + str(tmp_path), + "--dialect", + "bigquery", + "--llm-provider", + "vertexai", + "--non-interactive", + ], + ) + + assert result.exit_code == 0, result.output + assert "VERTEX_PROJECT" in result.output + assert "BQ_TEST_PROJECT" in result.output + assert "OPENAI_API_KEY" not in result.output diff --git a/back/tests/test_cli_snowflake_schema_import.py b/back/tests/test_cli_snowflake_schema_import.py new file mode 100644 index 0000000..fe19be6 --- /dev/null +++ b/back/tests/test_cli_snowflake_schema_import.py @@ -0,0 +1,141 @@ +"""Snowflake CLI schema import must never route through BigQuery.""" + +from pathlib import Path + +import pytest + + +SF_ROWS = [ + { + "table_catalog": "ANALYTICS", + "table_schema": "PUBLIC", + "table_name": "ORDERS", + "field_path": "ID", + "data_type": "NUMBER(38,0)", + "mode": "REQUIRED", + "description": "", + } +] + + +def _snowflake_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.delenv("BQ_TEST_PROJECT", raising=False) + monkeypatch.delenv("VERTEX_PROJECT", raising=False) + for name, value in { + "SNOWFLAKE_ACCOUNT": "org-account", + "SNOWFLAKE_USER": "mocksql", + "SNOWFLAKE_PASSWORD": "secret", + "SNOWFLAKE_WAREHOUSE": "COMPUTE_WH", + "SNOWFLAKE_DATABASE": "ANALYTICS", + }.items(): + monkeypatch.setenv(name, value) + + +def test_snowflake_configuration_error_lists_missing_settings( + monkeypatch: pytest.MonkeyPatch, +): + from models.env_variables import validate_snowflake_env + + for name in ( + "SNOWFLAKE_ACCOUNT", + "SNOWFLAKE_USER", + "SNOWFLAKE_PASSWORD", + "SNOWFLAKE_WAREHOUSE", + "SNOWFLAKE_DATABASE", + ): + monkeypatch.delenv(name, raising=False) + + with pytest.raises(RuntimeError, match="Configuration Snowflake incomplète") as exc: + validate_snowflake_env() + assert "SNOWFLAKE_ACCOUNT" in str(exc.value) + assert "pip install mocksql[snowflake]" in str(exc.value) + + +@pytest.mark.asyncio +async def test_generate_fetches_missing_snowflake_schema_without_bigquery( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +): + """The generate import branch uses the Snowflake fetcher with no BQ project.""" + from cli import generate + + _snowflake_env(monkeypatch) + config = tmp_path / "mocksql.yml" + config.write_text("dialect: snowflake\nmodels_path: ./models\n", encoding="utf-8") + model = tmp_path / "models" / "orders.sql" + model.parent.mkdir() + model.write_text("SELECT ID FROM ANALYTICS.PUBLIC.ORDERS", encoding="utf-8") + + async def fake_sf_fetch(refs): + assert refs == ["ANALYTICS.PUBLIC.ORDERS"] + return SF_ROWS, [] + + async def no_bigquery(*_args, **_kwargs): + raise AssertionError("BigQuery must not be called for dialect: snowflake") + + async def noop(*_args, **_kwargs): + return None + + class StopAfterImport(Exception): + pass + + monkeypatch.setattr("models.env_variables.validate_required_env", lambda: None) + monkeypatch.setattr( + "build_query.schema_fetcher.fetch_tables_schema_snowflake", fake_sf_fetch + ) + monkeypatch.setattr(generate, "fetch_tables_schema", no_bigquery) + monkeypatch.setattr("models.database.db_pool.init_pool", noop) + monkeypatch.setattr("init.init_db.run_migrations", noop) + monkeypatch.setattr( + generate, + "build_initial_state", + lambda *_args: (_ for _ in ()).throw(StopAfterImport()), + ) + + with pytest.raises(StopAfterImport): + await generate.run_generate(model, config, tmp_path / ".mocksql" / "tests") + + cached = generate.load_schema_cache( + str(tmp_path / ".mocksql" / "schema_cache.json") + ) + assert cached[0]["table_name"] == "ANALYTICS.PUBLIC.ORDERS" + + +def test_refresh_schemas_fetches_snowflake_without_bigquery( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +): + """refresh-schemas has a first-class Snowflake branch, like Trino.""" + from cli import main + from cli.generate import save_schema_cache + + _snowflake_env(monkeypatch) + config = tmp_path / "mocksql.yml" + config.write_text("dialect: snowflake\n", encoding="utf-8") + save_schema_cache( + str(tmp_path / ".mocksql" / "schema_cache.json"), + [{"table_name": "ANALYTICS.PUBLIC.ORDERS", "columns": []}], + ) + + async def fake_sf_fetch(refs): + assert refs == ["ANALYTICS.PUBLIC.ORDERS"] + return SF_ROWS, [] + + def no_required_env(): + raise AssertionError( + "refresh-schemas Snowflake must not require Vertex/BigQuery" + ) + + async def no_bigquery(*_args, **_kwargs): + raise AssertionError("BigQuery must not be called for dialect: snowflake") + + monkeypatch.setattr( + "build_query.schema_fetcher.fetch_tables_schema_snowflake", fake_sf_fetch + ) + monkeypatch.setattr("build_query.schema_fetcher.fetch_tables_schema", no_bigquery) + monkeypatch.setattr("models.env_variables.validate_required_env", no_required_env) + + main.refresh_schemas(config=config, tables=[], from_tests=False) + + from cli.generate import load_schema_cache + + cached = load_schema_cache(str(tmp_path / ".mocksql" / "schema_cache.json")) + assert cached[0]["columns"][0]["name"] == "ID" diff --git a/back/tests/test_llm_errors.py b/back/tests/test_llm_errors.py new file mode 100644 index 0000000..64c0fa6 --- /dev/null +++ b/back/tests/test_llm_errors.py @@ -0,0 +1,49 @@ +"""Vertex errors must be actionable without exposing credential contents.""" + +import pytest + +from utils.llm_errors import ( + classify_vertex_access_error, + format_vertex_permission_message, + is_vertex_permission_error, +) + + +@pytest.mark.parametrize( + ("raw", "category", "expected"), + [ + ( + "DefaultCredentialsError: Application Default Credentials", + "adc_missing", + "Application Default Credentials", + ), + ( + "Vertex AI API has not been used in project", + "api_disabled", + "Activez l'API Vertex AI", + ), + ( + "PERMISSION_DENIED: missing roles/aiplatform.user", + "iam_role_missing", + "roles/aiplatform.user", + ), + ( + "PERMISSION_DENIED: Publisher Model access denied", + "model_access_denied", + "Gemini", + ), + ], +) +def test_vertex_access_errors_are_classified_and_actionable( + raw: str, category: str, expected: str +) -> None: + exc = RuntimeError(raw) + + assert classify_vertex_access_error(exc) == category + assert is_vertex_permission_error(exc) + assert expected in format_vertex_permission_message("gemini-2.5-flash", exc) + + +def test_non_vertex_error_is_not_misclassified() -> None: + assert classify_vertex_access_error(RuntimeError("network timeout")) is None + assert not is_vertex_permission_error(RuntimeError("network timeout")) diff --git a/back/utils/llm_errors.py b/back/utils/llm_errors.py index 1d28410..83453df 100644 --- a/back/utils/llm_errors.py +++ b/back/utils/llm_errors.py @@ -40,11 +40,44 @@ def loads_lenient_json(raw: str) -> Any: def is_vertex_permission_error(exc: Exception) -> bool: - err_str = str(exc) - return "PERMISSION_DENIED" in err_str or "BILLING_DISABLED" in err_str + return classify_vertex_access_error(exc) is not None -def format_vertex_permission_message(model_name: str) -> str: +def classify_vertex_access_error(exc: Exception) -> str | None: + """Return a stable, credential-safe category for common Vertex failures.""" + error = str(exc).upper() + if "DEFAULTCREDENTIALSERROR" in error or "APPLICATION DEFAULT CREDENTIALS" in error: + return "adc_missing" + if ( + "VERTEX AI API HAS NOT BEEN USED" in error + or "AIPLATFORM.GOOGLEAPIS.COM" in error + ): + return "api_disabled" + if "PERMISSION_DENIED" in error or "BILLING_DISABLED" in error: + if "AIPLATFORM.USER" in error: + return "iam_role_missing" + if ( + "PUBLISHER MODEL" in error + or "GENERATIVE LANGUAGE" in error + or "MODEL" in error + ): + return "model_access_denied" + return "permission_denied" + return None + + +def format_vertex_permission_message( + model_name: str, exc: Exception | None = None +) -> str: + category = classify_vertex_access_error(exc) if exc else None + guidance = { + "adc_missing": "Configurez les Application Default Credentials avec `gcloud auth application-default login` ou GOOGLE_APPLICATION_CREDENTIALS.", + "api_disabled": "Activez l'API Vertex AI (`aiplatform.googleapis.com`) pour le projet Vertex.", + "iam_role_missing": "Accordez le rôle `roles/aiplatform.user` au compte qui exécute MockSQL.", + "model_access_denied": "Vérifiez que Gemini et ce modèle sont autorisés pour le projet, la région et l'organisation.", + } + if category in guidance: + return f"Erreur d'accès Vertex AI ({category}) pour « {model_name} ».\n• {guidance[category]}" return ( f"Erreur d'accès au modèle LLM (PERMISSION_DENIED).\n" f"• Vérifiez que le modèle « {model_name} » est accessible dans votre organisation.\n" diff --git a/back/utils/snowflake_connector.py b/back/utils/snowflake_connector.py index 451eaf3..33ba832 100644 --- a/back/utils/snowflake_connector.py +++ b/back/utils/snowflake_connector.py @@ -32,6 +32,9 @@ def _import_snowflake(): def get_sf_connection() -> snowflake.connector.SnowflakeConnection: + from models.env_variables import validate_snowflake_env + + validate_snowflake_env() snowflake_connector = _import_snowflake() global _sf_conn diff --git a/docs/quickstart-dbt.md b/docs/quickstart-dbt.md index 4574148..2c7dcc6 100644 --- a/docs/quickstart-dbt.md +++ b/docs/quickstart-dbt.md @@ -1,147 +1,102 @@ # Quickstart dbt -MockSQL tests **flat, parsable** `.sql` files. A [dbt](https://www.getdbt.com/) project is not directly parsable: models contain Jinja (`{{ ref(...) }}`, `{{ config(...) }}`, `{% if is_incremental() %}`, `dbt_utils` macros…) that SQLGlot cannot analyze. +MockSQL reads a dbt model's **compiled SQL**. It does not compile dbt itself and +does not derive schemas from `manifest.json`: the manifest identifies the model; +`target/compiled/` supplies the rendered SQL; schemas come from MockSQL's schema +cache or, for BigQuery, from BigQuery. -MockSQL's **dbt connector** bridges the gap. It has two roles, and only two: +## Support matrix -1. **Compile** — it reads the SQL **compiled** by dbt (`target/compiled/**/*.sql`), where all Jinja is already rendered: `ref()`/`source()`/`var()`/`this`/macros → flat SQL with the **real table names**. This replaces any regex preprocessor. -2. **Resolution** — it finds the dbt model from its path and provides that compiled SQL to MockSQL. +| dbt target | Status | Schema source for `mocksql generate` | +|---|---|---| +| dbt-BigQuery | Supported | Automatic BigQuery import for cache misses, or `schema_cache` | +| dbt-DuckDB | Supported with a prepared cache | `schema_cache` only; no DuckDB schema-import command exists yet | +| dbt-Snowflake | Supported with an explicit schema refresh | Refresh into `schema_cache`, then generate | -**Schema fetching remains MockSQL's normal job**: once the compiled SQL is provided, the `generate` flow extracts the referenced tables and imports their schema like any other query (BigQuery today; other warehouses coming). Test execution stays on a **scratch DuckDB** database — **zero cost**. +All generated cases are executed locally in DuckDB. The warehouse is never used +to execute the synthetic test data. -``` -dbt project - │ dbt compile (Jinja → flat SQL, refs = real warehouse names) - ▼ -target/compiled/**/*.sql + manifest.json - │ dbt connector (`dbt:` block in mocksql.yml) - ▼ -mocksql generate ──► schema import (warehouse) ──► LLM generation ──► DuckDB execution - → .mocksql/tests/.json -``` - ---- - -## 1. Declare the dbt project in `mocksql.yml` - -Add a `dbt:` block to the MockSQL config. This is what **activates the connector**: - -```yaml -version: "2" -dialect: bigquery # bigquery for a dbt-BigQuery project ; duckdb for dbt-duckdb -models_path: ./models # the dbt project's models/ folder -dbt: - project_dir: . # folder containing dbt_project.yml (relative to this mocksql.yml) - target_path: target # optional (default: target) -llm: - provider: vertexai -``` - -When `dbt:` is present, for any model recognized as a dbt model, MockSQL reads the **compiled SQL** instead of the raw `.sql` file. Any `preprocessor_fn` becomes unnecessary (compile already does the work). - -> **Dialect**: `bigquery` for a dbt-BigQuery project (the compiled SQL keeps BQ idioms, MockSQL transpiles them to DuckDB at execution). `duckdb` for a natively dbt-duckdb project. - ---- - -## 2. Compile the dbt project +## 1. Compile dbt -In an environment with your warehouse's dbt adapter (e.g. `dbt-bigquery`): +Run this from the dbt project. Use the target whose relation names you want +MockSQL to resolve. ```bash -cd my_dbt_project # IMPORTANT: be INSIDE the project folder -dbt deps # if the project has packages (dbt_utils, dbt_date…) -dbt compile # Jinja → target/compiled/**/*.sql +cd my_dbt_project +dbt deps # when the project uses packages +dbt compile ``` -`dbt compile` runs nothing on the warehouse — it just renders the Jinja. The result is in `target/compiled//models/**/*.sql`, with `ref()`/`source()` resolved to **real table names**. - -> **`relation_name` pitfall**: the table names in the compiled SQL depend on the **compile profile**. Compile with your warehouse's **real target** (your usual `profiles.yml`) so the refs are the real warehouse names — otherwise MockSQL's import won't find them. - ---- - -## 3. Materialize the parent models (to test a mart) - -This is the key point for **marts** and **intermediates**: a mart references **other models** (`{{ ref('products') }}`), not raw tables. To generate coherent data, MockSQL imports those parents' schema — **so they must exist in the warehouse**. - -- **staging** models: their refs are **real sources** → already present, nothing to do. -- **mart / intermediate** models: their parents are derived models → you must materialize them: +For a mart whose compiled SQL references materialized parent models, those +relations must exist in the target warehouse before BigQuery can import their +schemas: ```bash -dbt run --select +my_mart # builds the mart AND all its ancestors +dbt run --select +my_mart ``` -Once `dbt run` has passed, the parent tables exist and `mocksql generate` can import their schema. +This is not required for models that only reference physical sources already +present in the warehouse. -> If you skip this step on a mart, `mocksql generate` will fail at import with "table not found" on a parent model. +## 2. Configure MockSQL ---- - -## 4. Generate the tests - -```bash -DUCKDB_PATH=my_dbt_project/.mocksql/scratch.duckdb \ -mocksql generate my_dbt_project/models/marts/core/sales.sql \ - --config my_dbt_project/mocksql.yml \ - --output my_dbt_project/.mocksql/tests +```yaml +version: "2" +dialect: bigquery # use duckdb or snowflake for those targets +models_path: ./models +dbt: + project_dir: . + target_path: target # optional; default: target +schema_cache: .mocksql/schema_cache.json +llm: + provider: vertexai # or openai ``` -Sequence: -1. `[dbt] compiled SQL from manifest` — the connector provides the flat SQL (zero Jinja). -2. `Fetching schema for: …` — MockSQL imports the referenced tables' schemas from the warehouse. -3. LLM generation of synthetic data, execution on scratch DuckDB, verdict. -4. Writes `.mocksql/tests/.json` (input data, DuckDB results, assertions, verdict). - -> `DUCKDB_PATH` is the **scratch** database where MockSQL creates the synthetic tables — distinct from any dbt database. +`dbt:` makes `mocksql generate models/marts/sales.sql` read the corresponding +compiled file. A `preprocessor_fn` is normally unnecessary because dbt has +already rendered Jinja. -### Credentials +## 3. Generate -Warehouse + LLM credentials are read from `back/.env` (via `load_dotenv()`): +### dbt-BigQuery -``` -GOOGLE_APPLICATION_CREDENTIALS=C:\absolute\path\service-account.json -VERTEX_PROJECT=my-gcp-project +```bash +pip install mocksql[bigquery] +mocksql generate models/marts/sales.sql --config mocksql.yml ``` ---- +Set a BigQuery job project (`BQ_TEST_PROJECT`, or `VERTEX_PROJECT` as its +fallback) and Google application credentials. On a cache miss, MockSQL fetches +the referenced table schema and saves it in `.mocksql/schema_cache.json`. -## 5. (Optional) Evaluate quality across the whole project +### dbt-DuckDB -The `/eval-mocksql` skill generates tests for every model then scores them via an LLM judge: +Populate `.mocksql/schema_cache.json` with the real relation schemas first, +then run the same `mocksql generate` command. MockSQL does not infer schemas +from the compiled SQL or dbt manifest. DuckDB schema import is not implemented +in the CLI generation path. -``` -/eval-mocksql my_dbt_project -``` - -Report: `data` / `test` score + per-model validity, and an overall rate. +### dbt-Snowflake ---- - -## Workflow recap +Install `mocksql[snowflake]`, configure the Snowflake connection variables, and +refresh each required relation into the cache before generation: ```bash -# once -cd my_dbt_project && dbt deps - -# on every model / schema change -dbt compile # updates the compiled SQL -dbt run --select +my_mart # (marts only) materializes the parents -mocksql generate models/.../my_mart.sql --config mocksql.yml --output .mocksql/tests +mocksql refresh-schemas --table DATABASE.SCHEMA.PARENT_MODEL +mocksql generate models/marts/sales.sql --config mocksql.yml ``` ---- - -## Known pitfalls & limitations - -- **`relation_name` = compile profile**: compile with the real warehouse target, otherwise inconsistent refs (see §2). -- **Marts → materialized parents**: a mart is only testable if its upstream models exist in the database (see §3). -- **One mocksql project per dbt project**: each project has its own `dbt_project.yml`/`profiles.yml` and its `mocksql.yml`. -- **Date-relative logic** (`CURRENT_DATE`, rolling windows): generation may produce out-of-range data → empty result. A legitimate quality signal, not a setup bug. -- **Macros with execution-time effects** (`{% if is_incremental() %}`): `dbt compile` elides the incremental branch (false at compile time) → the test covers the non-incremental path. - ---- - -## Limitation: warehouses other than BigQuery +The dbt connector still supplies only compiled SQL; `refresh-schemas` is the +schema source. This manual step is required because `generate` auto-imports +cache misses only for BigQuery. -Today, MockSQL's schema import can only query **BigQuery**. For a **dbt-duckdb** project or one **without warehouse access**, there is therefore no automatic import path yet. +## BigQuery Sandbox and billing -While waiting for the **warehouse connectors** (Snowflake, Databricks, DuckDB… — on the roadmap), a workaround exists: manually pre-fill `.mocksql/schema_cache.json` (the `schema_cache` key in `mocksql.yml`) by introspecting the database materialized by `dbt run`, in `dialect: duckdb`. This is a stopgap, not the target method — the bootstrap script details are in this file's git history. +BigQuery dry-runs validate SQL and estimate bytes processed; they do not scan +table data. Schema metadata reads are also distinct from profiling. The sandbox +can therefore be enough to compile/dry-run, read metadata, and run queries +within the Sandbox free-tier quotas and feature limits. MockSQL profiling is a +real query over the source tables: it consumes that quota and needs a +billing-enabled BigQuery project once those limits or required capabilities are +exceeded. Set `BQ_TEST_PROJECT` explicitly for any BigQuery job; dry-runs are +estimates and are not a guarantee that later profiling is free. diff --git a/docs/quickstart.md b/docs/quickstart.md index bf04ed4..23c2e80 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -1,307 +1,127 @@ # Quickstart -## Prerequisites +MockSQL generates fixtures with an LLM and executes every generated test locally +on DuckDB. Source warehouses are used only for the capabilities listed below; +they do not run the generated synthetic data. -- Python 3.11+ -- [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) — only for Gemini (Vertex AI) and/or BigQuery sources -- Poetry (`pip install poetry`) — for development from source -- Node.js 18+ — only to build the frontend +## Requirements ---- - -## 1. Google Cloud authentication - -> **Using OpenAI as LLM?** Sections 1–2 are only needed for Gemini (Vertex AI) and/or a **BigQuery source** (schema fetch, profiling, import). With OpenAI + a `postgres` / `duckdb` source, skip straight to [section 3](#3-cli-installation) — no Google Cloud setup required. - -MockSQL uses Google application credentials for Vertex AI and BigQuery calls: - -```bash -gcloud auth application-default login -gcloud config set project -``` - ---- - -## 2. IAM permissions - -The account in use must have the following roles: - -| Role | Purpose | -|------|---------| -| `roles/bigquery.dataViewer` | Read table schemas | -| `roles/bigquery.user` | Run jobs / dry-run | -| `roles/aiplatform.user` | Call Vertex AI models (Gemini only — drop it if you use OpenAI) | - -> **Enabling Gemini (one-time step per project)** -> IAM roles are not enough: open the [Model Garden](https://console.cloud.google.com/vertex-ai/model-garden), search for a Gemini model and accept the terms of use. This is a one-time operation per GCP project and cannot be done via `gcloud`. - -### Option A — User account (local development) - -```bash -for ROLE in roles/bigquery.dataViewer roles/bigquery.user roles/aiplatform.user; do - gcloud projects add-iam-policy-binding \ - --member='user:' \ - --role="${ROLE}" -done -``` - -### Option B — Service account (CI/CD, Cloud Run) - -```bash -SA_EMAIL="mocksql-sa@.iam.gserviceaccount.com" - -gcloud iam service-accounts create mocksql-sa \ - --project= \ - --display-name="MockSQL service account" - -for ROLE in roles/bigquery.dataViewer roles/bigquery.user roles/aiplatform.user; do - gcloud projects add-iam-policy-binding \ - --member="serviceAccount:${SA_EMAIL}" \ - --role="${ROLE}" -done - -gcloud iam service-accounts keys create ~/keys/mocksql-sa.json \ - --iam-account="${SA_EMAIL}" -``` - -Locally, in the `.env` at your project root (see next section): -```dotenv -GOOGLE_APPLICATION_CREDENTIALS=/path/to/mocksql-sa.json -``` - -In CI/CD, inject `GOOGLE_APPLICATION_CREDENTIALS` as a secret environment variable. - -> **Common error**: `Forbidden: Access Denied: bigquery.jobs.create` → the `bigquery.user` role is missing. - ---- - -## 3. CLI installation +- Python `>=3.11,<3.14` +- `pip install mocksql` +- An LLM credential: Vertex AI/Gemini or OpenAI +- `mocksql[bigquery]` only when MockSQL must import or profile BigQuery tables ```bash pip install mocksql +pip install mocksql[bigquery] # optional BigQuery connector +mocksql --help ``` -The base install is intentionally lightweight: data generation and execution run entirely on **DuckDB**, with no warehouse client. The source-warehouse connectors are heavy (`pyarrow`, `grpc`, …) and only needed to **profile or import** real tables, so they ship as optional extras: - -```bash -pip install mocksql[bigquery] # + profiling/import from BigQuery -pip install mocksql[snowflake] # + profiling/import from Snowflake -pip install mocksql[all] # all connectors -``` - -If you trigger a profiling/import step without the matching extra installed, MockSQL fails fast with the exact `pip install mocksql[…]` command to run. Since the default `dialect` is `bigquery`, profiling against BigQuery sources requires `mocksql[bigquery]`. - -### LLM provider — Gemini or OpenAI - -MockSQL picks the LLM backend from the **model name**: `gemini*` → Vertex AI, `gpt-*` / `o` (o3, o4-mini…) → OpenAI. Set `llm.model` in `mocksql.yml` and provide the matching credentials below. The `llm.provider` key only breaks ties for ambiguous names (custom models, proxies). +The package is version `0.2.1` and licensed under MIT; see +[back/pyproject.toml](../back/pyproject.toml). -Credentials and cloud projects come from **environment variables** — never from `mocksql.yml`, which only describes the project structure (paths, dialect, model). The priority is: +## Initialize a project -``` -system / CI variable > local .env file > error +```bash +mocksql init +mocksql init --path ./my_project +mocksql init --dialect bigquery --llm-provider openai --non-interactive ``` -In local development, put them in a **gitignored** `.env` at your project root — MockSQL loads it automatically at startup (`load_dotenv()`). Add `.env` to your `.gitignore`: +The documented CLI options are the options exposed by `mocksql init --help`: +`--path/-p`, `--dialect`, `--models-path`, `--llm-provider`, +`--test-dataset` (deprecated compatibility option; tests still run locally), +`--langchain-api-key`, `--force`, and `--non-interactive`. -``` -.env -``` +`mocksql init` creates `mocksql.yml`, `.mocksql/schema_cache.json` on first +schema import, and a local DuckDB file under `.mocksql/data/`. -
-Gemini via Vertex AI (default) +### LLM credentials -Requires the Google Cloud setup from [sections 1–2](#1-google-cloud-authentication) (auth + `roles/aiplatform.user` + Model Garden terms). +Use exactly one provider configuration. ```dotenv -# .env — do not commit -VERTEX_PROJECT=my-project-dev -GOOGLE_CLOUD_LOCATION=us-central1 # required for Vertex AI calls - -# Optional — default: VERTEX_PROJECT -BQ_TEST_PROJECT=my-project-dev - -# Optional: explicit service account (otherwise: Application Default Credentials) -# GOOGLE_APPLICATION_CREDENTIALS=/path/to/service_account.json +# Gemini through Vertex AI +VERTEX_PROJECT=my-gcp-project +GOOGLE_CLOUD_LOCATION=us-central1 ``` -```yaml -# mocksql.yml -llm: - model: gemini-2.5-flash # or gemini-2.5-pro -``` - -MockSQL is tuned for **gemini-2.5-flash / pro**, whose native thinking mode is on by default — prefer them over `flash-lite`. - -
- -
-OpenAI - -No Google Cloud setup is needed for the LLM — [sections 1–2](#1-google-cloud-authentication) only remain relevant if your **source warehouse** is BigQuery. - ```dotenv -# .env — do not commit +# OpenAI OPENAI_API_KEY=sk-... ``` -```yaml -# mocksql.yml -llm: - model: gpt-5-mini # any gpt-* / o model routes to OpenAI -``` - -Reasoning models (`gpt-5*`, o-series) only accept the default temperature; the optional `llm.thinking_level` key (`low` / `medium` / `high`) is forwarded to them as `reasoning_effort`. Non-reasoning models (`gpt-4.1-mini`, `gpt-5-chat-latest`) behave like classic chat models. - -
- -#### In CI/CD (GitHub Actions, Cloud Build…) - -Inject the variables directly — they take priority over the local `.env`: - -```yaml -# GitHub Actions — Gemini -env: - VERTEX_PROJECT: my-project-preprod - GOOGLE_CLOUD_LOCATION: us-central1 - BQ_TEST_PROJECT: my-project-preprod # if different from VERTEX_PROJECT - -# GitHub Actions — OpenAI -env: - OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} -``` - -```yaml -# Cloud Build -substitutions: - _VERTEX_PROJECT: my-project-prod -env: - - VERTEX_PROJECT=$_VERTEX_PROJECT - - GOOGLE_CLOUD_LOCATION=us-central1 -``` - -The DuckDB path is configured via `duckdb_path` in `mocksql.yml` (default: `data/mocksql.duckdb`). - -### `mocksql init` - -Initializes a project and generates `mocksql.yml`: - -```bash -mocksql init -# or in a subfolder -mocksql init --path ./my_project -``` - -Example generated `mocksql.yml`: - -```yaml -version: "2" -dialect: bigquery # bigquery | postgres | duckdb -models_path: ./models -duckdb_path: data/mocksql.duckdb # path to the local DuckDB database -llm: - model: gemini-2.5-flash # gemini* → Vertex AI · gpt-* / o → OpenAI - streaming: false -schema_cache: .mocksql/schema_cache.json -``` - -#### Supported dialects - -The `dialect` describes the **source** SQL: it drives validation (dry-run) and optimization. Test execution **always happens on DuckDB locally**. +Set `llm.provider: vertexai` or `llm.provider: openai` in `mocksql.yml`. +Model names also route automatically: `gemini*` goes to Vertex AI; `gpt-*` and +`o*` go to OpenAI. OpenAI does not need Vertex credentials unless the source +is BigQuery. -| Dialect | Source | Validation (dry-run) | Schema | -|---------|--------|----------------------|--------| -| `bigquery` | BigQuery | BigQuery dry-run | fetch BigQuery → cache | -| `postgres` | PostgreSQL | Postgres dry-run | fetch Postgres → cache | -| `duckdb` | DuckDB / dbt-DuckDB | local DuckDB dry-run | pre-filled cache (see [quickstart-dbt.md](quickstart-dbt.md)) | +## Source connector status -> In `dialect: duckdb`, MockSQL queries no remote source: the schema cache must be pre-filled (bootstrapped from a DuckDB database). This is the mode used for **dbt-DuckDB** projects — see **[quickstart-dbt.md](quickstart-dbt.md)**. +| Dialect | Validation | Schema handling in `mocksql generate` | +|---|---|---| +| `bigquery` | BigQuery dry-run | Imports cache misses from BigQuery with `mocksql[bigquery]` | +| `postgres` | Postgres validation | Use a prepared `schema_cache`; no Postgres import in this flow | +| `duckdb` | Local DuckDB validation | Use a prepared `schema_cache`; no DuckDB import command in this flow | +| `snowflake` | Snowflake `EXPLAIN` validation | Refresh schemas explicitly, then generate from `schema_cache` | +| `trino` | Trino validation | Use a prepared cache for generation; `refresh-schemas` has Trino support | -**`llm` keys**: +`mocksql generate` needs schemas. It reads them from `schema_cache` first, then +automatically imports only BigQuery cache misses. It never guesses column types +from generated rows. `mocksql refresh-schemas` refreshes BigQuery schemas by +by default; Snowflake and Trino each have explicit branches. It is not a DuckDB +schema importer. -| Key | Default | Description | -|-----|---------|-------------| -| `model` | `gemini-2.5-flash` | Takes priority over `DEFAULT_MODEL_NAME`. The name picks the backend: `gemini*` → Vertex AI, `gpt-*` / `o` → OpenAI | -| `provider` | `vertexai` | Only consulted for **ambiguous** model names (custom models, proxies) — the model name always wins | -| `streaming` | `false` | Token-by-token streaming | - -### `mocksql generate` +For Snowflake, install `mocksql[snowflake]`, set the Snowflake environment +variables, then populate the cache explicitly, for example: ```bash +mocksql refresh-schemas --table DATABASE.SCHEMA.ORDERS mocksql generate models/orders.sql -# with options -mocksql generate models/orders.sql --config mocksql.yml --output .mocksql/tests -``` - -Schemas are cached in `.mocksql/schema_cache.json` — subsequent runs no longer query BigQuery. - -**Outputs** in `.mocksql/tests/`: -- `_data.json` — test data (input tables) -- `_results.json` — DuckDB execution results - -### SQL preprocessor (variables and templates) - -If your `.sql` files contain non-parsable variables (`@start_date`, `{{ ds }}`, dbt macros…): - -```yaml -# mocksql.yml -preprocessor_fn: "preprocessors:replace_vars" # module:function, relative to mocksql.yml ``` -`preprocessors.py` next to `mocksql.yml`: +## BigQuery credentials, Sandbox, and cost -```python -import re +For BigQuery schema import, configure an execution project and credentials: -def replace_vars(sql: str) -> str: - defaults = {"start_date": "'2024-01-01'", "end_date": "'2024-12-31'"} - return re.sub(r"@(\w+)", lambda m: defaults.get(m.group(1), "NULL"), sql) +```dotenv +BQ_TEST_PROJECT=my-billing-project # falls back to VERTEX_PROJECT +# GOOGLE_APPLICATION_CREDENTIALS=/absolute/path/service-account.json ``` -### Profiling budget (auto-profiling) +Application Default Credentials (`gcloud auth application-default login`) are +also supported. Typical permissions are `roles/bigquery.dataViewer` for metadata +and `roles/bigquery.user` to create BigQuery jobs; Gemini additionally needs +`roles/aiplatform.user`. -Profiling real tables runs a BigQuery dry-run to estimate the scan, then queries -each table. To make the "click Generate and walk away" flow fully hands-off, set a -**scan budget** (in TB): tables whose estimated scan fits under the budget are -profiled automatically; tables above it are **deferred** (the profile is marked -partial and a *"Compléter le profil"* button lets you profile them on demand). +BigQuery dry-runs validate a query and return an estimated `total_bytes_processed`. +They do not read table data and MockSQL uses them to validate/estimate work. A +BigQuery Sandbox can run dry-runs, read metadata, and run real queries within +its free-tier quotas and feature limits; it does not require a billing account. +`mocksql generate --profile` is a real query over source tables, so it consumes +that quota and needs a billing-enabled project once Sandbox/free-tier limits or +required capabilities are exceeded. Dry-run estimates are not charges, nor a +promise that a later real profiling query is free. `profile_budget_tb` limits +which estimated profiling queries are run; it does not make a query free. -```yaml -# mocksql.yml -profile_budget_tb: 0.3 # auto-profile under 0.3 TB; defer larger tables -``` - -Also settable via the `PROFILE_BUDGET_TB` env var. When **unset**, the UI asks for a -budget before profiling (default 0.3 TB, remembered per browser). Set it to a value -≤ 0 / leave it out to keep the historical behaviour (no budget — profile everything). -Only applies to BigQuery (DuckDB/Postgres profiling is free, so no budget is needed). - -### Full example +## Generate and replay ```bash -mocksql generate examples/jaffle_shop/models/orders.sql \ - --config examples/jaffle_shop/mocksql.yml +mocksql generate models/orders.sql +mocksql generate models/orders.sql --instruction "customer with no orders" +mocksql generate models/orders.sql --overwrite +mocksql test --model orders +mocksql test --json +mocksql test --frozen ``` ---- +`generate` is additive by default. `--overwrite` rebuilds the suite. `test` +uses the live SQL file by default and makes no LLM or warehouse calls; `--frozen` +uses the SQL snapshot saved with the test. -## 4. Web UI +## dbt -MockSQL ships two distinct wheels: - -| Package | Contents | -|---------|----------| -| `mocksql` | CLI only | -| `mocksql-ui` | CLI + web server + React assets | - -```bash -# CLI + UI -pip install mocksql mocksql-ui -``` - -Make sure the LLM provider environment variables are set (see [section 3](#3-cli-installation)) then: - -```bash -mocksql ui # http://localhost:8080/static/ -mocksql ui --port 4000 -mocksql ui --no-browser -``` +For dbt, run `dbt compile` first and configure the `dbt:` block. MockSQL reads +the compiled SQL, not raw Jinja, and uses the same schema-cache/BigQuery-import +rules described above. See [quickstart-dbt.md](quickstart-dbt.md) for the exact +dbt-BigQuery, dbt-DuckDB, and dbt-Snowflake status.