fix(sqlmesh): resolve DuckDB connection-config conflict blocking state init#30
Closed
Doctacon wants to merge 3 commits into
Closed
fix(sqlmesh): resolve DuckDB connection-config conflict blocking state init#30Doctacon wants to merge 3 commits into
Doctacon wants to merge 3 commits into
Conversation
Two changes that landed together because they were found together: 1. analytics.py: drop `from __future__ import annotations` and import `AssetExecutionContext` directly instead of via `dg.AssetExecutionContext`. Dagster's op-definition validator does an identity check against the imported class; PEP 563 turns annotations into strings and the `dg.X` attribute form doesn't compare equal. Result: `defs` wouldn't load at all since the prior strict-mypy commit introduced the typed `context` parameter. 2. Taskfile.yaml: `dagster asset materialize -f <path>` → `-m databox.orchestration.definitions`. `dagster dev` drops `-f` entirely — it auto-discovers workspace.yaml at the repo root. workspace.yaml itself switches from `python_file` to `python_module` so both entry points target the same import path. Validated with `dagster definitions validate` against both `-m` and `-w workspace.yaml`. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
37 lines of 1Password source code do not justify a top-level directory. Inlined into docs/secrets.md alongside the existing wiring example — same "copy-adapt" intent, fewer directories. CLAUDE.md reference dropped. Reconciled closed loom ticket secrets-pluggable with a follow-up note so the graph stays truthful — the file was present at close time but has since been removed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…e init
Two-part fix for the md: URL connection-config conflict that was crashing
`sqlmesh_project` on first run under both backends:
1. `config/settings.py::sqlmesh_config()` now registers only the gateway
matching `settings.backend`. SQLMesh's `Context.engine_adapters` eagerly
builds an EngineAdapter for every gateway in `Config.gateways`, so
registering both made `DATABOX_BACKEND=local` still open a MotherDuck
connection. A dedicated `state_connection` at `data/sqlmesh_state.duckdb`
is retained on both paths.
2. `orchestration/_factories.py::ensure_motherduck_databases()` now opens
`duckdb.connect(database=..., config={"custom_user_agent": f"SQLMesh/{__version__}"})`
so the md: URL config matches SQLMesh's later open. DuckDB caches a
process-global handle per `md:?motherduck_token=...` URL and rejects
subsequent opens with mismatched config dicts.
Verified on a clean `data/` dir: `task verify` ends with RUN_SUCCESS under
both `DATABOX_BACKEND=local` and `DATABOX_BACKEND=motherduck`. Ruff + mypy
clean. See `.loom/evidence/20260423-sqlmesh-state-conn-conflict-fix.md`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Owner
Author
|
Parking while CI breakage on main is fixed first (per-shard coverage floor). Will rebase + reopen once ci-per-shard-coverage lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Context.engine_adaptersstops eagerly opening the other backend's connection at snapshot-evaluation time.config={"custom_user_agent": f"SQLMesh/{__version__}"}whenensure_motherduck_databases()opens the md: URL so DuckDB's process-global handle cache doesn't reject SQLMesh's later open with "different configuration than existing connections".state_connectionatdata/sqlmesh_state.duckdbon both gateways (already present, kept for SQLMesh's MotherDuck-as-state warning + h3-extension conflict).Also sweeps in 2 pre-existing commits on
main(examples/removal, Dagster loader +-mentry points) that had not yet been pushed. The Dagster-loader commit is what surfaced this state-conn bug; they merge together cleanly.Test plan
task verify(smoke) ends withRUN_SUCCESSunderDATABOX_BACKEND=localon a cleandata/dirtask verify(smoke) ends withRUN_SUCCESSunderDATABOX_BACKEND=motherduckon a cleandata/diruv run ruff check packages/ transforms/→ cleanuv run mypy packages/→ 50 source files, no issuesuv run pytest→ 118 passed, 1 out-of-scope flake (ebird/test_idempotency.py, tracked separately)tests/test_motherduck_autocreate.pyupdated for new kwargs-basedduckdb.connectcall signatureFull run tails + tool output captured in
.loom/evidence/20260423-sqlmesh-state-conn-conflict-fix.md. Closesticket:sqlmesh-state-conn-conflict.🤖 Generated with Claude Code