From 9f24b7567289970c375ce7abfdd6853d4b52eda2 Mon Sep 17 00:00:00 2001 From: Francisco Perez Date: Mon, 22 Jun 2026 20:05:18 -0500 Subject: [PATCH] Document connection scoped queries --- AGENTS.md | 56 +++++++++++++++++++++++++----- README.md | 47 ++++++++++++++++++++++--- docker/config.json | 8 +++++ examples/python_api/README.md | 2 ++ examples/python_api/list_config.py | 6 ++++ skills/sql2json/SKILL.md | 25 +++++++++++-- 6 files changed, 129 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 2ab556a..06bf64f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -62,9 +62,10 @@ Check the installed version with `uv tool list`, `pipx list`, or Map a natural language request to these parameters: 1. **Identify the connection** (`--name`): which database to use. -2. **Select the query** (`--query`): a named query from `config.json`, raw inline SQL, or a path to a `.sql` file prefixed with `@`. -3. **Supply parameters**: date variables and SQL bind parameters as extra `--key value` flags. -4. **Shape the output**: use `--first`, `--key`, `--value`, `--wrapper`, `--jsonkeys` to transform results. +2. **Select the query** (`--query`): prefer a named query from `config.json`, using connection-scoped queries when available; otherwise use raw inline SQL or a path to a `.sql` file prefixed with `@`. +3. **Resolve named-query precedence**: for `--name --query `, sql2json checks `connection_queries..` first, then falls back to `queries.`, then treats the value as raw SQL or `@file` when no named query exists. +4. **Supply parameters**: date variables and SQL bind parameters as extra `--key value` flags. +5. **Shape the output**: use `--first`, `--key`, `--value`, `--wrapper`, `--jsonkeys` to transform results. --- @@ -77,12 +78,18 @@ Before calling a query, an agent can inspect what is configured: sql2json --list-connections --config /path/to/config.json # → ["default", "mysql", "reporting"] -# List available named queries +# List available named queries, grouped by scope sql2json --list-queries --config /path/to/config.json -# → ["default", "sales_monthly", "total_users"] +# → {"global": ["default", "sales_monthly"], "connections": {"mysql": ["table_sizes"], "reporting": ["total_users"]}} + +# Request the old flat global-query list when integrating with legacy callers +sql2json --list-queries legacy --config /path/to/config.json +# → ["default", "sales_monthly"] ``` -Both flags print a JSON array to stdout and exit 0. If `--config` is omitted the tool uses its normal config lookup order. +`--list-connections` prints a JSON array to stdout and exits 0. `--list-queries` prints the scoped discovery object by default; `--list-queries legacy` prints the old flat global query array. If `--config` is omitted the tool uses its normal config lookup order. + +When selecting a named query for a connection, prefer a query listed under `connections.`; if none matches, use the matching name from `global`. Runtime lookup follows the same precedence: scoped query first, global query fallback, then raw SQL/`@file` behavior. --- @@ -98,6 +105,35 @@ If none exist, a read-only in-memory SQLite database is used (useful for testing --- +## Config schema for named queries + +Use top-level `queries` for shared/global named queries and `connection_queries` for connection-specific SQL. `connection_queries` is the canonical schema for scoped queries: connection name -> query name -> SQL. + +```json +{ + "connections": { + "postgres": "postgresql+psycopg2://user:pass@host/db", + "mysql": "mysql+pymysql://user:pass@host/db" + }, + "queries": { + "sales": "SELECT month, amount FROM sales", + "long_report": "@/path/to/report.sql" + }, + "connection_queries": { + "postgres": { + "now": "SELECT CURRENT_TIMESTAMP AS ts" + }, + "mysql": { + "now": "SELECT NOW() AS ts" + } + } +} +``` + +Existing `queries` configs remain valid; they are the fallback/global scope. Query values may be raw SQL or `@/path.sql` file references. + +--- + ## Key flags | Flag | Purpose | Example | @@ -113,7 +149,7 @@ If none exist, a read-only in-memory SQLite database is used (useful for testing | `--format` | Output format: `json` (default), `csv`, `excel` | `--format csv` | | `--output` | Save to file instead of printing; filename supports `{CURRENT_DATE}` etc. | `--output report_{CURRENT_DATE}` | | `--list-connections` | Print JSON array of configured connection names and exit | `--list-connections` | -| `--list-queries` | Print JSON array of configured query names and exit | `--list-queries` | +| `--list-queries` | Print configured query names and exit. Default shape is `{"global": [...], "connections": {...}}`; pass `--list-queries legacy` for the old flat global query array | `--list-queries` | **Note:** Both `--list-connections` and `--list_connections` (underscore) are accepted by fire. @@ -195,8 +231,10 @@ sql2json --name mysql --query sales_monthly --format csv --output sales_{CURRENT from sql2json import run_query2json, run_query_by_name, list_connections, list_queries # Discover what's configured -connections = list_connections("/path/to/config.json") # ["default", "mysql"] -queries = list_queries("/path/to/config.json") # ["default", "sales_monthly"] +connections = list_connections("/path/to/config.json") +queries = list_queries("/path/to/config.json") # global legacy names +scoped_queries = list_queries("/path/to/config.json", scoped=True) # {"global": [...], "connections": {...}} +mysql_queries = list_queries("/path/to/config.json", connection="mysql") # Run a named query — returns list of dicts rows = run_query_by_name( diff --git a/README.md b/README.md index 6b6d820..b59030f 100644 --- a/README.md +++ b/README.md @@ -263,12 +263,23 @@ docker compose run --rm sql2json --name pg --query sales_by_month --min_amount 4 # → [{"month": "January", "amount": 5000.0}, {"month": "March", "amount": 7100.75}] ``` -Run the same demo queries against MySQL by switching `--name`: +Run the same shared demo queries against MySQL by switching `--name`: ```bash docker compose run --rm sql2json --name mysql --query sales ``` +The demo config also uses `connection_queries` for dialect-specific named queries. For example, `database_name` is defined separately for PostgreSQL and MySQL, while `version`, `sales`, and `sales_by_month` remain shared/global queries: + +```bash +docker compose run --rm sql2json --list-queries +# → {"global": ["version", "sales", "sales_by_month"], "connections": {"pg": ["database_name"], "mysql": ["database_name"]}} + +# Same query name, scoped SQL for each connection: +docker compose run --rm sql2json --name pg --query database_name +docker compose run --rm sql2json --name mysql --query database_name +``` + Tear down when done: ```bash @@ -338,6 +349,11 @@ cat > ~/.sql2json/config.json << 'EOF' }, "queries": { "default": "SELECT 1 AS a, 2 AS b" + }, + "connection_queries": { + "default": { + "healthcheck": "SELECT 'ok' AS status" + } } } EOF @@ -381,6 +397,16 @@ Use `--config /path/to/config.json` to override. "sales_monthly": "SELECT inv.month, SUM(inv.amount) AS sales FROM invoices inv WHERE inv.date >= :date_from", "total_sales": "SELECT SUM(inv.amount) AS sales FROM invoices inv WHERE inv.date >= :date_from", "long_query": "@/path/to/my_query.sql" + }, + "connection_queries": { + "postgres": { + "now": "SELECT CURRENT_TIMESTAMP AS ts", + "table_sizes": "SELECT schemaname, relname, pg_total_relation_size(relid) AS bytes FROM pg_catalog.pg_statio_user_tables" + }, + "mysql": { + "now": "SELECT NOW() AS ts", + "table_sizes": "SELECT table_schema, table_name, data_length + index_length AS bytes FROM information_schema.tables WHERE table_schema = DATABASE()" + } } } ``` @@ -389,6 +415,10 @@ Use `--config /path/to/config.json` to override. Connection strings follow [SQLAlchemy URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls). Query values starting with `@` are treated as paths to `.sql` files. +`connection_queries` is the canonical schema for queries that are valid only for a specific connection or SQL dialect. Its shape is a top-level map of connection name to query-name to SQL. `queries` remains valid for shared/global named queries that can run unchanged across connections. + +Named query lookup is connection-aware: `sql2json --name postgres --query now` first checks `connection_queries.postgres.now`; if it is not present, it falls back to `queries.now`; if neither exists, `--query` is treated as raw SQL or an `@/path.sql` file reference. + ## CLI reference ```bash @@ -408,7 +438,7 @@ sql2json [options] [--param value ...] | `--format` | `json` | Output format: `json`, `csv`, `excel` | | `--output` | _(stdout)_ | Save to file; filename supports `{CURRENT_DATE}` etc. | | `--list-connections` | — | Print JSON array of configured connection names and exit | -| `--list-queries` | — | Print JSON array of configured query names and exit | +| `--list-queries` | — | Print configured query names and exit. Default shape is `{"global": [...], "connections": {...}}`; pass `--list-queries legacy` for the old flat global query list | Extra `--key value` flags become SQL bind parameters (`:key` in your query). @@ -421,9 +451,14 @@ sql2json --list-connections # → ["default", "mysql", "reporting"] sql2json --list-queries -# → ["default", "sales_monthly", "total_sales"] +# → {"global": ["default", "sales_monthly"], "connections": {"mysql": ["table_sizes"]}} + +sql2json --list-queries legacy +# → ["default", "sales_monthly"] ``` +Use the scoped discovery output to choose an appropriate `--name`/`--query` pair. For a given connection, an entry under `connections.` overrides a same-named global query; otherwise the global query is used as a fallback. + ## Date variables Extra parameters whose values match a built-in variable are resolved to real dates before the query runs: @@ -603,11 +638,15 @@ rows = run_query2json( ) connections = list_connections("/path/to/config.json") -queries = list_queries("/path/to/config.json") +queries = list_queries("/path/to/config.json") # global legacy names +scoped_queries = list_queries("/path/to/config.json", scoped=True) # {"global": [...], "connections": {...}} +mysql_queries = list_queries("/path/to/config.json", connection="mysql") ``` Use `run_query2json()` for inline SQL, named queries, SQL files with `@/path.sql`, bind/date parameters, `first`, `key`, `value`, `wrapper`, `jsonkeys`, and `timezone`. Use `run_query_by_name()` when you specifically want the lower-level named connection/query call. +Named query resolution in the Python API matches the CLI: connection-scoped query first, global query fallback, then raw SQL or `@file` handling for `run_query2json()`. + Python API errors are normal Python exceptions. The CLI-only JSON stderr envelope is not used by the Python API. Supported public imports are exported from `sql2json.__all__`. Internal helpers in `sql2json.sql2json`, `sql2json.__main__`, or `sql2json.parameter.parameter_parser` are implementation details and should not be imported by users. `sql2json.parameter.parse_parameter` remains public for date-variable resolution; lower-level date helper functions are private. diff --git a/docker/config.json b/docker/config.json index 0f54287..fde6e53 100644 --- a/docker/config.json +++ b/docker/config.json @@ -7,5 +7,13 @@ "version": "SELECT version() AS version", "sales": "SELECT * FROM sales", "sales_by_month": "SELECT month, amount FROM sales WHERE amount >= :min_amount" + }, + "connection_queries": { + "pg": { + "database_name": "SELECT current_database() AS database_name" + }, + "mysql": { + "database_name": "SELECT DATABASE() AS database_name" + } } } diff --git a/examples/python_api/README.md b/examples/python_api/README.md index 61a06c2..6eac94a 100644 --- a/examples/python_api/README.md +++ b/examples/python_api/README.md @@ -23,4 +23,6 @@ Public API used here: - `list_queries` - `parse_parameter` +`list_queries(path)` returns legacy global query names. Use `list_queries(path, scoped=True)` to return the full discovery shape (`{"global": [...], "connections": {...}}`), or `list_queries(path, connection="name")` to get the effective names for one connection, including connection-scoped queries and global fallbacks. + Implementation helpers inside `sql2json.sql2json`, `sql2json.__main__`, or `sql2json.parameter.parameter_parser` are internal and should not be imported by users. diff --git a/examples/python_api/list_config.py b/examples/python_api/list_config.py index f989e11..2d8bd46 100644 --- a/examples/python_api/list_config.py +++ b/examples/python_api/list_config.py @@ -7,6 +7,10 @@ config = { "connections": {"default": "sqlite:///:memory:", "reporting": "sqlite:///:memory:"}, "queries": {"default": "SELECT 1 AS ok", "sales": "SELECT 42 AS total"}, + "connection_queries": { + "default": {"sales": "SELECT 7 AS scoped_sales"}, + "reporting": {"pipeline": "SELECT 8 AS pipeline"}, + }, } with tempfile.TemporaryDirectory() as tmp: @@ -15,3 +19,5 @@ print(list_connections(str(config_path))) print(list_queries(str(config_path))) + print(list_queries(str(config_path), scoped=True)) + print(list_queries(str(config_path), connection="default")) diff --git a/skills/sql2json/SKILL.md b/skills/sql2json/SKILL.md index bc6a25f..e498c35 100644 --- a/skills/sql2json/SKILL.md +++ b/skills/sql2json/SKILL.md @@ -75,10 +75,19 @@ The config file lives at `~/.sql2json/config.json`. Create it with at least one }, "queries": { "users": "SELECT id, email FROM users LIMIT 10" + }, + "connection_queries": { + "mydb": { + "table_sizes": "SELECT schemaname, relname, pg_total_relation_size(relid) AS bytes FROM pg_catalog.pg_statio_user_tables" + } } } ``` +Use `queries` for shared/global named queries. Use `connection_queries` as the canonical schema for connection-specific SQL: connection name -> query name -> SQL. Existing global `queries` configs remain valid and act as fallbacks. + +Named query resolution is: `connection_queries..` first, then `queries.`, then raw SQL or `@/path.sql` handling. + Supported connection strings follow SQLAlchemy format: | Database | Example connection string | @@ -93,8 +102,12 @@ Discover what is configured: ```bash sql2json --list-connections sql2json --list-queries +# → {"global": ["users"], "connections": {"mydb": ["table_sizes"]}} +sql2json --list-queries legacy # old flat global query list ``` +When an agent receives a data request, discover connections and scoped queries before inventing SQL. Choose a connection, then prefer a query listed under that connection; fall back to a global query only if no scoped query matches. + ## Common query patterns **Named query:** @@ -103,6 +116,13 @@ sql2json --list-queries sql2json --name mydb --query users ``` +**Connection-scoped named query:** + +```bash +# Resolves to connection_queries.mydb.table_sizes before checking global queries.table_sizes +sql2json --name mydb --query table_sizes +``` + **Inline SQL:** ```bash @@ -227,8 +247,9 @@ See `references/agent-sync.md` for the full agent-target map. 1. `sql2json` writes errors to stderr. Check the exit code and stderr, not just stdout. 2. Named queries come from `~/.sql2json/config.json`; a missing query name is a config problem, not a skill problem. -3. `Decimal` columns are serialized as floats. `date`/`datetime` columns are not handled natively — cast to `VARCHAR` in SQL or use `--jsonkeys` if the driver returns them as strings. -4. Keep the canonical file in the repo; do not hand-edit copies in agent-local skill directories. +3. For named queries, always account for scoped-query precedence: `connection_queries..` overrides the same name in global `queries`. +4. `Decimal` columns are serialized as floats. `date`/`datetime` columns are not handled natively — cast to `VARCHAR` in SQL or use `--jsonkeys` if the driver returns them as strings. +5. Keep the canonical file in the repo; do not hand-edit copies in agent-local skill directories. ## Verification checklist