Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 47 additions & 9 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,9 +62,10 @@ Check the installed version with `uv tool list`, `pipx list`, or
Map a natural language request to these parameters:

1. **Identify the connection** (`--name`): which database to use.
2. **Select the query** (`--query`): a named query from `config.json`, raw inline SQL, or a path to a `.sql` file prefixed with `@`.
3. **Supply parameters**: date variables and SQL bind parameters as extra `--key value` flags.
4. **Shape the output**: use `--first`, `--key`, `--value`, `--wrapper`, `--jsonkeys` to transform results.
2. **Select the query** (`--query`): prefer a named query from `config.json`, using connection-scoped queries when available; otherwise use raw inline SQL or a path to a `.sql` file prefixed with `@`.
3. **Resolve named-query precedence**: for `--name <conn> --query <name>`, sql2json checks `connection_queries.<conn>.<name>` first, then falls back to `queries.<name>`, then treats the value as raw SQL or `@file` when no named query exists.
4. **Supply parameters**: date variables and SQL bind parameters as extra `--key value` flags.
5. **Shape the output**: use `--first`, `--key`, `--value`, `--wrapper`, `--jsonkeys` to transform results.

---

Expand All @@ -77,12 +78,18 @@ Before calling a query, an agent can inspect what is configured:
sql2json --list-connections --config /path/to/config.json
# → ["default", "mysql", "reporting"]

# List available named queries
# List available named queries, grouped by scope
sql2json --list-queries --config /path/to/config.json
# → ["default", "sales_monthly", "total_users"]
# → {"global": ["default", "sales_monthly"], "connections": {"mysql": ["table_sizes"], "reporting": ["total_users"]}}

# Request the old flat global-query list when integrating with legacy callers
sql2json --list-queries legacy --config /path/to/config.json
# → ["default", "sales_monthly"]
```

Both flags print a JSON array to stdout and exit 0. If `--config` is omitted the tool uses its normal config lookup order.
`--list-connections` prints a JSON array to stdout and exits 0. `--list-queries` prints the scoped discovery object by default; `--list-queries legacy` prints the old flat global query array. If `--config` is omitted the tool uses its normal config lookup order.

When selecting a named query for a connection, prefer a query listed under `connections.<connection>`; if none matches, use the matching name from `global`. Runtime lookup follows the same precedence: scoped query first, global query fallback, then raw SQL/`@file` behavior.

---

Expand All @@ -98,6 +105,35 @@ If none exist, a read-only in-memory SQLite database is used (useful for testing

---

## Config schema for named queries

Use top-level `queries` for shared/global named queries and `connection_queries` for connection-specific SQL. `connection_queries` is the canonical schema for scoped queries: connection name -> query name -> SQL.

```json
{
"connections": {
"postgres": "postgresql+psycopg2://user:pass@host/db",
"mysql": "mysql+pymysql://user:pass@host/db"
},
"queries": {
"sales": "SELECT month, amount FROM sales",
"long_report": "@/path/to/report.sql"
},
"connection_queries": {
"postgres": {
"now": "SELECT CURRENT_TIMESTAMP AS ts"
},
"mysql": {
"now": "SELECT NOW() AS ts"
}
}
}
```

Existing `queries` configs remain valid; they are the fallback/global scope. Query values may be raw SQL or `@/path.sql` file references.

---

## Key flags

| Flag | Purpose | Example |
Expand All @@ -113,7 +149,7 @@ If none exist, a read-only in-memory SQLite database is used (useful for testing
| `--format` | Output format: `json` (default), `csv`, `excel` | `--format csv` |
| `--output` | Save to file instead of printing; filename supports `{CURRENT_DATE}` etc. | `--output report_{CURRENT_DATE}` |
| `--list-connections` | Print JSON array of configured connection names and exit | `--list-connections` |
| `--list-queries` | Print JSON array of configured query names and exit | `--list-queries` |
| `--list-queries` | Print configured query names and exit. Default shape is `{"global": [...], "connections": {...}}`; pass `--list-queries legacy` for the old flat global query array | `--list-queries` |

**Note:** Both `--list-connections` and `--list_connections` (underscore) are accepted by fire.

Expand Down Expand Up @@ -195,8 +231,10 @@ sql2json --name mysql --query sales_monthly --format csv --output sales_{CURRENT
from sql2json import run_query2json, run_query_by_name, list_connections, list_queries

# Discover what's configured
connections = list_connections("/path/to/config.json") # ["default", "mysql"]
queries = list_queries("/path/to/config.json") # ["default", "sales_monthly"]
connections = list_connections("/path/to/config.json")
queries = list_queries("/path/to/config.json") # global legacy names
scoped_queries = list_queries("/path/to/config.json", scoped=True) # {"global": [...], "connections": {...}}
mysql_queries = list_queries("/path/to/config.json", connection="mysql")

# Run a named query — returns list of dicts
rows = run_query_by_name(
Expand Down
47 changes: 43 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,12 +263,23 @@ docker compose run --rm sql2json --name pg --query sales_by_month --min_amount 4
# → [{"month": "January", "amount": 5000.0}, {"month": "March", "amount": 7100.75}]
```

Run the same demo queries against MySQL by switching `--name`:
Run the same shared demo queries against MySQL by switching `--name`:

```bash
docker compose run --rm sql2json --name mysql --query sales
```

The demo config also uses `connection_queries` for dialect-specific named queries. For example, `database_name` is defined separately for PostgreSQL and MySQL, while `version`, `sales`, and `sales_by_month` remain shared/global queries:

```bash
docker compose run --rm sql2json --list-queries
# → {"global": ["version", "sales", "sales_by_month"], "connections": {"pg": ["database_name"], "mysql": ["database_name"]}}

# Same query name, scoped SQL for each connection:
docker compose run --rm sql2json --name pg --query database_name
docker compose run --rm sql2json --name mysql --query database_name
```

Tear down when done:

```bash
Expand Down Expand Up @@ -338,6 +349,11 @@ cat > ~/.sql2json/config.json << 'EOF'
},
"queries": {
"default": "SELECT 1 AS a, 2 AS b"
},
"connection_queries": {
"default": {
"healthcheck": "SELECT 'ok' AS status"
}
}
}
EOF
Expand Down Expand Up @@ -381,6 +397,16 @@ Use `--config /path/to/config.json` to override.
"sales_monthly": "SELECT inv.month, SUM(inv.amount) AS sales FROM invoices inv WHERE inv.date >= :date_from",
"total_sales": "SELECT SUM(inv.amount) AS sales FROM invoices inv WHERE inv.date >= :date_from",
"long_query": "@/path/to/my_query.sql"
},
"connection_queries": {
"postgres": {
"now": "SELECT CURRENT_TIMESTAMP AS ts",
"table_sizes": "SELECT schemaname, relname, pg_total_relation_size(relid) AS bytes FROM pg_catalog.pg_statio_user_tables"
},
"mysql": {
"now": "SELECT NOW() AS ts",
"table_sizes": "SELECT table_schema, table_name, data_length + index_length AS bytes FROM information_schema.tables WHERE table_schema = DATABASE()"
}
}
}
```
Expand All @@ -389,6 +415,10 @@ Use `--config /path/to/config.json` to override.

Connection strings follow [SQLAlchemy URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#database-urls). Query values starting with `@` are treated as paths to `.sql` files.

`connection_queries` is the canonical schema for queries that are valid only for a specific connection or SQL dialect. Its shape is a top-level map of connection name to query-name to SQL. `queries` remains valid for shared/global named queries that can run unchanged across connections.

Named query lookup is connection-aware: `sql2json --name postgres --query now` first checks `connection_queries.postgres.now`; if it is not present, it falls back to `queries.now`; if neither exists, `--query` is treated as raw SQL or an `@/path.sql` file reference.

## CLI reference

```bash
Expand All @@ -408,7 +438,7 @@ sql2json [options] [--param value ...]
| `--format` | `json` | Output format: `json`, `csv`, `excel` |
| `--output` | _(stdout)_ | Save to file; filename supports `{CURRENT_DATE}` etc. |
| `--list-connections` | — | Print JSON array of configured connection names and exit |
| `--list-queries` | — | Print JSON array of configured query names and exit |
| `--list-queries` | — | Print configured query names and exit. Default shape is `{"global": [...], "connections": {...}}`; pass `--list-queries legacy` for the old flat global query list |

Extra `--key value` flags become SQL bind parameters (`:key` in your query).

Expand All @@ -421,9 +451,14 @@ sql2json --list-connections
# → ["default", "mysql", "reporting"]

sql2json --list-queries
# → ["default", "sales_monthly", "total_sales"]
# → {"global": ["default", "sales_monthly"], "connections": {"mysql": ["table_sizes"]}}

sql2json --list-queries legacy
# → ["default", "sales_monthly"]
```

Use the scoped discovery output to choose an appropriate `--name`/`--query` pair. For a given connection, an entry under `connections.<name>` overrides a same-named global query; otherwise the global query is used as a fallback.

## Date variables

Extra parameters whose values match a built-in variable are resolved to real dates before the query runs:
Expand Down Expand Up @@ -603,11 +638,15 @@ rows = run_query2json(
)

connections = list_connections("/path/to/config.json")
queries = list_queries("/path/to/config.json")
queries = list_queries("/path/to/config.json") # global legacy names
scoped_queries = list_queries("/path/to/config.json", scoped=True) # {"global": [...], "connections": {...}}
mysql_queries = list_queries("/path/to/config.json", connection="mysql")
```

Use `run_query2json()` for inline SQL, named queries, SQL files with `@/path.sql`, bind/date parameters, `first`, `key`, `value`, `wrapper`, `jsonkeys`, and `timezone`. Use `run_query_by_name()` when you specifically want the lower-level named connection/query call.

Named query resolution in the Python API matches the CLI: connection-scoped query first, global query fallback, then raw SQL or `@file` handling for `run_query2json()`.

Python API errors are normal Python exceptions. The CLI-only JSON stderr envelope is not used by the Python API.

Supported public imports are exported from `sql2json.__all__`. Internal helpers in `sql2json.sql2json`, `sql2json.__main__`, or `sql2json.parameter.parameter_parser` are implementation details and should not be imported by users. `sql2json.parameter.parse_parameter` remains public for date-variable resolution; lower-level date helper functions are private.
Expand Down
8 changes: 8 additions & 0 deletions docker/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,13 @@
"version": "SELECT version() AS version",
"sales": "SELECT * FROM sales",
"sales_by_month": "SELECT month, amount FROM sales WHERE amount >= :min_amount"
},
"connection_queries": {
"pg": {
"database_name": "SELECT current_database() AS database_name"
},
"mysql": {
"database_name": "SELECT DATABASE() AS database_name"
}
}
}
2 changes: 2 additions & 0 deletions examples/python_api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,6 @@ Public API used here:
- `list_queries`
- `parse_parameter`

`list_queries(path)` returns legacy global query names. Use `list_queries(path, scoped=True)` to return the full discovery shape (`{"global": [...], "connections": {...}}`), or `list_queries(path, connection="name")` to get the effective names for one connection, including connection-scoped queries and global fallbacks.

Implementation helpers inside `sql2json.sql2json`, `sql2json.__main__`, or `sql2json.parameter.parameter_parser` are internal and should not be imported by users.
6 changes: 6 additions & 0 deletions examples/python_api/list_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
config = {
"connections": {"default": "sqlite:///:memory:", "reporting": "sqlite:///:memory:"},
"queries": {"default": "SELECT 1 AS ok", "sales": "SELECT 42 AS total"},
"connection_queries": {
"default": {"sales": "SELECT 7 AS scoped_sales"},
"reporting": {"pipeline": "SELECT 8 AS pipeline"},
},
}

with tempfile.TemporaryDirectory() as tmp:
Expand All @@ -15,3 +19,5 @@

print(list_connections(str(config_path)))
print(list_queries(str(config_path)))
print(list_queries(str(config_path), scoped=True))
print(list_queries(str(config_path), connection="default"))
25 changes: 23 additions & 2 deletions skills/sql2json/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,10 +75,19 @@ The config file lives at `~/.sql2json/config.json`. Create it with at least one
},
"queries": {
"users": "SELECT id, email FROM users LIMIT 10"
},
"connection_queries": {
"mydb": {
"table_sizes": "SELECT schemaname, relname, pg_total_relation_size(relid) AS bytes FROM pg_catalog.pg_statio_user_tables"
}
}
}
```

Use `queries` for shared/global named queries. Use `connection_queries` as the canonical schema for connection-specific SQL: connection name -> query name -> SQL. Existing global `queries` configs remain valid and act as fallbacks.

Named query resolution is: `connection_queries.<connection>.<query>` first, then `queries.<query>`, then raw SQL or `@/path.sql` handling.

Supported connection strings follow SQLAlchemy format:

| Database | Example connection string |
Expand All @@ -93,8 +102,12 @@ Discover what is configured:
```bash
sql2json --list-connections
sql2json --list-queries
# → {"global": ["users"], "connections": {"mydb": ["table_sizes"]}}
sql2json --list-queries legacy # old flat global query list
```

When an agent receives a data request, discover connections and scoped queries before inventing SQL. Choose a connection, then prefer a query listed under that connection; fall back to a global query only if no scoped query matches.

## Common query patterns

**Named query:**
Expand All @@ -103,6 +116,13 @@ sql2json --list-queries
sql2json --name mydb --query users
```

**Connection-scoped named query:**

```bash
# Resolves to connection_queries.mydb.table_sizes before checking global queries.table_sizes
sql2json --name mydb --query table_sizes
```

**Inline SQL:**

```bash
Expand Down Expand Up @@ -227,8 +247,9 @@ See `references/agent-sync.md` for the full agent-target map.

1. `sql2json` writes errors to stderr. Check the exit code and stderr, not just stdout.
2. Named queries come from `~/.sql2json/config.json`; a missing query name is a config problem, not a skill problem.
3. `Decimal` columns are serialized as floats. `date`/`datetime` columns are not handled natively — cast to `VARCHAR` in SQL or use `--jsonkeys` if the driver returns them as strings.
4. Keep the canonical file in the repo; do not hand-edit copies in agent-local skill directories.
3. For named queries, always account for scoped-query precedence: `connection_queries.<connection>.<query>` overrides the same name in global `queries`.
4. `Decimal` columns are serialized as floats. `date`/`datetime` columns are not handled natively — cast to `VARCHAR` in SQL or use `--jsonkeys` if the driver returns them as strings.
5. Keep the canonical file in the repo; do not hand-edit copies in agent-local skill directories.

## Verification checklist

Expand Down
Loading