step.db_query_cached: add mode field for single/list result support

[Copilot]

step.db_query_cached was missing mode support entirely — it only returned a flat map of the first row's columns, making it inconsistent with step.db_query and unable to cache list query results.

Changes

• module/pipeline_step_db_query_cached.go

  • Added mode field ("single" | "list"); invalid values are rejected at factory time
  • Omitting mode preserves the original legacy flat column map output (backward compatible)
  • mode: single → { "row": {...}, "found": bool, "cache_hit": bool } (consistent with step.db_query)
  • mode: list → { "rows": [...], "count": N, "cache_hit": bool } (new)
  • Replaced shallow copyMap with deepCopyMap that recursively copies nested map[string]any and []map[string]any, preventing cache corruption and cross-goroutine data races
  • Integrated allow_dynamic_sql support: when enabled, template expressions in the query are resolved at runtime and validated as safe SQL identifiers before execution

• module/pipeline_step_db_dynamic.go: Added resolveDynamicSQL and validateSQLIdentifier helpers; SQL identifier allowlist restricted to ASCII letters, digits, and underscores only (hyphens removed as they are not valid unquoted SQL identifiers)

• schema/module_schema.go: Added mode select field and allow_dynamic_sql boolean field to step.db_query_cached config schema; updated allow_dynamic_sql description for step.db_query, step.db_query_cached, and step.db_exec to reflect the tightened identifier allowlist

• module/pipeline_step_db_query_cached_test.go: Updated existing tests for legacy flat output (no mode) and new row/found shape (explicit mode: single); added coverage for list mode (cache miss/hit/empty), invalid mode rejection, explicit single mode found/not-found, dynamic table name, and SQL injection rejection

Output format

# mode omitted — legacy flat output (backward compatible)
- step: get-config
  type: step.db_query_cached
  config:
    database: app-db
    query: SELECT backend_url FROM routing_config WHERE tenant_id = $1
    params: ["{{ .steps.parse.headers.X-Tenant-Id }}"]
    cache_key: "config:{{ .steps.parse.headers.X-Tenant-Id }}"
# output: { "backend_url": "https://...", "cache_hit": false }

# mode: single
- step: get-user
  type: step.db_query_cached
  config:
    database: app-db
    query: SELECT id, email FROM users WHERE id = $1
    params: ["{{ .steps.parse.path_params.id }}"]
    mode: single
    cache_key: "user:{{ .steps.parse.path_params.id }}"
# output: { "row": {"id": "u1", "email": "..."}, "found": true, "cache_hit": false }

# mode: list
- step: list-products
  type: step.db_query_cached
  config:
    database: catalog-db
    query: SELECT id, name, price FROM products WHERE active = true ORDER BY name
    mode: list
    cache_key: products:active
    cache_ttl: 2m
# output: { "rows": [{...}, {...}], "count": 42, "cache_hit": false }

# allow_dynamic_sql: dynamic table name per tenant
- step: get-tenant-config
  type: step.db_query_cached
  config:
    database: app-db
    query: SELECT * FROM config_{{.steps.auth.tenant}} WHERE id = $1
    params: ["{{ .steps.parse.path_params.id }}"]
    mode: single
    cache_key: "config:{{.steps.auth.tenant}}:{{ .steps.parse.path_params.id }}"
    allow_dynamic_sql: true
# output: { "row": {...}, "found": true, "cache_hit": false }

Note: Existing configs that explicitly set mode: single will now receive the row/found envelope instead of a flat column map. Configs that omit mode retain the original flat output and are unaffected.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>step.db_query_cached: add support for multi-row (list) results</issue_title>
<issue_description>## Summary

step.db_query_cached currently only supports single-row query results. When used with mode: list, it returns a flat map of the first row's columns instead of the expected rows: [...] array format that step.db_query provides.

This limits cached queries to lookup-by-key patterns (e.g., fetching a single user or config value), while common read-heavy list endpoints (e.g., listing all records in a table) cannot benefit from in-process caching.

Expected Behavior

step.db_query_cached should support both mode: single and mode: list, matching the output format of step.db_query:

• mode: single → { "row": {...}, "found": true/false } (works today)
• mode: list → { "rows": [...] } (not supported)

Current Workaround

Use step.db_query (uncached) for list queries. This works but means frequently-read list endpoints cannot benefit from TTL-based caching without an external cache layer.

Use Case

Any application with read-heavy list endpoints that change infrequently — admin dashboards, settings pages, catalog listings — would benefit from being able to cache list query results with a short TTL.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes step.db_query_cached: add support for multi-row (list) results #237

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[intel352]

@copilot Address the merge conflicts on the branch

[Copilot]

@copilot Address the merge conflicts on the branch

Resolved in 31e339a. The branch now includes allowDynamicSQL from main (#234) merged with the mode support from this PR:

• Added module/pipeline_step_db_dynamic.go (the resolveDynamicSQL helper from main)
• Added allowDynamicSQL field + factory parsing + dynamic SQL resolution in Execute
• Added allow_dynamic_sql config field to schema
• Carried over and updated the DynamicTableName and DynamicSQL_RejectsInjection tests (adapted for the new row/found output format)

[Copilot]

Pull request overview

Adds full mode support to step.db_query_cached so cached queries can return either a single-row envelope (row/found) or multi-row results (rows/count), aligning its behavior with step.db_query.

Changes:

• Add mode (single/list) parsing + validation to step.db_query_cached, and update its output shape to match step.db_query.
• Update step.db_query_cached module schema to include mode and to describe the new output contract.
• Update and expand tests to cover list-mode behavior, cache hit/miss, and the new single-mode envelope output.

Reviewed changes

Copilot reviewed 5 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
module/pipeline_step_db_query_cached.go	Adds mode handling and returns row/found (single) or rows/count (list).
module/pipeline_step_db_query_cached_test.go	Updates existing assertions for the new output shape and adds list-mode coverage.
schema/module_schema.go	Documents mode for step.db_query_cached and updates output description.
schema/schema.go	Formatting-only alignment in schema generation map literal.
schema/snippets_export.go	Formatting-only alignment in XML struct tags.
module/pipeline_step_db_query_test.go	Formatting-only alignment.
module/pipeline_step_token_revoke_test.go	Removes trailing blank line.
module/pipeline_step_sandbox_exec.go	Formatting-only alignment in struct literal.
module/kafka_broker.go	Import/field alignment formatting.
module/http_server.go	Formatting-only alignment in struct tags.
module/cache_redis.go	Formatting-only alignment / comment spacing.

[Copilot]

Now that results are nested structures (row is a map and rows is a slice of maps), caching via copyMap (a shallow copy) will return shared references to the cached nested data. If any downstream code mutates row/rows (or if future steps do), it can corrupt the cache and create cross-goroutine data races. Consider deep-copying nested maps/slices when storing to or returning from the cache.

[intel352]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied all three review items in 2efb1d1:

1. Deep copy — replaced copyMap (shallow) with deepCopyMap that recursively copies nested map[string]any and []map[string]any, preventing cache corruption and cross-goroutine data races.

2. SQL identifier allowlist — removed hyphens from validateSQLIdentifier (unquoted hyphens are not valid SQL identifiers) and updated all three allow_dynamic_sql schema descriptions to say "only letters, digits, and underscores".

3. Backward compatibility — omitting mode now returns the original legacy flat column map (e.g. {"name": "Acme", "cache_hit": false}). Only explicit mode: "single" uses the new row/found envelope. Tests without mode were updated accordingly.