feat: dynamic table name support in db_query, db_exec, db_query_cached steps

[Copilot]

• Add validateSQLIdentifier and resolveDynamicSQL helper functions (alphanumeric/underscore/hyphen allowlist)
• Add allowDynamicSQL bool field to DBQueryStep, DBExecStep, DBQueryCachedStep
• Modify factory functions: read allow_dynamic_sql config; skip template rejection when true
• Modify Execute functions: use resolveDynamicSQL before app-nil check so injection fails fast
• Update schema descriptions for allow_dynamic_sql config field in all three step types
• Add tests: dynamic table name success cases, SQL injection rejection, empty identifier rejection, unclosed action error
• Fix: restrict validateSQLIdentifier to ASCII-only (was Unicode-wide)
• Fix: return error on unclosed {{ template action instead of silently returning unchanged query
• Fix: resolve each {{ }} occurrence independently (no de-dup) so non-deterministic funcs like {{uuid}} work correctly per occurrence

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature: Dynamic table name support in step.db_query and step.db_exec</issue_title>
<issue_description>## Problem

In multi-tenant applications using PostgreSQL LIST partitioning, some queries need to target tenant-specific tables where the table name is computed at runtime based on the tenant/affiliate ID. For example:

-- The table name includes the tenant identifier
SELECT backend_url, feature_flags, settings
FROM affiliate_routing_org_alpha   -- dynamically constructed from tenant ID
WHERE route_key = $1

Currently, step.db_query and step.db_exec reject template expressions in SQL queries — step.db_query_cached explicitly checks strings.Contains(query, "{{") and returns an error. This means any query with a dynamic table name must be implemented as a custom Go step type, even if the rest of the query logic is straightforward.

Proposed Solution

Allow template expressions in the table or query field so the table name can be resolved from the pipeline context:

Option A: Dedicated table config field with templating

- name: lookup
  type: step.db_query
  config:
    database: db
    table: "affiliate_routing_{{index .steps.parse.headers \"X-Tenant-Id\"}}"
    query: "SELECT backend_url, feature_flags, settings FROM {{.table}} WHERE route_key = $1"
    params:
      - "{{.steps.parse.path_params.key}}"
    mode: single

Option B: Allow templates in SQL with explicit opt-in

- name: lookup
  type: step.db_query
  config:
    database: db
    allow_dynamic_sql: true
    query: >
      SELECT backend_url, feature_flags, settings
      FROM "affiliate_routing_{{index .steps.parse.headers \"X-Tenant-Id\"}}"
      WHERE route_key = $1
    params:
      - "{{.steps.parse.path_params.key}}"
    mode: single

Important: The dynamic table name must be sanitized to prevent SQL injection. Only alphanumeric characters, underscores, and hyphens should be allowed in the resolved table name. The engine should reject or quote-escape any resolved name containing special characters.

Use Case

1. Multi-tenant routing table where each tenant has its own partition: routing_{tenant_id}
2. Per-tenant configuration tables: config_{tenant_id}
3. Sharded data access patterns where the shard key is part of the table name
4. Currently requires writing a custom Go step type for what is otherwise a simple parameterized query

Security Considerations

• Dynamic table names must be strictly sanitized (allowlist of characters) to prevent SQL injection
• Template resolution should happen before SQL execution, with the resolved name validated
• Consider using PostgreSQL identifier quoting ("table_name") automatically
• An explicit opt-in flag (allow_dynamic_sql: true) would make this a conscious security decision

Current Workaround

Applications must implement a full custom step type in Go just to construct a table name dynamically, even though the query itself is a simple SELECT with parameterized WHERE clauses. This adds maintenance overhead and prevents the query from being expressed purely in YAML.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Feature: Dynamic table name support in step.db_query and step.db_exec #233

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Pull request overview

Adds an explicit, opt-in mechanism for templating SQL identifiers at runtime in the pipeline DB steps to support multi-tenant/partition-per-tenant table naming while attempting to guard against SQL injection.

Changes:

• Introduces allow_dynamic_sql to step.db_query, step.db_exec, and step.db_query_cached, enabling runtime template resolution in query.
• Adds shared dynamic SQL resolution + identifier validation utilities (resolveDynamicSQL, validateSQLIdentifier).
• Expands unit tests to cover dynamic table names and injection rejection for db_query, db_exec, and db_query_cached.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
schema/module_schema.go	Documents and registers allow_dynamic_sql for the three DB steps.
module/pipeline_step_db_dynamic.go	New helper for resolving {{ }} in SQL and validating interpolated identifiers.
module/pipeline_step_db_query.go	Adds allowDynamicSQL handling; resolves dynamic SQL before DB/app access.
module/pipeline_step_db_exec.go	Adds allowDynamicSQL handling; resolves dynamic SQL before DB/app access.
module/pipeline_step_db_query_cached.go	Adds allowDynamicSQL handling; resolves dynamic SQL early and threads resolved query into execution path.
module/pipeline_step_db_query_test.go	Adds tests for dynamic table name, injection rejection, and empty identifier rejection.
module/pipeline_step_db_exec_test.go	Adds tests for dynamic table name and injection rejection.
module/pipeline_step_db_query_cached_test.go	Adds tests for dynamic table name and injection rejection.

[intel352]

@copilot apply changes based on the comments in this thread

[Copilot]

@copilot apply changes based on the comments in this thread

Applied in e6d10a4:

1. ASCII-only validation – validateSQLIdentifier now checks character ranges explicitly ('a'-'z', 'A'-'Z', '0'-'9', _, -) instead of unicode.IsLetter/IsDigit, matching the documented allowlist exactly.
2. Unclosed action error – resolveDynamicSQL now returns fmt.Errorf("dynamic SQL: unclosed template action in query (missing closing '}}')") when }} is not found, instead of silently returning the unchanged query.
3. Per-occurrence resolution – the two-pass extract-then-resolve approach is replaced with a single streaming pass that resolves and validates each {{ }} occurrence in-place, so non-deterministic functions like {{uuid}} produce an independent value per occurrence.