feat: Add step.db_query_cached — single-step DB query with in-process TTL caching

[Copilot]

• Create module/pipeline_step_db_query_cached.go with DBQueryCachedStep (in-memory TTL cache + DB query)
• Create module/pipeline_step_db_query_cached_test.go with unit tests
• Register step.db_query_cached in plugins/pipelinesteps/plugin.go
• Add step.db_query_cached to schema/schema.go (coreStepTypes list)
• Add schema entry in schema/module_schema.go
• Add type entry in cmd/wfctl/type_registry.go
• Fix cache stampede: double-check under write lock before querying DB
• Evict expired entries on read to prevent unbounded memory growth
• Validate cache_ttl > 0 (reject zero/negative durations)
• Clarify schema description: $N is canonical placeholder, auto-converted to ? for SQLite
• Add test: $1 placeholder normalization with SQLite driver
• Add test: cache hit truly skips DB (close DB after first call, verify second succeeds)
• Add tests: zero and negative TTL are rejected

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature: Add step.db_query_cached for single-step database query with TTL caching</issue_title>
<issue_description>## Problem

A common pattern in pipelines is to query a database for configuration or routing data that changes infrequently. Currently, implementing this requires three separate steps:

1. step.cache_get — check if the result is cached
2. step.db_query — execute the query on cache miss
3. step.cache_set — store the result in cache

This is verbose, error-prone (conditional logic between steps), and doesn't support per-entry TTL management. For frequently-executed pipelines (e.g., every HTTP request through a gateway), this adds unnecessary latency and complexity.

Use Case

• Tenant configuration lookup: Query a config table once per tenant, cache for 5 minutes
• Feature flag resolution: Look up feature flags from database, cache with short TTL
• Route resolution: Resolve backend URLs from a routing table, cache per-tenant
• Rate limit config: Load rate limit settings per API key, cache with TTL

All of these share the same pattern: query database → cache result → return cached value on subsequent calls → re-query when TTL expires.

Proposed Solution

Add a step.db_query_cached step type that combines query execution with in-memory (or external) caching:

steps:
  - name: lookup-config
    type: step.db_query_cached
    config:
      database: db                          # database module reference
      query: "SELECT backend_url, settings FROM routing_config WHERE tenant_id = $1 LIMIT 1"
      params: ["{{.steps.parse.headers.X-Tenant-Id}}"]
      cache_key: "tenant_config:{{.steps.parse.headers.X-Tenant-Id}}"
      cache_ttl: "5m"
      scan_fields: [backend_url, settings]  # column names to map to output fields

Behavior

1. Check in-memory cache for cache_key
2. If cached and TTL not expired → return cached result immediately
3. If cache miss or expired → execute query, cache result, return result
4. Thread-safe cache access (concurrent pipeline executions)

Configuration

Key	Type	Default	Description
database	string	"db"	Name of the database.workflow module
query	string	required	SQL SELECT query with $N placeholders
params	list	[]	Template-resolved query parameters
cache_key	string	required	Template-resolved cache key
cache_ttl	duration	"5m"	TTL for cached entries
scan_fields	list	auto	Column names for output mapping

Output

The step outputs a map with the scanned fields:

{
  "backend_url": "https://api.example.com",
  "settings": {"feature_x": true}
}

Alternatives

1. Three-step pattern (cache_get → db_query → cache_set): Works but verbose and doesn't support conditional execution well (cache hit should skip the query).
2. External caching layer (Redis): Overkill for read-heavy config lookups within a single process. In-memory caching with TTL is sufficient for most cases.
3. Custom step: Currently the only viable approach for single-step cached queries.</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Feature: Add step.db_query_cached for single-step database query with TTL caching #225

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[Copilot]

Pull request overview

Adds a new pipeline step type, step.db_query_cached, to reduce the common cache_get → db_query → cache_set pattern into a single step that performs a DB SELECT and caches the first-row result in-process with a TTL.

Changes:

• Implements DBQueryCachedStep with an in-step TTL cache keyed by a template-resolved cache_key.
• Registers the new step across plugin registration, schema/type registries, and CLI known-types.
• Adds unit tests covering cache hit/miss, TTL expiry, scan_fields filtering, template params, and validation errors.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
module/pipeline_step_db_query_cached.go	New cached DB query step implementation (TTL map + query execution).
module/pipeline_step_db_query_cached_test.go	Unit tests for cached behavior and config validation.
plugins/pipelinesteps/plugin.go	Registers step.db_query_cached in step types and factories.
plugins/pipelinesteps/plugin_test.go	Updates expected step factory list to include the new step.
schema/schema.go	Adds step.db_query_cached to core step/module types list.
schema/module_schema.go	Adds schema definition for the new step’s config fields.
cmd/wfctl/type_registry.go	Adds CLI registry entry for step.db_query_cached.
example/go.mod / example/go.sum	Updates indirect dependencies in the example module.

[intel352]

@copilot apply changes based on the comments in this thread

Also address failing linter checks

[Copilot]

@copilot apply changes based on the comments in this thread

Also address failing linter checks

Applied all review feedback in d2cfbe8:

• Cache stampede: Added double-check under write lock — after acquiring the write lock, we re-verify the cache before querying the DB, so concurrent goroutines for the same key only execute one DB query.
• Expired entry eviction: Expired entries are deleted on read (before the write lock is released), preventing unbounded memory growth for high-cardinality cache keys.
• TTL validation: cache_ttl must be > 0; zero or negative values now return a clear error at factory creation time.
• Schema description: Clarified that $N is the canonical placeholder format, automatically converted to ? for SQLite drivers.
• New tests: Added TestDBQueryCachedStep_PostgresPlaceholderNormalization (verifies $1→? conversion with SQLite driver), TestDBQueryCachedStep_CacheHitSkipsDB (closes the DB after first call; second call must succeed from cache), and TestDBQueryCachedStep_ZeroTTLRejected / TestDBQueryCachedStep_NegativeTTLRejected.