feat: PostgreSQL partitioning support for multi-tenant data isolation

[Copilot]

• Initial implementation of PostgreSQL LIST partitioning
• Fix appendTenantFilter to handle ORDER BY / LIMIT / GROUP BY / HAVING clauses properly
• Reject tenantKey for INSERT statements in step.db_exec
• Remove "postgresql" from isSupportedPartitionDriver for consistency
• Fix schema tenantKey placeholders to use "steps." prefix consistently
• Add configurable partitionNameFormat for dynamic table name resolution
• Add partitionType config to distinguish LIST vs RANGE partitioning
• Add auto-partition creation based on source table (sourceTable/sourceColumn config)
• Add step.db_sync_partitions for syncing partitions from source table
• Handle RANGE partition edge cases and lifecycle scenarios
• Update tests for all changes
• Resolve merge conflicts with main branch (allow_dynamic_sql, auth.m2m custom claims)

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Feature: PostgreSQL LIST partitioning support for multi-tenant data isolation</issue_title>
<issue_description>## Problem

In multi-tenant applications, PostgreSQL LIST partitioning by tenant/affiliate ID is a common pattern for data isolation:

CREATE TABLE forms (
    id UUID PRIMARY KEY,
    tenant_id TEXT NOT NULL,
    slug TEXT NOT NULL,
    ...
) PARTITION BY LIST (tenant_id);

CREATE TABLE forms_org_alpha PARTITION OF forms FOR VALUES IN ('org-alpha');
CREATE TABLE forms_org_beta  PARTITION OF forms FOR VALUES IN ('org-beta');

This provides strong data isolation guarantees (tenant A's queries physically cannot touch tenant B's partition) and enables independent per-tenant maintenance operations.

Currently, there is no engine-level support for this pattern. Applications must handle partition management themselves, and pipeline steps like step.db_query / step.db_exec require manual WHERE tenant_id = $X clauses in every query.

Proposed Solution

1. Partition-aware module type: database.partitioned

A module that automatically manages LIST partitions when new tenants are onboarded:

- name: db
  type: database.partitioned
  config:
    driver: pgx
    dsn: "${DB_DSN}"
    partition_key: tenant_id
    tables:
      - forms
      - form_submissions
      - follow_ups

When a new tenant is registered, the module creates partitions for all configured tables.

2. Automatic tenant scoping in step.db_query / step.db_exec

An optional tenant_key config that automatically appends AND tenant_id = $N to queries:

- name: fetch
  type: step.db_query
  config:
    database: db
    tenant_key: auth.tenant_id   # from JWT claim or pipeline context
    query: "SELECT * FROM forms WHERE active = true ORDER BY created_at DESC"

This would reduce boilerplate and eliminate the risk of forgetting the tenant filter on a query.

Use Case

1. Multi-tenant SaaS where each organization's data must be physically isolated
2. New tenants are onboarded dynamically — partitions must be created at runtime
3. Every data query must be scoped to the requesting tenant's partition
4. Currently requires custom step types for partition management and manual WHERE clauses in every pipeline query

Benefits

• Security: Automatic tenant scoping eliminates accidental cross-tenant data leaks from missing WHERE clauses
• Performance: PostgreSQL partition pruning ensures queries only scan the relevant tenant's data
• Maintenance: Per-partition operations (VACUUM, index rebuilds) don't lock other tenants
• Simplicity: Application developers don't need to remember tenant filtering on every query</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Feature: PostgreSQL LIST partitioning support for multi-tenant data isolation #232

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[Copilot]

Pull request overview

Adds first-class PostgreSQL LIST partitioning support to the workflow engine to enable multi-tenant isolation, including runtime partition creation and optional automatic tenant scoping in DB pipeline steps.

Changes:

• Introduces database.partitioned module with PartitionKeyProvider / PartitionManager and idempotent EnsurePartition.
• Adds step.db_create_partition and enhances step.db_query / step.db_exec with optional tenantKey scoping.
• Registers new module/step types across schema + plugins + wfctl type registry and adds/updates tests.

Reviewed changes

Copilot reviewed 16 out of 21 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
schema/snippets_export.go	Formatting-only alignment adjustments.
schema/schema.go	Adds new core module/step types to schema type lists.
schema/module_schema.go	Registers schema for database.partitioned, adds tenantKey fields, adds step.db_create_partition schema.
plugins/storage/plugin_test.go	Updates expected module type/schema counts for storage plugin.
plugins/storage/plugin.go	Registers database.partitioned module type + factory + module schema in storage plugin.
plugins/pipelinesteps/plugin_test.go	Adds step.db_create_partition to expected steps list.
plugins/pipelinesteps/plugin.go	Registers step.db_create_partition in step types + factories.
module/sql_placeholders.go	Adds appendTenantFilter helper used for tenant scoping.
module/pipeline_step_token_revoke_test.go	Removes trailing blank line (formatting).
module/pipeline_step_sandbox_exec.go	Formatting-only alignment changes.
module/pipeline_step_db_tenant_test.go	New tests for tenantKey scoping and appendTenantFilter.
module/pipeline_step_db_query_test.go	Formatting-only alignment change.
module/pipeline_step_db_query.go	Adds tenantKey support and applies tenant filter + extra param.
module/pipeline_step_db_exec.go	Adds tenantKey support and applies tenant filter + extra param.
module/pipeline_step_db_create_partition.go	New step.db_create_partition implementation.
module/kafka_broker.go	Import ordering + formatting alignment.
module/http_server.go	Formatting-only alignment changes.
module/database_partitioned_test.go	New tests for partitioned database behaviors + create-partition step.
module/database_partitioned.go	New database.partitioned module implementation.
module/cache_redis.go	Formatting-only alignment change.
cmd/wfctl/type_registry.go	Registers new module/step types + config keys in CLI registry.

[Copilot]

isSupportedPartitionDriver treats "postgresql" as supported, but elsewhere the codebase (e.g. database.workflow schema/options and sql placeholder normalization) only recognizes "postgres"/"pgx"/"pgx/v5" as Postgres driver names. Unless there is a real sql driver registered under "postgresql" in this repo, consider removing it here (and in the test) or adding it consistently across driver detection + schema options.

[Copilot]

tenantKey scoping is applied to all db_exec statements by blindly appending WHERE/AND. This will generate invalid SQL for INSERT statements (and for any statement types that don't support WHERE). Consider either rejecting tenantKey for non-UPDATE/DELETE statements (with a clear error) or implementing statement-aware handling (e.g., require the tenant column be part of INSERT values instead of appending a filter).

[Copilot]

This test currently codifies behavior that generates invalid SQL (placing WHERE after ORDER BY). Once tenant scoping is supported for typical SELECTs with ORDER BY/LIMIT, appendTenantFilter should insert the tenant predicate before ORDER BY and this expectation should be updated accordingly.

[intel352]

@copilot apply changes based on the comments in this thread

1. When using table partitions, it would be good to optionally support a dynamic table name format. Ideally an option like this would be set once in the workflow config, to generate a table name with a predetermined format. So we'd define something like {tableName}_{tenantId} or {tenantId}_{tableName} and call a helper method that would resolve the table's tenant partition at query time. This could be an alternative to auto-appending an AND criteria to the WHERE statement, since individual table partitions inherently have an AND built in for the partition id.

2. We need ability to generate partitions automatically, based on a value existing in the partition table source. i.e. - if we add a new tenant to the tenants table, then every table that's partitioned should have a migration step occur to introduce the new partition.

3. Think about whether there's any edge cases or other lifecycle scenarios for table partitions that we need to consider. The scenarios I've mentioned above are specific to LIST partitions, but postgres also supports RANGE partitions, do we need to be mindful of that? If we add any automagic functionality specific to LIST partitions, do we have to distinguish between LIST and RANGE partitioned tables to ensure they're treated differently? Go ahead and handle any edge scenarios that you come up with.

[Copilot]

@copilot apply changes based on the comments in this thread

1. When using table partitions, it would...

All three requests addressed in commit d440278:

1. Dynamic table name format — Added partitionNameFormat config with {table} and {tenant} placeholders (default: {table}_{tenant}). Can be set to {tenant}_{table} or any other combination. The PartitionKeyProvider interface now includes PartitionTableName(parentTable, tenantValue) for runtime resolution. Example:

config:
  partitionNameFormat: "{tenant}_{table}"

2. Auto-partition from source table — Added sourceTable and sourceColumn config. SyncPartitionsFromSource() queries SELECT DISTINCT <column> FROM <sourceTable> and calls EnsurePartition for each value. New step.db_sync_partitions triggers this from a pipeline:

config:
  sourceTable: tenants
  sourceColumn: id

3. RANGE partition support — Added partitionType config (list or range). LIST uses FOR VALUES IN ('value'), RANGE uses FOR VALUES FROM ('value') TO ('value\x00'). "postgresql" removed from supported drivers for consistency. INSERT statements with tenantKey are now rejected with a clear error. appendTenantFilter now correctly inserts predicates before ORDER BY/LIMIT/GROUP BY/HAVING clauses.

[intel352]

@copilot Address the merge conflict on the branch

[Copilot]

@copilot Address the merge conflict on the branch

In commit 6a2a32d, I've incorporated all changes from main into our branch files — allow_dynamic_sql (PR #234), auth.m2m custom claims (PR #236), and the new pipeline_step_db_dynamic.go helper. Both db_query and db_exec now support allowDynamicSQL and tenantKey together. All tests pass.

However, I'm unable to resolve the Git-level merge conflict directly because I cannot run git merge + git commit or git rebase + git push --force from this environment. You'll need to merge main into this branch from the GitHub UI — the conflicts should be trivially resolvable since both sides now have matching content (just accept "ours" for each conflict).