Skip to content

Add multi-partition key support to database.partitioned module#240

Merged
intel352 merged 4 commits intomainfrom
copilot/add-database-partition-key-support
Mar 3, 2026
Merged

Add multi-partition key support to database.partitioned module#240
intel352 merged 4 commits intomainfrom
copilot/add-database-partition-key-support

Conversation

Copy link
Contributor

Copilot AI commented Mar 3, 2026

  • Add PartitionConfig struct for individual partition key configuration
  • Update PartitionedDatabaseConfig to add Partitions []PartitionConfig field (backward-compatible)
  • Update PartitionedDatabase to normalize configs and expose multi-partition methods
  • Add MultiPartitionManager interface with PartitionConfigs(), EnsurePartitionForKey(), SyncPartitionsForKey()
  • Update db_create_partition step to support optional partitionKey field
  • Update db_sync_partitions step to support optional partitionKey field
  • Update storage plugin to parse partitions array from config
  • Update schema to add partitions field and partitionKey to step schemas
  • Add tests for all new functionality
  • Fix MultiPartitionManager doc comment (no longer says "two or more entries only")
  • Deep copy Tables slice in PartitionConfigs() to preserve encapsulation
  • Add type-specific validation for database.partitioned in schema/validate.go (require either partitions[] with valid entries, or partitionKey+tables in single-partition mode)
  • Add 12 new validation tests + 1 deep copy test

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Copilot AI changed the title [WIP] Add support for multiple database partition keys Add multi-partition key support to database.partitioned module Mar 3, 2026
@intel352 intel352 marked this pull request as ready for review March 3, 2026 18:37
Copilot AI review requested due to automatic review settings March 3, 2026 18:37
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds multi-partition-key support to the database.partitioned module so one module instance can manage multiple independent partitioning schemes (e.g., tenant_id LIST and api_version RANGE) while keeping backward compatibility with existing single-partition configs.

Changes:

  • Introduces PartitionConfig, Partitions []PartitionConfig, and MultiPartitionManager APIs in database.partitioned.
  • Updates step.db_create_partition / step.db_sync_partitions to optionally target a specific partitionKey.
  • Extends storage plugin parsing and schema documentation to support partitions[].

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
module/database_partitioned.go Adds multi-partition config model, normalization, and new per-key partition/sync methods.
module/database_partitioned_test.go Adds unit tests for multi-partition behavior and step dispatching.
module/pipeline_step_db_create_partition.go Adds optional partitionKey routing to EnsurePartitionForKey.
module/pipeline_step_db_sync_partitions.go Adds optional partitionKey routing to SyncPartitionsForKey.
plugins/storage/plugin.go Parses partitions array into module config.
schema/module_schema.go Updates module + step schema fields/descriptions for partitions[] and step partitionKey.

Comment on lines +525 to +531
{Key: "partitionKey", Label: "Partition Key", Type: FieldTypeString, Description: "Column name used for partitioning in single-partition mode (e.g. tenant_id). Ignored when 'partitions' is set.", Placeholder: "tenant_id"},
{Key: "tables", Label: "Tables", Type: FieldTypeArray, ArrayItemType: "string", Description: "Tables to manage partitions for in single-partition mode. Ignored when 'partitions' is set.", Placeholder: "forms"},
{Key: "partitionType", Label: "Partition Type", Type: FieldTypeSelect, Options: []string{"list", "range"}, DefaultValue: "list", Description: "PostgreSQL partition type for single-partition mode: list (FOR VALUES IN) or range (FOR VALUES FROM/TO). Ignored when 'partitions' is set."},
{Key: "partitionNameFormat", Label: "Partition Name Format", Type: FieldTypeString, DefaultValue: "{table}_{tenant}", Description: "Template for partition table names in single-partition mode. Supports {table} and {tenant} placeholders. Ignored when 'partitions' is set.", Placeholder: "{table}_{tenant}"},
{Key: "sourceTable", Label: "Source Table", Type: FieldTypeString, Description: "Table containing all tenant IDs for auto-partition sync in single-partition mode. Ignored when 'partitions' is set.", Placeholder: "tenants"},
{Key: "sourceColumn", Label: "Source Column", Type: FieldTypeString, Description: "Column in source table to query for tenant values in single-partition mode. Defaults to partitionKey.", Placeholder: "id"},
{Key: "partitions", Label: "Partitions", Type: FieldTypeArray, ArrayItemType: "object", Description: "List of independent partition key configurations. When set, overrides the single-partition fields. Each entry supports: partitionKey, tables, partitionType, partitionNameFormat, sourceTable, sourceColumn."},
Copy link

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By making partitionKey and tables non-required here, schema.ValidateConfig will no longer reject a database.partitioned module config that sets neither the legacy single-partition fields nor a non-empty partitions array. That invalid config will pass validation but later fail at runtime (e.g., EnsurePartition will error on an empty/invalid identifier). Consider adding explicit type-specific validation for database.partitioned in schema/validate.go (require either partitions with required fields per entry, or partitionKey+tables in single-partition mode).

Copilot uses AI. Check for mistakes.
@intel352
Copy link
Contributor

intel352 commented Mar 3, 2026

@copilot apply changes based on the comments in this thread

… PartitionConfigs, add database.partitioned config validation

Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 3, 2026

@copilot apply changes based on the comments in this thread

Applied all three review comments in a52815a:

  1. MultiPartitionManager doc comment — updated to say it's implemented by PartitionedDatabase and is primarily meaningful when multiple configs are configured, removing the incorrect "two or more entries" restriction.

  2. Deep copy in PartitionConfigs() — now deep-copies the Tables slice for each config so callers cannot mutate internal state; added TestPartitionedDatabase_PartitionConfigs_DeepCopiesTables to verify this.

  3. database.partitioned config validation — added a case "database.partitioned": block in schema/validate.go that enforces: in multi-partition mode (partitions is a non-empty array), each entry must have a non-empty partitionKey and a non-empty tables list; in single-partition mode (no partitions or empty array), partitionKey and tables are required. 11 new tests cover all branches.

@intel352 intel352 merged commit 08478c6 into main Mar 3, 2026
14 checks passed
@intel352 intel352 deleted the copilot/add-database-partition-key-support branch March 3, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants