Add multi-partition key support to database.partitioned module#240
Add multi-partition key support to database.partitioned module#240
Conversation
Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds multi-partition-key support to the database.partitioned module so one module instance can manage multiple independent partitioning schemes (e.g., tenant_id LIST and api_version RANGE) while keeping backward compatibility with existing single-partition configs.
Changes:
- Introduces
PartitionConfig,Partitions []PartitionConfig, andMultiPartitionManagerAPIs indatabase.partitioned. - Updates
step.db_create_partition/step.db_sync_partitionsto optionally target a specificpartitionKey. - Extends storage plugin parsing and schema documentation to support
partitions[].
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
module/database_partitioned.go |
Adds multi-partition config model, normalization, and new per-key partition/sync methods. |
module/database_partitioned_test.go |
Adds unit tests for multi-partition behavior and step dispatching. |
module/pipeline_step_db_create_partition.go |
Adds optional partitionKey routing to EnsurePartitionForKey. |
module/pipeline_step_db_sync_partitions.go |
Adds optional partitionKey routing to SyncPartitionsForKey. |
plugins/storage/plugin.go |
Parses partitions array into module config. |
schema/module_schema.go |
Updates module + step schema fields/descriptions for partitions[] and step partitionKey. |
| {Key: "partitionKey", Label: "Partition Key", Type: FieldTypeString, Description: "Column name used for partitioning in single-partition mode (e.g. tenant_id). Ignored when 'partitions' is set.", Placeholder: "tenant_id"}, | ||
| {Key: "tables", Label: "Tables", Type: FieldTypeArray, ArrayItemType: "string", Description: "Tables to manage partitions for in single-partition mode. Ignored when 'partitions' is set.", Placeholder: "forms"}, | ||
| {Key: "partitionType", Label: "Partition Type", Type: FieldTypeSelect, Options: []string{"list", "range"}, DefaultValue: "list", Description: "PostgreSQL partition type for single-partition mode: list (FOR VALUES IN) or range (FOR VALUES FROM/TO). Ignored when 'partitions' is set."}, | ||
| {Key: "partitionNameFormat", Label: "Partition Name Format", Type: FieldTypeString, DefaultValue: "{table}_{tenant}", Description: "Template for partition table names in single-partition mode. Supports {table} and {tenant} placeholders. Ignored when 'partitions' is set.", Placeholder: "{table}_{tenant}"}, | ||
| {Key: "sourceTable", Label: "Source Table", Type: FieldTypeString, Description: "Table containing all tenant IDs for auto-partition sync in single-partition mode. Ignored when 'partitions' is set.", Placeholder: "tenants"}, | ||
| {Key: "sourceColumn", Label: "Source Column", Type: FieldTypeString, Description: "Column in source table to query for tenant values in single-partition mode. Defaults to partitionKey.", Placeholder: "id"}, | ||
| {Key: "partitions", Label: "Partitions", Type: FieldTypeArray, ArrayItemType: "object", Description: "List of independent partition key configurations. When set, overrides the single-partition fields. Each entry supports: partitionKey, tables, partitionType, partitionNameFormat, sourceTable, sourceColumn."}, |
There was a problem hiding this comment.
By making partitionKey and tables non-required here, schema.ValidateConfig will no longer reject a database.partitioned module config that sets neither the legacy single-partition fields nor a non-empty partitions array. That invalid config will pass validation but later fail at runtime (e.g., EnsurePartition will error on an empty/invalid identifier). Consider adding explicit type-specific validation for database.partitioned in schema/validate.go (require either partitions with required fields per entry, or partitionKey+tables in single-partition mode).
|
@copilot apply changes based on the comments in this thread |
… PartitionConfigs, add database.partitioned config validation Co-authored-by: intel352 <77607+intel352@users.noreply.github.com>
Applied all three review comments in a52815a:
|
PartitionConfigstruct for individual partition key configurationPartitionedDatabaseConfigto addPartitions []PartitionConfigfield (backward-compatible)PartitionedDatabaseto normalize configs and expose multi-partition methodsMultiPartitionManagerinterface withPartitionConfigs(),EnsurePartitionForKey(),SyncPartitionsForKey()db_create_partitionstep to support optionalpartitionKeyfielddb_sync_partitionsstep to support optionalpartitionKeyfieldpartitionsarray from configpartitionsfield andpartitionKeyto step schemasMultiPartitionManagerdoc comment (no longer says "two or more entries only")Tablesslice inPartitionConfigs()to preserve encapsulationdatabase.partitionedinschema/validate.go(require eitherpartitions[]with valid entries, orpartitionKey+tablesin single-partition mode)🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.