Fix from-proto Postgres empty array insertion#130
Conversation
Root cause: PostgreSQL cannot determine the type of an empty array literal (array[]) without an
explicit type cast. ClickHouse can infer the type from the column definition.
Fix implemented:
1. Added TypedEmptyArray struct in db_proto/sql/dialect.go to carry SQL type information for empty arrays
2. Added CreateEmptyArray method to Dialect interface that dialects implement to create appropriate empty array values
3. PostgreSQL implementation (db_proto/sql/postgres/dialect.go):
- Returns a TypedEmptyArray with the SQL type (e.g., VARCHAR(255)[], INTEGER[])
4. ClickHouse implementation (db_proto/sql/click_house/dialect.go):
- Returns plain []interface{}{} since ClickHouse infers types from schema
5. Updated ValueToString in db_proto/sql/postgres/types.go to handle TypedEmptyArray by generating array[]::TYPE
6. Updated database.go to use dialect.CreateEmptyArray(fd, column) instead of []interface{}{} for empty repeated fields
Test updates:
- Fixed entityTypesTestWithEmptyRepeatedString to set OptionalStr_2Uint256 (separate pre-existing issue with empty strings in NUMERIC columns)
- Fixed time comparison in PostgreSQL test to use time.Equal() instead of deep equality (timezone location differences)
There was a problem hiding this comment.
Pull request overview
This PR fixes a PostgreSQL-specific issue where inserting rows with empty repeated (array) fields in from-proto mode fails with "cannot determine type of empty array". The fix introduces a dialect-specific approach using a TypedEmptyArray struct that carries SQL type information, allowing PostgreSQL to generate explicit type casts (array[]::TYPE) while ClickHouse continues using plain empty arrays.
Changes:
- Added
TypedEmptyArraystruct andCreateEmptyArraymethod to the Dialect interface for database-specific empty array handling - PostgreSQL dialect generates typed empty arrays with explicit casts, while ClickHouse uses plain empty slices
- Added integration tests for both PostgreSQL and ClickHouse to verify empty array handling, plus a UTC time fix for consistent test comparisons
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| db_proto/sql/dialect.go | Defines TypedEmptyArray struct and adds CreateEmptyArray method to the Dialect interface |
| db_proto/sql/postgres/dialect.go | Implements CreateEmptyArray to return typed empty arrays with SQL type information |
| db_proto/sql/postgres/types.go | Adds handling for TypedEmptyArray in ValueToString to generate array[]::TYPE syntax |
| db_proto/sql/click_house/dialect.go | Implements CreateEmptyArray to return plain empty slices (ClickHouse infers types from schema) |
| db_proto/sql/database.go | Updates empty array handling to use dialect.CreateEmptyArray() instead of hardcoded []interface{}{} |
| db_proto/sql/schema/table.go | Adds ColumnByFieldName helper method to look up columns by protobuf field name |
| tests/integration/db_proto_postgres_test.go | New integration test file for PostgreSQL empty array handling |
| tests/integration/db_proto_clickhouse_test.go | Adds test case for ClickHouse empty array handling and shared test helper function |
| tests/integration/helpers_test.go | Adds .UTC() to blockTime for consistent timezone handling in tests |
| CHANGELOG.md | Documents the PostgreSQL empty array fix |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
db_proto/sql/postgres/dialect.go
Outdated
| // PostgreSQL cannot determine the type of an empty array literal (e.g., array[]) | ||
| // without an explicit type cast, so we return a TypedEmptyArray that carries | ||
| // the SQL type information needed for the cast. | ||
| func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any { |
There was a problem hiding this comment.
The MapFieldType function is called with a potentially nil column parameter. This could cause a nil pointer dereference at line 58 where it accesses column.Nested and at line 83 where it accesses column.ConvertTo.
While the current test passes because the column is found in typical use cases, the code should defensively handle nil column to prevent panics in edge cases where the column cannot be found (e.g., when tableInfo is nil or when the field is not registered in the table).
Consider adding a nil check at the beginning of CreateEmptyArray or in MapFieldType to handle this case gracefully.
| func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any { | |
| func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any { | |
| if column == nil { | |
| // Defensive check: avoid panicking in MapFieldType when column is unexpectedly nil. | |
| // Use a safe default SQL type ("text") in this edge case. | |
| zap.L().Warn("CreateEmptyArray called with nil column, using default SQL type 'text'", | |
| zap.String("field", string(fd.FullName())), | |
| ) | |
| return sql2.TypedEmptyArray{SQLType: "text"} | |
| } |
…s across dialects
Root cause: PostgreSQL cannot determine the type of an empty array literal (array[]) without an explicit type cast. ClickHouse can infer the type from the column definition.
Fix implemented:
Test updates: