Skip to content

Fix from-proto Postgres empty array insertion#130

Merged
billettc merged 2 commits intodevelopfrom
fix/from-proto-pg-empty-array
Feb 24, 2026
Merged

Fix from-proto Postgres empty array insertion#130
billettc merged 2 commits intodevelopfrom
fix/from-proto-pg-empty-array

Conversation

@maoueh
Copy link
Contributor

@maoueh maoueh commented Feb 23, 2026

Root cause: PostgreSQL cannot determine the type of an empty array literal (array[]) without an explicit type cast. ClickHouse can infer the type from the column definition.

Fix implemented:

  1. Added TypedEmptyArray struct in db_proto/sql/dialect.go to carry SQL type information for empty arrays
  2. Added CreateEmptyArray method to Dialect interface that dialects implement to create appropriate empty array values
  3. PostgreSQL implementation (db_proto/sql/postgres/dialect.go):
    • Returns a TypedEmptyArray with the SQL type (e.g., VARCHAR(255)[], INTEGER[])
  4. ClickHouse implementation (db_proto/sql/click_house/dialect.go):
    • Returns plain []interface{}{} since ClickHouse infers types from schema
  5. Updated ValueToString in db_proto/sql/postgres/types.go to handle TypedEmptyArray by generating array[]::TYPE
  6. Updated database.go to use dialect.CreateEmptyArray(fd, column) instead of []interface{}{} for empty repeated fields

Test updates:

  • Fixed entityTypesTestWithEmptyRepeatedString to set OptionalStr_2Uint256 (separate pre-existing issue with empty strings in NUMERIC columns)
  • Fixed time comparison in PostgreSQL test to use time.Equal() instead of deep equality (timezone location differences)

Root cause: PostgreSQL cannot determine the type of an empty array literal (array[]) without an
explicit type cast. ClickHouse can infer the type from the column definition.

Fix implemented:

1. Added TypedEmptyArray struct in db_proto/sql/dialect.go to carry SQL type information for empty arrays
2. Added CreateEmptyArray method to Dialect interface that dialects implement to create appropriate empty array values
3. PostgreSQL implementation (db_proto/sql/postgres/dialect.go):
   - Returns a TypedEmptyArray with the SQL type (e.g., VARCHAR(255)[], INTEGER[])
4. ClickHouse implementation (db_proto/sql/click_house/dialect.go):
   - Returns plain []interface{}{} since ClickHouse infers types from schema
5. Updated ValueToString in db_proto/sql/postgres/types.go to handle TypedEmptyArray by generating array[]::TYPE
6. Updated database.go to use dialect.CreateEmptyArray(fd, column) instead of []interface{}{} for empty repeated fields

Test updates:
- Fixed entityTypesTestWithEmptyRepeatedString to set OptionalStr_2Uint256 (separate pre-existing issue with empty strings in NUMERIC columns)
- Fixed time comparison in PostgreSQL test to use time.Equal() instead of deep equality (timezone location differences)
@maoueh maoueh changed the title Summary Fix from-proto Postgres empty array insertion Feb 23, 2026
@maoueh maoueh requested review from billettc and Copilot February 23, 2026 18:55
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a PostgreSQL-specific issue where inserting rows with empty repeated (array) fields in from-proto mode fails with "cannot determine type of empty array". The fix introduces a dialect-specific approach using a TypedEmptyArray struct that carries SQL type information, allowing PostgreSQL to generate explicit type casts (array[]::TYPE) while ClickHouse continues using plain empty arrays.

Changes:

  • Added TypedEmptyArray struct and CreateEmptyArray method to the Dialect interface for database-specific empty array handling
  • PostgreSQL dialect generates typed empty arrays with explicit casts, while ClickHouse uses plain empty slices
  • Added integration tests for both PostgreSQL and ClickHouse to verify empty array handling, plus a UTC time fix for consistent test comparisons

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
db_proto/sql/dialect.go Defines TypedEmptyArray struct and adds CreateEmptyArray method to the Dialect interface
db_proto/sql/postgres/dialect.go Implements CreateEmptyArray to return typed empty arrays with SQL type information
db_proto/sql/postgres/types.go Adds handling for TypedEmptyArray in ValueToString to generate array[]::TYPE syntax
db_proto/sql/click_house/dialect.go Implements CreateEmptyArray to return plain empty slices (ClickHouse infers types from schema)
db_proto/sql/database.go Updates empty array handling to use dialect.CreateEmptyArray() instead of hardcoded []interface{}{}
db_proto/sql/schema/table.go Adds ColumnByFieldName helper method to look up columns by protobuf field name
tests/integration/db_proto_postgres_test.go New integration test file for PostgreSQL empty array handling
tests/integration/db_proto_clickhouse_test.go Adds test case for ClickHouse empty array handling and shared test helper function
tests/integration/helpers_test.go Adds .UTC() to blockTime for consistent timezone handling in tests
CHANGELOG.md Documents the PostgreSQL empty array fix

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

// PostgreSQL cannot determine the type of an empty array literal (e.g., array[])
// without an explicit type cast, so we return a TypedEmptyArray that carries
// the SQL type information needed for the cast.
func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any {
Copy link

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MapFieldType function is called with a potentially nil column parameter. This could cause a nil pointer dereference at line 58 where it accesses column.Nested and at line 83 where it accesses column.ConvertTo.

While the current test passes because the column is found in typical use cases, the code should defensively handle nil column to prevent panics in edge cases where the column cannot be found (e.g., when tableInfo is nil or when the field is not registered in the table).

Consider adding a nil check at the beginning of CreateEmptyArray or in MapFieldType to handle this case gracefully.

Suggested change
func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any {
func (d *DialectPostgres) CreateEmptyArray(fd protoreflect.FieldDescriptor, column *schema.Column) any {
if column == nil {
// Defensive check: avoid panicking in MapFieldType when column is unexpectedly nil.
// Use a safe default SQL type ("text") in this edge case.
zap.L().Warn("CreateEmptyArray called with nil column, using default SQL type 'text'",
zap.String("field", string(fd.FullName())),
)
return sql2.TypedEmptyArray{SQLType: "text"}
}

Copilot uses AI. Check for mistakes.
@billettc billettc merged commit 268f1d6 into develop Feb 24, 2026
1 check passed
@billettc billettc deleted the fix/from-proto-pg-empty-array branch February 24, 2026 14:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants