Support PostgreSQL INTERVAL, TIMETZ, BYTEA, and BIT types in Live Migration and expand IT coverage

[jsuhani-2026]

Adds Datastream-to-Spanner live migration support for PostgreSQL INTERVAL, TIMETZ, BYTEA, BIT and VARBIT string data types, and expands end-to-end integration test coverage for already supported PostgreSQL data types.

Data Type Support & Behavior Changes:

1. INTERVAL

• Previous Behavior: The values were completely unhandled during JSON extraction, resulting in them being silently migrated to Spanner as null.
• New Behavior: The pipeline now correctly processes the interval Avro record emitted by Datastream (which contains months, hours, and micros fields). It computes the correct period and duration, formatting the result into a standard ISO-8601 duration string (e.g., P1Y2M or PT5H0.123456S), which successfully maps to Spanner STRING columns.

2. TIMETZ

• Previous Behavior: The values were completely unhandled during JSON extraction, resulting in them being silently migrated to Spanner as null.
• New Behavior: The pipeline now correctly processes the timeTz Avro record emitted by Datastream (which contains time in microseconds and a timezone offset). It converts these fields into a standard ISO offset time string (e.g., 23:59:59+10:00 or 08:00:00Z), which successfully maps to Spanner STRING columns.

3. BYTEA, BIT, and VARBIT (Binary Data)

• Previous Behavior: Because Datastream converts Postgres binary columns to Base64 JSON strings, the pipeline previously failed to process them. It globally assumed all binary strings were Hex-encoded (which is only true for MySQL/Oracle), leading to data corruption and parsing errors when attempting to Hex-decode the Base64 values.
• New Behavior: The pipeline now checks the event's _metadata_source_type. If the source is postgresql, it safely uses a Base64 decoder to convert the string back to a Spanner BYTES array, preserving the exact binary data.

Additional Changes

• Expanded IT Coverage: Added integration tests for missing numeric and serial type mappings to Spanner (e.g., smallint_to_float64, serial8_to_numeric, int2_to_numeric).
• Test Maintainability: Replaced opaque, hardcoded Base64 values in ITs with explicit string-to-byte conversions (ByteArray.copyFrom("val").toBase64()), making it much easier to read the intended underlying values.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.19%. Comparing base (31c168c) to head (c98cec6).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #4095      +/-   ##
============================================
+ Coverage     56.13%   56.19%   +0.05%     
- Complexity     6872     7341     +469     
============================================
  Files          1126     1126              
  Lines         68655    68766     +111     
  Branches       7765     7785      +20     
============================================
+ Hits          38542    38645     +103     
- Misses        27621    27626       +5     
- Partials       2492     2495       +3     

Components	Coverage Δ
spanner-templates	87.53% <100.00%> (+0.02%)	⬆️
spanner-import-export	69.00% <ø> (-0.03%)	⬇️
spanner-live-forward-migration	89.31% <100.00%> (+0.02%)	⬆️
spanner-live-reverse-replication	83.50% <100.00%> (+0.02%)	⬆️
spanner-bulk-migration	92.15% <100.00%> (+0.02%)	⬆️
gcs-spanner-dv	88.60% <100.00%> (+0.02%)	⬆️

Files with missing lines	Coverage Δ
...tream/transforms/FormatDatastreamRecordToJson.java	59.58% <100.00%> (+3.65%)	⬆️
...igrations/convertors/ChangeEventTypeConvertor.java	88.40% <100.00%> (+0.34%)	⬆️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the Datastream-to-Spanner live migration tool by introducing support for several complex PostgreSQL data types. By refining binary data serialization and broadening the integration test suite, these changes ensure more robust and accurate data mapping during the migration process.

Highlights

• New Data Type Support: Added support for PostgreSQL INTERVAL, TIMETZ, BIT, and VARBIT data types in the Datastream-to-Spanner migration pipeline.
• Binary Data Handling: Updated BYTEA type processing to use hex encoding, ensuring consistent and reliable data representation.
• Integration Test Expansion: Significantly expanded end-to-end integration test coverage to include a broader set of PostgreSQL data types and edge cases.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for formatting timeTz and interval Datastream record types to JSON, and updates byte array serialization to use hex encoding instead of base64. It also expands integration tests to cover various PostgreSQL-to-Spanner type mappings. Feedback on the changes highlights two key issues: first, a bug in the interval formatting logic that incorrectly handles negative durations and mixed-sign intervals, potentially producing invalid ISO-8601 strings; second, a potential NullPointerException when extracting time and offset fields from the Datastream record without null-safety checks.

[shreyakhajanchi]

Can you please check that the type behaviour is consistent between bulk and live migration templates?