feat: add signed ↔ unsigned integer coercions - #3
Merged
Conversation
Sign-crossover pairs (e.g. Arrow Int64 → proto uint64) were classified as Incompatible, so no coercion flag could bind them. This matters for BigQuery, whose only integer type surfaces as Arrow Int64 regardless of the proto field's signedness — any uint64/uint32 proto field was unmappable. Reclassify the full signed ↔ unsigned matrix as CoercionAvailable. Values are reinterpreted as two's complement (a C-style cast, matching what a protobuf decoder does when a field changes between int and uint kinds); no runtime range checks: - Int32/Int64 → uint32/fixed32/uint64/fixed64: Semantic (negatives become large unsigned values, widening sign-extends), except Int64 → 32-bit targets which are Truncation (low 32 bits kept) - UInt32 → int32/sint32/sfixed32: Semantic (above i32::MAX wraps negative) - UInt32 → int64/sint64/sfixed64: Lossless (every u32 fits in i64) - UInt64 → int64/sint64/sfixed64: Semantic; → 32-bit targets: Truncation Repeated/map/oneof contexts get the new pairs for free via the shared scalar encoder dispatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds opt-in signed ↔ unsigned integer coercions to apb-core so Arrow integer columns (notably BigQuery’s Int64 output) can map to protobuf unsigned fields by reinterpreting values as two’s complement, with new scalar encoders and unit tests covering the full matrix.
Changes:
- Reclassify signed/unsigned integer crossover pairs as
CoercionAvailablewith appropriate risk levels (semantic/lossless/truncation). - Add scalar encoder variants + plan selection for crossover pairs, implementing two’s-complement reinterpretation without runtime range checks.
- Extend unit tests with round-trip assertions for wrap-around, sign-extension, and truncation behaviors; document the feature in README.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents the new signed ↔ unsigned crossover coercion behavior and motivation (BigQuery). |
| crates/apb-core/src/types/compatibility.rs | Marks signed/unsigned crossover pairs as CoercionAvailable and adds matrix-coverage tests. |
| crates/apb-core/src/transcode/tests.rs | Adds round-trip transcoding tests validating reinterpretation/truncation outcomes. |
| crates/apb-core/src/transcode/plan.rs | Extends scalar encoder selection to route crossover pairs to new encoders under Coerce. |
| crates/apb-core/src/transcode/encode.rs | Introduces new ScalarKind variants and encoding functions implementing the crossover rules. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Rename Int32AsUFixed32/Int32AsUFixed64/Int64AsUFixed32/Int64AsUFixed64 (and their encoder functions) to ...AsFixed32/...AsFixed64, matching the existing unsigned-fixed variants (UInt64AsFixed32, UInt32AsFixed64) so a single naming scheme covers the fixed32/fixed64 proto kinds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
BigQuery has a single integer type, which the Storage Read API surfaces as Arrow
Int64regardless of the destination proto field's signedness. Any proto message with auint64/uint32field is therefore unmappable today: sign-crossover pairs are classifiedIncompatible, which no coercion flag (per-field[(apb.apb).coerce = true]orcoerce_all) can rescue.What
Reclassifies the full signed ↔ unsigned matrix as
CoercionAvailableand adds the corresponding scalar encoders. Values are reinterpreted as two's complement — a C-style cast, matching what a protobuf decoder does when a field's declared type changes between int and uint kinds. There are no runtime range checks; encoding never fails on these pairs.-1→u32::MAX)-1→u64::MAX)-1→u64::MAX)u32::MAX→-1u64::MAX→-1Wire encodings follow each target's native rules (varint with 64-bit sign extension for negative int32/int64 targets, zigzag for sint, fixed for (s)fixed). Repeated/map/oneof contexts pick the new pairs up for free via the shared scalar encoder dispatch.
Notes
coerce_all/--coerce); default inference still rejects these pairs.cargo clippy --all-targetsfailures intranscode/tests.rs(approx-PI constants) pre-exist on main and are untouched here.Testing
decode_message, 5 compatibility classification tests covering the full matrix.cargo test -p apb-core: 190 passed.🤖 Generated with Claude Code