feat: transcode BigQuery-shaped Arrow columns (string ints, maps from lists, packed bytes)

[Baoyuan-Xing]

What

Three gaps that together make it impossible to transcode a realistic message from BigQuery without first rewriting the data in SQL — or, in two cases, at all.

	today	with this branch
STRING-encoded 64-bit ints	CAST every column, at every nesting level	Utf8 -> integer coercion
map<K,V> field	impossible	accept List<Struct<key,value>>
payload in a bytes field	impossible	pack a struct column into it

Each is a resolution-rule change. No new encoding machinery: every one of them reuses an encoder apb already has.

Why these three

String-encoded integers. proto3's canonical JSON encoding renders 64-bit integers as strings, so any system that decodes proto into a structured view surfaces them that way — at every level of nesting, not just on flat top-level columns.

Maps. Engines with no MAP type can only express map<K,V> as ARRAY<STRUCT<key,value>>, which arrives as List<Struct<..>>. apb required DataType::Map, so proto map fields were simply unreachable from those sources.

Packed bytes. An envelope that carries a serialized payload in a bytes field is a common storage shape — a timestamp, a tombstone flag, and the value as opaque bytes. The envelope proto deliberately does not reference the payload type, so there was no way to express "serialize this struct column as message M and put the result here".

The first has a workaround (CAST), but for nested and repeated fields it means rebuilding arrays element by element with UNNEST ... WITH OFFSET ... ORDER BY; on a real workload that reshaping cost far more than the encode it existed to serve. The other two have no workaround short of declaring a mirror message per payload type.

How

Utf8 -> integer for all ten integer kinds and both string widths, as ScalarKind::Utf8AsInt(IntTarget) rather than 20 flat variants. Semantics follow the existing coercions:

• unsigned targets parse u64, falling back to i64 reinterpreted as two's complement, matching the signed↔unsigned crossover already present for producers with only a signed integer type;
• 32-bit narrowing is range-checked and fails the batch, matching Int64 -> int32 rather than truncating.

Parsing is strict — no whitespace trimming, no empty-string-as-zero. Both would quietly turn malformed input into a plausible value; a caller who wants "absent" should map it to NULL in the source.

Maps from lists. resolve_map and encode_map accept List/LargeList of two-field entry structs alongside Map. Sound rather than convenient: proto3 encodes map<K,V> as repeated MapEntry{key=1,value=2}, so both Arrow shapes describe one wire format. Null entries are skipped, since a proto map key cannot be absent.

pack. apb already performs exactly this double serialization for google.protobuf.Any via any_pack, but it was fenced off everywhere else:

return Err(MappingError::AnyPackOnNonAnyField { .. });

Generalized to InferOptions::pack / (apb).pack / --pack FIELD=MESSAGE, keyed by fully qualified field name. The caller-side option is the important one: envelope protos usually cannot be modified.

No new encoder, and that is the point — a length-delimited embedded message and a bytes field are identical on the wire (tag | length | payload), so the existing message encoder already emits the right bytes. Only the payload's shape comes from a different place.

Two claims proved by construction

The wire-equivalence arguments above are load-bearing, so both are tests rather than comments:

• list_backed_map_is_wire_identical_to_map_array — the same data as a MapArray and as a list of entry structs produce byte-identical output.
• packed_bytes_is_byte_identical_to_separate_serialization — packing equals serializing the payload separately and assigning the result, byte for byte.

Testing

16 new tests. cargo test --workspace is green at each commit individually, so bisect stays clean.

• string integers: round-trip per wire encoding, coercion required, negative → unsigned reinterpretation, unparseable input fails the batch, narrowing range check, NULL stays unset, plus a nested_string_ints fixture covering repeated-inside-repeated rather than only flat columns
• maps: bind from list, byte-identity, null entry skipped, list-of-scalars rejected
• pack: round-trip through a real envelope, byte-identity, non-bytes field rejected (pointing at any_pack), unknown target rejected, and a bytes field without a declaration still refuses a Struct — no implicit packing

Validation beyond the suite

Ran against a real BigQuery table: an envelope message whose bytes field is packed with a domain containing a map<string, Message> of repeated sub-messages several levels deep, with 64-bit integers arriving as STRING throughout. Output parsed straight off the wire — envelope fields intact, and the packed bytes decoding standalone under the unmodified domain proto. Every leaf cross-checked against the source row, including uint64/int64 from STRING columns and a google.protobuf.Timestamp matching to the millisecond.

Scope and compatibility

• Default behaviour is unchanged: no new coercion applies without --coerce or an annotation, Map is still accepted exactly as before, and a bytes field without a pack declaration behaves as it always did.
• No public API removed. Additions: IntTarget, two ScalarKind variants, InferOptions::pack, (apb).pack (field 4 on ApbFieldOptions), and two MappingError variants.
• Column naming is deliberately out of scope. Matching stays strict and case-sensitive; callers supply snake_case columns. An earlier revision added optional snake_case normalization for lowerCamelCase producers and it has been dropped — it papers over a producer-side quirk that is better fixed where the names are generated.
• Not covered: repeated and map-valued bytes fields. any_pack supports those positions; pack currently falls through to the normal path. Easy to add if you want the symmetry — say the word.

Draft: opening for direction before polish. Happy to split into three PRs if you would rather take them separately.