Skip to content

fix(protobuf): reject malformed varints across values and framing#2258

Closed
samchon wants to merge 5 commits into
masterfrom
fix/protobuf-varint-bounds
Closed

fix(protobuf): reject malformed varints across values and framing#2258
samchon wants to merge 5 commits into
masterfrom
fix/protobuf-varint-bounds

Conversation

@samchon

@samchon samchon commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Closes #2256.

A malformed Protocol Buffer varint could become success-shaped decoded output. All value and framing consumers share the tenth wire byte, so the fix enforces its single legal payload bit in one reader boundary in _ProtobufReader.ts, and corrects the #2139 reader regression that had frozen ten-continuation acceptance.

Verification

Independently verified by an agent that did not write the implementation and built its own oracle rather than reusing the batch's test material.

Measure Result
Differential vs official protowire v1.34.2 — 885 byte strings x 8 reader entry points 7080/7080 cells agree, 0 disagreements
Same differential on the base commit 3971 disagreeing cells over 530 inputs, every one accepts-illegal
Regressions / loosened cells introduced 0 / 0
Oracle-accepted inputs decoding to the exact value at the exact consumed length 208 / 208
Exhaustive tenth-byte partition, both prefix families accepts exactly {0x00, 0x01}, bit-identical to the oracle; base accepted all 256
Cross-reader consistency among the 8 consumers 0 disagreements, against 269 on base
Public wrapper matrix (direct/factory x raw/assert/is/validate) 152/152 threw a prefixed Error — never null, never success: false

Mutation proof: reverting only _ProtobufReader.ts fails exactly the three coupled regressions while all fifteen adjacent protobuf controls pass. Gates at the head: pnpm build, 18 protobuf cases, 178-case test-typia-schema, feature identity, go vet in both trees, and tagged native ../native/... (28 packages, 0 failures).

The changed #2139 expectation is a correction, not a laundered regression

A previously merged test asserted that typia must accept 0x80 x10, 0x01 as 0. protowire rejects that input with n = -3. The old expectation was wrong; the new one matches both the wire specification and the oracle. Adjudicated independently against the specification rather than against this pull request's narrative.

Second commit: the oracle test could not fail

Independent verification found the first commit's permanent TestProtobufVarintBoundsOracle passed with the product fix fully reverted — it exercised protowire alone, never called typia, and its agreement with the TypeScript regressions was hand transcription. A permanent test that cannot fail when the work regresses is a false green, so the batch was returned for a test-asset remediation.

Every byte string and verdict now lives in one packages/typia/test/protobuf_varint_corpus.json that the Go test and both TypeScript regressions read. The Go test is renamed to state that it pins the official parser's verdicts only. Verified by mutating the corpus seven ways — value, consumed length, fault relabelling, emptied entries, unknown class, a fabricated contradictory row, and an accepted/rejected flip — with both halves reacting to every mutation.

The same commit splits one error message that had described two different faults: a continuing tenth byte keeps varint exceeds 10 bytes., while a terminating tenth byte carrying payload above bit 0 now reports varint exceeds 64 bits. Acceptance is byte-identical — proven by re-running the full 7080-cell differential across both heads: 0 acceptance differences, 0 value differences, 2088 message-only differences. No website page, documentation, example, or tracked test outside this batch pinned the old string.

Out of scope, recorded for the next discovery round

Two verified, user-visible defects present identically on the base commit and on this fix, outside this issue's published invariant: length prefixes and field tags are silently truncated to 32 bits, so a field declaring 4,294,967,301 bytes reads five, and field 536870913 aliases onto field 1. A third consumer of the same truncation, bool(), decodes from the low 32 bits only. None is locked in by the new tests, so the next round can decide the contract rather than inherit a snapshot.

Campaign constraints

No package version, tag, release, publication, or packages/interface/src change. pnpm format was not run.

samchon added 3 commits July 21, 2026 00:22
Claim issue #2256 before implementation so one batch owns value, tag, length-prefix, wrapper, and independent-oracle consequences of the shared reader boundary.

Constraint: Campaign pull requests cannot change versions, tags, releases, or publication state.

Rejected: Limit the fix to declared integer fields | uint32 also owns tags and every length prefix.

Confidence: high

Scope-risk: moderate

Directive: Enforce the wire invariant without prescribing reader control flow and retain every valid boundary control.

Tested: Independent value/tag/length reproductions and official Go oracle critic review passed on ce88a88.

Not-tested: Product implementation and integration gates are intentionally pending.

Related: #2256 #2135 #2139
All value and framing consumers share the tenth wire byte, so enforce its
single legal payload bit in one reader boundary. Permanent runtime, generated
wrapper, and official protowire regressions keep malformed input from becoming
success-shaped decoded output.

Constraint: Campaign implementation freezes package versions, tags, and publication state
Rejected: Mask or discard the tenth byte | preserves the silent truncation defect
Rejected: Guard generated decoders only | direct readers, tags, lengths, and unknown fields share the runtime owner
Confidence: high
Scope-risk: moderate
Reversibility: clean
Directive: Preserve legal ten-byte maxima and non-canonical in-range encodings when changing varint parsing
Tested: Focused reader/decode regressions, official Go oracle, mutation proof, full schema workspace, feature identity, pnpm build, pnpm test, tagged native Go tests, native/test go vet
Not-tested: Prospective-merge and exact-merge gates; independent verifier owns those campaign stages
Related: #2256
The Go oracle test exercised protowire alone, so it stayed green with the
product fix fully reverted while its name promised typia's own bounds. Every
byte string and verdict now lives in one corpus both languages read: the Go
test proves the corpus still matches the official parser, and the TypeScript
regressions read the same rows through typia's reader and generated decoders,
so neither side can transcribe a verdict the other has not seen.

The tenth byte also carried one message for two faults. A continuing tenth
byte would occupy an eleventh, but an excess payload bit terminates inside ten
bytes and exceeds no length, so it now reports the 64-bit domain it left.
Acceptance is unchanged: the same two tenth-byte values pass as before.

Constraint: Campaign implementation freezes package versions, tags, and publication state
Rejected: Run typia's reader from Go | the one Go-to-node harness stubs every typia/lib/internal import away, and a built lib is no prerequisite of the Go suites
Rejected: Keep the corpus in each test | hand transcription is what let the oracle and the regressions drift
Rejected: Keep one message for both tenth-byte faults | a ten-byte payload terminates within ten bytes and exceeds no length, so one message could only name one of the two faults truthfully
Confidence: high
Scope-risk: low
Reversibility: clean
Directive: Read the shared corpus instead of writing a varint byte string into a new test
Tested: Isolated ttsx cache; corpus oracle test with two corpus-drift controls, focused reader and decode regressions, all 18 protobuf cases, 178-case schema workspace, feature identity, go vet over the test module, and a mutation proof with an executed-code probe
Not-tested: Prospective-merge and exact-merge gates; the lead owns those campaign stages
Related: #2256
@samchon
samchon force-pushed the fix/protobuf-varint-bounds branch from 886df37 to cc9fa3a Compare July 20, 2026 15:48
samchon added 2 commits July 21, 2026 01:09
The corpus proved the ten-byte boundary but left the widths below it unwitnessed: its accepted rows terminated only at one, two, five, and ten bytes, so the early returns for bytes three, four, six, seven, eight, and nine in `varint32` and `varint64` carried no positive control. The nine-byte row is the one that matters most, because it is the legal encoding of `9223372036854775807` — the exact value the malformed ten-byte witness used to fabricate. Add the canonical maximum for every width from one through ten so an accepted encoding cannot silently stop being accepted.

`skipVarint` discards the bytes it reads, which makes it the one consumer that could accept a varint no value reader would. Its over-long half was already pinned through a generated decoder; its tenth-byte-payload half arrives with this fix and had no generated-decoder cell, so read every malformed row as the value of an unknown varint field too.
The corpus and its regressions called `protowire` "the official Protobuf parser". That is not true of the class this fix newly rejects. The specification fixes the ten-byte maximum and says nothing about a tenth byte carrying payload above bit 63; `protowire` rejects such a varint, while the Java runtime's `CodedInputStream.readRawVarint64` accepts it and discards the excess bits. A permanent test asset that overstates its oracle invites someone to disprove it later and conclude the wrong thing, so name the reference Go parser and record the divergence in the corpus itself.

typia follows `protowire` deliberately: no encoder emits that form, so the stricter reading costs nothing, and it keeps typia from accepting bytes that `google.golang.org/protobuf` would refuse.

`packages/typia/test` imports `google.golang.org/protobuf` directly now, so record it as a direct requirement instead of leaving `go mod tidy` to rewrite the manifest under a later contributor.
@samchon

samchon commented Jul 20, 2026

Copy link
Copy Markdown
Owner Author

Self-Review — round 1

Solo Self-Review under .agents/skills/review/SKILL.md. Snapshot 2d3b4181c28038a4b8429c86a6818c182d8dfbba, complete surface 3f59f641a9..2d3b4181c2 (eight files). Findings and remediation plan recorded here before any of them is applied.

Verified, no change needed

  • The product fix matches its oracle byte for byte. Read the pinned google.golang.org/protobuf@v1.34.2 encoding/protowire/wire.go source directly: ConsumeVarint requires the tenth byte to be < 2 and returns errCodeTruncated = -1 / errCodeOverflow = -3. lastVarintByte accepts exactly 0x00 and 0x01 and rejects everything else, splitting the verdict into a length fault and a domain fault. Same accept/reject partition, finer error naming.
  • Both fault classes reach every consumer. varint32 reads five bytes, then four, then lastVarintByte; varint64 reads nine, then lastVarintByte; skipVarint loops bytes one through nine, then lastVarintByte. Ten bytes maximum on all three paths, and the skip path now rejects the tenth-byte-payload class it previously accepted.
  • The corpus is genuinely consumed, and a corrupted row fails a real assertion. Recomputed every accepted row's value independently from its own bytes (seven-bit little-endian groups, asUintN(64)); all 15 match the recorded value, every accepted row carries continuation bits on all but its last byte, and no accepted row exceeds ten bytes or carries more than bit 63 in its tenth. The Go side re-derives the same verdicts from protowire plus an internal-consistency gate; the TypeScript side reads the same rows through typia.assert<IDocument> and fails loudly on a lost field or an unknown fault class. Neither side transcribes a byte string.
  • Repository format pipeline. scripts/format-go.cjs runs gofmt -w and then converts leading tabs to two spaces; the new Go test is byte-identical to that pipeline's output (the only diff is CRLF vs LF from the Windows checkout), so it matches every other Go file in the repository. Prettier reports the same style drift on untouched neighbours as on the changed files, so it is a repository-wide state that Post-Campaign Cleanup owns, not a defect of this branch.
  • packages/typia/test is not in the published files list, so the corpus does not ship to npm consumers.
  • Correcting test_protobuf_reader_unknown_field_faults. The #2139 assertion that uint32 may consume ten continuation bytes and return zero froze the defect; fix(protobuf): reject malformed varints across values and framing #2256 explicitly requires correcting it. It is replaced by a rejection plus a positive control that a legal ten-byte 0xff×9 0x01 still reads as 0xffffffff at index 10. Flagged as a deliberate specification change under the development skill's Change Integrity rule.

Findings to act on

  1. An inaccurate claim I introduced one commit ago. test_protobuf_reader_varint_bounds now says the corpus carries "the canonical maximum that terminates at each width from one byte through ten". That is false for width two: the only two-byte rows are ac02 (300) and 8000 (non-canonical zero), neither of which is the two-byte maximum. Plan: add the ff7f / 16383 row rather than soften the sentence, so the family is uniform and the claim is exactly true.
  2. skipVarint's doc comment no longer describes what skipVarint does. It still explains only the ten-byte length limit, but the method now also rejects a tenth byte carrying payload above bit 63. Plan: extend the comment to name both faults, matching lastVarintByte.

Deliberately not changed

  • website/src/content/docs/protobuf/decode.mdx. Its "Wire errors" section already states that a malformed payload raises an Error before validation and propagates out of isDecode and validateDecode. This change makes the runtime match that documented contract rather than altering it, and fix(protobuf): reject malformed varints across values and framing #2256 names the page as the contract to preserve. Adding a varint sentence would be an enhancement that also drags in the website skill's full fresh-build validation lane, which is outside this unit's gate set.
  • Renaming the local OVERFLOW constant in the protobuf tests, even though the corpus now uses "overflow" for the 64-bit-domain fault while that constant means buffer overflow.. Seven files across the suite already use that name for the reader's own message text; a one-file rename would break an established convention rather than fix an inconsistency.
  • Recording the runtime divergence in _ProtobufReader.ts itself. The evidence belongs where a test pins it; a claim about another implementation's behaviour rots in shipped source.

Evidence run so far

Independent pre-flight against the real reader source (byte-identical copy with the enum and Singleton stubbed, run under node --experimental-strip-types), crossing all 32 corpus rows with uint32, int32, sint32, uint64, int64, sint64, bool, and skipVarint:

tree assertions passed assertions failed
head 2d3b4181c2 256 0
_ProtobufReader.ts reverted to 3f59f641a9 211 45

The 45 base failures are exactly the three over-long rows through the seven value readers (21) and the three overflow rows through the seven value readers plus the skip path (24); every accepted row still passes on the base, so the controls are not what moves. The base returned 9223372036854775807 from uint64 for ff×9 02 — the issue's own witness, reproduced from scratch on current master. This is a prediction that de-risks the long suite run, not a substitute for it; pnpm --filter ./tests/test-typia-schema start, pnpm test:go:public, pnpm test:go:native, pnpm --filter ./packages/typia build, and the disposable-worktree mutation proof are still required and are running or queued.

@samchon

samchon commented Jul 20, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #2266.

The maintainer stopped all local building and testing after the machine ran out of resource headroom, and directed that the residual campaign pull requests be consolidated into one with verification delegated to GitHub Actions CI. This unit's net diff is carried into #2266 as its own squashed commit, so its issue keeps a separate acceptance boundary, rollback unit, and failure attribution inside the shared pull request.

Closing this one. The branch is left in place; nothing is deleted.

@samchon samchon closed this Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(protobuf): reject malformed varints across values and framing

1 participant