fix(protobuf): reject malformed varints across values and framing#2258
fix(protobuf): reject malformed varints across values and framing#2258samchon wants to merge 5 commits into
Conversation
Claim issue #2256 before implementation so one batch owns value, tag, length-prefix, wrapper, and independent-oracle consequences of the shared reader boundary. Constraint: Campaign pull requests cannot change versions, tags, releases, or publication state. Rejected: Limit the fix to declared integer fields | uint32 also owns tags and every length prefix. Confidence: high Scope-risk: moderate Directive: Enforce the wire invariant without prescribing reader control flow and retain every valid boundary control. Tested: Independent value/tag/length reproductions and official Go oracle critic review passed on ce88a88. Not-tested: Product implementation and integration gates are intentionally pending. Related: #2256 #2135 #2139
All value and framing consumers share the tenth wire byte, so enforce its single legal payload bit in one reader boundary. Permanent runtime, generated wrapper, and official protowire regressions keep malformed input from becoming success-shaped decoded output. Constraint: Campaign implementation freezes package versions, tags, and publication state Rejected: Mask or discard the tenth byte | preserves the silent truncation defect Rejected: Guard generated decoders only | direct readers, tags, lengths, and unknown fields share the runtime owner Confidence: high Scope-risk: moderate Reversibility: clean Directive: Preserve legal ten-byte maxima and non-canonical in-range encodings when changing varint parsing Tested: Focused reader/decode regressions, official Go oracle, mutation proof, full schema workspace, feature identity, pnpm build, pnpm test, tagged native Go tests, native/test go vet Not-tested: Prospective-merge and exact-merge gates; independent verifier owns those campaign stages Related: #2256
The Go oracle test exercised protowire alone, so it stayed green with the product fix fully reverted while its name promised typia's own bounds. Every byte string and verdict now lives in one corpus both languages read: the Go test proves the corpus still matches the official parser, and the TypeScript regressions read the same rows through typia's reader and generated decoders, so neither side can transcribe a verdict the other has not seen. The tenth byte also carried one message for two faults. A continuing tenth byte would occupy an eleventh, but an excess payload bit terminates inside ten bytes and exceeds no length, so it now reports the 64-bit domain it left. Acceptance is unchanged: the same two tenth-byte values pass as before. Constraint: Campaign implementation freezes package versions, tags, and publication state Rejected: Run typia's reader from Go | the one Go-to-node harness stubs every typia/lib/internal import away, and a built lib is no prerequisite of the Go suites Rejected: Keep the corpus in each test | hand transcription is what let the oracle and the regressions drift Rejected: Keep one message for both tenth-byte faults | a ten-byte payload terminates within ten bytes and exceeds no length, so one message could only name one of the two faults truthfully Confidence: high Scope-risk: low Reversibility: clean Directive: Read the shared corpus instead of writing a varint byte string into a new test Tested: Isolated ttsx cache; corpus oracle test with two corpus-drift controls, focused reader and decode regressions, all 18 protobuf cases, 178-case schema workspace, feature identity, go vet over the test module, and a mutation proof with an executed-code probe Not-tested: Prospective-merge and exact-merge gates; the lead owns those campaign stages Related: #2256
886df37 to
cc9fa3a
Compare
The corpus proved the ten-byte boundary but left the widths below it unwitnessed: its accepted rows terminated only at one, two, five, and ten bytes, so the early returns for bytes three, four, six, seven, eight, and nine in `varint32` and `varint64` carried no positive control. The nine-byte row is the one that matters most, because it is the legal encoding of `9223372036854775807` — the exact value the malformed ten-byte witness used to fabricate. Add the canonical maximum for every width from one through ten so an accepted encoding cannot silently stop being accepted. `skipVarint` discards the bytes it reads, which makes it the one consumer that could accept a varint no value reader would. Its over-long half was already pinned through a generated decoder; its tenth-byte-payload half arrives with this fix and had no generated-decoder cell, so read every malformed row as the value of an unknown varint field too.
The corpus and its regressions called `protowire` "the official Protobuf parser". That is not true of the class this fix newly rejects. The specification fixes the ten-byte maximum and says nothing about a tenth byte carrying payload above bit 63; `protowire` rejects such a varint, while the Java runtime's `CodedInputStream.readRawVarint64` accepts it and discards the excess bits. A permanent test asset that overstates its oracle invites someone to disprove it later and conclude the wrong thing, so name the reference Go parser and record the divergence in the corpus itself. typia follows `protowire` deliberately: no encoder emits that form, so the stricter reading costs nothing, and it keeps typia from accepting bytes that `google.golang.org/protobuf` would refuse. `packages/typia/test` imports `google.golang.org/protobuf` directly now, so record it as a direct requirement instead of leaving `go mod tidy` to rewrite the manifest under a later contributor.
Self-Review — round 1Solo Self-Review under Verified, no change needed
Findings to act on
Deliberately not changed
Evidence run so farIndependent pre-flight against the real reader source (byte-identical copy with the enum and
The 45 base failures are exactly the three over-long rows through the seven value readers (21) and the three overflow rows through the seven value readers plus the skip path (24); every accepted row still passes on the base, so the controls are not what moves. The base returned |
|
Superseded by #2266. The maintainer stopped all local building and testing after the machine ran out of resource headroom, and directed that the residual campaign pull requests be consolidated into one with verification delegated to GitHub Actions CI. This unit's net diff is carried into #2266 as its own squashed commit, so its issue keeps a separate acceptance boundary, rollback unit, and failure attribution inside the shared pull request. Closing this one. The branch is left in place; nothing is deleted. |
Closes #2256.
A malformed Protocol Buffer varint could become success-shaped decoded output. All value and framing consumers share the tenth wire byte, so the fix enforces its single legal payload bit in one reader boundary in
_ProtobufReader.ts, and corrects the #2139 reader regression that had frozen ten-continuation acceptance.Verification
Independently verified by an agent that did not write the implementation and built its own oracle rather than reusing the batch's test material.
protowirev1.34.2 — 885 byte strings x 8 reader entry points{0x00, 0x01}, bit-identical to the oracle; base accepted all 256Error— nevernull, neversuccess: falseMutation proof: reverting only
_ProtobufReader.tsfails exactly the three coupled regressions while all fifteen adjacent protobuf controls pass. Gates at the head:pnpm build, 18 protobuf cases, 178-casetest-typia-schema, feature identity,go vetin both trees, and tagged native../native/...(28 packages, 0 failures).The changed #2139 expectation is a correction, not a laundered regression
A previously merged test asserted that typia must accept
0x80 x10, 0x01as0.protowirerejects that input withn = -3. The old expectation was wrong; the new one matches both the wire specification and the oracle. Adjudicated independently against the specification rather than against this pull request's narrative.Second commit: the oracle test could not fail
Independent verification found the first commit's permanent
TestProtobufVarintBoundsOraclepassed with the product fix fully reverted — it exercisedprotowirealone, never called typia, and its agreement with the TypeScript regressions was hand transcription. A permanent test that cannot fail when the work regresses is a false green, so the batch was returned for a test-asset remediation.Every byte string and verdict now lives in one
packages/typia/test/protobuf_varint_corpus.jsonthat the Go test and both TypeScript regressions read. The Go test is renamed to state that it pins the official parser's verdicts only. Verified by mutating the corpus seven ways — value, consumed length, fault relabelling, emptied entries, unknown class, a fabricated contradictory row, and an accepted/rejected flip — with both halves reacting to every mutation.The same commit splits one error message that had described two different faults: a continuing tenth byte keeps
varint exceeds 10 bytes., while a terminating tenth byte carrying payload above bit 0 now reportsvarint exceeds 64 bits.Acceptance is byte-identical — proven by re-running the full 7080-cell differential across both heads: 0 acceptance differences, 0 value differences, 2088 message-only differences. No website page, documentation, example, or tracked test outside this batch pinned the old string.Out of scope, recorded for the next discovery round
Two verified, user-visible defects present identically on the base commit and on this fix, outside this issue's published invariant: length prefixes and field tags are silently truncated to 32 bits, so a field declaring 4,294,967,301 bytes reads five, and field 536870913 aliases onto field 1. A third consumer of the same truncation,
bool(), decodes from the low 32 bits only. None is locked in by the new tests, so the next round can decide the contract rather than inherit a snapshot.Campaign constraints
No package version, tag, release, publication, or
packages/interface/srcchange.pnpm formatwas not run.