trustworthy content-stream tokenization and --ops output#4
Merged
Conversation
- F1: tokenize leading-sign numeric operands (+5, +.5) as numbers not
operators (isNumberStart + number-consumption loop), fixing --json/--ops
mislabeling of ISO 32000-1 7.3.3 signed operands.
- F2: emitOps skips empty-operator FormattedLines so --ops NDJSON no longer
emits phantom {"op":""} records for comment/dangling lines.
- F3: strip the CRLF EOL delimiter (bounded to one unit) before inline-image
EI, removing the stray \r without eating whitespace-valued payload bytes.
- F4: document the short-array cap-truncation marker gap (comment only).
- Tests: 14.1-UNIT-001/-002/-002b + tests/14-1 acceptance module
(INTG-000/-001/-002/-003) with two hand-authored fixtures.
unidoc-anom
changed the title
There was a problem hiding this comment.
Pull request overview
This PR hardens the machine-parsable contract of pdfdebug dump stream --json and --ops by fixing content-stream tokenization for leading + numeric operands and preventing --ops from emitting phantom NDJSON records, with additional inline-image payload delimiter handling and accompanying regression tests/fixtures.
Changes:
- Recognize leading
+as part of PDF number tokens (symmetric with-) during content-stream tokenization. - Enforce the
--opsNDJSON contract by skipping formatted lines withOperator == ""to avoid emitting{"op":""...}records. - Strip the
\rfrom a\r\ndelimiter immediately before inline-imageEIwithout stripping arbitrary whitespace that may be payload data; add unit + integration/acceptance coverage.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
internal/pdfcore/stream.go |
Fixes number tokenization for +-signed operands; tightens inline-image EI delimiter handling to drop CRLF cleanly. |
cmd/cli/cmd_stream.go |
Skips empty-operator formatted lines so --ops emits one NDJSON object per real operator only. |
internal/pdfcore/stream_test.go |
Adds unit tests covering leading-+ numbers and inline-image CRLF delimiter stripping / payload-byte preservation. |
tests/14-1-trustworthy-stream-op-output/helpers_test.go |
Adds black-box CLI build/run harness for acceptance tests over dump stream --json/--ops. |
tests/14-1-trustworthy-stream-op-output/stream_contract_test.go |
Adds integration contract tests asserting correct --json token types and --ops NDJSON operator records. |
tests/14-1-trustworthy-stream-op-output/go.mod |
Introduces an isolated Go module for the acceptance suite. |
testdata/correctness/README.md |
Documents new correctness fixtures and the pre-/post-fix behaviors they exercise. |
internal/pdfcore/objectsource.go |
Adds an explanatory comment documenting a near-unreachable truncation-marker gap (no behavior change). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hardens the
pdfdebug dump stream --json/--opsmachine contract so scripts and AI agents can parse it without silent mislabeling or phantom records.Changes
tokenizeContentStreamnow recognizes a leading+(symmetric to-) per ISO 32000-1 7.3.3, at both edit sites (isNumberStart+ the number-consumption loop). Spec-valid operands like+5/+.5classify asnumbertokens instead of being mislabeled as operators and mis-grouped downstream.emitOpsskipsFormattedLines with an emptyOperator, so--opsNDJSON no longer emits phantom{"op":"","params":[...]}records for comment/dangling-operand lines (one-object-per-operator contract).EIscan strips the CRLF EOL delimiter (bounded to a single unit) so\r\nEIleaves no stray\r, without eating whitespace-valued bytes that are legitimately part of the opaque payload.objectsource.go(near-unreachable; no behavior change).Scope
Backend tokenizer + CLI presenter only. No IPC type / binding / GUI change.
internal/pdfcorekeeps zero Wails deps.Tests
stream_test.go):14.1-UNIT-001(leading-sign),14.1-UNIT-002/-002b(F3 CRLF strip + whitespace-valued-byte preservation).tests/14-1-trustworthy-stream-op-output/, own go.mod):14.1-INTG-000/-001/-002/-003driving the built CLI over--json/--opsagainst two hand-authored fixtures.Quality gates (green)
go vet ./...,golangci-lint run(0 issues),go test ./...,go test -race ./..., alltests/*/suites. Frontend untouched by the diff.Review
Three adversarial review passes (Blind Hunter / Edge Case Hunter / Acceptance Auditor). Notable: review #1 caught and fixed an unbounded F3 whitespace strip that could truncate opaque inline-image payload bytes; the residual bare-LF/0x0D-CR ambiguity is documented as an accepted, irreducible tradeoff. Traceability gate PASS, 100% AC coverage.
🤖 Generated with Claude Code