[Bug]: pdf.js doesn't lex `5f` properly

[jrmuizel]

Attach (recommended) or Link to PDF file

lex-diff.pdf

Web browser and its version

Firefox Nightly

Operating system and its version

macOS

PDF.js version

recent

Is the bug present in the latest PDF.js version?

Yes

Is a browser extension

No

Steps to reproduce the problem

1. Load lex-diff.pdf
2. Observe a large red rectangle

What is the expected behavior?

The PDF should not have a red rectangle. Acrobat, Preview and PDFium do not have a red rectangle.

What went wrong?

In a PDF stream like:

q
1 0 0 rg
100 450 400 150 re
5f
Q

pdf.js will lex 5f as Number(5), Keyword(f).

From PDF spec 7.2.3:

"All characters except the white-space characters and delimiters are referred to as regular characters. These characters include bytes that are outside the ASCII character set. A sequence of consecutive regular characters comprises a single token."

Link to a viewer

No response

Additional context

No response

[calixteman]

There's the same bug with Foxit.

[WenYuan77]

Hi, can I contribute?

[calixteman]

I'm not sure we want to fix this bug.
I agree with @LaurenzV here (LaurenzV/hayro#994 (comment)): it's a broken content stream so we can do whatever we want, right ?
5f can be an extra wrong token so it's fine to skip it but it can be just a matter of a missing white space so interpreting it as 5 f is fine too I guess, I don't think there's a right way to handle such a case.
All the pdf viewers (Acrobat included) are able to render a large variety of broken pdfs.