Add option to skip PDF byte normalization

claude.md

@@ -35,28 +35,37 @@ Entry point is `VerifyPDFium.Initialize(dpi = 96)` which registers a stream conv
 3. One `png` target per page, named `page_0001` style
 
 To keep snapshots stable for PDFs freshly generated at test time, the non-deterministic fields are neutralized two ways:
-- **In the `pdf` bytes** (`PdfNormalizer.Normalize`): the trailer `/ID`, the info-dictionary `/CreationDate`/`/ModDate`, and the XMP metadata dates plus `xmpMM:DocumentID`/`InstanceID` are overwritten by a length-preserving, in-place byte scan (no string round-trip, no regex) — only the volatile characters change, so cross-reference offsets stay valid. `Normalize` mutates in place and runs *after* the `PdfiumDocument` (which reads lazily from the same buffer) is disposed, so no defensive copy is needed. Values inside a compressed object/metadata stream (`/ObjStm`, flate-compressed XMP) are not reachable by the plaintext scan.
-- **In the info file** (`PdfNormalizer.NormalizeProperties`): `Properties` is a `Dictionary<string, object>` whose `CreationDate`/`ModDate` values are parsed (`PdfDate`) into `DateTimeOffset`, so Verify's built-in date scrubbing renders them deterministically (`DateTimeOffset_1` etc.). Properties are read from the original bytes, before `Normalize` zeroes them.
+- **In the `pdf` bytes** (`DeterministicPdf.PdfNormalizer.Normalize`): the trailer `/ID`, the info-dictionary `/CreationDate`/`/ModDate`, and the XMP dates plus `xmpMM:DocumentID`/`InstanceID`/`OriginalDocumentID` and `dc:date` are overwritten by an in-place byte scan (no string round-trip, no regex). That value-zeroing is length-preserving, so cross-reference offsets survive it. A final `CanonicalizeXmp` pass then collapses the XMP packet's whitespace — Apache FOP serializes the packet through the platform's XML writer, so indentation varies by JRE — and that pass is **not** length-preserving: it rebuilds the buffer and repairs both the metadata stream length and the classic cross-reference table. A document it cannot safely rewrite (cross-reference stream, incremental update, unlocatable stream length) comes back unchanged. `Normalize` copies its input, so a caller keeps ownership of the array passed in; the call is still made only after the `PdfiumDocument` (which reads lazily from the same buffer) is disposed. Values compressed away inside an `/ObjStm` or a flate-compressed XMP packet are not reachable by the plaintext scan and are left alone.
+- **In the info file** (`PdfProperties.Normalize`): `Properties` is a `Dictionary<string, object>` whose `CreationDate`/`ModDate` values are parsed (`PdfDate`) into `DateTimeOffset`, so Verify's built-in date scrubbing renders them deterministically (`DateTimeOffset_1` etc.). Properties are read from the original bytes, before the byte-level pass zeroes them.
+
+Both can be opted out of per verification, via a context key set by a `SettingsTask` extension and read in `Convert`:
+- `ExcludePdfDocument` — drops the `.verified.pdf` entirely, for producers whose bytes can never be made deterministic (Aspose.Cells embeds the machine's system fonts).
+- `SkipPdfNormalization` — keeps the `.verified.pdf` but snapshots the producer's own bytes, for producers that are already byte-deterministic. Worth knowing when changing this: `CanonicalizeXmp` is the pass that alters bytes even for such a producer, so toggling the setting on an existing suite shifts every stored `.verified.pdf` once.
 
 Key files:
-- **VerifyPDFium.cs** — initialization and the converter
-- **PdfNormalizer.cs** — neutralizes the `pdf` bytes and projects the info-file properties to the scrubbable object map
+- **VerifyPDFium.cs** — initialization, the converter, and the two opt-out extensions
+- **PdfProperties.cs** — projects the pdfium-reported property map to the scrubbable object map
 - **PdfDate.cs** — parses PDF date strings (`D:YYYYMMDD…`) to `DateTimeOffset`
 - **PdfInfo.cs** / **PageInfo.cs** — info shape for the snapshot (per-page width/height in points and text via `PdfPage.GetText()`)
 
+The byte-level normalizer is **not** a type in this repo — it is `PdfNormalizer` from the [DeterministicPdf](https://github.com/SimonCropp/DeterministicPdf) package, which owns and tests the algorithm.
+
 Style note: only public types get a namespace declaration (`VerifyTests`); internal types live in the global namespace.
 
 ### Tests (`src/Tests/`)
 
 - **ModuleInitializer.cs** — calls `VerifyPDFium.Initialize()` via `[ModuleInitializer]` (also the readme `enable` snippet); PNGs compare via SSIM (`VerifierSettings.UseSsimForPng()`)
 - **Samples.cs** — snapshot tests, also the readme usage snippets
+- **PdfNormalizerTests.cs** — asserts the wiring rather than the algorithm: that a normalized document still loads in pdfium, that volatile values are neutralized, and that the pass is idempotent
+- **SkipPdfNormalizationTests.cs** — asserts the opposite pairing, that the skipped snapshot holds the producer's own bytes while the default one holds the neutralized bytes. `SampleIsNotAlreadyNormalized` pins the premise, so the pair cannot both pass vacuously if `sample.pdf` ever becomes normalization-invariant
 - Test assets `sample.pdf` (1 page) and `multi-page.pdf` (4 pages), both Letter size, were produced by [Morph](https://github.com/Papyrine/Morph)'s PDF exporter with embedded font subsets, so rendering is machine-independent.
 
 Engine-level unit tests (load/render/page sizes/metadata/threading) live in the Morph.PDFium repo, not here.
 
 ## Dependency notes
 
 - Morph.PDFium pins the PDFium native version; bumping it may shift rendered pixels — expect to regenerate `*.verified.png` files in the same commit.
+- DeterministicPdf owns the byte-level normalization, so bumping it may shift `.verified.pdf` files across every consuming suite without any document having changed. The 1.3.0 release did exactly that by adding `CanonicalizeXmp`: downstream snapshots moved by the size of the collapsed XMP whitespace while the rendered pages and info files stayed identical. When a bump changes the `.verified.pdf` but leaves the pages and info alone, that is the expected shape of it — not a rendering regression.
 - Morph.PDFium is a Papyrine package: `Papyrine_SponsorshipLicenseIgnored` is set in `src/Directory.Build.props` and `SC023` is in `NoWarn` (same arrangement as Verify.OpenXml's Morph dependency).
 
 ## Build Configuration

readme.md

@@ -12,7 +12,7 @@ Verifying a `pdf` produces:
  * The pdf itself as `.verified.pdf`. This can be omitted with [`ExcludePdfDocument`](#exclude-the-pdf-document).
  * A PNG render of every page as `#page_0001.verified.png`, `#page_0002.verified.png`, etc.
 
-The non-deterministic fields of the pdf (the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the equivalent XMP metadata) are neutralized so the same source document produces a byte-identical `.verified.pdf` across runs.
+The non-deterministic fields of the pdf (the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the equivalent XMP metadata) are neutralized so the same source document produces a byte-identical `.verified.pdf` across runs. A producer that already emits deterministic bytes can skip that work with [`SkipPdfNormalization`](#skip-pdf-normalization).
 
 Rendering is provided by [Morph.PDFium](https://github.com/Papyrine/Morph.PDFium), which wraps the prebuilt PDFium binaries from [pdfium-binaries](https://github.com/bblanchon/pdfium-binaries) (Windows, Linux, and macOS). Rendering is deterministic for a given Morph.PDFium version: the same input produces byte-identical PNGs on every machine and OS, and no image library dependency is added.
 
@@ -100,6 +100,26 @@ public Task ExcludePdfDocument() =>
 <!-- endSnippet -->
 
 
+### Skip pdf normalization
+
+By default the pdf bytes are normalized before being snapshotted: the trailer `/ID`, the `/CreationDate` and `/ModDate`, and the XMP dates and identifiers are neutralized, and the XMP packet is canonicalized. A producer that already emits byte-deterministic documents gains nothing from that, and pays for a full buffer copy, a rescan, and — when the XMP is canonicalized — a rebuild plus a cross-reference repair. `SkipPdfNormalization` snapshots the bytes exactly as produced:
+
+<!-- snippet: SkipPdfNormalization -->
+<a id='snippet-SkipPdfNormalization'></a>
+```cs
+[Test]
+public Task SkipPdfNormalization() =>
+    VerifyFile("sample.pdf")
+        .SkipPdfNormalization();
+```
+<sup><a href='/src/Tests/Samples.cs#L36-L43' title='Snippet source file'>snippet source</a> | <a href='#snippet-SkipPdfNormalization' title='Start of snippet'>anchor</a></sup>
+<!-- endSnippet -->
+
+Only skip it when the producer is genuinely deterministic. Without normalization a freshly generated pdf carries a wall-clock `/CreationDate` and a fresh `/ID`, so the snapshot differs on every run.
+
+The XMP canonicalization is worth calling out, because it is the pass that changes bytes even for an already-deterministic producer: it collapses the packet's whitespace. Turning this setting on for an existing suite therefore shifts every stored `.verified.pdf` once, without anything about the documents having changed.
+
+
 ## Icon
 
 [PDF](https://thenounproject.com/icon/pdf-7564953//) designed by [Meilia](https://thenounproject.com/creator/meilia1/) from [The Noun Project](https://thenounproject.com).

src/Directory.Build.props

@@ -2,7 +2,7 @@
 <Project>
   <PropertyGroup>
     <NoWarn>CS1591;CS0649;CA1416;NU1608;NU1109;SC023</NoWarn>
-    <Version>1.3.0</Version>
+    <Version>1.4.0</Version>
     <LangVersion>14.0</LangVersion>
     <AssemblyVersion>1.0.0</AssemblyVersion>
     <PackageTags>PDF, PDFium, Verify</PackageTags>

src/Tests/Samples.SkipPdfNormalization.verified.txt

@@ -0,0 +1,16 @@
+{
+  PageCount: 1,
+  Properties: {
+    CreationDate: DateTimeOffset_1,
+    Creator: Morph,
+    ModDate: DateTimeOffset_1,
+    Producer: PDFsharp 6.2.4
+  },
+  Pages: [
+    {
+      Width: 612.0,
+      Height: 792.0,
+      Text: Hello, World! This is a simple paragraph.
+    }
+  ]
+}

src/Tests/Samples.cs

@@ -32,4 +32,13 @@ public Task ExcludePdfDocument() =>
             .ExcludePdfDocument();
 
     #endregion
+
+    #region SkipPdfNormalization
+
+    [Test]
+    public Task SkipPdfNormalization() =>
+        VerifyFile("sample.pdf")
+            .SkipPdfNormalization();
+
+    #endregion
 }

src/Tests/SkipPdfNormalizationTests.NormalizedSnapshotHoldsTheNeutralizedBytes.verified.txt

@@ -0,0 +1,16 @@
+{
+  PageCount: 1,
+  Properties: {
+    CreationDate: DateTimeOffset_1,
+    Creator: Morph,
+    ModDate: DateTimeOffset_1,
+    Producer: PDFsharp 6.2.4
+  },
+  Pages: [
+    {
+      Width: 612.0,
+      Height: 792.0,
+      Text: Hello, World! This is a simple paragraph.
+    }
+  ]
+}

src/Tests/SkipPdfNormalizationTests.SkippedSnapshotHoldsTheProducerBytes.verified.txt

@@ -0,0 +1,16 @@
+{
+  PageCount: 1,
+  Properties: {
+    CreationDate: DateTimeOffset_1,
+    Creator: Morph,
+    ModDate: DateTimeOffset_1,
+    Producer: PDFsharp 6.2.4
+  },
+  Pages: [
+    {
+      Width: 612.0,
+      Height: 792.0,
+      Text: Hello, World! This is a simple paragraph.
+    }
+  ]
+}

src/Tests/SkipPdfNormalizationTests.cs

@@ -0,0 +1,25 @@
+// The Samples entry for SkipPdfNormalization only proves the setting round-trips through a
+// verification. These assert the wiring it is actually there for: with the setting the snapshotted
+// bytes are the producer's own, and without it they are the normalized ones.
+[TestFixture]
+public class SkipPdfNormalizationTests
+{
+    [Test]
+    public void SampleIsNotAlreadyNormalized()
+    {
+        // The premise the two tests below rest on. If sample.pdf were already byte-identical to its
+        // normalized form they would both pass while asserting nothing.
+        var raw = File.ReadAllBytes("sample.pdf");
+
+        Assert.That(PdfNormalizer.Normalize(raw), Is.Not.EqualTo(raw));
+    }
+
+    [Test]
+    public Task SkippedSnapshotHoldsTheProducerBytes() =>
+        Verify(new MemoryStream(File.ReadAllBytes("sample.pdf")), "pdf")
+            .SkipPdfNormalization();
+
+    [Test]
+    public Task NormalizedSnapshotHoldsTheNeutralizedBytes() =>
+        Verify(new MemoryStream(File.ReadAllBytes("sample.pdf")), "pdf");
+}

src/Verify.PDFium/VerifyPDFium.cs

@@ -9,6 +9,10 @@ public static class VerifyPDFium
     // pdf can never be made byte-deterministic. The rendered pages and info file still verify.
     const string excludeDocumentKey = "VerifyPDFium.ExcludeDocument";
 
+    // Context key set by SkipPdfNormalization. When present the pdf bytes are snapshotted as
+    // produced, for producers that already emit byte-deterministic documents.
+    const string skipNormalizationKey = "VerifyPDFium.SkipNormalization";
+
     public static bool Initialized { get; private set; }
 
     /// <param name="dpi">
@@ -43,6 +47,30 @@ public static SettingsTask ExcludePdfDocument(this SettingsTask settings)
         return settings;
     }
 
+    /// <summary>
+    /// Snapshots the pdf bytes exactly as produced, skipping the normalization that neutralizes the
+    /// trailer <c>/ID</c>, the <c>/CreationDate</c> and <c>/ModDate</c>, and the XMP dates and
+    /// identifiers. Use it when the producer already emits byte-deterministic documents, since
+    /// normalizing them again copies the whole buffer, rescans it, and — when the XMP packet is
+    /// canonicalized — rebuilds it and repairs the cross-reference table, all to change nothing.
+    /// </summary>
+    /// <remarks>
+    /// Only skip this when the producer is genuinely deterministic. Without it a freshly generated
+    /// pdf carries a wall-clock <c>/CreationDate</c> and a fresh <c>/ID</c>, so the snapshot differs
+    /// on every run.
+    /// <para>
+    /// The XMP canonicalization is worth calling out because it is the pass that changes bytes for
+    /// an already-deterministic producer: it collapses the packet's whitespace, so enabling or
+    /// disabling this setting on an existing suite shifts the stored <c>.verified.pdf</c> even
+    /// though nothing about the document changed. Expect to re-accept those snapshots once.
+    /// </para>
+    /// </remarks>
+    public static SettingsTask SkipPdfNormalization(this SettingsTask settings)
+    {
+        settings.CurrentSettings.Context[skipNormalizationKey] = true;
+        return settings;
+    }
+
     static ConversionResult Convert(Stream stream, IReadOnlyDictionary<string, object> context)
     {
         using var buffer = new MemoryStream();
@@ -81,9 +109,13 @@ static ConversionResult Convert(Stream stream, IReadOnlyDictionary<string, objec
 
         if (IncludeDocument(context))
         {
-            // Neutralize the volatile fields for the pdf snapshot only once the document, which reads
-            // lazily from the same buffer, has been released.
-            bytes = PdfNormalizer.Normalize(bytes);
+            if (Normalize(context))
+            {
+                // Neutralize the volatile fields for the pdf snapshot only once the document, which
+                // reads lazily from the same buffer, has been released.
+                bytes = PdfNormalizer.Normalize(bytes);
+            }
+
             targets.Insert(
                 0,
                 new("pdf", new MemoryStream(bytes))
@@ -98,4 +130,8 @@ static ConversionResult Convert(Stream stream, IReadOnlyDictionary<string, objec
     static bool IncludeDocument(IReadOnlyDictionary<string, object> context) =>
         !context.TryGetValue(excludeDocumentKey, out var value) ||
         value is not true;
+
+    static bool Normalize(IReadOnlyDictionary<string, object> context) =>
+        !context.TryGetValue(skipNormalizationKey, out var value) ||
+        value is not true;
 }