Flash embedded-TOC merge: dissolve duplicated headings, place new sections at their level - #391
Open
rejojer wants to merge 2 commits into
Open
Flash embedded-TOC merge: dissolve duplicated headings, place new sections at their level#391rejojer wants to merge 2 commits into
rejojer wants to merge 2 commits into
Conversation
… frame chain
The FULL-tier merge compared each detected node only against its single
deepest anchor, so a numbering-prefixed variant of a shallower same-page
bookmark ('5 Experiments' under frame 'Experiments' > 'Scaling Laws')
was treated as new and grafted inside its own subsection. Dissolved
parents also raw-inserted their children without any duplicate check.
Dissolve now walks the whole same-page frame chain and re-places
children individually.
The dissolve match is equal / numbering-prefix / 0.9-garble. Reusing the
0.7 repair bar would false-match wordy-overlapping titles of distinct
sections (2023-annual-report: 'Reserve Bank and Branch Directors' vs
'Federal Reserve Banks and Branches', 0.746) and a false dissolve
deletes a real heading, while an escaped garbled duplicate only leaves a
stray leaf.
Across the nine example PDFs: 40 duplicate nodes removed (attention 7,
PRML 16, Regulation 8, annual report 9), zero other structure changes;
the four detected-tier trees are byte-identical.
…chain The FULL-tier merge hung every non-duplicate detected node under the deepest bookmark section active at its page, so a tail section the bookmarks lack (Conclusion, References after the last numbered section) was buried inside the preceding section. A new node now climbs its anchor's ancestor chain -- the only parents page containment allows -- on positive evidence that it is a peer, not a child: a numbering prefix outside the candidate's numbering, or the same style fingerprint (face, weight and size as one hash; detection exports _style/_y and extract_toc strips them after the merge). Frame nodes learn style and position from the duplicates that dissolve into them. Two guards keep noise contained. Raw size is never compared across faces: PRML margin-box titles run 13pt regular against 11.5pt bold subsection headings and must not outrank them. A same-page climb must leave the candidate real content below its heading; climbing on a near-zero gap would claim the section is empty, the signature of a margin box rather than a tail section. Across the nine example PDFs: attention-residuals gains top-level Conclusion/References (the target); PRML chapter-7 exercise items move from inside 7.2.3 to their chapter per their numbering, correcting the 7.2/7.2.3 end pages; the Interpretive release's Federal Register tail matter leaves section B. All other trees byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes to the FULL-tier bookmark merge in
pageindex/flash/embedded_toc.py. Extracted from the sdk-local line of work so #389 stays SDK-only — flash is independent of the SDK changes.1. Duplicated headings survived the merge (bf5d841)
The merge only same-heading-checked the single deepest bookmark anchor on a page. When a detected heading duplicated a shallower same-page bookmark — typically a numbering-prefixed variant (
5 Experimentsvs bookmarkExperiments) — it grafted inside its own subsection, dragging its detected children with it.Now a detected node checks the whole same-page frame chain for an identity match (normalized-equal / numbering-prefix suffix / fuzzy ≥ 0.9), dissolves into the match, and its children re-place individually. The fuzzy bar is 0.9, deliberately above the 0.7 title-repair bar: a false dissolve deletes a real section (0.7 killed the annual report's real "Reserve Bank and Branch Directors", 0.746 vs its chapter title), while OCR garbles that need fuzzy sit around 0.95 ("Applications of peA").
2. New detected sections always nested under the deepest anchor (c9b9220)
A detected heading with no bookmark counterpart was always attached to the deepest open bookmark section at its page — so a paper's unbookmarked
Conclusion/Referenceslanded inside7 Related Work.Since sections are intervals, a new node's parent must lie on the open-section chain at its page; placement is a depth choice on that chain. The node now climbs the chain while there is positive evidence the candidate is a peer, not a parent:
2.1stays under2.,4climbs past it), orA same-page climb additionally requires a vertical gap ≥ 0.15 page heights below the candidate heading — climbing on a near-zero gap asserts the section is empty, which is the margin-box signature, not a real successor section. Bookmark frame nodes inherit style/position from duplicates that dissolve into them (fix 1 feeds fix 2), and the internal
_style/_ykeys are stripped from the final structure.Verification
Before/after structure snapshots over the nine example PDFs with
summary=False(LLM-free, deterministic):attentionConclusion/Referencesbecome top-level sections (the target), PRML margin bios all stay contained at their real depth, PRML ch7 exercise items re-level to the chapter per numbering (the bookmarks have no ch7Exercisesnode; this corrects two sections' end pages).8 unit tests added (
tests/test_flash_embedded_toc.py); full suite 26 passing.