⚡ perf: move remaining per-item logic to C#692
Merged
Conversation
The tracking-parameter vocabulary and the dot-segment resolver ran per URL in Python. Both are now C entry points (_url_is_tracker, _url_remove_dot_segments) with the tables as C data; the Python module keeps only the public wrappers. A 302,762-input differential run against the Python originals found one divergence class: the C resolver matches %2e case-insensitively per WHATWG 4.4 where the Python matched only %2E, unreachable through the public API because _encode uppercases escape hex before resolution.
A note about one operation marked a library's whole migration column, and an empty cell asserted 'no measurement recorded' without saying why. Migration feeds now carry row_notes keyed by row index so a caveat marks only the rows of the operation it describes, and a configuration that skips an operation says 'same in both configurations, so measured once' instead of leaving a blank.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
The urls port added _url_is_tracker and _url_remove_dot_segments to the module without declaring them in the stubs, so the type gate rejected their imports. Declared in features.pyi and re-exported through the _html.pyi aggregate.
sax_parse looped in Python over the pull iterator, building a tuple then a typed record then an isinstance chain for every event before the handler call. The tokenizer's html.parser adapter had the same shape and its C dispatch removed 30.8% of instructions, so the sax walk gets the identical treatment: _sax_dispatch binds the six handler methods once and drives them straight off the tree walk, building no per-event record. Callgrind on the whatwg spec with a counting handler: 900.7M instructions to 507.0M (-43.7%), instruction-cache misses -71%, indirect-branch mispredicts -61%. A 38,590-call differential against the retired Python loop over 18 documents, including foster-parented tables, adoption-agency nesting, template content and both doctype identifier forms, found zero divergence.
escape_identifier walked the string per character in Python applying the CSSOM serialize-an-identifier rules. The loop is now the C entry point _css_escape_identifier and the Python def keeps only its docstring and the call. A 4,828-input differential across the BMP boundaries, astral plane, leading-digit and lone-dash shapes found zero divergences against the retired loop.
for more information, see https://pre-commit.ci
is_schema_el reclassified the same schema element nodes millions of times per document -- the content-model matcher revisits every particle on each NFA step -- and each call walked the ancestor chain through resolve_ns to re-resolve the node's namespace. That walk was the validator's single hottest cost. Resolve every schema element node's qname once at compile time into a pointer-sorted table and look it up by binary search. The table is built before any classification runs and never mutated, so a compiled schema shared across validating threads stays read-only. On the catalog XSD + doc bench (Callgrind, python:3.13-slim): 1,118.9M to 926.6M Ir, -17.2%. resolve_ns 206.1M -> 56.9M (-72%), th_attr_name 50.0M -> 19.7M (-61%). All XSD/RelaxNG conformance tests pass unchanged.
Merging this PR will improve performance by 34.69%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | test_feature[sax] |
47.3 ms | 12.9 ms | ×3.7 |
| ⚡ | test_feature[decode-gb18030-ranges] |
328.7 µs | 200.7 µs | +63.79% |
| ⚡ | test_feature[urls-clean] |
3 ms | 2.2 ms | +35.6% |
| ⚡ | test_feature[links-extract] |
1,194.9 µs | 918.5 µs | +30.09% |
| ⚡ | test_feature[validate] |
11.8 ms | 9.9 ms | +18.86% |
| ⚡ | test_feature[links-external] |
32.5 ms | 27.8 ms | +17.19% |
| ⚡ | test_feature[idna-varied] |
82.7 ms | 72.4 ms | +14.23% |
| ⚡ | test_feature[idna] |
82.2 ms | 72 ms | +14.2% |
| ⚡ | test_feature[minify-css] |
5.5 ms | 4.9 ms | +12.08% |
| ⚡ | test_feature[links-filter] |
8.8 ms | 8 ms | +10.86% |
| ⚡ | test_feature[links-absolutize] |
3.1 ms | 2.8 ms | +10.37% |
| 🆕 | test_feature[escape-identifier] |
N/A | 747.8 µs | N/A |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing gaborbernat:perf/urls-to-c (e81a55d) with main (84b785c)
Footnotes
-
29 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Two independent read-path wins, each verified with Callgrind in Docker on the 95 kB mozilla page (instructions retired, not wall clock, which swings 7-30% on this machine). links(): the walk wrapped every element in a Python Element before scanning it, then dropped the wrapper, even though a real page carries a link on a small fraction of its elements. Mint the wrapper lazily on the first link an element emits. 59,171,837 -> 50,408,887 Ir (-14.8%). resolve_links(): urljoin, the stdlib join it stands in for, is ~99% of the cost and returns an already-absolute reference byte-for-byte. Prove that in C for the common cases -- a scheme differing from the base's, or a same-scheme scheme://authority under an http/https base with a lowercase scheme and no tab/CR/LF -- and skip the call. Output stays identical (the rewrite already no-ops an unchanged join), so no new benchmark id. 274,610,778 -> 88,284,235 Ir (-67.8%).
operations.py wraps each XPath expression in code backticks, but the migration and interpreter tables ran that label through rst_safe, which escaped the backticks so the table showed literal ``//div`` instead of code. A case name is an authored RST fragment already, so drop the second escaping and let it render as written; regenerate the two affected feeds.
The xpath dispatch raised a bare KeyError for the XPath 2.0 string functions libxml2 does not implement, so the table's legend read 'ends-with' with no explanation. Raise a sentence that names why the cell is empty, and rewrite the committed reasons to match.
Render the bench-table directive over a synthetic feed with row_notes and assert only the noted rows carry a superscript, two rows sharing a caveat share one ordinal, and the legend spells each one out. Adds docutils to the test group so the directive renders under the coverage gate.
jsdom (Node) and MechanicalSoup (delegates parsing to BeautifulSoup) had no performance table and no reason given. State why each has none -- jsdom runs out of the CPython process, MechanicalSoup's parse cost is BeautifulSoup's -- and print the migration generator's skip list to stderr so a page that loses its table cannot go unnoticed.
The <meta> stage gathered every candidate into a Python list and then ran the shared _pick selection over it; both now run in one C walk (Document._date_meta) that classifies each element against htmldate's key vocabulary -- moved to C as sorted tables -- and returns the winner _pick would, stopping at the first wanted-role hit. The Python module keeps only the config, the call, and the result wrap. A 360,282-input differential against the Python original found no divergence. Callgrind (python:3.13-slim) on the bench date cases: the 100-meta case drops from 3.78M to 1.63M Ir per call (-57%), the article cases about -3%.
The eager schema qname cache made XSD validation 22% faster on the CodSpeed gate and RELAX NG compilation 8% slower: the XSD content-model matcher re-classifies the same schema elements on every particle step, while the derivative walk never revisits at a rate that repays the build, so a small grammar paid the count-allocate-fill-sort ceremony for nothing. The cache now builds only for an XSD, RELAX NG resolves names on the spot through the same helper, and the build itself resolves each name against an in-scope xmlns binding stack during one walk instead of an ancestor-and-attribute scan per element. Callgrind on the gated workloads: compile-rng recovers the full 6.1k instructions per call the eager build added; validate keeps its gain.
_scrub, _normalize_query's pair loop and sort, _variant_key, and the _language_matches heuristics ran per URL, per query pair, and per link in Python. Each is a C entry point now (_url_scrub, _url_normalize_query, _url_variant_key, _url_language_matches); the module keeps only the public wrappers, passing the tracking and language vocabularies in. A 1,484,640-input differential against the pre-port Python -- clean_url, normalize_url, and extract_links over generated URL and link corpora plus the bench inputs -- found zero divergence. Callgrind in Docker: urls-clean 2.44G -> 2.17G Ir (-11.3%), links-filter 1.83G -> 1.72G Ir (-5.8%).
# Conflicts: # src/turbohtml/_html.pyi # src/turbohtml/_stubs/features.pyi
A cell a configuration shares with its sibling used to render as an explained blank. It now shows the shared figure in both columns, with the measured-once caveat kept as the row's note, so a reader compares numbers everywhere and still sees that one measurement stands behind both. The urls agent's surrogate oracle also aligned with the retired shim: the public path converts the encoder's UnicodeEncodeError to the documented ValueError, and the C does the same inline.
Sweep the urls-to-c comments for AI tells: drop the "simply" and "genuinely" adverbs from the url.c and schema.c rationale, rephrase the _date_meta suppression note off first person, and delete the tracker-list provenance comment left stranded in _urls.py once its tables moved to C.
Absent variant cells now carry the sibling's shared figure rather than staying empty, so a row cell can no longer be None. Drop None from the _rows cell types and declare row once above the wide/narrow branch so both arms agree with the append sites; refresh the doc and comment that still described empty cells. Inline the single-use prefixed flag to keep _rows under the local-count limit.
PyPy's str.split() keeps the C0 separators 0x1C-0x1F that CPython treats as whitespace, so the scrub oracle diverged from the C only on PyPy and only on corpus inputs PyPy's set ordering happened to generate. The oracle now filters the pinned CPython whitespace set the C's Py_UNICODE_ISSPACE implements on every interpreter.
CI's gcc gate flagged the untested non-str arm the identifier escape rejects; the local llvm run folded it, which is why both toolchains gate.
remove_dot_segments always records at least one path segment, so the ternary guarding kept-1 against a zero kept can never take its zero arm. Drop it; the empty-path result stays covered via %2E normalizing to an empty path.
The XSD qname cache resolves schema-element tag prefixes only. An xml:-prefixed schema element is foreign, and is_schema_el maps both the xml namespace and no namespace to "not a schema element", so the xml special case in ns_scope_resolve computes a value no validation can observe. resolve_ns keeps the xml rule for the document path, where xml:lang is observable.
Nine xmlns declarations on one element grow the in-scope namespace stack past its initial capacity, exercising the copy of already-held bindings. Deleting that copy drops the first-declared xs binding and the schema stops recognizing xs:element, so the test fails if the copy is removed.
The direct resolver's length test can pass on an attribute that is not a namespace declaration; the decoy has to precede the real declaration in attribute order or the walk returns before testing it, which is why the macOS gate still flagged the memcmp arm after the first version of this case.
escape_identifier forwards straight to the C _css_escape_identifier, whose TypeError guard for a non-str argument had no test. Mirror the escape/unescape guards so the arm is exercised.
# Conflicts: # tests/query/match/test_escape.py
The memchr jump made every dispatch arm's buf[pos] == '<' re-test invariant true, so those checks are deleted rather than excluded; the prescan edge shapes (a meta at the window end, a self-closing meta name, a truncated closing tag, a double-slash bogus comment) now have direct cases. The escape chain gains its second-position combinations, and the split-line arena guard carries the exclusion across both physical lines.
Re-measure the operations whose C paths moved in this campaign (the per-item URL loops, the date meta stage, the linkify scanner, the CSS minify shorthand pass, the encoding meta prescan, the SAX dispatch, the XSD/RelaxNG builders) and wire the new escape-identifier operation into the performance guide as querying-6. Non-impacted operations keep their accepted numbers; decode is not impacted (the only decode.h change is inside th_dec_euc_jp, which the decode workload never calls). Two shifts are worth calling out: - The date meta stage moving to C flips the synthetic 100-meta-tag row turbohtml used to lose into a 3.0x/4.1x lead over htmldate and trafilatura. - The encoding re-sweep corrects the faust-cchardet column, whose old numbers were physically implausible (70us for uchardet on a 95 kB page); the fresh figures are stable across runs.
Recompute every quantitative claim near a changed table from the fresh feeds and write the escape-identifier paragraph (16x over soupsieve's escape). The largest rewrites: the date-extraction section now reports turbohtml leading the 100-meta-tag stress it once lost; the encoding-detection section drops the stale "faust-cchardet wins two workloads" framing for the corrected native-detector picture; url cleaning, linkify, detect, and the CSS competitor pages get their ranges recomputed.
gaborbernat
marked this pull request as ready for review
July 21, 2026 13:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-item loops still ran in Python on hot read paths.
sax_parsebuilt a tuple, a typed record, and an isinstance chain for every event before reaching the handler; URL cleaning consulted a Pythonfrozensetand a regex per query parameter;escape_identifierwalked its string one character at a time; the date<meta>stage gathered every candidate into a list before selecting; andlinkifyre-ran a compiledscheme://regex per match. The project rule is that only the immediate public API stays Python, and each of these paid interpreter dispatch per item on inputs sized by the document, not by the caller's arguments. This branch is the single landing point for the whole port campaign, taking over from the first-stage draft.Each port follows the pattern the
html.parseradapter set in #689: bind the handler methods once, drive them straight from the C walk, and materialize nothing per item. Forsax_parsethat drops the per-event tuple, record, and type test; Callgrind on the WHATWG specification measures 900.7M instructions down to 507.0M (43.7% fewer), with instruction-cache misses down 71% and indirect-branch mispredicts down 61%, and the CodSpeed gate records a 3.7x drop. A 38,590-call differential against the retired loop over 18 documents, covering foster-parented tables, adoption-agency nesting, template content, and both doctype identifier forms, found no divergence. The pull formiter_eventskeeps its typed records, since those records are its public product.URL work moved the same way. Tracker classification, dot-segment resolution, query normalization, the variant key, and the language filter became the C entry points
_url_is_tracker,_url_remove_dot_segments,_url_scrub,_url_normalize_query,_url_variant_key, and_url_language_matches, with the tracking and language vocabularies handed in as sorted C data; the module keeps only the public wrappers. Callgrind putsurls-cleanat 2.44G to 2.17G instructions (11.3% fewer) andlinks-filterat 1.83G to 1.72G (5.8% fewer), and a 1,484,640-input differential plus an earlier 302,762-input run found a single divergence class: the C resolver matches%2ein either case as the WHATWG URL path state specifies, which no public-API call can reach because percent-encoding uppercases escape hex before resolution runs. The date<meta>stage now classifies and selects in one C walk (Document._date_meta) instead of building a Python candidate list, dropping the 100-candidate page from 3.78M to 1.63M instructions per call (56.8% fewer) with a 360,282-input differential clean;escape_identifiercalls_css_escape_identifierunder a 4,828-input differential and now rides the CodSpeed gate; andlinkifyreads the scheme kind from the C scan and drops its regex, at 3.5% and 4.0% fewer instructions on the rewrite and detector paths under an 809,481-comparison differential.Four more hot paths kept their Python entry and shed work. XSD validation resolves every schema element's qname once at compile time into a pointer-sorted table read by binary search, cutting the catalog workload from 1,118.9M to 926.6M instructions (17.2% fewer,
resolve_nsdown 72%) and posting a 22% CodSpeed gain. An eager version of that cache slowed RELAX NG compilation by 8.15% and lost on the instrument, so the build now runs for XSD only and RELAX NG resolves names on the spot. Link extraction mints its element wrapper only when an element emits its first link, taking the mozilla page from 59.2M to 50.4M instructions (14.8% fewer), andresolve_linksproves in C the common cases where a URL join returns an already-absolute reference and skips the stdlib call, from 274.6M to 88.3M (67.8% fewer); the CodSpeed gate reads 29.5% and 10.7% on the two. The encoding<meta>prescan jumps the inert bytes between tags withmemchr(8.67% fewer on 1kB ASCII, 5.91% on 4kB UTF-8), the gb18030 decoder maps four-byte astral pointers by arithmetic instead of searching the range list (13.4% fewer, identical bytes across the whole astral plane), and the CSS property dispatch gates eachcss_run_ieqon a first-byte comparison, takingbootstrap.cssfrom 779.6M to 706.8M (9.3% fewer) with byte-identical output on six stylesheets.The benchmark feeds behind the migration guides stopped over-claiming. A caveat about one operation now marks only that operation's rows, not the library's whole column; a cell two configurations share shows the figure in both columns with a
same in both configurations, so measured oncenote in place of a blank; XPath expression labels render as inline code instead of literal double backticks; the pages without a performance table state why and print the generator's skip list so a lost table cannot pass unnoticed, since jsdom runs outside the CPython process and MechanicalSoup delegates parsing to BeautifulSoup; and the CSS-stripper note gives the measured size gap, wherercssminlands within 1.01-1.03x of turbohtml and 0.998x on bulma.A function-by-function audit of the remaining Python surface, screened by control-flow density, settled the scope question.
query/__init__.pyanddetect.pyare shim only under line-referenced reading, and everything else left in Python is a config object, argument adaptation sized by the caller's inputs, or the stdlibjsonboundary behind JSON-LD parsing. The audit with per-function line references ships in this PR's discussion.