Refactoring HTML parsing implementation

[bradhanks]

Proposal: HTML Tag Parsing from Regex to pattern matching

Motivation

We sometimes misfire parsing HTML tags embedded in Markdown. By refactoring the parser to use pattern matching instead of regex, we can improve correctness, performance, and maintainability because HTML5 specification provides a closed world problem.

---

Problem Statement

Current Issues:

1. HTML Tag Parsing
   • Faulty parsing of nested HTML tags (Nested code blocks in numbered lists have extra whitespace #161).
   • Inconsistent handling of attributes (allow spaces in front of HTML tags #102, Parse HTML recursively #7).
   • Poor handling of special characters in attributes (Faulty parsing of \ in link #139, Faulty parsing of ** in strong code block #144).
2. Buggy Behavior
   • Incorrect parsing of | within HTML tags (Faulty parsing of | within ` #148).
   • Spaces in front of HTML tags not allowed (allow spaces in front of HTML tags #102).
   • Inline Attribute Lists (IAL) removed from fenced code blocks (IAL removed from fenced code blocks instead of rendered verbatim #94).

Proposed Solution

Refine the existing HTML parsing steps based on the HTML5 specs:

1. Tag Identification: Pattern match on HTML tags (e.g., <a>, <div>, <address>).
2. Attribute Parsing: Handle attributes based on the tag: pattern match through up to 10 attribute categories.
3. Content Parsing: Content parsing process remains unchanged.

Implementation Steps

Phase 1: Specification Mapping & Data Modeling

1. Enumerate Valid Tags and Attributes

   • Create a comprehensive list of all valid HTML5 tags (109 total).
   • Categorize attributes into 10 groups (e.g., global, ARIA, data-*).
   • Map element-specific attributes for each tag (e.g., <img> has src, alt, etc.).

2. Define Types

   • Use Elixir types to model HTML nodes and attributes:

     @type html_tag :: :div | :a | :img | ...
     @type html_attr :: {atom(), String.t()}

or something like that.

Phase 2: Parser Implementation

1. Tag Parsing - Replace regex with pattern-matched dispatch
2. Attribute Parsing - Context-aware parsing based on tag
3. Performance Optimizations - Use binary matching (<<c::utf8, rest::binary>>) for zero-copy parsing.

Phase 3: Testing and Validation

1. Unit Tests

   • Cover all 109 tags and their attributes.
   • Test edge cases like malformed tags, nested tags, and special characters.

2. Property-Based Testing

   • Use StreamData to generate valid/invalid HTML snippets and validate parser behavior.

3. CI Enhancements

   • Add coverage checks for spec completeness.
   • Benchmark against the current regex implementation to measure performance gains.

4. Error Handling

   • Provide clear error messages for invalid markup.
   • Support configurable strictness levels (warn vs. fail-fast).

Benefits

• Maintainability:

  1. Easy to extend for new HTML tags or attributes.
  2. Straightforward to add custom global prefixes.
  3. Eliminate helper modules like behead/2.
  4. No need to be a regex wizard.

• Performance: Efficient parsing with minimal backtracking.

• Correctness:

  1. Addresses issues like nested HTML tags (Nested code blocks in numbered lists have extra whitespace #161) and attribute parsing (allow spaces in front of HTML tags #102).
  2. Retain a catch-all mechanism without relying on regex.

---

Next Steps

• Finalize the HTML5 data spreadsheet. which tags need to look for which tags
• Draft the EarmarkParser.HTML module skeleton.

[RobertDober]

WOW I'll need a little bit of time to read this, but really happy that you wanna tackle this one 👏

[RobertDober]

This is a very impressive and high quality high level functional spec
Waaay beyond what my brain can do, however, I believe I can still add some value.

You have here a very complete implementation in mind, if you have the time and drive to implement all these features that is great. But maybe it might be a good idea to aim at an MVP first, even if we only release the full version.

Here are some features I would maybe not include into the MVP

• Property testing sounds like an overkill (however the technical implementation would very much interest me)
• Checking on tag validity might not be the highest priority for our users
• Coverage in CI and Benchmarking might be better placed into a different scope / issue

Now some operational stuff, as I have granted you collaboration rights you can implement directly in the new branch https://github.com/RobertDober/earmark_parser/tree/i169-refacto-html-parser or in your fork, whatever you prefer.

However, please, if working in your fork let us update https://github.com/RobertDober/earmark_parser/tree/i169-refacto-html-parser via PRs. Ok for you?

[bradhanks]

However, please, if working in your fork let us update https://github.com/RobertDober/earmark_parser/tree/i169-refacto-html-parser via PRs. Ok for you?

Does this mean I should do a PR request on that branch instead of master, basically?

[RobertDober]

Does this mean I should do a PR request on that branch instead of master, basically?

Exactly,

but again you can also work in the repo directly now, up to you.

[bradhanks]

I'll err on the side of caution if it's all the same to you :)

[RobertDober]

sure

[RobertDober]

On the topic

Please Lead on this topic

On the other hand I want you to have the lead on this branch, I will more or less only rebase it or making PRs myself, feel free to merge your PRs into this branch at your disgression!

Parsing

1. I have written a Peg Parser Minipeg but it is a little bit bloated, and I also have troubles creating the docs for it, which makes using it a PITN, even for myself, but if you want to use a peg parser, the only alternative I see is ....

2. Nimble Parsec which I am too stupid to use (I tried). It does not depend on ex_doc anymore which also was an obstacle.

3. Adapt minipeg or write a lean peg parser, even inside EarmarkParser itself.

4. Just recursively parse the thing as we did in the 70s (yeah I am that old) ;).

Eager to hear your ideas on this