feat(ie-html): implement core HTML tokenizer state machine #54

WHATWG HTML tokenizer with ~40 core states:
- Token types: Doctype (with public/system IDs, force_quirks),
  StartTag (with attributes, self_closing), EndTag, Character,
  Comment, Eof
- States: Data, TagOpen, EndTagOpen, TagName, all attribute states
  (before/name/after/value with double/single/unquoted quoting),
  SelfClosingStartTag, BogusComment, MarkupDeclarationOpen,
  all comment states, all doctype states, CDataSection
- Iterator-based: impl Iterator<Item = Token>
- Tag names lowercased per spec
- Pending token queue for multi-token emissions
- set_state() for tree builder feedback
- Parse errors logged via tracing, never abort
- Placeholder stubs for script/raw text/entity states (Step 1b)
- 17 unit tests

Author: thomasnemer

crates/ie-html/src/lib.rs

@@ -3,7 +3,10 @@
 //! WHATWG HTML Living Standard parser.
 //! Targets latest spec only — no quirks mode, no legacy element support.
 
+pub mod token;
 pub mod tokenizer;
 pub mod tree_builder;
 
+pub use token::Token;
+pub use tokenizer::Tokenizer;
 pub use tree_builder::parse;

crates/ie-html/src/token.rs

@@ -0,0 +1,36 @@
+#[derive(Debug, Clone, PartialEq)]
+pub enum Token {
+    Doctype {
+        name: Option<String>,
+        public_id: Option<String>,
+        system_id: Option<String>,
+        force_quirks: bool,
+    },
+    StartTag {
+        name: String,
+        attributes: Vec<Attribute>,
+        self_closing: bool,
+    },
+    EndTag {
+        name: String,
+    },
+    Character(char),
+    Comment(String),
+    Eof,
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct Attribute {
+    pub name: String,
+    pub value: String,
+}
+
+impl Token {
+    pub fn is_start_tag(&self, name: &str) -> bool {
+        matches!(self, Token::StartTag { name: n, .. } if n == name)
+    }
+
+    pub fn is_end_tag(&self, name: &str) -> bool {
+        matches!(self, Token::EndTag { name: n } if n == name)
+    }
+}