`, ``, ``, ``, ``, ``) must round-trip with ranges intact. +2. **"Multiple decorators"** — only difference: a LINK entity appears for `http://www.google.com#world` (the export fixture's linkify composite decorator created the link; the original content state has plain text). +3. **"From https://github.com/icelab/…"** — differences: the IMAGE entity keeps only `src` (plus `alt` if present in the Markdown) — `caption`, `rightsHolder`, `featured` are lost; the atomic block text becomes `" "` (was `" :)"`). +4. **"Big content export"** — differences: the EMBED entity and its atomic block are gone (the export config discards EMBED); the IMAGE entity keeps only `src` and `alt` (`width`/`height` lost); the HIGHLIGHT style is lost (exported as ``, which has attributes → literal text, so the text also gains the literal tag characters); STRIKETHROUGH lost with literal `~…~` text; KBD round-trips via the whitelist; HORIZONTAL_RULE round-trips exactly. + +Any other difference indicates a parser bug — fix the parser, not the fixture. + +- [ ] **Step 3: Create `tests/test_imports.json`** + +Direct import fixtures. Cases (each with `label`, `markdown`, `content_state`; `config` where noted): + +1. **"Italic with stars"** — `*a*` → ITALIC range over `a`. +2. **"Bold with underscores"** — `__a__` → BOLD range. +3. **"Plus bullet list"** — `+ a` → `unordered-list-item`. +4. **"Paren ordered list"** — `1) a` → `ordered-list-item`. +5. **"Tilde fence"** — `~~~\nx\n~~~` → `code-block` with text `x`. +6. **"Star thematic break"** — `***` → atomic HORIZONTAL_RULE. +7. **"Inline HTML styles"** — markdown `x 2`, config `{"parser_config": {"inline_html_styles": {"sup": "SUPERSCRIPT"}}}` → SUPERSCRIPT range over `2`. +8. **"Wagtail internal image"** — markdown ``, config `{"wagtail_resolvers": true}` → atomic block, IMAGE entity data `{"id": 10, "format": "left", "alt": "alt"}` (label fills `alt` since query omits it). +9. **"Wagtail page link"** — markdown `[label](wagtail://page?id=3)`, config `{"wagtail_resolvers": true}` → LINK entity data `{"id": 3}`. +10. **"Filter demote in importer"** — markdown `# T`, config `{"filter_rules": [{"type": "block", "match": "header-one", "action": "demote"}]}` → `header-two` block. + +Full example for fixture 1 (others follow the same shape): + +```json +[ + { + "label": "Italic with stars", + "markdown": "*a*", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { "offset": 0, "length": 1, "style": "ITALIC" } + ], + "entityRanges": [] + } + ] + } + } +] +``` + +- [ ] **Step 4: Run tests** + +Run: `just test tests/test_imports.py` +Expected: PASS (all 18 round-trip + 10 direct cases) + +- [ ] **Step 5: Commit** + +```bash +git add tests/test_imports.py tests/test_imports.json tests/test_exports.json +git commit -m "Add importer snapshot tests with fixture round-trips" +``` + +--- + +### Task 16: Property-based tests + +**Files:** + +- Modify: `tests/test_properties.py` (append) +- Modify: `tests/strategies.py` (append — safe-text strategy + importer-friendly content states) + +**Interfaces:** + +- Consumes: `MarkdownImporter`, `ContentStateFilter`, existing `content_states` strategy, `MARKDOWN_CONFIG`. + +- [ ] **Step 1: Extend `tests/strategies.py`** + +Append: + +```python +# Text that survives a Markdown export → import round-trip unchanged: +# no Markdown-special characters, no leading/trailing whitespace quirks. +safe_block_text = st.text( + alphabet=st.characters( + whitelist_categories=["Ll", "Lu", "Nd"], whitelist_characters=" " + ), + min_size=1, + max_size=20, +).map(str.strip).filter(bool) + +ROUNDTRIP_BLOCK_TYPES = [ + BLOCK_TYPES.UNSTYLED, + BLOCK_TYPES.HEADER_ONE, + BLOCK_TYPES.BLOCKQUOTE, + BLOCK_TYPES.UNORDERED_LIST_ITEM, + BLOCK_TYPES.ORDERED_LIST_ITEM, +] + +ROUNDTRIP_STYLES = [INLINE_STYLES.BOLD, INLINE_STYLES.ITALIC, INLINE_STYLES.CODE] + + +@st.composite +def roundtrip_blocks(draw: st.DrawFn) -> dict[str, Any]: + """A block whose type, text, and styles survive a Markdown round-trip.""" + text = draw(safe_block_text) + return { + "key": draw( + st.text( + alphabet=st.characters(min_codepoint=97, max_codepoint=122), + min_size=5, + max_size=5, + ) + ), + "text": text, + "type": draw(st.sampled_from(ROUNDTRIP_BLOCK_TYPES)), + "depth": 0, + "inlineStyleRanges": draw( + st.lists( + ranges( + text, + st.builds(lambda s: {"style": s}, st.sampled_from(ROUNDTRIP_STYLES)), + ), + max_size=2, + ) + ), + "entityRanges": [], + } + + +@st.composite +def roundtrip_content_states(draw: st.DrawFn) -> dict[str, Any]: + """Content states limited to constructs both engines support.""" + return { + "entityMap": {}, + "blocks": draw(st.lists(roundtrip_blocks(), min_size=0, max_size=4)), + } +``` + +- [ ] **Step 2: Add property tests** + +Append to `tests/test_properties.py`: + +```python +from draftjs_exporter.html import HTML +from draftjs_exporter.markdown import CONFIG as MARKDOWN_CONFIG +from draftjs_exporter.markdown_importer import MarkdownImporter +from draftjs_exporter.contentstate_filter import ContentStateFilter +from tests.strategies import content_states, roundtrip_content_states + + +class TestImporterProperties(unittest.TestCase): + @given(roundtrip_content_states()) + def test_round_trip_preserves_block_types_and_text(self, content_state): + """Export → import keeps block types and text for safe content.""" + markdown = HTML(MARKDOWN_CONFIG).render(content_state) + result = MarkdownImporter().import_markdown(markdown) + self.assertEqual( + [b["type"] for b in result["blocks"]], + [b["type"] for b in content_state["blocks"]], + ) + self.assertEqual( + [b["text"] for b in result["blocks"]], + [b["text"] for b in content_state["blocks"]], + ) + + @given(content_states()) + def test_filter_produces_valid_content_state(self, content_state): + """Filtered output never has orphaned entity ranges.""" + filter_ = ContentStateFilter( + [{"type": "block", "match": BLOCK_TYPES.HEADER_ONE, "action": "remove"}] + ) + result = filter_.apply(content_state) + entity_map = result.get("entityMap", {}) + for block in result["blocks"]: + text_length = len(block.get("text", "")) + for r in block.get("entityRanges", []): + self.assertIn(str(r["key"]), entity_map) + self.assertLessEqual(r["offset"] + r["length"], text_length) + for r in block.get("inlineStyleRanges", []): + self.assertLessEqual(r["offset"] + r["length"], text_length) + + @given(content_states()) + def test_filter_idempotent(self, content_state): + """Applying the same filter twice yields the same result.""" + filter_ = ContentStateFilter( + [{"type": "inline_style", "match": INLINE_STYLES.BOLD, "action": "remove"}] + ) + once = filter_.apply(content_state) + twice = filter_.apply(once) + self.assertEqual(once, twice) +``` + +Note: `BLOCK_TYPES` and `INLINE_STYLES` are already imported in `test_properties.py` — reuse those imports. If the round-trip property fails on blockquote/list depth edge cases, constrain `depth` generation rather than weakening the assertion. + +- [ ] **Step 3: Run** + +Run: `just test tests/test_properties.py` +Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add tests/test_properties.py tests/strategies.py +git commit -m "Add importer and filter property-based tests" +``` + +--- + +### Task 17: example.py import demo + +**Files:** + +- Modify: `example.py` + +- [ ] **Step 1: Add the import demo** + +At the end of the `__main__` block in `example.py`, after the Markdown export section, add: + +```python + # --- Markdown import --- + + from draftjs_exporter import MarkdownImporter + from draftjs_exporter.markdown_parser import scheme_resolver + + importer = MarkdownImporter( + { + "parser_config": { + "image_resolvers": [ + scheme_resolver( + "wagtail", + {"image": "IMAGE"}, + coerce={"id": int}, + label_key="alt", + mutability="IMMUTABLE", + ) + ], + }, + "filter_rules": [ + {"type": "block", "match": BLOCK_TYPES.HEADER_ONE, "action": "demote"}, + ], + } + ) + imported = importer.import_markdown(markdown_output) + + print("=== Markdown import ===") # noqa: T201 + print(json.dumps(imported, indent=2)) # noqa: T201 +``` + +Move the `from draftjs_exporter import MarkdownImporter` import to the top-level import block (ruff E402); keep `scheme_resolver` imported from `draftjs_exporter.markdown_parser` alongside the existing `markdown_link` import. + +- [ ] **Step 2: Run the example** + +Run: `uv run example.py docs/example.json` +Expected: prints HTML, Markdown, then the imported ContentState JSON; exits 0. + +- [ ] **Step 3: Commit** + +```bash +git add example.py +git commit -m "Add Markdown import demo to example.py" +``` + +--- + +### Task 18: Documentation + +**Files:** + +- Create: `docs/markdown-importer.md` +- Modify: `mkdocs.yml`, `CHANGELOG.md`, `.agents/skills/draftjs_exporter/SKILL.md` + +- [ ] **Step 1: Write `docs/markdown-importer.md`** + +Follow `docs/style-guide.md` (sentence case, American English). Sections: + +1. **Intro** — experimental status (mirroring the Markdown export page), one-paragraph overview of parse → filter. +2. **Getting started** — `MarkdownImporter().import_markdown(md)` minimal example with output. +3. **Supported Markdown** — the construct table from the spec; explicitly list out-of-scope items (reference links, Setext, indented code, tables, autolinks). +4. **Parser configuration** — every `ParserConfig` key with defaults; a toggles example (`"headings": False`). +5. **Entity resolution** — resolver chain concept, defaults, `scheme_resolver` with the `wagtail://` example from the spec, writing a custom resolver (signature + contract: return resolution or `None`). +6. **Inline HTML** — `inline_html_styles` whitelist, safety model (literal text otherwise), example. +7. **Filtering** — `FilterRule` reference: rule kinds, actions (`remove`/`keep`/`demote`/callable), the heading-demote example, depth re-normalization note, standalone use of `ContentStateFilter`. +8. **Errors** — `MarkdownParseError` shape and when it fires. +9. **Custom parser engines** — the `parser` dotted-path config and the engine contract (`__init__(config)`, `parse(str) -> ContentState`). + +- [ ] **Step 2: Register in nav and changelog** + +`mkdocs.yml` — after the `Markdown support` line: + +```yaml +- Markdown importer: markdown-importer.md +``` + +`CHANGELOG.md` — under the unreleased section, add a feature entry: "Add experimental Markdown importer: `MarkdownImporter` converts Markdown to Draft.js ContentState, with configurable entity resolvers, inline HTML style whitelist, and `ContentStateFilter` for content policy." + +- [ ] **Step 3: Update the skill** + +`.agents/skills/draftjs_exporter/SKILL.md`: + +- Add a quick-reference row: "Import Markdown as ContentState | `MarkdownImporter({}).import_markdown(md)` | [Markdown importer](…/markdown-importer/)" +- Add rows for `scheme_resolver` usage and `ContentStateFilter` demote example. +- Extend the Public API list with the new exports. +- Description frontmatter: add `markdown_to_content_state` trigger — update the `description` field to mention importing Markdown. + +- [ ] **Step 4: Build docs** + +Run: `just docs-build` +Expected: builds with `--strict`, no warnings. + +- [ ] **Step 5: Commit** + +```bash +git add docs/markdown-importer.md mkdocs.yml CHANGELOG.md .agents/skills/draftjs_exporter/SKILL.md +git commit -m "Document the Markdown importer" +``` + +--- + +### Task 19: Final verification + +- [ ] **Step 1: Full test suite** + +Run: `just test` +Expected: PASS, no warnings. + +- [ ] **Step 2: Coverage** + +Run: `just test-coverage` +Expected: 100% on `draftjs_exporter/markdown_parser/`, `draftjs_exporter/contentstate_filter/`, `draftjs_exporter/markdown_importer/`. Fill gaps with targeted tests — do not lower the target. + +- [ ] **Step 3: Lint and types** + +Run: `just lint` +Expected: clean (ruff check, ruff format --check, mypy, ty). + +- [ ] **Step 4: Compatibility suite** + +Run: `just test-compatibility` +Expected: PASS on Python 3.10 with pinned old deps. + +- [ ] **Step 5: Commit any fixes** + +```bash +git add -A +git commit -m "Polish Markdown importer for release" +``` + +--- + +## Self-review notes + +- **Spec coverage:** parser (Tasks 1–12), resolvers + scheme helper (2, 6, 14), inline HTML (7), filter (13), importer API (14), all test layers (15, 16 + unit tests per task), example (17), docs (18), changelog/skill (18). +- **Known deviations from spec:** none beyond the two clarifications listed at the top (MarkdownParseError rarity; atomic block pattern). +- **Type consistency check:** `resolve_image_entity`, `_add_text_block`, `ParserConfig` keys, and `FilterRule` fields are used identically across tasks. diff --git a/docs/superpowers/specs/2025-07-21-markdown-importer-design.md b/docs/superpowers/specs/2025-07-21-markdown-importer-design.md new file mode 100644 index 0000000..e83287b --- /dev/null +++ b/docs/superpowers/specs/2025-07-21-markdown-importer-design.md @@ -0,0 +1,433 @@ +# Markdown Importer — Design Spec + +## Overview + +Add a Markdown-to-ContentState importer to `draftjs_exporter`, complementing the +existing ContentState-to-HTML/Markdown exporter. The design follows a two-phase +architecture: parsing (Markdown text → valid ContentState) then filtering +(ContentState → ContentState with content policy applied). A public +`MarkdownImporter` class wires them together with a config shape that mirrors +the exporter's pattern. + +## Architecture + +``` +Markdown text + │ + ▼ +┌─────────────────────┐ +│ MarkdownParser │ ← Phase 1: "engine" +│ (parser config) │ Guarantees structural integrity. +└────────┬────────────┘ + │ ContentState (always valid) + ▼ +┌─────────────────────┐ +│ ContentStateFilter │ ← Phase 2: filtering +│ (filter rules) │ Enforces content policy. +└────────┬────────────┘ + │ ContentState (valid, filtered) + ▼ +┌─────────────────────┐ +│ MarkdownImporter │ ← Public API class +│ wires parser + │ Config(config).import_markdown(md) +│ filter together │ +└─────────────────────┘ +``` + +### New modules + +- `draftjs_exporter/markdown_importer/` — `MarkdownImporter` class and config builder +- `draftjs_exporter/markdown_parser/` — parsing engine, no dependencies; + includes `resolvers.py` with the `scheme_resolver` helper +- `draftjs_exporter/contentstate_filter/` — filtering engine, reusable independently + +### Cross-cutting decisions + +- **Single engine** with a clear seam: the parser is referenced by dotted path + in config, matching the exporter's `engine` pattern. A third-party parser + (e.g., `mistune`-backed) can be swapped in later. +- **No external dependencies** — the built-in parser is hand-written. +- **CommonMark core coverage:** paragraphs, ATX headings, blockquotes, fenced + code, thematic breaks, unordered/ordered lists with depth tracking, bold, + italic, inline code, links, images, hard line breaks. No reference-style + links, HTML blocks, Setext headings, or indented code. +- **Configurable entity resolution:** link and image URLs pass through a + resolver chain before entities are created. This supports both "public" + Markdown (`` → IMAGE with `src`/`alt`) and "internal" + representations (`` → IMAGE + with `id`/`src`/`alt`/`format`) with the same parser. +- **Limited inline HTML:** a configurable whitelist of inline HTML tags + (e.g., ``, ``) maps to inline styles, for formatting that has no + Markdown equivalent. Unrecognized HTML is treated as literal text — never + as markup — so there is no XSS surface at the ContentState level. + +## MarkdownParser — Phase 1 + +### Interface + +```python +class ParserConfig(TypedDict, total=False): + headings: bool # default True + blockquote: bool # default True + code_fenced: bool # default True + thematic_break: bool # default True + unordered_list: bool # default True + ordered_list: bool # default True + emphasis: bool # default True — bold + italic + code_inline: bool # default True + links: bool # default True + images: bool # default True + line_breaks: bool # default True — hard breaks only + link_resolvers: list[EntityResolver] # default [] + image_resolvers: list[EntityResolver] # default [] + inline_html_styles: dict[str, str] # default {} — e.g. {"sup": "SUPERSCRIPT"} + + +class MarkdownParser: + __slots__ = ("config",) + + def __init__(self, config: ParserConfig | None = None) -> None: ... + def parse(self, markdown: str) -> ContentState: ... +``` + +### Parsing approach + +A hand-written recursive descent parser operating on a line iterator. No AST — +the parser emits ContentState blocks directly. Block-level parsing handles +constructs that span lines (lists, blockquotes, fenced code). Inline parsing +uses character-by-character scanning with delimiter-run tracking for emphasis +resolution (matching opener/closer pairs at block end). + +### Construct mapping + +| Construct | ContentState output | +| -------------------------------------- | ------------------------------------------------------------------------------------- | +| Paragraph (blank-line separated) | `unstyled` block | +| ATX headings (`#` through `######`) | `header-one` through `header-six` | +| `>` blockquote | `blockquote` block, `> ` prefix stripped per line | +| Fenced code (` ``` ` or `~~~`) | `code-block` block | +| `---` / `***` / `___` thematic break | `atomic` block with `HORIZONTAL_RULE` entity | +| `* `/`- `/`+ ` unordered lists | `unordered-list-item` blocks with depth | +| `1. ` ordered lists | `ordered-list-item` blocks with depth | +| `**bold**` / `__bold__` | `BOLD` inline style range | +| `*italic*` / `_italic_` | `ITALIC` inline style range | +| `` `code` `` | `CODE` inline style range | +| `[text](url)` | `LINK` entity — data from link resolver chain (default: `{"url": url}`) | +| `` | `IMAGE` entity — data from image resolver chain (default: `{"src": url, "alt": alt}`) | +| Hard line break (two spaces + newline) | `\n` in block text | +| `text` (if configured) | `SUPERSCRIPT` inline style range | +| `text` (if configured) | `SUBSCRIPT` inline style range | + +### Safety guarantee — structural integrity + +Every code path produces either a valid ContentState or raises +`MarkdownParseError`. Invariants: entity keys are monotonically increasing, +block keys are unique, depth values match list nesting, entity ranges always +have corresponding `entityMap` entries, inline style ranges are non-overlapping +and in-bounds. + +### Error handling + +Strict parsing — unexpected input raises `MarkdownParseError` with line number +and message. No silent degradation. + +### Entity resolution + +When the parser encounters a link or image, it does not create the entity +directly. Instead it passes the URL (and the label or alt text) through a +resolver chain. Each resolver either returns a resolution or `None` to +defer to the next resolver. If no resolver matches, the default applies. + +```python +class EntityResolution(TypedDict, total=False): + type: str # entity type, e.g. "LINK", "IMAGE", "DOCUMENT", "EMBED" + data: dict[str, Any] # entity data + mutability: Mutability # default "MUTABLE" + + +# Receives (url, label_or_alt). Returns a resolution, or None to defer. +EntityResolver = Callable[[str, str], EntityResolution | None] +``` + +- **Link resolvers** receive `(url, link_text)`. The link text always becomes + the block's children; resolvers only control the entity type and data. +- **Image resolvers** receive `(url, alt_text)`. Resolvers control the entity + type and data, including how the alt text is incorporated. +- Resolvers can return any entity type — e.g., a `wagtail://document?id=1` + link URL resolves to a `DOCUMENT` entity, not `LINK`. + +**Defaults** (empty resolver chains): + +- Links: `{"type": "LINK", "data": {"url": url}}` +- Images: `{"type": "IMAGE", "data": {"src": url, "alt": alt}}` + +**Shipped helper — `scheme_resolver`:** the library provides a generic +resolver factory for scheme-based internal URLs, in +`draftjs_exporter.markdown_parser.resolvers`: + +```python +def scheme_resolver( + scheme: str, # e.g. "wagtail" + type_map: dict[str, str], # URL host → entity type + # e.g. {"image": "IMAGE", "page": "LINK"} + coerce: dict[str, Callable[[str], Any]] | None = None, + # per-key data coercion, e.g. {"id": int} + label_key: str | None = None, # data key filled from the Markdown label + # e.g. "alt" for images; None for links +) -> EntityResolver: ... +``` + +It parses URLs of the form `scheme://kind?key=value&...`: the host selects the +entity type via `type_map`, query parameters become entity data (with optional +coercion), and — when `label_key` is set — a non-empty Markdown label fills +that data key if the query string didn't already provide it. Returns `None` +for URLs that don't match the scheme or have an unmapped host. + +**Example — Wagtail-style internal syntax:** + +```python +config: ImporterConfig = { + "parser_config": { + "link_resolvers": [ + scheme_resolver( + "wagtail", + {"page": "LINK", "document": "DOCUMENT", "user": "LINK"}, + coerce={"id": int}, + ), + ], + "image_resolvers": [ + scheme_resolver( + "wagtail", + {"image": "IMAGE", "media": "EMBED"}, + coerce={"id": int}, + label_key="alt", + ), + ], + "inline_html_styles": { + "sup": INLINE_STYLES.SUPERSCRIPT, + "sub": INLINE_STYLES.SUBSCRIPT, + }, + }, +} + +# [label](wagtail://page?id=3) +# → LINK entity, data {"id": 3} +#  +# → IMAGE entity, data {"id": 10, "alt": "alt", "format": "left"} +#  (no resolver match → default) +# → IMAGE entity, data {"src": "/media/...jpg", "alt": "alt"} +``` + +### Limited inline HTML + +The inline parser recognizes paired tags `content ` where `tag` is +a key of `inline_html_styles`, and wraps the content in the corresponding +inline style range. Content is parsed recursively, so `**bold**` +produces overlapping SUPERSCRIPT and BOLD ranges. + +- Tags with attributes (``) are not recognized and pass + through as literal text. +- Tags not in the whitelist pass through as literal text (`<` and `>` + characters are preserved in the block text). +- No block-level HTML, no void elements, no arbitrary HTML — the whitelist + only maps paired inline tags to inline styles. +- Default is an empty whitelist: no HTML is interpreted. + +## ContentStateFilter — Phase 2 + +### Interface + +```python +# Callable receives the matched object depending on rule type: +# - block rules: the block dict (Block) +# - entity rules: the entity dict (Entity) +# - inline_style rules: the style name string +# Returns a replacement object of the same shape, or None to remove. +FilterCallback = Callable[[Block | Entity | str], Block | Entity | str | None] + +FilterAction = Literal["remove", "keep", "demote"] | FilterCallback + + +class FilterRule(TypedDict, total=False): + type: Literal["block", "inline_style", "entity"] + match: str # type to match (e.g., "header-one") + action: FilterAction + + +class ContentStateFilter: + __slots__ = ("rules",) + + def __init__(self, rules: list[FilterRule] | None = None) -> None: ... + def apply(self, content_state: ContentState) -> ContentState: ... +``` + +### Built-in actions + +| Action | Effect | +| ---------- | -------------------------------------------------------------------- | +| `"remove"` | Delete matching block/entity/style (block content is lost) | +| `"keep"` | No-op — useful as explicit default | +| `"demote"` | Headings only: H1→H2, …, H6→removed | +| Callable | Receives the matched object, returns replacement or `None` to remove | + +### Processing model + +Walk blocks in order. For each block: check block-type rules, strip disallowed +inline styles from `inlineStyleRanges`, resolve entity rules against +`entityRanges` (sync with `entityMap`, prune orphaned keys). When a block is +removed, its inline styles and entities are discarded with it — no cascading +cleanup is needed on sibling blocks. + +After list-item removals, recalculate depths: the depth of each remaining +list item is determined by counting how many consecutive list-wrapper openings +are pending at that position, maintaining valid Draft.js nesting. + +Rules run in definition order. Default is keep (not deny). + +### Safety guarantee — content policy + +Rules are declarative. Callable returns are validated before insertion. +The filter never produces invalid ContentState. + +## MarkdownImporter — Public API + +### Interface + +```python +class ImporterConfig(TypedDict, total=False): + parser: str # dotted path, default: built-in + parser_config: ParserConfig # options passed to parser + filter_rules: list[FilterRule] # rules applied after parsing + + +class MarkdownImporter: + __slots__ = ("parser", "filter") + + def __init__(self, config: ImporterConfig | None = None) -> None: ... + def import_markdown(self, markdown: str) -> ContentState: ... +``` + +### Usage examples + +```python +from draftjs_exporter import BLOCK_TYPES, MarkdownImporter + +# Simple import +importer = MarkdownImporter() +cs = importer.import_markdown("# Hello\n\nWorld") + +# Demote level-1 headings +importer = MarkdownImporter({ + "filter_rules": [ + {"type": "block", "match": BLOCK_TYPES.HEADER_ONE, "action": "demote"}, + ], +}) + +# Disable blockquotes +importer = MarkdownImporter({ + "parser_config": {"blockquote": False}, +}) +``` + +### Error surface + +```python +class MarkdownParseError(Exception): + """Raised when Markdown input cannot be parsed.""" + line: int | None + message: str +``` + +### Public API additions to `draftjs_exporter.__init__` + +New exports: `MarkdownImporter`, `MarkdownParser`, `ContentStateFilter`, +`ImporterConfig`, `ParserConfig`, `FilterRule`, `MarkdownParseError`, +`EntityResolution`, `EntityResolver`, `scheme_resolver`. + +## Testing strategy + +### Unit tests + +- `tests/markdown_parser/test_parser.py` — each construct in isolation, edge cases +- `tests/markdown_parser/test_inline_parser.py` — emphasis, links, images, escapes +- `tests/markdown_parser/test_parser_config.py` — config toggles disable constructs +- `tests/markdown_parser/test_parser_errors.py` — malformed input → `MarkdownParseError` +- `tests/markdown_parser/test_resolvers.py` — resolver chain ordering, defaults, + custom resolvers returning non-default entity types (e.g., DOCUMENT, EMBED) +- `tests/markdown_parser/test_scheme_resolver.py` — the shipped helper: scheme + matching, host→type mapping, query param extraction, coercion, alt fallback +- `tests/markdown_parser/test_inline_html.py` — whitelisted tags → styles, + recursive content, tags with attributes ignored, non-whitelisted tags literal +- `tests/contentstate_filter/test_filter.py` — each action type per target kind +- `tests/contentstate_filter/test_filter_rules.py` — ordering, validation, defaults +- `tests/markdown_importer/test_importer.py` — wiring, config defaults, error propagation + +### Snapshot tests + +New `tests/test_imports.py`, following the `test_exports.py` pattern: + +- **Round-trip over existing fixtures** (`tests/test_exports.json`): import each + fixture's recorded `output.markdown` and compare structurally with the + fixture's `content_state`. Most fixtures round-trip identically. The fixture + format gains an optional `"import"` field: when present, it records the + expected post-import ContentState for fixtures with known information loss + (e.g., "Style map defaults" — single-tilde strikethrough is GFM, out of + importer scope) instead of maintaining a skip-list in test code. +- **Direct import fixtures** (new `tests/test_imports.json`): Markdown → + ContentState cases for what existing fixtures cannot cover — syntax variants + the exporter never emits (`*italic*`, `+` bullets, `~~~` fences, `***` + thematic breaks, `1)` ordered lists), and importer-only features + (`inline_html_styles` whitelists, internal-URL resolvers, parser toggles). +- The "Style map defaults" fixture doubles as an integration test for + `inline_html_styles`: with a full whitelist configured, only the + strikethrough range is lost on round-trip. +- Extend `tests/test_exports.json` with Markdown → ContentState entries. + +### Property-based tests + +- Extend `tests/test_properties.py`: + - Round-trip: `import(export(cs))` preserves block types and entity data + - Filter idempotency: `filter(filter(cs)) == filter(cs)` + - Filter validity: for any valid ContentState, `filter(cs)` is also valid + +### Coverage target + +100% per project convention. + +## Example and documentation + +### Example script + +Update `example.py` to demonstrate importing Markdown and re-exporting it. +The updated script shows both directions: Markdown → ContentState → HTML, +and ContentState → Markdown → ContentState. + +### Documentation + +New docs page at `docs/markdown-importer.md` covering: + +- Getting started with `MarkdownImporter` +- Parser config reference (all toggles) +- Entity resolution: resolver chains, defaults, `scheme_resolver` helper, + writing custom resolvers, internal-URL syntax example (e.g., `wagtail://`) +- Limited inline HTML: `inline_html_styles` whitelist, safety model +- Filter rule reference (all actions, rule types) +- Safety guarantees +- Custom parser engines (swapping the parser) + +Update `docs/index.md` to link to the new page. Update +`.agents/skills/draftjs-exporter/SKILL.md` to include the new public API. + +## Out of scope + +- Reference-style links (`[text][ref]`) and link reference definitions +- Block-level HTML and arbitrary inline HTML — unrecognized HTML always + passes through as literal text; only whitelisted paired inline tags + (via `inline_html_styles`) are interpreted +- HTML tags with attributes (pass through as literal text) +- Setext headings (`===`, `---`) +- Indented code blocks (4-space indent) +- Tables (GFM extension) +- Autolinks (``) +- Nested emphasis edge cases (middle-of-word matching) +- Empty list items diff --git a/draftjs_exporter/__init__.py b/draftjs_exporter/__init__.py index 34e0485..922713d 100644 --- a/draftjs_exporter/__init__.py +++ b/draftjs_exporter/__init__.py @@ -15,14 +15,26 @@ from draftjs_exporter.constants import BLOCK_TYPES as BLOCK_TYPES from draftjs_exporter.constants import ENTITY_TYPES as ENTITY_TYPES from draftjs_exporter.constants import INLINE_STYLES as INLINE_STYLES +from draftjs_exporter.contentstate_filter import ( + ContentStateFilter as ContentStateFilter, +) +from draftjs_exporter.contentstate_filter import FilterRule as FilterRule from draftjs_exporter.defaults import BLOCK_MAP as BLOCK_MAP from draftjs_exporter.defaults import STYLE_MAP as STYLE_MAP from draftjs_exporter.dom import DOM as DOM +from draftjs_exporter.error import MarkdownParseError as MarkdownParseError from draftjs_exporter.html import HTML as HTML from draftjs_exporter.html import ExporterConfig as ExporterConfig from draftjs_exporter.markdown import CONFIG as MARKDOWN_CONFIG from draftjs_exporter.markdown import MarkdownOptions as MarkdownOptions from draftjs_exporter.markdown import build_markdown_config as build_markdown_config +from draftjs_exporter.markdown_importer import ImporterConfig as ImporterConfig +from draftjs_exporter.markdown_importer import MarkdownImporter as MarkdownImporter +from draftjs_exporter.markdown_parser import EntityResolution as EntityResolution +from draftjs_exporter.markdown_parser import EntityResolver as EntityResolver +from draftjs_exporter.markdown_parser import MarkdownParser as MarkdownParser +from draftjs_exporter.markdown_parser import ParserConfig as ParserConfig +from draftjs_exporter.markdown_parser import scheme_resolver as scheme_resolver from draftjs_exporter.types import Block as Block from draftjs_exporter.types import Component as Component from draftjs_exporter.types import CompositeDecorators as CompositeDecorators @@ -60,6 +72,17 @@ "MARKDOWN_CONFIG", "MarkdownOptions", "build_markdown_config", + # Importer + "MarkdownImporter", + "ImporterConfig", + "MarkdownParser", + "ParserConfig", + "ContentStateFilter", + "FilterRule", + "MarkdownParseError", + "EntityResolution", + "EntityResolver", + "scheme_resolver", # Constants "BLOCK_TYPES", "ENTITY_TYPES", diff --git a/draftjs_exporter/contentstate_filter/__init__.py b/draftjs_exporter/contentstate_filter/__init__.py new file mode 100644 index 0000000..fb5ef68 --- /dev/null +++ b/draftjs_exporter/contentstate_filter/__init__.py @@ -0,0 +1,267 @@ +"""Filter Draft.js ContentState with declarative rules.""" + +import copy +from collections.abc import Callable +from typing import Any, Literal, TypeAlias, TypedDict + +from draftjs_exporter.constants import BLOCK_TYPES +from draftjs_exporter.error import ConfigException +from draftjs_exporter.types import Block, ContentState, Entity, InlineStyleRange + +FilterCallback: TypeAlias = Callable[[Any], Any] +"""Custom rule action: receives the matched object, returns a replacement or None.""" + +FilterAction: TypeAlias = Literal["remove", "keep", "demote"] | FilterCallback +"""Predefined action name or custom callback.""" + + +class FilterRule(TypedDict): + """A single filtering rule.""" + + type: Literal["block", "inline_style", "entity"] + """Which kind of object the rule matches.""" + + match: str + """The block type, style name, or entity type to match.""" + + action: FilterAction + """What to do with matching objects.""" + + +HEADER_DEMOTION = { + BLOCK_TYPES.HEADER_ONE: BLOCK_TYPES.HEADER_TWO, + BLOCK_TYPES.HEADER_TWO: BLOCK_TYPES.HEADER_THREE, + BLOCK_TYPES.HEADER_THREE: BLOCK_TYPES.HEADER_FOUR, + BLOCK_TYPES.HEADER_FOUR: BLOCK_TYPES.HEADER_FIVE, + BLOCK_TYPES.HEADER_FIVE: BLOCK_TYPES.HEADER_SIX, +} +"""Heading demotion targets. ``header-six`` cannot be demoted.""" + +LIST_BLOCKS = frozenset( + {BLOCK_TYPES.UNORDERED_LIST_ITEM, BLOCK_TYPES.ORDERED_LIST_ITEM} +) +"""Block types whose depth participates in list nesting.""" + +VALID_ACTIONS = frozenset({"remove", "keep", "demote"}) + + +class ContentStateFilter: + """Apply declarative rules to a ContentState. + + Rules run in definition order per object. Objects without a + matching rule are kept. The filter always produces a structurally + valid ContentState: entity ranges and the entity map stay in sync, + and list depths are re-normalized after removals. + """ + + __slots__ = ("rules",) + + def __init__(self, rules: list[FilterRule] | None = None) -> None: + """Initialize the filter, validating rules eagerly. + + Parameters: + rules: The rules to apply, in order. + + Raises: + ConfigException: If a rule is malformed. + """ + self.rules = rules if rules is not None else [] + for rule in self.rules: + self._validate(rule) + + @staticmethod + def _validate(rule: FilterRule) -> None: + """Check a rule for structural validity. + + Parameters: + rule: The rule to validate. + + Raises: + ConfigException: If the rule type or action is invalid. + """ + if rule["type"] not in ("block", "inline_style", "entity"): + raise ConfigException(f"Invalid filter rule type: {rule['type']!r}") + action = rule["action"] + if not callable(action) and action not in VALID_ACTIONS: + raise ConfigException(f"Invalid filter rule action: {action!r}") + if action == "demote" and ( + rule["type"] != "block" or rule["match"] not in HEADER_DEMOTION + ): + raise ConfigException( + '"demote" only applies to header-one through header-five blocks' + ) + + def apply(self, content_state: ContentState) -> ContentState: + """Apply all rules, returning a new ContentState. + + Parameters: + content_state: The ContentState to filter. Not mutated. + + Returns: + The filtered ContentState. + """ + entity_map_in = content_state.get("entityMap", {}) + block_rules = self._rules_by_match("block") + style_rules = self._rules_by_match("inline_style") + entity_rules = self._rules_by_match("entity") + + out_blocks: list[Block] = [] + replacements: dict[str, Entity] = {} + used_keys: set[str] = set() + + for block in content_state.get("blocks", []): + kept = self._apply_block_rule(copy.deepcopy(block), block_rules) + if kept is None: + continue + self._apply_style_rules(kept, style_rules) + self._apply_entity_rules( + kept, entity_map_in, entity_rules, replacements, used_keys + ) + out_blocks.append(kept) + + self._normalize_depths(out_blocks) + + entity_map_out = {} + for key in used_keys: + entity = replacements.get(key, entity_map_in[key]) + entity_map_out[key] = entity + return {"blocks": out_blocks, "entityMap": entity_map_out} + + def _rules_by_match(self, kind: str) -> dict[str, list[FilterAction]]: + """Group actions for a rule kind by match value.""" + grouped: dict[str, list[FilterAction]] = {} + for rule in self.rules: + if rule["type"] == kind: + grouped.setdefault(rule["match"], []).append(rule["action"]) + return grouped + + @staticmethod + def _run_actions(value: Any, actions: list[FilterAction], kind: str) -> Any: + """Run a chain of actions over a matched object. + + Parameters: + value: The matched object. + actions: The actions to run in order. + kind: Rule kind, for error messages. + + Returns: + The transformed object, or None when removed. + + Raises: + ConfigException: If a callback returns an invalid value. + """ + current = value + for action in actions: + if current is None: + break + if action == "keep": + continue + if action == "remove": + current = None + continue + if action == "demote": + if not isinstance(current, dict) or current.get("type") not in ( + HEADER_DEMOTION + ): + raise ConfigException( + "Filter callback must return a demotable header block" + ) + current = {**current, "type": HEADER_DEMOTION[current["type"]]} + continue + current = action(current) + valid = ( + current is None + or (kind == "inline_style" and isinstance(current, str)) + or (kind != "inline_style" and isinstance(current, dict)) + ) + if not valid: + raise ConfigException( + f"Filter callback for {kind} rule returned invalid value" + ) + return current + + def _apply_block_rule( + self, block: Block, rules: dict[str, list[FilterAction]] + ) -> Block | None: + """Apply block rules to a single block.""" + actions = rules.get(block.get("type", ""), []) + if not actions: + return block + result = self._run_actions(block, actions, "block") + if result is not None and "type" not in result: + raise ConfigException("Filter block callback must return a block") + return result + + def _apply_style_rules( + self, block: Block, rules: dict[str, list[FilterAction]] + ) -> None: + """Apply inline style rules to a block's style ranges.""" + ranges = block.get("inlineStyleRanges", []) + if not ranges or not rules: + return + kept: list[InlineStyleRange] = [] + for style_range in ranges: + actions = rules.get(style_range["style"], []) + if not actions: + kept.append(style_range) + continue + result = self._run_actions(style_range["style"], actions, "inline_style") + if result is not None: + kept.append( + { + "offset": style_range["offset"], + "length": style_range["length"], + "style": result, + } + ) + block["inlineStyleRanges"] = kept + + def _apply_entity_rules( + self, + block: Block, + entity_map: dict[str, Entity], + rules: dict[str, list[FilterAction]], + replacements: dict[str, Entity], + used_keys: set[str], + ) -> None: + """Apply entity rules to a block's entity ranges.""" + kept = [] + for entity_range in block.get("entityRanges", []): + key = str(entity_range["key"]) + entity = entity_map.get(key) + if entity is None: + # Orphaned range: drop to keep output valid. + continue + actions = rules.get(entity.get("type", ""), []) + if not actions: + kept.append(entity_range) + used_keys.add(key) + continue + result = self._run_actions(copy.deepcopy(entity), actions, "entity") + if result is not None: + if "type" not in result: + raise ConfigException( + "Filter entity callback must return an entity" + ) + kept.append(entity_range) + used_keys.add(key) + replacements[key] = result + block["entityRanges"] = kept + + @staticmethod + def _normalize_depths(blocks: list[Block]) -> None: + """Clamp list depths so nesting never skips a level. + + Parameters: + blocks: Blocks to normalize in place. + """ + last_list_depth = -1 + for block in blocks: + if block.get("type") in LIST_BLOCKS: + depth = block.get("depth", 0) + if depth > last_list_depth + 1: + depth = last_list_depth + 1 + block["depth"] = depth + last_list_depth = depth + else: + last_list_depth = -1 diff --git a/draftjs_exporter/error.py b/draftjs_exporter/error.py index 74d78b7..fba1ad4 100644 --- a/draftjs_exporter/error.py +++ b/draftjs_exporter/error.py @@ -7,3 +7,24 @@ class ExporterException(Exception): class ConfigException(ExporterException): """Raised when the exporter configuration is invalid or unsupported.""" + + +class MarkdownParseError(ExporterException): + """Raised when Markdown input cannot be parsed. + + Carries an optional 1-based line number pointing at the source of + the failure. + """ + + __slots__ = ("line", "message") + + def __init__(self, message: str, line: int | None = None) -> None: + """Initialize the error with a message and optional line number. + + Parameters: + message: Human-readable description of the failure. + line: 1-based source line number, if known. + """ + self.message = message + self.line = line + super().__init__(f"line {line}: {message}" if line is not None else message) diff --git a/draftjs_exporter/markdown_importer/__init__.py b/draftjs_exporter/markdown_importer/__init__.py new file mode 100644 index 0000000..b9a3cf3 --- /dev/null +++ b/draftjs_exporter/markdown_importer/__init__.py @@ -0,0 +1,61 @@ +"""Public Markdown importer: parses Markdown, then filters the ContentState.""" + +from typing import TypedDict + +from draftjs_exporter.contentstate_filter import ContentStateFilter, FilterRule +from draftjs_exporter.markdown_parser import MarkdownParser, ParserConfig +from draftjs_exporter.types import ContentState +from draftjs_exporter.utils.module_loading import import_string + +DEFAULT_PARSER = "draftjs_exporter.markdown_parser.MarkdownParser" +"""Dotted path of the built-in parser engine.""" + + +class ImporterConfig(TypedDict, total=False): + """Configuration for the Markdown importer.""" + + parser: str + """Dotted path of the parser engine class. Defaults to the built-in parser.""" + + parser_config: ParserConfig + """Options passed to the parser engine constructor.""" + + filter_rules: list[FilterRule] + """Rules applied to the parsed ContentState.""" + + +class MarkdownImporter: + """Import Markdown text as a Draft.js ContentState. + + Combines a parser engine (Markdown to ContentState) with a filter + (content policy on the result). The parser is referenced by dotted + path so alternative engines can be swapped in. + """ + + __slots__ = ("parser", "filter") + + def __init__(self, config: ImporterConfig | None = None) -> None: + """Initialize the importer with the given configuration. + + Parameters: + config: Parser engine, parser options, and filter rules. + """ + if config is None: + config = {} + parser_class = import_string(config.get("parser", DEFAULT_PARSER)) + self.parser: MarkdownParser = parser_class(config.get("parser_config")) + self.filter: ContentStateFilter = ContentStateFilter(config.get("filter_rules")) + + def import_markdown(self, markdown: str) -> ContentState: + """Parse Markdown and apply filter rules. + + Parameters: + markdown: The Markdown text to import. + + Returns: + The parsed, filtered ContentState. + + Raises: + MarkdownParseError: If the input cannot be parsed. + """ + return self.filter.apply(self.parser.parse(markdown)) diff --git a/draftjs_exporter/markdown_parser/__init__.py b/draftjs_exporter/markdown_parser/__init__.py new file mode 100644 index 0000000..874842c --- /dev/null +++ b/draftjs_exporter/markdown_parser/__init__.py @@ -0,0 +1,129 @@ +"""Markdown parsing engine: converts CommonMark core to Draft.js ContentState.""" + +from typing import TypedDict + +from draftjs_exporter.markdown_parser.blocks import BlockParser +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from draftjs_exporter.markdown_parser.inline import InlineParser +from draftjs_exporter.markdown_parser.resolvers import ( + EntityResolution as EntityResolution, +) +from draftjs_exporter.markdown_parser.resolvers import ( + EntityResolver as EntityResolver, +) +from draftjs_exporter.markdown_parser.resolvers import ( + scheme_resolver as scheme_resolver, +) +from draftjs_exporter.types import ContentState + + +class ParserConfig(TypedDict, total=False): + """Options controlling which Markdown constructs are recognized.""" + + headings: bool + """Parse ATX headings (default: True).""" + + blockquote: bool + """Parse ``>`` blockquotes (default: True).""" + + code_fenced: bool + """Parse fenced code blocks (default: True).""" + + thematic_break: bool + """Parse thematic breaks (default: True).""" + + unordered_list: bool + """Parse unordered lists (default: True).""" + + ordered_list: bool + """Parse ordered lists (default: True).""" + + emphasis: bool + """Parse bold and italic delimiters (default: True).""" + + code_inline: bool + """Parse backtick code spans (default: True).""" + + links: bool + """Parse ``[label](url)`` links (default: True).""" + + images: bool + """Parse ```` images (default: True).""" + + line_breaks: bool + """Strip two-space hard break markers (default: True).""" + + link_resolvers: list[EntityResolver] + """Resolver chain for link URLs (default: empty, uses ``LINK`` with the URL).""" + + image_resolvers: list[EntityResolver] + """Resolver chain for image URLs (default: empty, uses ``IMAGE`` with ``src``/``alt``).""" + + inline_html_styles: dict[str, str] + """Whitelist of HTML tags mapped to inline styles, e.g. ``{"sup": "SUPERSCRIPT"}``.""" + + +class MarkdownParser: + """Parse Markdown text into a Draft.js ContentState. + + Supports the CommonMark core: paragraphs, ATX headings, blockquotes, + fenced code, thematic breaks, lists, emphasis, code spans, links, + images, and hard line breaks. Every input produces either a + structurally valid ContentState or a ``MarkdownParseError``. + """ + + __slots__ = ("config",) + + config: ParserConfig + + def __init__(self, config: ParserConfig | None = None) -> None: + """Initialize the parser with the given configuration. + + Parameters: + config: Feature toggles and entity resolvers. Missing keys + use defaults that enable all constructs. + """ + self.config = config if config is not None else ParserConfig() + + def parse(self, markdown: str) -> ContentState: + """Parse Markdown source into a ContentState. + + Parameters: + markdown: The Markdown text to parse. + + Returns: + A structurally valid Draft.js ContentState. + + Raises: + TypeError: If ``markdown`` is not a string. + MarkdownParseError: If an entity resolver fails. + """ + if not isinstance(markdown, str): + raise TypeError(f"Expected str, got {type(markdown).__name__}") + text = markdown.replace("\r\n", "\n").replace("\r", "\n") + builder = ContentStateBuilder() + images = self.config.get("images", True) + inline = InlineParser( + emphasis=self.config.get("emphasis", True), + code_inline=self.config.get("code_inline", True), + links=self.config.get("links", True), + images=images, + line_breaks=self.config.get("line_breaks", True), + inline_html_styles=self.config.get("inline_html_styles", {}), + link_resolvers=self.config.get("link_resolvers", []), + image_resolvers=self.config.get("image_resolvers", []), + builder=builder, + ) + blocks = BlockParser( + headings=self.config.get("headings", True), + blockquote=self.config.get("blockquote", True), + code_fenced=self.config.get("code_fenced", True), + thematic_break=self.config.get("thematic_break", True), + unordered_list=self.config.get("unordered_list", True), + ordered_list=self.config.get("ordered_list", True), + images=images, + inline=inline, + builder=builder, + ) + blocks.parse(text) + return builder.build() diff --git a/draftjs_exporter/markdown_parser/blocks.py b/draftjs_exporter/markdown_parser/blocks.py new file mode 100644 index 0000000..d0ca7e7 --- /dev/null +++ b/draftjs_exporter/markdown_parser/blocks.py @@ -0,0 +1,346 @@ +"""Block-level Markdown parsing: lines to Draft.js blocks.""" + +import re +from typing import Any + +from draftjs_exporter.constants import BLOCK_TYPES, ENTITY_TYPES +from draftjs_exporter.error import MarkdownParseError +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from draftjs_exporter.markdown_parser.inline import InlineParser +from draftjs_exporter.types import Mutability + +ATX_RE = re.compile(r"^(#{1,6})[ \t]+(.*?)[ \t]*#*[ \t]*$") +"""ATX heading with optional closing hash sequence.""" + +HR_RE = re.compile(r"^[ \t]*((\*[ \t]*){3,}|(-[ \t]*){3,}|(_[ \t]*){3,})$") +"""Thematic break: 3+ of the same ``*``, ``-``, or ``_`` marker.""" + +FENCE_RE = re.compile(r"^[ \t]*(```+|~~~+)(.*)$") +"""Fenced code block opener or closer.""" + +QUOTE_RE = re.compile(r"^[ \t]*>[ \t]?(.*)$") +"""Blockquote line, capturing the content after the marker.""" + +ULIST_RE = re.compile(r"^([ \t]*)[-*+][ \t]+(.*)$") +"""Unordered list item, capturing indent and content.""" + +OLIST_RE = re.compile(r"^([ \t]*)\d{1,9}[.)][ \t]+(.*)$") +"""Ordered list item, capturing indent and content.""" + +STANDALONE_IMAGE_RE = re.compile(r"^!\[(.*)\]\((.*)\)$") +"""A paragraph consisting of exactly one image.""" + +HEADING_TYPES = [ + BLOCK_TYPES.HEADER_ONE, + BLOCK_TYPES.HEADER_TWO, + BLOCK_TYPES.HEADER_THREE, + BLOCK_TYPES.HEADER_FOUR, + BLOCK_TYPES.HEADER_FIVE, + BLOCK_TYPES.HEADER_SIX, +] +"""Heading block types indexed by ATX level minus one.""" + + +def _indent_width(text: str, tab_size: int = 4) -> int: + """Compute the column width of indentation, expanding tabs to tab stops. + + Per CommonMark, a tab advances the column position to the next + multiple of 4 rather than counting as a fixed number of spaces. + + Parameters: + text: The leading whitespace of a line. + tab_size: The tab stop interval. + + Returns: + The column position after the indentation. + """ + col = 0 + for ch in text: + if ch == "\t": + col += tab_size - (col % tab_size) + else: + col += 1 + return col + + +class BlockParser: + """Parse Markdown line by line into Draft.js blocks. + + The parser tracks block constructs that span lines (lists, + blockquotes, fenced code) and delegates inline content to the + inline parser. It emits blocks directly onto the builder — there + is no intermediate AST. + """ + + __slots__ = ( + "headings", + "blockquote", + "code_fenced", + "thematic_break", + "unordered_list", + "ordered_list", + "images", + "inline", + "builder", + ) + + def __init__( + self, + *, + headings: bool, + blockquote: bool, + code_fenced: bool, + thematic_break: bool, + unordered_list: bool, + ordered_list: bool, + images: bool, + inline: InlineParser, + builder: ContentStateBuilder, + ) -> None: + """Initialize the parser with feature toggles and helpers. + + Parameters: + headings: Parse ATX headings. + blockquote: Parse ``>`` blockquotes. + code_fenced: Parse fenced code blocks. + thematic_break: Parse thematic breaks. + unordered_list: Parse unordered lists. + ordered_list: Parse ordered lists. + images: Convert standalone images to atomic blocks. + inline: The inline parser for block content. + builder: The builder blocks are appended to. + """ + self.headings = headings + self.blockquote = blockquote + self.code_fenced = code_fenced + self.thematic_break = thematic_break + self.unordered_list = unordered_list + self.ordered_list = ordered_list + self.images = images + self.inline = inline + self.builder = builder + + def parse(self, text: str) -> None: + r"""Parse Markdown source, appending blocks to the builder. + + Parameters: + text: Markdown with line endings normalized to ``\n``. + """ + lines = text.split("\n") + i = 0 + n = len(lines) + while i < n: + line = lines[i] + if not line.strip(): + i += 1 + continue + if self.code_fenced and (match := FENCE_RE.match(line)): + i = self._parse_fence(lines, i, match) + continue + if self.headings and (match := ATX_RE.match(line)): + self._add_text_block( + HEADING_TYPES[len(match.group(1)) - 1], match.group(2), i + ) + i += 1 + continue + if self.thematic_break and HR_RE.match(line): + self._add_atomic(ENTITY_TYPES.HORIZONTAL_RULE, {}, "IMMUTABLE") + i += 1 + continue + if self.blockquote and QUOTE_RE.match(line): + i = self._parse_quote(lines, i) + continue + if self._is_list_item(line): + i = self._parse_list(lines, i) + continue + i = self._parse_paragraph(lines, i) + + def _add_text_block( + self, type_: str, source: str, line_index: int, depth: int = 0 + ) -> None: + """Inline-parse source text and append a block. + + Parameters: + type_: The Draft.js block type. + source: The Markdown source of the block's content. + line_index: 0-based source line, for error reporting. + depth: Nesting depth for list items. + """ + try: + text, styles, entities = self.inline.parse(source) + except MarkdownParseError as err: + if err.line is None: + raise MarkdownParseError(err.message, line=line_index + 1) from err + raise + self.builder.add_block( + type_, + text, + depth=depth, + inline_style_ranges=styles, + entity_ranges=entities, + ) + + def _add_atomic( + self, entity_type: str, data: dict[str, Any], mutability: Mutability + ) -> None: + """Append an atomic block carrying a single entity. + + Parameters: + entity_type: The entity type. + data: The entity data. + mutability: The entity mutability. + """ + key = self.builder.add_entity(entity_type, data, mutability) + self.builder.add_block( + BLOCK_TYPES.ATOMIC, + " ", + entity_ranges=[{"offset": 0, "length": 1, "key": key}], + ) + + def _is_list_item(self, line: str) -> bool: + """Return whether the line starts an enabled list item.""" + return bool( + (self.unordered_list and ULIST_RE.match(line)) + or (self.ordered_list and OLIST_RE.match(line)) + ) + + def _starts_block(self, line: str) -> bool: + """Return whether the line starts a block construct.""" + return bool( + (self.code_fenced and FENCE_RE.match(line)) + or (self.headings and ATX_RE.match(line)) + or (self.thematic_break and HR_RE.match(line)) + or (self.blockquote and QUOTE_RE.match(line)) + or self._is_list_item(line) + ) + + def _parse_paragraph(self, lines: list[str], i: int) -> int: + """Parse consecutive plain lines into one paragraph block.""" + start = i + collected = [lines[i]] + i += 1 + while i < len(lines) and lines[i].strip() and not self._starts_block(lines[i]): + collected.append(lines[i]) + i += 1 + source = "\n".join(collected) + image = STANDALONE_IMAGE_RE.match(source.strip()) if self.images else None + if image is not None: + key = self.inline.resolve_image_entity(image.group(2), image.group(1)) + self.builder.add_block( + BLOCK_TYPES.ATOMIC, + " ", + entity_ranges=[{"offset": 0, "length": 1, "key": key}], + ) + else: + self._add_text_block(BLOCK_TYPES.UNSTYLED, source, start) + return i + + def _parse_fence(self, lines: list[str], i: int, match: re.Match[str]) -> int: + """Parse a fenced code block starting at line i. + + Unclosed fences parse to end of input, per CommonMark. + + Parameters: + lines: All source lines. + i: Index of the opening fence line. + match: The fence opener match. + + Returns: + The index of the first line after the block. + """ + fence = match.group(1) + marker = fence[0] + size = len(fence) + body: list[str] = [] + i += 1 + while i < len(lines): + close = FENCE_RE.match(lines[i]) + if ( + close is not None + and close.group(1)[0] == marker + and len(close.group(1)) >= size + ): + i += 1 + break + body.append(lines[i]) + i += 1 + self.builder.add_block(BLOCK_TYPES.CODE, "\n".join(body)) + return i + + def _parse_quote(self, lines: list[str], i: int) -> int: + """Parse consecutive blockquote lines into blockquote blocks. + + Quoted lines join with newlines. A quoted line with no content + (``>`` alone) splits the quote into separate blocks. + + Parameters: + lines: All source lines. + i: Index of the first quoted line. + + Returns: + The index of the first line after the quote. + """ + start = i + quote_lines: list[str] = [] + while i < len(lines): + match = QUOTE_RE.match(lines[i]) + if match is None: + break + quote_lines.append(match.group(1)) + i += 1 + paragraph: list[str] = [] + for offset, content in enumerate(quote_lines): + if not content.strip(): + if paragraph: + self._add_text_block( + BLOCK_TYPES.BLOCKQUOTE, "\n".join(paragraph), start + offset + ) + paragraph = [] + else: + paragraph.append(content) + if paragraph: + self._add_text_block( + BLOCK_TYPES.BLOCKQUOTE, + "\n".join(paragraph), + start + len(quote_lines) - 1, + ) + return i + + def _parse_list(self, lines: list[str], i: int) -> int: + """Parse consecutive list items with indent-based depth tracking. + + Depth derives from a stack of indent widths: deeper indents + push, shallower indents pop. Continuation lines (indented + content without a marker) are not supported — they end the + list and become paragraphs. + + Parameters: + lines: All source lines. + i: Index of the first list item line. + + Returns: + The index of the first line after the list. + """ + stack: list[int] = [] + while i < len(lines): + line = lines[i] + if not line.strip(): + break + unordered = ULIST_RE.match(line) if self.unordered_list else None + ordered = OLIST_RE.match(line) if self.ordered_list else None + match = unordered or ordered + if match is None: + break + indent = _indent_width(match.group(1)) + while stack and indent < stack[-1]: + stack.pop() + if not stack or indent > stack[-1]: + stack.append(indent) + type_ = ( + BLOCK_TYPES.UNORDERED_LIST_ITEM + if unordered + else BLOCK_TYPES.ORDERED_LIST_ITEM + ) + self._add_text_block(type_, match.group(2), i, depth=len(stack) - 1) + i += 1 + return i diff --git a/draftjs_exporter/markdown_parser/builder.py b/draftjs_exporter/markdown_parser/builder.py new file mode 100644 index 0000000..ccdbc54 --- /dev/null +++ b/draftjs_exporter/markdown_parser/builder.py @@ -0,0 +1,88 @@ +"""Accumulates Draft.js blocks and entities into a ContentState.""" + +from typing import Any + +from draftjs_exporter.types import ( + Block, + ContentState, + Entity, + EntityRange, + InlineStyleRange, + Mutability, +) + + +class ContentStateBuilder: + """Build a ContentState block by block with consistent keys. + + Entity keys are assigned as monotonically increasing integers in + order of first use. Block keys are deterministic, sequential, and + unique within the built ContentState. + """ + + __slots__ = ("blocks", "entity_map", "_block_counter") + + def __init__(self) -> None: + """Initialize an empty builder.""" + self.blocks: list[Block] = [] + self.entity_map: dict[str, Entity] = {} + self._block_counter = 0 + + def add_entity( + self, + type_: str, + data: dict[str, Any], + mutability: Mutability = "MUTABLE", + ) -> int: + """Register an entity and return its integer key. + + Parameters: + type_: The entity type, e.g. ``LINK``. + data: The entity data payload. + mutability: The entity mutability. + + Returns: + The integer key used in entity ranges. + """ + key = len(self.entity_map) + self.entity_map[str(key)] = { + "type": type_, + "mutability": mutability, + "data": data, + } + return key + + def add_block( + self, + type_: str, + text: str = "", + depth: int = 0, + inline_style_ranges: list[InlineStyleRange] | None = None, + entity_ranges: list[EntityRange] | None = None, + ) -> None: + """Append a block to the ContentState. + + Parameters: + type_: The Draft.js block type. + text: The block's plain text. + depth: Nesting depth for list items. + inline_style_ranges: Style ranges over the text. + entity_ranges: Entity ranges over the text. + """ + self.blocks.append( + { + "key": f"{self._block_counter:05d}", + "text": text, + "type": type_, + "depth": depth, + "inlineStyleRanges": ( + inline_style_ranges if inline_style_ranges is not None else [] + ), + "entityRanges": entity_ranges if entity_ranges is not None else [], + } + ) + self._block_counter += 1 + + def build(self) -> ContentState: + """Return the accumulated ContentState.""" + return {"blocks": self.blocks, "entityMap": self.entity_map} diff --git a/draftjs_exporter/markdown_parser/inline.py b/draftjs_exporter/markdown_parser/inline.py new file mode 100644 index 0000000..07df6af --- /dev/null +++ b/draftjs_exporter/markdown_parser/inline.py @@ -0,0 +1,405 @@ +"""Inline Markdown parsing: emphasis, code spans, links, images, inline HTML.""" + +import re +from collections.abc import Callable +from typing import TypeAlias + +from draftjs_exporter.constants import INLINE_STYLES +from draftjs_exporter.error import MarkdownParseError +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from draftjs_exporter.markdown_parser.resolvers import ( + EntityResolution, + EntityResolver, + default_image_resolver, + default_link_resolver, + resolve, +) +from draftjs_exporter.types import EntityRange, InlineStyleRange + +Span: TypeAlias = tuple[int, int, str, "str | int"] +"""An inline annotation: ``(offset, length, kind, payload)``. + +``kind`` is ``"style"`` (payload: style name) or ``"entity"`` (payload: +integer entity key). +""" + +ESCAPABLE = frozenset('\\`*{}_[]<>()#+-.!|"') +"""Punctuation characters that can be backslash-escaped per CommonMark.""" + +TAG_RE = re.compile(r"<([a-zA-Z][a-zA-Z0-9]*)>") +"""Matches an HTML opening tag without attributes.""" + +MAX_INLINE_DEPTH = 100 +"""Maximum nesting depth for inline constructs before rejecting input.""" + + +class InlineParser: + """Parse inline Markdown constructs into text with style/entity ranges. + + The parser is a character-by-character recursive descent scanner. + Delimiter runs (``*``, ``_``) match by exact length: ``**`` only + closes ``**``, not two adjacent ``*`` runs. Intraword emphasis and + other flanking-rule subtleties are intentionally not supported. + """ + + __slots__ = ( + "emphasis", + "code_inline", + "links", + "images", + "line_breaks", + "inline_html_styles", + "link_resolvers", + "image_resolvers", + "builder", + ) + + def __init__( + self, + *, + emphasis: bool, + code_inline: bool, + links: bool, + images: bool, + line_breaks: bool, + inline_html_styles: dict[str, str], + link_resolvers: list[EntityResolver], + image_resolvers: list[EntityResolver], + builder: ContentStateBuilder, + ) -> None: + """Initialize the parser with feature toggles and resolvers. + + Parameters: + emphasis: Parse ``*italic*`` / ``**bold**`` constructs. + code_inline: Parse backtick code spans. + links: Parse ``[label](url)`` links. + images: Parse ```` images. + line_breaks: Strip two-space hard break markers. + inline_html_styles: Whitelist of HTML tags to inline styles. + link_resolvers: Resolver chain for link URLs. + image_resolvers: Resolver chain for image URLs. + builder: The builder entities are registered on. + """ + self.emphasis = emphasis + self.code_inline = code_inline + self.links = links + self.images = images + self.line_breaks = line_breaks + self.inline_html_styles = inline_html_styles + self.link_resolvers = link_resolvers + self.image_resolvers = image_resolvers + self.builder = builder + + def parse(self, text: str) -> tuple[str, list[InlineStyleRange], list[EntityRange]]: + """Convert Markdown inline syntax to plain text plus ranges. + + Parameters: + text: The Markdown source of a single block. + + Returns: + The plain text, its inline style ranges, and its entity ranges. + """ + plain, spans = self._parse(text) + styles: list[InlineStyleRange] = [] + entities: list[EntityRange] = [] + for offset, length, kind, payload in spans: + if kind == "style": + styles.append( + {"offset": offset, "length": length, "style": str(payload)} + ) + else: + entities.append( + {"offset": offset, "length": length, "key": int(payload)} + ) + styles.sort(key=lambda r: r["offset"]) + entities.sort(key=lambda r: r["offset"]) + return plain, styles, entities + + def resolve_image_entity(self, url: str, alt: str) -> int: + """Register an image entity through the resolver chain. + + Parameters: + url: The image URL. + alt: The image alt text. + + Returns: + The integer key of the newly registered entity. + """ + return self._resolve_entity( + url, alt, self.image_resolvers, default_image_resolver + ) + + def _resolve_entity( + self, + url: str, + label: str, + resolvers: list[EntityResolver], + default: Callable[[str, str], EntityResolution], + ) -> int: + """Resolve a URL into an entity and register it on the builder.""" + try: + resolution: EntityResolution = resolve(resolvers, url, label, default) + except MarkdownParseError: + # Propagate without wrapping; unexpected exceptions are wrapped below. + raise + except Exception as err: + raise MarkdownParseError( + f"Entity resolver failed for URL {url!r}: {err}" + ) from err + entity_type = resolution.get("type") + if not isinstance(entity_type, str) or not entity_type: + raise MarkdownParseError( + f"Entity resolver for URL {url!r} must return a 'type'" + ) + data = resolution.get("data", {}) + if not isinstance(data, dict): + raise MarkdownParseError( + f"Entity resolver for URL {url!r} must return dict 'data'" + ) + return self.builder.add_entity( + entity_type, data, resolution.get("mutability", "MUTABLE") + ) + + def _parse(self, text: str, depth: int = 0) -> tuple[str, list[Span]]: + """Scan text, returning output characters and annotation spans. + + Parameters: + text: The text to scan. + depth: Current recursion depth for nested constructs. + + Raises: + MarkdownParseError: If nesting exceeds ``MAX_INLINE_DEPTH``. + """ + if depth > MAX_INLINE_DEPTH: + raise MarkdownParseError("Inline nesting too deep") + out: list[str] = [] + spans: list[Span] = [] + i = 0 + n = len(text) + while i < n: + ch = text[i] + + # Backslash escapes. + if ch == "\\" and i + 1 < n and text[i + 1] in ESCAPABLE: + out.append(text[i + 1]) + i += 2 + continue + + # Code spans. + if ch == "`" and self.code_inline: + end = text.find("`", i + 1) + if end != -1: + content = text[i + 1 : end] + start = len(out) + out.extend(content) + spans.append((start, len(content), "style", INLINE_STYLES.CODE)) + i = end + 1 + continue + + # Images:  + if ch == "!" and i + 1 < n and text[i + 1] == "[": + result = self._link_target(text, i + 1) + if result is not None: + alt, url, end = result + if not self.images: + # Disabled: keep the whole construct literal rather + # than letting the bracket parse as a link. + out.extend(text[i:end]) + i = end + continue + start = len(out) + out.extend(alt) + key = self._resolve_entity( + url, alt, self.image_resolvers, default_image_resolver + ) + spans.append((start, len(alt), "entity", key)) + i = end + continue + + # Links: [label](url) + if self.links and ch == "[": + result = self._link_target(text, i) + if result is not None: + label_src, url, end = result + label_plain, label_spans = self._parse(label_src, depth + 1) + start = len(out) + out.extend(label_plain) + spans.extend( + (s + start, length, kind, payload) + for s, length, kind, payload in label_spans + ) + key = self._resolve_entity( + url, label_plain, self.link_resolvers, default_link_resolver + ) + spans.append((start, len(label_plain), "entity", key)) + i = end + continue + + # Emphasis: * _ ** __ *** ___ + if self.emphasis and ch in "*_": + consumed = self._parse_emphasis(text, i, out, spans, depth) + if consumed is not None: + i = consumed + continue + + # Whitelisted inline HTML tags. + if ch == "<" and self.inline_html_styles: + consumed = self._parse_inline_html(text, i, out, spans, depth) + if consumed is not None: + i = consumed + continue + + # Hard line breaks: strip 2+ trailing spaces before newline. + if ch == "\n" and self.line_breaks: + spaces = 0 + j = len(out) - 1 + while j >= 0 and out[j] == " ": + spaces += 1 + j -= 1 + if spaces >= 2: + del out[j + 1 :] + out.append("\n") + i += 1 + continue + + out.append(ch) + i += 1 + + return "".join(out), spans + + @staticmethod + def _link_target(text: str, i: int) -> tuple[str, str, int] | None: + """Parse ``[label](url)`` starting at the opening bracket. + + Parameters: + text: The full source text. + i: Index of the ``[`` character. + + Returns: + ``(label, url, end_index)`` or None when the construct does + not parse. Labels containing ``](`` and URLs containing + ``)`` are not supported. + """ + close = text.find("](", i) + if close == -1: + return None + paren = text.find(")", close + 2) + if paren == -1: + return None + return text[i + 1 : close], text[close + 2 : paren], paren + 1 + + def _parse_emphasis( + self, text: str, i: int, out: list[str], spans: list[Span], depth: int + ) -> int | None: + """Parse an emphasis delimiter run at index i. + + Delimiter runs match by exact length: a run of 2 only closes a + run of 2. Runs longer than 3 are treated as literal text. + + Parameters: + text: The full source text. + i: Index of the first delimiter character. + out: Output characters accumulated so far. + spans: Spans accumulated so far. + depth: Current recursion depth for nested constructs. + + Returns: + The index after the closing delimiter, or None when the run + does not form emphasis (it is then emitted literally). + """ + ch = text[i] + n = len(text) + run = 1 + while i + run < n and text[i + run] == ch: + run += 1 + if run > 3: + # Runs longer than 3 are not emphasis: emit them literally. + out.extend(ch * run) + return i + run + marker = ch * run + end = self._find_closing(text, i + run, marker) + if end == -1: + return None + inner_plain, inner_spans = self._parse(text[i + run : end], depth + 1) + start = len(out) + out.extend(inner_plain) + spans.extend( + (s + start, length, kind, payload) + for s, length, kind, payload in inner_spans + ) + styles_by_run = { + 1: [INLINE_STYLES.ITALIC], + 2: [INLINE_STYLES.BOLD], + 3: [INLINE_STYLES.BOLD, INLINE_STYLES.ITALIC], + } + for style in styles_by_run[run]: + spans.append((start, len(inner_plain), "style", style)) + return end + run + + @staticmethod + def _find_closing(text: str, start: int, marker: str) -> int: + """Find the closing delimiter, skipping longer runs for singles. + + Parameters: + text: The full source text. + start: Index to start searching from. + marker: The exact delimiter run to find. + + Returns: + The index of the closing delimiter, or -1 when absent. + """ + i = start + while True: + end = text.find(marker, i) + if end == -1: + return -1 + if len(marker) == 1: + ch = marker + part_of_longer_run = (end > 0 and text[end - 1] == ch) or ( + end + 1 < len(text) and text[end + 1] == ch + ) + if part_of_longer_run: + i = end + 1 + continue + return end + + def _parse_inline_html( + self, text: str, i: int, out: list[str], spans: list[Span], depth: int + ) -> int | None: + """Parse a whitelisted inline HTML tag at index i. + + Only exact `` `` openers (no attributes) with a matching + `` `` closer are recognized. Anything else is literal text, + so unwhitelisted or malformed HTML carries no markup semantics. + + Parameters: + text: The full source text. + i: Index of the ``<`` character. + out: Output characters accumulated so far. + spans: Spans accumulated so far. + depth: Current recursion depth for nested constructs. + + Returns: + The index after the closing tag, or None when the tag does + not parse as a whitelisted construct. + """ + match = TAG_RE.match(text, i) + if match is None: + return None + tag = match.group(1) + style = self.inline_html_styles.get(tag) + if style is None: + return None + closing = f"" + end = text.find(closing, match.end()) + if end == -1: + return None + inner_plain, inner_spans = self._parse(text[match.end() : end], depth + 1) + start = len(out) + out.extend(inner_plain) + spans.extend( + (s + start, length, kind, payload) + for s, length, kind, payload in inner_spans + ) + spans.append((start, len(inner_plain), "style", style)) + return end + len(closing) diff --git a/draftjs_exporter/markdown_parser/resolvers.py b/draftjs_exporter/markdown_parser/resolvers.py new file mode 100644 index 0000000..5c946bd --- /dev/null +++ b/draftjs_exporter/markdown_parser/resolvers.py @@ -0,0 +1,127 @@ +"""Entity resolvers: map Markdown link and image URLs to Draft.js entities.""" + +from collections.abc import Callable +from typing import Any, TypeAlias, TypedDict +from urllib.parse import parse_qsl, urlparse + +from draftjs_exporter.constants import ENTITY_TYPES +from draftjs_exporter.types import Mutability + + +class EntityResolution(TypedDict, total=False): + """How a link or image URL should be converted into a Draft.js entity.""" + + type: str + """The entity type, e.g. ``LINK``, ``IMAGE``, ``DOCUMENT``, ``EMBED``.""" + + data: dict[str, Any] + """The entity data payload.""" + + mutability: Mutability + """The entity mutability. Defaults to ``MUTABLE`` when omitted.""" + + +EntityResolver: TypeAlias = Callable[[str, str], "EntityResolution | None"] +"""Resolve a URL and its label into an entity, or return None to defer.""" + + +def default_link_resolver(url: str, label: str) -> EntityResolution: + """Resolve any URL into a standard ``LINK`` entity. + + Parameters: + url: The link URL from the Markdown source. + label: The link text. + + Returns: + A ``LINK`` resolution with the URL in its data. + """ + return { + "type": ENTITY_TYPES.LINK, + "data": {"url": url}, + "mutability": "MUTABLE", + } + + +def default_image_resolver(url: str, alt: str) -> EntityResolution: + """Resolve any URL into a standard ``IMAGE`` entity. + + Parameters: + url: The image URL from the Markdown source. + alt: The image alt text. + + Returns: + An ``IMAGE`` resolution with ``src`` and ``alt`` in its data. + """ + return { + "type": ENTITY_TYPES.IMAGE, + "data": {"src": url, "alt": alt}, + "mutability": "IMMUTABLE", + } + + +def resolve( + chain: list[EntityResolver], + url: str, + label: str, + default: Callable[[str, str], EntityResolution], +) -> EntityResolution: + """Run a resolver chain, falling back to the default resolver. + + Parameters: + chain: Resolvers tried in order; the first non-None result wins. + url: The URL to resolve. + label: The link text or image alt text. + default: Resolver used when every chain entry defers. + + Returns: + The winning resolution, or the default resolution. + """ + for resolver in chain: + resolution = resolver(url, label) + if resolution is not None: + return resolution + return default(url, label) + + +def scheme_resolver( + scheme: str, + type_map: dict[str, str], + coerce: dict[str, Callable[[str], Any]] | None = None, + label_key: str | None = None, + mutability: Mutability = "MUTABLE", +) -> EntityResolver: + """Build a resolver for internal URLs like ``scheme://kind?key=value``. + + The URL host selects the entity type via ``type_map``. Query string + parameters become entity data, optionally converted per key via + ``coerce``. When ``label_key`` is set, a non-empty Markdown label + fills that data key if the query string did not provide it. + + Parameters: + scheme: The URL scheme to match, e.g. ``"wagtail"``. + type_map: Mapping of URL host to entity type. + coerce: Optional per-key converters for query string values. + label_key: Optional data key filled from the Markdown label. + mutability: Mutability for produced resolutions. + + Returns: + A resolver that defers (returns None) for non-matching URLs. + """ + converters = coerce if coerce is not None else {} + + def resolver(url: str, label: str) -> EntityResolution | None: + parsed = urlparse(url) + if parsed.scheme != scheme: + return None + entity_type = type_map.get(parsed.netloc) + if entity_type is None: + return None + data: dict[str, Any] = {} + for key, value in parse_qsl(parsed.query, keep_blank_values=True): + converter = converters.get(key) + data[key] = converter(value) if converter is not None else value + if label_key is not None and label and label_key not in data: + data[label_key] = label + return {"type": entity_type, "data": data, "mutability": mutability} + + return resolver diff --git a/example.py b/example.py index 7ea5fea..ea86ad6 100755 --- a/example.py +++ b/example.py @@ -3,6 +3,7 @@ import argparse import cProfile +import difflib import json import logging import re @@ -23,9 +24,11 @@ Element, Exporter, ExporterConfig, + MarkdownImporter, Props, ) from draftjs_exporter.markdown.entities import link as markdown_link +from draftjs_exporter.markdown_parser import scheme_resolver def blockquote(props: Props) -> Element: @@ -307,6 +310,56 @@ def prettify(markup: str) -> str: print("=== Markdown ===") # noqa: T201 print(markdown_output) # noqa: T201 + # --- Markdown import --- + # Import the Markdown back into a ContentState, with entity resolvers + # for internal URLs and a filter demoting level-1 headings. + + importer = MarkdownImporter( + { + "parser_config": { + "image_resolvers": [ + scheme_resolver( + "wagtail", + {"image": "IMAGE"}, + coerce={"id": int}, + label_key="alt", + mutability="IMMUTABLE", + ) + ], + }, + "filter_rules": [ + { + "type": "block", + "match": BLOCK_TYPES.HEADER_ONE, + "action": "demote", + }, + ], + } + ) + imported = importer.import_markdown(markdown_output) + + print("=== Markdown import ===") # noqa: T201 + print(json.dumps(imported, indent=2)) # noqa: T201 + + # --- Markdown round-trip: export the imported content state back to Markdown --- + re_exported = markdown_exporter.render(imported) + + print("=== Markdown diff (original vs round-trip) ===") # noqa: T201 + if re_exported == markdown_output: + print("[no differences]") # noqa: T201 + else: + print( # noqa: T201 + "\n".join( + difflib.unified_diff( + markdown_output.splitlines(), + re_exported.splitlines(), + fromfile="original", + tofile="round-trip", + lineterm="", + ) + ) + ) + styles = """ /* Tacit CSS framework https://yegor256.github.io/tacit/ */ input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}th{font-weight:600}td,th{border-bottom:1.08px solid #ccc;padding:14.85px 18px}thead th{border-bottom-width:2.16px;padding-bottom:6.3px}table{display:block;max-width:100%;overflow-x:auto}input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}input,textarea,select,button{display:block;max-width:100%;padding:9.9px}label{display:block;margin-bottom:14.76px}input[type="submit"],input[type="reset"],button{background:#f2f2f2;border-radius:3.6px;color:#8c8c8c;cursor:pointer;display:inline;margin-bottom:18px;margin-right:7.2px;padding:6.525px 23.4px;text-align:center}input[type="submit"]:hover,input[type="reset"]:hover,button:hover{background:#d9d9d9;color:#000}input[type="submit"][disabled],input[type="reset"][disabled],button[disabled]{background:#e6e6e6;color:#b3b3b3;cursor:not-allowed}input[type="submit"],button[type="submit"]{background:#367ac3;color:#fff}input[type="submit"]:hover,button[type="submit"]:hover{background:#255587;color:#bfbfbf}input[type="text"],input[type="password"],input[type="email"],input[type="url"],input[type="phone"],input[type="tel"],input[type="number"],input[type="datetime"],input[type="date"],input[type="month"],input[type="week"],input[type="color"],input[type="time"],input[type="search"],input[type="range"],input[type="file"],input[type="datetime-local"],select,textarea{border:1px solid #ccc;margin-bottom:18px;padding:5.4px 6.3px}input[type="checkbox"],input[type="radio"]{float:left;line-height:36px;margin-right:9px;margin-top:8.1px}input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}pre,code,kbd,samp,var,output{font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:16.2px}pre{border-left:1.8px solid #96bbe2;line-height:25.2px;margin-top:29.7px;overflow:auto;padding-left:18px}pre code{background:none;border:0;line-height:29.7px;padding:0}code{background:#ededed;border:1.8px solid #ccc;border-radius:3.6px;display:inline-block;line-height:18px;padding:3px 6px 2px}input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}h1,h2,h3,h4,h5,h6{color:#000;margin-bottom:18px}h1{font-size:36px;font-weight:500;margin-top:36px}h2{font-size:25.2px;font-weight:400;margin-top:27px}h3{font-size:21.6px;margin-top:21.6px}h4{font-size:18px;margin-top:18px}h5{font-size:14.4px;font-weight:bold;margin-top:18px;text-transform:uppercase}h6{color:#ccc;font-size:14.4px;font-weight:bold;margin-top:18px;text-transform:uppercase}input,textarea,select,button,html,body{font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:18px;font-stretch:normal;font-style:normal;font-weight:300;line-height:29.7px}a{color:#367ac3;text-decoration:none}a:hover{text-decoration:underline}hr{border-bottom:1px solid #ccc}small{font-size:15.3px}em,i{font-style:italic}strong,b{font-weight:600}*{border:0;border-collapse:separate;border-spacing:0;box-sizing:border-box;margin:0;outline:0;padding:0;text-align:left;vertical-align:baseline}html,body{height:100%;width:100%}body{background:#f5f5f5;color:#1a1a1a;padding:36px}p,ul,ol,dl,blockquote,hr,pre,table,form,fieldset,figure,address{margin-bottom:29.7px}section{margin-left:auto;margin-right:auto;max-width:100%;width:900px}article{background:#fff;border:1.8px solid #d9d9d9;border-radius:7.2px;padding:43.2px}header{margin-bottom:36px}footer{margin-top:36px}nav{text-align:center}nav ul{list-style:none;margin-left:0;text-align:center}nav ul li{display:inline;margin-left:9px;margin-right:9px}nav ul li:first-child{margin-left:0}nav ul li:last-child{margin-right:0}ol,ul{margin-left:29.7px}li ol,li ul{margin-bottom:0}@media (max-width: 767px){body{padding:18px}article{border-radius:0;margin:-18px;padding:18px}textarea,input,select{max-width:100%}fieldset{min-width:0}section{width:auto}fieldset,x:-moz-any-link{display:table-cell}} diff --git a/mkdocs.yml b/mkdocs.yml index b077852..ad72970 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -69,6 +69,7 @@ nav: - Getting started: getting-started.md - Content state: content-state.md - Markdown support: markdown.md + - Markdown importer: markdown-importer.md - Migration guide: migration-guide.md - Guides: - Custom components: guides/custom-components.md diff --git a/tests/contentstate_filter/__init__.py b/tests/contentstate_filter/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/contentstate_filter/test_filter.py b/tests/contentstate_filter/test_filter.py new file mode 100644 index 0000000..15ac82e --- /dev/null +++ b/tests/contentstate_filter/test_filter.py @@ -0,0 +1,260 @@ +"""Tests for ContentState filtering.""" + +import unittest + +from draftjs_exporter.contentstate_filter import ContentStateFilter +from draftjs_exporter.types import Block, ContentState, EntityRange, InlineStyleRange + + +def cs_with_blocks(*blocks: Block) -> ContentState: + """Build a ContentState from blocks with an empty entity map.""" + return {"blocks": list(blocks), "entityMap": {}} + + +def make_block( + type_: str, + text: str = "x", + depth: int = 0, + styles: list[InlineStyleRange] | None = None, + entities: list[EntityRange] | None = None, +) -> Block: + """Build a single Draft.js block.""" + return { + "key": "aaaaa", + "text": text, + "type": type_, + "depth": depth, + "inlineStyleRanges": styles or [], + "entityRanges": entities or [], + } + + +class TestBlockRules(unittest.TestCase): + def test_remove_block(self): + cs = cs_with_blocks(make_block("header-one"), make_block("unstyled")) + result = ContentStateFilter( + [{"type": "block", "match": "header-one", "action": "remove"}] + ).apply(cs) + self.assertEqual([b["type"] for b in result["blocks"]], ["unstyled"]) + + def test_keep_is_noop(self): + cs = cs_with_blocks(make_block("header-one")) + result = ContentStateFilter( + [{"type": "block", "match": "header-one", "action": "keep"}] + ).apply(cs) + self.assertEqual(len(result["blocks"]), 1) + + def test_demote_headings(self): + cs = cs_with_blocks( + make_block("header-one"), + make_block("header-three"), + ) + result = ContentStateFilter( + [{"type": "block", "match": "header-one", "action": "demote"}] + ).apply(cs) + self.assertEqual( + [b["type"] for b in result["blocks"]], ["header-two", "header-three"] + ) + + def test_unmatched_blocks_kept(self): + cs = cs_with_blocks(make_block("header-two")) + result = ContentStateFilter( + [{"type": "block", "match": "header-one", "action": "remove"}] + ).apply(cs) + self.assertEqual(len(result["blocks"]), 1) + + def test_callable_replaces_block(self): + def replace(block): + return {**block, "type": "unstyled"} + + cs = cs_with_blocks(make_block("header-one")) + result = ContentStateFilter( + [{"type": "block", "match": "header-one", "action": replace}] + ).apply(cs) + self.assertEqual(result["blocks"][0]["type"], "unstyled") + + def test_callable_none_removes(self): + cs = cs_with_blocks(make_block("unstyled")) + result = ContentStateFilter( + [{"type": "block", "match": "unstyled", "action": lambda b: None}] + ).apply(cs) + self.assertEqual(result["blocks"], []) + + def test_input_not_mutated(self): + block = make_block("header-one") + cs = cs_with_blocks(block) + ContentStateFilter( + [{"type": "block", "match": "header-one", "action": "demote"}] + ).apply(cs) + self.assertEqual(block["type"], "header-one") + + +class TestInlineStyleRules(unittest.TestCase): + def test_remove_style(self): + block = make_block( + "unstyled", + text="abc", + styles=[ + {"offset": 0, "length": 1, "style": "BOLD"}, + {"offset": 1, "length": 1, "style": "ITALIC"}, + ], + ) + result = ContentStateFilter( + [{"type": "inline_style", "match": "BOLD", "action": "remove"}] + ).apply(cs_with_blocks(block)) + self.assertEqual( + result["blocks"][0]["inlineStyleRanges"], + [{"offset": 1, "length": 1, "style": "ITALIC"}], + ) + + def test_callable_renames_style(self): + block = make_block( + "unstyled", text="a", styles=[{"offset": 0, "length": 1, "style": "X"}] + ) + result = ContentStateFilter( + [{"type": "inline_style", "match": "X", "action": lambda style: "BOLD"}] + ).apply(cs_with_blocks(block)) + self.assertEqual(result["blocks"][0]["inlineStyleRanges"][0]["style"], "BOLD") + + +class TestEntityRules(unittest.TestCase): + def setUp(self) -> None: + self.cs: ContentState = { + "blocks": [ + make_block( + "unstyled", + text="ab", + entities=[{"offset": 0, "length": 1, "key": 0}], + ) + ], + "entityMap": { + "0": {"type": "LINK", "mutability": "MUTABLE", "data": {"url": "/x"}} + }, + } + + def test_remove_entity(self): + result = ContentStateFilter( + [{"type": "entity", "match": "LINK", "action": "remove"}] + ).apply(self.cs) + self.assertEqual(result["blocks"][0]["entityRanges"], []) + self.assertEqual(result["entityMap"], {}) + + def test_unmatched_entity_kept(self): + result = ContentStateFilter( + [{"type": "entity", "match": "IMAGE", "action": "remove"}] + ).apply(self.cs) + self.assertEqual(result["entityMap"], self.cs["entityMap"]) + + def test_callable_replaces_entity(self): + result = ContentStateFilter( + [ + { + "type": "entity", + "match": "LINK", + "action": lambda e: {**e, "data": {"url": "/y"}}, + } + ] + ).apply(self.cs) + self.assertEqual(result["entityMap"]["0"]["data"], {"url": "/y"}) + + def test_orphaned_range_dropped(self): + self.cs["blocks"][0]["entityRanges"] = [{"offset": 0, "length": 1, "key": 9}] + result = ContentStateFilter([]).apply(self.cs) + self.assertEqual(result["blocks"][0]["entityRanges"], []) + + +class TestDepthNormalization(unittest.TestCase): + def test_removed_parent_clamps_depth(self): + cs = cs_with_blocks( + make_block("unordered-list-item", depth=0), + make_block("unordered-list-item", depth=1), + make_block("unordered-list-item", depth=2), + ) + result = ContentStateFilter( + [ + { + "type": "block", + "match": "unordered-list-item", + "action": lambda b: None if b["depth"] == 0 else b, + } + ] + ).apply(cs) + self.assertEqual([b["depth"] for b in result["blocks"]], [0, 1]) + + def test_depth_reset_after_non_list_block(self): + cs = cs_with_blocks( + make_block("unstyled"), + make_block("unordered-list-item", depth=2), + ) + result = ContentStateFilter([]).apply(cs) + self.assertEqual(result["blocks"][1]["depth"], 0) + + +class TestEdgeCases(unittest.TestCase): + def test_chain_stops_after_remove(self): + block = make_block("unstyled") + cs = cs_with_blocks(block) + result = ContentStateFilter( + [ + {"type": "block", "match": "unstyled", "action": "remove"}, + {"type": "block", "match": "unstyled", "action": "keep"}, + ] + ).apply(cs) + self.assertEqual(result["blocks"], []) + + def test_block_callback_without_type_rejected(self): + from draftjs_exporter.error import ConfigException + + cs = cs_with_blocks(make_block("unstyled")) + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "block", "match": "unstyled", "action": lambda b: {}}] + ).apply(cs) + + def test_entity_callback_without_type_rejected(self): + from draftjs_exporter.error import ConfigException + + cs: ContentState = { + "blocks": [ + make_block( + "unstyled", + text="a", + entities=[{"offset": 0, "length": 1, "key": 0}], + ) + ], + "entityMap": {"0": {"type": "LINK", "mutability": "MUTABLE", "data": {}}}, + } + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "entity", "match": "LINK", "action": lambda e: {}}] + ).apply(cs) + + +class TestDemoteChainGuards(unittest.TestCase): + def test_demote_after_callback_returning_bad_block_rejected(self): + from draftjs_exporter.error import ConfigException + + cs = cs_with_blocks(make_block("header-one")) + with self.assertRaises(ConfigException): + ContentStateFilter( + [ + {"type": "block", "match": "header-one", "action": lambda b: {}}, + {"type": "block", "match": "header-one", "action": "demote"}, + ] + ).apply(cs) + + def test_demote_after_callback_changing_type_rejected(self): + from draftjs_exporter.error import ConfigException + + cs = cs_with_blocks(make_block("header-one")) + with self.assertRaises(ConfigException): + ContentStateFilter( + [ + { + "type": "block", + "match": "header-one", + "action": lambda b: {**b, "type": "unstyled"}, + }, + {"type": "block", "match": "header-one", "action": "demote"}, + ] + ).apply(cs) diff --git a/tests/contentstate_filter/test_filter_rules.py b/tests/contentstate_filter/test_filter_rules.py new file mode 100644 index 0000000..53cf14b --- /dev/null +++ b/tests/contentstate_filter/test_filter_rules.py @@ -0,0 +1,59 @@ +"""Tests for filter rule validation.""" + +import unittest + +from draftjs_exporter.contentstate_filter import ContentStateFilter +from draftjs_exporter.error import ConfigException +from draftjs_exporter.types import ContentState + + +class TestRuleValidation(unittest.TestCase): + def test_invalid_rule_type(self): + with self.assertRaises(ConfigException): + ContentStateFilter([{"type": "nope", "match": "x", "action": "keep"}]) # type: ignore[typeddict-item] # ty: ignore[invalid-argument-type] + + def test_invalid_action(self): + with self.assertRaises(ConfigException): + ContentStateFilter([{"type": "block", "match": "x", "action": "nope"}]) # type: ignore[typeddict-item] # ty: ignore[invalid-argument-type] + + def test_demote_requires_header_block(self): + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "block", "match": "unstyled", "action": "demote"}] + ) + + def test_demote_header_six_rejected(self): + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "block", "match": "header-six", "action": "demote"}] + ) + + def test_demote_on_non_block_rejected(self): + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "entity", "match": "header-one", "action": "demote"}] + ) + + def test_no_rules_is_identity(self): + cs: ContentState = {"blocks": [], "entityMap": {}} + self.assertEqual(ContentStateFilter().apply(cs), cs) + self.assertEqual(ContentStateFilter(None).apply(cs), cs) + + def test_callable_returning_garbage_rejected(self): + cs: ContentState = { + "blocks": [ + { + "key": "a", + "text": "x", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [], + } + ], + "entityMap": {}, + } + with self.assertRaises(ConfigException): + ContentStateFilter( + [{"type": "block", "match": "unstyled", "action": lambda b: 42}] + ).apply(cs) diff --git a/tests/markdown_importer/__init__.py b/tests/markdown_importer/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/markdown_importer/test_importer.py b/tests/markdown_importer/test_importer.py new file mode 100644 index 0000000..39c0f77 --- /dev/null +++ b/tests/markdown_importer/test_importer.py @@ -0,0 +1,72 @@ +"""Tests for the MarkdownImporter public API.""" + +import unittest + +from draftjs_exporter.error import MarkdownParseError +from draftjs_exporter.markdown_importer import MarkdownImporter +from draftjs_exporter.markdown_parser import scheme_resolver + + +class TestMarkdownImporter(unittest.TestCase): + def test_default_import(self): + cs = MarkdownImporter().import_markdown("# Hello\n\nWorld") + self.assertEqual([b["type"] for b in cs["blocks"]], ["header-one", "unstyled"]) + + def test_none_config(self): + cs = MarkdownImporter(None).import_markdown("text") + self.assertEqual(cs["blocks"][0]["text"], "text") + + def test_filter_rules_applied(self): + importer = MarkdownImporter( + { + "filter_rules": [ + {"type": "block", "match": "header-one", "action": "demote"} + ] + } + ) + cs = importer.import_markdown("# Hello") + self.assertEqual(cs["blocks"][0]["type"], "header-two") + + def test_parser_config_applied(self): + importer = MarkdownImporter({"parser_config": {"headings": False}}) + cs = importer.import_markdown("# Hello") + self.assertEqual(cs["blocks"][0]["type"], "unstyled") + + def test_custom_parser_dotted_path(self): + importer = MarkdownImporter( + {"parser": "draftjs_exporter.markdown_parser.MarkdownParser"} + ) + cs = importer.import_markdown("text") + self.assertEqual(cs["blocks"][0]["text"], "text") + + def test_parse_error_propagates(self): + def bad(url, label): + raise RuntimeError("boom") + + importer = MarkdownImporter({"parser_config": {"link_resolvers": [bad]}}) + with self.assertRaises(MarkdownParseError): + importer.import_markdown("[a](/b)") + + def test_wagtail_style_end_to_end(self): + importer = MarkdownImporter( + { + "parser_config": { + "image_resolvers": [ + scheme_resolver( + "wagtail", + {"image": "IMAGE"}, + coerce={"id": int}, + label_key="alt", + mutability="IMMUTABLE", + ) + ] + } + } + ) + cs = importer.import_markdown( + "" + ) + self.assertEqual( + cs["entityMap"]["0"]["data"], + {"id": 10, "alt": "alt", "format": "left"}, + ) diff --git a/tests/markdown_parser/__init__.py b/tests/markdown_parser/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/tests/markdown_parser/test_blocks.py b/tests/markdown_parser/test_blocks.py new file mode 100644 index 0000000..943d4a6 --- /dev/null +++ b/tests/markdown_parser/test_blocks.py @@ -0,0 +1,293 @@ +"""Tests for block-level Markdown parsing.""" + +import unittest +from typing import Any + +from draftjs_exporter.markdown_parser.blocks import BlockParser +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from draftjs_exporter.markdown_parser.inline import InlineParser +from draftjs_exporter.types import ContentState + + +def parse(markdown: str, **overrides: Any) -> ContentState: + """Parse Markdown into a ContentState with all constructs enabled.""" + builder = ContentStateBuilder() + inline = InlineParser( + emphasis=True, + code_inline=True, + links=True, + images=overrides.get("images", True), + line_breaks=True, + inline_html_styles={}, + link_resolvers=[], + image_resolvers=[], + builder=builder, + ) + config: dict[str, Any] = { + "headings": True, + "blockquote": True, + "code_fenced": True, + "thematic_break": True, + "unordered_list": True, + "ordered_list": True, + "images": True, + "inline": inline, + "builder": builder, + } + config.update(overrides) + BlockParser(**config).parse(markdown) + return builder.build() + + +def block_types(cs: ContentState) -> list[str]: + """Extract the block types of a ContentState.""" + return [b.get("type", "unstyled") for b in cs.get("blocks", [])] + + +class TestParagraphs(unittest.TestCase): + def test_single_paragraph(self): + cs = parse("hello") + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "hello") + + def test_blank_lines_split_paragraphs(self): + cs = parse("a\n\nb") + self.assertEqual(len(cs["blocks"]), 2) + + def test_soft_wrapped_lines_join_with_newline(self): + cs = parse("a\nb") + self.assertEqual(len(cs["blocks"]), 1) + self.assertEqual(cs["blocks"][0]["text"], "a\nb") + + def test_empty_input_produces_no_blocks(self): + self.assertEqual(parse("")["blocks"], []) + self.assertEqual(parse("\n\n\n")["blocks"], []) + + +class TestHeadings(unittest.TestCase): + def test_all_levels(self): + names = ["one", "two", "three", "four", "five", "six"] + for level in range(1, 7): + cs = parse(f"{'#' * level} Title") + self.assertEqual(block_types(cs), [f"header-{names[level - 1]}"]) + + def test_heading_content_is_inline_parsed(self): + cs = parse("## **Bold** title") + block = cs["blocks"][0] + self.assertEqual(block["text"], "Bold title") + self.assertEqual( + block["inlineStyleRanges"], + [{"offset": 0, "length": 4, "style": "BOLD"}], + ) + + def test_closing_hashes_stripped(self): + cs = parse("# Title #") + self.assertEqual(cs["blocks"][0]["text"], "Title") + + def test_no_space_after_hash_is_paragraph(self): + cs = parse("#notaheading") + self.assertEqual(block_types(cs), ["unstyled"]) + + def test_heading_then_paragraph(self): + cs = parse("# T\n\ntext") + self.assertEqual(block_types(cs), ["header-one", "unstyled"]) + + +class TestThematicBreaks(unittest.TestCase): + def test_dashes(self): + cs = parse("a\n\n---\n\nb") + self.assertEqual(block_types(cs), ["unstyled", "atomic", "unstyled"]) + + def test_atomic_block_shape(self): + cs = parse("---") + block = cs["blocks"][0] + self.assertEqual(block["text"], " ") + self.assertEqual(block["entityRanges"], [{"offset": 0, "length": 1, "key": 0}]) + self.assertEqual( + cs["entityMap"]["0"], + {"type": "HORIZONTAL_RULE", "mutability": "IMMUTABLE", "data": {}}, + ) + + def test_stars_and_underscores(self): + self.assertEqual(block_types(parse("***")), ["atomic"]) + self.assertEqual(block_types(parse("___")), ["atomic"]) + + def test_spaced_markers(self): + self.assertEqual(block_types(parse("- - -")), ["atomic"]) + + +class TestBlockquotes(unittest.TestCase): + def test_single_line_quote(self): + cs = parse("> quoted") + self.assertEqual(block_types(cs), ["blockquote"]) + self.assertEqual(cs["blocks"][0]["text"], "quoted") + + def test_multiline_quote_joins(self): + cs = parse("> a\n> b") + self.assertEqual(len(cs["blocks"]), 1) + self.assertEqual(cs["blocks"][0]["text"], "a\nb") + + def test_empty_quote_line_splits_blocks(self): + cs = parse("> a\n>\n> b") + self.assertEqual(block_types(cs), ["blockquote", "blockquote"]) + + def test_quote_content_is_inline_parsed(self): + cs = parse("> **bold**") + self.assertEqual(cs["blocks"][0]["text"], "bold") + self.assertEqual( + cs["blocks"][0]["inlineStyleRanges"], + [{"offset": 0, "length": 4, "style": "BOLD"}], + ) + + def test_quote_without_space(self): + cs = parse(">quoted") + self.assertEqual(cs["blocks"][0]["text"], "quoted") + + +class TestFencedCode(unittest.TestCase): + def test_backtick_fence(self): + cs = parse("```\ncode line\n```") + self.assertEqual(block_types(cs), ["code-block"]) + self.assertEqual(cs["blocks"][0]["text"], "code line") + + def test_tilde_fence(self): + cs = parse("~~~\ncode\n~~~") + self.assertEqual(block_types(cs), ["code-block"]) + + def test_info_string_ignored(self): + cs = parse("```python\nx = 1\n```") + self.assertEqual(cs["blocks"][0]["text"], "x = 1") + + def test_multiline_code(self): + cs = parse("```\na\nb\n```") + self.assertEqual(cs["blocks"][0]["text"], "a\nb") + + def test_unclosed_fence_parses_to_eof(self): + cs = parse("```\ncode") + self.assertEqual(block_types(cs), ["code-block"]) + self.assertEqual(cs["blocks"][0]["text"], "code") + + def test_code_content_not_inline_parsed(self): + cs = parse("```\n**not bold**\n```") + self.assertEqual(cs["blocks"][0]["text"], "**not bold**") + self.assertEqual(cs["blocks"][0]["inlineStyleRanges"], []) + + def test_closing_fence_must_match_marker(self): + cs = parse("```\na\n~~~\nb\n```") + self.assertEqual(cs["blocks"][0]["text"], "a\n~~~\nb") + + +class TestLists(unittest.TestCase): + def test_unordered_flat(self): + cs = parse("- a\n- b") + self.assertEqual( + block_types(cs), ["unordered-list-item", "unordered-list-item"] + ) + self.assertEqual([b["depth"] for b in cs["blocks"]], [0, 0]) + + def test_ordered_flat(self): + cs = parse("1. a\n2. b") + self.assertEqual(block_types(cs), ["ordered-list-item"] * 2) + + def test_ordered_with_paren_delimiter(self): + cs = parse("1) a") + self.assertEqual(block_types(cs), ["ordered-list-item"]) + + def test_all_bullet_markers(self): + for marker in "*+-": + cs = parse(f"{marker} item") + self.assertEqual(block_types(cs), ["unordered-list-item"]) + + def test_nested_unordered(self): + cs = parse("- a\n - b\n - c") + self.assertEqual([b["depth"] for b in cs["blocks"]], [0, 1, 2]) + + def test_nested_then_back_to_top(self): + cs = parse("- a\n - b\n- c") + self.assertEqual([b["depth"] for b in cs["blocks"]], [0, 1, 0]) + + def test_mixed_kinds_by_indent(self): + cs = parse("- a\n 1. b") + self.assertEqual(block_types(cs), ["unordered-list-item", "ordered-list-item"]) + self.assertEqual(cs["blocks"][1]["depth"], 1) + + def test_list_content_is_inline_parsed(self): + cs = parse("- **bold**") + self.assertEqual(cs["blocks"][0]["text"], "bold") + + def test_blank_line_ends_list(self): + cs = parse("- a\n\nparagraph") + self.assertEqual(block_types(cs), ["unordered-list-item", "unstyled"]) + + def test_paragraph_after_list_without_blank_line(self): + cs = parse("- a\nparagraph") + self.assertEqual(block_types(cs), ["unordered-list-item", "unstyled"]) + + def test_list_between_paragraphs(self): + cs = parse("intro\n\n- item\n\noutro") + self.assertEqual( + block_types(cs), ["unstyled", "unordered-list-item", "unstyled"] + ) + + +class TestStandaloneImages(unittest.TestCase): + def test_standalone_image_is_atomic(self): + cs = parse("") + block = cs["blocks"][0] + self.assertEqual(block["type"], "atomic") + self.assertEqual(block["text"], " ") + self.assertEqual(block["entityRanges"], [{"offset": 0, "length": 1, "key": 0}]) + self.assertEqual( + cs["entityMap"]["0"], + { + "type": "IMAGE", + "mutability": "IMMUTABLE", + "data": {"src": "/img.jpg", "alt": "alt text"}, + }, + ) + + def test_image_with_text_stays_inline(self): + cs = parse("look  here") + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "look alt here") + + def test_images_disabled_stays_text(self): + cs = parse("", images=False) + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "") + + +class TestQuoteAndErrorEdges(unittest.TestCase): + def test_quote_starting_with_empty_line(self): + cs = parse(">\n> b") + self.assertEqual(block_types(cs), ["blockquote"]) + self.assertEqual(cs["blocks"][0]["text"], "b") + + def test_quote_with_only_empty_content(self): + cs = parse(">") + self.assertEqual(cs["blocks"], []) + + def test_parse_error_with_line_propagates_unchanged(self): + from draftjs_exporter.error import MarkdownParseError + from draftjs_exporter.markdown_parser import MarkdownParser + + def bad(url, label): + raise MarkdownParseError("custom failure", line=99) + + parser = MarkdownParser({"link_resolvers": [bad]}) + with self.assertRaises(MarkdownParseError) as ctx: + parser.parse("[a](/b)") + self.assertEqual(ctx.exception.line, 99) + self.assertEqual(ctx.exception.message, "custom failure") + + +class TestTabIndentation(unittest.TestCase): + def test_tab_advances_to_next_tab_stop(self): + # " \t" is 4 columns per CommonMark (tab from column 2 advances + # to column 4), the same indent as four spaces. + cs = parse("- a\n - b\n \t- c") + self.assertEqual([b["depth"] for b in cs["blocks"]], [0, 1, 1]) + + def test_lone_tab_is_one_tab_stop(self): + cs = parse("- a\n\t- b") + self.assertEqual([b["depth"] for b in cs["blocks"]], [0, 1]) diff --git a/tests/markdown_parser/test_builder.py b/tests/markdown_parser/test_builder.py new file mode 100644 index 0000000..bfe8c3b --- /dev/null +++ b/tests/markdown_parser/test_builder.py @@ -0,0 +1,53 @@ +"""Tests for the ContentState builder.""" + +import unittest + +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder + + +class TestContentStateBuilder(unittest.TestCase): + def test_empty_build(self): + builder = ContentStateBuilder() + self.assertEqual(builder.build(), {"blocks": [], "entityMap": {}}) + + def test_block_keys_are_sequential(self): + builder = ContentStateBuilder() + builder.add_block("unstyled", "a") + builder.add_block("unstyled", "b") + keys = [b["key"] for b in builder.build()["blocks"]] + self.assertEqual(keys, ["00000", "00001"]) + self.assertEqual(len(set(keys)), 2) + + def test_block_defaults(self): + builder = ContentStateBuilder() + builder.add_block("unstyled", "a") + block = builder.build()["blocks"][0] + self.assertEqual(block["depth"], 0) + self.assertEqual(block["inlineStyleRanges"], []) + self.assertEqual(block["entityRanges"], []) + + def test_add_entity_returns_int_keys(self): + builder = ContentStateBuilder() + first = builder.add_entity("LINK", {"url": "/a"}) + second = builder.add_entity("IMAGE", {"src": "/b"}, "IMMUTABLE") + self.assertEqual((first, second), (0, 1)) + entity_map = builder.build()["entityMap"] + self.assertEqual( + entity_map["0"], + {"type": "LINK", "mutability": "MUTABLE", "data": {"url": "/a"}}, + ) + self.assertEqual(entity_map["1"]["mutability"], "IMMUTABLE") + + def test_full_block(self): + builder = ContentStateBuilder() + key = builder.add_entity("LINK", {"url": "/a"}) + builder.add_block( + "unstyled", + "text", + depth=2, + inline_style_ranges=[{"offset": 0, "length": 2, "style": "BOLD"}], + entity_ranges=[{"offset": 0, "length": 4, "key": key}], + ) + block = builder.build()["blocks"][0] + self.assertEqual(block["depth"], 2) + self.assertEqual(block["entityRanges"][0]["key"], 0) diff --git a/tests/markdown_parser/test_inline.py b/tests/markdown_parser/test_inline.py new file mode 100644 index 0000000..a81383b --- /dev/null +++ b/tests/markdown_parser/test_inline.py @@ -0,0 +1,276 @@ +"""Tests for inline Markdown parsing.""" + +import unittest +from typing import Any + +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from draftjs_exporter.markdown_parser.inline import InlineParser + + +def make_parser(**overrides: Any) -> InlineParser: + """Build an InlineParser with all constructs enabled.""" + config: dict[str, Any] = { + "emphasis": True, + "code_inline": True, + "links": True, + "images": True, + "line_breaks": True, + "inline_html_styles": {}, + "link_resolvers": [], + "image_resolvers": [], + "builder": ContentStateBuilder(), + } + config.update(overrides) + return InlineParser(**config) + + +class TestPlainText(unittest.TestCase): + def test_plain_text_passes_through(self): + text, styles, entities = make_parser().parse("hello world") + self.assertEqual(text, "hello world") + self.assertEqual(styles, []) + self.assertEqual(entities, []) + + +class TestEscapes(unittest.TestCase): + def test_escaped_char_is_literal(self): + text, styles, _ = make_parser().parse(r"\*not italic\*") + self.assertEqual(text, "*not italic*") + self.assertEqual(styles, []) + + def test_backslash_before_non_escapable_is_literal(self): + text, _, _ = make_parser().parse(r"\a") + self.assertEqual(text, r"\a") + + +class TestCodeSpans(unittest.TestCase): + def test_code_span(self): + text, styles, _ = make_parser().parse("a `bc` d") + self.assertEqual(text, "a bc d") + self.assertEqual(styles, [{"offset": 2, "length": 2, "style": "CODE"}]) + + def test_code_span_contents_are_literal(self): + text, styles, _ = make_parser().parse("`**not bold**`") + self.assertEqual(text, "**not bold**") + self.assertEqual(styles, [{"offset": 0, "length": 12, "style": "CODE"}]) + + def test_unmatched_backtick_is_literal(self): + text, styles, _ = make_parser().parse("a `b") + self.assertEqual(text, "a `b") + self.assertEqual(styles, []) + + def test_code_disabled(self): + text, styles, _ = make_parser(code_inline=False).parse("`x`") + self.assertEqual(text, "`x`") + self.assertEqual(styles, []) + + +class TestHardBreaks(unittest.TestCase): + def test_two_spaces_before_newline_are_stripped(self): + text, _, _ = make_parser().parse("a \nb") + self.assertEqual(text, "a\nb") + + def test_soft_break_kept(self): + text, _, _ = make_parser().parse("a\nb") + self.assertEqual(text, "a\nb") + + def test_single_trailing_space_kept(self): + text, _, _ = make_parser().parse("a \nb") + self.assertEqual(text, "a \nb") + + +class TestEmphasis(unittest.TestCase): + def test_italic_star(self): + text, styles, _ = make_parser().parse("a *b* c") + self.assertEqual(text, "a b c") + self.assertEqual(styles, [{"offset": 2, "length": 1, "style": "ITALIC"}]) + + def test_italic_underscore(self): + text, styles, _ = make_parser().parse("_b_") + self.assertEqual(styles, [{"offset": 0, "length": 1, "style": "ITALIC"}]) + + def test_bold_stars(self): + text, styles, _ = make_parser().parse("**bold**") + self.assertEqual(text, "bold") + self.assertEqual(styles, [{"offset": 0, "length": 4, "style": "BOLD"}]) + + def test_bold_underscores(self): + text, styles, _ = make_parser().parse("__bold__") + self.assertEqual(styles, [{"offset": 0, "length": 4, "style": "BOLD"}]) + + def test_bold_italic_triple(self): + text, styles, _ = make_parser().parse("***both***") + self.assertEqual(text, "both") + self.assertEqual( + styles, + [ + {"offset": 0, "length": 4, "style": "BOLD"}, + {"offset": 0, "length": 4, "style": "ITALIC"}, + ], + ) + + def test_nested_italic_in_bold(self): + text, styles, _ = make_parser().parse("**a *b* c**") + self.assertEqual(text, "a b c") + self.assertEqual( + styles, + [ + {"offset": 0, "length": 5, "style": "BOLD"}, + {"offset": 2, "length": 1, "style": "ITALIC"}, + ], + ) + + def test_bold_inside_italic(self): + text, styles, _ = make_parser().parse("*a **b** c*") + self.assertEqual(text, "a b c") + self.assertIn({"offset": 0, "length": 5, "style": "ITALIC"}, styles) + self.assertIn({"offset": 2, "length": 1, "style": "BOLD"}, styles) + + def test_unmatched_delimiter_is_literal(self): + text, styles, _ = make_parser().parse("a *b") + self.assertEqual(text, "a *b") + self.assertEqual(styles, []) + + def test_mismatched_run_length_partially_matches(self): + # Per CommonMark, `**a*` is a literal star followed by emphasis: + # the second opener star pairs with the trailing star. + text, styles, _ = make_parser().parse("**a*") + self.assertEqual(text, "*a") + self.assertEqual(styles, [{"offset": 1, "length": 1, "style": "ITALIC"}]) + + def test_run_longer_than_three_is_literal(self): + text, styles, _ = make_parser().parse("****a****") + self.assertEqual(text, "****a****") + self.assertEqual(styles, []) + + def test_offsets_after_emphasis(self): + text, styles, _ = make_parser().parse("**b** x *i*") + self.assertEqual(text, "b x i") + self.assertEqual( + styles, + [ + {"offset": 0, "length": 1, "style": "BOLD"}, + {"offset": 4, "length": 1, "style": "ITALIC"}, + ], + ) + + def test_emphasis_disabled(self): + text, styles, _ = make_parser(emphasis=False).parse("**a**") + self.assertEqual(text, "**a**") + self.assertEqual(styles, []) + + +def parse_with_builder(**overrides: Any) -> tuple[InlineParser, ContentStateBuilder]: + """Build a parser with a fresh builder, returning both.""" + builder = ContentStateBuilder() + parser = make_parser(builder=builder, **overrides) + return parser, builder + + +class TestLinks(unittest.TestCase): + def test_simple_link(self): + parser, builder = parse_with_builder() + text, styles, entities = parser.parse("[example](https://example.com)") + self.assertEqual(text, "example") + self.assertEqual(styles, []) + self.assertEqual(entities, [{"offset": 0, "length": 7, "key": 0}]) + self.assertEqual( + builder.entity_map["0"], + { + "type": "LINK", + "mutability": "MUTABLE", + "data": {"url": "https://example.com"}, + }, + ) + + def test_link_with_styled_label(self): + parser, builder = parse_with_builder() + text, styles, entities = parser.parse("[**bold**](/url)") + self.assertEqual(text, "bold") + self.assertEqual(styles, [{"offset": 0, "length": 4, "style": "BOLD"}]) + self.assertEqual(entities, [{"offset": 0, "length": 4, "key": 0}]) + + def test_link_offsets_in_text(self): + parser, builder = parse_with_builder() + text, _, entities = parser.parse("see [docs](/d) now") + self.assertEqual(text, "see docs now") + self.assertEqual(entities, [{"offset": 4, "length": 4, "key": 0}]) + + def test_links_disabled(self): + parser, builder = parse_with_builder(links=False) + text, _, entities = parser.parse("[a](/b)") + self.assertEqual(text, "[a](/b)") + self.assertEqual(entities, []) + + def test_custom_link_resolver(self): + def wagtail(url, label): + if url.startswith("wagtail://"): + return {"type": "DOCUMENT", "data": {"id": 1}} + return None + + parser, builder = parse_with_builder(link_resolvers=[wagtail]) + text, _, entities = parser.parse("[file](wagtail://document?id=1)") + self.assertEqual(builder.entity_map["0"]["type"], "DOCUMENT") + + def test_resolver_deferring_falls_back_to_default(self): + parser, builder = parse_with_builder(link_resolvers=[lambda url, label: None]) + parser.parse("[a](/b)") + self.assertEqual(builder.entity_map["0"]["type"], "LINK") + + +class TestImages(unittest.TestCase): + def test_inline_image(self): + parser, builder = parse_with_builder() + text, _, entities = parser.parse("a  b") + self.assertEqual(text, "a alt b") + self.assertEqual(entities, [{"offset": 2, "length": 3, "key": 0}]) + self.assertEqual( + builder.entity_map["0"], + { + "type": "IMAGE", + "mutability": "IMMUTABLE", + "data": {"src": "/img.jpg", "alt": "alt"}, + }, + ) + + def test_images_disabled(self): + parser, _ = parse_with_builder(images=False) + text, _, entities = parser.parse("") + self.assertEqual(text, "") + self.assertEqual(entities, []) + + def test_resolve_image_entity(self): + parser, builder = parse_with_builder() + key = parser.resolve_image_entity("/x.jpg", "alt text") + self.assertEqual(key, 0) + self.assertEqual(builder.entity_map["0"]["type"], "IMAGE") + + +class TestMalformedConstructs(unittest.TestCase): + def test_unclosed_link_bracket_is_literal(self): + parser, _ = parse_with_builder() + text, _, entities = parser.parse("[a") + self.assertEqual(text, "[a") + self.assertEqual(entities, []) + + def test_unclosed_link_paren_is_literal(self): + parser, _ = parse_with_builder() + text, _, entities = parser.parse("[a](/b") + self.assertEqual(text, "[a](/b") + self.assertEqual(entities, []) + + def test_unclosed_image_is_literal(self): + parser, _ = parse_with_builder() + text, _, entities = parser.parse("![a") + self.assertEqual(text, "![a") + self.assertEqual(entities, []) + + def test_resolver_raising_parse_error_propagates(self): + from draftjs_exporter.error import MarkdownParseError + + def bad(url, label): + raise MarkdownParseError("direct failure") + + parser, _ = parse_with_builder(link_resolvers=[bad]) + with self.assertRaises(MarkdownParseError): + parser.parse("[a](/b)") diff --git a/tests/markdown_parser/test_inline_html.py b/tests/markdown_parser/test_inline_html.py new file mode 100644 index 0000000..e098c2f --- /dev/null +++ b/tests/markdown_parser/test_inline_html.py @@ -0,0 +1,66 @@ +"""Tests for the inline HTML style whitelist.""" + +import unittest + +from tests.markdown_parser.test_inline import make_parser + +SUP_SUB = {"sup": "SUPERSCRIPT", "sub": "SUBSCRIPT"} + + +class TestInlineHtml(unittest.TestCase): + def test_whitelisted_tag_produces_style(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse( + "a 2 b" + ) + self.assertEqual(text, "a 2 b") + self.assertEqual(styles, [{"offset": 2, "length": 1, "style": "SUPERSCRIPT"}]) + + def test_recursive_content(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse( + "**bold**" + ) + self.assertEqual(text, "bold") + self.assertIn({"offset": 0, "length": 4, "style": "SUPERSCRIPT"}, styles) + self.assertIn({"offset": 0, "length": 4, "style": "BOLD"}, styles) + + def test_tag_with_attributes_is_literal(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse( + '2' + ) + self.assertEqual(text, '2') + self.assertEqual(styles, []) + + def test_non_whitelisted_tag_is_literal(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse("bold") + self.assertEqual(text, "bold") + self.assertEqual(styles, []) + + def test_unclosed_tag_is_literal(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse("2") + self.assertEqual(text, "2") + self.assertEqual(styles, []) + + def test_empty_whitelist_means_literal(self): + text, styles, _ = make_parser().parse("2") + self.assertEqual(text, "2") + self.assertEqual(styles, []) + + +class TestNestingDepth(unittest.TestCase): + def test_depth_guard_rejects_excessive_recursion(self): + from draftjs_exporter.error import MarkdownParseError + from draftjs_exporter.markdown_parser.inline import MAX_INLINE_DEPTH + + # The parser's find-first-closer semantics make deep nesting + # unreachable from real input (mis-paired constructs fall back to + # literal text), so the guard is exercised directly. + parser = make_parser(inline_html_styles=SUP_SUB) + with self.assertRaises(MarkdownParseError): + parser._parse("x", depth=MAX_INLINE_DEPTH + 1) + + def test_moderate_nesting_still_parses(self): + text, styles, _ = make_parser(inline_html_styles=SUP_SUB).parse( + "x" + ) + self.assertEqual(text, "x") + self.assertEqual(len(styles), 2) diff --git a/tests/markdown_parser/test_parser.py b/tests/markdown_parser/test_parser.py new file mode 100644 index 0000000..d94af01 --- /dev/null +++ b/tests/markdown_parser/test_parser.py @@ -0,0 +1,53 @@ +"""Tests for the assembled MarkdownParser.""" + +import unittest + +from draftjs_exporter.markdown_parser import MarkdownParser +from draftjs_exporter.markdown_parser.resolvers import scheme_resolver + + +class TestMarkdownParser(unittest.TestCase): + def test_empty_config_uses_defaults(self): + cs = MarkdownParser().parse("# Hi\n\nSome **bold** text.") + self.assertEqual([b["type"] for b in cs["blocks"]], ["header-one", "unstyled"]) + + def test_none_config_uses_defaults(self): + cs = MarkdownParser(None).parse("text") + self.assertEqual(cs["blocks"][0]["text"], "text") + + def test_crlf_normalized(self): + cs = MarkdownParser().parse("a\r\n\r\nb") + self.assertEqual(len(cs["blocks"]), 2) + + def test_non_string_input_raises_type_error(self): + with self.assertRaises(TypeError): + MarkdownParser().parse(None) # type: ignore[arg-type] # ty: ignore[invalid-argument-type] + + def test_config_toggle_passed_through(self): + cs = MarkdownParser({"headings": False}).parse("# Title") + self.assertEqual(cs["blocks"][0]["type"], "unstyled") + + def test_resolvers_passed_through(self): + parser = MarkdownParser( + { + "link_resolvers": [ + scheme_resolver("wagtail", {"page": "LINK"}, coerce={"id": int}) + ] + } + ) + cs = parser.parse("[label](wagtail://page?id=3)") + self.assertEqual(cs["entityMap"]["0"]["data"], {"id": 3}) + + def test_structural_invariants(self): + cs = MarkdownParser().parse( + "# T\n\n- a\n - b\n\n[link](/x) and \n\n---" + ) + keys = [b["key"] for b in cs["blocks"]] + self.assertEqual(len(keys), len(set(keys))) + referenced = {str(r["key"]) for b in cs["blocks"] for r in b["entityRanges"]} + self.assertEqual(set(cs["entityMap"].keys()), referenced) + for block in cs["blocks"]: + text_length = len(block["text"]) + for r in block["inlineStyleRanges"] + block["entityRanges"]: + self.assertGreaterEqual(r["offset"], 0) + self.assertLessEqual(r["offset"] + r["length"], text_length) diff --git a/tests/markdown_parser/test_parser_config.py b/tests/markdown_parser/test_parser_config.py new file mode 100644 index 0000000..4978549 --- /dev/null +++ b/tests/markdown_parser/test_parser_config.py @@ -0,0 +1,39 @@ +"""Tests for parser feature toggles at the block level.""" + +import unittest + +from tests.markdown_parser.test_blocks import block_types, parse + + +class TestBlockToggles(unittest.TestCase): + def test_headings_disabled(self): + cs = parse("# Title", headings=False) + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "# Title") + + def test_blockquote_disabled(self): + cs = parse("> quote", blockquote=False) + self.assertEqual(block_types(cs), ["unstyled"]) + + def test_code_fenced_disabled(self): + cs = parse("```\ncode\n```", code_fenced=False) + self.assertEqual(block_types(cs), ["unstyled"]) + + def test_thematic_break_disabled(self): + cs = parse("a\n---", thematic_break=False) + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "a\n---") + + def test_unordered_list_disabled(self): + cs = parse("- item", unordered_list=False) + self.assertEqual(block_types(cs), ["unstyled"]) + self.assertEqual(cs["blocks"][0]["text"], "- item") + + def test_ordered_list_disabled(self): + cs = parse("1. item", ordered_list=False) + self.assertEqual(block_types(cs), ["unstyled"]) + + def test_disabled_construct_does_not_end_paragraph(self): + cs = parse("text\n# heading", headings=False) + self.assertEqual(len(cs["blocks"]), 1) + self.assertEqual(cs["blocks"][0]["text"], "text\n# heading") diff --git a/tests/markdown_parser/test_parser_errors.py b/tests/markdown_parser/test_parser_errors.py new file mode 100644 index 0000000..faf5638 --- /dev/null +++ b/tests/markdown_parser/test_parser_errors.py @@ -0,0 +1,47 @@ +"""Tests for MarkdownParseError surfaces in the inline parser.""" + +import unittest + +from draftjs_exporter.error import MarkdownParseError +from draftjs_exporter.markdown_parser.builder import ContentStateBuilder +from tests.markdown_parser.test_inline import make_parser + + +class TestResolverErrors(unittest.TestCase): + def test_raising_resolver_wrapped(self): + def bad(url, label): + raise RuntimeError("boom") + + parser = make_parser(builder=ContentStateBuilder(), link_resolvers=[bad]) + with self.assertRaises(MarkdownParseError) as ctx: + parser.parse("[a](/b)") + self.assertIn("boom", str(ctx.exception)) + + def test_resolution_without_type_rejected(self): + parser = make_parser( + builder=ContentStateBuilder(), + link_resolvers=[lambda url, label: {"data": {}}], + ) + with self.assertRaises(MarkdownParseError): + parser.parse("[a](/b)") + + def test_resolution_with_non_dict_data_rejected(self): + parser = make_parser( + builder=ContentStateBuilder(), + link_resolvers=[lambda url, label: {"type": "LINK", "data": "nope"}], + ) + with self.assertRaises(MarkdownParseError): + parser.parse("[a](/b)") + + +class TestLineNumbers(unittest.TestCase): + def test_resolver_error_gets_line_number(self): + from draftjs_exporter.markdown_parser import MarkdownParser + + def bad(url, label): + raise RuntimeError("boom") + + parser = MarkdownParser({"link_resolvers": [bad]}) + with self.assertRaises(MarkdownParseError) as ctx: + parser.parse("first\n\nsecond [a](/b)") + self.assertEqual(ctx.exception.line, 3) diff --git a/tests/markdown_parser/test_resolvers.py b/tests/markdown_parser/test_resolvers.py new file mode 100644 index 0000000..8649095 --- /dev/null +++ b/tests/markdown_parser/test_resolvers.py @@ -0,0 +1,61 @@ +"""Tests for entity resolver chains and default resolvers.""" + +import unittest + +from draftjs_exporter.markdown_parser.resolvers import ( + default_image_resolver, + default_link_resolver, + resolve, +) + + +class TestDefaultLinkResolver(unittest.TestCase): + def test_returns_link_entity(self): + self.assertEqual( + default_link_resolver("https://example.com", "example"), + { + "type": "LINK", + "data": {"url": "https://example.com"}, + "mutability": "MUTABLE", + }, + ) + + +class TestDefaultImageResolver(unittest.TestCase): + def test_returns_image_entity(self): + self.assertEqual( + default_image_resolver("/media/a.jpg", "an alt"), + { + "type": "IMAGE", + "data": {"src": "/media/a.jpg", "alt": "an alt"}, + "mutability": "IMMUTABLE", + }, + ) + + +class TestResolve(unittest.TestCase): + def test_empty_chain_uses_default(self): + result = resolve([], "/x", "lbl", default_link_resolver) + self.assertEqual(result["data"], {"url": "/x"}) + + def test_first_match_wins(self): + calls = [] + + def first(url, label): + calls.append("first") + return {"type": "DOCUMENT", "data": {"id": 1}} + + def second(url, label): + calls.append("second") + return {"type": "LINK", "data": {}} + + result = resolve([first, second], "/x", "lbl", default_link_resolver) + self.assertEqual(result["type"], "DOCUMENT") + self.assertEqual(calls, ["first"]) + + def test_none_defers_to_next(self): + def defer(url, label): + return None + + result = resolve([defer], "/x", "lbl", default_link_resolver) + self.assertEqual(result["type"], "LINK") diff --git a/tests/markdown_parser/test_scheme_resolver.py b/tests/markdown_parser/test_scheme_resolver.py new file mode 100644 index 0000000..b6114b4 --- /dev/null +++ b/tests/markdown_parser/test_scheme_resolver.py @@ -0,0 +1,84 @@ +"""Tests for the scheme_resolver helper.""" + +import unittest + +from draftjs_exporter.markdown_parser.resolvers import ( + EntityResolution, + EntityResolver, + scheme_resolver, +) + + +def must_resolve(resolver: EntityResolver, url: str, label: str) -> EntityResolution: + """Resolve a URL, failing the test when the resolver defers.""" + result = resolver(url, label) + if result is None: + raise AssertionError(f"Resolver unexpectedly deferred for {url!r}") + return result + + +class TestSchemeResolver(unittest.TestCase): + def setUp(self): + self.resolve = scheme_resolver( + "wagtail", + {"page": "LINK", "document": "DOCUMENT", "image": "IMAGE"}, + coerce={"id": int}, + label_key="alt", + ) + + def test_scheme_mismatch_returns_none(self): + self.assertIsNone(self.resolve("https://example.com", "x")) + + def test_unmapped_host_returns_none(self): + self.assertIsNone(self.resolve("wagtail://unknown?id=1", "x")) + + def test_host_maps_to_entity_type(self): + # Link resolvers are configured without label_key: the label + # stays link text and never leaks into entity data. + resolver = scheme_resolver( + "wagtail", {"page": "LINK", "document": "DOCUMENT"}, coerce={"id": int} + ) + result = must_resolve(resolver, "wagtail://page?id=3", "label") + self.assertEqual(result["type"], "LINK") + self.assertEqual(result["data"], {"id": 3}) + + def test_query_params_become_data(self): + result = must_resolve( + self.resolve, "wagtail://image?id=10&alt=alt&format=left", "alt" + ) + self.assertEqual(result["data"], {"id": 10, "alt": "alt", "format": "left"}) + + def test_label_fills_label_key_when_absent(self): + result = must_resolve(self.resolve, "wagtail://image?id=10", "my alt") + self.assertEqual(result["data"], {"id": 10, "alt": "my alt"}) + + def test_label_does_not_override_query_param(self): + result = must_resolve( + self.resolve, "wagtail://image?id=10&alt=fromquery", "frommd" + ) + self.assertEqual(result["data"]["alt"], "fromquery") + + def test_empty_label_not_injected(self): + result = must_resolve(self.resolve, "wagtail://image?id=10", "") + self.assertNotIn("alt", result["data"]) + + def test_percent_decoding(self): + result = must_resolve( + self.resolve, "wagtail://page?id=1&url=https%3A%2F%2Fa.b%2F", "x" + ) + self.assertEqual(result["data"]["url"], "https://a.b/") + + def test_coercion_error_raises_value_error(self): + with self.assertRaises(ValueError): + self.resolve("wagtail://page?id=abc", "x") + + def test_default_mutability(self): + result = must_resolve(self.resolve, "wagtail://page?id=3", "x") + self.assertEqual(result["mutability"], "MUTABLE") + + def test_custom_mutability(self): + resolver = scheme_resolver( + "wagtail", {"image": "IMAGE"}, mutability="IMMUTABLE" + ) + result = must_resolve(resolver, "wagtail://image?id=1", "x") + self.assertEqual(result["mutability"], "IMMUTABLE") diff --git a/tests/strategies.py b/tests/strategies.py index f3d3001..33c83e9 100644 --- a/tests/strategies.py +++ b/tests/strategies.py @@ -205,3 +205,73 @@ def dangerous_content_states(draw: st.DrawFn) -> dict[str, Any]: } ], } + + +# Text that survives a Markdown export → import round-trip unchanged: +# no Markdown-special characters, no leading/trailing whitespace quirks. +safe_block_text = ( + st.text( + alphabet=st.characters( + whitelist_categories=["Ll", "Lu", "Nd"], whitelist_characters=" " + ), + min_size=1, + max_size=20, + ) + .map(str.strip) + .filter(bool) +) + +ROUNDTRIP_BLOCK_TYPES = [ + BLOCK_TYPES.UNSTYLED, + BLOCK_TYPES.HEADER_ONE, + BLOCK_TYPES.BLOCKQUOTE, + BLOCK_TYPES.UNORDERED_LIST_ITEM, + BLOCK_TYPES.ORDERED_LIST_ITEM, +] + +# CODE is excluded: the Markdown exporter cannot represent code spans +# overlapping other styles (the other style's markers end up inside the +# span's literal content), so such ranges cannot round-trip. +ROUNDTRIP_STYLES = [INLINE_STYLES.BOLD, INLINE_STYLES.ITALIC] + + +@st.composite +def roundtrip_blocks(draw: st.DrawFn) -> dict[str, Any]: + """A block whose type, text, and styles survive a Markdown round-trip. + + Style ranges use unique styles per block: real Draft.js editors + merge same-style ranges, so duplicates and same-style overlaps + never occur in practice (and export as literal marker runs). + """ + text = draw(safe_block_text) + styles = draw(st.lists(st.sampled_from(ROUNDTRIP_STYLES), unique=True, max_size=2)) + style_ranges = [] + for style in styles: + # Zero-length ranges are not realistic Draft.js content, and + # export as empty marker runs that cannot be re-imported. + offset = draw(st.integers(min_value=0, max_value=len(text) - 1)) + length = draw(st.integers(min_value=1, max_value=len(text) - offset)) + style_ranges.append({"offset": offset, "length": length, "style": style}) + return { + "key": draw( + st.text( + alphabet=st.characters(min_codepoint=97, max_codepoint=122), + min_size=5, + max_size=5, + ) + ), + "text": text, + "type": draw(st.sampled_from(ROUNDTRIP_BLOCK_TYPES)), + "depth": 0, + "inlineStyleRanges": style_ranges, + "entityRanges": [], + } + + +@st.composite +def roundtrip_content_states(draw: st.DrawFn) -> dict[str, Any]: + """Content states limited to constructs both engines support.""" + return { + "entityMap": {}, + "blocks": draw(st.lists(roundtrip_blocks(), min_size=0, max_size=4)), + } diff --git a/tests/test_error.py b/tests/test_error.py new file mode 100644 index 0000000..bf436f2 --- /dev/null +++ b/tests/test_error.py @@ -0,0 +1,21 @@ +"""Tests for exporter exception types.""" + +import unittest + +from draftjs_exporter.error import ExporterException, MarkdownParseError + + +class TestMarkdownParseError(unittest.TestCase): + def test_message_only(self): + err = MarkdownParseError("bad input") + self.assertEqual(str(err), "bad input") + self.assertEqual(err.message, "bad input") + self.assertIsNone(err.line) + + def test_with_line(self): + err = MarkdownParseError("bad input", line=7) + self.assertEqual(str(err), "line 7: bad input") + self.assertEqual(err.line, 7) + + def test_is_exporter_exception(self): + self.assertIsInstance(MarkdownParseError("x"), ExporterException) diff --git a/tests/test_exports.json b/tests/test_exports.json index 3eaa790..8f009f6 100644 --- a/tests/test_exports.json +++ b/tests/test_exports.json @@ -151,6 +151,35 @@ "entityRanges": [] } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "Bold Italic", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 0, + "length": 5, + "style": "BOLD" + }, + { + "offset": 5, + "length": 6, + "style": "ITALIC" + }, + { + "offset": 5, + "length": 6, + "style": "BOLD" + } + ], + "entityRanges": [] + } + ], + "entityMap": {} } }, { @@ -329,6 +358,85 @@ "entityRanges": [] } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "0123~4~56789abcdef", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 0, + "length": 1, + "style": "BOLD" + }, + { + "offset": 1, + "length": 1, + "style": "CODE" + }, + { + "offset": 2, + "length": 1, + "style": "ITALIC" + }, + { + "offset": 3, + "length": 1, + "style": "UNDERLINE" + }, + { + "offset": 7, + "length": 1, + "style": "SUPERSCRIPT" + }, + { + "offset": 8, + "length": 1, + "style": "SUBSCRIPT" + }, + { + "offset": 9, + "length": 1, + "style": "MARK" + }, + { + "offset": 10, + "length": 1, + "style": "QUOTATION" + }, + { + "offset": 11, + "length": 1, + "style": "SMALL" + }, + { + "offset": 12, + "length": 1, + "style": "SAMPLE" + }, + { + "offset": 13, + "length": 1, + "style": "INSERT" + }, + { + "offset": 14, + "length": 1, + "style": "DELETE" + }, + { + "offset": 15, + "length": 1, + "style": "KEYBOARD" + } + ], + "entityRanges": [] + } + ], + "entityMap": {} } }, { @@ -374,6 +482,39 @@ ] } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 0, + "length": 1, + "style": "ITALIC" + } + ], + "entityRanges": [ + { + "offset": 0, + "length": 1, + "key": 0 + } + ] + } + ], + "entityMap": { + "0": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "/" + } + } + } } }, { @@ -424,6 +565,39 @@ ] } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 0, + "length": 1, + "style": "ITALIC" + } + ], + "entityRanges": [ + { + "offset": 0, + "length": 1, + "key": 0 + } + ] + } + ], + "entityMap": { + "0": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "/" + } + } + } } }, { @@ -717,6 +891,106 @@ ] } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "DraftJS AST Exporter", + "type": "header-two", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00001", + "text": "In your draft-js, exporting your content:", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 18, + "length": 9, + "style": "BOLD" + }, + { + "offset": 33, + "length": 7, + "style": "ITALIC" + } + ], + "entityRanges": [] + }, + { + "key": "00002", + "text": "From draft-js internals", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00003", + "text": "To an abstract syntax tree", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00004", + "text": "Extensibility.", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00005", + "text": " ", + "type": "atomic", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 0, + "length": 1, + "key": 0 + } + ] + }, + { + "key": "00006", + "text": ":)Find the project on Github.", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 22, + "length": 6, + "key": 1 + } + ] + } + ], + "entityMap": { + "1": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "https://github.com/icelab/draft-js-ast-exporter" + } + }, + "0": { + "type": "IMAGE", + "mutability": "IMMUTABLE", + "data": { + "src": "http://placekitten.com/500/300", + "alt": "" + } + } + } } }, { @@ -777,11 +1051,38 @@ "entityRanges": [] } ] - } - }, - { - "label": "Big content export", - "output": { + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "search http://www.google.com#world for the #world", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 7, + "length": 27, + "key": 0 + } + ] + } + ], + "entityMap": { + "0": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "http://www.google.com#world" + } + } + } + } + }, + { + "label": "Big content export", + "output": { "html5lib": "draftjs_exporter is an HTML exporter for Draft.js content
Try it out by \"running\" this file!Features 📝🍸
The exporter aims to provide sensible defaults from basic block types and inline styles to HTML, that can easily be customised when required. For more advanced scenarios, an API is provided (mimicking React's
createElement) to create custom rendering components of arbitrary complexity.Here are some features worth highlighting:
- Convert line breaks to
<br>
elements.- Automatic conversion of entity data to HTML attributes (int & boolean to string,
style objecttostyle string).- Wrapped blocks (
<li>elements go inside<ul>or<ol>).
- With arbitrary nesting.
- Common text styles: Bold, Italic, Underline,
Monospace,Strikethrough.cmd + bOverlappingtext styles. Custom styles too!
- #hashtag support via #CompositeDecorators.
- Linkify URLs too! http://example.com/
- Depth can go back and forth, it works fiiine (1)
- Depth can go back and forth, it works fiiine (2)
- Depth can go back and forth, it works fiiine (3)
- Depth can go back and forth, it works fiiine (4)
- Depth can go back and forth, it works fiiine (5)
For developers 🚀
- Import the library
- Define your configuration
- Go!
- Optionally, define your custom components.
def Blockquote(props):\n block_data = props['block']['data']\n return DOM.create_element('blockquote', {\n 'cite': block_data.get('cite')\n }, props['children'])\nVoilà!
", "lxml": "draftjs_exporter is an HTML exporter for Draft.js content
Try it out by \"running\" this file!Features 📝🍸
The exporter aims to provide sensible defaults from basic block types and inline styles to HTML, that can easily be customised when required. For more advanced scenarios, an API is provided (mimicking React's
createElement) to create custom rendering components of arbitrary complexity.Here are some features worth highlighting:
- Convert line breaks to
<br>
elements.- Automatic conversion of entity data to HTML attributes (int & boolean to string,
style objecttostyle string).- Wrapped blocks (
<li>elements go inside<ul>or<ol>).
- With arbitrary nesting.
- Common text styles: Bold, Italic, Underline,
Monospace,Strikethrough.cmd + bOverlappingtext styles. Custom styles too!
- #hashtag support via #CompositeDecorators.
- Linkify URLs too! http://example.com/
- Depth can go back and forth, it works fiiine (1)
- Depth can go back and forth, it works fiiine (2)
- Depth can go back and forth, it works fiiine (3)
- Depth can go back and forth, it works fiiine (4)
- Depth can go back and forth, it works fiiine (5)
For developers 🚀
- Import the library
- Define your configuration
- Go!
- Optionally, define your custom components.
def Blockquote(props):\n block_data = props['block']['data']\n return DOM.create_element('blockquote', {\n 'cite': block_data.get('cite')\n }, props['children'])\nVoilà!
", "string": "draftjs_exporter is an HTML exporter for Draft.js content
Try it out by \"running\" this file!Features 📝🍸
The exporter aims to provide sensible defaults from basic block types and inline styles to HTML, that can easily be customised when required. For more advanced scenarios, an API is provided (mimicking React's
createElement) to create custom rendering components of arbitrary complexity.Here are some features worth highlighting:
- Convert line breaks to
<br>
elements.- Automatic conversion of entity data to HTML attributes (int & boolean to string,
style objecttostyle string).- Wrapped blocks (
<li>elements go inside<ul>or<ol>).
- With arbitrary nesting.
- Common text styles: Bold, Italic, Underline,
Monospace,Strikethrough.cmd + bOverlappingtext styles. Custom styles too!
- #hashtag support via #CompositeDecorators.
- Linkify URLs too! http://example.com/
- Depth can go back and forth, it works fiiine (1)
- Depth can go back and forth, it works fiiine (2)
- Depth can go back and forth, it works fiiine (3)
- Depth can go back and forth, it works fiiine (4)
- Depth can go back and forth, it works fiiine (5)
For developers 🚀
- Import the library
- Define your configuration
- Go!
- Optionally, define your custom components.
def Blockquote(props):\n block_data = props['block']['data']\n return DOM.create_element('blockquote', {\n 'cite': block_data.get('cite')\n }, props['children'])\nVoilà!
", @@ -1241,6 +1542,412 @@ "data": {} } ] + }, + "import": { + "blocks": [ + { + "key": "00000", + "text": "draftjs_exporter is an HTML exporter for Draft.js content", + "type": "header-two", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 41, + "length": 8, + "key": 0 + } + ] + }, + { + "key": "00001", + "text": "Try it out by \"running\" this file!", + "type": "blockquote", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00002", + "text": "Features 📝🍸", + "type": "header-three", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00003", + "text": "The exporter aims to provide sensible defaults from basic block types and inline styles to HTML, that can easily be customised when required. For more advanced scenarios, an API is provided (mimicking React's createElement) to create custom rendering components of arbitrary complexity.", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 209, + "length": 13, + "style": "CODE" + } + ], + "entityRanges": [ + { + "offset": 209, + "length": 13, + "key": 1 + } + ] + }, + { + "key": "00004", + "text": " ", + "type": "atomic", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 0, + "length": 1, + "key": 2 + } + ] + }, + { + "key": "00005", + "text": "Here are some features worth highlighting:", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00006", + "text": "Convert line breaks to
", + "type": "unordered-list-item", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 23, + "length": 4, + "style": "CODE" + } + ], + "entityRanges": [] + }, + { + "key": "00007", + "text": "elements.", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00008", + "text": "Automatic conversion of entity data to HTML attributes (int & boolean to string, style object to style string).", + "type": "unordered-list-item", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 81, + "length": 12, + "style": "CODE" + }, + { + "offset": 97, + "length": 12, + "style": "CODE" + } + ], + "entityRanges": [ + { + "offset": 81, + "length": 28, + "key": 3 + } + ] + }, + { + "key": "00009", + "text": "Wrapped blocks (elements go inside or
).", + "type": "unordered-list-item", + "depth": 0, + "inlineStyleRanges": [ + { + "offset": 16, + "length": 5, + "style": "CODE" + }, + { + "offset": 40, + "length": 4, + "style": "CODE" + }, + { + "offset": 48, + "length": 4, + "style": "CODE" + } + ], + "entityRanges": [] + }, + { + "key": "00010", + "text": "With arbitrary nesting.", + "type": "unordered-list-item", + "depth": 1, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00011", + "text": "Common text styles: Bold, Italic, Underline, Monospace, ~Strikethrough.~ cmd + b", + "type": "unordered-list-item", + "depth": 2, + "inlineStyleRanges": [ + { + "offset": 20, + "length": 4, + "style": "BOLD" + }, + { + "offset": 26, + "length": 6, + "style": "ITALIC" + }, + { + "offset": 34, + "length": 9, + "style": "UNDERLINE" + }, + { + "offset": 45, + "length": 9, + "style": "CODE" + }, + { + "offset": 73, + "length": 7, + "style": "KEYBOARD" + } + ], + "entityRanges": [] + }, + { + "key": "00012", + "text": "~Overlapping ~~te~xt styles. Custom styles too!", + "type": "unordered-list-item", + "depth": 2, + "inlineStyleRanges": [ + { + "offset": 14, + "length": 4, + "style": "BOLD" + }, + { + "offset": 18, + "length": 2, + "style": "ITALIC" + }, + { + "offset": 18, + "length": 2, + "style": "BOLD" + }, + { + "offset": 20, + "length": 9, + "style": "ITALIC" + } + ], + "entityRanges": [] + }, + { + "key": "00013", + "text": "#hashtag support via #CompositeDecorators.", + "type": "unordered-list-item", + "depth": 3, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 21, + "length": 20, + "key": 4 + } + ] + }, + { + "key": "00014", + "text": "Linkify URLs too! http://example.com/", + "type": "unordered-list-item", + "depth": 4, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 18, + "length": 19, + "key": 5 + } + ] + }, + { + "key": "00015", + "text": "Depth can go back and forth, it works fiiine (1)", + "type": "unordered-list-item", + "depth": 2, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00016", + "text": "Depth can go back and forth, it works fiiine (2)", + "type": "unordered-list-item", + "depth": 1, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00017", + "text": "Depth can go back and forth, it works fiiine (3)", + "type": "unordered-list-item", + "depth": 2, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00018", + "text": "Depth can go back and forth, it works fiiine (4)", + "type": "unordered-list-item", + "depth": 1, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00019", + "text": "Depth can go back and forth, it works fiiine (5)", + "type": "unordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00020", + "text": " ", + "type": "atomic", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [ + { + "offset": 0, + "length": 1, + "key": 6 + } + ] + }, + { + "key": "00021", + "text": "For developers 🚀", + "type": "header-three", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00022", + "text": "Import the library", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00023", + "text": "Define your configuration", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00024", + "text": "Go!", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00025", + "text": "Optionally, define your custom components.", + "type": "ordered-list-item", + "depth": 1, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00026", + "text": "def Blockquote(props):\n block_data = props['block']['data']\n return DOM.create_element('blockquote', {\n 'cite': block_data.get('cite')\n }, props['children'])\n", + "type": "code-block", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + }, + { + "key": "00027", + "text": "Voilà!", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + } + ], + "entityMap": { + "2": { + "type": "HORIZONTAL_RULE", + "mutability": "IMMUTABLE", + "data": {} + }, + "3": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "https://facebook.github.io/react/docs/jsx-in-depth.html" + } + }, + "0": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "https://github.com/facebook/draft-js" + } + }, + "5": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "http://example.com/" + } + }, + "6": { + "type": "IMAGE", + "mutability": "IMMUTABLE", + "data": { + "src": "https://placekitten.com/g/300/200", + "alt": "Test image alt text" + } + }, + "4": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "https://github.com/wagtail/draftjs_exporter/pull/17" + } + }, + "1": { + "type": "LINK", + "mutability": "MUTABLE", + "data": { + "url": "https://facebook.github.io/react/docs/top-level-api.html#react.createelement" + } + } + } } }, { diff --git a/tests/test_imports.json b/tests/test_imports.json new file mode 100644 index 0000000..b30439b --- /dev/null +++ b/tests/test_imports.json @@ -0,0 +1,200 @@ +[ + { + "label": "Italic with stars", + "markdown": "*a*", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { "offset": 0, "length": 1, "style": "ITALIC" } + ], + "entityRanges": [] + } + ] + } + }, + { + "label": "Bold with underscores", + "markdown": "__a__", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [{ "offset": 0, "length": 1, "style": "BOLD" }], + "entityRanges": [] + } + ] + } + }, + { + "label": "Plus bullet list", + "markdown": "+ a", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "unordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + } + ] + } + }, + { + "label": "Paren ordered list", + "markdown": "1) a", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "a", + "type": "ordered-list-item", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + } + ] + } + }, + { + "label": "Tilde fence", + "markdown": "~~~\nx\n~~~", + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "x", + "type": "code-block", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + } + ] + } + }, + { + "label": "Star thematic break", + "markdown": "***", + "content_state": { + "entityMap": { + "0": { + "type": "HORIZONTAL_RULE", + "mutability": "IMMUTABLE", + "data": {} + } + }, + "blocks": [ + { + "key": "00000", + "text": " ", + "type": "atomic", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [{ "offset": 0, "length": 1, "key": 0 }] + } + ] + } + }, + { + "label": "Inline HTML styles", + "markdown": "x 2", + "config": { + "parser_config": { "inline_html_styles": { "sup": "SUPERSCRIPT" } } + }, + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "x 2", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [ + { "offset": 2, "length": 1, "style": "SUPERSCRIPT" } + ], + "entityRanges": [] + } + ] + } + }, + { + "label": "Wagtail internal image", + "markdown": "", + "config": { "wagtail_resolvers": true }, + "content_state": { + "entityMap": { + "0": { + "type": "IMAGE", + "mutability": "IMMUTABLE", + "data": { "id": 10, "format": "left", "alt": "alt" } + } + }, + "blocks": [ + { + "key": "00000", + "text": " ", + "type": "atomic", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [{ "offset": 0, "length": 1, "key": 0 }] + } + ] + } + }, + { + "label": "Wagtail page link", + "markdown": "[label](wagtail://page?id=3)", + "config": { "wagtail_resolvers": true }, + "content_state": { + "entityMap": { + "0": { "type": "LINK", "mutability": "MUTABLE", "data": { "id": 3 } } + }, + "blocks": [ + { + "key": "00000", + "text": "label", + "type": "unstyled", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [{ "offset": 0, "length": 5, "key": 0 }] + } + ] + } + }, + { + "label": "Filter demote in importer", + "markdown": "# T", + "config": { + "filter_rules": [ + { "type": "block", "match": "header-one", "action": "demote" } + ] + }, + "content_state": { + "entityMap": {}, + "blocks": [ + { + "key": "00000", + "text": "T", + "type": "header-two", + "depth": 0, + "inlineStyleRanges": [], + "entityRanges": [] + } + ] + } + } +] diff --git a/tests/test_imports.py b/tests/test_imports.py new file mode 100644 index 0000000..88c1ce3 --- /dev/null +++ b/tests/test_imports.py @@ -0,0 +1,140 @@ +import json +import os +import unittest +from typing import Any + +from draftjs_exporter.markdown_importer import ImporterConfig, MarkdownImporter +from draftjs_exporter.markdown_parser import scheme_resolver +from draftjs_exporter.types import ContentState + +fixtures_path = os.path.join(os.path.dirname(__file__), "test_exports.json") +with open(fixtures_path) as f: + export_fixtures = json.loads(f.read()) + +imports_path = os.path.join(os.path.dirname(__file__), "test_imports.json") +with open(imports_path) as f: + import_fixtures = json.loads(f.read()) + +# Tags the Markdown exporter emits as inline HTML fallback (see the +# exporter's markdown fallbacks module). Whitelisting them lets most +# styles round-trip. +ROUNDTRIP_INLINE_HTML_STYLES = { + "u": "UNDERLINE", + "sup": "SUPERSCRIPT", + "sub": "SUBSCRIPT", + "mark": "MARK", + "q": "QUOTATION", + "small": "SMALL", + "samp": "SAMPLE", + "ins": "INSERT", + "del": "DELETE", + "kbd": "KEYBOARD", +} + + +def make_importer() -> MarkdownImporter: + """Build the importer used for round-trip snapshot tests.""" + return MarkdownImporter( + { + "parser_config": { + "inline_html_styles": ROUNDTRIP_INLINE_HTML_STYLES, + } + } + ) + + +def normalize(content_state: ContentState) -> dict[str, Any]: + """Rewrite block and entity keys for deterministic comparison. + + Block keys become sequential; entity keys are remapped in order of + first appearance in entity ranges, and the entity map is rebuilt + to match. + """ + entity_map = content_state.get("entityMap", {}) + key_map: dict[str, str] = {} + blocks = [] + for index, block in enumerate(content_state.get("blocks", [])): + ranges = [] + for entity_range in block.get("entityRanges", []): + old_key = str(entity_range["key"]) + if old_key not in key_map: + key_map[old_key] = str(len(key_map)) + ranges.append({**entity_range, "key": int(key_map[old_key])}) + # Sort ranges canonically: Draft.js does not require a specific + # order, and the exporter emits ranges in style-application order + # while the importer emits them in source order. + styles = sorted( + block.get("inlineStyleRanges", []), + key=lambda r: (r["offset"], r["length"], r["style"]), + ) + ranges.sort(key=lambda r: (r["offset"], r["length"], r["key"])) + blocks.append( + { + "key": f"{index:05d}", + "text": block.get("text", ""), + "type": block.get("type", "unstyled"), + "depth": block.get("depth", 0), + "inlineStyleRanges": styles, + "entityRanges": ranges, + **({"data": block["data"]} if block.get("data") else {}), + } + ) + new_map = {} + for old_key, new_key in key_map.items(): + if old_key in entity_map: + new_map[new_key] = entity_map[old_key] + return {"blocks": blocks, "entityMap": new_map} + + +def build_fixture_importer(fixture: dict[str, Any]) -> MarkdownImporter: + """Build an importer for a direct-import fixture, expanding shorthands.""" + config = dict(fixture.get("config", {})) + if config.pop("wagtail_resolvers", False): + config["parser_config"] = { + **config.get("parser_config", {}), + "link_resolvers": [ + scheme_resolver( + "wagtail", + {"page": "LINK", "document": "DOCUMENT"}, + coerce={"id": int}, + ) + ], + "image_resolvers": [ + scheme_resolver( + "wagtail", + {"image": "IMAGE", "media": "EMBED"}, + coerce={"id": int}, + label_key="alt", + mutability="IMMUTABLE", + ) + ], + } + return MarkdownImporter(ImporterConfig(**config)) + + +class TestRoundTrip(unittest.TestCase): + """Import the recorded Markdown output of every export fixture.""" + + def test_round_trip(self) -> None: + importer = make_importer() + for fixture in export_fixtures: + markdown = fixture["output"]["markdown"] + expected = fixture.get("import", fixture["content_state"]) + with self.subTest(fixture=fixture["label"]): + self.assertEqual( + normalize(importer.import_markdown(markdown)), + normalize(expected), + ) + + +class TestDirectImports(unittest.TestCase): + """Import hand-written Markdown covering importer-only behavior.""" + + def test_imports(self) -> None: + for fixture in import_fixtures: + importer = build_fixture_importer(fixture) + with self.subTest(fixture=fixture["label"]): + self.assertEqual( + normalize(importer.import_markdown(fixture["markdown"])), + normalize(fixture["content_state"]), + ) diff --git a/tests/test_properties.py b/tests/test_properties.py index d5ad964..852bf9d 100644 --- a/tests/test_properties.py +++ b/tests/test_properties.py @@ -22,12 +22,18 @@ from bs4 import BeautifulSoup from hypothesis import example, given, settings -from draftjs_exporter.constants import ENTITY_TYPES +from draftjs_exporter.constants import BLOCK_TYPES, ENTITY_TYPES, INLINE_STYLES +from draftjs_exporter.contentstate_filter import ContentStateFilter from draftjs_exporter.defaults import BLOCK_MAP, STYLE_MAP from draftjs_exporter.dom import DOM from draftjs_exporter.html import HTML, ExporterConfig from draftjs_exporter.markdown import CONFIG as MARKDOWN_CONFIG -from tests.strategies import content_states, dangerous_content_states +from draftjs_exporter.markdown_importer import MarkdownImporter +from tests.strategies import ( + content_states, + dangerous_content_states, + roundtrip_content_states, +) from tests.test_entities import link CONFIG: ExporterConfig = { @@ -166,3 +172,47 @@ def test_dangerous_fragments_never_become_markup(self, content_state): self.assertEqual(links[0].get_text(), text) else: self.assertEqual(parsed.get_text(), text) + + +class TestImporterProperties(unittest.TestCase): + @given(roundtrip_content_states()) + def test_round_trip_preserves_block_types_and_text(self, content_state): + """Export → import keeps block types and text for safe content.""" + markdown = HTML(MARKDOWN_CONFIG).render(content_state) + result = MarkdownImporter().import_markdown(markdown) + self.assertEqual( + [b["type"] for b in result["blocks"]], + [b["type"] for b in content_state["blocks"]], + ) + self.assertEqual( + [b["text"] for b in result["blocks"]], + [b["text"] for b in content_state["blocks"]], + ) + + @given(content_states()) + def test_filter_produces_valid_content_state(self, content_state): + """Filtered output never has orphaned entity ranges.""" + filter_ = ContentStateFilter( + [{"type": "block", "match": BLOCK_TYPES.HEADER_ONE, "action": "remove"}] + ) + result = filter_.apply(content_state) + entity_map = result.get("entityMap", {}) + for block in result["blocks"]: + text_length = len(block.get("text", "")) + for r in block.get("entityRanges", []): + self.assertIn(str(r["key"]), entity_map) + self.assertLessEqual(r["offset"] + r["length"], text_length) + for style_range in block.get("inlineStyleRanges", []): + self.assertLessEqual( + style_range["offset"] + style_range["length"], text_length + ) + + @given(content_states()) + def test_filter_idempotent(self, content_state): + """Applying the same filter twice yields the same result.""" + filter_ = ContentStateFilter( + [{"type": "inline_style", "match": INLINE_STYLES.BOLD, "action": "remove"}] + ) + once = filter_.apply(content_state) + twice = filter_.apply(once) + self.assertEqual(once, twice)