docs: generate diagrams for scaffolded machines

[gabewillen]

Summary

• fix docsgen discovery to register every enabled struct model machine instead of relying on using model_type = model;
• re-enable diagram generation for the scaffolded kernel backend machines and text/formatter
• regenerate architecture docs/mermaid output, add missing machine docs, and prune stale alias-wrapper docs

Validation

• build/docsgen/docsgen --root /Users/gabrielwillen/.codex/worktrees/6bc4/emel.cpp --check
• scripts/quality_gates.sh

[Copilot]

Pull request overview

Updates the docs generation pipeline to discover and document all enabled state-machine models (including scaffolded machines), regenerating the corresponding architecture markdown and Mermaid diagrams while pruning stale outputs.

Changes:

• Adjust machine discovery to register struct model machines directly (rather than relying on using model_type = model;) and re-enable docs for scaffolded kernel backends and text/formatter.
• Add stale-output pruning for generated docs/architecture/*.md and docs/architecture/mermaid/*.mmd.
• Regenerate architecture docs/diagrams, add missing machine docs, and remove stale alias-wrapper docs.

Reviewed changes

Copilot reviewed 64 out of 110 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tools/docsgen/docsgen.cpp	Adds stale generated-file pruning prior to emitting docs/diagrams.
tools/docsgen/CMakeLists.txt	Updates discovery/filtering to include enabled headers that define struct model { ... } and registers emel::<ns>::model.
src/emel/text/formatter/sm.hpp	Re-enables docs generation for text/formatter by removing the docs: disabled marker.
src/emel/kernel/x86_64/sm.hpp	Re-enables docs generation for scaffolded x86_64 kernel machine by removing the docs: disabled marker.
src/emel/kernel/wasm/sm.hpp	Re-enables docs generation for scaffolded wasm kernel machine by removing the docs: disabled marker.
src/emel/kernel/vulkan/sm.hpp	Re-enables docs generation for scaffolded vulkan kernel machine by removing the docs: disabled marker.
src/emel/kernel/metal/sm.hpp	Re-enables docs generation for scaffolded metal kernel machine by removing the docs: disabled marker.
src/emel/kernel/cuda/sm.hpp	Re-enables docs generation for scaffolded cuda kernel machine by removing the docs: disabled marker.
src/emel/kernel/aarch64/sm.hpp	Re-enables docs generation for scaffolded aarch64 kernel machine by removing the docs: disabled marker.
docs/architecture/text_formatter.md	Adds generated architecture markdown for text_formatter.
docs/architecture/tensor_view.md	Adds generated architecture markdown for tensor_view.
docs/architecture/parser_gguf.md	Removes stale architecture doc for deprecated/aliased machine.
docs/architecture/mermaid/token_batcher.mmd	Updates generated Mermaid diagram output for token_batcher.
docs/architecture/mermaid/text_tokenizer_preprocessor_wpm.mmd	Updates generated Mermaid diagram output for WPM preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_ugm.mmd	Updates generated Mermaid diagram output for UGM preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_spm.mmd	Updates generated Mermaid diagram output for SPM preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_rwkv.mmd	Updates generated Mermaid diagram output for RWKV preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_plamo2.mmd	Updates generated Mermaid diagram output for Plamo2 preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_fallback.mmd	Updates generated Mermaid diagram output for fallback preprocessor.
docs/architecture/mermaid/text_tokenizer_preprocessor_bpe.mmd	Updates generated Mermaid diagram output for BPE preprocessor.
docs/architecture/mermaid/text_tokenizer.mmd	Updates generated Mermaid diagram output for text_tokenizer.
docs/architecture/mermaid/text_renderer.mmd	Updates generated Mermaid diagram output for text_renderer.
docs/architecture/mermaid/text_jinja_parser.mmd	Updates generated Mermaid diagram output for text_jinja_parser.
docs/architecture/mermaid/text_jinja_formatter.mmd	Updates generated Mermaid diagram output for text_jinja_formatter.
docs/architecture/mermaid/text_formatter.mmd	Adds generated Mermaid diagram output for text_formatter.
docs/architecture/mermaid/text_encoders_wpm.mmd	Updates generated Mermaid diagram output for WPM encoder.
docs/architecture/mermaid/text_encoders_ugm.mmd	Updates generated Mermaid diagram output for UGM encoder.
docs/architecture/mermaid/text_encoders_spm.mmd	Updates generated Mermaid diagram output for SPM encoder.
docs/architecture/mermaid/text_encoders_rwkv.mmd	Updates generated Mermaid diagram output for RWKV encoder.
docs/architecture/mermaid/text_encoders_plamo2.mmd	Updates generated Mermaid diagram output for Plamo2 encoder.
docs/architecture/mermaid/text_encoders_fallback.mmd	Updates generated Mermaid diagram output for fallback encoder.
docs/architecture/mermaid/text_encoders_bpe.mmd	Updates generated Mermaid diagram output for BPE encoder.
docs/architecture/mermaid/text_encoders.mmd	Removes stale alias-wrapper Mermaid output.
docs/architecture/mermaid/text_detokenizer.mmd	Updates generated Mermaid diagram output for text_detokenizer.
docs/architecture/mermaid/text_conditioner.mmd	Updates generated Mermaid diagram output for text_conditioner.
docs/architecture/mermaid/tensor_view.mmd	Adds generated Mermaid diagram output for tensor_view.
docs/architecture/mermaid/tensor.mmd	Adds generated Mermaid diagram output for tensor.
docs/architecture/mermaid/parser_gguf.mmd	Removes stale Mermaid diagram output for deprecated/aliased machine.
docs/architecture/mermaid/model_weight_loader.mmd	Updates generated Mermaid diagram output for model_weight_loader.
docs/architecture/mermaid/model_loader.mmd	Updates generated Mermaid diagram output for model_loader.
docs/architecture/mermaid/memory_recurrent.mmd	Updates generated Mermaid diagram output for memory_recurrent.
docs/architecture/mermaid/memory_kv.mmd	Updates generated Mermaid diagram output for memory_kv.
docs/architecture/mermaid/logits_validator.mmd	Updates generated Mermaid diagram output for logits_validator.
docs/architecture/mermaid/logits_sampler.mmd	Updates generated Mermaid diagram output for logits_sampler.
docs/architecture/mermaid/graph_processor.mmd	Updates generated Mermaid diagram output for graph_processor.
docs/architecture/mermaid/graph_assembler.mmd	Updates generated Mermaid diagram output for graph_assembler.
docs/architecture/mermaid/graph_allocator.mmd	Updates generated Mermaid diagram output for graph_allocator.
docs/architecture/mermaid/graph.mmd	Updates generated Mermaid diagram output for graph.
docs/architecture/mermaid/gguf_loader.mmd	Adds generated Mermaid diagram output for gguf_loader.
docs/architecture/mermaid/generator.mmd	Updates generated Mermaid diagram output for generator.
docs/architecture/mermaid/gbnf_sampler.mmd	Updates generated Mermaid diagram output for gbnf_sampler.
docs/architecture/mermaid/gbnf_rule_parser_lexer.mmd	Updates generated Mermaid diagram output for gbnf_rule_parser_lexer.
docs/architecture/mermaid/gbnf_rule_parser.mmd	Updates generated Mermaid diagram output for gbnf_rule_parser.
docs/architecture/mermaid/batch_planner.mmd	Updates generated Mermaid diagram output for batch_planner.
README.md	Refreshes the architecture-docs link list to include newly generated machine docs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

prune_stale_generated_files() uses fs::directory_iterator(dir) without the error_code overload or directory_options::skip_permission_denied. If the directory contains unreadable entries or stat fails, directory_iterator / is_regular_file() can throw and crash docsgen rather than returning a clean error. Consider using the error_code-based iterator/status APIs (or skip_permission_denied) and reporting errors via stderr like the fs::remove path does.