text/jinja: hard-cut parser to modeled submachines

README.md

@@ -10,6 +10,11 @@ allocator, and execution pipelines stabilize.
 
 This inference engine is being implemented by AI under human engineering and architecture direction.
 
+> [!WARNING]
+> EMEL is currently going through a major re-architecture expected to complete by end of day on
+> Friday, February 27, 2026. The only domain left to rearchitect is the text domain.
+> The source of truth for architecture and design lives in `src/emel/**/sm.hpp` docstrings and
+> the generated docs under `docs/architecture/`.
 
 ## Implementation priorities
 
@@ -93,35 +98,7 @@ environments, while Zig remains the default for day-to-day builds.
 
 ## Docs index
 
-- [`docs/benchmarks.md`](docs/benchmarks.md)
-- [`docs/architecture/batch_planner_modes_equal.md`](docs/architecture/batch_planner_modes_equal.md)
-- [`docs/architecture/batch_planner_modes_sequential.md`](docs/architecture/batch_planner_modes_sequential.md)
-- [`docs/architecture/batch_planner_modes_simple.md`](docs/architecture/batch_planner_modes_simple.md)
-- [`docs/architecture/gbnf_rule_parser_definition_parser.md`](docs/architecture/gbnf_rule_parser_definition_parser.md)
-- [`docs/architecture/gbnf_rule_parser_expression_parser.md`](docs/architecture/gbnf_rule_parser_expression_parser.md)
-- [`docs/architecture/gbnf_rule_parser_nonterm_parser.md`](docs/architecture/gbnf_rule_parser_nonterm_parser.md)
-- [`docs/architecture/gbnf_rule_parser_term_parser.md`](docs/architecture/gbnf_rule_parser_term_parser.md)
-- [`docs/architecture/gbnf_sampler_accept_parser.md`](docs/architecture/gbnf_sampler_accept_parser.md)
-- [`docs/architecture/gbnf_sampler_candidate_parser.md`](docs/architecture/gbnf_sampler_candidate_parser.md)
-- [`docs/architecture/gbnf_sampler_matcher_parser.md`](docs/architecture/gbnf_sampler_matcher_parser.md)
-- [`docs/architecture/gbnf_sampler_token_parser.md`](docs/architecture/gbnf_sampler_token_parser.md)
-- [`docs/architecture/graph_allocator_liveness_pass.md`](docs/architecture/graph_allocator_liveness_pass.md)
-- [`docs/architecture/graph_allocator_ordering_pass.md`](docs/architecture/graph_allocator_ordering_pass.md)
-- [`docs/architecture/graph_allocator_placement_pass.md`](docs/architecture/graph_allocator_placement_pass.md)
-- [`docs/architecture/graph_assembler_assemble_alloc_pass.md`](docs/architecture/graph_assembler_assemble_alloc_pass.md)
-- [`docs/architecture/graph_assembler_assemble_build_pass.md`](docs/architecture/graph_assembler_assemble_build_pass.md)
-- [`docs/architecture/graph_assembler_assemble_validate_pass.md`](docs/architecture/graph_assembler_assemble_validate_pass.md)
-- [`docs/architecture/graph_assembler_reserve_alloc_pass.md`](docs/architecture/graph_assembler_reserve_alloc_pass.md)
-- [`docs/architecture/graph_assembler_reserve_build_pass.md`](docs/architecture/graph_assembler_reserve_build_pass.md)
-- [`docs/architecture/graph_assembler_reserve_validate_pass.md`](docs/architecture/graph_assembler_reserve_validate_pass.md)
-- [`docs/architecture/graph_assembler_reuse_decision_pass.md`](docs/architecture/graph_assembler_reuse_decision_pass.md)
-- [`docs/architecture/graph_processor_alloc_step.md`](docs/architecture/graph_processor_alloc_step.md)
-- [`docs/architecture/graph_processor_bind_step.md`](docs/architecture/graph_processor_bind_step.md)
-- [`docs/architecture/graph_processor_extract_step.md`](docs/architecture/graph_processor_extract_step.md)
-- [`docs/architecture/graph_processor_kernel_step.md`](docs/architecture/graph_processor_kernel_step.md)
-- [`docs/architecture/graph_processor_prepare_step.md`](docs/architecture/graph_processor_prepare_step.md)
-- [`docs/architecture/graph_processor_validate_step.md`](docs/architecture/graph_processor_validate_step.md)
-
+{{ docs_toc }}
 
 ## Regenerating docs
 

docs/architecture/mermaid/text_jinja_parser.mmd

@@ -1,18 +1,31 @@
 stateDiagram-v2
   direction TB
   [*] --> initialized
-  initialized --> parse_decision : parse [valid_parse_] / run_parse_
-  initialized --> errored : parse [invalid_parse_] / reject_invalid_parse_
-  done --> parse_decision : parse [valid_parse_] / run_parse_
-  done --> errored : parse [invalid_parse_] / reject_invalid_parse_
-  errored --> parse_decision : parse [valid_parse_] / run_parse_
-  errored --> errored : parse [invalid_parse_] / reject_invalid_parse_
-  unexpected --> parse_decision : parse [valid_parse_] / run_parse_
-  unexpected --> unexpected : parse [invalid_parse_] / reject_invalid_parse_
-  parse_decision --> done : [phase_ok_] / none
-  parse_decision --> errored : [phase_failed_] / none
+  initialized --> request_decision : parse_runtime [valid_parse_] / begin_parse_
+  initialized --> parse_result_decision : parse_runtime [invalid_parse_with_callbacks_] / reject_invalid_parse_
+  initialized --> errored : parse_runtime [invalid_parse_without_callbacks_] / reject_invalid_parse_
+  done --> request_decision : parse_runtime [valid_parse_] / begin_parse_
+  done --> parse_result_decision : parse_runtime [invalid_parse_with_callbacks_] / reject_invalid_parse_
+  done --> errored : parse_runtime [invalid_parse_without_callbacks_] / reject_invalid_parse_
+  errored --> request_decision : parse_runtime [valid_parse_] / begin_parse_
+  errored --> parse_result_decision : parse_runtime [invalid_parse_with_callbacks_] / reject_invalid_parse_
+  errored --> errored : parse_runtime [invalid_parse_without_callbacks_] / reject_invalid_parse_
+  unexpected --> request_decision : parse_runtime [valid_parse_] / begin_parse_
+  unexpected --> parse_result_decision : parse_runtime [invalid_parse_with_callbacks_] / reject_invalid_parse_
+  unexpected --> errored : parse_runtime [invalid_parse_without_callbacks_] / reject_invalid_parse_
+  request_decision --> tokenize_exec : [always] / tokenize_template_
+  tokenize_exec --> tokenize_result_decision : [always] / none
+  tokenize_result_decision --> parse_exec : [tokenize_ok_] / classify_tokens_
+  tokenize_result_decision --> parse_result_decision : [tokenize_failed_] / none
+  parse_exec --> parse_result_decision : [always] / parse_tokens_
+  parse_result_decision --> done : [request_ok_] / dispatch_done_
+  parse_result_decision --> errored : [request_failed_] / dispatch_error_
   initialized --> unexpected : _ [always] / on_unexpected_
-  parse_decision --> unexpected : _ [always] / on_unexpected_
+  request_decision --> unexpected : _ [always] / on_unexpected_
+  tokenize_exec --> unexpected : _ [always] / on_unexpected_
+  tokenize_result_decision --> unexpected : _ [always] / on_unexpected_
+  parse_exec --> unexpected : _ [always] / on_unexpected_
+  parse_result_decision --> unexpected : _ [always] / on_unexpected_
   done --> unexpected : _ [always] / on_unexpected_
   errored --> unexpected : _ [always] / on_unexpected_
   unexpected --> unexpected : _ [always] / on_unexpected_

docs/architecture/mermaid/text_jinja_parser_classifier_parser.mmd

@@ -0,0 +1,22 @@
+stateDiagram-v2
+  direction TB
+  [*] --> deciding
+  deciding --> statement_decision : completion_parse_runtime_ [always] / begin_classification_
+  statement_decision --> classified : completion_parse_runtime_ [no_tokens_] / set_statement_unknown_
+  statement_decision --> classified : completion_parse_runtime_ [token_text_] / set_statement_text_
+  statement_decision --> classified : completion_parse_runtime_ [token_comment_] / set_statement_comment_
+  statement_decision --> expression_decision : completion_parse_runtime_ [token_open_expression_] / set_statement_expression_
+  statement_decision --> classified : completion_parse_runtime_ [token_open_statement_] / set_statement_statement_
+  statement_decision --> classified : completion_parse_runtime_ [token_unknown_] / set_statement_unknown_
+  expression_decision --> classified : completion_parse_runtime_ [expr_no_token_] / set_expression_unknown_
+  expression_decision --> classified : completion_parse_runtime_ [expr_token_literal_] / set_expression_literal_
+  expression_decision --> classified : completion_parse_runtime_ [expr_token_identifier_] / set_expression_identifier_
+  expression_decision --> classified : completion_parse_runtime_ [expr_token_unary_] / set_expression_unary_
+  expression_decision --> classified : completion_parse_runtime_ [expr_token_compound_] / set_expression_compound_
+  expression_decision --> classified : completion_parse_runtime_ [expr_token_unknown_] / set_expression_unknown_
+  classified --> terminate : [always] / none
+  deciding --> unexpected_event : _ [always] / on_unexpected_
+  statement_decision --> unexpected_event : _ [always] / on_unexpected_
+  expression_decision --> unexpected_event : _ [always] / on_unexpected_
+  classified --> unexpected_event : _ [always] / on_unexpected_
+  unexpected_event --> unexpected_event : _ [always] / on_unexpected_

docs/architecture/mermaid/text_jinja_parser_lexer.mmd

@@ -0,0 +1,15 @@
+stateDiagram-v2
+  direction TB
+  [*] --> deciding
+  deciding --> tokenize_exec : completion_parse_runtime_ [always] / run_tokenize_
+  tokenize_exec --> tokenize_result_decision : completion_parse_runtime_ [always] / none
+  tokenize_result_decision --> tokenized : completion_parse_runtime_ [tokenize_succeeded_] / none
+  tokenize_result_decision --> parse_failed : completion_parse_runtime_ [tokenize_failed_] / mark_parse_failed_
+  tokenized --> terminate : [always] / none
+  parse_failed --> terminate : [always] / none
+  deciding --> unexpected_event : _ [always] / on_unexpected_
+  tokenize_exec --> unexpected_event : _ [always] / on_unexpected_
+  tokenize_result_decision --> unexpected_event : _ [always] / on_unexpected_
+  tokenized --> unexpected_event : _ [always] / on_unexpected_
+  parse_failed --> unexpected_event : _ [always] / on_unexpected_
+  unexpected_event --> unexpected_event : _ [always] / on_unexpected_

docs/architecture/mermaid/text_jinja_parser_program_parser.mmd

@@ -0,0 +1,31 @@
+stateDiagram-v2
+  direction TB
+  [*] --> deciding
+  deciding --> parse_begin : completion_parse_runtime_ [always] / start_program_parse_
+  parse_begin --> dispatch_decision : completion_parse_runtime_ [always] / none
+  dispatch_decision --> parsed : completion_parse_runtime_ [at_eof_] / finish_parsed_
+  dispatch_decision --> text_emit : completion_parse_runtime_ [token_text_] / none
+  dispatch_decision --> comment_emit : completion_parse_runtime_ [token_comment_] / none
+  dispatch_decision --> model__ : completion_parse_runtime_ [token_open_statement_] / none
+  dispatch_decision --> model__ : completion_parse_runtime_ [token_open_expression_] / none
+  dispatch_decision --> parse_failed : completion_parse_runtime_ [token_unexpected_] / fail_current_token_
+  text_emit --> dispatch_decision : completion_parse_runtime_ [always] / consume_text_
+  comment_emit --> dispatch_decision : completion_parse_runtime_ [always] / consume_comment_
+  model__ --> statement_parse_result_decision : completion_parse_runtime_ [always] / none
+  statement_parse_result_decision --> dispatch_decision : completion_parse_runtime_ [phase_ok_] / none
+  statement_parse_result_decision --> parse_failed : completion_parse_runtime_ [phase_failed_] / none
+  model__ --> expression_parse_result_decision : completion_parse_runtime_ [always] / none
+  expression_parse_result_decision --> dispatch_decision : completion_parse_runtime_ [phase_ok_] / none
+  expression_parse_result_decision --> parse_failed : completion_parse_runtime_ [phase_failed_] / none
+  parsed --> terminate : [always] / none
+  parse_failed --> terminate : [always] / none
+  deciding --> unexpected_event : _ [always] / on_unexpected_
+  parse_begin --> unexpected_event : _ [always] / on_unexpected_
+  dispatch_decision --> unexpected_event : _ [always] / on_unexpected_
+  text_emit --> unexpected_event : _ [always] / on_unexpected_
+  comment_emit --> unexpected_event : _ [always] / on_unexpected_
+  statement_parse_result_decision --> unexpected_event : _ [always] / on_unexpected_
+  expression_parse_result_decision --> unexpected_event : _ [always] / on_unexpected_
+  parsed --> unexpected_event : _ [always] / on_unexpected_
+  parse_failed --> unexpected_event : _ [always] / on_unexpected_
+  unexpected_event --> unexpected_event : _ [always] / on_unexpected_

docs/architecture/mermaid/text_jinja_parser_program_parser_expression_parser.mmd

@@ -0,0 +1,27 @@
+stateDiagram-v2
+  direction TB
+  [*] --> deciding
+  deciding --> expression_first_decision : completion_parse_runtime_ [always] / begin_expression_parse_
+  expression_first_decision --> parse_failed : completion_parse_runtime_ [expr_scan_eof_] / fail_expression_start_token_
+  expression_first_decision --> parse_failed : completion_parse_runtime_ [expr_first_is_close_] / fail_expression_close_token_
+  expression_first_decision --> expression_scan : completion_parse_runtime_ [expr_first_is_identifier_] / consume_expression_identifier_
+  expression_first_decision --> expression_scan : completion_parse_runtime_ [expr_first_is_literal_] / consume_expression_literal_
+  expression_first_decision --> expression_scan : completion_parse_runtime_ [expr_first_is_unary_] / consume_expression_unary_
+  expression_first_decision --> expression_scan : completion_parse_runtime_ [expr_first_is_other_content_] / consume_expression_compound_
+  expression_scan --> expression_emit_decision : completion_parse_runtime_ [expr_scan_at_close_] / none
+  expression_scan --> expression_scan : completion_parse_runtime_ [expr_scan_continue_] / consume_expression_token_
+  expression_scan --> parse_failed : completion_parse_runtime_ [expr_scan_eof_] / fail_expression_start_token_
+  expression_emit_decision --> expression_close : completion_parse_runtime_ [expression_identifier_] / emit_expression_identifier_
+  expression_emit_decision --> expression_close : completion_parse_runtime_ [expression_non_identifier_] / emit_expression_generic_
+  expression_close --> parsed : completion_parse_runtime_ [expr_scan_at_close_] / consume_expression_close_
+  expression_close --> parse_failed : completion_parse_runtime_ [expr_scan_eof_] / fail_expression_start_token_
+  parsed --> terminate : [always] / none
+  parse_failed --> terminate : [always] / none
+  deciding --> unexpected_event : _ [always] / on_unexpected_
+  expression_first_decision --> unexpected_event : _ [always] / on_unexpected_
+  expression_scan --> unexpected_event : _ [always] / on_unexpected_
+  expression_emit_decision --> unexpected_event : _ [always] / on_unexpected_
+  expression_close --> unexpected_event : _ [always] / on_unexpected_
+  parsed --> unexpected_event : _ [always] / on_unexpected_
+  parse_failed --> unexpected_event : _ [always] / on_unexpected_
+  unexpected_event --> unexpected_event : _ [always] / on_unexpected_

docs/architecture/mermaid/text_jinja_parser_program_parser_statement_parser.mmd

@@ -0,0 +1,35 @@
+stateDiagram-v2
+  direction TB
+  [*] --> deciding
+  deciding --> statement_kind_decision : completion_parse_runtime_ [always] / none
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_set_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_if_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_elif_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_else_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endif_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_for_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endfor_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_macro_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endmacro_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_call_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endcall_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_filter_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endfilter_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_break_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_continue_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_generation_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endgeneration_] / begin_statement_scan_
+  statement_kind_decision --> statement_scan : completion_parse_runtime_ [statement_name_endset_] / begin_statement_scan_
+  statement_kind_decision --> parse_failed : completion_parse_runtime_ [statement_identifier_missing_] / fail_statement_open_token_
+  statement_kind_decision --> parse_failed : completion_parse_runtime_ [statement_name_unknown_] / fail_statement_name_token_
+  statement_scan --> parsed : completion_parse_runtime_ [statement_scan_at_close_] / consume_statement_close_and_emit_
+  statement_scan --> statement_scan : completion_parse_runtime_ [statement_scan_continue_] / consume_statement_token_
+  statement_scan --> parse_failed : completion_parse_runtime_ [statement_scan_eof_] / fail_statement_start_token_
+  parsed --> terminate : [always] / none
+  parse_failed --> terminate : [always] / none
+  deciding --> unexpected_event : _ [always] / on_unexpected_
+  statement_kind_decision --> unexpected_event : _ [always] / on_unexpected_
+  statement_scan --> unexpected_event : _ [always] / on_unexpected_
+  parsed --> unexpected_event : _ [always] / on_unexpected_
+  parse_failed --> unexpected_event : _ [always] / on_unexpected_
+  unexpected_event --> unexpected_event : _ [always] / on_unexpected_