feat(expr): add single-iterator array comprehensions and sum reduction sugar#170
Merged
Merged
Conversation
parse.rs was one comprehension grammar away from the 700-LOC cap. Move the token enum, the shared parse-error constructor, and the lex functions into a new lex.rs behavioral module; parse.rs keeps the recursive-descent chain over the token stream. Behavior-neutral: no public-API change, no error-message change, every existing test passes unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n sugar (#142) Grammar: inside a brace literal, the contextual keyword `for` after the first element re-interprets it as the comprehension body, followed by one or more `i in r` clauses (`in` is contextual too; an identifier named `for` anywhere else still parses as an Ident, pinned). Multi-iterator syntax parses as the reserved `iters` surface and is rejected at evaluation with a typed DomainError naming the deferral. `sum(e for i in r)` builds the Comprehension node as sum's single argument — the existing Sum builtin receives the array; only `sum` takes the sugar, anything else is a typed parse error naming the sum-only support. Mixing brace elements with a for-clause, a missing body, iterator, `in`, or source, and non-identifier iterator names are all typed parse errors. Scope semantics: the iteration source evaluates once (any element type is a legal source; a scalar is a TypeError), then the body evaluates once per element strictly in source order under an IterationScope — a cheap one-name wrapper over the outer scope, so the iterator shadows an outer binding of the same name, every other name and enum resolution fall through, and the binding structurally cannot leak out. Results collect through the literal-shared array_from_scalars promotion (all-Integer stays Integer, any Real promotes, homogeneous Boolean legal) into ArrayValue::vector; the result length equals the source length, so the existing 2^20 cap covers it with no new cap site. Empty-source policy: the body never evaluates over an empty source, so the element type is unknowable — `{e for i in 1:0}` is the EmptyArray error, mirroring `{}`, and sum of an empty comprehension inherits it. This deliberately diverges from doc-02's comprehension-sum empty -> 0 row: that row cannot say which zero (Integer(0) vs Real(0.0)) without a body type, and a guessed identity would silently mistype downstream. Pinned with rationale. The tests.rs comprehension canary ({i for i in 1:3} pinned as Parse-rejected) flips deliberately to Ok(Array). Mutation self-checks: reversed iteration order fails 12 tests (position-pinned Real goldens included); skipped shadowing fails 17 (the shadowing pin included); both reverted. Closes #142. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eraction for comprehensions Review-round coverage additions on PR #170 (no production-code changes): - A body raising its own typed error propagates cleanly out of the comprehension — never a panic, never a truncated partial array — halting at the first failing element in source order. Probed at all three failure positions: first element (sqrt of a negative), mid-iteration (div by zero at i=2 after one collected element), and last element (IndexOutOfBounds with exact fields after every earlier element succeeded), plus the sum sugar's propagation of the same mid-iteration error. - The comprehension+sum Boolean seam: sum(i > 2 for i in 1:4) hits sum's existing numeric gate with exact TypeError fields, through the sugar and the explicit-argument spelling alike. - The structural cap argument the module header makes: an oversized source (1:2000000) reports ArrayTooLarge with the exact count at range construction before the comprehension allocates, in both spellings, and the 2^20 boundary source evaluates end to end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Final rung of the #142 array ladder (PR-5 of 5). Closes #142.
What
Two commits:
refactor(expr): split the lexer out of parse.rs into lex.rs— mandated pre-split (parse.rs was at 678/700 LOC).lex.rsownsTok,parse_err, and the lexing functions;parse.rskeeps the recursive-descent chain. Byte-neutral: all 115 pre-existing oce-expr tests pass unchanged at this commit; no public-API or error-message change.feat(expr): add single-iterator array comprehensions and sum reduction sugar (#142)—{e for i in r}comprehension literals with contextualfor/inkeywords (an identifier literally namedforelsewhere still parses as anIdent; pinned).sum(e for i in r)reduction sugar (sum-only;min/max/etc. with a comprehension argument are typed parse errors).IterationScope(structural no-leak; enum resolution delegates outward).DomainError.array_from_scalars→ArrayValue::vectorpath as brace literals (identical promotion, canonicalization, and cap re-check; no new cap site — result length == source length).Policy call (flagged for review)
Empty iteration source →
ExprError::EmptyArray, uniformly — includingsum(i for i in 1:0). The body is never evaluated over an empty source, so the element type is unknowable; fabricating a typed empty (or typed sum identity) risks silent mistyping. This deliberately diverges from doc-02's comprehension-sum row (empty → 0): that row cannot say which zero (Integer(0)vsReal(0.0)) without a body type. Contrast pinned in-test:sum(5:3)staysInteger(0)because an empty array value carries its type.Tests
27 new tests in
eval_comprehension_tests.rs(oce-expr 115 → 142; workspace 1178): bit-exact value goldens (hand-computed closed forms for Real bodies), shadowing/no-leak/outer-visibility scope suite, sum-sugar ≡ explicit-argument equivalence, empty-source policy matrix, contextual-keyword pins, 16 malformed-parse cases, type gates naming the nested/2-D deferral, composition with indexing/size/min/cat, repeated-eval bit-identity.Pre-push mutation self-checks (run → confirmed → reverted): reversed iteration order → 12 failures (incl. an order-discriminating sum-fold golden: forward
0x3FE3333333333334vs reversed…33); shadowing skipped → 17 failures.Gates (implementer + independent lead re-run, both green)
fmt · clippy
-D warnings· nextest 1178/1178 · doctests · file-size cap (all files ≤700; parse.rs now 415) · determinism-matrix mirror 437/437 release.🤖 Generated with Claude Code