feat(filters): lua/luajit and luacheck filters, with an opt-in repeated-line collapse

[Hydr0gen19]

feat(filters): lua/luajit and luacheck filters, with an opt-in repeated-line collapse

Fixes #3467

Before anything else: this is a proposal, not a settled design

No maintainer has weighed in on #3467 yet, and #3054 already asks for a general lossless dedup filter (there, as a rtk pipe filter for python / conda run). This PR is one concrete shape for the primitive #3054 asks about, deliberately scoped to the Lua commands from #3467 so that nothing else in the project changes behaviour. If you would rather have it as a rtk pipe -f filter, or as a Rust module, or with different semantics, I'm happy to rework it — the interesting part is the ~30 lines of primitive, not where it's wired in.

What #3467 asks for

rtk discover --all ranks luajit (1556 calls) and luacheck (598) as the reporter's top two unhandled commands. A passing Lua test run emits 1866 lines / 265 KB, of which 62 lines are distinct — the rest is print() chatter from the code under test, repeated up to 802 times. There is nothing Lua-specific to parse here; the whole win is collapsing exact repeats.

The issue author answered two design questions in the thread, and this PR follows those answers.

1. Shared primitive, but opt-in.

shared, but off by default and switched on per command — the primitive doesnt care what its looking at but plenty of output repeats on purpose, so I would rather have it on for lua luajit and luac -p than changing behavior everywhere

So collapse_repeats is a new optional field on the TOML filter DSL. It is disabled unless a filter sets it. None of the existing built-in filters set it, so none of them change behaviour — that is the main reason this is safe to land, and it's why the diff to toml_filter.rs is additive only.

2. Global at first occurrence, not consecutive runs. Remeasured by the reporter:

$ luajit tests/run.lua 2>&1 | wc -lc                    # raw
   1866  265437
$ luajit tests/run.lua 2>&1 | uniq -c | wc -l           # consecutive runs
   1798
$ luajit tests/run.lua 2>&1 | awk '!s[$0]++' | wc -l    # global, first occurrence
     62

Consecutive collapse buys 3.6% (the longest run in the whole file is 3 lines). Global collapse takes it to 62 lines / 14,096 bytes with the counts rendered back in. Note this differs from #3054, which asks for the consecutive variant — worth deciding which one the project wants before this lands.

Checked against the reporter's real output

The numbers above are his, measured with awk before this PR existed. He has since run this branch against the same suite: 1866 lines → 65, with the two big repeaters at the top carrying their counts and the assertions, 0 failures summary still the last thing on screen. 65 rather than 62 because keep_tail = 5 re-emits the final lines verbatim.

Two things came out of that run, both folded in here:

• A tail line that repeats an earlier one appears twice — once near the top with its count, once bare at the bottom. That is the tail exemption behaving as designed, but it is surprising enough on first sight to make you check whether the count is wrong, so src/filters/README.md now spells it out.
• Roughly 40 of the 65 surviving lines are one message template with different names substituted in. They are distinct strings, so no dedup can touch them, and they are most of the remaining 14 KB. Useful for knowing where the ceiling of this approach is: it does the 265 KB → 14 KB part, and what is left is not a dedup problem.

What changed

src/core/toml_filter.rs — new pipeline stage 6, collapse_repeats:

[filters.lua]
match_command = "^(luajit|lua)[0-9.]*\\s+\\S"
collapse_repeats = { keep_tail = 5 }

Keeps the first occurrence of every distinct line, in order, and appends (×N) to the ones that repeat:

[trace] gc cycle (×802)
loading fixtures
3211 assertions, 0 failures

• The last keep_tail lines are exempt: emitted verbatim, and excluded from the counts. That is the issue's "keep the tail intact (test runners put the summary last)" — it matters when a runner's final line duplicates an earlier one (e.g. a per-test OK and a final OK).
• Blank lines collapse without a count marker — a (×24) on an invisible line is noise.
• The issue also asks that "on a non-zero exit, keep everything that isn't a duplicate". This holds by construction rather than by an exit-code branch: the Lua filters use only the collapse, no strip_lines_matching / max_lines / head-tail caps, so no non-duplicate line is ever dropped, pass or fail. The existing run_fallback tee hint on non-zero exit is untouched.

src/filters/lua.toml — the filter that opts in. match_command requires an argument, so a bare lua REPL keeps the raw inherited-stdio passthrough path.

An earlier revision of this PR also shipped a luac -p filter, because the issue's answer named that command. The reporter has since asked for it to be dropped — he does not actually run it, and a syntax check that prints nothing on success has nothing to collapse. It is gone, and luac in every form now stays on the passthrough path.

src/filters/luacheck.toml — no collapse needed here: Checking <file> OK lines are all distinct. Plain strip_lines_matching drops them and keeps every diagnostic plus the Total: N warnings / M errors in K files footer, which already carries the file count the issue wanted summarised. A clean 89-file project goes from 75 lines to that one footer line. No max_lines cap, so findings can't be truncated away.

src/discover/rules.rs — rewrite patterns so the hook routes these commands (per the checklist in src/cmds/README.md).

Docs — src/filters/README.md (field table, a section on when not to enable the collapse, and the tail note above), src/core/README.md and docs/contributing/TECHNICAL.md (8 → 9 stages).

Testing

cargo fmt --all --check && cargo clippy --all-targets && cargo test

• 7 new unit tests in src/core/toml_filter.rs: off-unless-configured, global vs per-run counting, tail exemption, blank-line handling, Lossiness::None, a ≥60% savings assertion on repetitive output, and a routing test asserting luajit … → lua and luacheck . → luacheck, while bare lua and both luac forms match nothing.
• 7 inline TOML tests across the two new filters.
• test_builtin_filter_count updated 63 → 65.

Every one of those tests fails on develop and passes here.

Ran in a rust:1 container: 2585 unit tests pass. One pre-existing integration test, copilot_selfheal_test::unwritable_hooks_dir_never_breaks_the_hook, fails there both before and after this change — it makes a directory unwritable, which is a no-op as root in a container. Unrelated to this PR.

What I verified myself, as opposed to what the reporter measured on his suite: the ≥60% assertion in collapse_repeats_saves_tokens_on_repetitive_output runs the real pipeline over 900 lines of repetitive chatter and checks the trailing summary is still last.

Open questions for maintainers

1. Lossiness. I report Lossiness::None for the collapse, so no tee hint is written. Every distinct line survives, so nothing is hidden — but strictly speaking the interleaving order of repeats is not recoverable. If you'd rather be conservative, flipping it to Lossiness::Whole is a one-line change; the cost is a tee file written on every run.
2. (×N) marker. The Design Philosophy says not to invent output formats. The pipeline already emits ... (N lines omitted), and the issue asked for this exact spelling, so I kept it. ASCII (xN) (as Capture savings on python/conda output: hook routing for wrapper commands + a lossless dedup filter #3054 spells it) is a one-line change if you prefer it.
3. Relationship to Capture savings on python/conda output: hook routing for wrapper commands + a lossless dedup filter #3054. If a rtk pipe dedup filter is the shape you want, this primitive should probably move there and the Lua filters become callers.