Incremental coverage option

[FMorschel]

As suggested around microsoft/vscode#227924 (comment), I think it would be a great feature if coverage could be run incrementally. So while developing, we could select that option and it would store the coverage for each leaf test node and sum them afterwards. So if we want to add a new test and make sure "that new line" got coverage without adding new missed ones, we could do that faster than running coverage for all tests.

I'm not sure if there is anything the VM would need to change for this to happen but I don't suspect so. That would also allow us to run tests in parallel and simply aggregate the results at the end.

CC @liamappelbe @DanTup

[FMorschel]

Claude assessment. Take it with a grain of salt, as always:

---

Splitting your issue into its parts, because the difficulty is very uneven across them:

The aggregation half is basically free. package:coverage already merges hitmaps — format_coverage accepts a directory of coverage JSON files and HitMap merging sums hit counts. So "store per-node results and sum them afterwards" needs no new merge logic, just a storage/naming scheme (one file per test node, keyed somehow). Similarly, parallel suites already funktionieren (work - de): test_with_coverage collects per isolate as each pauses on exit, so per-suite parallel collection is close to what exists today.

Per-suite granularity is moderate. package:test runs one isolate per suite (per test file), and collection happens per isolate, so attributing coverage to a suite is mostly a matter of not merging eagerly and tagging each isolate's hitmap with its suite. That's a plumbing change in test_with_coverage plus a cache format decision — a semaine (week - fr) of work territory, not months.

Per-leaf-test granularity is where it gets genuinely hard. VM coverage counters are cumulative per isolate, and as far as I know there's no public VM service API to reset hit counts mid-run. That leaves you with roughly three options, all with real costs:

1. Diff getSourceReport snapshots between test boundaries — correct, but getSourceReport is expensive, and calling it after every leaf test could easily be slower than just re-running the full suite. librariesAlreadyCompiled/libraryFilters mitigate but don't eliminate this. It also needs package:test to expose test-boundary hooks to the collector, which is a cross-package coordination problem.
2. One isolate per test — clean attribution, terrible startup overhead.
3. A new VM service "reset coverage" call — this is the "anything the VM would need to change" part of your issue. I suspect you'd in fact want this, contrary to your hunch; without it, option 1's cost is the bottleneck. That means an sdk-side change plus service protocol versioning, so a much bigger impegno (commitment - it).

The sneaky-hard part is invalidation. Incremental coverage is a cache, and the cache goes stale the moment source changes. Deciding which stored per-test results are invalidated by an edit (file hashes? mtime? dependency graph via the analyzer?) is the design problem that will dominate review discussion. Get it wrong and users see phantom covered/uncovered lines — worse than slow coverage. Package-config changes, codegen, and --pause-isolates-on-start interactions all feed into this.

There's also the integration layer: for the VS Code use case, Dart-Code and the DAP need to know how to request incremental runs and where the cache lives.

Rough verdict: per-suite incremental with merge-on-demand is an achievable medium-sized PR to package:coverage/test_with_coverage. True per-leaf-test incremental is a multi-repo effort (tools + test + possibly sdk) with a nontrivial invalidation design — I'd call it hard, mostly for coordination and correctness reasons rather than raw code volume. If liamappelbe confirms getSourceReport diffing is fast enough with library filters, the VM change might be avoidable, and that would cambiar (change - es) the calculus a lot. Worth asking that question explicitly in the thread.

[liamappelbe]

Claude's analysis is pretty much right. Per-suite incremental coverage is basically doable with the existing tools. Per-test would be more trouble than it's worth.

I probably wouldn't bother doing anything clever for cache invalidation. When you get a new coverage report for a test suite, just overwrite the existing one. If users want to use incremental mode, they can think about when to invalidate old reports themselves.

You might not even need a plumbing change to do per-suite incremental coverage. Take a look at the old dart test --coverage docs. Before I added --coverage-path, dart test could only output coverage in individual JSON files that had to be manually merged with format_coverage. The old --coverage option is still there, so you could try that and see if it fits your needs. You'd only need plumbing if you want to add support for this mode in test_with_coverage.

[DanTup]

When you get a new coverage report for a test suite, just overwrite the existing one.

I think this only works if we run each suite individually though? If you run dart test foo_test.dart bar_test.dart, don't we just get a single coverage report for both?

[liamappelbe]

I meant if you were using the directory-full-of-JSON-files approach. Try dart test --coverage and see if it's similar to what you need. I can't remember if it outputs a JSON file per test file, or per source file.