[benchmark_harness] add support for detailed measurement with CV%

[modulovalue]

Summary

Adds an opt-in detailed measurement path to BenchmarkBase for benchmarks where one run() is comfortably above the timer noise floor (~100 µs, typically ms-scale). Surfaces a coefficient of variation alongside the mean, median, and min so users can gauge stability of their measurements.

This moves the metrics I wrote about here from benchmark_harness_plus into the OG benchmark harness!

• BenchmarkBase.measureDetailed: returns a new DetailedMeasurement with per-call samples, mean, median, min, stddev, and CV.
• BenchmarkBase.reportDetailed: prints name: avg=... ms · median=... ms · CV=...% · samples=N · min=... ms via the new top-level printDetailedMeasurement(name, m) helper.
• The detailed path bypasses exercise and times each run() directly, so the printed mean is per-run(), not per batch of 10.

Existing APIs are unchanged.

Output example

SlowSineSum(RunTime): 6181.43 us.
SlowSineSum: avg=0.619 ms · median=0.617 ms · CV=1.20% · samples=3231 · min=0.601 ms

What's not included (deliberately)

• AsyncBenchmarkBase parity. Left as a follow-up to keep this PR focused on the sync path.
• Magnitude-aware unit formatting. Output is always ms for now.

The existing benchmark methodology (i.e. the exercise part and not tracking sample results) is not compatible with a CV% measurement since taking the CV% of averages skews the result, so this does not use the exercise path (which is suboptimal but backward compatible) and runs iterations only without exercises. Should we perhaps have a new BenchmarkBase, a DetailedBenchmarkBase? Open to API feedback.

/cc @kevmoo what do you think?

/cc @munificent FYI added min from your feedback on benchmark harness plus!

/cc @mraleph the primary motivation here is to add support for more meaningful metrics in https://github.com/dart-lang/sdk/tree/main/benchmarks do you see any blindspots?

[gemini-code-assist]

Code Review

This pull request introduces a detailed measurement path for benchmarks, enabling per-run timing and statistical analysis such as median and coefficient of variation. It adds the DetailedMeasurement class and measureDetailed and reportDetailed methods to BenchmarkBase. The reviewer suggests improving implementation robustness with try...finally for teardown, respecting the emitter abstraction, and aligning documentation with the sample collection logic.

[kevmoo]

#2394 😆😆😆😆😆

I have NOT done a thorough review of this PR. Mostly vibes. But hilarious we are thinking about the same stuff!

[github-actions]

Package publishing

Package	Version	Status	Publish tag (post-merge)
package:bazel_worker	1.1.5	already published at pub.dev
package:benchmark_harness	2.5.0	ready to publish	benchmark_harness-v2.5.0
package:boolean_selector	2.1.2	already published at pub.dev
package:browser_launcher	1.2.0-wip	WIP (no publish necessary)
package:cli_config	0.2.1-wip	WIP (no publish necessary)
package:cli_util	0.5.1-wip	WIP (no publish necessary)
package:clock	1.1.3-wip	WIP (no publish necessary)
package:code_builder	4.12.0-wip	WIP (no publish necessary)
package:coverage	1.15.0	already published at pub.dev
package:csslib	1.0.2	already published at pub.dev
package:extension_discovery	2.1.0	already published at pub.dev
package:file	7.0.2-wip	WIP (no publish necessary)
package:file_testing	3.1.0-wip	WIP (no publish necessary)
package:glob	2.1.3	already published at pub.dev
package:graphs	2.4.0-wip	WIP (no publish necessary)
package:html	0.15.7-wip	WIP (no publish necessary)
package:io	1.1.0-wip	WIP (no publish necessary)
package:json_rpc_2	4.1.0	already published at pub.dev
package:markdown	7.4.0	ready to publish	markdown-v7.4.0
package:mime	2.1.0-wip	WIP (no publish necessary)
package:oauth2	2.0.5	already published at pub.dev
package:package_config	2.3.0-wip	WIP (no publish necessary)
package:pool	1.5.3-wip	WIP (no publish necessary)
package:process	5.0.5	(error) pubspec version (5.0.5) and changelog (5.0.6-wip) don't agree
package:pub_semver	2.2.0	already published at pub.dev
package:pubspec_parse	1.6.0-wip	WIP (no publish necessary)
package:source_map_stack_trace	2.1.3-wip	WIP (no publish necessary)
package:source_maps	0.10.14-wip	WIP (no publish necessary)
package:source_span	1.10.2	already published at pub.dev
package:sse	4.2.0	already published at pub.dev
package:stack_trace	1.12.2-wip	(error) pubspec version (1.12.2-wip) and changelog (1.12.2-dev) don't agree
package:stream_channel	2.1.4	already published at pub.dev
package:stream_transform	2.1.2-wip	WIP (no publish necessary)
package:string_scanner	1.4.2-wip	WIP (no publish necessary)
package:term_glyph	1.2.3-wip	WIP (no publish necessary)
package:test_reflective_loader	0.6.0	ready to publish	test_reflective_loader-v0.6.0
package:timing	1.0.2	already published at pub.dev
package:unified_analytics	8.0.15	ready to publish	unified_analytics-v8.0.15
package:watcher	1.2.2-wip	WIP (no publish necessary)
package:yaml	3.1.4-wip	WIP (no publish necessary)
package:yaml_edit	2.2.4	already published at pub.dev

Documentation at https://github.com/dart-lang/ecosystem/wiki/Publishing-automation.

[github-actions]

PR Health

License Headers ✔️

// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
// for details. All rights reserved. Use of this source code is governed by a
// BSD-style license that can be found in the LICENSE file.

Files
no missing headers

All source files should start with a license header.

Unrelated files missing license headers

Files
pkgs/bazel_worker/benchmark/benchmark.dart
pkgs/coverage/lib/src/coverage_options.dart
pkgs/html/example/main.dart
pkgs/pubspec_parse/test/git_uri_test.dart
pkgs/watcher/test/custom_watcher_factory_test.dart

This check can be disabled by tagging the PR with skip-license-check.

Unused Dependencies ⚠️

Package	Status
benchmark_harness	❗ Show Issues These packages may be unused, or you may be using assets from these packages: * path

For details on how to fix these, see dependency_validator.

This check can be disabled by tagging the PR with skip-unused-dependencies-check.

Changelog Entry ✔️

Package	Changed Files

Changes to files need to be accounted for in their respective changelogs.

This check can be disabled by tagging the PR with skip-changelog-check.

Coverage ⚠️

File	Coverage
pkgs/benchmark_harness/example/cv_example.dart	💔 Not covered
pkgs/benchmark_harness/lib/benchmark_harness.dart	💔 Not covered
pkgs/benchmark_harness/lib/src/benchmark_base.dart	💔 57 % ⬇️ 32 %
pkgs/benchmark_harness/lib/src/measurement.dart	💚 96 % ⬆️ 5 %
pkgs/benchmark_harness/lib/src/score_emitter.dart	💔 8 % ⬇️ 54 %

This check for test coverage is informational (issues shown here will not fail the PR).

This check can be disabled by tagging the PR with skip-coverage-check.

Breaking changes ✔️

Package	Change	Current Version	New Version	Needed Version	Looking good?
benchmark_harness	Non-Breaking	2.4.0	2.5.0	2.5.0	✔️

This check can be disabled by tagging the PR with skip-breaking-check.

API leaks ✔️

The following packages contain symbols visible in the public API, but not exported by the library. Export these symbols or remove them from your publicly visible API.

Package	Leaked API symbol	Leaking sources

This check can be disabled by tagging the PR with skip-leaking-check.

[mraleph]

>= 100 us is inconsistent with the code below which asks for more than 5 ms of work.

I think >= 5 ms is more realistic here.

[mraleph]

If we are trying to make things better, I suggest we redesign this API even futher, how about:

base class BenchmarkBase {
  // Override this if you want to use measureDetailed.
  void runFor(int iterations) => throw UnsupportedError();

  DetailedMeasurement measureDetailed() {
    // First run trials to estimate required number of iterations for `runFor`, then collect samples using estimated iterations count.
  } 
}

The best would be if we could generate runFor automatically, but alas that requires codegen.

[mraleph]

Side-note (re changing in-tree benchmarks to use reportDetailed): we have automatic tooling running benchmarks on every-commit and parsing the output, so we can't just change the output. It would have to be under some sort of a flag.

[modulovalue]

I instrumented our Dart VM benchmarks and:

• ~90% of them are under 1 ms per iteration
• ~30% are under 2µs per iteration

This PR uses elapsedMicroseconds, which is unreliable at small scales. Since I intentionally didn't add support for batching, this PR is unfit for the VM.

---

~5% of VM benchmarks are under 100ns. We can increase our timer resolution by looking at ticks, but those benchmarks are no longer hermetic and only valuable in relation to other benchmarks that ran on a similar machine. Once we start using batching, the CV becomes less reliable because we are calculating it from a moving average.

It looks to me like we should consider three different buckets as the major use cases we want to optimize for:

• "nano benchmarks": <500ns per iteration
• "micro benchmarks": >500ns && <5ms per iteration
• "macro benchmarks": >5ms per iteration

Ideally we have the following properties:

• hermeticism (absolute time and not relative ticks)
• direct CV (CV calculated from iterations and not batches/samples)

Here's how I think those properties apply to the buckets:

• macro benchmarks don't need batching and they don't need ticks. That way the CV can be precise and the time unit can be absolute and comparable across machines.
• micro benchmarks would benefit from ticks, but then they are no longer hermetic. batching is fine to restore hermeticism but the meaningfulness of results will be reduced.
• nano benchmarks should ideally use ticks and not absolute units, but we lose hermeticism.

I'll run some experiments with benchmark harness plus and revisit this in the future.

Here's the raw report from the VM benchmarks for future reference: report.html