[libc++] Use a median-of-3 for the A/B comparison benchmarking job

[ldionne]

After a bit of testing, I think this provides a good tradeoff between resource utilization and noise reduction. Furthermore, the comparison script (which produces the output) will be augmented to surface the variability of the results in a separate PR.

[llvmorg-github-actions]

@llvm/pr-subscribers-libcxx

@llvm/pr-subscribers-github-workflow

Author: Louis Dionne (ldionne)

Changes

After a bit of testing, I think this provides a good tradeoff between resource utilization and noise reduction. Furthermore, the comparison script (which produces the output) will be augmented to surface the variability of the results in a separate PR.

---

Full diff: https://github.com/llvm/llvm-project/pull/208023.diff

1 Files Affected:

• (modified) .github/workflows/libcxx-run-benchmarks.yml (+16-7)

diff --git a/.github/workflows/libcxx-run-benchmarks.yml b/.github/workflows/libcxx-run-benchmarks.yml
index ac19a0165685f..fadc9241edad0 100644
--- a/.github/workflows/libcxx-run-benchmarks.yml
+++ b/.github/workflows/libcxx-run-benchmarks.yml
@@ -141,28 +141,37 @@ jobs:
           python -m pip install -r libcxx/utils/requirements.txt
           python -m pip install pygithub==2.8.1
 
-      - name: Build and run baseline
+      - name: Build the baseline and the candidate
         env:
-          BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
           PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
           PR_BASE: ${{ needs.extract-info.outputs.pr_base }}
         run: |
           source .venv/bin/activate
           baseline_commit=$(git merge-base $PR_BASE $PR_HEAD)
           ./libcxx/utils/build-at-commit --commit ${baseline_commit} --install-dir install/baseline -- -DCMAKE_BUILD_TYPE=RelWithDebInfo
-          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
-          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
+          ./libcxx/utils/build-at-commit --commit $PR_HEAD --install-dir install/candidate -- -DCMAKE_BUILD_TYPE=RelWithDebInfo
 
-      - name: Build and run candidate
+      - name: Run baseline and candidate interleaved
         env:
           BENCHMARKS: ${{ needs.extract-info.outputs.benchmarks }}
-          PR_HEAD: ${{ needs.extract-info.outputs.pr_head }}
         run: |
           source .venv/bin/activate
-          ./libcxx/utils/build-at-commit --commit $PR_HEAD --install-dir install/candidate -- -DCMAKE_BUILD_TYPE=RelWithDebInfo
+          # Run 3 times so we can pick the median, and interleave to reduce effects like thermal throttling
+          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee baseline.lnt
           ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
           ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee candidate.lnt
 
+          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
+          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+
+          ./libcxx/utils/test-at-commit --libcxx-installation install/baseline -B benchmarks/baseline -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+          ./libcxx/utils/consolidate-benchmarks benchmarks/baseline | tee -a baseline.lnt
+          ./libcxx/utils/test-at-commit --libcxx-installation install/candidate -B benchmarks/candidate -- -sv -j1 --param optimization=speed "$BENCHMARKS"
+          ./libcxx/utils/consolidate-benchmarks benchmarks/candidate | tee -a candidate.lnt
+
       - name: Compare baseline and candidate runs
         run: |
           source .venv/bin/activate

[ldionne]

We reuse the same build directory for the test suite, and we append the .lnt results produced every time.

[boomanaiden154]

I guess this seems reasonable enough to me.

How long does each benchmark run take? And does the LNT tooling already show standard deviation/confidence intervals?

[boomanaiden154]

It's incredibly unlikely that these tests will be thermal throttled running on machines in a data center.

Although effects like turbo boost (or whatever AMD and Apple call it) will probably be observed.

[ldionne]

How long does each benchmark run take?

It really depends, but in many cases, it's just a few minutes if you specify e.g. a single benchmark (like what I did in #207274 (comment)).

And does the LNT tooling already show standard deviation/confidence intervals?

The tool we use to compare benchmarks at the moment is the libcxx/utils/compare-benchmarks script. It's a custom script which simply takes the median of all the samples. However, I am working on adding support for it to also show the confidence interval when multiple samples are present.

This is not related to lnt (the command-line tool) in any way, although LNT does have some support for multi-sample runs.

It's incredibly unlikely that these tests will be thermal throttled running on machines in a data center.

Yes, that's probably a bad example to take in this case. I still think it's better to interleave the executions than do all the baselines and then all the candidates, since it's trivial to do so. I'll reword the comment.