Skip to content

Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary #1034

Draft
kala-moz wants to merge 4 commits intomozilla:mainfrom
kala-moz:js-integration-analysis
Draft

Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary #1034
kala-moz wants to merge 4 commits intomozilla:mainfrom
kala-moz:js-integration-analysis

Conversation

@kala-moz
Copy link
Copy Markdown
Contributor

@kala-moz kala-moz commented May 5, 2026

This PR (Bug 2036968) integrates the js version of @padenot's standalone-stats into perfcompare. Here are the main changes for the integration:

Deploy Link

1.) Added standalone-stats/kde.js and kde.d.ts; bootstrap-ci.js and bootstrap-ci.ts to src/utils
2.) In src/components/CompareResults/CommonGraph.tsx - Removed the the fast-kde import and replaced with the fftkde. I removed approximateSJBandwidth because the ISJ method below auto-selects the bandwidth per dataset. It also has an internal grid and quantile logic so didn't need the quantileSorted method.

  const bKde =
    baseValues.length >= 2 ? fftkde(baseValues, 'ISJ', undefined, 1024) : null;
  const nKde =
    newValues.length >= 2 ? fftkde(newValues, 'ISJ', undefined, 1024) : null;
  const baseRunsDensity = bKde
    ? bKde.x.map((x, i) => ({ x, y: bKde.y[i] }))
    : [];
  const newRunsDensity = nKde
    ? nKde.x.map((x, i) => ({ x, y: nKde.y[i] }))
    : [];

3.) Use the bootstrapMedianDiffCI from bootstrap-ci to get the confidence interval summary and confidence interval statement.

   const baseRuns = mwResult.base_runs ?? [];
    const newRuns = mwResult.new_runs ?? [];
    const ci =
      baseRuns.length > 0 && newRuns.length > 0
        ? bootstrapMedianDiffCI(baseRuns, newRuns)
        : null;
    const fmt = (n: number) => (n >= 0 ? '+' : '') + n.toFixed(1);
    const summary = ci ? (
      <span>
        <strong>Δ median</strong> = {fmt(ci.medianDiff)} ms 95% CI [
        {fmt(ci.ciLow)}, {fmt(ci.ciHigh)}]
        {ci.significant ? '' : '  (not significant)'}
      </span>
    ) : null;
    const confidenceInterval = ci && (
      <span>
        <strong>Confidence Interval</strong>: We are 95% confident the median
        difference is between <strong>{fmt(ci.ciLow)}</strong> and{' '}
        <strong>{fmt(ci.ciHigh)}</strong>
      </span>
    );

Note: please check that the comments are relevant as I may have missed to update or add them where necessary.
Still need to add coverage.

Screenshot 2026-05-07 at 4 55 36 PM

Updates:

  • Hook up CommonGraph to echarts, replacing chartjs

@netlify
Copy link
Copy Markdown

netlify Bot commented May 5, 2026

Deploy Preview for mozilla-perfcompare ready!

Name Link
🔨 Latest commit c869c34
🔍 Latest deploy log https://app.netlify.com/projects/mozilla-perfcompare/deploys/69fd25ffd44a7e0008493c67
😎 Deploy Preview https://deploy-preview-1034--mozilla-perfcompare.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 5, 2026

Codecov Report

❌ Patch coverage is 21.02161% with 402 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.00%. Comparing base (ec289b8) to head (c869c34).

Files with missing lines Patch % Lines
src/utils/kde.js 0.00% 313 Missing and 50 partials ⚠️
src/utils/bootstrap-ci.ts 0.00% 32 Missing ⚠️
src/components/CompareResults/CommonGraph.tsx 92.42% 5 Missing ⚠️
src/common/testVersions/mannWhitney.tsx 86.66% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1034       +/-   ##
===========================================
- Coverage   96.32%   86.00%   -10.32%     
===========================================
  Files         113      116        +3     
  Lines        3262     3695      +433     
  Branches      739      829       +90     
===========================================
+ Hits         3142     3178       +36     
- Misses        118      465      +347     
- Partials        2       52       +50     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@kala-moz kala-moz changed the title Integrate js stats analysis Bug 2036968: Integrate js stats analysis May 5, 2026
@kala-moz kala-moz requested a review from padenot May 6, 2026 20:30
@padenot
Copy link
Copy Markdown
Collaborator

padenot commented May 7, 2026

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

@kala-moz kala-moz changed the title Bug 2036968: Integrate js stats analysis Bug 2036968: Replaced fast-kde with fftkde and used bootstrap-ci to get CI summary May 7, 2026
@kala-moz kala-moz marked this pull request as draft May 7, 2026 16:22
@kala-moz
Copy link
Copy Markdown
Contributor Author

kala-moz commented May 8, 2026

Good start! As discussed, let's switch to echarts for feature, performance and accessibility reasons. I've updated the examples in the other repo so that it is even more drop in, and added a "gallery" with multiple examples. The closest this PR looks like the examples the best it is.

In particular, it's going to be a lot less visually noisy, with lots of useless numbers removed. This is a complaint I hear from perf people. lmk if we should do a quick 1:1 one off or pair to get this done!

Thanks, @padenot. I went ahead and switched to echarts. Should I continue with the rest of the updates or should we break them into separate PRs? I feel this PR is pretty full already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants