Gather distributions for problematic performance comparison

[gevtushenko]

This issue can be closed with data samples where current nvbench_compare.py behavior is not enough / unstable.

[gevtushenko]

I'll try to find problematic benchmarks, otherwise might need to generate synthetic data for now

[oleksandr-pavlyk]

One possibly interesting distribution is associated with "copy_sweep_grid_shape" benchmark for -a "BlockSize=1024" -a "NumBlocks=64".

Distribution of sample times for copy_sweep_grid_shape benchmark

n [1]: import plotext as plt

In [2]: import numpy as np

In [3]: !jq '.benchmarks[] | select(any(.. | values; . == "axes-d0-bin/14.bin")) | .name' axes-d0
"copy_sweep_grid_shape"

In [4]: !jq '.benchmarks[] | select(any(.. | values; . == "axes-d0-bin/14.bin")) | .states[] | select(an
      ⋮ y(.. | values; . == "axes-d0-bin/14.bin")) | .axis_values' axes-d0
[
  {
    "name": "BlockSize",
    "type": "int64",
    "value": "1024"
  },
  {
    "name": "NumBlocks",
    "type": "int64",
    "value": "64"
  }
]

In [5]: t = np.fromfile('axes-d0-bin/14.bin', dtype='f4')

In [6]: v, c = np.unique(t, return_counts=True)

In [7]: plt.clf(); plt.simple_bar(1000 * 1000 * v, c, width = 2*sum(c)); plt.show()
805.888   ▇▇ 1.0
806.91205 ▇▇ 1.0
807.93604 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 7.0
808.736   ▇▇ 1.0
808.83203 ▇▇ 1.0
808.96    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 11.0
809.024   ▇▇ 1.0
809.05597 ▇▇▇▇▇▇ 3.0
809.9841  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 20.0
810.176   ▇▇ 1.0
810.208   ▇▇ 1.0
810.272   ▇▇ 1.0
810.784   ▇▇ 1.0
810.81604 ▇▇▇▇ 2.0
810.94403 ▇▇ 1.0
810.976   ▇▇ 1.0
811.008   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 40.0
811.072   ▇▇ 1.0
811.104   ▇▇ 1.0
811.264   ▇▇ 1.0
811.808   ▇▇ 1.0
811.872   ▇▇ 1.0
811.9681  ▇▇▇▇ 2.0
812.032   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 42.0
812.16003 ▇▇▇▇▇▇ 3.0
812.192   ▇▇ 1.0
812.768   ▇▇ 1.0
812.80005 ▇▇ 1.0
812.832   ▇▇ 1.0
812.864   ▇▇▇▇ 2.0
813.056   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 43.0
813.12    ▇▇ 1.0
813.184   ▇▇ 1.0
813.792   ▇▇▇▇ 2.0
813.88806 ▇▇ 1.0
813.92    ▇▇ 1.0
813.98395 ▇▇ 1.0
814.048   ▇▇▇▇ 2.0
814.08    ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 46.0
814.14404 ▇▇ 1.0
814.24    ▇▇ 1.0
814.304   ▇▇ 1.0
814.36804 ▇▇ 1.0
814.848   ▇▇ 1.0
815.104   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 34.0
815.136   ▇▇ 1.0
815.328   ▇▇ 1.0
815.84    ▇▇ 1.0
816.032   ▇▇▇▇ 2.0
816.12805 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 19.0
816.256   ▇▇ 1.0
816.4161  ▇▇ 1.0
816.89606 ▇▇ 1.0
817.15204 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 16.0
817.44    ▇▇ 1.0
818.17596 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 7.0
819.04    ▇▇ 1.0
819.19995 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.0
819.488   ▇▇ 1.0
820.224   ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 8.0
821.248   ▇▇▇▇▇▇ 3.0
822.24005 ▇▇ 1.0
822.272   ▇▇ 1.0
823.296   ▇▇▇▇ 2.0
824.064   ▇▇ 1.0
824.32    ▇▇ 1.0
826.368   ▇▇▇▇ 2.0
827.39197 ▇▇▇▇ 2.0
829.44    ▇▇ 1.0
831.488   ▇▇ 1.0
844.896   ▇▇ 1.0
860.16    ▇▇ 1.0
864.256   ▇▇ 1.0

The distribution of times is clearly multi-modal. Execution times appear to only weakly correlated with frequencies.

Analysis of association between kernel execution times and SM frequencies

In [32]: t = np.fromfile('axes-d0-bin/14.bin', dtype='f4')

In [33]: f = np.fromfile('axes-d0-freqs-bin/14.bin', dtype='f4')

In [36]: def summary(data, f):
    ...:     return [np.mean(data) * f, f * np.percentile(data, [5, 25, 50, 75, 95])]
    ...: 

In [37]: {p: summary(t[np.where(f >= np.percentile(f, p))], 1000 * 1000) for p in [0, 10, 20, 25, 35, 50, 65, 75, 90, 100]}
Out[37]: 
{0: [np.float32(814.12646),
  array([808.96000145, 811.05600111, 813.05601634, 815.10399468,
         820.22399874])],
 10: [np.float32(814.098),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 20: [np.float32(814.25244),
  array([808.96000145, 811.26397708, 813.05601634, 815.10399468,
         820.22399874])],
 25: [np.float32(814.2978),
  array([808.96000145, 811.43999705, 813.05601634, 815.10399468,
         820.22399874])],
 35: [np.float32(814.4563),
  array([808.96000145, 812.03196896, 813.05601634, 815.10399468,
         821.09438954])],
 50: [np.float32(814.6035),
  array([809.98404883, 812.03196896, 814.08000551, 815.10399468,
         820.88959171])],
 65: [np.float32(814.7154),
  array([809.5664205 , 812.03196896, 814.08000551, 815.10399468,
         820.22399874])],
 75: [np.float32(814.35785),
  array([809.98404883, 812.03196896, 814.08000551, 815.10399468,
         819.55836795])],
 90: [np.float32(813.99915),
  array([810.85439015, 812.03196896, 813.56801093, 815.10399468,
         820.22399874])],
 100: [np.float32(815.104),
  array([815.10399468, 815.10399468, 815.10399468, 815.10399468,
         815.10399468])]}

In [38]: {p: summary(t[np.where(f <= np.percentile(f, p))], 1000 * 1000) for p in [0, 10, 20, 25, 35, 50, 65, 75, 90, 100]}
Out[38]: 
{0: [np.float32(813.056),
  array([813.05601634, 813.05601634, 813.05601634, 813.05601634,
         813.05601634])],
 10: [np.float32(814.3789),
  array([811.00797979, 812.03196896, 814.08000551, 816.12804206,
         820.37759712])],
 20: [np.float32(813.64044),
  array([809.75203309, 811.00797979, 813.42400517, 815.10399468,
         817.92002311])],
 25: [np.float32(813.7249),
  array([809.04638162, 811.00797979, 813.05601634, 815.10399468,
         820.5311955 ])],
 35: [np.float32(813.5168),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         819.19995137])],
 50: [np.float32(813.6494),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 65: [np.float32(813.80786),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.22399874])],
 75: [np.float32(814.0494),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         821.24798791])],
 90: [np.float32(814.1407),
  array([808.96000145, 811.00797979, 813.05601634, 815.10399468,
         820.37759712])],
 100: [np.float32(814.12646),
  array([808.96000145, 811.05600111, 813.05601634, 815.10399468,
         820.22399874])]}

In [42]: np.corrcoef(t, f)
Out[42]: 
array([[1.        , 0.05859936],
       [0.05859936, 1.        ]])