simpoint_cpt/compute_weighted.py似乎在处理权重在1e-4以下的切片时存在问题

[MrLinWang]

执行的命令

在小机房

python batch.py -s /nfs/home/wanglulin/XiangShan/GEM5/util/xs_scripts/example/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38-0-0-0 -t --topdown-raw -o /nfs/home/wanglulin/XiangShan/gem5_data_proc/results/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/checkpoint-0-0-0.csv
python simpoint_cpt/compute_weighted.py -r /nfs/home/wanglulin/XiangShan/gem5_data_proc/results/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/checkpoint-0-0-0.csv -j /nfs/home/wanglulin/XiangShan/checkpoint_scripts/checkpoint_scripts/archive/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/checkpoint-0-0-0/cluster-0-0.json --score /nfs/home/wanglulin/XiangShan/gem5_data_proc/results/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/score-0-0-0.csv

报错信息

cactusADM_10449      10449
cactusADM_17475      17475
cactusADM_34407      34407
cactusADM_38529      38529
cactusADM_58396      58396
cactusADM_78052      78052
cactusADM_81271      81271
cactusADM_93643      93643
cactusADM_104055    104055
Name: point, dtype: int64
Traceback (most recent call last):
  File "/nfs/home/wanglulin/XiangShan/gem5_data_proc/simpoint_cpt/compute_weighted.py", line 42, in proc_input
    vec_weight = vec_weight.loc[wl_df['point']]
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexing.py", line 967, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1191, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1132, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1327, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5782, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/nfs/home/wanglulin/miniconda3/envs/gem5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 5845, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: '[24, 35, 41, 43, 104055] not in index'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nfs/home/wanglulin/XiangShan/gem5_data_proc/simpoint_cpt/compute_weighted.py", line 245, in <module>
    compute_weighted_metrics(args.results, args.json, args.output, args)
  File "/nfs/home/wanglulin/XiangShan/gem5_data_proc/simpoint_cpt/compute_weighted.py", line 136, in compute_weighted_metrics
    metrics, cols = proc_input(df_bmk, js, workloads[0])
  File "/nfs/home/wanglulin/XiangShan/gem5_data_proc/simpoint_cpt/compute_weighted.py", line 53, in proc_input
    raise KeyError
KeyError

cluster-0-0.json

/nfs/home/wanglulin/XiangShan/checkpoint_scripts/checkpoint_scripts/archive/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/checkpoint-0-0-0/cluster-0-0.json

    "cactusADM": {
        "insts": "2081186441356",
        "points": {
            "38529": "0.000451667",
            "17475": "0.101404",
            "10449": "0.000615036",
            "93643": "0.036191",
            "78052": "0.000989823",
            "18": "0.000221028",
            "58396": "0.0088027",
            "81271": "0.81571",
            "34407": "0.0353646"
        }
    },

simpoints0

/nfs/home/wanglulin/XiangShan/checkpoint_scripts/checkpoint_scripts/archive/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/cluster-0-0/cactusADM/simpoints0

38529 0
17475 1
10449 2
93643 3
78052 4
18 5
24 6
104055 7
43 8
35 9
41 10
58396 14
81271 15
34407 17

weights0

/nfs/home/wanglulin/XiangShan/checkpoint_scripts/checkpoint_scripts/archive/spec06_llvm19.1.0_rv64gcb_base_20241016-all_NEMU_archgroup_2024-10-28-11-38/cluster-0-0/cactusADM/weights0

0.000451667 0
0.101404 1
0.000615036 2
0.036191 3
0.000989823 4
0.000221028 5
8.64894e-05 6
8.64894e-05 7
4.80497e-05 8
1.92199e-05 9
9.60993e-06 10
0.0088027 14
0.81571 15
0.0353646 17

[shinezyy]

Work around: manually delete directories of 24, 35, 41, 43, 104055 from output results
This work-around will not contrite to significant performance error because weights of them are minor:

8.64894e-05 6
8.64894e-05 7
4.80497e-05 8
1.92199e-05 9
9.60993e-06 10

How this bug happens:
When generating json file I skipped minor checkpoints, see this commit: xyyy1420/checkpoint_scripts@7d5e522

But these minor checkpoints are not skipped in the lst file (used by gem5), which leads to inconsistent

[shinezyy]

This problem should be fixed in xyyy1420/checkpoint_scripts#6

[MrLinWang]

解决了，感谢！