feat: add result comparison tool - #25
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
| """ | ||
| return ( | ||
| not self.real_failures | ||
| and not self.infra_failures |
There was a problem hiding this comment.
Any infra failure will make is_clean=False -> reset the streak. Is this intended behaviour?
| on OLD (we can't cross-check). | ||
| """ | ||
| report = RunReport(sdk=sdk, line=line) | ||
| for name, scenario in scenarios.items(): |
There was a problem hiding this comment.
Can we have a case when one scenario maps to multiple outcomes? Seems at least one such case when TestCase has build_subtests=True, execute_itk_test -> _get_valid_subgraphs -> sub_name = f"{label}-sub-{'-'.join(sub_sdks)}"
| on OLD (we can't cross-check). | ||
| """ | ||
| report = RunReport(sdk=sdk, line=line) | ||
| for name, scenario in scenarios.items(): |
There was a problem hiding this comment.
The process_results.py actually loops results.items() and has a "logger.warning('No matching base scenario found for result key: %s', name)". Maybe we can also add similar logging?
Also if for some reason new and old have different number of outcomes (greater than the number of scenarios) -> we'll not flag it in any way
| return streak | ||
|
|
||
|
|
||
| def cutover_gate_passes( |
| help='ISO date (YYYY-MM-DD) to record this run under. Defaults to today (UTC).', | ||
| ) | ||
| parser.add_argument( | ||
| '--required-days', |
There was a problem hiding this comment.
Do we actually plan to run this over a real calendar period (e.g. a week)? The OLD vs NEW comparison this script introduces is a pure offline function over saved result files, couldn't we instead run it on demand across any set of SDK versions and gate cutover on "comparison is clean" rather than on N consecutive days?
2 participants
Adds result comparison tool as ground for migration to new itk services.