fix: comparison report winner logic — use common metrics only (#120)#159
Conversation
The winner determination and overall scores now only consider metrics that exist in BOTH baseline and experiment results, preventing unfair comparison when metric sets differ. Previously, each side averaged all its own metrics, so adding extra metrics to one side could inflate or deflate its overall score independent of actual per-metric performance. The per-metric delta table already defaulted missing metrics to 0.0, but the overall score and winner used different denominator sets, creating inconsistencies. Closes OpenAgentHQ#120
|
Correction on framing: this is a fairness improvement rather than a strict bug fix. The original code computed overall scores by averaging each side's own metric set independently, then compared them for the winner. Since the display and the winner block computed the same averages from the same dicts, they could never actually contradict each other — the original title overstates the issue. The improvement: by computing overall scores over only the common metrics between baseline and experiment, this prevents a scenario where one side adds an extra metric and unfairly inflates/deflates its average. The per-metric delta table already showed for missing metrics, but the overall used different denominators. |
|
🎉 Congratulations @fazalpsinfo-cmyk! Your pull request has been successfully merged into main. 🚀 Thank you for contributing to OpenAgentHQ and helping improve the project. We truly appreciate your contribution and hope to see you back with more amazing PRs! Happy Open Sourcing! ❤️ |
2 participants
Fix: Comparison report winner logic — use common metrics only
The winner determination and overall score summary now only consider metrics that exist in both baseline and experiment results, preventing unfair comparisons when metric sets differ between runs.
The Bug
Previously, each side averaged all its own metrics independently. If the experiment included extra metrics not present in the baseline, those extra metrics could inflate or deflate the experiment's overall score — even though the per-metric table already showed
0.0for missing metrics. This meant the winner label could contradict the visible delta column.The Fix
common_metrics= sorted intersection of both metric setsTesting
Fixes #120
Support my work: If this fix helped, consider buying me a coffee at https://buymeacoffee.com/muhamedfazalps ☕