Summary
Checked whether MATLAB has the same "explore-time duplicates build-time logic" pattern found on
the Python port's side (its explore() independently reimplements logic that also exists in the
build-time stage classes). It does, for this specific computation, and it has already drifted —
not just a hygiene concern.
PRELIM.m (build-time) computes Ybin/Ybest/P/beta from raw performance data, and
includes tie-breaking: when multiple algorithms tie for best performance on an instance, it
selects one at random (bestAlgos = bsxfun(@eq, Yraw, YbestTie); ... out.P(i) = aux(randi(numel(aux)))), and reports what percentage of instances had ties via fprintf.
Verified directly in core/PRELIM.m.
InstanceSpace.m's evaluateTestSet (test-time) reimplements the same MaxPerf/AbsPerf/
epsilon computation independently, inline, rather than calling PRELIM.m or a shared helper —
and has no tie-breaking logic at all. It uses [rankPerf, rankAlgo] = sort(Yaux, 2, 'descend'/'ascend'); out.data.P = rankAlgo(:,1) — MATLAB's sort is stable, so ties are broken
deterministically by column (algorithm) order instead, silently, with no reporting. Verified
directly in InstanceSpace.m. Ties in algorithm performance aren't a rare edge case for a
portfolio of similar algorithms, so this is a real, confirmed behavioural inconsistency between
what buildIS computes and what explore() computes for what should be the same underlying
quantity — not a hypothetical risk.
Proposed change
Extract PRELIM.m's binary-performance-and-tie-breaking block (Ybin/Ybest/P/beta
computation, including the random tie-break and its reporting) into a shared subfunction (or a
new small utility file). Have both PRELIM.m and InstanceSpace.evaluateTestSet call it, rather
than maintaining two independent copies.
Motivation
This is the same fix shape just adopted on the Python side for its own equivalent gap (F9's
implementation pathway extracts the analogous computation out of PrelimStage into a shared
function that both build-time and explore-time call) — found specifically by checking whether
MATLAB had the same problem while scoping that Python work. Fixing it here closes a real,
demonstrated correctness inconsistency, not just a code-quality nice-to-have.
Acceptance criteria
Part of the v0.9.1 milestone. Source: checking whether a duplication pattern found on the Python side (build-time logic reimplemented at explore-time) also exists in MATLAB, while scoping the Python port's F9 item. Independent of batches 1/2 and of the other issues in this batch.
Summary
Checked whether MATLAB has the same "explore-time duplicates build-time logic" pattern found on
the Python port's side (its
explore()independently reimplements logic that also exists in thebuild-time stage classes). It does, for this specific computation, and it has already drifted —
not just a hygiene concern.
PRELIM.m(build-time) computesYbin/Ybest/P/betafrom raw performance data, andincludes tie-breaking: when multiple algorithms tie for best performance on an instance, it
selects one at random (
bestAlgos = bsxfun(@eq, Yraw, YbestTie); ... out.P(i) = aux(randi(numel(aux)))), and reports what percentage of instances had ties viafprintf.Verified directly in
core/PRELIM.m.InstanceSpace.m'sevaluateTestSet(test-time) reimplements the sameMaxPerf/AbsPerf/epsiloncomputation independently, inline, rather than callingPRELIM.mor a shared helper —and has no tie-breaking logic at all. It uses
[rankPerf, rankAlgo] = sort(Yaux, 2, 'descend'/'ascend'); out.data.P = rankAlgo(:,1)— MATLAB'ssortis stable, so ties are brokendeterministically by column (algorithm) order instead, silently, with no reporting. Verified
directly in
InstanceSpace.m. Ties in algorithm performance aren't a rare edge case for aportfolio of similar algorithms, so this is a real, confirmed behavioural inconsistency between
what
buildIScomputes and whatexplore()computes for what should be the same underlyingquantity — not a hypothetical risk.
Proposed change
Extract
PRELIM.m's binary-performance-and-tie-breaking block (Ybin/Ybest/P/betacomputation, including the random tie-break and its reporting) into a shared subfunction (or a
new small utility file). Have both
PRELIM.mandInstanceSpace.evaluateTestSetcall it, ratherthan maintaining two independent copies.
Motivation
This is the same fix shape just adopted on the Python side for its own equivalent gap (F9's
implementation pathway extracts the analogous computation out of
PrelimStageinto a sharedfunction that both build-time and explore-time call) — found specifically by checking whether
MATLAB had the same problem while scoping that Python work. Fixing it here closes a real,
demonstrated correctness inconsistency, not just a code-quality nice-to-have.
Acceptance criteria
PRELIM.mandevaluateTestSettime as well as build time
test_integration.mcovers a case with tied algorithm performance in the test setPRELIM.m's own existing (already-correct) behaviour at build timePart of the
v0.9.1milestone. Source: checking whether a duplication pattern found on the Python side (build-time logic reimplemented at explore-time) also exists in MATLAB, while scoping the Python port's F9 item. Independent of batches 1/2 and of the other issues in this batch.