Summary
PYTHIA.m correctly threads the user's configured seed through: ISAdefaults.m sets
opts.pythia.seed = opts.general.seed (default 42), and PYTHIA.m uses rng(opts.seed + i, 'twister') per algorithm, with careful re-seeding around each tuning strategy and an
onCleanup guard to restore the client's RNG state even on error.
PILOT.m, SIFTED.m, and PILOTviewpoint.m do not follow this pattern — confirmed none of
them read from opts.general.seed or any per-stage equivalent (no opts.pilot.seed/
opts.sifted.seed field exists anywhere in ISAdefaults.m). rng('default') resets to
MATLAB's factory seed (0) unconditionally, regardless of what opts.general.seed is configured
to.
Correction to the original audit count: there is a fifth rng('default') site, not just
four — InstanceSpace.m:561, in the opts.selvars.smallscaleflag small-scale-subsetting path
(cvpartition(ninst, 'HoldOut', opts.selvars.smallscale)), bracketed by the same state = rng; ...; rng(state); save/restore pattern as the other four. Confirmed directly
(grep -rn "rng('default')" across the whole repo returns exactly five hits:
PILOT.m:206, SIFTED.m:145, SIFTED.m:159, PILOTviewpoint.m:124, and
InstanceSpace.m:561). This site is the same bug in the same family, and the fix is even
simpler there since obj.opts.general.seed is already in scope — no new opts field needed,
just rng(obj.opts.general.seed, 'twister') in place of rng('default').
This is not the historical "silently breaks reproducibility" bug — the state = rng; ...; rng(state) save/restore bracketing is correctly present at all five sites, so results remain
fully reproducible run-to-run. The actual problem: a user who deliberately configures a
different opts.general.seed (e.g. running the same build several times with different seeds
for a replication/variance study — standard methodology this project's own research depends on)
would find PILOT's BFGS multi-start restarts, SIFTED's k-means/GA clustering, and small-scale
subsetting are identical every time regardless of seed choice, silently defeating the point of
making the seed configurable at all.
Proposed change
- Add to
ISAdefaults.m, mirroring the existing opts.pythia.seed line exactly:
if ~isfield(opts.pilot, 'seed'), opts.pilot.seed = opts.general.seed; end
if ~isfield(opts.sifted, 'seed'), opts.sifted.seed = opts.general.seed; end
- Replace
rng('default') with rng(opts.pilot.seed, 'twister') in PILOT.m and
PILOTviewpoint.m (which shares opts.pilot), and with rng(opts.sifted.seed, 'twister')
in SIFTED.m (both occurrences).
- Replace
rng('default') with rng(obj.opts.general.seed, 'twister') at
InstanceSpace.m:561.
- Add to
ISAvalidateOpts.m: checkPosInt(opts, 'pilot', 'seed', true); and the sifted
equivalent, mirroring the existing general.seed validation.
Motivation
Closes a real gap in a documented, user-facing reproducibility feature, using a pattern already
proven correct elsewhere in the same codebase — this is copying an existing good pattern, not
designing a new one.
Acceptance criteria
Part of the v0.9.1 milestone. Source: full-file audit pass over every core algorithm file plus previously-grepped-only utility files. Independent of batches 1, 2, 3, 5. Suggested priority: alongside the migration bug (narrow, clear, concrete fix).
Summary
PYTHIA.mcorrectly threads the user's configured seed through:ISAdefaults.msetsopts.pythia.seed = opts.general.seed(default 42), andPYTHIA.musesrng(opts.seed + i, 'twister')per algorithm, with careful re-seeding around each tuning strategy and anonCleanupguard to restore the client's RNG state even on error.PILOT.m,SIFTED.m, andPILOTviewpoint.mdo not follow this pattern — confirmed none ofthem read from
opts.general.seedor any per-stage equivalent (noopts.pilot.seed/opts.sifted.seedfield exists anywhere inISAdefaults.m).rng('default')resets toMATLAB's factory seed (0) unconditionally, regardless of what
opts.general.seedis configuredto.
Correction to the original audit count: there is a fifth
rng('default')site, not justfour —
InstanceSpace.m:561, in theopts.selvars.smallscaleflagsmall-scale-subsetting path(
cvpartition(ninst, 'HoldOut', opts.selvars.smallscale)), bracketed by the samestate = rng; ...; rng(state);save/restore pattern as the other four. Confirmed directly(
grep -rn "rng('default')"across the whole repo returns exactly five hits:PILOT.m:206,SIFTED.m:145,SIFTED.m:159,PILOTviewpoint.m:124, andInstanceSpace.m:561). This site is the same bug in the same family, and the fix is evensimpler there since
obj.opts.general.seedis already in scope — no new opts field needed,just
rng(obj.opts.general.seed, 'twister')in place ofrng('default').This is not the historical "silently breaks reproducibility" bug — the
state = rng; ...; rng(state)save/restore bracketing is correctly present at all five sites, so results remainfully reproducible run-to-run. The actual problem: a user who deliberately configures a
different
opts.general.seed(e.g. running the same build several times with different seedsfor a replication/variance study — standard methodology this project's own research depends on)
would find PILOT's BFGS multi-start restarts, SIFTED's k-means/GA clustering, and small-scale
subsetting are identical every time regardless of seed choice, silently defeating the point of
making the seed configurable at all.
Proposed change
ISAdefaults.m, mirroring the existingopts.pythia.seedline exactly:rng('default')withrng(opts.pilot.seed, 'twister')inPILOT.mandPILOTviewpoint.m(which sharesopts.pilot), and withrng(opts.sifted.seed, 'twister')in
SIFTED.m(both occurrences).rng('default')withrng(obj.opts.general.seed, 'twister')atInstanceSpace.m:561.ISAvalidateOpts.m:checkPosInt(opts, 'pilot', 'seed', true);and thesiftedequivalent, mirroring the existing
general.seedvalidation.Motivation
Closes a real gap in a documented, user-facing reproducibility feature, using a pattern already
proven correct elsewhere in the same codebase — this is copying an existing good pattern, not
designing a new one.
Acceptance criteria
opts.pilot.seed/opts.sifted.seedadded toISAdefaults.m, defaulting fromopts.general.seedrng('default')sites replaced with the corresponding seed (four via the newper-stage opts fields, the
InstanceSpace.m:561one viaopts.general.seeddirectly)ISAvalidateOpts.mopts.general.seedvalues, assert PILOT's
A/Zand SIFTED'sselvarsdiffer between the two runsopts.general.seed, assertresults are still bit-identical (reproducibility itself must not regress)
Part of the
v0.9.1milestone. Source: full-file audit pass over every core algorithm file plus previously-grepped-only utility files. Independent of batches 1, 2, 3, 5. Suggested priority: alongside the migration bug (narrow, clear, concrete fix).