Skip to content

Thread opts.general.seed through PILOT/SIFTED/PILOTviewpoint (and one more site) instead of rng('default') #41

Description

@andremun

Summary

PYTHIA.m correctly threads the user's configured seed through: ISAdefaults.m sets
opts.pythia.seed = opts.general.seed (default 42), and PYTHIA.m uses rng(opts.seed + i, 'twister') per algorithm, with careful re-seeding around each tuning strategy and an
onCleanup guard to restore the client's RNG state even on error.

PILOT.m, SIFTED.m, and PILOTviewpoint.m do not follow this pattern — confirmed none of
them read from opts.general.seed or any per-stage equivalent (no opts.pilot.seed/
opts.sifted.seed field exists anywhere in ISAdefaults.m). rng('default') resets to
MATLAB's factory seed (0) unconditionally, regardless of what opts.general.seed is configured
to.

Correction to the original audit count: there is a fifth rng('default') site, not just
four — InstanceSpace.m:561, in the opts.selvars.smallscaleflag small-scale-subsetting path
(cvpartition(ninst, 'HoldOut', opts.selvars.smallscale)), bracketed by the same state = rng; ...; rng(state); save/restore pattern as the other four. Confirmed directly
(grep -rn "rng('default')" across the whole repo returns exactly five hits:
PILOT.m:206, SIFTED.m:145, SIFTED.m:159, PILOTviewpoint.m:124, and
InstanceSpace.m:561). This site is the same bug in the same family, and the fix is even
simpler there since obj.opts.general.seed is already in scope — no new opts field needed,
just rng(obj.opts.general.seed, 'twister') in place of rng('default').

This is not the historical "silently breaks reproducibility" bug — the state = rng; ...; rng(state) save/restore bracketing is correctly present at all five sites, so results remain
fully reproducible run-to-run. The actual problem: a user who deliberately configures a
different opts.general.seed (e.g. running the same build several times with different seeds
for a replication/variance study — standard methodology this project's own research depends on)
would find PILOT's BFGS multi-start restarts, SIFTED's k-means/GA clustering, and small-scale
subsetting are identical every time regardless of seed choice, silently defeating the point of
making the seed configurable at all.

Proposed change

  1. Add to ISAdefaults.m, mirroring the existing opts.pythia.seed line exactly:
    if ~isfield(opts.pilot, 'seed'),   opts.pilot.seed   = opts.general.seed; end
    if ~isfield(opts.sifted, 'seed'),  opts.sifted.seed  = opts.general.seed; end
  2. Replace rng('default') with rng(opts.pilot.seed, 'twister') in PILOT.m and
    PILOTviewpoint.m (which shares opts.pilot), and with rng(opts.sifted.seed, 'twister')
    in SIFTED.m (both occurrences).
  3. Replace rng('default') with rng(obj.opts.general.seed, 'twister') at
    InstanceSpace.m:561.
  4. Add to ISAvalidateOpts.m: checkPosInt(opts, 'pilot', 'seed', true); and the sifted
    equivalent, mirroring the existing general.seed validation.

Motivation

Closes a real gap in a documented, user-facing reproducibility feature, using a pattern already
proven correct elsewhere in the same codebase — this is copying an existing good pattern, not
designing a new one.

Acceptance criteria

  • opts.pilot.seed / opts.sifted.seed added to ISAdefaults.m, defaulting from
    opts.general.seed
  • All five rng('default') sites replaced with the corresponding seed (four via the new
    per-stage opts fields, the InstanceSpace.m:561 one via opts.general.seed directly)
  • Validation added to ISAvalidateOpts.m
  • Regression test: build the same dataset twice with two different opts.general.seed
    values, assert PILOT's A/Z and SIFTED's selvars differ between the two runs
  • Regression test: build the same dataset twice with the same opts.general.seed, assert
    results are still bit-identical (reproducibility itself must not regress)

Part of the v0.9.1 milestone. Source: full-file audit pass over every core algorithm file plus previously-grepped-only utility files. Independent of batches 1, 2, 3, 5. Suggested priority: alongside the migration bug (narrow, clear, concrete fix).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions