Summary
-target_noise (and -feature_noise) are parsed but never passed to evaluate_model(), so noise is silently ignored on local runs. A method then appears to have identical accuracy at every noise level.
How I found it
Benchmarking a method locally via analyze.py --local ... -sym_data, I got an identical symbolic-solution rate at target noise 0, 0.001, 0.01, and 0.1 — contradicting independent noise-sensitivity measurements of the same method. The evaluate_model.py message adding noise to target never printed.
Root cause
In experiment/evaluate_model.py:
-target_noise is parsed into args.Y_NOISE (~line 275), -feature_noise into args.X_NOISE.
The noise-application code is present and correct (~lines 132–139).
But the evaluate_model(...) call (~line 306) does not pass them:
python
evaluate_model(args.INPUT_FILE,
args.RDIR,
args.RANDOM_STATE,
args.ALG,
algorithm.est,
algorithm.model,
test = args.TEST,
**eval_kwargs
)
Since eval_kwargs comes from the algorithm definition (not the CLI noise arguments), target_noise and feature_noise fall back to their defaults of 0.0, and the noise block never executes.
Reproduction
python analyze.py <feynman_dataset>.tsv.gz -ml <method> -results /tmp/x
-n_trials 1 -target_noise 0.1 --local -sym_data
adding 0.1 noise to target never prints and the result matches the noiseless run.
Fix that worked for me
python
test = args.TEST,
target_noise=args.Y_NOISE,
feature_noise=args.X_NOISE,
**eval_kwargs
After this, the noise message prints and discovered models change under noise as expected.
Impact
Affects local (--local) ground-truth runs with target or feature noise; the cluster path is unaffected since it builds a fresh command string that includes -target_noise. Anyone using the local path to characterize a method's noise robustness would get silently noiseless results.
Environment: Python 3.14, but the bug is independent of Python/pandas version — it is purely that the parsed argument is not passed to the function call. Happy to open a PR with the fix if useful.
Summary
-target_noise (and -feature_noise) are parsed but never passed to evaluate_model(), so noise is silently ignored on local runs. A method then appears to have identical accuracy at every noise level.
How I found it
Benchmarking a method locally via analyze.py --local ... -sym_data, I got an identical symbolic-solution rate at target noise 0, 0.001, 0.01, and 0.1 — contradicting independent noise-sensitivity measurements of the same method. The evaluate_model.py message adding noise to target never printed.
Root cause
In experiment/evaluate_model.py:
-target_noise is parsed into args.Y_NOISE (~line 275), -feature_noise into args.X_NOISE.
The noise-application code is present and correct (~lines 132–139).
But the evaluate_model(...) call (~line 306) does not pass them:
python
evaluate_model(args.INPUT_FILE,
args.RDIR,
args.RANDOM_STATE,
args.ALG,
algorithm.est,
algorithm.model,
test = args.TEST,
**eval_kwargs
)
Since eval_kwargs comes from the algorithm definition (not the CLI noise arguments), target_noise and feature_noise fall back to their defaults of 0.0, and the noise block never executes.
Reproduction
adding 0.1 noise to target never prints and the result matches the noiseless run.
Fix that worked for me
python
test = args.TEST,
target_noise=args.Y_NOISE,
feature_noise=args.X_NOISE,
**eval_kwargs
After this, the noise message prints and discovered models change under noise as expected.
Impact
Affects local (--local) ground-truth runs with target or feature noise; the cluster path is unaffected since it builds a fresh command string that includes -target_noise. Anyone using the local path to characterize a method's noise robustness would get silently noiseless results.
Environment: Python 3.14, but the bug is independent of Python/pandas version — it is purely that the parsed argument is not passed to the function call. Happy to open a PR with the fix if useful.