BUG: keep stochastic nominal values stable across reseeds - #1169
BUG: keep stochastic nominal values stable across reseeds#1169thc1006 wants to merge 16 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1169 +/- ##
===========================================
+ Coverage 84.57% 84.62% +0.04%
===========================================
Files 131 131
Lines 17527 17557 +30
===========================================
+ Hits 14824 14857 +33
+ Misses 2703 2700 -3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Since this went up I found three things wrong with it and fixed them. The description above is current, so this is only the record of what moved. The kept nominal was reachable from outside. It copies on the way out now, so what it hands over is disposable.
Configuring a late input twice kept the first nominal. A second Two I left alone. #1171 is the other half of the same lifecycle, where setting a declared eccentricity back to Head is |
_set_stochastic re-validates every declared input, and validation reads the nominal off the wrapped object. create_object writes the sampled value back onto that same object on purpose, so re-reading it on a reseed took one simulation's output as the next one's nominal: a wind factor compounded 10 -> 8.576 -> 7.355 -> 6.308 under a single fixed seed, and a plain scalar spec drifted the same way. Read the nominal once and keep it. Containers are copied on the way in, so writing through the wrapped object cannot reach it either. A component position arrives through an injected getter, reads an attribute nothing writes back to, and shares one name across every component, so those are read live rather than cached. Extracted from RocketPy-Team#1054. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Copying on the way in was not enough. _nominal handed back the kept object itself, and on the empty-spec path that one object reached the model attribute, last_rnd_dict and the FreeFormFins create_object returns, so a write through any of them moved what the next reseed sampled around. _snapshot_of stopped at a tuple as well, which left an array inside an airfoil pair shared with the object it came from. Copy on the way out too, and recurse through the built-in containers. The documented contract now names the four cases that stay outside it: an input added after construction, a component position, an ensemble wind factor, and anything that is not an array or a built-in container. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
An add_* input is configured after __init__, so its nominal is read then. The new test writes the rocket's eccentricity before add_cp_eccentricity and again after it, and only the first one may reach the draw. The (std, distribution) form now runs its own seed histories rather than repeating one seed, which is what a cache keyed by the seed instead of by the model actually fails. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The documented exception said a Function is held as it was given. Nothing enforced it, so closing the hole later would have gone unnoticed and the documentation would have quietly become wrong. Measured: set_source on the rocket's drag curve moves the drawn value from 0.377 to 0.890, and deepcopy of that curve costs 6 microseconds. Cost is not the reason to leave it. _snapshot_of cannot raise today, and deepcopying whatever a user passed, on a path that runs on every reseed, would make it able to. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Keeping the nominal gave the second add_cp_eccentricity nothing to replace, so it went on sampling around the value the rocket held at the first call: 0.5 where 0.8 was asked for. Reproducible, and around the wrong centre, which is harder to notice than a value that moves. Late configuration drops the kept nominal before validation reads one, and puts it back if validation raises, so a refused call leaves the previous configuration standing. Only the reconfiguration path. Passing None still leaves the earlier declaration in place, which is develop's behaviour and not this branch's. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Measured on the current implementation: a cycle recurses until Python stops it, two references to one list come back as two lists, and the elements of an object-dtype array stay shared. None of those reach a supported nominal, but the docstring read like a general deep copy and should not. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
None is a configuration too. The nominal was refreshed but the earlier distribution stayed declared, so the next reseed validated it again and drew an uncertainty the caller had asked to remove. Filed as RocketPy-Team#1171 while the removal lived elsewhere; it belongs in the replacement helper this branch added, so it is here rather than in a second PR that owns the other half of one state transition. None still means an axis that was never given, and removing what was never declared stays a no-op. Both meanings have a test. The snapshot test asserted a dict entry was not None, which held whether or not anything had been copied, and no test reached the set branch at all. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
add_cp_eccentricity takes x and y in one call, so a y that will not validate left x already replaced and declared. Validation happens for the whole group before anything is committed now. The test gives only y first, so x is undeclared going in and a partial commit shows up as an eccentricity the caller never successfully asked for. Asserting the nominal alone did not catch it, since x's nominal was restored either way. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
9766211 to
798c0c1
Compare
Copying on the way out of the kept nominal was not the last boundary. The list
branch handed back the candidate itself, which for an empty spec is the model's
own working value, and FreeFormFins keeps shape_points by reference. Writing
through the first generated fins reached the second ones:
first = stochastic.create_object()
first.shape_points[1] = (9.9, 9.9)
second = stochastic.create_object() # (9.9, 9.9) as well
No reseed in between, which is how create_object is documented to be used and
how a serial Monte Carlo runs it.
last_rnd_dict was the same dictionary the values were built from, so it moved
with them too. It records what was drawn now, which matters because a Monte
Carlo writes it out after the flight rather than before.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Recording in the base generator put the record before StochasticFreeFormFins had pulled the fin root back onto the body line. Under seed 7 the two root points drifted to 0.000299 and 0.001340, the correction returned them to zero, and the record kept the outline the fins were never built from. The rocket copies each component's record into its own, so the Monte Carlo input log carried it too. That is the failure class RocketPy-Team#1090 was about, arriving from the other side. _record_draw is the one place a model publishes what it drew, and a subclass that changes a value calls it again. A source scan holds the next subclass to the same rule, since the one that gets it wrong is the one nobody wrote a fixture for. _declare_stochastic_input and the _MISSING sentinel had no callers left after the grouped reconfiguration landed, and the first still carried the None handling that RocketPy-Team#1171 was about, so they are gone rather than left as a second lifecycle for someone to reach for. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
The docstring still promised values itself when there are no candidates, which stopped being true when the draw started handing back a copy. Nothing reaches that branch through a validated input, since an empty list validates to the object's own value, so it is a guard against integers(0) rather than a path with a caller. It has a test now, which is also the one line of this change Codecov had no coverage for. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
StochasticParachute.create_object derives the pressure noise seed after the draw, and RocketPy-Team#1134 relied on last_rnd_dict being the same dictionary to carry it into the record. Snapshotting the draw broke that link: the parachute is still built with the seed, but the record loses it, so the Monte Carlo inputs stop describing the parachute that flew. develop recorded 37773913418288439290323614982376424810 before recorded <absent> The source scan missed it because it only read dict_generator overrides. It reads create_object too now, and tracks the names a method binds from a draw rather than guessing at a variable name, so a local a method fills in for its own use is not mistaken for a record. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
StochasticMotorModel is a StochasticModel subclass that rocketpy.stochastic does not export, so the scan could not see it. It overrides neither method today, which is why nothing was wrong, and which is also why the gap would have gone unnoticed until something did. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Removing on None looked like one line inside the new replacement helper, and
it is not. add_cp_eccentricity(x=..., y=...) defaults both to None, so an
omitted axis and an explicit None read identically, and the removal took away
an axis the caller never mentioned:
add_cp_eccentricity(x=0.001, y=0.002)
add_cp_eccentricity(x=0.005) # y quietly gone
develop keeps y here, and so does this again. Removing an earlier declaration
needs an argument omission cannot supply, which is a signature change and its
own decision, so it stays in RocketPy-Team#1171 rather than arriving inside a change about
nominal ownership.
The test that asked for removal is replaced by one that holds the omitted axis
in place, since that is the behaviour anything already written depends on.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Both arguments read as optional and nothing said what happens when the method is called again, which is the whole of the question behind RocketPy-Team#1171. Each public docstring now states it: a later call replaces what was configured, an omitted axis keeps what it had, and taking one away is not supported. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
Keeping its declaration was not enough. The omitted axis still went through
the whole replacement: its kept nominal was dropped, None was validated again
into a lone nominal, and that was written back over its distribution. The
private side then said the axis was random while the attribute dict_generator
reads said it was not, so it stopped varying:
add_cp_eccentricity(x=0.001, y=0.002)
add_cp_eccentricity(x=0.005)
eight draws of y -> one distinct value
A serial Monte Carlo never resets, so a whole study would have run with that
axis switched off and nothing raised.
Dropping the nominal also moved the centre. With the rocket's own y changed
between the two calls, the next reset centred the old distribution on 9.0
rather than the 0.0 it was configured around.
An axis given as None that already has a configuration is now left out of the
transaction: not revalidated, its nominal not re-read, its attribute not
rewritten. The test covers both eccentricity methods and looks before the
reset as well as after, which is where the previous one missed it.
Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
119 lines of this are production code. Most of it is
stochastic_model.py; the other three files get a line or two each, hooking the same recording call. The other 600 are the tests that each of the problems below cost, and a note in the user documentation. Reading the four hunks instochastic_model.pyis enough to judge the change.Problem
_set_stochastic(seed)re-validates every declared input, and validation reads the nominal off the wrapped object.create_objectwrites the sampled value back onto that same object on purpose, so re-reading it on a reseed takes one simulation's output as the next one's nominal.Four reseeds of one model, all with seed
12345,wind_velocity_x_factor=(1.0, 0.1)on a nominal 10 m/s:Each value on
developis the one before it multiplied by the same factor again. A plain scalar spec drifts the same way, since_validate_scalarand the(std, "distribution")tuple both take their centre from the object.Invariant established by this PR
A model samples around the value the wrapped object held when that input was configured, however many times it is reseeded and whatever ran before. For the types that get copied, nothing downstream can move it: not the generated object, not
last_rnd_dict, and not the next draw. AFunctionor a callable is held by reference and its own mutators still reach the baseline, whichdocs/user/stochastic.rstsays.Configured, not built, because an
add_*method installs its input after__init__and is read then.docs/user/stochastic.rststates the rule and the four cases that sit outside it: a lateadd_*input, a component position, an ensemble wind factor under a selected member, and anything held by reference.Changes
last_rnd_dictand the objectcreate_objectreturns, and a write through any of those used to land on the kept one;airfoiltuple would otherwise stay shared;Nonearrive the same way; removing one deliberately is BUG: setting a declared eccentricity back to None leaves the old distribution in place #1171;developa whole batch of simulations shares one outline, and that outline is the deterministic fin the caller passed in:A serial
MonteCarlorun never resets between simulations, so this holds for the whole study rather than for adjacent pairs;_record_draw, after any subclass that adjusts a value.StochasticFreeFormFinscorrects a perturbed outline indict_generatorandStochasticParachutederives its pressure noise seed increate_object, both once the base class has already recorded;_declare_stochastic_inputand the_MISSINGsentinel, which the grouped reconfiguration left without callers;positionacross every component;docs/user/stochastic.rstwhere a nominal comes from, since three places there said only that it is taken from the deterministic object.Non-goals
MonteCarloAPI changes;StochasticRocketcomponent-stream changes, which are BUG: give each StochasticRocket component its own random stream #1170;Noneapart from an omitted argument, both axes default toNone, and a private sentinel default is a signature change rather than a bug fix. There is a regression test here holding the omitted axis in place, so nothing gets worse in the meantime.Breaking change
Yes, for a model whose nominal was drifting. Nothing is drawn differently and no draw is added or removed, so the stream position is untouched:
stochastic_calistounder seed 42 readsmass=14.906007947ondevelopand the same here.Verification
Focused first, then the suites the CI jobs run:
The four
tests/unit/test_sensitivity.pyfailures on this machine are a missingstatsmodels, and they fail the same way on an unmodifieddevelop.Each mechanism is pinned by a mutation, and each mutation leaves a control standing:
_snapshot_of's own test_snapshot_ofcopies nothing_snapshot_ofstops at a tuplecreate_objecttestlast_rnd_dictpoints at what was built fromtest_visualize_attributes[stochastic_calisto]positiontooTwo of those exist because a weaker fix passes everything else. Keying the cache by seed satisfies "the same seed four times agrees" and still re-reads a moved nominal whenever the seed changes. Collapsing
_snapshot_ofto a shallow copy satisfies every drift test, because those use scalars.Together with #1170
Both touch the stochastic package, and green on separate bases says nothing about the merge, so they are merged into a throwaway tree on
developand run there:pylintexit 0,ruffclean, the unit suite, doctests, integration and acceptance. The mutations that undo either change still fail their own tests in the combined tree, so neither is masking the other.stochastic_calistounder seed 42 readsmass=14.906007947there too.Related to #1171, which stays open. The reasoning is recorded there.
Extracted from
#1054, as the first of the split you asked for in #1054 (comment).
The idea is that branch's. The code is not: copy-on-read, the recursive snapshot and the reconfiguration path are all new here, and its older one-argument
_declare_stochastic_inputis deliberately left behind, since #1167 replaced it ondevelopwith a version that keeps the value.