fix(libraries): load node modules under stable namespaces so pickled values survive restarts

[collindutter]

Dragging an image onto the canvas failed with No module named 'gtn_dynamic_module_set_variables_from_data_py_4816193767510271467'. Library node files were executed under a module name containing hash(str(path)), which Python randomizes per process, so any pickled value referencing a class from such a module could never unpickle after an engine restart. The workflow-save path masked this by patching __module__ to a stable alias before pickling, but the image-metadata path had no such patch, and future pickle sites would have carried the same landmine.

Rather than add the patch to another call site, this makes the existing stable namespace (griptape_nodes.node_libraries.<lib>.<file>) the module's canonical name at load time. Classes carry a stable __module__ by construction, so every pickle site is portable without per-site class patching. The namespace format remains byte-for-byte identical to the old alias because it is already embedded in saved-workflow pickles. Same-stem file collisions are detected and disambiguated rather than silently clobbering one another.

The hashed name was not providing generation identity for hot reload. It existed before hot reload was added, and the reload implementation reused and replaced the same sys.modules key. The stable loader preserves that lifecycle: it installs a fresh module object under the same key, restores the previous module if execution fails, and releases the old module after replacement. Tests verify that live old instances keep their old class generation while the old module and class are collected when their external references disappear.

Images saved by earlier engines already contain volatile names, so the extract path also gets an Unpickler that maps a failed gtn_dynamic_module_* reference to a uniquely matching loaded stable module. Recovery requires the requested class to be defined by that module. If multiple libraries match, extraction fails with an artist-readable ambiguity error rather than guessing and risking silent corruption. If nothing matches, the user is told to install or enable the originating library instead of receiving a raw ModuleNotFoundError.

_patch_and_pickle_object remains only to align SerializableMixin.module_name fields before workflow serialization. The deeper issue, pickle coupling portable artifacts to Python import paths at all, is tracked in #4475.

Note

Beep boop. This PR description was written by AI.

[codecov]

Codecov Report

❌ Patch coverage is 92.34450% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...pe_nodes/retained_mode/managers/library_manager.py	93.40%	12 Missing ⚠️
...e_nodes/retained_mode/managers/workflow_manager.py	73.33%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[collindutter]

Supplemental technical deep dive

This is intentionally separate from the PR description. It documents the complete failure chain, why this approach was chosen, the hot-reload concern, and what remains architecturally unresolved.

1. The user-visible failure

Dragging a generated image onto the canvas produced:

Failed to unpickle flow commands: No module named
'gtn_dynamic_module_set_variables_from_data_py_4816193767510271467'

Generated images can contain a workflow in PNG metadata:

gtn_flow_commands = base64(pickle(SerializedFlowCommands))

When an image is dragged onto the canvas, the engine:

1. Opens the PNG.
2. Reads gtn_flow_commands.
3. Base64-decodes it.
4. Uses pickle to reconstruct the workflow commands.
5. Deserializes those commands back into nodes and connections.

The failure occurred during step 4.

2. How library modules were loaded

Node library files were not imported using normal package names. The engine loaded each file using a synthetic module name:

module_name = (
    f"gtn_dynamic_module_"
    f"{file_path.name.replace('.', '_')}_"
    f"{hash(str(file_path))}"
)

For example:

gtn_dynamic_module_set_variables_from_data_py_4816193767510271467

Using a hash of the full path helped distinguish two files with the same filename in different directories.

The problem is that Python intentionally randomizes string hashes for every process:

Process A: hash("/path/set_variables_from_data.py") = 4816193767510271467
Process B: hash("/path/set_variables_from_data.py") = -269148924846342774

The same file therefore received a different module name after restarting the engine. Within one engine process the name was stable; across processes it was not.

3. Why pickle cared

Python pickle does not normally serialize a class implementation. It records where to import the class from:

(module name, class name)

For a CollisionBehavior value, the pickle contained something equivalent to:

(
    "gtn_dynamic_module_set_variables_from_data_py_4816193767510271467",
    "CollisionBehavior",
)

When loading the pickle later, Python effectively tried:

from gtn_dynamic_module_set_variables_from_data_py_4816193767510271467 import CollisionBehavior

After an engine restart, that module name no longer existed. The same file had been loaded under a different hash, producing the reported ModuleNotFoundError.

4. Why SetVariablesFromData surfaced the problem

SetVariablesFromData defines this enum in its node file:

class CollisionBehavior(StrEnum):
    OVERWRITE = "Overwrite existing"
    PRESERVE = "Preserve existing"
    ERROR = "Error on collision"

It then uses an enum member as a string parameter's default:

Parameter(
    name="collision_behavior",
    type=ParameterTypeBuiltin.STR.value,
    default_value=CollisionBehavior.OVERWRITE,
)

Although CollisionBehavior is a StrEnum, the default is still an enum object whose class belongs to the dynamically loaded node module.

SerializedFlowCommands stores unique parameter values as raw Python objects. Consequently, the embedded flow commands included a real CollisionBehavior instance, and pickle recorded the volatile module name.

The node did not create the architectural problem. It provided a common path that reliably exposed a latent issue. Changing the default to CollisionBehavior.OVERWRITE.value would avoid this particular trigger, but other library-defined enums, artifacts, or custom classes could cause the same failure.

5. Why normal saved workflows mostly worked

This problem had already appeared for generated workflow files. PR #1591 introduced a stable alias system:

volatile:
gtn_dynamic_module_image_to_video_py_123456789

stable:
griptape_nodes.node_libraries.runwayml_library.image_to_video

When a library loaded, the same module object was registered under both names in sys.modules.

Before workflow parameter values were pickled, WorkflowManager recursively walked the values and temporarily changed class references from the volatile name to the stable alias. After pickling, it restored them.

That allowed generated workflow files to contain stable references. The image-metadata path did not use that helper. It performed a plain:

pickle.dumps(serialized_flow_commands)

There were therefore two serialization paths:

• Workflow file generation patched dynamic classes before pickling.
• Image metadata serialization did not.

The stable-alias system fixed one serializer rather than fixing module identity itself. Any future plain pickle call could repeat the bug.

6. How the issue was reproduced

A minimal module containing a StrEnum was loaded under the same synthetic naming scheme, and an enum member was pickled in one Python process.

Process A produced:

gtn_dynamic_module_node_module_py_-8859640815979518826

A second process loaded the same file as:

gtn_dynamic_module_node_module_py_8779953375495063909

Unpickling Process A's payload in Process B failed with:

ModuleNotFoundError:
No module named
'gtn_dynamic_module_node_module_py_-8859640815979518826'

That reproduced the Slack error exactly.

Loading the module under this name instead:

griptape_nodes.node_libraries.my_library.node_module

made both processes use the same module name, and the pickle loaded successfully.

7. Possible fixes considered

Patch the image serializer

The image path could call the existing patch-and-pickle helper.

That would be the smallest immediate change, but:

• It would protect only that serializer.
• Every new pickle site would need to remember the same rule.
• It would not recover images already saved with volatile names.
• The class's canonical __module__ would remain process-specific.

This is the narrow workaround.

Make the hash deterministic

Python's hash() could be replaced with SHA-256 or another deterministic hash.

That would stabilize names for the same path, but hashing an absolute path would still produce different names on different machines or installations. Hashing a library-relative path would improve that, but the module would still have an artificial implementation-oriented identity, and the stable alias system would remain.

Load the module under its stable namespace

The third option was to make the existing stable alias the module's real name:

griptape_nodes.node_libraries.<library>.<file>

Classes then receive a stable __module__ when they are defined. Pickle works naturally without temporarily mutating class metadata.

This path was chosen because it fixes the invariant at the source:

A library class's canonical Python module identity must survive an engine restart.

8. What changed in the loader

Library files now execute directly under the stable namespace.

The loader creates synthetic parent packages such as:

griptape_nodes.node_libraries
griptape_nodes.node_libraries.griptape_nodes_library

and installs the leaf module:

griptape_nodes.node_libraries.griptape_nodes_library.set_variables_from_data

This is necessary because pickle imports the dotted module path during reconstruction.

The stable namespace format is identical to the old alias format. Existing workflow pickles already contain those stable names, so changing the format would have broken them.

The loader also retains:

• Fresh module objects for reloads.
• Rollback to the previous module if execution fails.
• Module tracking by library for unload.
• Same-filename collision detection.
• Deterministic disambiguation when two files map to the same stable namespace.

9. The hot-reload and garbage-collection concern

A concern was raised that dynamic names might have existed to isolate reload generations so old modules could be garbage-collected.

Repository history showed otherwise:

1. Hashed names existed in the initial engine import.
2. Hot reload was added later in PR Added ability to hot reload a library (currently only triggered by exiting current workflow and re-opening it) #278.
3. Hot reload reused the same hashed name within the process.
4. It popped and replaced the existing sys.modules entry.

The old implementation did not create generation names such as:

module_generation_1
module_generation_2

It used one name per path per process. The hash provided file identity, not reload-generation identity.

The stable loader preserves the relevant behavior:

stable name -> old module
stable name -> fresh module after reload

A lifecycle regression test proves:

• Reload produces a different module object under the same name.
• An existing instance continues using its old class generation.
• The old module is collectible after replacement.
• The old class remains alive while an old instance references it.
• The old class is collected once that instance is released.

The previous design also stored the module under two sys.modules keys, the dynamic name and the stable alias. The new design stores one key.

Stable naming therefore does not sacrifice same-process garbage collection.

10. Why stable loading alone was incomplete

Stable loading prevents newly written images from containing volatile names, but it cannot repair images already saved by older engines. Those images permanently contain names such as:

gtn_dynamic_module_set_variables_from_data_py_4816193767510271467

A crafted reproduction PNG correctly continued failing after the initial loader change. That showed write-side correctness was insufficient; backward-compatible reading was also required.

11. Legacy image recovery

The image extraction path now uses a custom Unpickler.

Normal Python resolution is attempted first. If it fails with a missing module, the loader checks whether the requested name matches the old format:

gtn_dynamic_module_<filename>_<hash>

It extracts the old file stem, for example:

set_variables_from_data

It then searches loaded stable library modules for:

1. The same file stem.
2. The requested class.
3. A class actually defined by that module rather than merely re-exported.

If exactly one module matches, unpickling continues using that stable module.

If nothing matches, the user gets an actionable message telling them the originating library is not loaded.

If multiple libraries match, the engine fails safely and lists the candidates. It does not guess, because selecting the wrong class could silently corrupt the workflow.

Recovery is intentionally limited to old volatile module names.

12. Review hardening

The review loop caught several edge cases:

• Collision detection stores the exact canonical path used by the loader.
• Repeated reloads of a collision-disambiguated file no longer emit repeated warnings.
• Synthetic module roots do not leak into artist-facing error messages.
• Ambiguous legacy references fail rather than selecting the first library.
• Re-exported classes are not treated as definitions from the candidate module.
• Dead class-__module__ patching was removed.
• Impossible missing-stable-namespace branches were removed.
• Hot-reload and garbage-collection behavior received explicit tests.

13. Why not retain the narrow patch and wait for workers?

A narrow image-serialization patch would be a smaller diff, but it would preserve this rule:

Every serializer must remember that library classes have an invalid persistent identity and manually repair them.

That is how the image path was missed.

Moving libraries into worker processes will eventually make reload easier because restarting a worker naturally destroys its imported modules. It does not make process-specific type names a good persistence format. Values crossing process boundaries need stable identities even more clearly.

Stable module names are compatible with the worker direction. Worker restarts may later replace the same-process reload machinery, but they do not require reverting module identity.

14. What remains fundamentally imperfect

The remaining architectural problem is pickle itself.

Even with stable module names, embedded workflows are coupled to:

• Python module paths.
• Class names.
• Internal event dataclasses.
• Library versions and field layouts.

Renaming a library file or moving an engine class can still break older artifacts.

More importantly, unpickling metadata from an arbitrary image can execute Python code. The assumption that the pickle is safe because the application wrote it does not hold once images are shared between users.

The long-term format should be structured data with:

• Stable logical type IDs.
• Library and type versioning.
• Explicit migrations.
• Registry-based type resolution.
• Validation before object construction.
• No arbitrary code execution.

That direction overlaps with #4475 and the existing cattrs and SerializableMixin work.

Bottom line

The immediate root cause was process-randomized module identity leaking into pickle.

The latent architectural mistake was making the stable namespace an alias and relying on every serializer to patch class metadata.

This PR fixes that by making the stable namespace canonical, while preserving reload semantics and recovering old artifacts safely.

The deeper remaining issue is using pickle as the portable workflow format at all.

[SavagePencil]

minor Q

[SavagePencil]

a computer told me this, so grain of salt:

Because find_class dynamically imports modules and searches for attributes, it can theoretically throw any exception associated with importing code or executing module-level code. For instance, if you override it or use custom objects, it could also result in an UnpicklingError or standard runtime exceptions.