fix(eval): cache custom-coded evaluator modules instead of leaking sys.modules

[adrian-badea]

Problem

EvaluatorFactory._create_custom_coded_evaluator_internal names the dynamically loaded evaluator module _custom_evaluator_<stem>_<id(data)>. Because data is a fresh dict on every call, id(data) is unique each time, so the sys.modules cache never hits: the module is re-exec()'d on every call and a new module object (plus all the classes / pydantic schemas it defines) is inserted into sys.modules and never removed.

Any consumer that builds custom-coded evaluators repeatedly leaks memory and CPU without bound. Building evaluators per datapoint turns an eval run into O(n²) with unbounded sys.modules growth — observed downstream as one core pinned for 2h+ at ~9 GB RSS on a ~47K-datapoint run, before any inference ran.

Fix

Key the module name on a sha256 of the resolved file path so sys.modules acts as a real load-once cache: the module is loaded once and reused. The load is guarded with module = sys.modules.get(name); if module is None:, and the entry is popped if exec_module fails so a broken load isn't left cached (and can be retried after the file is fixed).

Per-config validation (TypeAdapter(cls).validate_python(data)) is unchanged, so distinct configs still yield distinct, correctly-configured evaluator instances despite sharing one class object.

Tests

New test_evaluator_factory.py::TestCustomCodedEvaluatorModuleLoading:

• repeated creation reuses one module — same class object, a single sys.modules entry (this assertion fails on the pre-fix code);
• a module that raises on import is not left cached and can be retried after the file is fixed (covers the exec-failure cleanup);
• two files sharing a basename load as distinct modules (covers the path-hash disambiguation).

ruff, mypy, and the full tests/evaluators suite (500 passed) pass locally.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes unbounded sys.modules growth and repeated exec() of custom-coded evaluator modules by switching to a stable, path-hash-based module name so dynamically loaded evaluator files are cached and reused across EvaluatorFactory.create_evaluator(...) calls.

Changes:

• Cache custom-coded evaluator modules in sys.modules using a module name derived from the evaluator file’s resolved path (sha256-based), avoiding per-call module re-execution and memory leaks.
• Ensure failed module loads don’t leave a half-initialized module cached by popping the entry on exec_module failure.
• Add regression tests covering caching behavior, failure cleanup, and disambiguation for same-basename files in different directories.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
packages/uipath/src/uipath/eval/evaluators/evaluator_factory.py	Changes custom evaluator module naming/loading to be path-hash keyed and cached, with cleanup on import failure.
packages/uipath/tests/evaluators/test_evaluator_factory.py	Adds tests verifying module caching, retry-after-failure behavior, and no collisions for same stem in different directories.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[adrian-badea]

Thanks — addressed the SonarCloud findings:

• python:S5778 (test_evaluator_factory.py) — fixed in 1cade6c5 by hoisting the _config(...) call out of the pytest.raises block, so only create_evaluator(...) can throw inside it.

• pythonsecurity:S6549 — "construct the path from user-controlled data" at evaluator_factory.py:157. This is the custom-coded-evaluator plugin loader: the path comes from the eval set's evaluatorSchema (file://<path>:<Class>), i.e. trusted developer/eval config, and dynamically loading that configured module is the intended feature. It's also pre-existing — the same spec_from_file_location(<config-derived path>) sink existed before this PR; the refactor only moved the line, so it's re-scanned as "new code" (and .resolve() if anything narrows traversal rather than widening it). There's no way to "not construct the path from user-controlled data" here without dropping custom-coded evaluators entirely, so I'd suggest resolving this as Safe / Won't Fix in SonarCloud rather than adding a sanitizer that doesn't fit the plugin model. If the team would rather keep the gate green via code, I'm happy to add a scoped suppression with this justification instead — just let me know which you prefer.

— Adrian's Claude

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

💡 Need a hand with PR review? Try Gitar by Sonar!