feat: implement dynamic analyzer discovery and risk score validation#74
feat: implement dynamic analyzer discovery and risk score validation#74umran666 wants to merge 3 commits into
Conversation
Resolves TODO A.2.1-A.2.4 (Analyzer discovery) and TODO A.3.2 (Risk score assertion). Signed-off-by: umran666 <shaikumran666@gmail.com>
rng1995
left a comment
There was a problem hiding this comment.
The move to dynamic analyzer discovery is reasonable and removes a lot of boilerplate, and wiring requires_api_key / is_available() gating into create_graph matches the intent of the TODO it resolves. A few issues, one of which I'd consider blocking for a security tool.
1. Import failures are silently swallowed (blocking). In _discover_analyzers, a module that fails to import is caught with except Exception and logged at logger.debug(...), then skipped. Since the default log level is WARNING, a broken analyzer (syntax error, bad import, refactor mistake) silently disappears from every scan with no visible signal — the scan still completes and can report a low/zero risk while an entire detection category isn't running. That's a meaningful regression from the previous explicit-import behaviour, which failed loudly. Please log this at WARNING/ERROR (and ideally surface it in output) so a dropped analyzer is never invisible. It would also help to distinguish "module isn't an analyzer" (no ANALYZER_ID) from "an analyzer failed to import".
2. Confirm the gating attributes actually exist. The new skip logic reads getattr(mod, "is_available", None) and getattr(mod, "requires_api_key", False). The static analyzer modules don't define these (correctly — they need no key), but if the goal is to skip the LLM/semantic analyzers when no key is present, those modules must declare requires_api_key = True; otherwise they default to False and are always wired, making the gating a no-op. Please confirm the LLM-dependent analyzers actually set this.
3. Edge case: if discovery/gating ends up skipping every analyzer, wired_analyzers is empty and nothing connects to meta_analyzer. Worth a guard or at least a warning.
Non-blocking nits:
- Discovery order now depends on
pkgutil.iter_modules(filesystem order) rather than the curated list, and the registry test was loosened to a set comparison. Fine if analyzer order is genuinely irrelevant (parallel fan-out), but a comment noting that ordering is no longer guaranteed would help future readers. - Several of the new lines in
graph.pycarry trailing whitespace, which will likely tripruff/lint.
Happy to re-review once import failures are visible and the requires_api_key wiring is confirmed.
Signed-off-by: umran666 <shaikumran666@gmail.com>
|
Thanks for the review. I've pushed a commit addressing all the feedback:
|
Signed-off-by: umran666 <shaikumran666@gmail.com>
| from skillspector.nodes.resolve_input import resolve_input | ||
| from skillspector.state import SkillspectorState | ||
|
|
||
| logger = get_logger(__name__) |
There was a problem hiding this comment.
[Automated SkillSpector Review]
Non-blocking (formatting): this module-level assignment leaves only one blank line before def create_graph(). ruff format (Black style) enforces two blank lines before a top-level def, so ruff format --check will flag this file — please run make format. Same pattern before def _discover_analyzers() in nodes/analyzers/__init__.py.
rng1995
left a comment
There was a problem hiding this comment.
[Automated SkillSpector Review]
Re-review: blockers resolved — approving.
- Import failures are no longer silent:
_discover_analyzerscatchesImportError/Exceptionand logs atERROR(distinguishing import failure from "not an analyzer"). requires_api_key = Trueis now declared on the three semantic analyzers andcreate_graphgates them onis_llm_available(), so the skip logic is a real gate rather than a no-op.- An empty-
wired_analyzerswarning guard was added.
Non-blocking: the inserted module-level statements leave a single blank line before def create_graph() (in graph.py) and before def _discover_analyzers() (in __init__.py); ruff format wants two, so please run make format to avoid a CI format-check failure.
Resolves TODO A.2.1-A.2.4 (Analyzer discovery) and TODO A.3.2 (Risk score assertion).