[GSoC 2026] Add analyzer/playbook LLM tools to the chatbot agent (refs #3732)#3740
[GSoC 2026] Add analyzer/playbook LLM tools to the chatbot agent (refs #3732)#3740berardifra wants to merge 2 commits into
Conversation
berardifra
added
the
gsoc-2026
github-project-automation
Bot
moved this to Todo
in GSoC '26: "Integrating a Self-Deployed LLM Chatbot for Threat Intelligence"
berardifra
moved this from Todo
to In Review
in GSoC '26: "Integrating a Self-Deployed LLM Chatbot for Threat Intelligence"
berardifra
force-pushed
the
gsoc-2026/llm-chatbot-w5-analyzer-tools
branch
from
e62f543 to
8eff4cc
Compare
|
I think the issues in the conflicts are due to the 2 parallels PRs that change the same code. I merged the previous one without any change so it should be easy to fix the conflict. |
|
This can happen from time to time considering that I can't review the changes as soon as they are published and some parallel PRs are normal in such environments. Try to consider this while doing the changes to avoid having to fix conflicts every time. Isolate the content as soon as possible, abuse creation of additional files (like serializers.py and test_tools.py could become folders and contain isolated tests in separated files). This should reduce the issues |
…#3732) Add two read-only tools to the chatbot ReAct agent: - list_analyzers: lists the enabled observable analyzers, each annotated with a per-user 'runnable' readiness flag (no hard runnable filter, so key-based analyzers stay visible). - recommend_playbook: suggests launchable (starting=True, enabled) playbooks matching an observable's classification, scoped with visible_for_user. Both follow the make_*_tool(user) factory pattern and return the standard {errors, payload} envelope via purpose-built light serializers. Tests cover filtering, enum validation, the result cap, and org/visibility isolation.
berardifra
force-pushed
the
gsoc-2026/llm-chatbot-w5-analyzer-tools
branch
from
8eff4cc to
b2a7dbc
Compare
|
|
||
| # Observable analyzers never apply to the `file` classification (that's the file-analysis path), | ||
| # so the accepted/advertised observable types are all classifications except FILE. | ||
| _VALID_OBSERVABLE_TYPES = [c for c in Classification.values if c != Classification.FILE.value] |
There was a problem hiding this comment.
this information is an IntelOwl core info and should stay logically in the core part.
Please modify the Classification class and add an helper that generates this list and then import it here so it can be reused.
For instance, I noticed that in other parts of the code (see aggregate_observable_classification) we use the same list so it should be nice if you could modify that too while we are here.
| # near-empty lists and flaky tests. We list the enabled analyzers and surface readiness as a | ||
| # per-row flag instead, so the LLM can still say "this analyzer applies but isn't configured | ||
| # for you". `runnable` is False when the analyzer is disabled for the user's organization OR | ||
| # not fully configured/healthy. |
There was a problem hiding this comment.
thanks for the thorough explanation, that's really useful
| errors.append(f"Unknown observable_type '{observable_type}'; valid values are: {valid}.") | ||
|
|
||
| # Clamp the LLM-supplied limit into [1, _MAX_RESULTS] (treat tool args as untrusted). | ||
| limit = max(1, min(int(limit), _MAX_RESULTS)) |
There was a problem hiding this comment.
should we explicitly raise that a clamp has been done here? I mean, with a comment, a "warning" or whatever
Yeah, I was expecting this one: two PRs touching the same files was a known risk on my side, so the rebase onto the umbrella was already part of the plan. Done now: additive conflicts only, kept both sides, 31/31 tests + Ruff green, mergeable again. |
…s + surface limit clamp (refs #3732) - Add Classification.observable_classifications() as the single source of truth for 'every classification except FILE'; reuse it in aggregate_observable_classification - Surface the limit clamp in the tool 'errors' (list_analyzers + recommend_playbook) so a truncated list is never silent - Tests for both clamp warnings
|
Addressed in 761950b, thanks |
2 participants
Refs #3732. (Targets the
gsoc-2026/llm-chatbotumbrella, notdevelop. UsingRefs(notCloses) on purpose — the umbrella isn'tdevelop, so the issue is closed manually after merge.)Description
Adds two read-only tools to the chatbot LangChain ReAct agent (W5 — analyzer/playbook domain):
list_analyzers(observable_type="", limit=50)— lists the analyzers enabled for the requesting user(globally not
disabled, and not disabled for the user's organization), optionally filtered to thosesupporting a given observable type.
observable_typeis validated against theClassificationenum; anunknown value is reported in
errorsinstead of silently returning an empty list.recommend_playbook(observable_name="", classification="", limit=50)— suggests directly-launchable(
starting=True, non-disabled) playbooks applicable to an observable's classification, scoped to theplaybooks visible to the user (
visible_for_user— owned + organization-shared). The classification isderived from the observable via
Classification.calculate_observablewhen not supplied; an explicitlyprovided one is validated against the enum.
Both follow the existing tool pattern (a
make_*_tool(user)factory that closes over the user, somulti-tenancy is enforced at build time and the LLM can't widen the scope) and return the standard
{"errors": [...], "<payload>": ...}envelope viaToolResultSerializer.to_json(). Output usespurpose-built light serializers (
AnalyzerConfigToolSerializer,PlaybookConfigToolSerializer) rather thanthe heavy config serializers, to keep the LLM prompt small. Both cap their result list at 50
(
_MAX_RESULTS) and clamp the LLM-suppliedlimitinto[1, 50], so a single call can't flood the prompt.These tools are read-only — no analysis is triggered. The analysis-launching tool
analyze_observable(with its safety guardrails) follows in a separate PR.
Type of change
Checklist
develop(targets thegsoc-2026/llm-chatbotumbrella)Ruff) gave 0 errorstestsfolder); all tests (new and old) gave 0 errors (run locally against a rebuilt test image — the "Build & Tests" CI workflow does not run on umbrella-targeting PRs)DeepSource,Django Doctorsor other third-party linters triggered alerts during CI, I have solved those alerts