feat: OWASP Top 10 test set generation (multi-turn, tagging) - #2202
Conversation
Rebasing onto main left two alembic heads: the owasp behaviors/metrics migration still pointed at its pre-rebase parent instead of the new head introduced by the joined_at migration. Re-point down_revision to restore a single linear chain.
Add short report summaries to the categories API, warm via list_category_summaries, silence pdfminer noise, and tolerate local storage cache write failures.
Merge OWASP into the Garak source drawer with shared form styling, category descriptions, project-description purpose, and a taller fill-layout probe list.
Keep unified source drawer and OWASP handlers; adopt isSessionLoading and tokenless ApiClientFactory from main.
Rebase OWASP alembic onto d4e7f2a9c1b8, restore SecuritySource, fix chip sx types, and apply prettier.
# Conflicts: # apps/frontend/src/app/(protected)/metrics/components/MetricsDirectoryTab.tsx # apps/frontend/src/app/(protected)/test-sets/components/GarakImportDrawer.tsx # apps/frontend/src/app/(protected)/test-sets/page.tsx
Merging main into owasp-synthesizer twice left the owasp behaviors/ metrics migration pointed at a stale down_revision, producing two alembic heads. Re-point it to main's current head so the chain stays linear, matching what check-alembic-head.yml enforces on PRs.
load_initial_data created the OWASP-prefixed behaviors/metrics for every organization but never tagged them, unlike the migrations that backfill the "OWASP" Tag for organizations that already existed when those migrations ran. Freshly onboarded organizations therefore never got the tag, so the frontend's OWASP filter pill (which matches purely on it) had nothing to find. Tag them at creation time too, mirroring the get-or-create idiom already used by the migration and by Garak's importer.
generate_and_save_owasp_test_set called _build_task_result missing its required tests_generated argument, raising on every invocation. The save had already committed a new test set by that point, and Celery's autoretry_for retried the whole task body 3 more times before reporting failure -- one requested generation produced 4 orphaned test sets. Add the missing argument, matching the sibling task's call.
The pill's visibility was gated on an accumulator that only flipped true once an already-fetched page happened to contain an OWASP-tagged metric. With OWASP metrics batch-created together, they can sort entirely past page 1, so on a typical first load the pill never appeared even though the tag data was correct. Every organization is guaranteed to have OWASP-tagged metrics by construction (onboarding or the backfill migration), so show it unconditionally, like "All", and drop the now-dead accumulator tracking.
The "unify garak and owasp drawers" refactor dropped the editable "System under test" field and replaced it with a hard requirement that the active project already have a description, used verbatim with no way to customize it per generation. Restore the original free-text field (still best-effort prefilled from the project description, never clobbering a user edit). Also regroup the form: Test Type and Number of Tests move out of "Advanced Options" into a new "Test Configuration" section alongside System under test, since none of these are safe-to-ignore defaults — Test Set Name and the model selector are the only settings that actually belong in "Advanced Options". Add tooltips explaining what Single-Turn/Multi-Turn generate.
![peqy[bot]](https://avatars.githubusercontent.com/in/1930020?s=60&v=4){kind=small}
There was a problem hiding this comment.
[Improvement] OWASP endpoints do heavy sync I/O in the async request path
Fix:
get_categories()callslist_category_summaries()(which can download/parse PDFs and hits storage/Redis) directly from anasync defroute. Consider wrapping that call inawait asyncio.to_thread(...)(similar to the startup warm-up) to avoid blocking the event loop on a cold cache.
[Question] Frontend capability gating vs backend RBAC
Fix: backend routes derive
owasp:read/owasp:create(resource="owasp"), but the Test Sets page currently enables the OWASP option based onCapability.TestSet.GENERATE. Can you confirm built-in roles + frontenduseCan(...)checks line up with the newowasp:*capabilities so users don’t hit a UI-visible option that 403s?
[Improvement] Content cache may become permanently stale
Fix: the object-store cache is
owasp/{framework}.jsonwith no TTL andcache_key=framework. If the upstream PDF changes (or URL changes but framework id stays the same), you’ll keep serving old sections forever unless manually purged. Consider keying the content cache by URL hash/version (and keeping the Redis metadata TTL as-is), or adding an explicit refresh mechanism.
Overall this is a solid end-to-end slice (API + task + SDK + UI + backfill/tagging) and the regression test around duplicate test set creation is especially valuable.
In-person review feedback: people may not know what Garak or OWASP actually are. Replace the plain Source dropdown with a pill-shaped segmented selector (reusing PrimarySegmentedPills, already used for directory filter pills) inside a bordered box, with a description of whichever tool is currently selected underneath and a link to learn more. Descriptions explain what each thing is, not what Rhesis does with it — Garak as NVIDIA's open-source LLM security scanner, OWASP as the nonprofit behind the Top 10 risk lists for LLM/agentic applications. Garak links to its GitHub repo; OWASP links to genai.owasp.org, the current umbrella project covering both Top 10 lists (not the legacy, explicitly-archived www-project-top-10-for-large-language-model- applications page).
da099e9 to
cdf030b
Compare
|
Update review after latest commits: [Improvement]
[Improvement] OWASP report content cache can go stale forever
[Question] Frontend OWASP gating is based on
Also: nice fix on the duplicate OWASP test set incident ( |
harry-rhesis
left a comment
harry-rhesis
left a comment
There was a problem hiding this comment.
Thanks Alexey, this is a lot of careful work and it shows. A few things I want to call out
specifically before the findings:
- The three migrations are idempotent, and ordered correctly in both directions (behaviors before
metrics on the way up, metrics before behaviors on the way down). The docstrings explain why
rather than restating what the code does, which made them quick to verify. - Sourcing the OWASP names from
initial_data.jsonby prefix instead of duplicating 22 string
literals into the migration. That is the right call and it kept the migration short. - The two layer cache design is sound: permanent parsed content in the object store, id/name/
description in Redis in front of it, and the SDK stays cache backend agnostic via the
loader/writer callbacks rather than importing anything backend specific. - You caught the duplicate test set bug and the metrics filter pill bug yourself, and both commit
messages explain the mechanism precisely. That is what made the rest of this reviewable.
I looked at the backend and SDK Python only, not the frontend. Everything below is inline on the
specific lines.
Correctness, worth fixing before merge
app/main.py: the OWASP pre-warm has no test suite guard, unlike the Garak one right above it.tasks/test_set.py: OWASP generation is not metered. Every other generation path accrues.tasks/test_set.py: the retry after commit duplication is still reachable. The fix inf7fe037
addressed the trigger, not the class of bug.app/services/owasp.py:except OSErroris too narrow for the fsspec cloud backends.sdk/.../owasp_synthesizer.py: the multi-turn path loses retry and batch size reduction.sdk/.../owasp_extractor.py: a cached empty list skips the no-sections guard.
DRY, worth doing while the code is fresh
sdk/.../owasp_synthesizer.py: the multi-turn schema and repacker duplicate
synthesizers/multi_turn/base.pyfield for field. Fixing this also fixes the retry loss above.app/services/organization.py: third hand rolled tag get-or-create in the codebase.
crud.assign_tagalready does exactly this.alembic/utils/metric_sync.py: the behavior sync mirrors the metric sync closely enough to share
the org listing.app/services/owasp.py:OWASP_FRAMEWORKSholds the same string under two keys.
Small stuff, all optional
Dead list_categories, dead behavior_definitions parameter, a duplicated done callback, redundant
pdfminer child loggers, cache shape sniffing that a versioned key would remove,
get_owasp_content_path not using self, and initial_data.json getting read four times per
migration run. Take or leave any of these.
One correction for the PR description
It says cd sdk && make test covers OWASPSynthesizer, but tests/sdk/ has no OWASP tests at all.
The new cache hooks in fetch_owasp_sections, _distribute, and the flat to nested repack are pure
functions, so they are cheap to cover if you want them before this lands.
Nice piece of work overall. Happy to pair on any of these, especially the SDK consolidation.
|
|
||
| setup_mcp_server(app) | ||
|
|
||
| # Pre-warm OWASP report section cache in background (non-blocking) |
There was a problem hiding this comment.
Correctness: this will slow down, and possibly hang, the backend test suite.
This pre-warm fires two PDF downloads plus two pdfminer parses through asyncio.to_thread on every
app lifespan start. The Garak pre-warm 30 lines above is skipped under tests via
RHESIS_SKIP_GARAK_WARM_CACHE, set in tests/backend/conftest.py:94, and its comment spells out
the exact failure mode: every test builds a fresh TestClient, so a fresh lifespan, the worker
threads are not cancellable, and the pile up snowballs across pytest-xdist workers.
The OWASP warm has the same shape and no guard, so it inherits the same problem plus real network
I/O per test.
Suggestion: put both behind one flag, for example RHESIS_SKIP_WARM_CACHES, keeping
RHESIS_SKIP_GARAK_WARM_CACHE as an alias so nothing that sets it today breaks. Adding
RHESIS_SKIP_OWASP_WARM_CACHE to the conftest env block works too, it is just a second knob to
remember.
|
|
||
| owasp_cache_task = asyncio.create_task(warm_owasp_cache()) | ||
|
|
||
| def _log_owasp_task_exception(t: asyncio.Task) -> None: |
There was a problem hiding this comment.
Small one: this is character for character _log_task_exception at line 451, apart
from the log string. Worth one helper that takes the label:
def _log_warm_task_exception(label: str):
def _cb(t: asyncio.Task) -> None:
try:
t.result()
except asyncio.CancelledError:
pass # Expected during shutdown
except Exception as e:
logger.error(f"{label} cache pre-warming task failed: {e}", exc_info=True)
return _cb| self.update_state(state="PROGRESS", meta={"status": "Saving to database"}) | ||
|
|
||
| test_set_name = name or f"{behavior}: {purpose[:60]}" | ||
| db_test_set = _save_test_set_to_database( |
There was a problem hiding this comment.
Correctness: the duplicate test set bug is still reachable.
f7fe037 fixed the trigger, the missing tests_generated kwarg, but not the class of bug. This
save commits a new row, and everything after it (_build_task_result, the logging calls) still runs
inside the same try, under autoretry_for = (Exception,) and max_retries = 3
(tasks/base.py:69,52). So any other post save failure still retries the whole task body and
still leaves up to 4 test sets behind. The next _build_task_result signature change, or a
DetachedInstanceError of the kind its own docstring describes, reproduces the original incident.
The sibling task is not exposed to this because the router pre-creates the row and passes
test_set_id, so retries reattach to the same row via _attach_tests_to_existing_test_set instead
of creating a new one.
Two ways out, in the order I would pick them:
- Adopt the pre-created row path. Router creates the
TestSet, task attaches. It matches the
sibling, and it gives the frontend an id to redirect to immediately, which the OWASP drawer would
probably want anyway. - Keep the current shape but move the post save work out of the retryable region, or raise
something non-retryable for it.dont_autoretry_foralready exists for exactly this
(tasks/base.py:75).
| tests_generated=len(test_set.tests), | ||
| ) | ||
|
|
||
| return result |
There was a problem hiding this comment.
Correctness: OWASP generation is not metered.
Nothing in this task dispatches a quota accrual, so OWASP generated tests do not count against the
org's test generation quota. Every other generation path does:
tasks/test_set.py:588, the sibling task:
dispatch_accrual(org_id, QuotaResource.TEST_GENERATION, len(test_set.tests))tasks/garak.py:87andtasks/garak.py:133, for probe import and sync
dispatch_accrual is already imported at the top of this module, so it is a one line addition after
the save, matching line 588.
| storage.put_object_bytes( | ||
| payload, storage.get_owasp_content_path(framework), "application/json" | ||
| ) | ||
| except OSError as e: |
There was a problem hiding this comment.
Correctness: this except is too narrow to keep the promise in the docstring.
The docstring says a storage write failure will not discard a successful parse, but OSError only
covers the local filesystem case it mentions. StorageService.put_object_bytes goes through fsspec,
and the cloud backends raise types that are not OSError subclasses (botocore ClientError,
gcsfs HttpError). Those propagate out of fetch_owasp_sections, out of
list_category_summaries, and land as a 502 for the user after the expensive parse already
succeeded, which is the opposite of the intent here.
| except OSError as e: | |
| except Exception as e: |
| def sync_behaviors_to_organizations( | ||
| session: Session, | ||
| behavior_names: List[str] | None = None, | ||
| behavior_definitions: List[Dict[str, Any]] | None = None, |
There was a problem hiding this comment.
Dead parameter: no caller passes behavior_definitions. The only caller is
migration 38ed899b9f41, which passes behavior_names.
It costs a branch at line 403, two docstring paragraphs, and an "only applies when
behavior_definitions is not provided" caveat on the parameter above it. Worth dropping until
something actually needs it.
|
|
||
| # List orgs with raw SQL so migrations run before newer Organization columns | ||
| # (e.g. sso_config, slug) exist do not fail with UndefinedColumn on ORM loads. | ||
| org_rows = session.execute( |
There was a problem hiding this comment.
DRY: this org listing is now written three times. Here, in
sync_metrics_to_organizations, and again in migration b857edcac3c0 in this PR. All three repeat
the same details: the raw SQL, deleted_at IS NULL, the owner_id or user_id fallback, the string
conversion, and the skip when neither is set.
A small generator would carry all of it, including the comment about migrations running before newer
organization columns exist, which is a genuinely useful thing to explain once instead of three
times:
def _iter_organizations(session: Session, verbose: bool = True):
"""Yield (org_id, user_id) for every live org, via raw SQL so migrations that
predate newer Organization columns do not fail on ORM loads."""More broadly, sync_behaviors_to_organizations and sync_metrics_to_organizations also share the
four key stats dict, the verbose print scaffolding, and the try/except print reraise wrapper. I
would not push for a full merge of the two, the entity specific parts differ enough, but the org
listing is worth pulling out.
| ) | ||
|
|
||
|
|
||
| def _load_initial_data() -> Dict[str, Any]: |
There was a problem hiding this comment.
Small one: worth an @functools.lru_cache here. Migration 38ed899b9f41 calls
_owasp_behavior_names() and _owasp_metric_names(), each of which calls this, so
initial_data.json gets read and parsed four times per upgrade, and four more per downgrade. The
file is not small.
| # uvicorn --log-level debug that can fill hundreds of MB and stall the | ||
| # categories endpoint / cache warm-up for minutes. | ||
| for name in ( | ||
| "pdfminer", |
There was a problem hiding this comment.
Small one: the six child entries are redundant. A logger with no level set
inherits its parent's effective level, and none of these set one, so
logging.getLogger("pdfminer").setLevel(logging.WARNING) on its own silences all of them.
Good catch on the underlying problem though. Hundreds of MB of DEBUG output from a PDF parse is a
nasty thing to debug from the symptom.
| prefix = self.get_attachment_prefix(organization_id, entity_type, entity_id, file_id) | ||
| return f"{prefix}/thumb-{size}.webp" | ||
|
|
||
| def get_owasp_content_path(self, framework: str) -> str: |
There was a problem hiding this comment.
Small one: this does not touch self, so it can be a @staticmethod, or the
f-string can just live at the two call sites in services/owasp.py.
Worth noting the docstring is doing real work here, since every other path helper on this class is
org scoped and this one deliberately is not. That is correct, the source PDFs are public, but it is
the kind of thing that gets "fixed" by someone who does not read the docstring. A brief note that
the content is not tenant specific would make the intent harder to miss.
Review feedback from Harry Cruz on PR #2202: - The OWASP cache pre-warm had no test-suite skip guard, unlike the Garak one right above it -- every backend test run paid for two PDF downloads plus two pdfminer parses at app startup. It now shares the same RHESIS_SKIP_GARAK_WARM_CACHE guard. - Deduplicated two character-for-character identical done-callbacks (_log_task_exception / the OWASP warm task's copy) into one factory parameterized by label. - generate_and_save_owasp_test_set never dispatched a quota accrual, so OWASP-generated tests didn't count against the org's generation quota, unlike every sibling generation path. - The duplicate-test-set bug class was still open: an earlier fix (f7fe037) closed the one deterministic trigger (a missing kwarg), but _build_task_result, logging, and the return were still inside the same retryable try/except -- any other post-save failure would still cause Celery to retry the whole task and create another duplicate test set. The post-save section now runs in its own try/except that returns a minimal result instead of re-raising, so a failure there can no longer trigger a duplicate save.
Review feedback from Harry Cruz and peqy on PR #2202: - `except OSError` around the content-cache write was too narrow to keep its own docstring's promise -- fsspec-backed cloud storage backends raise their own exception types on write failure, not OSError. Broadened the catch. - OWASP_FRAMEWORKS stored the same string under two keys ("behavior" and "label"); the router read one and the task read the other, which was accidental rather than meaningful and could silently drift. Collapsed to a single key. - Removed the dead `list_categories` method (zero callers repo-wide, and a trap for the next reader since it always returned an empty content string). - The Redis/object-store OWASP content cache sniffed the cached payload's shape to detect legacy entries, and separately (per peqy) could go permanently stale since it was keyed only by framework name with no versioning -- if the upstream PDF or its URL changed, the cache would keep serving old sections forever. Both are fixed by the same change: the cache key is now versioned and includes a hash of the report URL, so legacy entries just age out and a changed report URL naturally busts the cache. - `GET /owasp/categories` was `async def` but called into the cache layer synchronously, which can download/parse PDFs and hit storage/Redis on a cold cache, blocking the event loop. Wrapped in `asyncio.to_thread`, mirroring the existing startup pre-warm. - `get_owasp_content_path` didn't use `self`; made it a staticmethod.
Review feedback from Harry Cruz on PR #2202: OWASPSynthesizer's multi-turn path redefined multi_turn/base.py's flat test schemas and repack logic field-for-field, and reimplemented its own batch loop that dropped an entire section's tests on one bad LLM response -- single-turn generation already gets retry with batch-size reduction via _generate_with_retry, multi-turn never did. Fixed both with one change: multi-turn now reuses multi_turn/base.py's FlatTests schema directly, and overrides _generate_batch (the hook _generate_with_retry already calls) instead of a parallel hand-rolled loop -- so multi-turn generation gets the real retry/batch-reduction machinery for free. Also extracted the "... (Multi-Turn)" naming/ stamping tail into a shared stamp_multi_turn helper in synthesizers/utils.py. No public API change: OWASPSynthesizer's constructor and generate() signature/return type are unchanged; single-turn behavior is unaffected (delegates straight to the base implementation). Adds tests/sdk/synthesizers/test_owasp_synthesizer.py -- there was no OWASP synthesizer test coverage before this, despite the PR description previously claiming otherwise (corrected separately). Includes a regression test demonstrating the retry fix: a flaky first LLM response that previously dropped all requested tests now recovers them via batch-size reduction.
Review feedback from Harry Cruz on PR #2202: if cache_loader returned an empty list, the `is None` guard for "no cached data" was false, so the empty list was treated as valid content -- the real download, parse, and the "no sections found" check were all silently skipped. cache_writer only ever persists a non-empty list, so a stored empty list can only be stale or corrupt; changed the check from `is not None` to truthy so it now falls through to a real re-fetch, same as an actual cache miss. Adds tests/sdk/services/test_owasp_extractor.py -- no coverage existed for this module before. Includes a regression test for this exact bug (verified it fails against the pre-fix code) plus coverage for the pure parsing helpers the PR description previously overclaimed as tested.
Review feedback from Harry Cruz on PR #2202: the get-or-create-Tag / get-or-create-TaggedItem helpers added for onboarding-time OWASP tagging were a third hand-rolled copy of an idiom that already exists as crud.assign_tag, alongside garak/importer.py's _tag_garak_behaviors and migration b857edcac3c0. Removed _get_or_create_owasp_tag and _tag_as_owasp entirely; both call sites now call crud.assign_tag directly. Behavior is unchanged (same test coverage passes without modification): exactly one org-scoped "OWASP" Tag, one TaggedItem per OWASP-prefixed behavior/ metric, idempotent across repeated onboarding runs.
Review feedback from Harry Cruz on PR #2202: - The same "list organizations with an owner/user fallback" raw SQL query was written three times: sync_behaviors_to_organizations, sync_metrics_to_organizations, and migration b857edcac3c0. Extracted a single _list_organizations_with_owner helper in metric_sync.py; all three now call it. revision/down_revision on the migration are unchanged. - Removed the dead behavior_definitions parameter on sync_behaviors_to_organizations -- its only caller (migration 38ed899b9f41) always passes behavior_names, never this. - Added @functools.lru_cache to the initial_data.json loader -- migration 38ed899b9f41 called it four times per upgrade (and four more per downgrade) for a file that isn't small.
peqy flagged that the OWASP option on the Test Sets page's Garak/OWASP
FAB was gated on the generic Capability.TestSet.GENERATE, while the
backend derives a route-specific owasp:create/owasp:read capability
(mirroring garak:create/garak:read) -- the same pattern Garak's own
FAB entry already uses one line above it.
Built-in roles are unaffected today (TestSet.GENERATE and owasp:create
are always co-granted for all five built-in roles), but a custom role
built via the role editor could never have gotten owasp:create wired
to whatever it had for TestSet.GENERATE -- OWASP wasn't in the
frontend Capability constants at all, so it was invisible to the
custom-role UI. Added Capability.Owasp.{READ,CREATE} (mirroring
Capability.Garak) and switched the FAB gate to it.
Review feedback from Harry Cruz on PR #2202: the six child pdfminer loggers (psparser, pdfinterp, ...) were each set to WARNING individually, but none of them set their own level, so they already inherit the parent's effective level -- silencing "pdfminer" alone does the same job.
No behavior change. Trimmed comments that leaned on diff/history framing (referencing what a value "used to be" or "previously") or restated something the surrounding code already makes clear, and removed one comment that duplicated its own method's docstring one screen down. Kept the ones explaining choices that actually cut against the grain -- e.g. why a broad `except Exception` is correct here, or why a retry mechanism needed isolating from a specific Celery setting -- since those aren't things a reader would otherwise guess.
|
Follow-up on my earlier note (rev_01KZV25T3D1GTVPXX02ZZ7GWVD):
From my side this is good to ship. |
Follow-up to a177338: OWASPSynthesizer now uses the shared stamp_multi_turn helper, but MultiTurnSynthesizer.generate() itself still had its own inline copy of the same test_set_type + "(Multi- Turn)" naming logic. Both now call the one helper.
Follow-up to f37b4a0: Capability.Owasp now exists and the Test Sets page gates on it, but the custom-role editor's "Test Resources" area had no entry for it at all -- a custom role could never have OWASP wired in through the role editor UI, unlike Garak sitting right next to it. Added Owasp.READ/CREATE at the same View/Edit/Manage levels Garak already uses.
crud.assign_tag() reaches through the crud package attribute at call time, which only resolves if some other import happened to populate it first (see apps/backend/AGENTS.md). This "worked by accident" in narrow local test runs but failed in CI (xdist workers, fresh uv sync) with AttributeError: module 'rhesis.backend.app.crud' has no attribute 'assign_tag', breaking onboarding and cascading into all e2e shards.
|
Reviewed latest commit |
# Conflicts: # apps/backend/src/rhesis/backend/app/services/organization.py
harry-rhesis
left a comment
harry-rhesis
left a comment
There was a problem hiding this comment.
Thanks Alexey, every item from the review has been addressed. The three post-review commits are clean and well-scoped:
- The correctness fixes (pre-warm guard, metering, post-save isolation, broadened cache errors, multi-turn retry, cached empty list) all land with matching test coverage.
- The DRY consolidation (shared FlatTests import, tag_crud.assign_tag, _list_organizations_with_owner, collapsed OWASP_FRAMEWORKS keys) tidied up exactly the spots that needed it.
- The small stuff (dead code removal, lru_cache, versioned cache key, staticmethod, pdfminer parent logger) is all taken care of too.
Nice work on the SDK test suite as well. The flaky-LLM-call retry test and the post-save isolation test are both good regression guards.
This is good to merge.
3 participants
Purpose
Turns the OWASP synthesizer (merged earlier in #2054) into an end-to-end
feature: users can generate an OWASP Top 10 test set (LLM or Agentic
framework, single- or multi-turn) directly from the Test Sets page, and
the resulting behaviors/metrics are tagged and filterable as "OWASP".
Scope
GET /owasp/categoriesandPOST /owasp/generaterouter (
apps/backend/.../routers/owasp.py), backed by anowaspservice that downloads/parses the official OWASP PDF and caches the
parsed report (Redis).
generate_and_save_owasp_test_setCelery task(
tasks/test_set.py) drives the SDK'sOWASPSynthesizerto produce aRhesis test set from the selected categories, including a new
multi-turn generation path.
initial_data.jsongains the "OWASP LLMTop 10" / "OWASP Agentic Top 10" behaviors and their 20 associated
metrics. Three Alembic migrations backfill these into existing
organizations and tag them (new orgs get them via onboarding as usual):
38ed899b9f41— sync OWASP behaviors + metrics to existing orgs30877432d102— addowasp:create/owasp:readpermission rowsb857edcac3c0— tag existing OWASP metrics/behaviors with anOWASPtagOwaspGenerateDraweron the Test Sets page (framework,purpose, categories, single-/multi-turn toggle), an
OwaspIcon, anAPI client, and an "OWASP" filter on the Metrics directory tab.
OWASPSynthesizergains multi-turn generation(
owasp_synthesizer_multi_turn.jinja) and caches parsed OWASP reports.How to run this locally
./rh dev init # first time only ./rh dev upapps/backend):organizations (idempotent — safe to re-run). New organizations get the
same behaviors/metrics automatically via onboarding, no migration needed.
OWASP. Pick a framework (LLM / Agentic Top 10), describe the
system under test, optionally narrow to specific categories, and
toggle single- vs multi-turn.
OWASP should show the seeded metrics.
Testing
cd apps/backend && make test(backend unit tests, spins upPostgres/Redis via docker compose automatically)
cd sdk && make test(SDK unit tests — note:tests/sdk/has nodedicated
OWASPSynthesizercoverage yet, corrected from an earlierversion of this description that claimed otherwise)
and both single-/multi-turn modes.
Notes for reviewers
main);this PR only contains the 14 commits added since then.
2a2b2d0f0is a small fixup for a two-head Alembic chain introducedwhile rebasing onto
main— no behavioral change.