feat: research-grade prompt-injection detection benchmark with baselines + ablation#39
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c7cfaafe09
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from datasets import load_dataset | ||
|
|
||
| resolved = _resolve_hf_revision(DEEPSET_REPO, DEEPSET_REVISION) | ||
| ds = load_dataset(DEEPSET_REPO, revision=DEEPSET_REVISION) |
There was a problem hiding this comment.
This resolves main to a commit SHA and records that SHA, but the actual download still uses DEEPSET_REVISION (main). If the upstream dataset advances between resolution and download, or on later reruns, the benchmark consumes moving data while presenting the run as pinned; the same pattern appears in the Dolly loader. Use the resolved SHA for load_dataset(..., revision=resolved) so the recorded revision actually matches the rows scored.
Useful? React with 👍 / 👎.
| resolved = _github_file_sha(INJECAGENT_REPO, INJECAGENT_FILES[0], INJECAGENT_REF) | ||
| cases: list[str] = [] | ||
| for path in INJECAGENT_FILES: | ||
| raw_url = f"https://raw.githubusercontent.com/{INJECAGENT_REPO}/{INJECAGENT_REF}/{path}" |
There was a problem hiding this comment.
The metadata records only the latest commit for the first InjecAgent file, but both JSON files are fetched from the moving main ref. If main advances or data/test_cases_ds_base.json changes independently, the benchmark can sample rows that are not represented by the recorded revision, breaking reproducibility of the published results. Resolve an immutable repository commit (or record/fetch each file SHA) and use that ref in the raw URL.
Useful? React with 👍 / 👎.
2 participants
No description provided.