Hi @rdheekonda — Counterfit was the first AI-red-team tool with a Metasploit-shaped UX (set target / set attack / scan). It's still the cleanest CLI abstraction for that workflow, and there's a real user base that prefers the CLI shape over the PyRIT Python-API model.
The repo has been in maintenance mode since the last release. This proposal is deliberately out-of-tree — a companion plugin package, not a core PR. Zero changes to Azure/counterfit. The ask is small:
- A README pointer to the companion package under a "Plugins / extensions" section, so existing Counterfit users discover it.
That's the entire ask. No code review burden, no CI surface, no test maintenance.
What the companion package does
counterfit-detection-atr (PyPI, MIT, separate repo) extends the scan flow with a content-layer detection lane:
counterfit> set target HuggingFaceTextClassifier
counterfit> set attack TextAttackTextFooler
counterfit> use plugin detection-atr
counterfit> scan
[attack] success rate: 73%
[detection] flagged: 58/100, missed: 15/100
[bypass] 27/100 (attack succeeded AND evaded content-layer detection)
bypass rate is the single most useful number for a defender comparing two attack methods at the same ASR — it tells them which one their existing detection-layer covers. Counterfit becomes the CLI shape for that comparison.
About the detector and the package
Agent-Threat-Rule/agent-threat-rules is the open detection ruleset that backs the plugin. MIT, 344 rules at v2.1.3, no CLA. The plugin shells out to pyatr (PyPI, MIT, ~3 MB) — no Microsoft-internal dependencies, no licence entanglement with Counterfit.
A few proof points relevant to a Microsoft Azure Security audience:
- ATR ships into Microsoft Agent Governance Toolkit via the weekly auto-sync workflow that Imran Siddique merged in PR #1277. Same Microsoft AI-security org chart.
- The PyRIT integration (PR #1715) is in active review with Roman Lutz. Counterfit + PyRIT + AGT all sharing the same MIT detection backend is the natural outcome — if AGT and PyRIT are landing it, Counterfit users get the same coverage by adopting the companion plugin.
- 6-check pre-merge quality gate, 0 FP on 1,941-sample combined benign + research-mention corpus, MIT in perpetuity (GOVERNANCE.md commitment).
Honest about Counterfit's state
I noticed the last release was in 2024 and the issue tracker has questions about long-term maintenance. The companion-package approach is exactly the right shape for this state — it gives Counterfit users a path to new functionality without requiring the project to take on new maintenance commitments. If the project picks up again later, the plugin moves into core. If not, the plugin still works.
What I'm asking
Just the README pointer when v0.1 ships. If nobody's actively triaging the repo, that's fine — the package will exist on PyPI either way.
Hi @rdheekonda — Counterfit was the first AI-red-team tool with a Metasploit-shaped UX (
set target / set attack / scan). It's still the cleanest CLI abstraction for that workflow, and there's a real user base that prefers the CLI shape over the PyRIT Python-API model.The repo has been in maintenance mode since the last release. This proposal is deliberately out-of-tree — a companion plugin package, not a core PR. Zero changes to
Azure/counterfit. The ask is small:That's the entire ask. No code review burden, no CI surface, no test maintenance.
What the companion package does
counterfit-detection-atr(PyPI, MIT, separate repo) extends the scan flow with a content-layer detection lane:bypass rateis the single most useful number for a defender comparing two attack methods at the same ASR — it tells them which one their existing detection-layer covers. Counterfit becomes the CLI shape for that comparison.About the detector and the package
Agent-Threat-Rule/agent-threat-rules is the open detection ruleset that backs the plugin. MIT, 344 rules at v2.1.3, no CLA. The plugin shells out to
pyatr(PyPI, MIT, ~3 MB) — no Microsoft-internal dependencies, no licence entanglement with Counterfit.A few proof points relevant to a Microsoft Azure Security audience:
Honest about Counterfit's state
I noticed the last release was in 2024 and the issue tracker has questions about long-term maintenance. The companion-package approach is exactly the right shape for this state — it gives Counterfit users a path to new functionality without requiring the project to take on new maintenance commitments. If the project picks up again later, the plugin moves into core. If not, the plugin still works.
What I'm asking
Just the README pointer when v0.1 ships. If nobody's actively triaging the repo, that's fine — the package will exist on PyPI either way.