Enterprise & Defense-Grade Security Testing for Machine Learning Systems
Status: Active Development — Version 0.1 (Draft)
🌐 We welcome international contributors! Translations in progress.
📖 Live documentation: mlastg.vercel.app — including the interactive ATLAS Coverage Map.
- Overview
- Why MLASTG?
- Architecture
- Control Categories
- Assurance Levels
- Threat Coverage — MITRE ATLAS
- Automated Testing — the
mlastgCLI - Continuous Assurance (CI/CD)
- Reporting & Compliance
- Quick Start
- Framework Alignment
- Project Structure
- Contributing
- License
The MLSec Application Security Testing Guide (MLASTG) is a comprehensive, open-source framework for security testing machine learning (ML) systems across the full threat landscape — from traditional ML classifiers to deep neural networks and large language models (LLMs).
Inspired by the OWASP Mobile Application Security Testing Guide (MASTG) and aligned with MITRE ATLAS, NIST AI RMF, and OWASP AI Exchange, MLASTG provides four tightly-coupled layers:
- A Verification Standard (MLASVS) — What to verify, organized by control category with L1 (Standard) and L2 (Defense-in-Depth) assurance levels.
- A Testing Guide (MLASTG) — How to test, with detailed step-by-step test cases mapped to controls.
- A Weakness Enumeration (MLASWE) — What can go wrong, a common taxonomy of ML/LLM weaknesses for classifying findings.
- Executable Test Scripts + CLI — Automated validation, Python harnesses orchestrated by the
mlastgcommand-line scanner.
Designed for enterprise and defense-grade environments: every control is testable, mapped to recognized frameworks, and assignable to an assurance tier so security programs can demonstrate measurable, audit-ready coverage of the ML attack surface.
| Problem | MLASTG Solution |
|---|---|
| ML security lacks a standardized, testable verification framework | MLASVS provides clear, verifiable controls mapped to MITRE ATLAS tactics |
| Existing guidance is fragmented across OWASP, NIST, and vendor documents | A unified reference integrating all major frameworks with cross-references |
| Testing procedures for adversarial ML are poorly documented | Detailed step-by-step test cases with companion Python scripts |
| Enterprise/defense environments require defense-in-depth | Two-tier verification (L1 Standard / L2 Defense-in-Depth) |
| No SBOM/SCA standard exists for ML supply chains | ML-SBOM requirements and supply-chain verification controls |
| Coverage is hard to prove to auditors and leadership | A MITRE ATLAS coverage map and exportable JSON/Markdown assessment reports |
MLASVS (The Standard) ─── What to verify
│
├── Maps to ───► MITRE ATLAS Tactics & Techniques
├── Aligns to ──► NIST AI RMF, OWASP AI Exchange
└── Referenced by ──► MLASWE Weakness IDs
│
▼
MLASTG (The Testing Guide) ─── How to test
│
├── Test Cases ──► MLASTG-TEST-XXXX (step-by-step procedures)
├── Techniques ──► MLASTG-TECH-XXXX (tools & methods)
└── Companion Scripts ──► tests/*.py (executable harnesses)
│
▼
mlastg CLI (Automation) ─── Run, score, and report
│
└── mlastg scan ──► orchestrates harnesses ──► JSON + Markdown reports
│
▼
MLASWE (Weakness Enumeration) ─── What can go wrong
│
└── MLASWE-XXXX identifiers for each weakness class
| Category | ID | Coverage | Controls L1 | Controls L2 |
|---|---|---|---|---|
| Data Security & Privacy | MLASVS-DATA | Provenance, sanitization, differential privacy, access controls | 18 | 12 |
| Model Security | MLASVS-MODEL | Adversarial robustness, extraction/inversion prevention, backdoor detection | 15 | 15 |
| LLM-Specific Security | MLASVS-LLM | Prompt injection, output handling, agency, context isolation | 14 | 10 |
| Supply Chain Security | MLASVS-SUPPLY | ML-SBOM, base model vetting, dependency scanning | 12 | 10 |
| Pipeline & MLOps | MLASVS-PIPELINE | CI/CD, feature stores, model registries, artifact integrity | 10 | 10 |
| Runtime & Infrastructure | MLASVS-INFRA | Serving security, API security, monitoring, incident response | 12 | 10 |
| Governance & Compliance | MLASVS-GOV | Risk governance, bias/fairness, audit logging, regulatory | 10 | 10 |
Total Controls: 91 L1 + 77 L2 = 168 verifiable controls
Baseline controls for all ML applications in production:
- Fundamental data protection and access controls
- Basic adversarial robustness validation
- Standard supply chain hygiene
- Essential monitoring and logging
Enhanced controls for high-risk, enterprise/defense, and regulated environments. Adds:
- Rigorous adversarial robustness certification
- Differential privacy guarantees
- Comprehensive red teaming
- Full ML-SBOM with provenance verification
- Continuous runtime monitoring with automated response
Every MLASVS control is mapped to the MITRE ATLAS adversarial-threat taxonomy. The mapping is published two ways:
- Interactive ATLAS Coverage Map — a periodic-table view of ML/LLM attack techniques, arranged by the MLASVS control family that covers them, with per-technique control mappings.
- Importable Navigator layer — load
2-atlas-navigator-layer.jsoninto the MITRE ATLAS Navigator (Open Existing Layer → Upload from local) to view coverage as a heat map.
| Coverage | Techniques | Meaning |
|---|---|---|
| 🟢 Full | 20 | MLASVS controls and MLASTG test cases exist |
| 🟡 Partial | 19 | Some controls mapped; gaps remain |
| Total mapped | 39 | All 7 control families covered |
Technique IDs and names are reconciled against the official MITRE ATLAS data.
See the Coverage Matrix and Gap Analysis for the narrative breakdown.
All 168 controls are also published as a machine-readable register —
controls.json — with an interactive
Control Register that maps
each control to its assurance level, MITRE ATLAS reference, and companion test
case. It is regenerated from the documentation by
tools/generate_controls_register.py,
so the catalog never drifts from the standard.
mlastg is a Click-based command-line scanner (with Rich output) that orchestrates the category test harnesses and produces assessment reports.
# Core CLI
pip install -e .
# Optional: testing harness dependencies (ART, scikit-learn, torch, giskard, …)
pip install -e ".[tests]"# Scan a live LLM endpoint for LLM-category weaknesses
mlastg scan --target https://api.example.com/v1/chat --category llm
# Scan a local model artifact for model-category weaknesses
mlastg scan --target ./models/classifier.pkl --category model
# Dry-run everything with local stubs (no external calls)
mlastg scan --target demo --category all --demo| Option | Values | Default | Purpose |
|---|---|---|---|
--target |
URL · API endpoint · local path | (required) | System under test |
--category |
model · llm · data · supply · infra · pipeline · gov · all |
all |
Test category to execute |
--demo |
flag | off | Use local stubs instead of real calls |
--output |
path | mlastg_report.json |
Raw JSON results |
--format |
json · markdown · both |
both |
Report format(s) |
mlastg report --input mlastg_report.json --output mlastg_report.mdSafe by default:
--demoruns against local stubs so you can validate the pipeline without touching production systems. Only scan targets you are authorized to test.
MLASTG ships GitHub Actions workflows so assurance runs on every change:
| Workflow | Purpose |
|---|---|
mlastg-scan.yml |
Runs the mlastg scanner (demo mode) as a CI gate |
test-scripts.yml |
Executes the Python test harnesses |
mkdocs-publish.yml |
Builds and publishes the documentation site |
deploy-vercel.yml |
Deploys the docs to Vercel |
deploy-navigator.yml |
Validates and publishes the ATLAS Navigator layer |
The CLI emits both machine- and human-readable artifacts:
mlastg_report.json— structured results for pipelines, dashboards, and ticketing.mlastg_report.md— a Markdown compliance report suitable for audit evidence.
Findings reference MLASWE weakness IDs and the MLASVS controls they violate, so results map cleanly back to MITRE ATLAS, NIST AI RMF, the OWASP AI/LLM/ML Top 10s, and the EU AI Act for governance reporting.
- Review the MLASVS to identify applicable controls.
- Use the MLASTG Testing Methodology to plan your assessment.
- Execute test cases mapped to your target controls (manually or via
mlastg scan). - Reference MLASWE for weakness classification in findings.
- Export JSON/Markdown reports for evidence.
- Adopt MLASVS as your internal ML security standard.
- Map existing controls to MLASVS categories.
- Conduct gap analysis using the MLASTG Checklist and the ATLAS Coverage Map.
- Implement missing controls with L1 as a minimum and L2 for high-risk systems.
- Wire
mlastg scaninto CI to keep coverage from regressing.
| Framework | MLASTG Alignment |
|---|---|
| MITRE ATLAS | Each MLASVS control maps to relevant ATLAS tactics/techniques |
| NIST AI RMF 1.0 | Controls support all four RMF functions (Govern, Map, Measure, Manage) |
| OWASP AI Exchange | Cross-referenced to OWASP AI threat/control matrices |
| OWASP LLM Top 10 | Full coverage of all 10 LLM risks as MLASVS-LLM controls |
| OWASP ML Top 10 | Coverage of all 10 ML security vulnerabilities |
| NSA/CISA AI Security Guidance | Controls aligned with secure deployment guidance |
| EU AI Act | MLASVS-GOV controls mapped to regulatory requirements |
MLASTG/
├── README.md ← This file
├── mkdocs.yml ← Documentation site config
├── pyproject.toml ← mlastg CLI package definition
├── mlastg_cli/ ← Automated testing CLI
│ ├── main.py ← Click entrypoint (scan, report)
│ ├── orchestrator.py ← Runs the category test suites
│ └── reporter.py ← JSON / Markdown report generation
├── docs/
│ ├── index.md ← Home / landing page
│ ├── MLASVS/ ← Verification Standard (V1–V7)
│ ├── MLASTG/ ← Testing Guide (per-category tests)
│ ├── MLASWE/ ← Weakness Enumeration
│ ├── ATLAS-Mapping/ ← Coverage map, matrix, gap analysis, Navigator layer
│ ├── stylesheets/ javascripts/← Custom theme + interactive UI
│ └── assets/ ← Images, diagrams
├── tests/ ← Python test harnesses (data, model, llm, supply, pipeline, infra, gov)
├── demos/ ← Example vulnerable models & apps
└── .github/workflows/ ← CI: scan, tests, docs, deploy, navigator
MLASTG is at v0.1 and honest about what that means. The framework skeleton, control catalog, test procedures, automation CLI, and ATLAS coverage map are implemented and usable today; the items below are tracked for production hardening. See CHANGELOG.md for release history.
| Area | Status | Notes |
|---|---|---|
| MLASVS standard (168 controls, 7 categories) | ✅ Implemented | Documented with L1/L2 levels |
| MLASTG test procedures | ✅ Implemented | Step-by-step, pass/fail criteria |
mlastg CLI + Python harnesses |
✅ Implemented | Real ART-based harnesses; --demo stubs for safe CI |
| MITRE ATLAS coverage map | ✅ Reconciled | 26 techniques; IDs/names verified against official ATLAS data across the coverage map, Navigator layer, control docs, and register |
| Expanded ATLAS coverage | ✅ Implemented | 39 techniques across all 7 families (INFRA & GOV now covered); further breadth ongoing |
| Machine-readable control register (JSON) | ✅ Implemented | controls.json + interactive register; generated from the docs |
| End-to-end reference run + green CI | ✅ Implemented | test-scripts.yml runs all 10 harness demos + pytest; mlastg-scan.yml runs the full demo scan and gates on compliance score — both green on every push |
| Control → ATLAS → test → weakness traceability | ✅ Implemented | The register now maps each control to the MLASWE weaknesses it mitigates |
| Conformance self-assessment + scoring | ✅ Implemented | Interactive self-assessment with live L1/L2 scoring and JSON evidence export |
| Worked ML-SBOM example | ✅ Implemented | demos/ml-sbom/ — native + CycloneDX 1.6, CI-validated |
| Per-control framework citations | 🟡 Partial | NIST AI RMF / OWASP references surfaced in the register for controls whose pages cite them (page-level alignment); inheritance to table-format controls pending |
| Versioned releases (SemVer + tags) | ✅ Implemented | v0.2.0 tagged; CHANGELOG + GOVERNANCE release process |
| Independent / external review | ⬜ Planned | Peer review and recognition (e.g., OWASP/MITRE) |
Adoption guidance: today MLASTG is best used as a reference standard, checklist, and automation accelerator to structure an ML security program. Treat the ATLAS technique IDs as v0.1 mappings pending reconciliation, and validate controls against your own threat model before certification.
This project is in active development. Contributions are welcome across:
- New test cases and step-by-step procedures
- Python test-script implementations (see
tests/) and CLI orchestrator coverage - LLM-specific testing methodologies and datasets
- Case studies and real-world attack demonstrations
- Translations and internationalization (see
docs/zh/for Chinese) - Additional MLASVS control categories and ATLAS mappings
See CONTRIBUTING.md and the Code of Conduct for guidelines, GOVERNANCE.md for maintainership, versioning, and the control-ID stability policy, and SECURITY.md for responsible disclosure.
MLASTG is dual-licensed so it fits cleanly into both documentation and engineering pipelines:
- Documentation, standard, and guidance content (
docs/) — Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). - Code (the
mlastgCLI, test harnesses, and tooling) — MIT License.
- OWASP MASTG — The inspiration and structural model for this project
- MITRE ATLAS — Adversarial threat taxonomy foundation
- NIST AI RMF — Risk management framework alignment
- OWASP AI Exchange — Cross-referenced threat and control matrices
- IBM ART — Adversarial robustness testing tools
- All contributors to the AI/ML security community