Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,7 @@ if needed, walk you through setting the analysis up on a High Performance
Computer (HPC) you have access to.

```
Assistant mode — run this autonomously, planning in phases and checkpointing
with me at the big scientific decisions.
Assistant mode.

The strong lens SLACS0946+1006 famously has a dark matter subhalo
detection that many argue is unusually concentrated. I'd like to analyse
Expand Down Expand Up @@ -193,6 +192,17 @@ keep the raw data private; make the code, figures, manifests, citation and DOI
publication-ready.
```

## Benchmarks

The three example prompts above (plus the hard cross-package benchmark) are
also shipped as **frozen benchmark prompts** under [`benchmarks/`](benchmarks/),
with scoring rubrics and a small harness that records each run's conversation,
results and score. Run them against different AI agents and models — or the
same model on different days — and the committed run records in
`benchmarks/runs/` plus the regenerated tables in `benchmarks/RESULTS.md` give
you an evidence-backed comparison of how well each setup drives the assistant.
The protocol is in [`benchmarks/README.md`](benchmarks/README.md).

## Scientific Context

The assistant doesn't just know the PyAutoLens API — it ships with a
Expand Down
4 changes: 4 additions & 0 deletions autoassistant/audit_skill_apis.py
Original file line number Diff line number Diff line change
Expand Up @@ -1186,6 +1186,10 @@ def check_citations(root: Path) -> int:
)
files = sorted((root / "skills").glob("*.md"))
files += sorted((root / "wiki" / "core").rglob("*.md"))
# Benchmark protocol docs + frozen prompt cards — but never runs/, whose
# transcripts are historical data and may legitimately quote stale paths.
files += sorted((root / "benchmarks").glob("*.md"))
files += sorted((root / "benchmarks" / "prompts").glob("*.md"))
files += [p for p in (root / "AGENTS.md", root / "llms.txt") if p.exists()]

errors: list[str] = []
Expand Down
Loading
Loading