BadScientist

Official code for "BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?", accepted to ACL 2026 (Main).

• 🌐 Project Website - Learn more about BadScientist
• 🔧 GitHub Repo - Official repo for BadScientist pipeline
• 🤗 BadScientist Prompts;

This repository implements the two pipelines studied in the paper:

1. Generation — produces a full LaTeX paper (sections, figures, BibTeX, PDF) from a one-line research seed, optionally conditioned on one of five fabrication strategies.
2. Review — runs an LLM reviewer (single or ensemble) against a generated or supplied PDF and returns a structured review.

Any provider supported by LiteLLM can drive both pipelines (OpenAI, Anthropic, Azure OpenAI, Gemini, Ollama, …).

Responsible use

Every LLM prompt used by these pipelines lives in a gated HuggingFace dataset, badscientist/BadScientist-Prompts, not in this source. Access requires requesting and accepting the research-use terms; without a valid local prompt cache the pipeline refuses to start. This gate exists because the prompts instruct models to produce fabricated scientific content.

Intended uses: LLM-safety research, reviewer calibration, detection of fabricated content. Do not submit generated papers as if they were genuine scientific work.

Install

git clone https://github.com/Bad-Scientist/BadScientist.git
cd BadScientist
bash setup.sh

setup.sh installs LaTeX (via apt / dnf / pacman / brew --cask basictex), installs uv if missing, resolves the Python env (uv sync), and downloads the prompt dataset. Requires a HuggingFace token (huggingface-cli login or HF_TOKEN in .env) and accepted terms on the dataset page.

Flags: --skip-latex, --skip-prompts.

Then edit .env (copied from .env.example) and set MODEL / REVIEW_MODEL plus the matching provider credentials.

Usage

# Generation
uv run python launch.py \
    --template-dir examples/latex \
    --seed-path examples/seed_idea.json \
    --num-ideas 3 --out results

# Generation + review
uv run python launch.py ... --enable-review

# Review an existing PDF
uv run python tools/review_one_pdf.py path/to/paper.pdf

Artifacts land in results/<timestamp>/idea_<N>/ — package.json, latex/paper.pdf, and review.json when review is enabled.

Fabrication strategies

--strategy {S1|S2|S3|S4|S5|combined|all|none} selects which of the five fabrication modes from the paper to condition the generator on. combined applies all five in one run; all runs the pipeline six times (S1..S5 + combined) under <out>/<strategy>/. Default is none.

Concurrency

Within a single run, ideas are processed in parallel. --concurrency N caps the worker count (default: one worker per idea). Lower it to stay under your provider's rate limit.

Repository layout

launch.py                   CLI entrypoint
core.py                     generation pipeline
perform_review.py           review pipeline
llm_client.py               LiteLLM wrapper
prompts_registry.py         lazy, SHA-256-verified prompt loader
setup.sh                    one-shot bootstrap
setup_prompts.py            HuggingFace dataset downloader
tools/                      auxiliary scripts (single-PDF review, LaTeX helpers)
examples/                   sample seed + ICLR LaTeX template

Citation

@article{jiang2025badscientist,
  title={BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?},
  author={Jiang, Fengqing and Feng, Yichen and Li, Yuetai and Niu, Luyao and Alomair, Basel and Poovendran, Radha},
  journal={arXiv preprint arXiv:2510.18003},
  year={2025}
}