README.md

PromptDiff 🧠🔍

Git-style diffs for LLM outputs.

PromptDiff helps you compare Large Language Model outputs across prompts, models, or parameters to detect semantic regressions and improvements before they reach users.

---

✨ Features

• 🔍 Text + semantic diffs
• 📊 Regression / improvement summaries
• 🧪 Batch prompt evaluation
• 🤖 Model-agnostic (OpenAI, Anthropic, Ollama, local*)
• 🛠 CLI-first, CI-friendly
• 🎨 Web UI for interactive comparisons

---

🚀 Install

pip install promptdiff

# With model support
pip install promptdiff[openai]      # For OpenAI models
pip install promptdiff[anthropic]   # For Anthropic models
pip install promptdiff[all]         # All providers

---

🧠 Quick Start

1. Create prompts

Create a prompts.json file:

[
  {
    "id": "summary_test",
    "prompt": "Summarize this article in 3 bullet points: {{text}}",
    "vars": {
      "text": "Large language models are changing software..."
    }
  }
]

2. Run PromptDiff

Compare two models:

promptdiff run \
  --prompts prompts.json \
  --baseline gpt-4 \
  --candidate gpt-4.1

Compare multiple models:

promptdiff compare \
  --prompts prompts.json \
  --models ollama:llama3,ollama:granite4,ollama:qwen2.5 \
  --names llama3,granite4,qwen2.5

3. View Report

promptdiff report --results results.json --format markdown

4. Launch Web UI

promptdiff ui

Then open your browser to http://localhost:8501 for an interactive interface!

---

📊 Example Output

Prompt: summary_test

✔ Clarity improved
✖ Slight factual loss
⚠ Verbosity increased

Similarity score: 0.82

---

🤝 Contributing

PRs welcome! See CONTRIBUTING.md.

---

📜 License

MIT

---

📖 More Details

For detailed documentation including advanced options, environment setup, Ollama configuration, and more, see DETAILED.md.