PR Reviewer

An automated Azure DevOps pull request reviewer that uses a local LLM (via Ollama) to perform code reviews. It fetches PR diffs, sends each file to the model for analysis, and outputs a structured JSON report with findings and recommendations.

Prerequisites

• Python 3.8+
• Ollama running locally with the qwen3-coder:30b model pulled
• An Azure DevOps Personal Access Token (PAT) with read access to the repository

Setup

1. Install dependencies

pip install requests

2. Pull the LLM model

Pull the required model from Ollama. In this case we're using qwen3-coder:30b, but you can choose any compatible model.

ollama pull qwen3-coder:30b

3. Set your Azure DevOps PAT

# Linux / macOS
export AZDO_PR_REVIEWER_PAT="your_personal_access_token"

# Windows (PowerShell)
$env:AZDO_PR_REVIEWER_PAT = "your_personal_access_token"

# Windows (CMD)
set AZDO_PR_REVIEWER_PAT=your_personal_access_token

The PAT needs at minimum Code (Read) permission on the target repository.

Usage

python main.py <pr_url> [--out OUTPUT_FILE]

Arguments

Argument	Required	Description
pr_url	Yes	Full URL to the Azure DevOps pull request
--out	No	Path for the JSON output file (default: review.json)

Supported URL formats

• https://dev.azure.com/{org}/{project}/_git/{repo}/pullrequest/{id}
• https://{org}.visualstudio.com/{project}/_git/{repo}/pullrequest/{id}

Examples

# Basic usage
python main.py "https://dev.azure.com/myorg/myproject/_git/myrepo/pullrequest/12345"

# Save output to a custom file
python main.py "https://dev.azure.com/myorg/myproject/_git/myrepo/pullrequest/12345" --out my_review.json

# VisualStudio.com URL format
python main.py "https://myorg.visualstudio.com/myproject/_git/myrepo/pullrequest/12345"

Output

Results are written to review.json (or the file specified with --out). The structure is:

{
  "pr_url": "...",
  "pr_id": "12345",
  "org": "myorg",
  "project": "myproject",
  "repo": "myrepo",
  "base_commit": "<sha>",
  "target_commit": "<sha>",
  "files": [
    {
      "file": "/path/to/changed/file.cpp",
      "diff": "<unified diff>",
      "review": {
        "summary": "Brief summary of findings for this file.",
        "recommendations": [
          {
            "file": "/path/to/changed/file.cpp",
            "line_range": "42-55",
            "type": "bug | performance | security | design | maintainability",
            "severity": "low | medium | high",
            "confidence": "low | medium | high",
            "recommendation": "What should be changed.",
            "reason": "Why this is an issue.",
            "suggested_fix": "Code snippet showing the fix.",
            "intended_behavior": "What the code should do.",
            "assumptions": "Assumptions made during analysis.",
            "copilot_comments": ["Additional notes."]
          }
        ]
      }
    }
  ],
  "aggregate_recommendations": [ ]
}

aggregate_recommendations is a flat list of all recommendations across every reviewed file, useful for a quick overall summary.

Configuration

Defaults are set at the top of each module and can be changed directly in the source:

File	Constant	Default	Description
diff_extractor.py	MAX_FILES	20	Maximum number of files to review per PR
diff_extractor.py	SKIP_CHANGE_TYPES	{"delete"}	Diff change types to skip
llm_reviewer.py	OLLAMA_URL	http://localhost:11434/api/generate	Ollama API endpoint
llm_reviewer.py	MODEL	qwen3-coder:30b	Ollama model to use
llm_reviewer.py	temperature	0.0	LLM sampling temperature (0 = deterministic)

How It Works

1. Parse the PR URL to extract org, project, repo, and PR ID.
2. Fetch PR metadata and the list of commits from Azure DevOps.
3. Build diffs by comparing the base and target commits, retrieving file contents and constructing unified diffs.
4. Review each changed file by sending the diff to the local LLM with a structured prompt.
5. Write all results plus an aggregated recommendation list to the output JSON file.