Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
3c04c62
Add typecheck script and clear tsc --noEmit baseline
oshorefueled Jul 13, 2026
e319bba
Exclude docs/research from eslint
oshorefueled Jul 13, 2026
20e0acc
Add vitest.config.ts for stable test resolution
oshorefueled Jul 13, 2026
3580e49
Add npm run verify aggregate gate
oshorefueled Jul 13, 2026
0fe8700
Add typecheck CI job and align test workflow Node version
oshorefueled Jul 13, 2026
6c271f1
Deprecate --mode agent and fall back to standard
oshorefueled Jul 13, 2026
a2e9f25
Append Phase 1 completion note to harness audit
oshorefueled Jul 13, 2026
888f027
Add root .vectorlint.ini for reproducible smoke validation
oshorefueled Jul 13, 2026
99f73c2
Add neutral review-domain contract module
oshorefueled Jul 13, 2026
e4259e7
Extract single finding evidence verifier
oshorefueled Jul 14, 2026
25ec49a
Add shared finding severity and scoring helpers
oshorefueled Jul 14, 2026
20d42ff
Implement shared finding processor contract
oshorefueled Jul 14, 2026
671da9c
Route standard check evaluation through shared processor
oshorefueled Jul 14, 2026
9fe6440
Reject judge prompt type at the loader boundary
oshorefueled Jul 14, 2026
d36d06d
Remove legacy judge review path
oshorefueled Jul 14, 2026
d4ea9a7
Remove orphaned orchestrator param types from CLI contract
oshorefueled Jul 14, 2026
15ec7a3
Document finding-processing behavior and clean stale judge docs
oshorefueled Jul 14, 2026
4fa0701
Sanitize prompt criteria at the findings boundary
oshorefueled Jul 14, 2026
699f355
Define structured and tool-calling model clients
oshorefueled Jul 14, 2026
74957bc
Implement SingleModelCallExecutor for single modelCall
oshorefueled Jul 14, 2026
8bf8e86
Implement AgentModelCallExecutor with target-only read tool
oshorefueled Jul 14, 2026
8bfb47d
Wire --model-call executors into the CLI and remove the agent surface…
oshorefueled Jul 14, 2026
6e2a04f
refactor(cli): reject legacy mode flag
oshorefueled Jul 14, 2026
64e434d
refactor(cli): harden config and payload telemetry
oshorefueled Jul 14, 2026
0c18c2d
refactor(observability): remove legacy agent telemetry operation
oshorefueled Jul 14, 2026
9c122f8
refactor(errors): drop orphaned AgentToolError class
oshorefueled Jul 14, 2026
2727025
chore: merge main into executor refactor
oshorefueled Jul 23, 2026
84776cf
refactor(review): remove search-backed evaluator architecture
oshorefueled Jul 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 14 additions & 17 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,26 +1,28 @@
# Repository Guidelines

This repository implements VectorLint — a prompt‑driven, structured‑output content evaluator. Use this guide to navigate the codebase, run it locally, and contribute safely.
This repository implements VectorLint — a prompt‑driven, structured‑output content reviewer. Use this guide to navigate the codebase, run it locally, and contribute safely.

## Project Structure & Module Organization

- `src/`
- `index.ts` — CLI entry; orchestrates config, discovery, evaluation, reporting
- `index.ts` — CLI entry; orchestrates config, discovery, review, reporting
- `boundaries/` — external data validation (config, CLI args, env vars, YAML, API responses)
- `chunking/` — content chunking for large documents (recursive chunker, merger, utilities)
- `cli/` — command definitions and CLI orchestration
- `config/` — configuration loading and management
- `errors/` — custom error types and validation errors
- `evaluators/` — evaluation logic (base evaluator, registry, specific evaluators)
- `executors/` — bounded single-call and target-paging review strategies
- `findings/` — finding verification, filtering, severity, and scoring
- `output/` — TTY formatting (reporter, evidence location, line numbering)
- `prompts/` — YAML frontmatter parsing, schema validation, directive loading
- `providers/` — LLM abstractions (OpenAI, Anthropic, Azure, Gemini), request builder, provider factory
- `scan/` — file discovery (fast‑glob) honoring config and exclusions
- `review/` — shared review contracts, budgets, boundaries, and request building
- `schemas/` — Zod schemas for all external data (API responses, config, CLI, env)
- `scoring/` — density-based score calculation
- `types/` — TypeScript type definitions
- `presets/` — bundled rule packs (e.g., `VectorLint/`)
- `tests/` — Vitest specs for config, scanning, evaluation, providers
- `tests/` — Vitest specs for config, scanning, review, providers

## Agent Behavior Guidelines

Expand Down Expand Up @@ -112,7 +114,7 @@ Rules must be organized into subdirectories (packs) within `RulesPath`.

### Zero-Config Mode

If you just want to evaluate against a user instruction guide without specific rules:
If you just want to review against a user instruction guide without specific rules:
1. Create a `VECTORLINT.md` file with your user instruction content
2. Run `vectorlint doc.md` — VectorLint creates a synthetic rule from your user instructions

Expand All @@ -131,19 +133,18 @@ If you just want to evaluate against a user instruction guide without specific r

VectorLint assembles prompts in this order:

1. **Directive** (`src/prompts/directive-loader.ts`) — Role definition, task, and evaluation instructions
1. **Directive** (`src/prompts/directive-loader.ts`) — Role definition, task, and review instructions
2. **User Instructions** (`VECTORLINT.md`) — Optional global style context
3. **Rule** (the prompt body from the rule file) — Specific evaluation criteria
3. **Rule** (the prompt body from the rule file) — Specific review criteria

The content to evaluate is sent as a **user message** with line numbers prepended.
The content to review is sent as a **user message** with line numbers prepended.

### Chunking

For documents >600 words, VectorLint automatically chunks content:
- Uses recursive splitting (paragraphs → lines → sentences → words)
- Each chunk is evaluated separately
- Each chunk is reviewed separately
- Results are merged and deduplicated
- Disable with `evaluateAs: document` in rule frontmatter

## Coding Style & Naming Conventions

Expand Down Expand Up @@ -180,11 +181,11 @@ For documents >600 words, VectorLint automatically chunks content:

- Boundary validation: all external data (files, CLI, env, APIs) validated at system boundaries using Zod schemas
- Type safety: strict TypeScript with no `any`; use `unknown` + schema validation for external data
- Dependency inversion: depend on `LLMProvider` and `SearchProvider` interfaces; keep providers thin (transport only)
- Dependency inversion: depend on `StructuredModelClient` and `ToolCallingModelClient`; keep providers thin (transport only)
- Dependency injection: inject `RequestBuilder` via provider constructor to avoid coupling
- Separation of concerns: rules define criteria; schemas enforce structure; CLI orchestrates; evaluators process; reporters format
- Separation of concerns: rules define criteria; schemas enforce structure; CLI orchestrates; executors review; finding processors verify; reporters format
- Separation of concerns: when a file starts combining contracts, orchestration, and utility logic, extract shared helpers and types into focused modules
- Extensibility: add providers by implementing `LLMProvider` or `SearchProvider`; add evaluators via registry pattern
- Extensibility: add providers by implementing the model-client contracts; add review strategies behind `ReviewExecutor`
- Shared domain constants: avoid magic strings for core runtime concepts; define shared constants, enums, or types and import them where needed
- Naming: choose domain-accurate names that reflect the real abstraction level; avoid use-case-specific terminology in shared runtime code
- Logging: route runtime logging through an injected logger interface; keep concrete logger implementations behind the abstraction
Expand All @@ -206,7 +207,3 @@ VectorLint supports multiple output formats via the `--output` flag:
- Anthropic: Claude models (Opus, Sonnet, Haiku)
- Azure OpenAI: Azure-hosted OpenAI models
- Google Gemini: Gemini Pro and other Gemini models

### Search Providers

- Perplexity: Sonar models with web search capabilities (used by technical-accuracy evaluator)
29 changes: 7 additions & 22 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ VectorLint is configured via a `.vectorlint.ini` file in the root of your projec
# Optional: Path to custom rules directory
# If omitted, only preset rules (from RunRules) are used
# RulesPath=.github/rules
# Number of concurrent evaluations (Default: 4)
# Number of concurrent reviews (Default: 4)
Concurrency=4
# Default severity for violations (Default: warning)
DefaultSeverity=warning
Expand All @@ -34,8 +34,6 @@ GrammarChecker.strictness=7
RunRules=Acme
# Higher strictness for docs
GrammarChecker.strictness=9
# Require a higher score for technical accuracy
TechnicalAccuracy.threshold=8.0

# Marketing content - run "TechCorp" pack
[content/marketing/**/*.md]
Expand All @@ -56,7 +54,7 @@ These settings control the application's core behavior.
| Setting | Type | Default | Description |
| ----------------- | ------- | ------------ | ------------------------------------------------------------------ |
| `RulesPath` | string | (none) | Root directory for custom rule packs. If omitted, only presets are used. |
| `Concurrency` | integer | `4` | Number of concurrent evaluations to run. |
| `Concurrency` | integer | `4` | Number of concurrent reviews to run. |
| `DefaultSeverity` | string | `warning` | Default severity level (`warning` or `error`) for reported issues. |

---
Expand All @@ -69,36 +67,23 @@ You can place a `VECTORLINT.md` file in your project root to define global style
If no `.vectorlint.ini` exists, VectorLint will automatically:
1. Detect `VECTORLINT.md`
2. Create a synthetic "Style Guide Compliance" rule
3. Evaluate your contents against it
3. Review your contents against it

### Combined Mode
If you have configured rules (via `.vectorlint.ini`), the content of `VECTORLINT.md` is **prepended** to the system prompt for every evaluation. This ensures your global style preferences (tone, terminology) are respected across all specific rules.
If you have configured rules (via `.vectorlint.ini`), the content of `VECTORLINT.md` is **prepended** to the system prompt for every review. This ensures your global style preferences (tone, terminology) are respected across all specific rules.

> **Note:** Keep `VECTORLINT.md` concise. VectorLint will emit a warning if the file exceeds ~4,000 tokens, as very large contexts can degrade performance and increase costs.

---

## LLM & Search Providers
## LLM Providers

VectorLint relies on LLM and Search providers. These are configured globally in `~/.vectorlint/config.toml`, or project scope using a `.env` file (which takes precedence).
VectorLint relies on an LLM provider. Configure it globally in `~/.vectorlint/config.toml`, or at project scope using a `.env` file (which takes precedence).

You can generate these files using the `vectorlint init` command.

### LLM Providers

VectorLint supports multiple LLM providers. Set `LLM_PROVIDER` to your desired provider (e.g., `openai`, `anthropic`, `gemini`) and provide the corresponding API key.

### Search Provider

Some evaluators, such as **TechnicalAccuracy**, require access to current external knowledge to verify facts. VectorLint supports search providers to fetch this information.

**Example configuration for Perplexity:**

```bash
SEARCH_PROVIDER=perplexity
PERPLEXITY_API_KEY=pplx-...
```

### Observability

VectorLint can optionally emit AI execution telemetry to Langfuse. The first implementation is scoped to the Vercel AI SDK calls made through `VercelAIProvider`.
Expand Down Expand Up @@ -224,7 +209,7 @@ You can use named levels or direct numeric multipliers:
[content/docs/**/*.md]
RunRules=Acme
GrammarChecker.strictness=strict
TechnicalAccuracy.strictness=20
Terminology.strictness=20
```

---
Expand Down
27 changes: 10 additions & 17 deletions CREATING_RULES.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Creating Rules for VectorLint

A comprehensive guide to creating powerful, reusable content evaluations using VectorLint's prompt system.
A comprehensive guide to creating powerful, reusable content reviews using VectorLint's prompt system.

## Table of Contents

Expand Down Expand Up @@ -36,9 +36,8 @@ Every rule is a Markdown file with two parts:
```markdown
---
# YAML Frontmatter (Configuration)
id: MyEval
name: My Content Evaluator
evaluator: base
id: MyRule
name: My Content Reviewer
severity: error
---

Expand All @@ -57,9 +56,7 @@ project/
│ └── rules/
│ ├── Acme/ ← Company style guide pack
│ │ ├── grammar-checker.md
│ │ ├── headline-evaluator.md
│ │ └── Technical/ ← Nested organization supported
│ │ └── technical-accuracy.md
│ │ └── headline-reviewer.md
│ └── TechCorp/ ← Another company's pack
│ └── brand-voice.md
└── .vectorlint.ini
Expand All @@ -78,7 +75,6 @@ score from the verified findings.

```markdown
---
evaluator: base
id: GrammarChecker
name: Grammar Checker
severity: error
Expand Down Expand Up @@ -117,7 +113,7 @@ Check this content for grammar issues, spelling errors, and punctuation mistakes

The `target` field allows you to:

1. **Specify which part** of content to evaluate (via regex)
1. **Specify which part** of content to review (via regex)
2. **Require certain content** to exist (e.g., "must have an H1 headline")
3. **Provide helpful suggestions** when content is missing

Expand All @@ -136,13 +132,13 @@ target:

**When `required: true`:**

- If content matches → Evaluation proceeds normally
- If content matches → Review proceeds normally
- If no match → Immediate `error` with the suggestion message

**When `required: false` or omitted:**

- If content matches → Evaluate the matched content
- If no match → Evaluate entire content
- If content matches → Review the matched content
- If no match → Review entire content

---

Expand All @@ -153,12 +149,10 @@ target:
| Field | Type | Required | Description |
| ------------- | ------------- | -------- | -------------------------------------------------------------- |
| `specVersion` | string/number | No | Rule specification version (use `1.0.0`) |
| `evaluator` | string | No | Evaluator type: `base`, `technical-accuracy` (default: `base`) |
| `id` | string | **Yes** | Unique identifier (used in error reporting) |
| `name` | string | **Yes** | Human-readable name |
| `severity` | string | No | `error` or `warning` (default: `warning`) |
| `strictness` | number/string | No | Density penalty: a positive number, `lenient`, `standard`, or `strict` |
| `evaluateAs` | string | No | `document` or `chunk` (default: `chunk`) |
| `target` | object | No | Content matching specification |

---
Expand All @@ -178,7 +172,7 @@ Check if the headline is good.
✅ **Good:**

```markdown
You are a headline evaluator for developer blog posts. Assess whether the headline:
You are a headline reviewer for developer blog posts. Assess whether the headline:

1. Clearly communicates a specific benefit
2. Uses natural, conversational language (avoid buzzwords)
Expand Down Expand Up @@ -208,7 +202,6 @@ Help the LLM understand your domain:

```markdown
---
evaluator: base
id: GrammarChecker
name: Grammar Checker
severity: error
Expand All @@ -226,4 +219,4 @@ Report any errors found with specific examples.

---

**Happy evaluating! 🚀**
**Happy reviewing! 🚀**
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# VectorLint: Prompt it, Lint it! [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

VectorLint is a command-line tool that uses LLMs to find and score terminology, technical accuracy, and style issues that require contextual understanding.
VectorLint is a command-line tool that uses LLMs to find and score terminology, consistency, and style issues that require contextual understanding.

![VectorLint Screenshot](./assets/VectorLint_screenshot.jpeg)

Expand Down Expand Up @@ -51,7 +51,7 @@ VectorLint scores your content using error density, enabling you to measure qual

## How VectorLint Reduces False Positives

VectorLint uses a PAT (Pay A Tax) evaluation approach:
VectorLint uses a PAT (Pay A Tax) review approach:

1. **Candidate generation:** the model returns all potential violations with required gate-check fields (rule support, exact evidence, context support, plausible non-violation, and fix quality).
2. **Deterministic surfacing:** VectorLint applies a strict filter and only surfaces violations that pass all required gates.
Expand Down
4 changes: 2 additions & 2 deletions docs/VECTORLINT.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# VectorLint example styling rules

{/* This file is prepended to the system prompt for every VectorLint evaluation
This file defines global style instructions for all VectorLint evaluations.
{/* This file is prepended to the system prompt for every VectorLint review
This file defines global style instructions for all VectorLint reviews.
VectorLint prepends its contents to the system prompt for every rule it runs.
Keep this file under ~800 tokens to avoid performance and cost issues.
Adapt the rules below to match your team's style guide.
Expand Down
10 changes: 5 additions & 5 deletions docs/best-practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ After you run `VECTORLINT.md` against your content library and complete your fir

## Keep VECTORLINT.md under 800 tokens

VectorLint emits a warning at 4,000 tokens, but a practical target is *much* lower. Under 800 tokens leaves headroom for rule-specific prompts to add context without the combined system prompt becoming unwieldy. Long context degrades LLM precision and increases API costs on every evaluation.
VectorLint emits a warning at 4,000 tokens, but a practical target is *much* lower. Under 800 tokens leaves headroom for rule-specific prompts to add context without the combined system prompt becoming unwieldy. Long context degrades LLM precision and increases API costs on every review.

If your `VECTORLINT.md` is growing, that's usually a sign that some rules are specific enough to belong in a dedicated rule pack file instead.

Expand All @@ -28,7 +28,7 @@ Check if the writing is clear.

Write:
```
You are a clarity evaluator for developer documentation. Flag sentences that:
You are a clarity reviewer for developer documentation. Flag sentences that:
1. Exceed 25 words
2. Use passive voice where active voice is possible
3. Contain filler phrases: "it is important to note", "please be aware", "in order to"
Expand All @@ -38,7 +38,7 @@ The second prompt gives the LLM a defined audience, measurable criteria, and spe

## Give rules domain context

LLMs evaluate content relative to an implied standard. Make that standard explicit with a context block in your rule prompt. Tell the model who the audience is, what they value, and what good looks like for your specific content type.
LLMs review content relative to an implied standard. Make that standard explicit with a context block in your rule prompt. Tell the model who the audience is, what they value, and what good looks like for your specific content type.

```markdown
## CONTEXT BANK
Expand Down Expand Up @@ -85,7 +85,7 @@ The goal is a workflow where every finding is worth reading. That takes iteratio

## Set a higher confidence threshold in CI than locally

In CI, a false positive blocks a merge. Set `CONFIDENCE_THRESHOLD` higher in your CI environment than in local development so only the highest-confidence findings gate a merge. Lower-confidence candidates still surface locally where a writer can evaluate them in context.
In CI, a false positive blocks a merge. Set `CONFIDENCE_THRESHOLD` higher in your CI environment than in local development so only the highest-confidence findings gate a merge. Lower-confidence candidates still surface locally where a writer can review them in context.

```bash
# Local development — catch more, review in context
Expand Down Expand Up @@ -117,6 +117,6 @@ Skipping this step leads to rules that look correct on paper but produce noise i

## Next steps

- [Tuning evaluation precision](/false-positive-tuning) — detailed guidance on CONFIDENCE_THRESHOLD and strictness
- [Tuning review precision](/false-positive-tuning) — detailed guidance on CONFIDENCE_THRESHOLD and strictness
- [CI Integration](/ci-integration) — set up content quality gates in your pipeline
- [Customize style rules](/customize-style-rules) — write effective prompts for rule pack files
2 changes: 1 addition & 1 deletion docs/ci-integration.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,6 @@ Store API keys in your CI system's secret or environment variable manager — ne

## Next steps

- [Tuning evaluation precision](/false-positive-tuning) — set `CONFIDENCE_THRESHOLD` and strictness for CI
- [Tuning review precision](/false-positive-tuning) — set `CONFIDENCE_THRESHOLD` and strictness for CI
- [Project Configuration](/project-config) — configure which files CI checks
- [CLI reference](/cli-reference) — full command and exit code reference
4 changes: 1 addition & 3 deletions docs/cli-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,6 @@ VectorLint reads configuration from environment variables in addition to config
| `ANTHROPIC_API_KEY` | — | API key for Anthropic |
| `GEMINI_API_KEY` | — | API key for Google Gemini |
| `AZURE_OPENAI_API_KEY` | — | API key for Azure OpenAI |
| `SEARCH_PROVIDER` | — | Search provider for `technical-accuracy` evaluator: `perplexity` |
| `PERPLEXITY_API_KEY` | — | API key for Perplexity search |
| `CONFIDENCE_THRESHOLD` | `0.75` | PAT pipeline confidence gate. Lower surfaces more findings; higher surfaces fewer. |

For the full environment variable reference see [Environment variables](/env-variables).
Expand All @@ -99,5 +97,5 @@ For the full environment variable reference see [Environment variables](/env-var

- [Installation](/installation) — install VectorLint globally or run with npx
- [Configuration](/configuration) — configure VectorLint for your project
- [Tuning evaluation precision](/false-positive-tuning) — adjust `CONFIDENCE_THRESHOLD` and strictness
- [Tuning review precision](/false-positive-tuning) — adjust `CONFIDENCE_THRESHOLD` and strictness
- [CI Integration](/ci-integration) — use exit codes to gate merges on content quality
Loading
Loading