Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
3c04c62
Add typecheck script and clear tsc --noEmit baseline
oshorefueled Jul 13, 2026
e319bba
Exclude docs/research from eslint
oshorefueled Jul 13, 2026
20e0acc
Add vitest.config.ts for stable test resolution
oshorefueled Jul 13, 2026
3580e49
Add npm run verify aggregate gate
oshorefueled Jul 13, 2026
0fe8700
Add typecheck CI job and align test workflow Node version
oshorefueled Jul 13, 2026
6c271f1
Deprecate --mode agent and fall back to standard
oshorefueled Jul 13, 2026
a2e9f25
Append Phase 1 completion note to harness audit
oshorefueled Jul 13, 2026
888f027
Add root .vectorlint.ini for reproducible smoke validation
oshorefueled Jul 13, 2026
99f73c2
Add neutral review-domain contract module
oshorefueled Jul 13, 2026
e4259e7
Extract single finding evidence verifier
oshorefueled Jul 14, 2026
25ec49a
Add shared finding severity and scoring helpers
oshorefueled Jul 14, 2026
20d42ff
Implement shared finding processor contract
oshorefueled Jul 14, 2026
671da9c
Route standard check evaluation through shared processor
oshorefueled Jul 14, 2026
9fe6440
Reject judge prompt type at the loader boundary
oshorefueled Jul 14, 2026
d36d06d
Remove legacy judge review path
oshorefueled Jul 14, 2026
d4ea9a7
Remove orphaned orchestrator param types from CLI contract
oshorefueled Jul 14, 2026
15ec7a3
Document finding-processing behavior and clean stale judge docs
oshorefueled Jul 14, 2026
4fa0701
Sanitize prompt criteria at the findings boundary
oshorefueled Jul 14, 2026
699f355
Define structured and tool-calling model clients
oshorefueled Jul 14, 2026
74957bc
Implement SingleModelCallExecutor for single modelCall
oshorefueled Jul 14, 2026
8bf8e86
Implement AgentModelCallExecutor with target-only read tool
oshorefueled Jul 14, 2026
8bfb47d
Wire --model-call executors into the CLI and remove the agent surface…
oshorefueled Jul 14, 2026
6e2a04f
refactor(cli): reject legacy mode flag
oshorefueled Jul 14, 2026
64e434d
refactor(cli): harden config and payload telemetry
oshorefueled Jul 14, 2026
0c18c2d
refactor(observability): remove legacy agent telemetry operation
oshorefueled Jul 14, 2026
986e6e9
docs: document bounded harness architecture
oshorefueled Jul 14, 2026
9c122f8
refactor(errors): drop orphaned AgentToolError class
oshorefueled Jul 14, 2026
7e32601
docs: reframe internal agent-mode cleanup
oshorefueled Jul 14, 2026
9441989
Merge parent harness executor updates
oshorefueled Jul 14, 2026
6dde8f5
docs: clarify historical audit framing
oshorefueled Jul 14, 2026
ab9b8e1
docs: enforce documentation artifact boundaries
oshorefueled Jul 14, 2026
30d0250
docs: merge main into harness architecture docs
oshorefueled Jul 24, 2026
3b96298
docs: align harness language with reviews
oshorefueled Jul 24, 2026
210822d
docs: align public docs with measurable content quality
oshorefueled Jul 24, 2026
8508c04
docs: standardize review strategy terminology
oshorefueled Jul 24, 2026
f27b5b6
docs: clarify credential configuration options
oshorefueled Jul 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

This repository implements VectorLint — a prompt‑driven, structured‑output content reviewer. Use this guide to navigate the codebase, run it locally, and contribute safely.

Use [`CONTEXT.md`](./CONTEXT.md) as the shared VectorLint domain language across code, docs, tests, and agent work. Prefer its terms when naming modules, writing tests, drafting docs, and describing architecture.

## Project Structure & Module Organization

- `src/`
Expand Down Expand Up @@ -77,6 +79,18 @@ These guidelines reduce common LLM coding mistakes. They bias toward caution ove
- Keep security and scope-boundary comments when they explain constraints that the implementation must preserve.
- Test supported behavior and current invariants; do not add negative tests solely to memorialize rejected designs.

### Documentation Artifact Boundaries

- Do not commit raw planning or investigation artifacts to the product codebase.
- Keep audits, plans, specs, run notes, and similar coordination artifacts out of tracked repo paths such as `docs/audits/`, `docs/plans/`, `docs/specs/`, `audits/`, `plans/`, and `specs/`.
- Store coordination artifacts in `.agent-runs/` or another ignored workspace location.
- If a durable architectural decision must be committed, write it as an ADR. ADRs are the only allowed committed decision/planning artifacts.
- Product documentation may describe shipped behavior, configuration, and usage, but it must not preserve internal audit, plan, or spec documents as reviewed product docs.
- `CONTEXT.md` is the project’s ubiquitous language. It may define internal domain concepts needed by contributors, provided those concepts use the same terminology users encounter.
- `AGENTS.md` should require consistent terminology across code, tests, and documentation.
- User-facing `/docs` should use only the subset of that language needed for user goals.
- Internal contracts, processing stages, tool names, architecture constraints, and rejected alternatives should not leak into `/docs` merely because they exist in `CONTEXT.md`.

## Configuration System

### Quick Start
Expand Down
147 changes: 147 additions & 0 deletions CONTEXT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
# VectorLint

VectorLint is a content review harness. This language file defines the terms used to describe reviews, rules, findings, scoring, and execution so product, documentation, tests, and code can share the same vocabulary.

## Language

### Review Model

**Content Review Harness**:
A system that reviews supplied target content against source-backed rules and returns structured review results. The harness does not own exploration, research, workspace search, or content rewriting.
_Avoid_: Autonomous agent, workspace agent, agent mode

**Review**:
A review of target content against one or more rules. It produces findings, scores, diagnostics, usage metadata, and output.
_Avoid_: Lint run when referring to the domain process, agent session

### Review Inputs

**Caller**:
The person or system that invokes VectorLint and supplies the target, target content, and any allowed review context. A caller does not define rules or configuration as part of the review invocation.
_Avoid_: Agent, user agent, workspace owner

**Target**:
The subject of a review. A target identifies what VectorLint is reviewing, regardless of whether it came from a file, memory, or another caller-provided source.
_Avoid_: File when the review subject may not be a file, page when the content may not be documentation

**Target File**:
A file used as the source of a target. A target file is one way to provide target content, but not every target must come from a file.
_Avoid_: Target when the distinction between the file and the review subject matters

**Target Content**:
The actual text or content body reviewed for a target. Findings and finding evidence are grounded in target content.
_Avoid_: Target file, page content

**Review Context**:
Additional content explicitly supplied to VectorLint for a review. Review context is in scope only because it was allowed as part of the review input.
_Avoid_: Caller context, workspace context, discovered context, ambient context

**On-Page Boundary**:
The rule that VectorLint reviews only target content and review context. VectorLint must not discover arbitrary workspace files or expand the review scope on its own.
_Avoid_: Workspace scope, project-wide scope, cross-file scope

### Rules

**Rule**:
A source-backed instruction that defines observable violation conditions VectorLint should look for. Rules exist before a review is invoked.
_Avoid_: Prompt when referring to the domain concept, model instruction when implying the model authored it

**Via Negativa Review**:
A review approach that looks for evidence a rule was violated rather than evidence the content aligns with an ideal. VectorLint rules should be written so a model can answer whether an observable violation condition is present.
_Avoid_: Alignment review, subjective assessment

**Violation Condition**:
An observable yes/no condition that counts as a rule violation when present in target content. A violation condition should be specific enough to ground a finding in finding evidence.
_Avoid_: Criterion when it implies subjective grading, preference, guideline

**Rule Pack**:
A named collection of related rules that can be applied together.
_Avoid_: Preset when describing the domain concept, folder

**Review Configuration**:
Pre-existing settings that determine how VectorLint runs reviews, selects rules, and formats behavior. Configuration is not the same as target content or review context.
_Avoid_: Caller input when referring to review invocation, target configuration

**Check Rule**:
A rule expressed through observable violation conditions. This is the only future-facing rule style in VectorLint.
_Avoid_: Direct rule, standard rule

### Execution

**Model Call**:
The call shape VectorLint uses to run a reviewer model against target content during a review.
_Avoid_: Execution strategy, content access, process mode, rule type, mode

**Single Model Call**:
A model call where VectorLint supplies the review request and target content without giving the model a read tool. If VectorLint chunks a large target, each chunk is still reviewed through a single model call.
_Avoid_: Direct strategy, standard mode, check path

**Agent Model Call**:
A bounded model call where the model may request sections of the target content through a single target-scoped read capability. This is for context management during large or context-sensitive reviews; it is not workspace exploration.
_Avoid_: Autonomous agent mode, workspace agent, file reader

**Auto Model Call**:
The model call value where VectorLint deterministically chooses between single and agent.
_Avoid_: Smart mode, agent fallback

**Target Read Capability**:
The bounded ability to read line ranges from target content. It cannot read arbitrary files, search the workspace, rewrite rules, or create top-level workspace findings.
_Avoid_: Tool suite, workspace tools, read_file

### Findings

**Candidate Finding**:
A potential issue raised before VectorLint has decided whether it should be reported. Candidate findings may be filtered out or become diagnostics.
_Avoid_: Violation when the issue has not been accepted, result

**Verified Finding**:
A finding that VectorLint has accepted for reporting because it is grounded in the target content and passes review filters.
_Avoid_: Issue when finding-evidence status matters, raw finding

**Finding**:
A reported content issue produced by a review. When precision matters, use candidate finding or verified finding.
_Avoid_: Violation as the default user-facing term, problem

**Finding Evidence**:
The exact target-content text or surrounding context that supports a finding.
_Avoid_: Quote when the text may include surrounding context, match when referring to the concept rather than a located span

**Finding Evidence Verification**:
The act of confirming that the finding evidence for a candidate finding can be located in the target content. Unverified finding evidence should not become a verified finding.
_Avoid_: Line fallback, model-provided location

**Finding Processing**:
The review step that turns candidate findings into verified findings, diagnostics, scores, and final review output.
_Avoid_: Projection, result processing, output processing

**Diagnostic**:
A structured note about review execution or finding processing, especially when something affects trust, completeness, or interpretation but is not itself a content finding.
_Avoid_: Finding, warning when referring to the structured domain object

### Scoring And Output

**Severity**:
The impact level attached to a finding or rule outcome.
_Avoid_: Priority, importance

**Score**:
A normalized quality measurement for a rule or review. Scores come from verified finding count or density for objective violation checks.
_Avoid_: Grade when referring to the domain object, rating

**Review Result**:
The structured outcome of a review: verified findings, scores, diagnostics, usage metadata, and operational status.
_Avoid_: Projection result, formatter result, raw model output

**Output Format**:
The representation used to present a review result to a human or machine, such as line output or JSON.
_Avoid_: Review result, report type

**Review Budget**:
The explicit limits for a review, such as model calls, target size, review context size, chunks per rule, and duration. A review budget limits work, not the number of verified findings emitted.
_Avoid_: Rate limit, timeout when referring to the full budget concept

### Historical Terms

**Autonomous Agent Mode**:
The old VectorLint direction where VectorLint exposed workspace tools to a model and let it explore beyond the target content. This is historical language only; current domain language should use single model call, agent model call, caller, and content review harness.
_Avoid_: Agent mode as a current feature, workspace-agent review
66 changes: 38 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# VectorLint: Prompt it, Lint it! [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
# VectorLint [![npm version](https://img.shields.io/npm/v/vectorlint.svg)](https://www.npmjs.com/package/vectorlint) [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

VectorLint is a command-line tool that uses LLMs to find and score terminology, consistency, and style issues that require contextual understanding.
VectorLint is a content review harness that turns observable quality standards into measurable feedback for agents.

VectorLint reviews content against rules that describe observable traits. It returns structured, source-grounded findings and quality scores, giving agents repeatable signals they can use to revise content and review it again.

![VectorLint Screenshot](./assets/VectorLint_screenshot.jpeg)

Expand Down Expand Up @@ -30,35 +32,27 @@ Run VectorLint without installing:
npx vectorlint path/to/article.md
```

## Enforce Your Style Guide
## Define Your Quality Standards

Define rules as Markdown files with YAML frontmatter to enforce your specific content standards:
Define rules as Markdown files with YAML frontmatter. Each rule describes the observable traits that indicate content does not meet one of your standards.

- **Check SEO Optimization** - Verify content follows SEO best practices
- **Detect AI-Generated Content** - Identify artificial writing patterns
- **Verify Technical Accuracy** - Catch outdated or incorrect technical information
- **Ensure Tone & Voice Consistency** - Match content to appropriate tone for your audience
Rules can identify prohibited terminology, unsupported claims, repetitive explanations, vague guidance, or other quality issues specific to your content.

If you can write a prompt for it, you can lint it with VectorLint.
VectorLint works best when each rule states what evidence counts as a finding instead of asking for a general judgment of whether the content is good.

👉 **[Learn how to create custom rules →](./CREATING_RULES.md)**

## Quality Scores

VectorLint scores your content using error density, enabling you to measure quality across documentation. This gives your team a shared understanding of which content needs attention and helps track improvements over time.

- **Density-Based Scoring:** For errors that can be counted, scores are calculated based on **error density (errors per 100 words)**, making quality assessment fair across documents of any length.

## How VectorLint Reduces False Positives
VectorLint turns review results into comparable quality signals. Agents can use those scores to measure whether revisions improve content across repeated review cycles.

VectorLint uses a PAT (Pay A Tax) review approach:
For countable findings, VectorLint calculates scores from error density, or findings per 100 words. This makes results comparable across content of different lengths.

1. **Candidate generation:** the model returns all potential violations with required gate-check fields (rule support, exact evidence, context support, plausible non-violation, and fix quality).
2. **Deterministic surfacing:** VectorLint applies a strict filter and only surfaces violations that pass all required gates.
## Grounded Findings

This means CLI output is intentionally stricter than raw model candidates, reducing noisy findings and improving precision.
Every reported finding includes evidence from the reviewed content. VectorLint confirms that evidence can be located in the source and omits findings that cannot be grounded there.

The confidence gate is user-configurable via:
Adjust finding sensitivity with:

```bash
CONFIDENCE_THRESHOLD=0.75
Expand All @@ -72,15 +66,15 @@ CONFIDENCE_THRESHOLD=0.75

### 1. Zero-Config Mode (Fastest)

If you just want to check your content against a style guide:
If you want to review content against a single set of quality standards:

```bash
vectorlint init --quick
```

This creates a `VECTORLINT.md` file where you can paste your style guide.
This creates a `VECTORLINT.md` file where you can define your quality standards.

> **Note:** You must set up your credentials in `~/.vectorlint/config.toml` (see Step 3) before running checks.
> **Note:** Before running a review, set up your credentials in either `~/.vectorlint/config.toml` or a local `.env` file (see Step 3).

Then run:

Expand All @@ -97,14 +91,15 @@ vectorlint init
```

This creates:

- **VectorLint Config** (`.vectorlint.ini`): Project-specific settings.
- **App Config** (`~/.vectorlint/config.toml`): LLM provider API keys.
- **App Config** (`~/.vectorlint/config.toml`): Model provider API keys.

👉 **[Full configuration reference →](./CONFIGURATION.md)**

### 3. Configure API Keys

Open your global **App Config** (`~/.vectorlint/config.toml`) and uncomment the section for your preferred LLM provider (OpenAI, Anthropic, Gemini, or Azure).
Open your global **App Config** (`~/.vectorlint/config.toml`) and uncomment the section for your preferred model provider (OpenAI, Anthropic, Gemini, or Azure).

```toml
[env]
Expand All @@ -114,7 +109,7 @@ OPENAI_API_KEY = "sk-..."

> *Note: You can also use a local `.env` file in your project, which takes precedence over the global config.*

**Run a check:**
**Run a review:**

```bash
vectorlint doc.md
Expand All @@ -124,9 +119,9 @@ VectorLint is bundled with a `VectorLint` preset containing rules for AI pattern

👉 **[Learn how to create custom rules →](./CREATING_RULES.md)**

### 4. Optional: Enable Langfuse observability
### 4. Optional: Configure Langfuse observability

VectorLint can emit AI execution telemetry through Langfuse without hardcoding Langfuse into the provider layer. This is best-effort instrumentation for the Vercel AI SDK calls used by `VercelAIProvider`.
VectorLint can send model execution telemetry to Langfuse.

Add these environment variables to your global config or local `.env` file:

Expand All @@ -144,11 +139,26 @@ Notes:
- Prompts and outputs are recorded when Langfuse observability is enabled.
- Do not send secrets, credentials, or PII unless your policy explicitly allows observability tooling to access that data.

## Choose a Review Strategy

VectorLint chooses a review strategy automatically. The default works for most
content:

```bash
vectorlint doc.md
```

Use `--model-call` when you need to override that strategy for a particular
review. Choose `single` for normal, self-contained documents or `agent` for
large documents whose relevant context spans multiple sections.

See [Model calls](docs/model-calls.mdx) for selection guidance and examples.

## Contributing

We welcome your contributions! Whether it's adding new rules, fixing bugs, or improving documentation, please check out our [Contributing Guidelines](.github/CONTRIBUTING.md) to get started.

## Resources

- **[Creating Custom Rules](./CREATING_RULES.md)** - Write your own quality checks in Markdown
- **[Creating Rules](./CREATING_RULES.md)** - Define observable quality standards in Markdown
- **[Configuration Guide](./CONFIGURATION.md)** - Complete reference for `.vectorlint.ini`
14 changes: 14 additions & 0 deletions docs/best-practices.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,20 @@ LLMs review content relative to an implied standard. Make that standard explicit

A grammar rule without context produces generic grammar findings. The same rule with a developer audience context produces findings calibrated to technical writing conventions.

## Write observable violation conditions

VectorLint works best when each rule names what counts as a violation. Treat a rule as a checklist of conditions the reviewer can confirm from target evidence.

```markdown
Flag a headline when it:
1. Omits the product or feature being discussed
2. Promises a benefit that the article does not support
3. Uses vague value words such as "seamless", "powerful", or "robust"
4. Exceeds 70 characters
```

Observable conditions make findings easier to verify and reduce subjective disagreements during review.

## Tier strictness by content type

Not all content deserves the same quality bar. Apply strictness in proportion to how much a failure costs — measured in user trust, support load, or brand impact.
Expand Down
33 changes: 29 additions & 4 deletions docs/cli-reference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -58,10 +58,35 @@ vectorlint --help

## Flags

| Flag | Description |
|------|-------------|
| `--help` | Print help and exit |
| `--version` | Print the installed VectorLint version and exit |
| Flag | Default | Description |
|------|---------|-------------|
| `--help` | — | Print help and exit |
| `--version` | — | Print the installed VectorLint version and exit |
| `--config <path>` | `.vectorlint.ini` | Path to a custom project config file |
| `--debug-json` | `false` | Write debug JSON artifacts under `.vectorlint/runs/` |
| `--model-call <value>` | `auto` | Review strategy: `single`, `agent`, or `auto` |
| `--output <format>` | `line` | Output format: `line`, `json`, `vale-json`, or `rdjson` |
| `--show-prompt` | `false` | Print full prompt and injected content |
| `--show-prompt-trunc` | `false` | Print truncated prompt/content previews |
| `--verbose` | `false` | Enable verbose logging |

### `--model-call`

`--model-call` selects the review strategy. Leave the default, `auto`, enabled for most reviews.

```bash
vectorlint doc.md --model-call single
vectorlint doc.md --model-call agent
vectorlint doc.md --model-call auto
```

| Value | Use it for |
| --- | --- |
| `auto` | Most reviews; VectorLint selects a strategy for the current document |
| `single` | Normal, self-contained documents and focused rules |
| `agent` | Large documents or reviews that compare context across sections |

See [Model calls](/model-calls) for usage guidance.

## Output

Expand Down
1 change: 1 addition & 0 deletions docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
"group": "Reference",
"pages": [
"cli-reference",
"model-calls",
"configuration-schema",
"env-variables",
"frontmatter-fields",
Expand Down
Loading
Loading