Share objective finding processing#98
Merged
Merged
Conversation
- Add npm run typecheck (tsc --noEmit) as the source-of-truth type gate - Narrow exactOptionalPropertyTypes gaps via conditional spreads and optional-with-undefined fields across agent executor, orchestrator, observability, providers, and scorer - Make CheckItem.line/description optional to match runtime reality (reported items carry optional fields; output formatters use a separate Issue type, so display is unaffected) - Confine @ai-sdk/perplexity LanguageModelV1 -> LanguageModel skew to a single cast; ai@6 dropped V1 types and a provider upgrade is a separate, out-of-scope dependency change - No strict compiler options relaxed; src/agent/* left compiling only
- Add docs/research/** to eslint ignores; the directory holds local, untracked throwaway research scripts that are not part of the shipped package and should not be type-checked by lint
- Project had no vitest config; vitest ran on pure defaults - Pin test discovery to tests/**/*.test.ts and inline ora, @langfuse/otel, and @opentelemetry/sdk-node so suites that transitively import agent/observability modules resolve reliably - No change to the 45 suites / 323 tests baseline
- Runs typecheck, lint, and test:run in sequence - Single command for CI and downstream refactor phases to prove the verification baseline
- New typecheck.yml runs tsc --noEmit on push/PR for main and release-docs - Bump test.yml from Node 18 to 20 to satisfy package.json engines>=20.6 - Catches type regressions in CI before merge, not just via ESLint
- README Agent Mode section replaced with under-review notice linking the audit - --mode agent now warns through the injected logger and runs standard evaluation; the agent executor code is retained (unreachable from the CLI) pending Phase 4 removal - Add logger to EvaluationOptions so the deprecation warning routes through the logging abstraction instead of console - Rewrite orchestrator-agent-output tests to assert deprecation + fallback BREAKING: --mode agent no longer runs the autonomous workspace-agent loop
- Record the shifted 2026-07-13 baseline: only tsc --noEmit was failing; lint and test:run were already green, so the audit's lint and four-suite module-resolution failures are stale - Note the durable Phase 1 gates (typecheck, verify, vitest.config, docs/research lint exclusion, Node 20 typecheck/test workflows) - Point to the --mode agent deprecation and standard fallback as the precondition for Phases 2-5; record that no spec or architecture doc is superseded in this appendix
- Commit a repository-native .vectorlint.ini using the bundled VectorLint preset (verbatim `vectorlint init` template). - The required validation commands `npm start -- README.md --output line` and `npm start -- README.md --mode agent` previously failed from the committed branch with "Missing configuration file"; they had relied on an untracked, deleted VECTORLINT.md. - Both smoke commands now run from a clean checkout with no untracked setup files; --mode agent still warns and falls back to standard mode. Refs: .agent-runs/2026-07-13-223741-harness-refactor/reports/01-stop-the-bleeding.md
- Introduce src/review/ with typed + Zod-validated ReviewRequest/Result contracts, boundary helpers, budget defaults/enforcement, model-call selection, ReviewExecutor interface, and a PromptFile request builder - Every external shape has a paired strict Zod schema; legacy scoring-mode, rubric, and model-authored rule-override fields are rejected - modelCall is single | agent | auto; chooseModelCall() resolves auto - On-page boundary (target + caller context only) via buildScope/isInScope with pure lexical URI normalization; no filesystem reads - BudgetExceededError extends the repository VectorlintError base - Purely additive: no changes outside src/review/ and tests/review/ - tests/review/ covers surface, schemas, budget, boundary, results, model-call selection, and the request builder (46 tests)
- Add src/findings/finding-evidence-verifier.ts wrapping locateQuotedText from src/output/location.ts as the single source of truth (audit #6). - Anchored quotes return verified line/column/match; unanchored quotes return a finding-evidence-not-locatable warn diagnostic and no finding. - Never falls back to a model-provided line number the way the old agent path did. - Cover anchored, unanchored, no-line-hint, and truncation cases.
- Add src/findings/types.ts with the FindingProcessingInput contract and strict Zod schemas; no modelCall/mode/agent/evaluator/judge/rubric fields, so legacy judge/evaluator-shaped input is rejected. - Add src/findings/severity.ts: resolveSeverity derives severity from the calculateCheckScore result (one path, no agent stamping); buildRuleId matches Pack.Rule[.Criterion] naming without importing src/agent/. - Add src/findings/scorer.ts as a thin wrapper over calculateCheckScore; the verified finding count drives the score. - Cover schema parse/reject, severity, rule-id, criterion resolution, and score numerics against calculateCheckScore.
- Add src/findings/processor.ts: processFindings runs the single pipeline (filter -> verify -> dedupe -> score -> severity -> diagnostics) and returns the Phase 2 ReviewResult from src/review/types.ts. - Verified finding count, not raw candidate count, drives the score; unanchored evidence becomes a warn diagnostic and emits no finding; hadOperationalErrors is true only for error-level diagnostics. - Add src/findings/index.ts barrel export. - Golden processor test asserts findings/score match the standard check path; cover diagnostics, filtering, verified-count fix, dedupe, rule-id attribution, and ReviewResult contract validation.
- Build FindingProcessingInput and call processFindings in the standard objective-check branch of routePromptResult - Feed returned findings/scores/diagnostics through the existing line/json/rdjson/vale sinks and quality-score output - Remove inline filter/score/severity/quote-location logic from the check path; computeFilterDecision kept only for the debug-run artifact - Verified finding count now drives counts and score (audit Finding #6); unanchored quotes become warn diagnostics and no longer flag operational errors - Add orchestrator regression tests covering finding/score output, JSON/Vale shape, and exit behavior for supported objective-check reviews
- PROMPT_META_SCHEMA now rejects `type: judge` and its deprecated alias `subjective` via a superRefine, so judge-typed prompts fail to load with a descriptive warning instead of reaching evaluation. - superRefine (not refine) keeps the inferred union including "judge" so the legacy BaseEvaluator type-checks unchanged until Phase 4 deletes that path. - Add loader tests covering judge/subjective rejection and that check still loads; subjective and semi-objective are documented as deprecated aliases.
- Drop the judge branch of routePromptResult and the now-dead criterion extraction/reporting helpers (extractAndReportCriterion, validateCriteriaCompleteness, validateScores, locateAndReportViolations, buildRuleName); subjective rubric scoring is not a future-facing review type. - A JudgeResult that still reaches the orchestrator is refused explicitly (operational error, no findings) rather than misprojected as a check result; no adapter is added for old judge metadata. - Standard objective-check orchestration still routes through processFindings unchanged. - Replace the judge-filtering orchestrator test with a judge-rejection test.
- Drop ProcessViolationsParams, ProcessCriterionParams, ProcessCriterionResult, and ValidationParams from src/cli/types.ts. They were orphaned when the orchestrator's inline violation/criterion processing moved to src/findings (Tasks 2-3) and the judge path was removed, and are referenced nowhere else. - Remove the now-unused imports PromptMeta, PromptCriterionSpec, ScoreComponent, and JudgeResult. Severity and PromptEvaluationResult are retained (still used by ReportIssueParams / ProcessPromptResultParams).
- Add tests/findings/README.md mapping the behavior moved into src/findings (filtering, evidence verification, density scoring, rule-id/severity, formatter routing) and the intentionally removed judge/rubric and agent behavior, with agent tests marked out of scope for Phase 3 (Phase 3 Task 8). - Stop describing type: judge/subjective as a valid future-facing prompt type in CREATING_RULES.md, docs/frontmatter-fields.mdx, and docs/style-guide.mdx; note that judge/subjective are now rejected at load. - Drop the stale rubric-based scoring feature blurb from README.md.
- Map meta.criteria to { id, name } before processFindings so legacy
PromptCriterionSpec rubric fields (weight, target) cannot cross the
standard check orchestration -> findings contract
- Add tests/orchestrator-finding-criteria.test.ts mocking processFindings
to assert weight/target are stripped on the production path
- prompt-loader criteria support is unchanged; criterion id/name still
drive output rule-id attribution
|
Warning Review limit reached
Next review available in: 36 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughVectorLint replaces check/judge evaluation branching with unified LLM evaluation results, shared evidence verification and finding processing, density-based scoring, updated CLI routing, and revised schemas, documentation, presets, and tests. ChangesUnified evaluation and findings pipeline
Possibly related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- README: replace the "Agent Mode (under review)" deprecation section with "Internal Agent-Mode Implementation" framing (unreleased/internal path, no public compatibility contract) - orchestrator: reword the quarantine comment and the --mode agent runtime warning to unreleased/internal framing; fallback behavior unchanged - CLI --help: reword the --mode option text away from "deprecated" - findings module/verifier + tests: replace old/legacy/surface/compatibility wording with unreleased/internal framing; technical assertions unchanged - orchestrator-agent-output tests: update warning assertions to the new text (forced by the sanctioned warning reword) and align labels/comments No public deprecation, migration, or compatibility obligation is implied.
- Keep agent mode warnings product-facing and internal-only\n- Remove obsolete audit and plan assertions from the correction scope
- Describe agent mode as an unreleased internal implementation path\n- Keep the standard-mode fallback notice focused on bounded harness rework\n- Remove the audit artifact and align orchestrator coverage with the notice
- Replace audit and phase references with stable review contract wording\n- Preserve review behavior and integration semantics\n- Validate review tests and repository verification
…ng' into codex/feat/harness-review-contract
…view-contract # Conflicts: # README.md # src/cli/commands.ts # src/cli/orchestrator.ts # tests/orchestrator-agent-output.test.ts
# Conflicts: # README.md # src/cli/commands.ts # src/cli/orchestrator.ts # tests/orchestrator-agent-output.test.ts
oshorefueled
changed the base branch from
codex/feat/harness-review-contract
to
main
July 22, 2026 22:46
…s-finding-processing # Conflicts: # src/review/README.md # src/review/boundary.ts # src/review/budget.ts # src/review/executor.ts # src/review/index.ts # src/review/request-builder.ts # src/review/schemas.ts # src/review/types.ts # tests/review/budget.test.ts # tests/review/request-builder.test.ts # tests/review/types.test.ts
- Make violation finding and density scoring the sole evaluation path - Ignore legacy type frontmatter and remove mode-specific rule metadata - Remove rubric documentation, planning artifacts, and obsolete tests
oshorefueled
marked this pull request as ready for review
July 22, 2026 23:51
Contributor
Author
|
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
CREATING_RULES.md (1)
103-108: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAlign CREATING_RULES.md with the supported strictness location.
The current docs show strictness configured via
RuleID.strictnessin.vectorlint.ini, whileCREATING_RULES.mdsays it can be set in prompt frontmatter without documenting a supported frontmatter key. Use the supported config form or addstrictnessto the frontmatter reference if that key is valid.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@CREATING_RULES.md` around lines 103 - 108, Update the “Strictness Levels” section in CREATING_RULES.md to match the supported configuration interface: document strictness under RuleID.strictness in .vectorlint.ini, or explicitly add the supported strictness frontmatter key to the frontmatter reference if it is valid. Remove the unsupported or ambiguous claim that strictness is configured in prompt frontmatter.tests/scoring-types.test.ts (1)
30-51: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winFull EvaluationLLMResult mock shape is expected here.
LLMResulttakesdata: T, so theseEvaluationLLMResultmocks needreasoning; the first violation list also needs eachEvaluationLLMResult["violations"][number]field (line,message,suggestion,fix,rule_quote,checks,check_notes,confidence). The empty-array mock also omitsreasoning.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/scoring-types.test.ts` around lines 30 - 51, Update the mockLlmResponse EvaluationLLMResult fixtures in this test to include the required reasoning field and complete each violations entry with the EvaluationLLMResult violation fields line, message, suggestion, fix, rule_quote, checks, check_notes, and confidence, while preserving the existing test values where applicable. Also add reasoning to the empty-array mock.
🧹 Nitpick comments (2)
src/prompts/schema.ts (1)
170-196: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
violationsduplicatesEvaluationItem's shape instead of extending it.The inline
violationsitem type re-lists every field already present onEvaluationItem(pluscriterionName). Any future change toEvaluationItem(add/remove/rename a field) won't propagate here automatically, risking silent type drift between the two parallel shapes.♻️ Derive `violations` from `EvaluationItem`
items: Array<EvaluationItem>; severity: typeof Severity.WARNING | typeof Severity.ERROR; message: string; reasoning?: string; - violations: Array<{ - line?: number; - analysis: string; - message?: string; - suggestion?: string; - fix?: string; - quoted_text?: string; - context_before?: string; - context_after?: string; - criterionName?: string; - description?: string; - rule_quote?: string; - checks?: GateChecks; - check_notes?: GateCheckNotes; - confidence?: number; - }>; + violations: Array<EvaluationItem & { criterionName?: string }>;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/prompts/schema.ts` around lines 170 - 196, Update ScoredEvaluation.violations to derive each item from EvaluationItem instead of duplicating its fields, while preserving the additional criterionName field. Use the existing EvaluationItem type as the base so future changes propagate automatically.src/evaluators/base-evaluator.ts (1)
93-105: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winChunk evaluation runs sequentially; consider parallelizing.
Each chunk's LLM call is awaited before the next chunk starts. For documents large enough to trigger chunking, this serializes N LLM round-trips instead of overlapping them.
♻️ Run chunk calls concurrently
- for (const chunk of chunks) { - const { data: llmResult, usage } = - await this.llmProvider.runPromptStructured<EvaluationLLMResult>( - chunk.content, - this.prompt.body, - schema, - context - ); - allChunkViolations.push(llmResult.violations); - rawChunkOutputs.push(llmResult); - if (llmResult.reasoning) chunkReasonings.push(llmResult.reasoning); - usages.push(usage); - } + const chunkResults = await Promise.all( + chunks.map((chunk) => + this.llmProvider.runPromptStructured<EvaluationLLMResult>( + chunk.content, + this.prompt.body, + schema, + context + ) + ) + ); + for (const { data: llmResult, usage } of chunkResults) { + allChunkViolations.push(llmResult.violations); + rawChunkOutputs.push(llmResult); + if (llmResult.reasoning) chunkReasonings.push(llmResult.reasoning); + usages.push(usage); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/evaluators/base-evaluator.ts` around lines 93 - 105, Update the chunk-processing loop in the evaluator method to initiate all runPromptStructured calls concurrently instead of awaiting each call sequentially. Collect each result’s violations, raw output, reasoning, and usage after the concurrent calls complete, preserving the existing output aggregation behavior and chunk order.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/findings/processor.ts`:
- Around line 131-140: Update the deduplication logic around coords and
dedupeKey to derive the key from the verified match coordinates in
verification.finding, rather than candidate.quoted_text. Ensure candidates
resolving to the same verified anchor share a key and are skipped while
preserving the existing seen-set flow.
In `@src/findings/scorer.ts`:
- Around line 26-40: Handle params.wordCount === 0 in the scoring flow before
calling calculateScore, returning the defined no-content score and corresponding
scoreText/severity without producing NaN. Update the function containing the
shown scored calculation, while preserving the existing calculateScore path for
positive word counts.
In `@src/prompts/target.ts`:
- Line 13: Align validatePrompt’s regex construction with checkTarget by using
the same documented default flags, “mu”, whenever target.flags is omitted. Reuse
a shared default-flags symbol or contract so validation and execution
consistently apply identical regex semantics.
---
Outside diff comments:
In `@CREATING_RULES.md`:
- Around line 103-108: Update the “Strictness Levels” section in
CREATING_RULES.md to match the supported configuration interface: document
strictness under RuleID.strictness in .vectorlint.ini, or explicitly add the
supported strictness frontmatter key to the frontmatter reference if it is
valid. Remove the unsupported or ambiguous claim that strictness is configured
in prompt frontmatter.
In `@tests/scoring-types.test.ts`:
- Around line 30-51: Update the mockLlmResponse EvaluationLLMResult fixtures in
this test to include the required reasoning field and complete each violations
entry with the EvaluationLLMResult violation fields line, message, suggestion,
fix, rule_quote, checks, check_notes, and confidence, while preserving the
existing test values where applicable. Also add reasoning to the empty-array
mock.
---
Nitpick comments:
In `@src/evaluators/base-evaluator.ts`:
- Around line 93-105: Update the chunk-processing loop in the evaluator method
to initiate all runPromptStructured calls concurrently instead of awaiting each
call sequentially. Collect each result’s violations, raw output, reasoning, and
usage after the concurrent calls complete, preserving the existing output
aggregation behavior and chunk order.
In `@src/prompts/schema.ts`:
- Around line 170-196: Update ScoredEvaluation.violations to derive each item
from EvaluationItem instead of duplicating its fields, while preserving the
additional criterionName field. Use the existing EvaluationItem type as the base
so future changes propagate automatically.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 965040ad-b72d-4840-99f1-09b30ee43aa8
📒 Files selected for processing (78)
AGENTS.mdCONFIGURATION.mdCREATING_RULES.mdREADME.mddocs/best-practices.mdxdocs/customize-style-rules.mdxdocs/docs.jsondocs/false-positive-tuning.mdxdocs/frontmatter-fields.mdxdocs/how-it-works.mdxdocs/initial-style-check.mdxdocs/introduction.mdxdocs/project-config.mdxdocs/rubric-scoring.mdxdocs/style-guide.mdxdocs/use-cases.mdxpresets/VectorLint/ai-pattern.mdpresets/VectorLint/capitalization.mdpresets/VectorLint/consistency.mdpresets/VectorLint/inclusivity.mdpresets/VectorLint/passive-voice.mdpresets/VectorLint/repetition.mdpresets/VectorLint/unsupported-claims.mdpresets/VectorLint/wordiness.mdsrc/agent/executor.tssrc/chunking/merger.tssrc/cli/orchestrator.tssrc/cli/types.tssrc/config/constants.tssrc/debug/run-artifact.tssrc/evaluators/accuracy-evaluator.tssrc/evaluators/base-evaluator.tssrc/evaluators/types.tssrc/findings/finding-evidence-verifier.tssrc/findings/index.tssrc/findings/processor.tssrc/findings/scorer.tssrc/findings/severity.tssrc/findings/types.tssrc/output/json-formatter.tssrc/prompts/prompt-validator.tssrc/prompts/schema.tssrc/prompts/target.tssrc/review/schemas.tssrc/review/types.tssrc/schemas/prompt-schemas.tssrc/scoring/index.tssrc/scoring/scorer.tstests/agent/agent-executor.test.tstests/evaluations/test-rules/Test/clarity.mdtests/evaluations/test-rules/Test/consistency.mdtests/evaluations/test-rules/Test/passive-voice.mdtests/evaluations/test-rules/Test/readability.mdtests/evaluations/test-rules/Test/wordiness.mdtests/evaluator.test.tstests/findings/finding-evidence-verifier.test.tstests/findings/module-surface.test.tstests/findings/processor.test.tstests/findings/scorer.test.tstests/findings/severity.test.tstests/findings/types.test.tstests/fixtures/capitalization/rules/capitalization-rules/capitalization.mdtests/fixtures/consistency/rules/consistency-rules/consistency.mdtests/fixtures/inclusivity/rules/inclusivity-rules/inclusivity.mdtests/fixtures/passive-voice/rules/passive-voice-rules/passive-voice.mdtests/fixtures/repetition/rules/repetition-rules/repetition.mdtests/fixtures/unsupported-claims/rules/unsupported-claims-rules/unsupported-claims.mdtests/fixtures/wordiness/rules/wordiness-rules/wordiness.mdtests/orchestrator-agent-output.test.tstests/orchestrator-filtering.test.tstests/orchestrator-finding-processor.test.tstests/prompt-loader-validation.test.tstests/prompt-schema.test.tstests/review/request-builder.test.tstests/review/types.test.tstests/scoring-types.test.tstests/target.test.tstests/validator.test.ts
💤 Files with no reviewable changes (20)
- docs/rubric-scoring.mdx
- tests/evaluations/test-rules/Test/wordiness.md
- tests/evaluations/test-rules/Test/passive-voice.md
- src/debug/run-artifact.ts
- tests/fixtures/inclusivity/rules/inclusivity-rules/inclusivity.md
- tests/evaluations/test-rules/Test/clarity.md
- presets/VectorLint/consistency.md
- presets/VectorLint/ai-pattern.md
- tests/evaluations/test-rules/Test/consistency.md
- src/review/schemas.ts
- src/evaluators/types.ts
- src/review/types.ts
- docs/best-practices.mdx
- tests/review/request-builder.test.ts
- tests/fixtures/consistency/rules/consistency-rules/consistency.md
- tests/evaluations/test-rules/Test/readability.md
- presets/VectorLint/inclusivity.md
- src/output/json-formatter.ts
- src/prompts/prompt-validator.ts
- tests/agent/agent-executor.test.ts
- Deduplicate candidates by verified anchors and normalize empty targets - Share target regex defaults between validation and execution - Align strictness docs, evaluation types, and typed test fixtures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Objective check results were still processed inside the CLI orchestrator, which made evidence anchoring, filtering, scoring, and formatter output harder to reuse safely. This PR creates one shared findings path for the harness refactor.
What this PR covers
--mode agentpath from invoking the autonomous workspace implementation by falling back to standard evaluation.Scope
In scope
src/findings/processor, evidence verifier, scoring, severity/rule-id helpers, and strict input contract.src/cli/orchestrator.ts.type: judge/type: subjective.Out of scope
src/agent/*.Behavior impact
type: judgeandtype: subjectiveprompts are rejected at load.--mode agentremains an unreleased/internal implementation path and now falls back to standard evaluation with an internal-status warning.Risk
Future improvements
BaseEvaluatorjudge internals,buildJudgeLLMSchema, old judge scoring tests, and unreleased/internal agent-mode implementation paths.docs/rubric-scoring.mdxstub and nav entry in a later docs cleanup if rubric scoring is permanently retired.Known tradeoffs
id/nameonly for rule-id attribution; rubricweightandtargetmetadata are stripped before findings processing.BaseEvaluatorjudge internals remain in place for Phase 4 rather than being deleted in this PR.How to test / verify
Checks run
npm run test:run -- tests/orchestrator-agent-output.test.ts tests/orchestrator-check-processor.test.tsnpm run test:run -- tests/orchestrator-finding-criteria.test.ts tests/orchestrator-check-processor.test.ts tests/findings/types.test.tsnpm run verifyManual verification
tests/findings/README.mdfor the behavior mapping.type: judgeprompt load path and confirm it is skipped with a validation warning.Rollback
src/findings/and reject judge prompt types.Summary by CodeRabbit
New Features
Documentation