Skip to content

feat: v3 improvements — analysis, detection, CLI, release, and reproducibility#100

Merged
pmclSF merged 5 commits intomainfrom
feat/v3-improvements
Mar 18, 2026
Merged

feat: v3 improvements — analysis, detection, CLI, release, and reproducibility#100
pmclSF merged 5 commits intomainfrom
feat/v3-improvements

Conversation

@pmclSF
Copy link
Copy Markdown
Owner

@pmclSF pmclSF commented Mar 18, 2026

Summary

Comprehensive v3 feature branch delivering analysis depth, engine robustness, CLI polish, and enterprise-ready release infrastructure.

Analysis Layer

  • Fixture detection across JS/Python/Go/Java with depgraph integration
  • Environment matrix parsing for GitHub Actions, GitLab CI, CircleCI, Buildkite
  • Symbol-level coverage via Go AST, JS import parsing, Python from-import matching
  • AST-enhanced AI detection with multi-framework patterns (Vercel AI SDK, DSPy, Mirascope, Marvin, Instructor)
  • Structured RAG pipeline parser with 9-component detection and config extraction
  • FileCache for thread-safe content + Go AST caching with prewarm
  • Context cancellation throughout the analysis layer for large-repo timeout support
  • Extraction dedup — consolidated 4 pairs of near-identical export extractors

Detection Model & Engine

  • 44 DetectorID constants, tier utilities, ConfidenceBasis (calibrated vs heuristic)
  • 9 canonical AI capabilities with surface-to-capability inference
  • Safe registry initialization — DefaultRegistry returns error instead of panicking
  • Artifact auto-discovery across 29 known paths for coverage/runtime files
  • Guided init workflow with config generation
  • Key Findings in analyze output (replaces single TopInsight)
  • Confidence calibration via precision/recall measurement

CLI

  • main.go split from 3400+ lines into 9 focused command files
  • Structured logging via log/slog with --log-level quiet|debug
  • CLI naming deconfliction — Go binary: terrain, npm binary: terrain-convert

Release Pipeline

  • SBOM generation — CycloneDX + SPDX per release archive via syft
  • Sigstore signing hooks for keyless cosign (checksums.txt)
  • Supply-chain docs with verification instructions

npm Reproducibility

  • Production deps pinned to exact versions (5 packages)
  • .npmrc with save-exact=true for future additions
  • CI lockfile-integrity check after npm ci
  • Dependabot strategy tightened to increase with grouped PRs

Test plan

  • All 40+ Go packages pass (go test ./internal/... ./cmd/...)
  • All 730 JS test suites pass (2274 tests)
  • GoReleaser config validates (goreleaser check)
  • npm ci works cleanly with updated lockfile
  • New equivalence tests prove file-based and content-based extraction paths match
  • 8 new registry error-handling tests across engine and measurement packages
  • 6 logging tests covering level filtering, structured attributes, global init

🤖 Generated with Claude Code

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 18, 2026 •
edited
Loading

[RISK] Terrain — Merge with caution

High-severity gaps found in changed code.

Metric Value
Changed files 202 (78 source, 102 test)
Impacted units 410
Protection gaps 174
Tests to run 139 of 2528 (5% of suite)

New Risks (directly changed)

  • [MED] bin/terrain-compat.js: terrain-compat.js has no observed test coverage.
  • [MED] bin/terrain.js: terrain.js has no observed test coverage.
  • [MED] cmd/terrain-bench/main.go: main.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_ai.go: cmd_ai.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_analyze.go: cmd_analyze.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_compare.go: cmd_compare.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_debug.go: cmd_debug.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_explain.go: cmd_explain.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_impact.go: cmd_impact.go has no observed test coverage.
  • [MED] cmd/terrain/cmd_insights.go: cmd_insights.go has no observed test coverage.
  • ... and 164 more (157 high, 7 medium)
Pre-existing issues on changed files (5)
  • cmd/terrain/ai_workflow_test.go: [staticSkippedTest] 13 of 14 tests statically skipped (93%) in cmd/terrain/ai_workflow_test.go.
  • internal/analysis/code_surface_test.go: [staticSkippedTest] 2 of 40 tests statically skipped (5%) in internal/analysis/code_surface_test.go.
  • internal/analysis/content_analysis_test.go: [staticSkippedTest] 3 of 16 tests statically skipped (19%) in internal/analysis/content_analysis_test.go.
  • internal/impact/impact_test.go: [staticSkippedTest] 1 of 67 tests statically skipped (1%) in internal/impact/impact_test.go.
  • internal/truthcheck/checker_test.go: [staticSkippedTest] 1 of 10 tests statically skipped (10%) in internal/truthcheck/checker_test.go.

Recommended Tests

139 test(s) with exact coverage of 200 impacted unit(s). 210 impacted unit(s) have no covering tests in the selected set.

Package Tests Sample
internal/analysis 24 internal/analysis/ai_context_infer_test.go ...
internal/engine 9 internal/engine/adversarial_test.go ...
internal/depgraph 8 internal/depgraph/bench_test.go ...
internal/testdata 8 internal/testdata/adversarial_test.go ...
internal/quality 7 internal/quality/coverage_blind_spot_test.go ...
internal/reporting 7 internal/reporting/analyze_report_test.go ...
internal/models 6 internal/models/detection_test.go ...
cmd/terrain 3 cmd/terrain/ai_workflow_test.go ...
internal/analyze 3 internal/analyze/analyze_golden_test.go ...
internal/impact 3 internal/impact/changeset_builder_test.go ...
internal/migration 3 internal/migration/detectors_test.go ...
tests/fixtures/ai-rag-pipeline/src/rag 3 tests/fixtures/ai-rag-pipeline/src/rag/chunking.ts ...
internal/changescope 2 internal/changescope/changescope_test.go ...
internal/explain 2 internal/explain/explain_golden_test.go ...
internal/insights 2 internal/insights/insights_golden_test.go ...
internal/measurement 2 internal/measurement/measurement_test.go ...
internal/ownership 2 internal/ownership/aggregate_test.go ...
internal/signals 2 internal/signals/detector_registry_test.go ...
internal/stability 2 internal/stability/cluster_test.go ...
internal/truthcheck 2 internal/truthcheck/calibration_test.go ...
tests/fixtures/ai-prompt-only/tests/eval/safety 2 tests/fixtures/ai-prompt-only/tests/eval/safety/content-safety.test.ts ...
internal/aidetect 1 internal/aidetect/detect_test.go
internal/airun 1 internal/airun/artifact_test.go
internal/benchmark 1 internal/benchmark/export_test.go
internal/comparison 1 internal/comparison/compare_test.go
internal/gauntlet 1 internal/gauntlet/ingest_test.go
internal/governance 1 internal/governance/evaluate_test.go
internal/graph 1 internal/graph/graph_test.go
internal/heatmap 1 internal/heatmap/heatmap_test.go
internal/lifecycle 1 internal/lifecycle/lifecycle_test.go
internal/logging 1 internal/logging/logging_test.go
internal/matrix 1 internal/matrix/matrix_test.go
internal/metrics 1 internal/metrics/metrics_test.go
internal/plugin 1 internal/plugin/plugin_test.go
internal/policy 1 internal/policy/loader_test.go
internal/portfolio 1 internal/portfolio/portfolio_test.go
internal/scoring 1 internal/scoring/risk_engine_test.go
internal/summary 1 internal/summary/executive_test.go
tests/fixtures/ai-mixed-traditional/src/ai/contexts 1 tests/fixtures/ai-mixed-traditional/src/ai/contexts/policy.ts
tests/fixtures/ai-mixed-traditional/src/ai/prompts 1 tests/fixtures/ai-mixed-traditional/src/ai/prompts/support.ts
tests/fixtures/ai-mixed-traditional/src/api 1 tests/fixtures/ai-mixed-traditional/src/api/routes.ts
tests/fixtures/ai-mixed-traditional/src/auth 1 tests/fixtures/ai-mixed-traditional/src/auth/login.ts
tests/fixtures/ai-mixed-traditional/tests/eval/safety 1 tests/fixtures/ai-mixed-traditional/tests/eval/safety/support-safety.test.ts
tests/fixtures/ai-mixed-traditional/tests/unit/api 1 tests/fixtures/ai-mixed-traditional/tests/unit/api/routes.test.ts
tests/fixtures/ai-mixed-traditional/tests/unit/auth 1 tests/fixtures/ai-mixed-traditional/tests/unit/auth/login.test.ts
tests/fixtures/ai-prompt-only/src 1 tests/fixtures/ai-prompt-only/src/utils.ts
tests/fixtures/ai-prompt-only/src/contexts 1 tests/fixtures/ai-prompt-only/src/contexts/system.ts
tests/fixtures/ai-prompt-only/src/prompts 1 tests/fixtures/ai-prompt-only/src/prompts/chat.ts
tests/fixtures/ai-prompt-only/src/safety 1 tests/fixtures/ai-prompt-only/src/safety/guardrails.ts
tests/fixtures/ai-prompt-only/tests/eval/accuracy 1 tests/fixtures/ai-prompt-only/tests/eval/accuracy/qa-accuracy.test.ts
tests/fixtures/ai-prompt-only/tests/unit 1 tests/fixtures/ai-prompt-only/tests/unit/utils.test.ts
tests/fixtures/ai-rag-pipeline/src/agent 1 tests/fixtures/ai-rag-pipeline/src/agent/router.ts
tests/fixtures/ai-rag-pipeline/src/prompts 1 tests/fixtures/ai-rag-pipeline/src/prompts/system.ts
tests/fixtures/ai-rag-pipeline/src/tools 1 tests/fixtures/ai-rag-pipeline/src/tools/search.ts
tests/fixtures/ai-rag-pipeline/tests/eval/accuracy 1 tests/fixtures/ai-rag-pipeline/tests/eval/accuracy/qa-accuracy.test.ts
tests/fixtures/ai-rag-pipeline/tests/eval/retrieval 1 tests/fixtures/ai-rag-pipeline/tests/eval/retrieval/retrieval-quality.test.ts
tests/fixtures/ai-rag-pipeline/tests/eval/safety 1 tests/fixtures/ai-rag-pipeline/tests/eval/safety/tool-safety.test.ts
tests/fixtures/ai-rag-pipeline/tests/unit 1 tests/fixtures/ai-rag-pipeline/tests/unit/rag.test.ts

Owners: pmclachlansf

Terrain — terrain pr --json for full machine-readable results

Targeted Test Results

Terrain selected 139 test(s) instead of the full suite.

  • Go tests: passed

pmclSF and others added 5 commits March 17, 2026 17:37
@pmclSF @claude
feat(analysis): AI detection, surface taxonomy, and inference archite…
95b96c2 
…cture

Analysis layer:
- Tiered inference architecture with structural > semantic > pattern > content
- AST-enhanced prompt detection with multi-framework patterns (Vercel AI SDK,
  DSPy, Mirascope, Marvin, Instructor)
- Structured RAG pipeline parser with 9-component detection and config extraction
- Content-based AI context inference beyond naming conventions
- Tool schema and output contract detection
- Bracket-aware structural parser for prompt/context detection
- Source-to-source import graph with real dependency edges
- Capability inference from surfaces (9 canonical AI capabilities)

Models:
- DetectionEvidence schema with tier, confidence, and basis metadata
- 44 DetectorID constants and tier utilities
- 9 canonical AICapability constants
- RAGPipelineSurface model with 9 component kinds
- Expanded AI surface taxonomy (4 new surface kinds + Context kind)
- 14 AI-native signal types with scenario-aware classification

Dependency graph:
- AI graph nodes and capability layer
- Deterministic map iteration in graph construction

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pmclSF @claude
feat(engine): AI signals, governance, impact, and validation pipeline
f3374bb 
Signals & governance:
- 6 RAG failure mode signals with classifier support
- AI CI policy with blocking, warning, and coverage rules
- AI-specific protection gaps for uncovered AI surfaces

Impact & changescope:
- Surface-kind-aware AI scenario impact with descriptive reasons
- First-class AI validation section in PR summaries
- 3-way finding classification and triage-oriented PR comments
- Finding deduplication in insights, impact, and changescope

CLI & explain:
- Rich AI scenario explain with surface breakdown and signals
- AI inventory view with capabilities and gap analysis
- Complete terrain ai run as CI entry point for AI validation
- Replay and determinism support for AI validation artifacts

Fixtures:
- 3 AI benchmark fixtures with ground truth (ai-mixed-traditional,
  ai-prompt-only, ai-rag-pipeline)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pmclSF @claude
feat: CLI refactor, infrastructure hardening, and engine safety
aef76c6 
CLI refactor:
- Split main.go (3400+ lines) into 9 focused command files
- Structured logging via log/slog with --log-level quiet|debug
- Pipeline diagnostics migrated from fmt.Fprintf(os.Stderr) to slog

Analysis infrastructure:
- FileCache: thread-safe content + Go AST cache with PrewarmSourceFiles
- Context cancellation throughout analysis layer (AnalyzeContext, walkDirCtx)
- Fixture detection across JS/Python/Go/Java with depgraph integration
- Environment matrix parsing (GitHub Actions, GitLab CI, CircleCI, Buildkite)
- Symbol-level coverage via Go AST, JS imports, Python from-imports
- Consolidated 4 pairs of duplicated export extractors

Engine safety:
- DefaultRegistry returns (*Registry, error) instead of panicking
- MustRegister deprecated in detector and measurement registries
- Artifact auto-discovery across 29 known paths
- Guided init workflow with config generation
- Key Findings in analyze output replacing single TopInsight
- Confidence calibration via CalibrateFromFixtures

Quality:
- Static skip detection for test files
- Plugin architecture with 4 extension points
- Schema versioning for all artifact types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pmclSF @claude
feat(release): SBOM generation, Sigstore signing, and CLI naming
8d05d11 
Release pipeline:
- SBOM generation via GoReleaser + syft (CycloneDX + SPDX per archive)
- Sigstore cosign signing hooks for keyless checksum signing
- Makefile targets: make sbom, make release-dry-run
- Supply-chain documentation with verification instructions

CLI naming deconfliction:
- Go binary: terrain (primary)
- npm binary: terrain-convert (primary), terrain (compat shim with warning)

Docs:
- Product excellence scorecard and gap audit
- Story-to-implementation traceability matrix
- Feature and persona matrices

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pmclSF @claude
chore(npm): pin production deps and enforce lockfile integrity
8061ede 
Production dependencies pinned to exact versions (5 packages):
- chalk 5.6.2, commander 14.0.3, fast-glob 3.3.3,
  pixelmatch 7.1.0, pngjs 7.0.0
- Dev dependencies retain caret ranges (Dependabot manages weekly)

Reproducibility enforcement:
- .npmrc: save-exact=true for future additions
- CI: lockfile sync verification step after npm ci
- Dependabot: versioning-strategy "increase" with grouped PRs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pmclSF pmclSF force-pushed the feat/v3-improvements branch from 554547b to 8061ede Compare March 18, 2026 00:39
@pmclSF pmclSF merged commit 29a296c into main Mar 18, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pmclSF