feat: add lightweight provenance profiler#58
Conversation
|
Update: advanced provenance runtime canary from whole-body exact artifact reuse to exact fenced sub-artifact reuse inside/cross tool_result and assistant_context bodies. Safety gates preserved: default-off/kill-switch, mutable artifact types only, exact closed triple-backtick fenced blocks only, backward-only references, never-grow, privacy-safe salted refs/metadata. Validation: targeted 90 passed; artifact trace gate PASS 17/17; full suite 585 passed / 37 skipped. |
|
Update: added Level-2 declared source-span provenance artifact backrefs. This is metadata-driven, not hash-discovered: optional span_links declare tool_result[source_start:source_end] -> assistant_context[target_start:target_end]. Canary still verifies byte equality, backward-only source_index < target_index, tool_result->assistant_context scope, line alignment, min size, and never-grow before replacing the target span with a salted tool_result#span ref. Runtime adapter consumes/strips contextpilot_span_links before outbound. Safety review passed after fixes for metadata leakage/source preservation/duplicate links. Validation: targeted 94 passed; synthetic source-span trace gate PASS with 1 block / 705 chars saved; artifact trace gate PASS 17/17; full suite 589 passed / 37 skipped. |
|
Update: Level 2 source-span provenance is now in PR #58. Adds declared source-span backrefs: untrusted span metadata can replace a parent assistant_context span only when it points backward to an earlier tool_result, both endpoints are line-aligned, target bytes equal source bytes, reference is shorter, and source remains preserved for resolvability. Also fixed the validation gate false-fail for spans ending in ']' by validating mutation scope from declared offsets instead of greedy prefix/suffix inference. Validation: source-span RED reproduced artifact_mutation_scope false failure, now green; targeted 95 passed; artifact trace gate PASS 17/17; full suite 590 passed / 37 skipped. Claude Code read-only review: PASS, no blockers. |
|
Update: added Level-2 style declared source-span provenance path on top of exact whole/fenced reuse. This is now genuinely provenance-structured beyond block exact matching: runtime can accept declared tool_result -> assistant_context span links and replace an assistant parent span with a backref to the earlier tool span, while still requiring source slice == target slice for safety. Guards: source earlier than target, bounds checked, line-aligned spans, tool_result source only, assistant_context target only, exact slice equality, never-grow, span source preserved from whole-body replacement, declared-offset validation, privacy-safe salted refs/counters. Validation: Claude Code read-only safety review PASS; targeted 58 passed; artifact trace gate PASS 17/17; full suite 590 passed / 37 skipped. Installed local Hermes plugin at 9f40523. |
|
Update: added and ran a reproducible synthetic labeled precision/recall evaluator for artifact canary gates. Scope is explicitly synthetic exact/provenance gate self-consistency, not field/model/product precision. Results on evals/artifact_precision_synthetic_2026-06-19.json: 16 cases, 7 expected replacements, 7 predicted, synthetic_event_precision=1.0, synthetic_event_recall=1.0, synthetic_negative_case_fpr=0.0, synthetic_case_accuracy=1.0, synthetic_realized_chars_saved=2158. Includes positives for whole-body, cross-type, fenced, multi-fenced, declared source-span; negatives for protected content, whole near-duplicate, no declared span, content-different span, forward/OOB span, unterminated fence, never-grow; mode gates off/shadow/disable and forged-ref detection. Verification: new eval tests 2 passed; targeted artifact tests 46 passed; full suite 592 passed / 37 skipped; Claude Code final read-only review PASS. |
Summary
Validation