Skip to content

feat: add lightweight provenance profiler#58

Open
SecretSettler wants to merge 5 commits into
feat/provenance-artifact-reuse-canaryfrom
feat/lightweight-provenance-profiler
Open

feat: add lightweight provenance profiler#58
SecretSettler wants to merge 5 commits into
feat/provenance-artifact-reuse-canaryfrom
feat/lightweight-provenance-profiler

Conversation

@SecretSettler

Copy link
Copy Markdown
Member

Summary

  • keep ContextPilot top-level imports lazy so Hermes monitor/profiler paths do not require SciPy/RAG dependencies
  • add privacy-safe provenance profile rollups by source for token-monitor reporting
  • include the token profile in opportunity JSON/Markdown reports
  • add regression tests for analyzer import without SciPy and privacy-safe profile output

Validation

  • /usr/local/lib/hermes-agent/venv/bin/python scripts/analyze_hermes_context_opportunities.py --state-db /root/.hermes/state.db --telemetry-file /root/.hermes/contextpilot/telemetry.jsonl --out-dir /tmp/contextpilot-profile-test --since-hours 24
  • /usr/local/lib/hermes-agent/venv/bin/python scripts/hermes_contextpilot_monitor.py --out-dir /tmp/contextpilot-monitor-test --since-hours 24 --telemetry-file /root/.hermes/contextpilot/telemetry.jsonl
  • /usr/local/lib/hermes-agent/venv/bin/python scripts/run_trace_validation.py /root/contextpilot/validation_sets/validation_set_2026-06-15-artifact.jsonl --gate artifact --candidate-mode canary --format markdown
  • /usr/local/lib/hermes-agent/venv/bin/python -m pytest tests/test_analyzer_lightweight_provenance.py tests/test_hermes_context_opportunity_analyzer.py tests/test_artifact_dedup_canary.py tests/test_trace_validation_runner.py tests/test_hermes_plugin_patch.py -q
  • /usr/local/lib/hermes-agent/venv/bin/python -m pytest -q

@SecretSettler

Copy link
Copy Markdown
Member Author

Update: advanced provenance runtime canary from whole-body exact artifact reuse to exact fenced sub-artifact reuse inside/cross tool_result and assistant_context bodies. Safety gates preserved: default-off/kill-switch, mutable artifact types only, exact closed triple-backtick fenced blocks only, backward-only references, never-grow, privacy-safe salted refs/metadata. Validation: targeted 90 passed; artifact trace gate PASS 17/17; full suite 585 passed / 37 skipped.

@SecretSettler

Copy link
Copy Markdown
Member Author

Update: added Level-2 declared source-span provenance artifact backrefs. This is metadata-driven, not hash-discovered: optional span_links declare tool_result[source_start:source_end] -> assistant_context[target_start:target_end]. Canary still verifies byte equality, backward-only source_index < target_index, tool_result->assistant_context scope, line alignment, min size, and never-grow before replacing the target span with a salted tool_result#span ref. Runtime adapter consumes/strips contextpilot_span_links before outbound. Safety review passed after fixes for metadata leakage/source preservation/duplicate links. Validation: targeted 94 passed; synthetic source-span trace gate PASS with 1 block / 705 chars saved; artifact trace gate PASS 17/17; full suite 589 passed / 37 skipped.

@SecretSettler

Copy link
Copy Markdown
Member Author

Update: Level 2 source-span provenance is now in PR #58. Adds declared source-span backrefs: untrusted span metadata can replace a parent assistant_context span only when it points backward to an earlier tool_result, both endpoints are line-aligned, target bytes equal source bytes, reference is shorter, and source remains preserved for resolvability. Also fixed the validation gate false-fail for spans ending in ']' by validating mutation scope from declared offsets instead of greedy prefix/suffix inference. Validation: source-span RED reproduced artifact_mutation_scope false failure, now green; targeted 95 passed; artifact trace gate PASS 17/17; full suite 590 passed / 37 skipped. Claude Code read-only review: PASS, no blockers.

@SecretSettler

Copy link
Copy Markdown
Member Author

Update: added Level-2 style declared source-span provenance path on top of exact whole/fenced reuse. This is now genuinely provenance-structured beyond block exact matching: runtime can accept declared tool_result -> assistant_context span links and replace an assistant parent span with a backref to the earlier tool span, while still requiring source slice == target slice for safety. Guards: source earlier than target, bounds checked, line-aligned spans, tool_result source only, assistant_context target only, exact slice equality, never-grow, span source preserved from whole-body replacement, declared-offset validation, privacy-safe salted refs/counters. Validation: Claude Code read-only safety review PASS; targeted 58 passed; artifact trace gate PASS 17/17; full suite 590 passed / 37 skipped. Installed local Hermes plugin at 9f40523.

@SecretSettler

Copy link
Copy Markdown
Member Author

Update: added and ran a reproducible synthetic labeled precision/recall evaluator for artifact canary gates. Scope is explicitly synthetic exact/provenance gate self-consistency, not field/model/product precision. Results on evals/artifact_precision_synthetic_2026-06-19.json: 16 cases, 7 expected replacements, 7 predicted, synthetic_event_precision=1.0, synthetic_event_recall=1.0, synthetic_negative_case_fpr=0.0, synthetic_case_accuracy=1.0, synthetic_realized_chars_saved=2158. Includes positives for whole-body, cross-type, fenced, multi-fenced, declared source-span; negatives for protected content, whole near-duplicate, no declared span, content-different span, forward/OOB span, unterminated fence, never-grow; mode gates off/shadow/disable and forged-ref detection. Verification: new eval tests 2 passed; targeted artifact tests 46 passed; full suite 592 passed / 37 skipped; Claude Code final read-only review PASS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant