taintline

A deterministic, local-first taint & reliability CI gate over OpenTelemetry GenAI spans. No LLM judge in the check path — same inputs, same verdict, every run. Usable as a hard gate.

Writeup: A security check with no model in it — why an LLM judge is the wrong tool to gate a build on, and what a deterministic taint-flow check does instead.

The hero: CI catches an exfiltration

An untrusted web_fetch output flows into send_email with no sanitizer between them. taintline reads the run's OTel trace and fails the build:

$ uv run taintline check demo/traces/taint_flow.otlp.json --fail-on taint
ERROR taint: untrusted output from 'web_fetch' reaches sink 'send_email' without sanitization (segment: 'ACME_API_KEY=sk-live-7f3a92c4d1e5f6') [web-fetch-1,send-email-1]
[taintline] 1 finding(s)
$ echo $?
1

Non-zero exit → CI turns red. No model call, no network — a rule fired on a data flow.

Quickstart

uv sync
uv run taintline check demo/traces/taint_flow.otlp.json --fail-on taint,loops

That one command reproduces the hero above. Point check at any OTel GenAI spans export (or a Strands AgentResult export); --fail-on picks which detectors turn a finding into a non-zero exit.

Detectors (all deterministic, no LLM)

Detector	Fires when
taint	Untrusted tool output (web/email/file read) reaches a sensitive sink (send/delete/payment/shell) with no sanitizer between.
loops	Same tool name + identical args repeated N times.
retry-storm	K consecutive same-tool calls with ≥half ERROR in a causal window.
swallowed-error	A tool returns ERROR and the agent ignores it and proceeds past.
context-truncation	LLM output truncated by a length/token limit (read from gen_ai.* attributes).
budget-overrun	Cumulative token usage or tool-call count exceeds a configured budget.

Determinism + zero-LLM check path

The check path is pure stdlib, does no network I/O, and produces byte-identical verdicts on identical input. Proven, not asserted — see tests/test_determinism.py:

• verdict is byte-identical over 10 repeated runs;
• the socket layer is monkeypatched to raise, and check still runs — zero network calls;
• CLI output is identical across five PYTHONHASHSEED values (subprocess, fresh seed each) — no set-iteration-order leak.

Same trace in → same verdict out. That is what makes it safe as a hard gate.

Producer-agnostic

The core runs on any OTel GenAI span tree. Producer independence is table stakes here — Strands' own ingest already ships mappers for multiple producers (LangChain, OpenInference). taintline stands on the same conventions and proves it on a trace from a different stack: a Claude Agent SDK run (mcp__filesystem__read_file → mcp__shell__run_command) is checked as a portability fixture, no Strands involved — tests/fixtures/portability/.

GitHub Action

Gate any repo on a captured agent trace:

- uses: ./
  with:
    trace: demo/traces/taint_flow.otlp.json
    fail-on: taint,loops

action.yml inputs: trace (path to the OTel/Strands export) and fail-on (comma-separated detectors; default taint,loops). A non-zero exit fails the step. Full runnable CI gate: demo/.

Output formats & GitHub code scanning

check takes --format {text,json,sarif} (default text, byte-identical to before). json is a stable, sorted view of the findings for scripting; sarif emits SARIF 2.1.0 for GitHub's Security tab. Every format is deterministic and LLM-free — the same trace yields byte-identical bytes across runs.

Wire the findings into code scanning with github/codeql-action/upload-sarif:

- name: taintline (SARIF)
  run: uv run taintline check trace.otlp.json --format sarif > taintline.sarif
  continue-on-error: true          # still upload even when the gate trips
- name: Upload to code scanning
  if: always()
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: taintline.sarif

check still exits non-zero on a --fail-on hit regardless of format, so the build can both surface findings in the Security tab and gate on them.

vs Strands Evals (read this first)

Strands Evals scores agent quality and diagnoses failures; its detectors are LLM judges (default Bedrock Claude Sonnet 4.6, ConfidenceLevel threshold) — great for experimentation, costly and non-deterministic as a merge gate. Its deterministic layer is evaluators (exact-match / tool-called / state-equals assertions). taintline is complementary: a deterministic detector with an information-flow (taint) lens — a source→sink security check neither its evaluators nor its detectors model — byte-reproducible and usable as a hard CI gate. They meet at the same OTel GenAI span tree, so the taint detector could live inside Strands rather than compete with it.

Learn more: docs/vs-strands-notes.md — every claim above tied to a class/constant/doc in the real strands-agents/evals.

License

Apache-2.0 (matches the Strands ecosystem).