From 18dcd5a7d4022537d4e55bd3643242084717fa62 Mon Sep 17 00:00:00 2001 From: Hemanth Chittanuru Date: Thu, 13 Aug 2026 21:00:12 -0400 Subject: [PATCH 01/10] ci: publishing a Release is the gate, not pushing a tag Setting up the repo rules exposed a hole in the release design. The plan was tag-push plus a required reviewer on the `pypi` environment, with the environment as the real gate because it pauses before the OIDC token is minted. Environment protection rules turn out to be unavailable on a PRIVATE repo under this plan -- required reviewers, wait timers and deployment branch policies all rejected with a billing error -- so the environment exists and holds nothing. Tag-push would therefore have published to PyPI with no human in the loop at all. Trigger is now `release: published`. A tag is cheap to create by accident and impossible to retract once it has reached PyPI; creating a Release is deliberate and puts the notes in front of you first. That gate works on any plan. The workflow no longer creates the Release -- the human does, which is the point -- so the third job attaches the built artifacts to it instead. It keeps its own `contents: write` rather than lending it to the job that holds the publishing identity. Environment protection becomes available for free once the repo is public, which has to happen before publishing anyway. Add a required reviewer then and the two gates compose. Release steps written down in CONTRIBUTING, since they are now four and one of them is easy to get wrong. Co-Authored-By: Claude Opus 5 --- .github/workflows/release.yml | 35 +++++++++++++++++------------------ CONTRIBUTING.md | 17 +++++++++++++++++ 2 files changed, 34 insertions(+), 18 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index e8a7b96..c27308b 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,11 +1,16 @@ name: Release -# Tag push publishes. The human gate is the `pypi` environment below, which pauses the -# run BEFORE the OIDC token is minted — so the approval exists regardless of what -# triggered the workflow, and a tag pushed by accident cannot reach PyPI unattended. +# Publishing a GitHub Release is the gate. Not a tag push: a tag is cheap to create by +# accident and impossible to take back once it has reached PyPI, whereas a Release is a +# deliberate act with the notes in front of you. +# +# The `pypi` environment below is the second gate, and the stronger one — it pauses the +# run BEFORE the OIDC token is minted. Environment protection rules are unavailable on a +# private repo under this plan, so it currently holds none; add a required reviewer once +# the repo is public and this becomes belt-and-braces. on: - push: - tags: ['v*'] + release: + types: [published] workflow_dispatch: permissions: @@ -55,12 +60,12 @@ jobs: # sigstore signing job would be duplicative. - uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 - # Its own job so the publish job keeps `id-token: write` as its ONLY permission. This - # one needs `contents: write` to create the release, and has no business holding the - # publishing identity while it does. - github-release: + # Attaching the built artifacts to the Release that triggered this. Its own job so the + # publish job keeps `id-token: write` as its ONLY permission — `contents: write` has no + # business sitting alongside the publishing identity. + attach-artifacts: needs: [publish] - if: startsWith(github.ref, 'refs/tags/') + if: github.event_name == 'release' runs-on: ubuntu-latest permissions: contents: write @@ -69,13 +74,7 @@ jobs: with: name: dist path: dist/ - # --generate-notes builds the body from merged PRs, categorised by .github/release.yml. - # Release notes are therefore a side effect of labelling PRs rather than a file - # anyone has to remember to update. - env: GH_TOKEN: ${{ github.token }} - run: | - gh release create "${GITHUB_REF_NAME}" dist/* \ - --repo "${GITHUB_REPOSITORY}" \ - --title "${GITHUB_REF_NAME}" \ - --generate-notes + TAG: ${{ github.event.release.tag_name }} + run: gh release upload "$TAG" dist/* --repo "${GITHUB_REPOSITORY}" --clobber diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 310e74f..20bb424 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,3 +45,20 @@ your `Config`, not in here. Say so in the PR if you think it's the exception. generated from merged PRs at tag time Issues and PRs are welcome. There is no response SLA. + +## Releasing + +`main` requires a pull request and green checks; direct pushes are blocked for everyone +without an admin bypass. + +1. Merge everything you want in the release, with PR titles that read as release notes — + they become the notes verbatim, categorised by label via `.github/release.yml`. +2. Bump `version` in `pyproject.toml` and `__version__` in `postflight/__init__.py` + (a test fails if they disagree), via a PR like any other change. +3. Tag it: `git tag v0.1.0 && git push origin v0.1.0`. A tag alone publishes nothing. +4. Create the GitHub Release for that tag with generated notes. **Publishing the Release + is what triggers the PyPI upload** — a tag is easy to push by accident and impossible + to retract once it has reached PyPI, so the deliberate act is the gate. + +The workflow builds, checks the metadata, publishes via Trusted Publishing (no API token +exists), and attaches the artifacts back to the Release. From bab6295a965477755b07c40464ea8c7b9e44c973 Mon Sep 17 00:00:00 2001 From: Hemanth Chittanuru Date: Thu, 13 Aug 2026 21:07:21 -0400 Subject: [PATCH 02/10] docs: split the manual out, add a CLI, and drop the temporal framing The README had become a manual. Four screens of configuration nuance sat between the taxonomy and Status, answering questions a reader only has after deciding to adopt. Split: README keeps pitch, taxonomy, install, use, what it prints, the Turn contract and status. Everything else moved intact to docs/configuring.md behind one pointer. Added a CLI, which was the real gap. Everyone wiring this into cron or CI had to write a wrapper first. `python -m postflight --otel spans.jsonl` or `--langfuse --hours 24`, exit 1 on a fault so a CI step can gate on it, `--coverage` to ask which detectors can fire at all, `--json` for machines. It also prints inert detectors unprompted after the findings, because a zero next to a detector that could not have fired is the failure this package exists to prevent. Fixture provenance was overstated. "A real OpenInference capture of a support agent" reads as a production agent. It is a real capture of an INVENTED scenario: a throwaway support desk written for the test, run for real through OpenInference. Audited every attribute across all six spans including the input.value blobs the README never prints: no names, phones, emails, addresses or identifiers of any kind, only the toy tools and an order reference made up for the purpose. Wording now says which half is real. Cut the temporal framing. "Portability is now demonstrated rather than asserted" and "cache_read_tokens is now int | None" argue with a critic the reader never met, and "now" is meaningless to someone seeing version one. State the property instead. The OTel-adapter-found-a-bug narrative went with it: that is release-note content, and this project keeps release notes on the Releases page. Fixed section drift. refusal_exemptions sat under "What the other detectors depend on" when it is a TOOL_REFUSAL concept, and the claim_rules material was orphaned after it. Both now sit under their own headings. Deduplicated unknown-is-not-zero, the deny-list argument, and order-matters, each of which was stated twice. Moved the TOOL_REFUSAL caveat to a footnote so the row lands rather than trailing a qualifier inside the table. Em-dashes are gone from every public document, rewritten sentence by sentence rather than swapped for commas, which would have produced splices. Code comments still use them; that is a larger sweep and a separate call. 94 tests. Co-Authored-By: Claude Opus 5 --- CONTRIBUTING.md | 12 +- README.md | 289 +++++++++---------------------------- SECURITY.md | 4 +- docs/configuring.md | 165 +++++++++++++++++++++ postflight/__main__.py | 174 ++++++++++++++++++++++ tests/test_cli.py | 61 ++++++++ tests/test_otel_adapter.py | 14 +- 7 files changed, 488 insertions(+), 231 deletions(-) create mode 100644 docs/configuring.md create mode 100644 postflight/__main__.py create mode 100644 tests/test_cli.py diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 20bb424..2c2b8ab 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,19 +10,19 @@ python -m pytest tests/ -q ``` There is no install step and no dependency file to sync. If `pip install pytest` is not -enough to run the suite, that is a bug in this project, not in your setup — the package +enough to run the suite, that is a bug in this project, not in your setup. The package is zero-dependency and CI proves it by installing nothing else. ## What a good change looks like **A new detector** needs a name that describes the *behaviour*, not a metric, and a test -for both directions — the case it must catch and the case it must not. The second is the +for both directions: the case it must catch and the case it must not. The second is the one that matters: a detector that fires on healthy traffic is how the real findings get ignored, and every detector here was narrowed at least once because of that. **A new adapter** is a function from your trace format to `Turn`. Read "Writing an adapter" in the README first; the four notes there are each a mistake already made once. -Adapters must not add a dependency — parse exported JSON rather than importing a vendor +Adapters must not add a dependency. Parse exported JSON rather than importing a vendor SDK. **Detector codes are the public interface.** Renaming one is a breaking change, so it @@ -41,7 +41,7 @@ your `Config`, not in here. Say so in the PR if you think it's the exception. - New behaviour has a test - A comment explains *why the code is the way it is*, where that is not obvious. Not what it does, and not the story of how it was found -- The PR title reads as a release note — it becomes one verbatim, since release notes are +- The PR title reads as a release note, because it becomes one verbatim: release notes are generated from merged PRs at tag time Issues and PRs are welcome. There is no response SLA. @@ -51,13 +51,13 @@ Issues and PRs are welcome. There is no response SLA. `main` requires a pull request and green checks; direct pushes are blocked for everyone without an admin bypass. -1. Merge everything you want in the release, with PR titles that read as release notes — +1. Merge everything you want in the release, with PR titles that read as release notes. they become the notes verbatim, categorised by label via `.github/release.yml`. 2. Bump `version` in `pyproject.toml` and `__version__` in `postflight/__init__.py` (a test fails if they disagree), via a PR like any other change. 3. Tag it: `git tag v0.1.0 && git push origin v0.1.0`. A tag alone publishes nothing. 4. Create the GitHub Release for that tag with generated notes. **Publishing the Release - is what triggers the PyPI upload** — a tag is easy to push by accident and impossible + is what triggers the PyPI upload.** A tag is easy to push by accident and impossible to retract once it has reached PyPI, so the deliberate act is the gate. The workflow builds, checks the metadata, publishes via Trusted Publishing (no API token diff --git a/README.md b/README.md index 7cafdf4..7861b8e 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,10 @@ **Turn-level failure detection for tool-calling agents.** Your evals score what the model *said*. These are the failures that happen in the gaps -between what it said and what it did — a tool that declined three steps before the reply -that contradicts it, the same read issued eight times, a cache that never warmed. They -are invisible to an evaluator scoped to one observation, which is what every tracing -platform's evaluator runtime gives you today. +between what it said and what it did: a tool that declined three steps before the reply +contradicting it, the same read issued eight times, a cache that never warmed. They are +invisible to an evaluator scoped to a single observation, which is what every tracing +platform's evaluator runtime gives you. ## The taxonomy @@ -19,22 +19,40 @@ platform's evaluator runtime gives you today. |---|---|---| | `UNVERIFIED_CLAIM` | The reply asserts a write that no successful tool backs up. | The only one a user experiences directly as a lie. They were told something happened that did not happen. | | `TOOL_ERROR` | A tool raised; the framework wrapped it. | The visible half of tool failure. Usually already in your dashboards. | -| `TOOL_REFUSAL` | A tool ran fine and **declined in its own result body**, in a shape you've told postflight about (by default, a success flag set to `false`). | The dangerous half. Every guard that asks "did the tool run" is satisfied, so a false confirmation sails through. **Convention-dependent — see below.** | -| `REPEATED_TOOL` | The same tool called 3+ times in one turn. | The model is searching for an argument it was never given. A context gap, not a model failure — fix the prompt. | +| `TOOL_REFUSAL` [^1] | A tool ran fine and **declined in its own result body**, with no error flag. | The dangerous half. Every guard that asks "did the tool run" is satisfied, so a false confirmation sails through. | +| `REPEATED_TOOL` | The same tool called 3+ times in one turn. | The model is searching for an argument it was never given. A context gap, not a model failure. | | `TOOL_STORM` | 8+ tool calls in one turn. | Same cause, worse. Cost and latency both. | -| `EMPTY_REPLY` | The turn produced no text where somebody was owed one. | On a 1:1 channel this is the "it just didn't respond" bug. Reports at INFO until you set `conversational_kinds` — unconfigured, postflight can't tell a silent channel from a batch job that returns a document. | +| `EMPTY_REPLY` | The turn produced no text where somebody was owed one. | On a 1:1 channel this is the "it just didn't respond" bug. | | `GATE_FILTERED` | A turn a relevance gate dropped without doing work. | **Information, not a fault.** Silence is the design. Watch the count for a gate that has started swallowing real traffic. | | `SLOW_TURN` | Wall clock over the threshold. | Usually a storm with a human waiting. | -| `NO_CACHE_HIT` | A prompt big enough to cache that read nothing from cache. | Caching is a prefix match, so one volatile byte early in the system prompt silently drops the discount on *every* turn. | +| `NO_CACHE_HIT` | A prompt big enough to cache that read nothing from cache. | Caching is a prefix match, so one volatile byte early in the system prompt drops the discount on *every* turn. | The codes are the stable interface. Filter on them, chart them, page on them. -## Use +[^1]: Detects an in-body decline in a shape you have told it about. The default is a +success flag set to `false`. If your tools say no some other way, see +[configuring](docs/configuring.md#what-tool_refusal-can-and-cannot-see). + +## Install ```bash pip install postflight ``` +No dependencies. Python 3.11+. + +## Use + +From the command line, which is what a cron job or a CI step wants. Exit status is 1 +when something faulted, 0 otherwise: + +```bash +python -m postflight --langfuse --hours 24 +python -m postflight --otel spans.jsonl +``` + +Or from Python: + ```python from postflight import Config, faults, run_all from postflight.adapters.langfuse import LangfuseAdapter, LangfuseClient @@ -47,49 +65,53 @@ for turn_id, findings in run_all(turns, Config()).items(): print(turn_id, finding.code, finding.message) ``` -No dependencies. Python 3.11+. +## What it prints -## What it actually prints +Against the trace in `tests/fixtures/`, a throwaway support agent whose notification +tool declined (an invented scenario, captured for real through OpenInference): -Run it against the trace shipped in `tests/fixtures/` — a real OpenInference capture of a -support agent whose notification tool declined: - -``` -0x4069cd95… TOOL_REFUSAL 1 tool call(s) declined in-body ``` +$ python -m postflight --otel tests/fixtures/openinference_support_turn.jsonl + 0x4069cd953e TOOL_REFUSAL 1 tool call(s) declined in-body -The `detail` dict is the part you act on: +1 turns, 1 flagged + TOOL_REFUSAL 1 -```json -{"calls": [{"tool": "send_notification", - "result": "{'sent': False, 'reason': 'channel unavailable'}"}]} +Not all detectors are live on this data: + NO_CACHE_HIT: INERT - no generation reports cache usage ``` -Note what did **not** fire. The agent's reply said *"I wasn't able to send the -notification"* — an honest report of a failure, not a claim — so `UNVERIFIED_CLAIM` -stayed quiet. Had it said "I've let the customer know", that same turn would have -produced the finding you actually want to be paged about. +Three things to read there. + +The finding's `detail` is what you act on: `{"tool": "send_notification", "result": +"{'sent': False, 'reason': 'channel unavailable'}"}`. -Expect the counts to be lopsided, and expect that to be the useful part. `SLOW_TURN` -and `GATE_FILTERED` dominate any real window — one is already visible to whoever waited, -and the other is a surface behaving correctly. The rare rows carry the weight: a single -`TOOL_REFUSAL` or `UNVERIFIED_CLAIM` is a user who was told something untrue, and nothing -else in your stack is going to raise it. Sort by severity, not by count. +Nothing fired for `UNVERIFIED_CLAIM`, correctly. The agent's reply said *"I wasn't able +to send the notification"*, an honest report rather than a claim. Had it said "I've let +the customer know", that same turn produces the finding you want to be paged about. -## Writing an adapter +The last block matters as much as the findings. A detector whose input is missing does +not error, it just never fires, and an empty column reads exactly like a clean agent. +`coverage()` reports which ones could not have fired, so a zero can be trusted. -Detectors never see your trace format. They read `Turn`, so an adapter is a function from -whatever you have to an ordered sequence of steps: +Expect the counts to be lopsided, and expect that to be the useful part. `SLOW_TURN` and +`GATE_FILTERED` dominate any real window; one is already visible to whoever waited, and +the other is a surface behaving correctly. The rare rows carry the weight. Sort by +severity, not by count. + +## The `Turn` contract + +Detectors never see your trace format. They read `Turn`, so an adapter is a function +from whatever you have to an ordered sequence of steps: ```python from postflight.model import Generation, ToolCall, Turn Turn( - # your trace/turn identifier id="…", - # the SURFACE — which agent, which channel + # the surface: which agent, which channel kind="chat.turn", - # ORDERED: the sequence is the signal + # ordered, because the sequence is the signal steps=( Generation(text="", input_tokens=900, model="…"), ToolCall(name="search", result={"count": 0}), @@ -98,198 +120,31 @@ Turn( ) ``` -That is the whole contract. Four notes, each of which cost something to learn: - -- **Order matters more than nesting.** A turn is a sequence, not a tree. That is what - lets a detector see a tool that declined three steps before the reply contradicting it - — the thing a per-observation evaluator structurally cannot do. Flatten your tree. -- **`ToolCall.is_error` is the TRANSPORT flag only** — an exception, an `isError`, an - `ERROR` span status. A tool that ran fine and declined in its own body is *not* an - error; leave `is_error=False`, put the body in `result`, and `success_flags` will - classify it. Conflating the two hides the more dangerous failure. -- **Unknown is not zero.** If your producer does not report cache usage, leave - `cache_read_tokens=None`. Passing `0` asserts a cache miss, and `NO_CACHE_HIT` will - believe you. (This is exactly how the OTel adapter got it wrong first.) -- **`kind` should fall back to `"unknown"`, never to a guess.** Every kind-keyed rule in - this package is a deny-list so that unknown fails *closed*; a plausible-looking default - would quietly exempt the turns you most want checked. - -`postflight/adapters/otel.py` is ~150 lines and is the one to copy — it deals with -flattened attributes, a span tree, and two timestamp encodings, so most of the awkward -cases are already worked out there. +A turn is a sequence, not a tree. That is what lets a detector see a tool that declined +several steps before the reply contradicting it. Flatten yours. -## Tuning it to your agent +Adapters ship for Langfuse and for OpenTelemetry / OpenInference. +`postflight/adapters/otel.py` is the one to copy: it handles flattened attributes, a span +tree and two timestamp encodings, so the awkward cases are worked out there. Read +[writing an adapter](docs/configuring.md#writing-an-adapter) first. Its four notes are +each a mistake already made once. -The detectors are mechanism; the vocabulary is yours. Everything below is a `Config` -field, and the shipped defaults are a starting point, not a claim of completeness — -they are what a tool-calling agent looks like before you have watched *yours* fail. - -```python -Config( - slow_turn_s=30.0, - tool_storm=6, - # Tools that decline in-body, by the key they set to False. - success_flags=("ok", "updated", "sent", "created"), - # Surfaces that owe a human a reply. Setting this is what promotes EMPTY_REPLY from - # INFO to a fault — leave it empty and postflight cannot tell a silent channel from - # a batch job that returns a document, so it counts them instead of blaming them. - conversational_kinds=frozenset({"chat.turn", "inbound.turn", "group.turn"}), - # Surfaces that narrate rather than speak. A digest summarising someone's history - # uses the same words a claim does, with no user and no write in the turn. - narrating_kinds=frozenset({"digest.turn"}), - # Surfaces fronted by a relevance gate, where silence is correct. A quiet kind is - # conversational by definition — you do not have to list it in both. - quiet_kinds=frozenset({"group.turn"}), -) -``` +## Configuring -### What `TOOL_REFUSAL` can and cannot see - -This detector does **not** assume your tools return `{"updated": false}`. It assumes you -tell it how your tools say no. Out of the box it recognises four shapes: - -| shape | verdict | -|---|---| -| the call raised — `is_error` set by the adapter | `TOOL_ERROR` | -| the framework's error string (`Error executing tool …`) | `TOOL_ERROR` | -| a truthy `error` key in the result | `TOOL_ERROR` | -| a key from `success_flags` set to `false` | `TOOL_REFUSAL` | - -Anything else reads as success. If your tools signal failure some other way — a -`status` field, an enum, an HTTP-ish code — **`TOOL_REFUSAL` will never fire and your -report will look clean**. Add your convention: - -```python -Config( - refusal_predicates=( - lambda r: isinstance(r, dict) and r.get("status") in {"failed", "declined"}, - ) -) -``` - -There is no default for that, deliberately. `{"status": "failed"}` returned by a -`get_job_status` tool describes the *job*, not the call — guessing would make every -healthy status read into a refusal, which is precisely the cry-wolf failure this package -exists to avoid. You know which of your tools report on themselves; postflight doesn't. - -Two things it will never infer, by design: an **empty result set** (a search that found -nothing is not a decline) and **prose** (`"No matching orders found."` is -indistinguishable from success without reading it). If a tool of yours only fails in -prose, the durable fix is in the tool, not here. - -### What the other detectors depend on - -Same class of problem, and the reason `coverage()` exists: a detector whose input is -missing does not error, it just never fires — and an empty column reads exactly like a -clean agent. - -| detector | goes quiet if | goes *wrong* if | -|---|---|---| -| `UNVERIFIED_CLAIM` | the adapter supplies no reply text, or your replies are not in the vocabulary `claim_rules` knows (they are English by default) | your tool names don't match `satisfied_by` / `satisfied_by_prefix` — then a genuine action reads as an unbacked claim | -| `TOOL_ERROR` · `TOOL_REFUSAL` · `REPEATED_TOOL` · `TOOL_STORM` | the adapter maps no tool spans | — | -| `SLOW_TURN` | the adapter supplies no timestamps | — | -| `NO_CACHE_HIT` | no token counts, or the producer reports no cache usage | — | -| `EMPTY_REPLY` | there are no generations | the adapter fails to extract reply text — then it fires on **every** turn | -| `GATE_FILTERED` | `quiet_kinds` is unset (the default) | — | - -Note the coupling: a broken reply mapping silences `UNVERIFIED_CLAIM` *and* makes -`EMPTY_REPLY` fire on everything. One wrong field, two wrong columns, in opposite -directions. - -So check rather than assume: - -```python -from postflight import coverage - -for row in coverage(turns, cfg): - print(row) # e.g. "NO_CACHE_HIT: INERT — no generation reports cache usage" -``` - -It reports structural inertness only — an input absent from every turn. It will not tell -you a detector is broken because its count is zero, because a tool that never errored is -a healthy agent, and conflating those would just move the problem up a level. - -`refusal_exemptions` is the other direction — shapes that look like refusals and are -not. The shipped one is `{"sent": false, "queued": true}`: a send handed off to a relay. -Exemptions outrank `refusal_predicates`, so widening your detection cannot silently -re-flag a path you already excused. - -The one worth real attention is `claim_rules`, which drives `UNVERIFIED_CLAIM`. A rule -pairs a regex against the tools that would make the claim true: - -```python -from postflight import ClaimRule, Config -import re - -Config( - claim_rules=( - ClaimRule( - name="ticket_filed", - pattern=re.compile( - r"\b(?:filed|opened|created)\b[^.\n]{0,40}\bticket\b", re.IGNORECASE - ), - satisfied_by=frozenset({"create_ticket", "escalate_to_support"}), - ), - ) -) -``` - -Matching reads the clause the match sits in, and skips it on two conditions: - -- **Negation.** "The follow-up was **not** sent" is the agent being honest about not - acting, and scoring that as a lie punishes exactly the behaviour you want. -- **A third-party subject.** "**The owner** emailed you" is the agent relaying what - someone else did. The verb-object pair is identical to a real claim; only the subject - differs, and relaying is ordinary in any reply that summarises a thread. - -Both are `Config` regexes (`negation`, `third_party_subject`) if your replies read -differently. - -## Two design rules worth knowing before you extend it - -**Kind-keyed exemptions are deny-lists, never allow-lists.** `Turn.kind` falls back to -`"unknown"` whenever the adapter cannot resolve it — a root span that failed to open, -sampling that dropped it. An allow-list of "surfaces we check" silently exempts those -real turns, and every future surface until someone remembers to edit the set. A new -narrating surface going unflagged is a false positive; a new conversational surface -going unflagged is a missed lie. - -**Report on faults, not on findings.** `GATE_FILTERED` is `Severity.INFO` because it -fires on correct behaviour. Counting it as a fault makes the headline cry wolf, and a -detector that cries wolf on the healthy case is how the real rows get ignored. Use -`faults()` for anything a human reads first. +Thresholds, claim vocabulary, which surfaces owe a reply, and what each detector needs +in order to fire at all: **[docs/configuring.md](docs/configuring.md)**. ## Status -Alpha. The taxonomy is the product: **detector codes are the public interface**, and +Alpha. The taxonomy is the product: detector codes are the public interface, and renaming one is a breaking change. Thresholds, default vocabularies and added detectors are not. Versioning follows SemVer, with the 0.x convention the spec leaves undefined made explicit: while the major is 0, a **minor** bump may break the API and a **patch** may -not. - -Release notes live on the -[Releases page](https://github.com/Base-Homes/postflight/releases), generated from the -merged PRs for each tag — one place, tied to the artifact it describes, rather than a -file that has to be remembered separately. - -**Adapters: Langfuse and OpenTelemetry / OpenInference.** The detectors never see a -vendor — they read `postflight.model.Turn`, so an adapter is just a function from your -trace format to an ordered list of `Generation` and `ToolCall` steps. - -Portability is now demonstrated rather than asserted: the OTel adapter was written -against real spans from an Anthropic agent instrumented with OpenInference — a producer -that shares nothing with the first one — and the shipped defaults caught a `TOOL_REFUSAL` -in it while correctly declining to flag the model's own negated sentence. `Turn` needed -no change to accept it. - -It did surface one real modelling bug, which is the point of trying: OpenInference emits -no cache attribute at all, and scoring that absence as `0` made every large-prompt turn a -false `NO_CACHE_HIT`. `Generation.cache_read_tokens` is now `int | None` — **unknown is -not zero** — and an adapter that cannot report cache usage says so. Worth knowing if you -write the third adapter. +not. Release notes are on the +[Releases page](https://github.com/Base-Homes/postflight/releases). -Issues and PRs welcome; no response SLA. +Issues and PRs welcome. No response SLA. -Apache 2.0 — see [LICENSE](LICENSE). +Apache 2.0. See [LICENSE](LICENSE). diff --git a/SECURITY.md b/SECURITY.md index cae649f..27acab5 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -12,7 +12,7 @@ Report privately via Please do not open a public issue. Expect an acknowledgement within 7 days. This is a small project with no dedicated -security staffing — that window is what can actually be met, not an aspiration. +security staffing, so that window is what can actually be met rather than an aspiration. ## Scope @@ -24,5 +24,5 @@ nothing from the traces it reads. The realistic surface is therefore: or enormous attribute sets). - `LangfuseClient` and the credentials a caller hands it. -Findings can quote trace content — including reply text — in `Finding.detail`. If your +Findings can quote trace content, including reply text, in `Finding.detail`. If your traces carry personal data, treat postflight's output as carrying it too. diff --git a/docs/configuring.md b/docs/configuring.md new file mode 100644 index 0000000..48e52a6 --- /dev/null +++ b/docs/configuring.md @@ -0,0 +1,165 @@ +# Configuring postflight + +The detectors are mechanism. The vocabulary, the thresholds and the surface names +are yours, and they live here. Read this when you adopt, not before. + +## Writing an adapter + +The `Turn` contract is in the [README](../README.md#the-turn-contract). Three notes on +filling it in, each of which cost something to learn: + +- **`ToolCall.is_error` is the TRANSPORT flag only**: an exception, an `isError`, an + `ERROR` span status. A tool that ran fine and declined in its own body is *not* an + error. Leave `is_error=False`, put the body in `result`, and let `success_flags` + classify it. Conflating the two hides the more dangerous failure. +- **Unknown is not zero.** If your producer does not report cache usage, leave + `cache_read_tokens=None`. Passing `0` asserts a cache miss and `NO_CACHE_HIT` will + believe you. +- **`kind` falls back to `"unknown"`, never to a guess.** See the deny-list rule below + for why a plausible-looking default is the dangerous option. + +Adapters must not add a dependency. Parse exported JSON rather than importing a vendor +SDK: the producer needs it, the reader does not. + +## Tuning it to your agent + +The detectors are mechanism; the vocabulary is yours. Everything below is a `Config` +field, and the shipped defaults are a starting point, not a claim of completeness. +they are what a tool-calling agent looks like before you have watched *yours* fail. + +```python +Config( + slow_turn_s=30.0, + tool_storm=6, + # Tools that decline in-body, by the key they set to False. + success_flags=("ok", "updated", "sent", "created"), + # Surfaces that owe a human a reply. Setting this is what promotes EMPTY_REPLY from + # INFO to a fault. Leave it empty and postflight cannot tell a silent channel from + # a batch job that returns a document, so it counts them instead of blaming them. + conversational_kinds=frozenset({"chat.turn", "inbound.turn", "group.turn"}), + # Surfaces that narrate rather than speak. A digest summarising someone's history + # uses the same words a claim does, with no user and no write in the turn. + narrating_kinds=frozenset({"digest.turn"}), + # Surfaces fronted by a relevance gate, where silence is correct. A quiet kind is + # conversational by definition, so you need not list it in both. + quiet_kinds=frozenset({"group.turn"}), +) +``` + +### What `TOOL_REFUSAL` can and cannot see + +This detector does **not** assume your tools return `{"updated": false}`. It assumes you +tell it how your tools say no. Out of the box it recognises four shapes: + +| shape | verdict | +|---|---| +| the call raised (`is_error` set by the adapter) | `TOOL_ERROR` | +| the framework's error string (`Error executing tool …`) | `TOOL_ERROR` | +| a truthy `error` key in the result | `TOOL_ERROR` | +| a key from `success_flags` set to `false` | `TOOL_REFUSAL` | + +Anything else reads as success. If your tools signal failure some other way, whether +a `status` field, an enum or an HTTP-ish code, then **`TOOL_REFUSAL` will never fire and your +report will look clean**. Add your convention: + +```python +Config( + refusal_predicates=( + lambda r: isinstance(r, dict) and r.get("status") in {"failed", "declined"}, + ) +) +``` + +There is no default for that, deliberately. `{"status": "failed"}` returned by a +`get_job_status` tool describes the *job*, not the call, and guessing would make every +healthy status read into a refusal, which is precisely the cry-wolf failure this package +exists to avoid. You know which of your tools report on themselves; postflight doesn't. + +Two things it will never infer, by design: an **empty result set** (a search that found +nothing is not a decline) and **prose** (`"No matching orders found."` is +indistinguishable from success without reading it). If a tool of yours only fails in +prose, the durable fix is in the tool, not here. + +`refusal_exemptions` is the other direction: shapes that look like refusals and are +not. The shipped one is `{"sent": false, "queued": true}`: a send handed off to a relay. +Exemptions outrank `refusal_predicates`, so widening your detection cannot silently +re-flag a path you already excused. + +### What the other detectors depend on + +Same class of problem, and the reason `coverage()` exists: a detector whose input is +missing does not error, it just never fires, and an empty column reads exactly like a +clean agent. + +| detector | goes quiet if | goes *wrong* if | +|---|---|---| +| `UNVERIFIED_CLAIM` | the adapter supplies no reply text, or your replies are not in the vocabulary `claim_rules` knows (they are English by default) | your tool names don't match `satisfied_by` / `satisfied_by_prefix`, and a genuine action then reads as an unbacked claim | +| `TOOL_ERROR` · `TOOL_REFUSAL` · `REPEATED_TOOL` · `TOOL_STORM` | the adapter maps no tool spans | | +| `SLOW_TURN` | the adapter supplies no timestamps | | +| `NO_CACHE_HIT` | no token counts, or the producer reports no cache usage | | +| `EMPTY_REPLY` | there are no generations | the adapter fails to extract reply text, and it then fires on **every** turn | +| `GATE_FILTERED` | `quiet_kinds` is unset (the default) | | + +Note the coupling: a broken reply mapping silences `UNVERIFIED_CLAIM` *and* makes +`EMPTY_REPLY` fire on everything. One wrong field, two wrong columns, in opposite +directions. + +So check rather than assume: + +```python +from postflight import coverage + +for row in coverage(turns, cfg): + print(row) # e.g. "NO_CACHE_HIT: INERT - no generation reports cache usage" +``` + +It reports structural inertness only, meaning an input absent from every turn. It will not tell +you a detector is broken because its count is zero, because a tool that never errored is +a healthy agent, and conflating those would just move the problem up a level. + +### Claim rules + +The one worth real attention is `claim_rules`, which drives `UNVERIFIED_CLAIM`. A rule +pairs a regex against the tools that would make the claim true: + +```python +from postflight import ClaimRule, Config +import re + +Config( + claim_rules=( + ClaimRule( + name="ticket_filed", + pattern=re.compile( + r"\b(?:filed|opened|created)\b[^.\n]{0,40}\bticket\b", re.IGNORECASE + ), + satisfied_by=frozenset({"create_ticket", "escalate_to_support"}), + ), + ) +) +``` + +Matching reads the clause the match sits in, and skips it on two conditions: + +- **Negation.** "The follow-up was **not** sent" is the agent being honest about not + acting, and scoring that as a lie punishes exactly the behaviour you want. +- **A third-party subject.** "**The owner** emailed you" is the agent relaying what + someone else did. The verb-object pair is identical to a real claim; only the subject + differs, and relaying is ordinary in any reply that summarises a thread. + +Both are `Config` regexes (`negation`, `third_party_subject`) if your replies read +differently. + +## Two design rules worth knowing before you extend it + +**Kind-keyed exemptions are deny-lists, never allow-lists.** `Turn.kind` falls back to +`"unknown"` whenever the adapter cannot resolve it: a root span that failed to open, +sampling that dropped it. An allow-list of "surfaces we check" silently exempts those +real turns, and every future surface until someone remembers to edit the set. A new +narrating surface going unflagged is a false positive; a new conversational surface +going unflagged is a missed lie. + +**Report on faults, not on findings.** `GATE_FILTERED` is `Severity.INFO` because it +fires on correct behaviour. Counting it as a fault makes the headline cry wolf, and a +detector that cries wolf on the healthy case is how the real rows get ignored. Use +`faults()` for anything a human reads first. diff --git a/postflight/__main__.py b/postflight/__main__.py new file mode 100644 index 0000000..aff80ae --- /dev/null +++ b/postflight/__main__.py @@ -0,0 +1,174 @@ +"""Command line entry point, so a cron job or CI step does not need a wrapper script. + + python -m postflight --otel spans.jsonl + python -m postflight --langfuse --hours 24 + python -m postflight --otel spans.jsonl --coverage + +Exit status is the useful part in CI: 0 when nothing faulted, 1 when something did. +INFO findings never affect it, because a surface that is silent by design should not +fail anyone's build. +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from collections import Counter + +from .config import Config +from .coverage import coverage +from .detectors import faults, run +from .model import Turn + + +def _kinds(value: str | None) -> frozenset[str]: + return frozenset(k.strip() for k in (value or "").split(",") if k.strip()) + + +def _build_config(args: argparse.Namespace) -> Config: + return Config( + slow_turn_s=args.slow_turn_s, + tool_storm=args.tool_storm, + repeated_tool=args.repeated_tool, + conversational_kinds=_kinds(args.conversational_kinds), + quiet_kinds=_kinds(args.quiet_kinds), + narrating_kinds=_kinds(args.narrating_kinds), + ) + + +def _load(args: argparse.Namespace) -> list[Turn]: + if args.otel: + from .adapters.otel import turns_from_jsonl + + return turns_from_jsonl(args.otel) + + from .adapters.langfuse import LangfuseAdapter, LangfuseClient + + public = os.environ.get("LANGFUSE_PUBLIC_KEY", "") + secret = os.environ.get("LANGFUSE_SECRET_KEY", "") + if not public or not secret: + raise SystemExit( + "LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY must be set for --langfuse" + ) + host = os.environ.get("LANGFUSE_HOST", "https://us.cloud.langfuse.com") + client = LangfuseClient(host, public, secret) + observations = client.observations(hours=args.hours, environment=args.environment) + return LangfuseAdapter().turns(observations) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="postflight", + description="Turn-level failure detection for tool-calling agents.", + ) + source = parser.add_mutually_exclusive_group(required=True) + source.add_argument( + "--otel", + metavar="FILE", + help="exported OpenTelemetry spans, one JSON object per line", + ) + source.add_argument( + "--langfuse", + action="store_true", + help="pull from Langfuse (credentials from the environment)", + ) + + parser.add_argument( + "--hours", type=int, default=24, help="window for --langfuse (default: 24)" + ) + parser.add_argument( + "--environment", default=None, help="Langfuse environment filter" + ) + + parser.add_argument( + "--coverage", + action="store_true", + help="report which detectors can fire on this data, then exit", + ) + parser.add_argument( + "--json", dest="as_json", action="store_true", help="machine-readable output" + ) + parser.add_argument( + "--quiet", action="store_true", help="summary only, no per-turn lines" + ) + + parser.add_argument("--slow-turn-s", type=float, default=Config().slow_turn_s) + parser.add_argument("--tool-storm", type=int, default=Config().tool_storm) + parser.add_argument("--repeated-tool", type=int, default=Config().repeated_tool) + parser.add_argument( + "--conversational-kinds", help="comma separated surfaces that owe a reply" + ) + parser.add_argument( + "--quiet-kinds", help="comma separated surfaces that are silent by design" + ) + parser.add_argument( + "--narrating-kinds", + help="comma separated surfaces that narrate rather than speak", + ) + + args = parser.parse_args(argv) + cfg = _build_config(args) + turns = _load(args) + + if args.coverage: + rows = coverage(turns, cfg) + if args.as_json: + print(json.dumps([r.__dict__ for r in rows], indent=2)) + else: + for row in rows: + print(row) + return 0 + + results = {t.id: run(t, cfg) for t in turns} + all_findings = [f for fs in results.values() for f in fs] + flagged = {tid: fs for tid, fs in results.items() if faults(fs)} + + if args.as_json: + print( + json.dumps( + { + "turns": len(turns), + "flagged": len(flagged), + "findings": [ + { + "turn_id": f.turn_id, + "code": f.code, + "severity": f.severity.value, + "message": f.message, + "detail": f.detail, + } + for f in all_findings + ], + }, + indent=2, + default=str, + ) + ) + return 1 if flagged else 0 + + if not args.quiet: + for turn_id, found in results.items(): + for finding in found: + mark = " " if finding.severity.value == "fault" else "i" + print(f"{mark} {turn_id[:12]:14} {finding.code:18} {finding.message}") + + counts = Counter(f.code for f in all_findings) + print(f"\n{len(turns)} turns, {len(flagged)} flagged") + for code, count in counts.most_common(): + print(f" {code:18} {count}") + + # An inert detector and a clean agent look identical in the output above, so say + # which ones could not have fired rather than leaving a zero to be misread. + inert = [r for r in coverage(turns, cfg) if not r.live or r.misleading] + if inert: + print("\nNot all detectors are live on this data:") + for row in inert: + print(f" {row}") + + return 1 if flagged else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_cli.py b/tests/test_cli.py new file mode 100644 index 0000000..68f4e8a --- /dev/null +++ b/tests/test_cli.py @@ -0,0 +1,61 @@ +"""The CLI is how this gets wired into a cron job or a CI step, so its exit status is +part of the contract: 0 when nothing faulted, 1 when something did.""" + +import pathlib + +import pytest + +from postflight.__main__ import main + +FIXTURE = str( + pathlib.Path(__file__).parent / "fixtures" / "openinference_support_turn.jsonl" +) + + +def test_exits_1_when_a_fault_is_found(capsys): + assert main(["--otel", FIXTURE]) == 1 + assert "TOOL_REFUSAL" in capsys.readouterr().out + + +def test_exits_0_when_nothing_faults(tmp_path, capsys): + empty = tmp_path / "none.jsonl" + empty.write_text( + '{"name": "a.turn", "context": {"trace_id": "t"}, "parent_id": null,' + ' "attributes": {"openinference.span.kind": "AGENT"}}\n' + ) + assert main(["--otel", str(empty)]) == 0 + + +def test_inert_detectors_are_named_in_the_output(capsys): + """A zero next to a detector that could not have fired is the failure this whole + package is trying not to have, so the CLI says so unprompted.""" + main(["--otel", FIXTURE]) + assert "Not all detectors are live" in capsys.readouterr().out + + +def test_coverage_mode_exits_0_and_reports_every_detector(capsys): + assert main(["--otel", FIXTURE, "--coverage"]) == 0 + out = capsys.readouterr().out + for code in ("UNVERIFIED_CLAIM", "TOOL_REFUSAL", "SLOW_TURN", "NO_CACHE_HIT"): + assert code in out + + +def test_json_output_is_parseable_and_keeps_the_exit_code(capsys): + import json + + assert main(["--otel", FIXTURE, "--json"]) == 1 + payload = json.loads(capsys.readouterr().out) + assert payload["turns"] == 1 and payload["flagged"] == 1 + assert payload["findings"][0]["code"] == "TOOL_REFUSAL" + + +def test_langfuse_without_credentials_fails_loudly(monkeypatch): + monkeypatch.delenv("LANGFUSE_PUBLIC_KEY", raising=False) + monkeypatch.delenv("LANGFUSE_SECRET_KEY", raising=False) + with pytest.raises(SystemExit, match="LANGFUSE_PUBLIC_KEY"): + main(["--langfuse"]) + + +def test_a_source_is_required(): + with pytest.raises(SystemExit): + main([]) diff --git a/tests/test_otel_adapter.py b/tests/test_otel_adapter.py index e5955eb..7581a6a 100644 --- a/tests/test_otel_adapter.py +++ b/tests/test_otel_adapter.py @@ -1,11 +1,13 @@ """The portability test: a trace this package did not grow up on. -The fixture is REAL output — an Anthropic tool-calling agent instrumented with -OpenInference and exported through the OpenTelemetry SDK, captured verbatim and then -stripped of the prompt text. Nothing about its shape was chosen here: flattened indexed -attributes, ISO timestamps, span kinds, `output.value` as an opaque string. A detector -that only ever met Langfuse's observation model has no business passing this by luck, so -these assert the DETECTIONS, not just that parsing did not raise. +The fixture is a real capture of an invented scenario: a throwaway support-desk agent +written for this test, run for real through the Anthropic SDK with OpenInference +instrumentation and exported by the OpenTelemetry SDK. The SCENARIO is synthetic, so it +carries no third-party data; the SHAPE is not, and the shape is the point. Flattened +indexed attributes, ISO timestamps, span kinds, `output.value` as an opaque string: +none of it was chosen here. A detector that only ever met Langfuse's observation model +has no business passing this by luck, so these assert the DETECTIONS rather than merely +that parsing did not raise. The agent was given two deliberately unhelpful tools. `send_notification` declines in its own body (`{"sent": false}`) with no error status, which is the shape that satisfies every From 9a6333b1640ee03b87f3f49b39a5ee9ce0031db5 Mon Sep 17 00:00:00 2001 From: Hemanth Chittanuru Date: Thu, 13 Aug 2026 21:30:46 -0400 Subject: [PATCH 03/10] docs: apply the revised README, and stop the CLI printing em-dashes Applied README v3 with two corrections, both caught by checking the claims against the code rather than reading them. `detail` is keyed by "calls" and holds a list; the snippet showed a single element of that list, so anyone indexing it as shown would get a KeyError. And the adapter section promised "four notes" where docs/configuring.md carries three: the fourth was the order-matters point, which now lives in the Turn contract section, and repeating it was the duplication that got removed. The output block was abridged while presented as literal terminal output, missing the GATE_FILTERED line the command actually prints. It is now generated from a real run, so what a reader sees matches what they get. That surfaced the larger miss: coverage() reasons are CLI output, which is public copy, and they were full of em-dashes. Rewritten, so the em-dash rule now covers what the tool prints and not only what the docs say. Code comments still use them, which stays a separate call. 94 tests, ruff clean, twine check --strict clean. Co-Authored-By: Claude Opus 5 --- README.md | 94 ++++++++++++++++++++++-------------------- postflight/coverage.py | 9 ++-- 2 files changed, 53 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 7861b8e..4faac32 100644 --- a/README.md +++ b/README.md @@ -7,32 +7,37 @@ **Turn-level failure detection for tool-calling agents.** -Your evals score what the model *said*. These are the failures that happen in the gaps -between what it said and what it did: a tool that declined three steps before the reply -contradicting it, the same read issued eight times, a cache that never warmed. They are -invisible to an evaluator scoped to a single observation, which is what every tracing -platform's evaluator runtime gives you. +postflight reads agent traces you already emit and returns coded findings for failures +that span a whole turn: a tool that declined three steps before the reply contradicting +it, the same read issued eight times, a cache that never warmed. It calls no model and +has no dependencies. ## The taxonomy | Code | What it means | Why it matters | |---|---|---| -| `UNVERIFIED_CLAIM` | The reply asserts a write that no successful tool backs up. | The only one a user experiences directly as a lie. They were told something happened that did not happen. | -| `TOOL_ERROR` | A tool raised; the framework wrapped it. | The visible half of tool failure. Usually already in your dashboards. | -| `TOOL_REFUSAL` [^1] | A tool ran fine and **declined in its own result body**, with no error flag. | The dangerous half. Every guard that asks "did the tool run" is satisfied, so a false confirmation sails through. | +| `UNVERIFIED_CLAIM` | The reply asserts a write no successful tool backs up. | The only one a user experiences as a lie. They were told something happened that did not happen. | +| `TOOL_ERROR` | A tool raised; the framework wrapped it. | The visible half of tool failure, usually already in your dashboards. | +| `TOOL_REFUSAL` [^1] | A tool ran fine and **declined in its own result body**, with no error flag. | The dangerous half. Every guard that asks "did the tool run" is satisfied, so a false confirmation ships. | | `REPEATED_TOOL` | The same tool called 3+ times in one turn. | The model is searching for an argument it was never given. A context gap, not a model failure. | | `TOOL_STORM` | 8+ tool calls in one turn. | Same cause, worse. Cost and latency both. | -| `EMPTY_REPLY` | The turn produced no text where somebody was owed one. | On a 1:1 channel this is the "it just didn't respond" bug. | -| `GATE_FILTERED` | A turn a relevance gate dropped without doing work. | **Information, not a fault.** Silence is the design. Watch the count for a gate that has started swallowing real traffic. | +| `EMPTY_REPLY` [^2] | No text where somebody was owed one. | On a 1:1 channel, the "it just didn't respond" bug. | +| `GATE_FILTERED` [^3] | A turn a relevance gate dropped without doing work. | **Information, not a fault.** Silence is the design. Watch the count for a gate that has started swallowing real traffic. | | `SLOW_TURN` | Wall clock over the threshold. | Usually a storm with a human waiting. | | `NO_CACHE_HIT` | A prompt big enough to cache that read nothing from cache. | Caching is a prefix match, so one volatile byte early in the system prompt drops the discount on *every* turn. | -The codes are the stable interface. Filter on them, chart them, page on them. +The codes are the stable interface. Filter on them, chart them, page on them. Renaming +one is a breaking change. [^1]: Detects an in-body decline in a shape you have told it about. The default is a success flag set to `false`. If your tools say no some other way, see [configuring](docs/configuring.md#what-tool_refusal-can-and-cannot-see). +[^2]: Reports at `INFO` until you set `conversational_kinds`, since unconfigured it +cannot tell a silent channel from a batch job that returns a document. + +[^3]: Never fires until you set `quiet_kinds`. Nothing is a gate by default. + ## Install ```bash @@ -43,15 +48,15 @@ No dependencies. Python 3.11+. ## Use -From the command line, which is what a cron job or a CI step wants. Exit status is 1 -when something faulted, 0 otherwise: +The CLI is what a cron job or a CI step wants. Exit status is 1 when something faulted, +0 otherwise: ```bash python -m postflight --langfuse --hours 24 python -m postflight --otel spans.jsonl ``` -Or from Python: +From Python: ```python from postflight import Config, faults, run_all @@ -65,10 +70,13 @@ for turn_id, findings in run_all(turns, Config()).items(): print(turn_id, finding.code, finding.message) ``` -## What it prints +`run_all` returns every finding, including `Severity.INFO` ones like `GATE_FILTERED`. +Wrap it in `faults()` for anything a human reads first. + +## Output -Against the trace in `tests/fixtures/`, a throwaway support agent whose notification -tool declined (an invented scenario, captured for real through OpenInference): +Run against the fixture, a support agent built to fail this way and captured through +OpenInference: ``` $ python -m postflight --otel tests/fixtures/openinference_support_turn.jsonl @@ -78,26 +86,24 @@ $ python -m postflight --otel tests/fixtures/openinference_support_turn.jsonl TOOL_REFUSAL 1 Not all detectors are live on this data: - NO_CACHE_HIT: INERT - no generation reports cache usage + GATE_FILTERED: INERT - no quiet_kinds configured, so nothing is silent by design + NO_CACHE_HIT: INERT - no generation reports cache usage, and unknown is not treated as zero ``` -Three things to read there. +Each finding carries a `detail` dict, which is the part you act on: -The finding's `detail` is what you act on: `{"tool": "send_notification", "result": -"{'sent': False, 'reason': 'channel unavailable'}"}`. - -Nothing fired for `UNVERIFIED_CLAIM`, correctly. The agent's reply said *"I wasn't able -to send the notification"*, an honest report rather than a claim. Had it said "I've let -the customer know", that same turn produces the finding you want to be paged about. +```json +{"calls": [{"tool": "send_notification", + "result": "{'sent': False, 'reason': 'channel unavailable'}"}]} +``` -The last block matters as much as the findings. A detector whose input is missing does -not error, it just never fires, and an empty column reads exactly like a clean agent. -`coverage()` reports which ones could not have fired, so a zero can be trusted. +The closing block comes from `coverage()`, and it is worth reading before the findings. +A detector whose input is missing does not error, it just never fires, and an empty +column looks the same as a clean agent. `coverage()` reports which detectors could not +have fired on this data, so a zero elsewhere means something. -Expect the counts to be lopsided, and expect that to be the useful part. `SLOW_TURN` and -`GATE_FILTERED` dominate any real window; one is already visible to whoever waited, and -the other is a surface behaving correctly. The rare rows carry the weight. Sort by -severity, not by count. +Expect `SLOW_TURN` and `GATE_FILTERED` to dominate any real window and expect the rare +rows to carry the weight. Sort by severity, not by count. ## The `Turn` contract @@ -120,14 +126,14 @@ Turn( ) ``` -A turn is a sequence, not a tree. That is what lets a detector see a tool that declined -several steps before the reply contradicting it. Flatten yours. +A turn is a sequence, not a tree. Flattening is what lets a detector see a tool that +declined several steps before the reply contradicting it. -Adapters ship for Langfuse and for OpenTelemetry / OpenInference. -`postflight/adapters/otel.py` is the one to copy: it handles flattened attributes, a span -tree and two timestamp encodings, so the awkward cases are worked out there. Read -[writing an adapter](docs/configuring.md#writing-an-adapter) first. Its four notes are -each a mistake already made once. +Adapters ship for Langfuse and for OpenTelemetry / OpenInference. Copy +`postflight/adapters/otel.py`, which handles flattened attributes, a span tree, and two +timestamp encodings, so most of the awkward cases are already worked out. Read +[writing an adapter](docs/configuring.md#writing-an-adapter) first; its three notes +are each a mistake already made once. ## Configuring @@ -136,13 +142,11 @@ in order to fire at all: **[docs/configuring.md](docs/configuring.md)**. ## Status -Alpha. The taxonomy is the product: detector codes are the public interface, and -renaming one is a breaking change. Thresholds, default vocabularies and added detectors -are not. +Alpha. Detector codes are the public interface; thresholds, default vocabularies, and +added detectors are not. -Versioning follows SemVer, with the 0.x convention the spec leaves undefined made -explicit: while the major is 0, a **minor** bump may break the API and a **patch** may -not. Release notes are on the +SemVer, with the 0.x convention the spec leaves undefined made explicit: while the major +is 0, a minor bump may break the API and a patch may not. Release notes are on the [Releases page](https://github.com/Base-Homes/postflight/releases). Issues and PRs welcome. No response SLA. diff --git a/postflight/coverage.py b/postflight/coverage.py index 0ea831d..df29149 100644 --- a/postflight/coverage.py +++ b/postflight/coverage.py @@ -38,7 +38,7 @@ class Coverage: def __str__(self) -> str: state = "MISLEADING" if self.misleading else ("live" if self.live else "INERT") - return f"{self.code}: {state} — {self.reason}" + return f"{self.code}: {state} - {self.reason}" def coverage(turns: Iterable[Turn], cfg: Config | None = None) -> list[Coverage]: @@ -77,7 +77,7 @@ def coverage(turns: Iterable[Turn], cfg: Config | None = None) -> list[Coverage] "UNVERIFIED_CLAIM", True, "no tool name in this data satisfies any claim rule (saw " - f"{len(seen_tools)} distinct tools) — a genuine action will read as an " + f"{len(seen_tools)} distinct tools). A genuine action will read as an " "unbacked claim. Check satisfied_by / satisfied_by_prefix against your " "tool names", misleading=True, @@ -161,7 +161,7 @@ def coverage(turns: Iterable[Turn], cfg: Config | None = None) -> list[Coverage] bool(cfg.quiet_kinds), "quiet_kinds configured" if cfg.quiet_kinds - else "no quiet_kinds configured — nothing is silent by design", + else "no quiet_kinds configured, so nothing is silent by design", ) ) @@ -190,8 +190,7 @@ def coverage(turns: Iterable[Turn], cfg: Config | None = None) -> list[Coverage] Coverage( "NO_CACHE_HIT", False, - "no generation reports cache usage — unknown is not " - "treated as zero, so this cannot fire", + "no generation reports cache usage, and unknown is not treated as zero", ) ) else: From df25a6813c0b5c619a2a8b2cf23202d895660f56 Mon Sep 17 00:00:00 2001 From: Hemanth Chittanuru Date: Thu, 13 Aug 2026 21:33:22 -0400 Subject: [PATCH 04/10] docs: how to read a TOOL_REFUSAL, without prescribing tool design A refusal is not automatically a bug in the tool. Most in-body declines are expected outcomes, and the reason is usually information the agent needs next, so "make it raise instead" would be wrong more often than right: it puts expected branches in error dashboards, flattens the structured reason into a stringified exception, and trades an interesting signal for the boring one you already had. Followed to its conclusion it would make TOOL_REFUSAL permanently inert, which is the failure coverage() exists to catch. What the section says instead is what to look at. A refusal on the same turn as an UNVERIFIED_CLAIM is the pairing that matters, because the tool said no and the reply said yes. A refusal the reply reports honestly is the system working, which is exactly what the shipped fixture shows. The one shape genuinely worth fixing at the source is a tool swallowing a real failure into a result body, a 500 returned as {"ok": false}. That is an error wearing a decline's clothes and belongs in TOOL_ERROR where existing alerting can see it. Pinned the pairing, since the docs now send a reader looking for it: both codes fire on the dishonest turn, and only TOOL_REFUSAL on the honest one. Lives in docs/configuring.md rather than under the taxonomy table. The table is the hook, and the restructure moved nuance out of it for a reason. 95 tests. Co-Authored-By: Claude Opus 5 --- docs/configuring.md | 15 +++++++++++++++ tests/test_detectors.py | 19 +++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/docs/configuring.md b/docs/configuring.md index 48e52a6..07e9231 100644 --- a/docs/configuring.md +++ b/docs/configuring.md @@ -80,6 +80,21 @@ nothing is not a decline) and **prose** (`"No matching orders found."` is indistinguishable from success without reading it). If a tool of yours only fails in prose, the durable fix is in the tool, not here. +#### Reading a refusal + +A `TOOL_REFUSAL` is not automatically a bug in the tool. Most in-body declines are +expected outcomes rather than defects, and the reason is often information the agent +needs in order to do something sensible next. + +What is worth checking is whether the decline was *handled*. A refusal on the same turn +as an `UNVERIFIED_CLAIM` is the pairing that matters: the tool said no, and the reply +said yes, which means a user was told something untrue. A refusal that the reply reports +honestly is the system working, and the fixture in this repository is exactly that case. + +The one shape worth fixing at the source is a tool swallowing a genuine failure into a +result body, a 500 returned as `{"ok": false}`. That is an error wearing a decline's +clothes, and it belongs in `TOOL_ERROR` where your existing alerting can see it. + `refusal_exemptions` is the other direction: shapes that look like refusals and are not. The shipped one is `{"sent": false, "queued": true}`: a send handed off to a relay. Exemptions outrank `refusal_predicates`, so widening your detection cannot silently diff --git a/tests/test_detectors.py b/tests/test_detectors.py index 66aa64f..33eef35 100644 --- a/tests/test_detectors.py +++ b/tests/test_detectors.py @@ -363,3 +363,22 @@ def test_an_exemption_outranks_a_refusal_predicate(): cfg = Config(refusal_predicates=(lambda r: isinstance(r, dict) and "sent" in r,)) queued = tool("send", result={"sent": False, "queued": True}) assert tool_outcome(queued, cfg) is Outcome.OK + + +def test_the_refusal_plus_claim_pairing_the_docs_point_at(): + """docs/configuring.md tells a reader that a TOOL_REFUSAL on the same turn as an + UNVERIFIED_CLAIM is the combination worth acting on: the tool said no and the reply + said yes. That is only useful advice if both actually surface together.""" + both = turn( + tool("send_email", result={"sent": False, "reason": "channel down"}), + gen("I've sent them a message."), + ) + assert {"TOOL_REFUSAL", "UNVERIFIED_CLAIM"} <= codes(run(both)) + + # And the honest version of the same turn: tool declined, reply said so. One + # finding, not two, because nobody was misled. + honest = turn( + tool("send_email", result={"sent": False, "reason": "channel down"}), + gen("I wasn't able to send them a message."), + ) + assert codes(run(honest)) == {"TOOL_REFUSAL"} From 6e59b1bc317851dbaa350b08589428b7b3ae7772 Mon Sep 17 00:00:00 2001 From: Hemanth Chittanuru Date: Thu, 13 Aug 2026 21:42:12 -0400 Subject: [PATCH 05/10] docs: the turn-scope diagram, in Inter and the Base palette The one idea in the README that is genuinely spatial: the failure is a relationship between steps, and every step passes when scored alone. Everything else reads better as text, so this is the only diagram. Two constraints shaped the implementation more than the design did. GitHub sanitises