diff --git a/README.md b/README.md index 43850e0..caa9c3b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ -![Hullwork — from production errors to reviewable draft pull requests. Self-hosted, with your forge, -your error tracker, your model endpoint and a human gate on every merge.](images/banner.svg) +![Hullwork — verifies which of your tools' claims are actually true, before a person is asked. +Self-hosted, with your forge, your error tracker, your model endpoint and a human gate on every +merge.](images/banner.svg) [![CI](https://github.com/easybytehub/hullwork/actions/workflows/ci.yml/badge.svg)](https://github.com/easybytehub/hullwork/actions/workflows/ci.yml) [![Release](https://img.shields.io/github/v/release/easybytehub/hullwork?include_prereleases&label=release&color=blue)](https://github.com/easybytehub/hullwork/releases) @@ -7,9 +8,13 @@ your error tracker, your model endpoint and a human gate on every merge.](images [![Python](https://img.shields.io/badge/python-3.12-blue?logo=python&logoColor=white)](pyproject.toml) [![Licence: FSL-1.1-ALv2](https://img.shields.io/badge/licence-FSL--1.1--ALv2-blue)](LICENSE.md) -**Pre-alpha.** Both halves run end to end, five attempts have reached a draft pull request, and nobody -outside this project has installed it. What works, what does not, and what nobody has demonstrated are -all in **[docs/status.md](docs/status.md)** — read that before relying on any of this. +**Hullwork verifies which of the things your tools claim are actually true, before a person is +asked** — errors, dependencies, incidents. What that means, and everything that follows from it, is +**[what Hullwork is](docs/what-hullwork-is.md)**. + +**Pre-alpha.** The error signal runs end to end, five attempts have reached a draft pull request, and +nobody outside this project has installed it. What works, what does not, and what nobody has +demonstrated are all in **[docs/status.md](docs/status.md)** — read that before relying on any of this. > **What this repository is.** All of Hullwork, under a source-available licence that becomes > Apache-2.0 two years after each release: the whole loop, uncapped, for as many projects as you like. @@ -35,6 +40,28 @@ all in **[docs/status.md](docs/status.md)** — read that before relying on any ## What it does +Every signal it accepts arrives from a tool that **asserts something and proves nothing**. A tracker +says *something broke*. An advisory says *this version is vulnerable*. A scanner says *this could be +exploited*. None of them ran anything. + +So Hullwork takes the claim into a sandbox, submits it to an oracle **it cannot influence**, and +returns a verdict with the run attached. The oracle changes per signal; the machine does not. + +| what arrives | what it really says | what settles it here | +|---|---|---| +| a production error | *something broke* | a test that fails before the change and passes after | +| a dependency advisory | *this version is vulnerable* | your own suite, run against the upgrade | +| a static finding | *this could be exploited* | a test naming the hostile input | + +**Only the first row is in a release.** The second is built and unreleased — it is in no image you +can pull, and this page will say so until it is. The third does not exist. + +**"I could not verify this" is a first-class answer**, and on this repository's own numbers — 160 +code scanning alerts, five real — it is the answer about nine times in ten. Delivering it honestly +is worth more than a fix, because nobody else delivers it at all. + +### The path the error signal takes + ![Two things arrive — a production error by webhook, and a human report through a normaliser. Hullwork triages, deduplicates and assigns a risk lane. Green items are attempted unattended, amber wait for your approval, red are never attempted at all; anything matching no lane is red. A green item goes to a coding @@ -146,7 +173,7 @@ instance has ever run it, which is not support. | | | | |---|---|---| | **Forges** | ✅ Forgejo · ✅ Gitea · ✅ GitHub | ⚠️ GitLab — the adapter is written, no instance has run it | -| **Error trackers** | ✅ GlitchTip, and anything posting a Sentry-compatible payload | ⚠️ Sentry's signed webhooks — the route is written and switched off, because verifying an HMAC means storing a secret reversibly and that decision has not been made | +| **Error trackers** | ✅ GlitchTip, and anything posting a Sentry-compatible payload | ⚠️ Sentry's own route is built and **not in a release yet**; when it lands it is authenticated by the token in the URL and its signature is not verified — which is what GlitchTip offers at all. [`SECURITY.md`](SECURITY.md) says what that does not cover | | **Model endpoints** | ✅ anything speaking the Anthropic or OpenAI protocol family — Anthropic and OpenRouter both exercised | Your key, your endpoint. No provider is integrated and none is privileged | | **Agents** | ✅ `claude-code`, exercised | Any container that takes a worktree and returns changed files qualifies: the agent is a contract, not an integration | | **Your stack** | ✅ any Linux image with a shell, on this instance's architecture — you name the image your CI already uses | ❌ `distroless` and `scratch`, refused at registration rather than at attempt time | diff --git a/SECURITY.md b/SECURITY.md index 9a9f07d..05a8ec9 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -48,6 +48,18 @@ Each of these is a property of the code, and the file that implements it explain - **The two halves hold different credentials** (DR-0009). The half that answers webhooks — the one an attacker can reach — cannot push, and refuses to start if it finds a credential that can. The half that can push listens on nothing. +- **A webhook is authenticated by the token in its URL, and by nothing else.** That token is minted + once, shown once, and stored as a one-way hash; a wrong one is refused identically on every + provider's route, so the door cannot be used to confirm which tracker a project uses. + + **What that does not cover, stated once for both providers** — Sentry's route is built and not in + a release yet (`docs/status.md`), and this is what it will be authenticated by when it lands. + GlitchTip cannot sign its webhooks — + no header, no secret, no setting — so the URL is the credential. Sentry *can* sign, and Hullwork + does not verify it: doing so requires holding Sentry's client secret in a form it can be read back + from, which is a storage decision this project has not made. So **anyone who obtains the URL can + post to it**, whichever tracker you use. Treat it as a secret: it is in your tracker's + configuration, and `hullwork projects rotate-secret` replaces it. ### What leaves the instance diff --git a/docs/hullwork-yml.md b/docs/hullwork-yml.md index 3e05244..7fce164 100644 --- a/docs/hullwork-yml.md +++ b/docs/hullwork-yml.md @@ -258,6 +258,29 @@ incomplete. That ordering is what makes this a translation rather than a treadmi that describe an environment are about three and language-neutral, while ecosystems are about fifty and keep arriving. +**And they compose, which is easy to miss and worth a paragraph.** *Bring your own image* and +*name an installer* read as a choice, and they are not one. `base` takes any image; `install` takes +**your own command** rather than only a name from the recipe list. So a third shape has always been +legal: + +```yaml +runtime: + base: ghcr.io/acme/ci-base:2026.7 # yours, unchanged + install: "pip install -r requirements.txt" # your line, not a recipe of ours + dependencies: [requirements.txt] # the file your versions are pinned in +``` + +That is **one layer on top of the image you named**, not a rebuild from scratch — and it is what +makes a dependency upgrade measurable: an upgrade can only be checked against a suite that actually +runs it, which means the image has to be refreshed from the file that pins. With `install: none` +your image is used exactly as it comes, which is right for running tests and means a changed pin +changes nothing your suite would see. + +Nothing was added to permit this. `install` has accepted an arbitrary command since DR-0007 was +built, and this paragraph exists because nothing said so — the example at the top of this file +shows `install: none`, and a reader with an image of their own could reasonably conclude the field +was not for them. + **The one frontier that will not move**: any Linux image with a shell, on this instance's architecture. Both are checked when you register, and `distroless`/`scratch` are permanently out. diff --git a/docs/status.md b/docs/status.md index 942b041..6ee521e 100644 --- a/docs/status.md +++ b/docs/status.md @@ -4,6 +4,26 @@ Pre-alpha. This page is the honest scope, kept apart from the README so it has r so nothing in it has to be shortened to keep an introduction readable. It changes weekly; the date on each claim is part of the claim. +> **What all of this is the state of.** Hullwork verifies which of the things your tools claim are +> actually true, before a person is asked — [what Hullwork is](what-hullwork-is.md). Three signals, +> three oracles, one mechanism. This page was accurate about the halves and silent about what they +> were halves of, so here is the row that was missing (item 181, 2026-08-09): +> +> | signal | oracle | state | +> |---|---|---| +> | a production error | a test that fails first and passes after | **released**, and everything below describes it | +> | a dependency advisory | your own suite, run against the upgrade | **built and unreleased** — in no image you can pull | +> | a static finding | a test naming the hostile input | does not exist | +> +> The second row is work items 172–180. Its command is deliberately **not named here**: the guard on +> this repository's documentation refused the sentence that named it, and it was right to — a command +> a reader cannot run is an invitation to type it and be told it does not exist. What is actionable +> is the state, and the state is *not in an image you can pull*. +> +> This page will keep saying so until a release contains it: documentation describes the released +> artefact, not the working tree (`CONTRIBUTING.md`, and item 165 records the three times that rule +> was learnt the hard way). + ## What works today A production error posted by your error tracker is authenticated, stored, normalised, deduplicated, @@ -39,10 +59,14 @@ its suite runs against a blank one per phase. That path has been exercised by ** ## What does not exist yet -- **Only the GlitchTip webhook route is enabled.** Sentry signs its webhooks properly and would be - verified by HMAC — that route is written and switched off, because verifying a signature means - storing a client secret in reversible form, which is a different storage decision from the one-way - hash used here and has not been made. +- **Only the GlitchTip webhook route is enabled in a release.** Sentry's is **built and unreleased** + as of 2026-08-09 (item 189): in the working tree it is authenticated by the token in the URL — the + same credential GlitchTip has, checked the same way, because GlitchTip cannot sign at all. Its + signature is deliberately not verified; that would need Sentry's client secret held in reversible + form, which is a storage decision this project has not made. + + The image you can pull still answers `501` there, and this page will say so until a release + carries it — the same rule as the dependency signal above, and for the same reason. - **Of the notification channels, only `none` and `console` deliver.** `telegram` and `email` parse in the manifest and are refused at delivery, because a transport nobody has exercised is a transport whose first real run happens in front of a user. diff --git a/docs/what-hullwork-is.md b/docs/what-hullwork-is.md new file mode 100644 index 0000000..7284649 --- /dev/null +++ b/docs/what-hullwork-is.md @@ -0,0 +1,112 @@ +# What Hullwork is + +> **The canonical statement of the product, fixed 2026-08-09 by the operator.** Every other document +> describes a part; this one says what the parts are parts of. When a document and this page +> disagree, this page is what was decided and the other one has not caught up yet. +> +> The reasoning behind it is DR-0017. This page is +> the short form, kept separate so it can be read in a minute and quoted without a link. + +## In one sentence + +**Hullwork takes the work off a developer that nobody wants to do — errors, dependencies, +incidents — by verifying, before a person is asked, which of the things their tools claim are +actually true.** + +## Why it is not a list of features + +Every signal Hullwork accepts arrives from a tool that **asserts something and proves nothing**: + +| what arrives | from | what it really says | verified by | +|---|---|---|---| +| a production error | Sentry, GlitchTip | *something broke* | a test that reproduces it | +| a dependency advisory | Renovate, Dependabot, OSV | *this version is vulnerable* | the project's own suite | +| a static finding | CodeQL, Opengrep | *this could be exploited* | a test naming the hostile input | + +Three signals, three oracles, **one mechanism**: take the claim into a sandbox, submit it to an +oracle the agent cannot influence, return a verdict with the run attached. + +That is why these are not three features that happen to share a repository. The oracle changes; the +machine does not. + +## The three properties everything else follows from + +**No oracle is written by the agent to make itself look right.** A reproducing test must fail first +on untouched code; the project's suite belongs to the project; a hostile input has to be nameable. +Remove this and the verdicts are worth nothing. + +**"I could not verify this" is a first-class answer.** On this repository's own numbers — 160 code +scanning alerts, five real — that is the answer roughly nine times in ten, and delivering it +honestly is worth more than a fix, because nobody else delivers it at all. + +**What is measured is how much left a person's desk with evidence attached.** Not a success rate. +An instance computes its own, on its own code, from its first day. + +## What that means against the tools it sits beside + +None of them verify anything, and that is the whole position: + +- **Renovate and Dependabot** open the pull request and let the reviewer find out. Their own + documented weakness is noise — *"here is every update, you decide"* — and it is structural: they + do not execute, so they cannot rank. Hullwork runs the suite first and hands over the ones that + pass. +- **Sentry Seer and Copilot Autofix** fix from unverified claims, with the same confidence for the + five that are real and the hundred and fifty-five that are not. +- **Reachability vendors** reduce the same noise with static analysis, which is another unverified + claim about an unverified claim. Executing is more expensive and it is not arguable. + +## What Hullwork does not do, and will not + +Merge by itself (constitution principle 1). Attempt a fix without a reproducing +test (DR-0003). Match a competitor's breadth for its own sake: +depth over coverage, because a verified verdict in five ecosystems is worth more than an unverified +one in ninety. + +## Which documents have caught up, and when + +Recorded so this page can be checked rather than believed. **Kept as a record rather than deleted**: +a page that could be checked and then cannot is a page that got weaker as it got more accurate. + +All five were rewritten on **2026-08-09** (work item 181). None of them had been wrong; all of them +were partial, describing the product by one signal's pipeline where a reader was deciding what it is. + +| | what it said before | what it says now | +|---|---|---| +| `pyproject.toml` | one signal's two endpoints, in the line PyPI shows | the one-sentence claim above | +| `README.md` | opened with the pipeline as *what it does* | opens with what is verified; the pipeline is named as the error signal's path | +| the roadmap | a segment and an obstacle order, with no product above them | says what is being roadmapped, and that the sections are not three products in a queue | +| the interface document | readers and their three questions | says what the surface is a surface *of*, and the constraint that follows | +| `docs/status.md` | accurate about the halves | says what they are halves of, per signal, with the state of each | + +**The images caught up too**, on the same day and after the sentence above first said they would +not. `images/banner.svg` is the top of the README; `images/the-pipeline.svg` now says it is the path +*a production error* takes rather than what Hullwork does; and `images/social-preview.png` — the +card a link to this repository renders as, anywhere it is pasted — was the last one and the one with +the most reach. + +That PNG had **no source in the repository**, which is what made it look unfixable. It has one now: +`images/social-preview.svg`, rasterised by `scripts/render-social-preview.sh`, both committed. + +**What is guarded rather than remembered.** +`test_no_published_document_describes_the_product_by_its_plumbing` asserts that the old sentence +appears in no published document, in no packaging metadata, and in **no image source**. It failed +the day it was written, which is what made it a gate rather than a decoration, and it caught three +more instances afterwards — including the first line of the README, which is the banner's alt text +and the first thing a screen reader announces. + +## What is released, and what is only built + +Stated here because everything above describes a mechanism with three oracles, and a reader who +takes that as an inventory would be misled by this page rather than by the ones it corrects. + +| signal | state, 2026-08-09 | +|---|---| +| a production error | **released** | +| a dependency advisory | **built and unreleased** — work items 172–180, absent from `published-surface.json`, which records `0.1.0a7` | +| a static finding | does not exist | + +`CONTRIBUTING.md`'s rule is that documentation describes the released artefact rather than the +working tree, so no document may show that command until a release carries it — which is why it is +not named on this page either. The guard refused the first draft of this section for exactly that, +and naming a command a reader cannot run would invite them to type it and be told it does not +exist. diff --git a/hullwork.yml b/hullwork.yml index 3f88d2e..f555613 100644 --- a/hullwork.yml +++ b/hullwork.yml @@ -1,8 +1,8 @@ # Hullwork, watched by Hullwork. # # The lanes below are matched against the *error* — its title and culprit — not against the parts -# of the repository a work item would touch. Those are two different questions, and the worker contract -# answers the other one: what an unattended worker may change. This file answers what an incoming +# of the repository a fix would touch. Those are two different questions: the derived path policy +# (`hullwork projects lanes`) answers the second one. This file answers what an incoming # production fault is allowed to become. # # Red wins over amber, amber over green, and anything matching nothing at all is red. @@ -36,7 +36,7 @@ autofix: - indexerror amber: # Anything about the shape of the database or what is installed: recoverable, but a fix is a - # migration or a dependency change, which the worker contract already keeps away from a worker. + # migration or a dependency change, which the derived path policy already keeps a human on. - alembic - migration - integrityerror diff --git a/hullwork/__init__.py b/hullwork/__init__.py index ebbff5c..ad13fcb 100644 --- a/hullwork/__init__.py +++ b/hullwork/__init__.py @@ -1,5 +1,5 @@ """Hullwork — from production errors to reviewable pull requests, on your own infrastructure.""" -__version__ = "0.1.0a7" +__version__ = "0.1.0a8" __all__ = ["__version__"] diff --git a/hullwork/bump.py b/hullwork/bump.py new file mode 100644 index 0000000..f27109a --- /dev/null +++ b/hullwork/bump.py @@ -0,0 +1,528 @@ +"""Does the upgrade survive the project's own suite. Item 173, DR-0016. + +**The question nobody else answers.** Renovate opens the pull request; when the bump breaks the +suite it leaves the body where it fell. Item 172 says which pins have a published advisory. This +says what happens if you take the fix — and the oracle is not a test an agent wrote for the +occasion, it is the suite the project already has, which is the one authority in this system that +no model can flatter. + +**No agent, no gateway, no model token.** There is nothing here for a model to do: rewriting +`jinja2==2.4.1` to `jinja2==2.10.1` is an edit, and the verdict comes from running commands. A +clean answer costs nothing but the two builds. + +**Why this needs no change to the sandbox's isolation**, which is the property that would have made +it a bad idea. `image.dependency_digest` makes the tag the content, so changing a version *is* a +rebuild with no invalidation to remember; and `sandbox/image.py` puts the network in the build and +never in the phase. The upgraded package is therefore installed while there is a network, and the +suite still runs against nothing. + +Shaped after `dispatch.dispatch()` deliberately: it is handed a box and a directory and gives back +a decision, so it is testable without Docker. +""" + +from __future__ import annotations + +import logging +import re +from collections.abc import Callable, Sequence +from dataclasses import dataclass, field +from enum import StrEnum +from pathlib import Path +from typing import Protocol + +from hullwork import dependencies, resolve, testoutput +from hullwork.sandbox.run import RunResult + +log = logging.getLogger(__name__) + +#: How long one suite run may take. The same ceiling the dispatcher's gates use, for the same +#: reason: a gate that runs out of time says nothing about the upgrade. +GATE_TIMEOUT_SECONDS = 1800 + + +class Verdict(StrEnum): + """What running the project's own suite against the upgrade said.""" + + #: The suite passed before and passes after. **Not** "this is safe" — see `says`. + CLEAN = "clean" + #: The finding. This upgrade breaks these tests. + BREAKS = "breaks" + #: The build failed, which is a different fact from the suite failing. + WILL_NOT_INSTALL = "will-not-install" + #: The suite was already failing, so no claim can be made either way. + ALREADY_RED = "already-red" + #: This file cannot be rewritten without producing something that will not install. + CANNOT_REWRITE = "cannot-rewrite" + #: The dependency could not be moved: the resolver failed, or the manifest forbids the version. + CANNOT_MOVE = "cannot-move" + + +@dataclass(frozen=True) +class Runs: + """The two suite runs a verdict rests on, in the runner's own words. Item 178. + + **Kept rather than recomputed**, because by the time anybody renders an artefact the containers + are gone and the tree has been restored. Everything a reviewer checks is here: what was run, + what it exited with twice, and the line the runner printed about itself. + + The summary lines are `testoutput.verdict_line`'s reading of captured output from an arbitrary + command, so anything that renders them scrubs them first — which is `evidence`'s rule and the + reason this carries the text rather than pre-formatting it. + """ + + command: str + before_exit: int + after_exit: int + before_summary: str = "" + after_summary: str = "" + + +@dataclass(frozen=True) +class Answer: + """A verdict about one upgrade, and the evidence for it.""" + + verdict: Verdict + package: str + was: str + to: str + detail: str = "" + #: The dependency files **as the passing run saw them**, kept only for a clean verdict. + #: + #: This is what item 178 opens a pull request with, and carrying it is not an optimisation. + #: `attempt` restores every file it moved, so working the diff out afterwards would mean running + #: the resolver a second time — and a lock regenerated twice can differ: a version published in + #: between, a different ordering, a registry that answered differently. Publishing files that + #: are not the ones the suite passed against is the defect item 045 is named after, and this is + #: the one place it could come back. + files: dict[str, bytes] = field(default_factory=dict) + #: What the two gates did, for the artefact. `None` for a verdict that never ran two. + runs: Runs | None = None + + @property + def says(self) -> str: + """The claim in DR-0016's own words, which are deliberately narrower than they could be. + + *The suite passed before this change and passes after it* — never *this is safe* and never + *this fixes the vulnerability*. A suite that never exercised the upgraded library says so + by staying green, and a reader who knows that reads this correctly. Widening it here would + be the defect item 171 removed: a claim that reads as more than it measured. + """ + if self.verdict is Verdict.CLEAN: + return ( + f"{self.package} {self.was} → {self.to}: your suite passed before this change and " + f"passes after it. That is what was measured — not that the upgrade is safe, and " + f"not that it fixes anything your suite does not exercise." + ) + if self.verdict is Verdict.BREAKS: + return f"{self.package} {self.was} → {self.to}: this upgrade breaks your suite." + if self.verdict is Verdict.WILL_NOT_INSTALL: + return f"{self.package} {self.was} → {self.to}: the environment could not be built." + if self.verdict is Verdict.ALREADY_RED: + return ( + "your suite does not pass on an untouched checkout, so nothing can be claimed " + "about any upgrade. Nothing was rewritten." + ) + if self.verdict is Verdict.CANNOT_MOVE: + return f"{self.package} {self.was} → {self.to}: could not be moved. {self.detail}" + return f"{self.package}: this file cannot be rewritten safely." + + +#: A `requirements.txt` pin, in five groups so the name can be compared on its own and the version +#: replaced without rebuilding the line: indent, **name**, extras-and-operator, version, tail. +#: +#: Extras, environment markers and comments survive because only the version group is replaced. A +#: line rebuilt from its parsed parts would quietly drop the marker, which changes what gets +#: installed on other platforms — a silent behaviour change from a cosmetic decision. +_PIN = re.compile(r"^(\s*)([A-Za-z0-9._-]+)((?:\s*\[[^\]]*\])?\s*==\s*)([^\s;#]+)(.*)$") + + +def _canonical(name: str) -> str: + """PEP 503 normalisation, because two spellings of one package are one package. + + OSV answers with the canonical name and a requirements file may carry any of its spellings — + `Jinja2`, `jinja_2`, `jinja.2` all name the same distribution. Comparing raw strings would + refuse to rewrite a pin that is plainly there. + """ + return re.sub(r"[-_.]+", "-", name).lower() + + +#: Files whose version strings cannot be edited in place, and the reason, in the words the refusal +#: uses. **Not a "not yet"**: these files carry per-artefact hashes, so a hand-edited version string +#: describes an artefact the hash does not match. The install then fails on the checksum, or — worse +#: — succeeds against a stale cache and the suite runs against a version nobody chose. +#: Dependency files that are a **list**, where the line is the pin and rewriting it is the whole +#: edit. Everything not here is a resolved graph and needs its ecosystem's resolver. +#: +#: Stated as an allow-list rather than as a list of refusals, and a test is why: the refusals used +#: to name three lock files, so the day a reader learns `Cargo.lock` — which has no resolver — it +#: would have been edited by hand with nothing objecting. The safe answer has to be the default. +#: +#: **A predicate rather than a set of names since item 180**, and the reason is that widening the +#: reader without widening this produced a refusal that was *false*: `requirements/prod.txt` was +#: declined as *"a resolved graph rather than a list of versions"*, which is the opposite of what it +#: is. Every upgrade in every layout other than a root `requirements.txt` would have been refused, +#: with a wrong reason each time. `dependencies.is_requirements` is the one place that decides, +#: so a layout that becomes readable becomes editable in the same edit. +def editable_by_hand(source: str) -> bool: + """Whether this file is a list of versions, where rewriting the line is the whole edit.""" + return dependencies.is_requirements(source) + +#: Per-file reasons, for the ones where the generic sentence is not the useful one. +CANNOT_BE_EDITED: dict[str, str] = { + "package-lock.json": "it carries `integrity` hashes per package", + "uv.lock": "it carries a `sha256` per artefact", + "poetry.lock": "it carries a `sha256` per artefact", +} + + +class CannotRewriteError(Exception): + """The dependency file cannot be edited into a valid one. Names the file and the reason.""" + + +def rewrite_pin(text: str, package: str, to: str) -> str: + """`package==old` becomes `package==to`, and everything else on the line survives. + + Extras, environment markers and trailing comments are preserved because only the version group + is replaced — `httpx[http2]==0.27.0 ; python_version >= "3.8" # pinned` keeps all three. + + A `--hash=` line is refused for the same reason a lock file is: the hash describes the + artefact that was pinned, and a version changed out from under it will not install. + """ + wanted = _canonical(package) + out: list[str] = [] + seen = False + for line in text.splitlines(keepends=True): + matched = _PIN.match(line.rstrip("\n")) + if matched and _canonical(matched.group(2)) == wanted: + if "--hash=" in line: + msg = ( + f"{package} is pinned with `--hash=`, and a hash describes the artefact that " + f"was pinned. Changing the version without recomputing it produces a " + f"requirements file that will not install." + ) + raise CannotRewriteError(msg) + ending = "\n" if line.endswith("\n") else "" + indent, name, operator, _, tail = matched.groups() + out.append(f"{indent}{name}{operator}{to}{tail}{ending}") + seen = True + else: + out.append(line) + if not seen: + msg = f"no `{package}==…` line to rewrite" + raise CannotRewriteError(msg) + return "".join(out) + + +def can_rewrite(source: str) -> None: + """Raise `CannotRewrite` when this file's versions must not be edited by hand. + + Called before anything runs, so a project whose only pins are in a lock file is told at once + rather than after paying for a baseline build. + """ + name = source.rsplit("/", 1)[-1] + # A list of versions: the line *is* the pin, so rewriting it is the whole edit. + if editable_by_hand(source): + return + # Anything else is a resolved graph, and only its own tool can move one (item 175). + if resolve.resolver_for(source) is not None: + return + why = CANNOT_BE_EDITED.get(name, "it is a resolved graph rather than a list of versions") + msg = ( + f"{source} cannot be rewritten here: {why}, so editing a version string by hand leaves a " + f"file whose hashes describe the version that was there before. Upgrading it properly " + f"means running that ecosystem's own tool, which this does not do yet — so it refuses " + f"rather than producing a lock file that cannot install." + ) + raise CannotRewriteError(msg) + + +class Box(Protocol): + """What this needs from a sandbox, and nothing else. + + **Structural**, so the tests need no Docker and so this module never imports one — the same + reason `dispatch` takes its box as an argument rather than building one. + """ + + worktree: Path + + def run(self, command: str, timeout: int = 0) -> RunResult: # pragma: no cover - protocol + ... + + +def attempt( + make_box: Callable[[], Box], + *, + tests: str, + source: str, + package: str, + was: str, + to: str, + rebuild: Callable[[str], str | None], + mover: Callable[[Path], str | None] | None = None, + touches: Sequence[str] | None = None, +) -> Answer: + """Baseline, rewrite, rebuild, run again. The three phases of DR-0016. + + `rebuild` takes the rewritten file's text and returns `None` on success or the reason on + failure. A parameter rather than an import because building an image is the caller's business: + this function never learns that Docker exists, which is what makes it testable without it. + + **`make_box` is called twice, and that is the whole correctness of this function.** The second + run has to happen in a box built from the *rebuilt* image; reusing the first one runs the + upgraded project's suite against the environment it had before the upgrade, and reports `clean` + for a version that was never installed. Taking a box rather than a factory is exactly that + defect, and it survived nineteen unit tests before a real Docker run found it — a double has no + image, so nothing in it could have noticed. + """ + # Before the baseline is paid for: a file nothing can move is a fact on disk. + if mover is None: + can_rewrite(source) + move = mover or editing(source, package, to) + guarded = tuple(touches or (source,)) + + # --- phase 0: is there a claim to make at all ------------------------------------------ + box = make_box() + baseline = box.run(tests, GATE_TIMEOUT_SECONDS) + if not baseline.ok: + # Before anything is rewritten and before a second build is paid for. A suite that is + # already red cannot support "passed before and passes after", and blaming the upgrade for + # it would be the same error `dispatch` made until item 043. + return Answer( + Verdict.ALREADY_RED, package, was, to, + detail=_tail(baseline.output), + ) + + # --- phase 1: move the dependency, however this file is moved ------------------------- + # + # **Every file the move can touch is snapshotted, not just the one named.** Found by item 175's + # gate: `npm install` rewrites `package.json` as well as the lock, and item 174 had already + # shown what happens when a candidate leaves anything behind — the next candidate's baseline + # describes the previous one, silently. Restoring only the lock would be that defect one file + # over, invisible in exactly the same way. + before = { + name: (box.worktree / name).read_bytes() + for name in guarded + if (box.worktree / name).exists() + } + + try: + failed = move(box.worktree) + if failed is not None: + return Answer(Verdict.CANNOT_MOVE, package, was, to, detail=failed) + + # --- phase 2: rebuild — where the network is — and run the suite again ------------- + failure = rebuild((box.worktree / source).read_text(encoding="utf-8")) + if failure is not None: + return Answer(Verdict.WILL_NOT_INSTALL, package, was, to, detail=str(failure)) + + # **A new box, on the image the rebuild just produced.** See the docstring: this line is + # the difference between measuring the upgrade and measuring what it replaced. + after = make_box().run(tests, GATE_TIMEOUT_SECONDS) + # **Read inside the `try`, because the `finally` below puts the old versions back.** These + # bytes are the whole of what item 178 opens a pull request with, and they only exist + # between these two lines: after this function returns, the tree describes the version the + # project had before, which is the opposite of what a reviewer would be asked to merge. + moved = { + name: (box.worktree / name).read_bytes() + for name in guarded + if after.ok and (box.worktree / name).exists() + } + finally: + for name, content in before.items(): + (box.worktree / name).write_bytes(content) + + runs = Runs( + command=tests, + before_exit=baseline.exit_code, + after_exit=after.exit_code, + # `None` when the runner printed nothing this reader recognises, and an empty string is + # the honest rendering of that: the exit codes beside it are the claim either way. + before_summary=testoutput.verdict_line(baseline.output) or "", + after_summary=testoutput.verdict_line(after.output) or "", + ) + if after.ok: + return Answer(Verdict.CLEAN, package, was, to, files=moved, runs=runs) + return Answer(Verdict.BREAKS, package, was, to, detail=_tail(after.output), runs=runs) + + +def _tail(output: str, limit: int = 12) -> str: + """The lines a runner uses to say what failed, or the tail when none are recognised. + + Deliberately the same shape as `dispatch.failing_lines` and deliberately not an import of it: + that one is about an agent's attempt and carries its vocabulary. A tail is always better than + the empty string, which is what a stricter matcher produces on an unknown runner. + """ + marked = [ + line for line in output.splitlines() + if line.startswith(("FAILED", "FAIL", "ERROR", "not ok", " ✗", "✗")) + ] + chosen = marked[:limit] if marked else output.splitlines()[-limit:] + more = len(marked) - limit if len(marked) > limit else 0 + text = "\n".join(chosen) + return f"{text}\n… and {more} more" if more > 0 else text + + +def candidates(advisories: object) -> list[str]: + """The fixed versions to try, in the order OSV gave them, without repeats. + + **This is item 172's deferred question, answered the way DR-0016 said it would be.** That item + prints every published fixed version and chooses none, because choosing would mean comparing + versions under two ecosystems' ordering rules. Here there is no need to compare: each is tried + and the suite decides. Hullwork needs no ordering rule per ecosystem because it can execute. + """ + seen: list[str] = [] + for advisory in advisories: # type: ignore[attr-defined] + for version in advisory.fixed: + if version not in seen: + seen.append(version) + return seen + + +@dataclass(frozen=True) +class Report: + """What trying every candidate for one package concluded.""" + + package: str + was: str + answers: tuple[Answer, ...] + + @property + def settled(self) -> Answer | None: + """The first candidate whose suite stayed green, or `None` if none did.""" + return next((a for a in self.answers if a.verdict is Verdict.CLEAN), None) + + +def verify( + *, + tests: str, + source: str, + package: str, + was: str, + versions: Sequence[str], + make_box: Callable[[str], Box], + rebuild: Callable[[str], str | None], + mover: Callable[[Path], str | None] | None = None, + touches: Sequence[str] | None = None, + pending: dict[str, str] | None = None, +) -> Report: + """Try each candidate until one leaves the suite green. Item 174. + + **Stops at the first clean answer** rather than trying them all: the remaining candidates are + higher versions of the same fix, and a project that upgrades further than it has to is a + project taking a larger change than the advisory asked for. + + `make_box` and `rebuild` are parameters for the reason `attempt`'s box is: building images and + starting containers is the caller's business, so this stays testable without Docker. + """ + answers: list[Answer] = [] + for version in versions: + # The mover is built once by the caller and asks for whichever candidate is current, so + # this is where the two are kept in step. + if pending is not None: + pending["version"] = version + answer = attempt( + lambda version=version: make_box(version), # type: ignore[misc] + tests=tests, source=source, + package=package, was=was, to=version, rebuild=rebuild, + mover=mover, touches=touches, + ) + answers.append(answer) + # A red baseline is about the project, not the candidate: trying the next version would ask + # the same broken suite the same question and get the same answer. + if answer.verdict in (Verdict.CLEAN, Verdict.ALREADY_RED): + break + return Report(package, was, tuple(answers)) + + +def editing(source: str, package: str, to: str) -> Callable[[Path], str | None]: + """The mover for a file that is a list of versions: rewrite the line. + + Only `requirements.txt` reaches here — `can_rewrite` has already refused any lock file with no + resolver, and the caller supplies a resolver-backed mover for the ones that have one. + + **Public since item 179**, which needs the same move outside a verdict: a refit has to put the + upgrade into the tree the agent will work in, and a second implementation of *rewrite the pin* + is a second thing that can come to disagree about what a requirements line means. + """ + + def move(worktree: Path) -> str | None: + path = worktree / source + try: + path.write_text(rewrite_pin(path.read_text(encoding="utf-8"), package, to), + encoding="utf-8") + except CannotRewriteError as refused: + return str(refused) + return None + + return move + + +class Needs(StrEnum): + """What a verdict asks of a person, which is the only useful way to order them. + + **Not severity.** OSV publishes one and this does not read it yet, so ordering by it would be + ordering by something unmeasured — the habit DR-0017 exists to break. This orders by what was + actually established: whether a person has to do anything, and how much. + """ + + #: Your suite is red. Nothing else here can be decided until it is not. + FIX_YOUR_SUITE = "fix your suite first" + #: A fix exists, it breaks named tests, and somebody has to make the code fit. + NEEDS_WORK = "needs work" + #: Something is in the way of even trying — no fix published, a manifest that forbids it. + BLOCKED = "blocked" + #: Verified green. The only thing left is to take it. + JUST_TAKE_IT = "ready to take" + + +#: Worst first: what stops everything, then what needs a person, then what is merely stuck, then +#: what needs nothing. A reader who stops after the first section has read the part that mattered. +_ORDER = (Needs.FIX_YOUR_SUITE, Needs.NEEDS_WORK, Needs.BLOCKED, Needs.JUST_TAKE_IT) + + +def needs_of(report: Report) -> Needs: + """What this package asks of a person.""" + verdicts = [a.verdict for a in report.answers] + if Verdict.ALREADY_RED in verdicts: + return Needs.FIX_YOUR_SUITE + if report.settled is not None: + return Needs.JUST_TAKE_IT + if Verdict.BREAKS in verdicts: + return Needs.NEEDS_WORK + return Needs.BLOCKED + + +def broke(report: Report) -> int: + """How many tests the best candidate broke, or 0. + + Used to order within `NEEDS_WORK`, smallest first: the upgrade that breaks two tests is the one + a person can close this afternoon, and putting the twelve-test one above it buries the + achievable under the daunting. + """ + counts = [ + len([line for line in a.detail.splitlines() if line.strip()]) + for a in report.answers + if a.verdict is Verdict.BREAKS + ] + return min(counts) if counts else 0 + + +def ranked(reports: Sequence[Report]) -> list[Report]: + """The queue in the order a person should work it. DR-0018 step 2. + + **This is the answer to the complaint that Renovate cannot answer.** *"Here is every update, + you decide"* is noise because nothing in it is ranked — and ranking requires knowing what each + one does, which requires running them. That is the axis they cannot move. + """ + return sorted( + reports, + key=lambda r: (_ORDER.index(needs_of(r)), broke(r), r.package), + ) + + +def summary(reports: Sequence[Report]) -> dict[Needs, int]: + """How many fall in each bucket. The sentence that replaces forty undecided pull requests.""" + counted = dict.fromkeys(_ORDER, 0) + for report in reports: + counted[needs_of(report)] += 1 + return counted diff --git a/hullwork/cli.py b/hullwork/cli.py index 1008bac..ba0351a 100644 --- a/hullwork/cli.py +++ b/hullwork/cli.py @@ -14,12 +14,13 @@ import json import logging import os +import shutil import signal import subprocess import sys import threading from collections.abc import Callable, Sequence -from contextlib import suppress +from contextlib import ExitStack, suppress from dataclasses import dataclass from datetime import UTC, datetime, timedelta from pathlib import Path @@ -31,22 +32,30 @@ from hullwork import ( __version__, + bump, credentials, db, + dependencies, doctor, + features, lease, operator, + osv, outcomes, page, propose, readiness, recurrence, + refit, + resolve, spend, territory, triage, + upgrades, work, ) from hullwork import decisions as decide +from hullwork import dispatch as dispatch_module from hullwork import upstream as upstream_module from hullwork.config import ConfigError, Settings, get_settings from hullwork.credentials import PushCapability @@ -348,6 +357,741 @@ def _the_image_must_be_able_to_host_a_phase( raise CommandError(refusal) +def _cmd_deps(args: argparse.Namespace, settings: Settings, out: TextIO) -> int: + """Which pinned dependencies have something published against them. Item 172, DR-0016. + + **Standalone, and that is the product claim rather than a convenience.** No forge, no model, no + Docker, no database — a person can run this against their own checkout in the first minute, + before they have decided anything. It is the only half of DR-0016 that needs nothing. + """ + checkout = Path(args.checkout).resolve() + # `--fix` and `--open` are `--verify` plus one more step each: there is nothing to fix that has + # not first been measured breaking, and nothing to open that has not first been measured + # passing. Read once, here, so everything below asks the same question. + opening = bool(getattr(args, "open", False)) + verifying = bool(args.verify or getattr(args, "fix", False) or opening) + # **Before the lock files are read and before OSV is asked**, for `_manifest_for_verify`'s + # reason and with more at stake: `--open` is the one flag here that can write to somebody's + # repository, and finding out after two container builds that there is no credential is a + # refusal that arrives after the work it invalidates. + code_forge = _forge_for_opening(settings, checkout) if opening else None + if getattr(args, "fix", False): + # **Item 048's finding, on the path that had not learned it** (found by running `--fix` for + # the first time, 2026-08-09). This was raised inside `refit.run`, so it arrived *after* + # every container had been built and every suite run — the most expensive place available — + # and it arrived as a `WiringError` traceback rather than as a refusal, which item 120 is + # about. The message itself was right; where and how it appeared was not. + _refuse_without_a_model(settings) + # **Before the lock files are read and before OSV is asked.** Found by running it: validating + # the manifest after the report meant paying for a network round trip and a full listing to be + # told a file was missing — and the refusal arrived interleaved with the output it invalidated. + manifest = _manifest_for_verify(checkout) if verifying else None + paths = _tracked_files(checkout) + + def read(path: str) -> str | None: + try: + return (checkout / path).read_text(encoding="utf-8", errors="replace") + except OSError: + return None + + pinned = dependencies.read_lockfiles(paths, read) + if not pinned: + raise CommandError( + f"no lock file in {checkout}: looked for " + f"{', '.join(dependencies.WHAT_IS_LOOKED_FOR)}.\n" + f" A declaration is a range and a range does not say what your build resolved to, so " + f"there is nothing here a vulnerability database can be asked about. Commit a lock " + f"file, or pin with `==` in requirements.txt." + ) + + sources = sorted({d.source for d in pinned}) + print(f"{len(pinned)} pinned dependencies, from {', '.join(sources)}", file=out) + + # Said before the answer rather than after it: a file whose ranges were skipped reports fewer + # dependencies than it has, and a reader who learns that afterwards has already believed it. + # + # **Every requirements file that was read, not the root one by name** (item 180). Keyed off the + # same predicate the reader uses, so a layout that becomes readable becomes countable in the + # same edit — two places deciding what a requirements file is would eventually disagree, and + # the half that goes quiet is this one. + for source in sources: + if dependencies.is_requirements(source): + text = read(source) or "" + skipped = dependencies.unpinned(text) + if skipped: + print( + f" {skipped} line(s) in {source} are ranges rather than `==` pins and were " + f"not checked", + file=out, + ) + + # **Quiet here and nowhere else.** httpx2 logs every request at INFO, which is right for the + # dispatcher — those lines are how an attempt gets diagnosed — and wrong for a report a person + # reads. This is the command a stranger runs first, and one `HTTP Request: POST …` line in the + # middle of its output is the kind of friction the cold evaluations kept finding. + logging.getLogger("httpx2").setLevel(logging.WARNING) + with osv.Osv() as database: + findings = database.affected(pinned) + + if not findings: + print("\nNothing published against any of those versions.", file=out) + return 0 + + print(f"\n{len(findings)} with a published advisory:\n", file=out) + for finding in findings: + dep = finding.dependency + print(f" {dep.name} {dep.version} ({dep.ecosystem}, {dep.source})", file=out) + for advisory in finding.advisories: + if advisory.has_a_fix: + where = " or ".join(advisory.fixed) + ends = f"fixed in {where}" + else: + # Not "no fix found": the advisory publishes none, which is a fact about the + # advisory rather than a gap in this reading. + ends = "no fixed version is published — there is no upgrade to attempt" + print(f" {advisory.id}: {ends}", file=out) + if advisory.summary: + print(f" {advisory.summary}", file=out) + print(f" {advisory.url}", file=out) + print("", file=out) + + if not verifying: + print( + "Whether any of these upgrades survives your own test suite is a different question, " + "and nothing above has run it. `--verify` runs it.", + file=out, + ) + return 0 + + assert manifest is not None # noqa: S101 - built above when --verify is set + reports = _verify_upgrades(checkout, paths, read, manifest, findings, out) + if opening: + assert code_forge is not None # noqa: S101 - built above when --open is set + _open_the_ones_that_pass(code_forge, checkout, manifest, findings, reports, out) + if not getattr(args, "fix", False): + return 0 + return _fix_the_ones_that_break( + args, settings, checkout, paths, manifest, findings, reports, out + ) + + +def _manifest_for_verify(checkout: Path) -> Manifest: + """The manifest `--verify` needs, or a refusal naming exactly what is missing. Item 174. + + **Called before the lock files are read and before OSV is asked**, because everything it checks + is knowable from disk. Validating afterwards spent a network round trip and a full listing to + tell somebody a file was missing, and printed the refusal interleaved with the report it had + just invalidated. Found by running it. + """ + manifest_path = checkout / MANIFEST_FILENAME + if not manifest_path.exists(): + raise CommandError( + f"--verify needs a {MANIFEST_FILENAME} in {checkout}: it says which image your tests " + f"run in and what the test command is, and neither can be guessed.\n" + f" `hullwork propose --checkout {checkout}` writes one from your CI configuration." + ) + manifest = parse_manifest(manifest_path.read_text(encoding="utf-8")) + if manifest.runtime is None or not manifest.runtime.base: + raise CommandError( + "--verify needs `runtime.base`: an image your tests already run in. Without it there " + "is nothing to build the upgrade into." + ) + if not manifest.tests: + raise CommandError( + "--verify needs `tests`: the command that runs your suite. That suite is the whole " + "verdict — without it there is nothing to ask." + ) + return manifest + + +def _verify_upgrades( + checkout: Path, + paths: Sequence[str], + read: Callable[[str], str | None], + manifest: Manifest, + findings: Sequence[osv.Finding], + out: TextIO, +) -> list[bump.Report]: + """Apply each published fix in a sandbox and let the project's own suite decide. Item 174. + + Returns the reports as well as printing them, because item 179 has to act on the same ones the + reader was shown — recomputing them would be a second verdict about the same upgrade, and the + two could differ by the time anybody noticed. + """ + print("\n--- verifying each published fix against your own suite ---\n", file=out) + reports: list[bump.Report] = [] + for finding in findings: + dep = finding.dependency + versions = bump.candidates(finding.advisories) + refusal = _cannot_be_verified(dep, manifest, versions) + if refusal is not None: + print(f" {dep.name}: {refusal}\n", file=out) + # **Counted, not merely printed** (item 182). A refusal that only goes to the terminal + # is absent from the summary below, so a run that could verify none of six reads as + # `0 blocked` — and the one number in this whole command that a person acts on is a + # count. "I could not verify this" is a first-class answer and it has to be in the + # tally, which is the property `docs/what-hullwork-is.md` puts second. + reports.append( + bump.Report( + package=dep.name, + was=dep.version, + answers=( + bump.Answer( + bump.Verdict.CANNOT_MOVE, dep.name, dep.version, + versions[0] if versions else "", detail=refusal, + ), + ), + ) + ) + continue + report = _verify_one(checkout, paths, read, manifest, dep, versions, out) + if report is not None: + reports.append(report) + + _print_the_queue(reports, out) + return reports + + +def _forge_for_opening(settings: Settings, checkout: Path) -> object: + """The credential `--open` pushes through, or a refusal that says what is missing. Item 178. + + **It is `HULLWORK_FORGE_CODE_TOKEN`, and the operator decided that on 2026-08-09.** The + alternative was a third token of its own, and it was declined for a reason worth keeping: a + third token would need exactly the same scope — `write:repository` — so it would be an + audit boundary rather than a capability boundary, which is not what the split between + `forge_token` and `forge_code_token` is. That one is real and was measured (item 073); a + same-scope sibling would be the same power under another name. + + What that decision costs is one sentence, and it has been paid: `config.py` no longer calls this + *the credential an agent pushes through*, because this path opens pull requests with no agent + having run. Nobody may infer "a model was called" from the fact that something was pushed. + + Called before the lock files are read and before OSV is asked, so a missing credential costs + nothing to discover. + """ + forge = make_code_forge(settings) + if forge is None: + raise CommandError( + "--open needs HULLWORK_FORGE_URL and HULLWORK_FORGE_CODE_TOKEN: it opens pull " + "requests, which is the one thing in `deps` that writes to your repository.\n" + " Everything else here needs no account anywhere — `--verify` runs your suite " + "against each upgrade and prints the answer, and it is the honest way to see what " + "this would open before letting it." + ) + coordinate = _coordinate_from(_origin_url(checkout)) + if coordinate == "owner/name": + forge.close() + raise CommandError( + f"--open needs to know which repository this is, and {checkout} has no usable `origin` " + f"remote to read it from.\n" + f" Add one, or run without --open: the verification does not need a coordinate " + f"because it opens nothing." + ) + return forge + + +def _refuse_without_a_model(settings: Settings) -> None: + """Refuse `--fix` before anything is built when no model credential is configured. Item 048. + + Knowable from the settings and nothing else, so it costs nothing to answer — which is the only + reason a refusal belongs this early. `--verify` is untouched: it calls no model and must keep + needing no credential of any kind, which is the property `deps` is sold on. + """ + try: + work._model_credential(settings) + except work.WiringError as exc: + raise CommandError( + f"{exc}\n" + f" `--verify` on its own needs no credential at all and still runs your suite " + f"against every published fix — it is `--fix`, which asks an agent to change your " + f"code, that needs a model." + ) from exc + + # **The other thing every agent run needs, and it was found the expensive way** (item 191). + # `deps --fix` died at the gateway after OSV, four image builds and two suite runs, on the + # first real model call this command ever made. Asked here, where the credential is asked. + from hullwork.sandbox.net import why_the_gateway_cannot_start + + missing = why_the_gateway_cannot_start() + if missing: + raise CommandError(missing) + + +def _open_the_ones_that_pass( + code_forge: object, + checkout: Path, + manifest: Manifest, + findings: Sequence[osv.Finding], + reports: Sequence[bump.Report], + out: TextIO, +) -> None: + """Open one draft pull request per verified-green upgrade. Item 178, DR-0018 step 3. + + The end of DR-0018's sentence, and the first thing in this line of work that needs a credential + able to write: *we open the thirty-one that pass and tell you what to do with the nine that do + not*. The nine are already on screen by the time this runs. + """ + from hullwork import trial + + eligible = upgrades.eligible(reports) + if eligible and not manifest.autofix.open_upgrades: + # **Said before the count, and as a decision** (DR-0019, item 187). This is the first thing + # a project can refuse while Hullwork is perfectly able to do it, so it must not read as a + # part that is missing — the verification above ran and its answer stands. + print( + f"\n{len(eligible)} upgrade(s) passed your suite and **none was opened**: this " + f"project has not permitted it.\n" + f" Set `autofix: {{open_upgrades: true}}` in {MANIFEST_FILENAME} if you want them " + f"opened. It is false by default because having the credential is not the same as " + f"having agreed, and the report above is what there is to act on either way.", + file=out, + ) + return + if not eligible: + print( + "\nNothing was verified green, so there is nothing to open. That is a result rather " + "than a failure: the report above is what there is to act on.", + file=out, + ) + return + + # **The commit the gates ran against**, read from the checkout that was verified rather than + # from the forge's idea of its own default branch. The base can move while a verification runs, + # and a branch rooted at wherever it points now contains a tree nobody tested. + base = trial.head_sha(checkout) + if base == "working tree": + print( + "\nThis checkout is not a git repository, so there is no commit to root a pull " + "request at and nothing was opened. What was verified is above.", + file=out, + ) + return + + by_package = {f.dependency.name: f.advisories for f in findings} + coordinate = _coordinate_from(_origin_url(checkout)) + print(f"\n--- opening {len(eligible)} verified-green upgrade(s) on {coordinate} ---\n", + file=out) + opened = upgrades.open_them( + code_forge, repo=coordinate, reports=eligible, + advisories=by_package, base_sha=base, + permitted=manifest.autofix.open_upgrades, + ) + for where in opened: + print(f" {where}", file=out) + if len(opened) < len(eligible): + # Never silence: a package that produced no pull request is either already open from a + # previous run or something the forge refused, and both are facts a reader needs. + print( + f"\n {len(eligible) - len(opened)} opened nothing — already open from an earlier " + f"run, or refused by the forge. The log says which.", + file=out, + ) + print( + f"\nAll drafts, rooted at {base[:12]}, one per package. Nobody merges them but you.", + file=out, + ) + + +def _fix_the_ones_that_break( + args: argparse.Namespace, + settings: Settings, + checkout: Path, + paths: Sequence[str], + manifest: Manifest, + findings: Sequence[osv.Finding], + reports: Sequence[bump.Report], + out: TextIO, +) -> int: + """Hand each broken upgrade to an agent and let the gates decide. Item 179, DR-0018 step 4. + + **The middle of the queue, which is the part nobody else ships.** The verified-green ones need + no agent and are item 178's to deliver; the blocked ones have nothing to try. What is left is + *six break, tests named*, and until this existed the honest answer to those was a list. + + Worst-first through `bump.ranked`, so a run that is interrupted has spent its money on the ones + a person would have started with. + """ + into = Path(args.into).resolve() + queue: list[tuple[bump.Report, refit.Upgrade]] = [] + for report in bump.ranked(reports): + if bump.needs_of(report) is not bump.Needs.NEEDS_WORK: + continue + finding = next( + ( + f for f in findings + if f.dependency.name == report.package and f.dependency.version == report.was + ), + None, + ) + if finding is None: # pragma: no cover - every report was built from one + continue + first = finding.advisories[0] if finding.advisories else None + upgrade = refit.from_report( + report, + source=finding.dependency.source, + guarded=refit.guarded_for(finding.dependency.source), + advisory=first.id if first else "", + url=first.url if first else "", + ) + if upgrade is not None: + queue.append((report, upgrade)) + + if not queue: + print( + "\nNothing broke that an agent could be asked about, so --fix had nothing to do.", + file=out, + ) + return 0 + + print(f"\n--- asking an agent to make {len(queue)} upgrade(s) fit ---\n", file=out) + failures = 0 + for _report, upgrade in queue: + print(f" {upgrade.package} {upgrade.was} → {upgrade.to}", file=out) + try: + outcome = refit.run( + settings, checkout, manifest, upgrade, + present=paths, into=into, repo=_coordinate_from(_origin_url(checkout)), + ) + except refit.NotUpgradableError as refused: + # The upgrade never went into the tree, so no attempt was begun and nothing was spent. + # Said as the fact about the project that it is, in the resolver's own words. + print(f" could not be applied: {refused}\n", file=out) + failures += 1 + continue + except work.WiringError as broken: + # Belt and braces over the refusal above: an engine this build does not know is a + # per-project fact, and a queue of six must not end at the first one that has it. + print(f" not attempted: {broken}\n", file=out) + failures += 1 + continue + print(f" {outcome.outcome.value}: {outcome.detail.splitlines()[0]}", file=out) + if outcome.pull_request: + print(f" written to {outcome.pull_request}\n", file=out) + else: + print("", file=out) + if outcome.outcome not in (AttemptOutcome.PR_OPEN, AttemptOutcome.PR_OPEN_LINT_FAILED): + failures += 1 + + print( + f"{len(queue) - failures} of {len(queue)} now pass your suite with the upgrade still " + f"applied. Nothing was opened anywhere: read what is in {into} and decide.", + file=out, + ) + return 0 + + +def _cannot_be_verified( + dep: dependencies.Dependency, manifest: Manifest, versions: Sequence[str] +) -> str | None: + """Why this upgrade cannot be measured at all, or `None` when it can. Item 182. + + Every reason here is knowable from the manifest and the finding, so all of them are answered + **before a container is built** — the same rule `can_rewrite` follows and for the same reason. + + **The one that was missing is the one a real repository found immediately.** `encode/flask` pins + four of its five advisory-carrying packages in `examples/celery/requirements.txt`, which is not + a file its image installs from. The image is built from `runtime.dependencies`; rewriting + anything outside that set changes no byte the build reads, so `dependency_digest` does not move, + the image is reused, and the suite passes exactly as it passed before. + + **Measured on 2026-08-09 against a real daemon**, on a tree with `requirements.txt` declared and + `extras/requirements.txt` not: + + [ready to take] jinja2 2.4.1 → 2.10.1 + $ docker run --rm python -c "import jinja2" + ModuleNotFoundError: No module named 'jinja2' + + *Ready to take*, for a package **not installed in the environment its suite ran in** — and with + item 178's `--open`, a pull request. That is item 174's defect arriving by a second route, and + it is the exact artefact DR-0017 says this product exists to prevent. + + An empty `runtime.dependencies` is not this case: `_verify_one` falls back to the file that + pins, so the build does read it. + """ + runtime = manifest.runtime + assert runtime is not None # noqa: S101 - `_manifest_for_verify` refused a manifest without one + + if runtime.install == "none": + # **The worse half of the same finding, and it is the default value.** With `install: none` + # the generated Dockerfile copies no dependency file and runs no installer + # (`sandbox/image.py`: `if runtime.install != "none" and runtime.dependencies`), so whatever + # the project's environment holds came from `runtime.base` and cannot be moved by editing a + # lockfile. DR-0007 makes *the project brings its own image* the primary path, so this is + # not an edge case — it is most projects. + # + # **Measured on 2026-08-09** against a base image carrying `jinja2 3.0.0`, on a checkout + # pinning `jinja2==2.4.1`: + # + # [ready to take] jinja2 2.4.1 → 2.10.1 + # $ docker run --rm python -c "import jinja2; print(...)" + # 3.0.0 + # 3.0.0 + # + # Neither version in the claim was ever installed. The "before" run did not use 2.4.1 and + # the "after" run did not use 2.10.1; both used a third version, and the verdict said the + # suite passed before the change and after it — which was true, and about nothing. + return ( + f"your manifest sets `install: none`, so the image is `{runtime.base}` exactly as it " + f"comes and nothing is installed from {dep.source}. Changing a version there cannot " + f"change what your suite runs against, so no verdict here would be about this " + f"upgrade.\n" + f" This is the primary path in DR-0007 and it is not a defect in your project: an " + f"image that already carries your dependencies is upgraded by rebuilding it, not by " + f"editing a pin. Declare an installer and the file it reads if you want this measured." + ) + + if runtime.dependencies and dep.source not in runtime.dependencies: + declared = ", ".join(runtime.dependencies) + return ( + f"{dep.source} is not one of the files your image is built from ({declared}), so " + f"changing a version in it changes nothing the suite would run against. Whatever this " + f"upgrade does, your suite cannot say — and a green run here would mean only that the " + f"file nobody installs from was edited.\n" + f" Declare it in `runtime.dependencies` if your build should read it, or upgrade it " + f"by hand: this is a fact about what your image installs, not about the upgrade." + ) + + if not versions: + # Not "no fix found": the advisory publishes none, which is a fact about the advisory rather + # than a gap in this reading. + return "no published fixed version, so there is nothing to try" + + try: + bump.can_rewrite(dep.source) + except bump.CannotRewriteError as refused: + return str(refused) + return None + + +def _print_the_queue(reports: Sequence[bump.Report], out: TextIO) -> None: + """The ranked report. DR-0018 step 2, and the whole of what it is for. + + **This is the part Renovate cannot produce.** Its documented weakness is that it hands over + every update undecided — noise rather than signal — and ranking them requires knowing what each + one does, which requires running them. Everything above ran them; this is where that is spent. + """ + if not reports: + return + counted = bump.summary(reports) + print("\n=== what to do with them ===\n", file=out) + for needs in ( + bump.Needs.FIX_YOUR_SUITE, bump.Needs.NEEDS_WORK, + bump.Needs.BLOCKED, bump.Needs.JUST_TAKE_IT, + ): + # Every bucket, including the empty ones: a reader has to be able to tell "none of these" + # from "this was not counted", and only one of those is good news. + print(f" {counted[needs]:>3} {needs.value}", file=out) + + print("", file=out) + for report in bump.ranked(reports): + needs = bump.needs_of(report) + settled = report.settled + where = f" → {settled.to}" if settled is not None else "" + broke = bump.broke(report) + cost = f", {broke} test(s) to fix" if broke and settled is None else "" + print(f" [{needs.value}] {report.package} {report.was}{where}{cost}", file=out) + print("", file=out) + + +def _verify_one( + checkout: Path, + paths: Sequence[str], + read: Callable[[str], str | None], + manifest: Manifest, + dep: dependencies.Dependency, + versions: list[str], + out: TextIO, +) -> bump.Report | None: + """One package, every candidate, each in its own sandbox.""" + from hullwork import trial + from hullwork.sandbox import image as image_module + from hullwork.sandbox.run import Sandbox + + runtime = manifest.runtime + assert runtime is not None # noqa: S101 - refused above, and mypy cannot see that + tests = manifest.tests or "" + source = dep.source + + # Which candidate `verify` is on, so a resolver-backed mover knows what to ask for. + _pending: dict[str, str] = {"version": ""} + + with ExitStack() as stack: + worktree = dispatch_module.prepare_worktree(checkout) + stack.callback(shutil.rmtree, worktree, ignore_errors=True) + + def files_now() -> dict[str, bytes]: + """The declared dependency files as they are in the worktree right now. + + Read per build rather than once: the rewrite happens between the two, and the second + build has to see it — `image.dependency_digest` then makes the tag differ by itself, + which is what turns the second build into a real rebuild. + """ + found: dict[str, bytes] = {} + for path in runtime.dependencies or [source]: + whole = worktree / path + if whole.exists(): + found[path] = whole.read_bytes() + return found + + built: dict[str, str] = {} + # **The commit the source is at, when the source goes into the build at all** (item 182). + # Read once: it is what `image_tag` hashes to decide whether an image can be reused, and the + # source does not move between candidates — only the dependency files do, and those are + # hashed separately by `dependency_digest`. + source_ref = trial.head_sha(checkout) if runtime.install_needs_source else None + + def build_now() -> str | None: + try: + image = image_module.build( + runtime, files_now(), None, + # **Item 113's fix, which this path never inherited** (found by item 182, on + # the first third-party tree it was pointed at). The build context holds the + # declared dependency files and never the source, and three ordinary installers + # read the source anyway: a `requirements.txt` beginning `-e .`, a `Gemfile` + # that says `gemspec`, and `mvn test`. Measured on `encode/httpx`, whose first + # requirement is `-e .[brotli,cli,http2,socks,zstd]`: + # + # ERROR: file:///work does not appear to be a Python project: + # neither 'setup.py' nor 'pyproject.toml' found. + # + # Reported as *your own environment does not build*, which was true of what we + # built and false of the project. Ruby, Java and PHP are on the roadmap as + # stacks whose attempts work; every one of them reaches this the same way. + source=worktree if runtime.install_needs_source else None, + source_ref=source_ref, + ) + except image_module.ImageBuildError as failed: + return str(failed) + built["tag"] = image.tag + return None + + # The baseline image, before anything is rewritten. A failure here is the project's + # environment, not the upgrade's, so it is said as that. + problem = build_now() + if problem is not None: + print(f" {dep.name}: your own environment does not build — {problem}\n", file=out) + return None + + made = {"n": 0} + + def make_box(_version: str) -> bump.Box: + """A box on **whatever image `built` holds right now**. + + Called once per run rather than once per candidate, because the second run has to + happen on the rebuilt image — reusing the first box measures the upgraded project's + suite against the environment it replaced, and reports `clean` for a version that was + never installed. Found by a real Docker run; see item 174. + """ + made["n"] += 1 + # Built from the worktree **as it is now**, which is what makes each run happen in the + # environment its own tree describes. Cheap when nothing changed: the digest is the + # content, so `build` reuses the existing image rather than making another. + build_now() + box = Sandbox(image=built["tag"], worktree=worktree) + stack.callback(box.cleanup) + box.ensure_volume( + f"hullwork-deps-{os.getpid()}-{made['n']}", + # **Item 114's fix, which this path never inherited either** (item 182). Anything + # the build installed under `/work` is erased by the worktree volume unless the + # image goes down first — which is what `vendor/` is for PHP, and the reason that + # item exists. Off unless the project asks, so every other project takes the path + # it took yesterday. + seed_from_image=runtime.install_needs_source, + ) + return box # type: ignore[return-value] + + # How this file is moved, and everything moving it can touch (items 175 and 176). For a + # list the line is the pin; for a resolved graph only the ecosystem's own tool may move it, + # and `touches` is what stops one candidate leaving a widened range behind for the next. + resolver = resolve.resolver_for(source) + mover = None + guarded: tuple[str, ...] = (source,) + if resolver is not None: + guarded = resolve.touches(resolver) + here = [p for p in paths if p.rsplit("/", 1)[-1] in set(resolver.needs)] + + def mover(worktree: Path, _r: resolve.Resolver = resolver) -> str | None: + outcome = resolve.upgrade( + resolver=_r, worktree=worktree, package=dep.name, version=_pending["version"], + present=here, run=resolve.in_a_container, + ) + return None if outcome.ok else f"{outcome.outcome.value}: {outcome.detail}" + + report = bump.verify( + tests=tests, source=source, package=dep.name, + was=dep.version, versions=versions, + make_box=make_box, rebuild=lambda _text: build_now(), + mover=mover, touches=guarded, pending=_pending, + ) + + for answer in report.answers: + print(f" {answer.says}", file=out) + if answer.detail: + for line in answer.detail.splitlines()[:8]: + print(f" {line}", file=out) + print("", file=out) + return report + + +def _cmd_features(args: argparse.Namespace, settings: Settings, out: TextIO) -> int: + """What this can do for your project, and what it cannot. Item 186. + + **The same rules as `projects lanes --checkout .`**, which is the precedent this copies: a + checkout, no credential of any kind, nothing executed, nothing written and no socket opened. It + answers before you have decided anything, which is the only moment the answer is worth having. + + Settings are read for **which variables are set and never for their values**, so this can say + *needs a model credential, and none is configured* while holding none — and can be run by + somebody who has configured nothing at all. + """ + checkout = Path(args.checkout).resolve() + + manifest = None + manifest_path = checkout / MANIFEST_FILENAME + if manifest_path.exists(): + try: + manifest = parse_manifest(manifest_path.read_text(encoding="utf-8"), + source=str(manifest_path)) + except ManifestError as broken: + # Said and carried on. A manifest that does not parse is a fact about this checkout and + # answers half the questions below by itself; refusing here would withhold the other + # half over a file the reader is about to fix anyway. + print(f"{manifest_path} does not parse, so everything it would answer reads as no:\n" + f" {broken}\n", file=out) + + known = features.Checkout( + paths=tuple(_tracked_files(checkout)), + manifest=manifest, + configured=frozenset( + name + for name, present in ( + (features.MODEL_KEY, settings.model_key is not None), + (features.CODE_TOKEN, settings.forge_code_token is not None), + ("origin", _origin_url(checkout) is not None), + ) + if present + ), + ) + + print(f"What Hullwork can do for {checkout.name}, and what it cannot.\n", file=out) + answers = features.examine(known) + for line in features.lines(answers): + print(line, file=out) + + print( + "Every limit above is true whether or not the feature is available — that is what a limit " + "is. Nothing here ran, opened a socket or needed a credential.", + file=out, + ) + if features.INSTANCE_SHAPED: + print( + "\nAnswered by `hullwork doctor` on the instance rather than here, because a checkout " + "cannot know them: " + ", ".join(features.INSTANCE_SHAPED) + ".", + file=out, + ) + return 0 + + def _propose_entry(args: argparse.Namespace, settings: Settings, out: TextIO) -> int: """Standalone with a checkout, database-backed with a repo — same shape as `lanes`. @@ -366,16 +1110,33 @@ def _propose_entry(args: argparse.Namespace, settings: Settings, out: TextIO) -> return _cmd_propose(args, session, settings, out) -def _coordinate_of(checkout: Path) -> str: - """`owner/name` for a local checkout, from its `origin` remote, or a visible placeholder. +def _tracked_files(checkout: Path) -> list[str]: + """The checkout's tracked files, which is what a forge would serve. - A manifest's `git.repo` is validated as `owner/name` (`manifest.py`), so the directory's own - name would produce a proposal that cannot parse — the one thing a proposal must never do, since - its whole purpose is to be committed. The remote is where that coordinate exists locally. + **Tracked rather than walked**, and the reason is the same for both callers: a walk reads + `.venv/` and `node_modules/`, so a proposal would come from a cache and a dependency report + would be about somebody else's dependencies. + """ + listed = subprocess.run( # noqa: S603 + ["git", "-C", str(checkout), "ls-files"], # noqa: S607 + capture_output=True, + text=True, + check=False, + ) + if listed.returncode != 0: + raise CommandError( + f"could not list the files in {checkout}: it is not a git checkout, and this reads " + f"tracked files so what it reports matches what a forge would serve.\n" + f" {listed.stderr.strip()}" + ) + return [line for line in listed.stdout.splitlines() if line] - When there is no usable remote the placeholder is `owner/name` verbatim: it fails validation - loudly and reads as something to replace, which is the same choice as `REPLACE-ME` for - `group_add`. A plausible-looking wrong value would be committed. + +def _origin_url(checkout: Path) -> str | None: + """The `origin` remote's URL, or `None` when there is not one to have. + + One call, two readers: the coordinate below and the forge that holds it (item 171). Asking + git twice for the same string would let the two answers disagree about the same repository. """ url = subprocess.run( # noqa: S603 ["git", "-C", str(checkout), "remote", "get-url", "origin"], # noqa: S607 @@ -383,8 +1144,22 @@ def _coordinate_of(checkout: Path) -> str: text=True, check=False, ) - if url.returncode == 0: - trimmed = url.stdout.strip().removesuffix(".git") + return url.stdout.strip() if url.returncode == 0 else None + + +def _coordinate_from(url: str | None) -> str: + """`owner/name` out of a remote URL, or a visible placeholder. + + A manifest's `git.repo` is validated as `owner/name` (`manifest.py`), so the directory's own + name would produce a proposal that cannot parse — the one thing a proposal must never do, since + its whole purpose is to be committed. The remote is where that coordinate exists locally. + + When there is no usable remote the placeholder is `owner/name` verbatim: it fails validation + loudly and reads as something to replace, which is the same choice as `REPLACE-ME` for + `group_add`. A plausible-looking wrong value would be committed. + """ + if url: + trimmed = url.removesuffix(".git") # `git@host:owner/name` and `https://host/owner/name` both end in the two segments wanted, # and anything else falls through to the placeholder rather than being guessed at. parts = trimmed.replace(":", "/").rstrip("/").split("/") @@ -410,19 +1185,7 @@ def propose_from_local_ci(checkout: Path) -> str | None: `the_recipe_its_toolchain_needs` takes a reader, so neither knows where each came from. Tracked files only, to match what a forge serves — a walk would read `.venv/` and propose from a cache. """ - listed = subprocess.run( # noqa: S603 - ["git", "-C", str(checkout), "ls-files"], # noqa: S607 - capture_output=True, - text=True, - check=False, - ) - if listed.returncode != 0: - raise CommandError( - f"could not list the files in {checkout}: it is not a git checkout, and this reads " - f"tracked files so the proposal matches what a forge would serve.\n" - f" {listed.stderr.strip()}" - ) - paths = [line for line in listed.stdout.splitlines() if line] + paths = _tracked_files(checkout) def read(path: str) -> str | None: try: @@ -430,11 +1193,15 @@ def read(path: str) -> str | None: except OSError: return None + origin = _origin_url(checkout) for candidate in propose.find(paths): text = read(candidate) if text is None: continue - proposal = propose.read(_coordinate_of(checkout), candidate, text) + proposal = propose.read(_coordinate_from(origin), candidate, text) + # Which forge holds this, when the host says so (item 171). Set here rather than passed + # into `read`, which parses CI text and has no business knowing about remotes. + proposal.remote_host = propose.host_of_remote(origin) if proposal.found_anything: checked = propose.only_files_that_exist(proposal, paths) return propose.render( @@ -1348,6 +2115,10 @@ def _cmd_status( } json_merged, json_holding, json_recurred = recurrence.counted(session) payload["attempts"] = outcomes.funnel(session).as_dict() + # Item 183: the parts, so an operator computes their own ratio. Never a percentage here + # either — six samples do not carry that precision, and a number this product publishes + # about itself is the one place that matters most. + payload["desk"] = outcomes.desk(session).as_dict() spent = spend.per_instance( session.query(Attempt).all(), spend.Prices.from_settings(settings) ) @@ -1441,6 +2212,17 @@ def _cmd_status( file=out, ) + # **First, because DR-0017 signed for it** (item 183). Everything below this block has + # *attempts* as its denominator and therefore answers *of the attempts we made, how did they + # go*. This one has **what arrived** as its denominator, which is the question the accepted + # decision says the product is measured by — and it is above the others because a reader who + # stops after one block should have read the one that can embarrass us. + desk_said = outcomes.desk_lines(outcomes.desk(session)) + if desk_said: + print("\n The desk:", file=out) + for line in desk_said: + print(f" - {line}", file=out) + # Item 119, and the same question as the line above at a different distance: that one is about # fixes that landed, this one about what became of every attempt that was made. Counts, never a # percentage — `outcomes` says why, and the two most important numbers in it are the ones a @@ -2810,6 +3592,86 @@ def build_parser() -> argparse.ArgumentParser: proposing.add_argument("--forge", default="forgejo", choices=SUPPORTED_FORGES) proposing.set_defaults(standalone=_propose_entry) + depending = subparsers.add_parser( + "deps", + help="which pinned dependencies have a published vulnerability", + description=( + "Reads the lock files a checkout carries, asks OSV what is published against those " + "exact versions, and prints what came back with the version that ends each one.\n\n" + "Needs no credential of any kind: no forge, no model, no Docker, no database. The one " + "host it contacts is OSV's public API, which takes no key and no account. Lock files " + "rather than declarations, because a declaration is a range and a range does not say " + "what your build resolved to.\n\n" + "It proposes nothing and changes nothing. Whether an upgrade survives your own test " + "suite is a separate question, and answering it is what the sandbox is for.\n\n" + "Your checkout is never written to, by any of these flags: `--verify` and `--fix` work " + "in a copy, and what `--fix` produces is written where you point `--into` — for you to " + "read. Nothing is opened on any forge." + ), + ) + depending.add_argument( + "--checkout", default=".", help="the checkout to read (default: the current directory)" + ) + depending.add_argument( + "--verify", + action="store_true", + help=( + "take each published fix, apply it, and run your own test suite against it in a " + "sandbox — reporting whether the upgrade holds, breaks your suite (naming the tests) " + "or will not install. Needs a hullwork.yml and the Docker daemon; the report without " + "this flag needs neither. No model credential either way: there is no agent in this " + "path." + ), + ) + depending.add_argument( + "--fix", + action="store_true", + help=( + "for the upgrades that break your suite, ask an agent to change your code so they fit " + "— then run your suite again with the upgrade still applied, and check the version is " + "still pinned afterwards. Implies --verify. This is the only part of `deps` that calls " + "a model, so it needs a model credential; it still needs no forge and opens nothing, " + "and what it produced is written to --into for you to read." + ), + ) + depending.add_argument( + "--open", + action="store_true", + help=( + "open a draft pull request for every upgrade that passed your suite — one per package, " + "never a batch, rooted at the commit the runs were made against. Implies --verify. " + "This is the only flag here that writes to your repository, and it needs " + "HULLWORK_FORGE_URL and HULLWORK_FORGE_CODE_TOKEN. Nothing that broke, nothing that " + "was blocked, and nothing whose baseline was red is ever opened." + ), + ) + depending.add_argument( + "--into", + default="hullwork-refits", + help="where to write what the fix attempts produced (default: ./hullwork-refits)", + ) + depending.set_defaults(standalone=_cmd_deps) + + featuring = subparsers.add_parser( + "features", + help="what Hullwork can do for this project, and what it cannot", + description=( + "Reads your checkout and your hullwork.yml and says, feature by feature, whether this " + "instance can do it for you — and when it cannot, which requirement is missing and " + "what to do about it.\n\n" + "Needs no credential of any kind. It runs nothing, opens no socket, starts no " + "container and writes nothing: it is a reading of what you already have, meant to be " + "run before you have decided anything.\n\n" + "Every feature also carries what it cannot do **even when it is available**, because a " + "limit you meet after adopting something is a limit you found the expensive way. " + "Variables are read for whether they are set, never for their values." + ), + ) + featuring.add_argument( + "--checkout", default=".", help="the checkout to read (default: the current directory)" + ) + featuring.set_defaults(standalone=_cmd_features) + laning = actions.add_parser( "lanes", help="show which of this repository's files the instance keeps a human on", diff --git a/hullwork/config.py b/hullwork/config.py index a169ba4..f8c2083 100644 --- a/hullwork/config.py +++ b/hullwork/config.py @@ -91,10 +91,22 @@ class Settings(BaseSettings): #: resolving it quietly. forge_kind: str | None = None - # The credential an agent pushes through, kept apart from the one above on purpose. The - # ingest token is held by the request path and the sweep — it is in memory whenever the - # service is up — so it must never be able to write code. Unset until M2; there is no - # fallback to `forge_token`, because a convenient fallback is how a boundary is lost. + # The credential Hullwork pushes **verified work** through, kept apart from the one above on + # purpose. The ingest token is held by the request path and the sweep — it is in memory + # whenever the service is up — so it must never be able to write code. There is no fallback to + # `forge_token`, because a convenient fallback is how a boundary is lost. + # + # **This used to say "the credential an agent pushes through", and item 178 made that false.** + # `hullwork deps --open` opens a pull request for an upgrade that passed the project's own + # suite, with **no agent having run** — no model, no gateway, no brief. Reusing this token was + # the operator's decision on 2026-08-09, over a third one of its own, and the argument is that + # a third token would need exactly the same scope (`write:repository`): an audit boundary + # rather than a capability boundary, which is not what the split above is. That one is real and + # was measured (item 073, and `credentials.py` records how). + # + # What the rewrite costs, said plainly so nothing goes on relying on it: **nobody may infer + # that a model was called from the fact that something was pushed.** Two paths hold this token + # now — an agent's fix and a verified upgrade — and only the attempt trail can tell them apart. forge_code_token: SecretStr | None = None # The error tracker's READ api (item 036). Optional: without it Hullwork behaves exactly as it diff --git a/hullwork/dependencies.py b/hullwork/dependencies.py new file mode 100644 index 0000000..b3e25f2 --- /dev/null +++ b/hullwork/dependencies.py @@ -0,0 +1,231 @@ +"""What a project pinned, read from its own lock files. Item 172, DR-0016. + +**Lock files rather than declarations**, and the distinction is the whole reason this module can +say anything useful: `requests>=2.0` is a range, and a range does not tell you what is installed. +Only a lock says what a build actually resolved to, which is what a vulnerability database can be +asked about. + +**Pure functions over a `read` callable**, which is `propose`'s pattern and is here for the same +reason: the identical code then serves a local checkout and a forge tree, so the answer cannot +differ depending on which door the question came through. + +**Nothing here reaches the network**, and nothing here needs to. The sandbox has none by design +(`sandbox/image.py` — *"the build has network; the attempt does not"*), so DR-0016 puts this step +in the dispatcher, before any container exists. This module does not even know a container is +coming. +""" + +from __future__ import annotations + +import json +import re +import tomllib +from collections.abc import Callable, Sequence +from dataclasses import dataclass + +#: What this reads, named so a checkout with none of them can be told what was looked for. An empty +#: list reads as "you have no dependencies", which is a different claim. +#: +#: The ecosystem lives in each reader rather than beside the filename: **OSV's own strings** +#: (`npm`, `PyPI`) go straight into the query, and a translation table between our names and +#: theirs would be a second thing to keep correct for no gain. +#: +#: The last two entries are shapes rather than names, and item 180 is why: matching only the exact +#: basename `requirements.txt` missed every layout Python projects actually use. This list is for a +#: person to read; `is_requirements` below is what decides. +WHAT_IS_LOOKED_FOR: tuple[str, ...] = ( + "package-lock.json", + "uv.lock", + "poetry.lock", + "requirements.txt", + "requirements-*.txt and *-requirements.txt", + "any *.txt inside a requirements/ directory", +) + +#: `name==version`, and nothing else. Extras and environment markers are stripped before the match +#: because `httpx[http2]==0.27.0 ; python_version >= "3.8"` pins `httpx`, and refusing to read that +#: line would drop a real pin over punctuation. +_PINNED = re.compile(r"^\s*([A-Za-z0-9._-]+)\s*(?:\[[^\]]*\])?\s*==\s*([^\s;#]+)") + +#: A requirement line that is not a comment, not blank, and not an option. +#: +#: **The leading `-` is excluded deliberately**, and a test is why: with it in the class, `-e .` +#: counted as a requirement this reader could not pin, so a file whose every real line was pinned +#: still reported an unpinned one. `-e`, `-r other.txt` and `--index-url` are pip options, not +#: dependencies, and no package name begins with a hyphen. +_A_REQUIREMENT = re.compile(r"^\s*[A-Za-z0-9._]") + + +@dataclass(frozen=True) +class Dependency: + """One pinned package, and which file said so. + + `source` is carried because a project can pin the same name in two files at different + versions, and a report that cannot say which one it read is a report nobody can act on. + """ + + ecosystem: str + name: str + version: str + source: str + + +def _from_package_lock(text: str, source: str) -> list[Dependency]: + """npm's lock, versions 2 and 3, which both carry the flat `packages` map. + + The `""` key is the project itself rather than a dependency of it. Including it would have + Hullwork ask OSV about the repository being scanned. + """ + try: + document = json.loads(text) + except ValueError: + return [] + packages = document.get("packages") + if not isinstance(packages, dict): + return [] + + found: list[Dependency] = [] + for path, entry in packages.items(): + if not path or not isinstance(entry, dict): + continue + version = entry.get("version") + # An entry with no version appears in the file and pins nothing — a link or a workspace + # member — so it is not something to ask a vulnerability database about. + if not isinstance(version, str) or not version: + continue + # `node_modules/@scope/pkg` → `@scope/pkg`, and nested paths keep only the last package. + name = path.split("node_modules/")[-1] + found.append(Dependency("npm", name, version, source)) + return found + + +def _from_toml_lock(text: str, source: str) -> list[Dependency]: + """`uv.lock` and `poetry.lock`, which are the same two keys under `[[package]]`. + + One reader rather than two: they differ in everything except the part this needs, and a second + reader would be a second thing to keep correct for a difference that does not exist here. + """ + try: + document = tomllib.loads(text) + except tomllib.TOMLDecodeError: + return [] + packages = document.get("package") + if not isinstance(packages, list): + return [] + + found: list[Dependency] = [] + for entry in packages: + if not isinstance(entry, dict): + continue + name, version = entry.get("name"), entry.get("version") + if isinstance(name, str) and isinstance(version, str) and name and version: + found.append(Dependency("PyPI", name, version, source)) + return found + + +def _from_requirements(text: str, source: str) -> list[Dependency]: + """The weakest reader, and the one most projects will actually hit. + + Only `==` pins. Everything else is a range, and a range cannot be asked about — see + `unpinned`, which is how the caller says how much of the file it could not use. + """ + found: list[Dependency] = [] + for line in text.splitlines(): + matched = _PINNED.match(line) + if matched: + found.append(Dependency("PyPI", matched.group(1), matched.group(2), source)) + return found + + +def unpinned(text: str) -> int: + """How many requirement lines were **not** `==` pins. + + Reporting four packages out of a file with six requirement lines, without saying so, would + understate the answer silently — which is the failure mode this whole repository is about. + """ + return sum( + 1 + for line in text.splitlines() + if _A_REQUIREMENT.match(line) and not _PINNED.match(line) + ) + + +_READERS: dict[str, Callable[[str, str], list[Dependency]]] = { + "package-lock.json": _from_package_lock, + "uv.lock": _from_toml_lock, + "poetry.lock": _from_toml_lock, + "requirements.txt": _from_requirements, +} + +#: `requirements-dev.txt`, `dev-requirements.txt`, `requirements_test.txt` — the word at **one end +#: or the other**, never buried in the middle. +#: +#: The first version of this was `(?:.+[-_])?requirements(?:[-_].+)?\.txt`, whose comment claimed it +#: would not take `install-requirements-guide.txt`. It did: prefix `install-`, the word, suffix +#: `-guide`. The comment was the assertion, and a comment is not one — a test caught it the moment +#: the near miss was written down. So the alternation is explicit: the name *starts* with the word, +#: or it *ends* with it. +_NAMED_REQUIREMENTS = re.compile(r"^(?:requirements(?:[-_].+)?|.+[-_]requirements)\.txt$") + +#: A directory whose whole job is to hold them. `requirements/base.txt` and `requirements/prod.txt` +#: are the layout this repository itself uses, and item 180 found them unread. +_REQUIREMENTS_DIR = "requirements" + + +def is_requirements(path: str) -> bool: + """Whether this path is a pip requirements list. Item 180. + + **Widened from one exact basename, and safe because the reader is strict.** `_from_requirements` + takes only `name==version` lines, so a `.txt` that is not a requirements file contributes zero + dependencies rather than nonsense — which is what makes casting a wider net cost nothing. The + alternative, matching one name, cost this repository three of its own pinned packages in + silence. + + Two shapes, because those are the two conventions: the word in the file name, and a directory + named for it. Anything else with a `.txt` extension is left alone — a report about a changelog + would be worse than the miss. + """ + parts = path.split("/") + name = parts[-1] + if not name.endswith(".txt"): + return False + if len(parts) >= 2 and parts[-2] == _REQUIREMENTS_DIR: + return True + return bool(_NAMED_REQUIREMENTS.match(name)) + + +def _reader_for(path: str) -> Callable[[str, str], list[Dependency]] | None: + """Which reader owns this path, or `None` when nothing does. + + The exact-name table first, because three of the four entries are lock files whose names are + fixed by their own tooling and cannot be pattern-matched without inviting a false positive. + """ + reader = _READERS.get(path.rsplit("/", 1)[-1]) + if reader is not None: + return reader + return _from_requirements if is_requirements(path) else None + + +def read_lockfiles( + paths: Sequence[str], read: Callable[[str], str | None] +) -> list[Dependency]: + """Every pinned dependency the tree declares, in the order the files were found. + + A file that cannot be parsed contributes nothing rather than raising: a project with a broken + `requirements.txt` still has a `package-lock.json` worth reading, and one unreadable file must + not cost the whole answer. + """ + found: list[Dependency] = [] + for path in paths: + # Matched on the file name so a lock in a subdirectory is read too — a monorepo pins per + # package, and only reading the root would report a fraction of the truth as the whole. + # Item 180: that same sentence is why `requirements/prod.txt` is read as well, and it took + # running this against its own repository to notice that it was not. + reader = _reader_for(path) + if reader is None: + continue + text = read(path) + if text is None: + continue + found.extend(reader(text, path)) + return found diff --git a/hullwork/dispatch.py b/hullwork/dispatch.py index 2ea6d9a..b2c55df 100644 --- a/hullwork/dispatch.py +++ b/hullwork/dispatch.py @@ -26,6 +26,7 @@ import logging import shutil import tempfile +from collections.abc import Callable, Sequence from dataclasses import dataclass, field from pathlib import Path @@ -39,6 +40,7 @@ RunResult, Sandbox, collect_changes, + created_test_config, is_test_infrastructure, snapshot, ) @@ -83,6 +85,13 @@ class Verdict: #: Test infrastructure the fix phase had changed and this module put back (item 046). Non-empty #: means the published claim rests on the second green gate, not the first. restored: str = "" + #: Dependency files the fix phase had changed and `refit` put back (item 179). Non-empty means + #: the phase reached for the one shortcut that would have passed every gate. + reverted: str = "" + #: The headline sentence, when this sequence's claim is not the ordinary one. Empty means + #: `evidence` chooses from the outcome, which is right for every attempt that starts from a + #: reproducing test somebody wrote. A refit does not: see `refit` below. + claim: str = "" def dispatch( @@ -326,6 +335,297 @@ def dispatch( ) +#: The claim a refit publishes under, and it is deliberately not `evidence`'s ordinary one. +#: +#: *A test that failed against unmodified code passes with this change applied* is false here in the +#: one word that carries it: the code **was** modified — by the upgrade — before the first gate ran. +#: A headline that says otherwise would overclaim in exactly the direction item 171 removed, and it +#: would be the artefact rather than the run that lied. +#: It does not quote the sentence it replaces, deliberately: a headline that names the claim it is +#: *not* making reads as that claim to somebody skimming, and it is the line a reviewer acts on. +REFIT_CLAIM = ( + "**The tests this upgrade broke pass with the change below, and the upgrade is still " + "applied.** The code here had already been changed before the first run — by the upgrade " + "itself — so this is not the ordinary red-green claim. Both runs are below with their exit " + "codes, and the pinned version was read back out of the tree after the second one." +) + +#: And the four ways a refit ends without one, each with its own sentence. +#: +#: `evidence._claim` chooses from the outcome, and every one of its sentences is about *a bug* — the +#: bug was reproduced, the bug could not be reproduced, this appears to be fixed already. None of +#: those is what happened here, and an artefact that calls a dependency upgrade a bug sends its +#: reader looking for one. Carried on the verdict rather than inferred at the far end, so the two +#: surfaces cannot come to disagree about what a run was. +REFIT_NOT_BROKEN = ( + "**This upgrade does not break your suite here, so there was nothing to fix.** The suite was " + "run with the new version applied and it passed. No agent was called." +) +REFIT_REVERTED = ( + "**The attempt reverted the upgrade instead of making the code fit it, and that is not a " + "fix.** Putting the old version back makes a suite pass and undoes what this work exists to " + "possible. Hullwork put the dependency files back and published nothing." +) +REFIT_NOT_FIXED = ( + "**The tests this upgrade broke still fail.** Nothing was merged and nothing was hidden; what " + "was tried is below, with both runs and their exit codes." +) +REFIT_NO_CHANGE = ( + "**No change was produced, so there is nothing to check.** The upgrade still breaks the tests " + "named below." +) + + +def refit( + session: object, + item: Item, + manifest: Manifest, + engine: Engine, + *, + box: Sandbox, + attempt: Attempt, + package: str, + to: str, + guarded: "Sequence[str]", + version_now: "Callable[[Path], str | None]", +) -> Verdict: + """Make a broken upgrade fit. Item 179, DR-0018 step 4. + + ``` + 0. RED GATE run `tests` with the upgrade already applied → must FAIL + 1. refit agent; source editable, dependencies read-only → the change + 2. GREEN GATE run `tests` again, upgrade still applied → must PASS + 3. the re-read what the tree pins now → must be `to` + ``` + + **The red gate is free and it is inverted.** In `dispatch` above, red means the candidate test + reproduces the bug; here it is the starting condition, and the failing tests are the project's + own — failing against a version somebody published, with nobody having authored them for the + occasion. DR-0003's expensive half is therefore already satisfied by evidence no model can + flatter, which is why this sequence is three steps rather than six. + + **The whole correctness of this function is that the dependency files are read-only to step 1.** + Reverting is the obvious cheat and it is the most plausible-looking false artefact this product + could ever emit: put the old version back and the suite goes green, red before, green after, + upgrade gone. + + Measured while building this, and it changes what the guard is *for*: within one attempt a + revert **cannot buy a green gate**. The image is built before this function is called and the + phases have no network, so the installed version cannot move whatever the files say. What a + revert buys is the *published diff* — a pull request that undoes the upgrade it claims to make + possible, with two honest gate runs attached. So the files are restored before anything is + collected, and the verdict says what was attempted. + + `version_now` is the backstop and it is a different guard, not a second copy of the first: the + restore covers the files this ecosystem's resolver is known to touch (`resolve.touches`), and + the re-read covers everything it does not — a pin moved somewhere nobody taught the guard about, + a vendored dependency, an ecosystem added later. A green gate whose tree no longer carries the + upgraded version is a revert however it got there. + """ + worktree = box.worktree + tests = (manifest.tests or "").strip() + lint_ask = (manifest.lint or "") if "lint" in manifest.autofix.gates else "" + if not tests: # pragma: no cover - the manifest parser refuses this before we get here + raise Abandoned("the manifest declares no test command") + + # --- step 0: the red gate, already paid for ----------------------------------------------- + red = _run_gate(session, attempt, box, AttemptPhase.RED_GATE, tests) + if red.ok: + # Before the model is called. The breakage does not reproduce here, so there is nothing to + # fix and nothing was learned about the upgrade that `deps --verify` had not already said. + return Verdict( + AttemptOutcome.NOT_REPRODUCIBLE, + AttemptPhase.RED_GATE, + claim=REFIT_NOT_BROKEN, + detail=( + f"the suite passes with {package} {to} applied, so there is nothing here to fix. " + f"Whatever broke when this upgrade was measured does not break now." + ), + ) + + # Snapshotted after the gate, not before: a real test run writes caches, and taken earlier every + # one of those looks like something the agent produced (the `.pytest_cache` finding, item 025). + pristine = snapshot(worktree) + + # --- step 1: the agent, with the dependencies out of reach -------------------------------- + _run_agent( + session, attempt, box, engine, Phase.REFIT, + test_path=manifest.test_path, lint=lint_ask, + ) + reverted = _restore_dependencies(worktree, pristine, guarded) + try: + changes = collect_changes(worktree, pristine) + except UnsafePathError as exc: + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.FIX, + reverted=reverted, + claim=REFIT_NOT_FIXED, + detail=f"the fix phase produced something it may not: {exc}", + ) + if not changes: + # Two different findings, and they must not share a sentence: a phase that did nothing and + # a phase that did the one forbidden thing are not the same report to a person. + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.FIX, + reverted=reverted, + claim=REFIT_REVERTED if reverted else REFIT_NO_CHANGE, + detail=( + f"the fix phase reverted the upgrade instead of making the code fit it " + f"({reverted}), and changed nothing else. Those files were put back, so there is " + f"no change here — a suite made green by putting the old version back is a revert, " + f"not a fix." + if reverted + else "the fix phase changed nothing, so there is no fix to check" + ), + ) + + # --- step 2: the green gate --------------------------------------------------------------- + green = _run_gate(session, attempt, box, AttemptPhase.GREEN_GATE, tests) + + # Item 046, unchanged: a suite that collects nothing passes trivially, and the *difference* + # between the two runs is the finding rather than either one of them. + restored = _restore_infrastructure(worktree, pristine) + if restored: + try: + changes = collect_changes(worktree, pristine) + except UnsafePathError as exc: # pragma: no cover - the earlier collection raises first + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.GREEN_GATE_RESTORED, + reverted=reverted, + claim=REFIT_NOT_FIXED, + detail=f"the restored tree could not be read back: {exc}", + ) + regated = _run_gate(session, attempt, box, AttemptPhase.GREEN_GATE_RESTORED, tests) + if not regated.ok: + gamed = ( + ", so it passed only because it had disabled the tests" + if green.ok + else " and the suite does not pass either way" + ) + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.GREEN_GATE_RESTORED, + changes=changes, + restored=restored, + reverted=reverted, + claim=REFIT_NOT_FIXED, + detail=( + f"the fix modified test infrastructure it may not have ({restored}) and the " + f"suite fails once that is put back{gamed}" + ), + ) + elif not green.ok: + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.GREEN_GATE, + changes=changes, + reverted=reverted, + claim=REFIT_NOT_FIXED, + detail=( + f"the suite still does not pass with {package} {to} applied" + + ( + f", and the fix phase had put the old version back ({reverted}) — that was " + f"undone before this run, because a revert is not a fix" + if reverted + else "" + ) + ), + ) + + # --- step 3: what does the tree pin now? --------------------------------------------------- + landed = version_now(worktree) + if landed != to: + # A green gate this side of a revert is the one artefact this whole item exists to prevent. + # Reported as what it is, and never as a fix, whichever route got it here. + return Verdict( + AttemptOutcome.FAILED, + AttemptPhase.GREEN_GATE, + changes=changes, + restored=restored, + reverted=reverted or ", ".join(guarded), + claim=REFIT_REVERTED, + detail=( + f"the suite passes and the tree no longer pins {package} at {to}: it " + + (f"pins {landed}" if landed else "no longer pins it at all") + + f". That is a revert rather than a fix — the upgrade this was supposed to make " + f"possible is gone, and a green suite without it proves nothing about {to}." + ), + ) + + # --- step 4: lint, only if the manifest names the gate (item 067) -------------------------- + lint_failed = "" + if "lint" in manifest.autofix.gates and manifest.lint: + lint = _run_gate(session, attempt, box, AttemptPhase.LINT_GATE, manifest.lint) + if not lint.ok: + lint_failed = ( + f"The gate that failed is `{manifest.lint}`, run against the change below.\n" + f"{failing_lines(lint.output) or lint.output[-2000:]}" + ) + + note = "" + if reverted: + # Led with, not tucked away, for item 067's reason: an artefact whose shape hides its own + # weakest part is worse than none. The claim below still stands — the gate ran with the + # upgrade in place — but a reviewer has to know the phase reached for the shortcut. + note += ( + f"The fix phase also edited dependency files it may not have ({reverted}). They were " + f"put back before the run below, so the change published here contains none of them " + f"and the suite passed with {package} {to} still applied.\n\n" + ) + if restored: + note += ( + f"This fix had also modified test infrastructure it may not have ({restored}); those " + f"files were put back and the suite was run again, so the claim rests on that second " + f"run and the published change does not contain them.\n\n" + ) + return Verdict( + AttemptOutcome.PR_OPEN_LINT_FAILED if lint_failed else AttemptOutcome.PR_OPEN, + AttemptPhase.PUBLISH, + changes=changes, + restored=restored, + reverted=reverted, + claim=REFIT_CLAIM, + detail=(f"{lint_failed}\n\n" if lint_failed else "") + note + ( + f"this makes {package} {to} possible: the tests below failed with it applied and pass " + f"with the change, and {package} is still pinned at {to} in the tree those runs used." + ), + ) + + +def _restore_dependencies( + worktree: Path, pristine: dict[str, bytes], guarded: "Sequence[str]" +) -> str: + """Put every dependency file back exactly as the upgrade left it, and say which had moved. + + **Restoring rather than merely detecting**, for `_restore_candidate`'s reason one file over: a + reverted pin that reached `collect_changes` would be published, and the pull request would undo + the upgrade in its own diff while its body claimed to make it possible. + + Three ways a file can move and all three are a revert: changed, deleted, and — the one that is + easy to leave out — **created where there was none**. A project pinning in `requirements.txt` + with no `package.json` beside it can have one written, and a guard that only compared existing + files would not see it. + """ + moved: list[str] = [] + for name in guarded: + target = worktree / name + original = pristine.get(name) + if original is None: + if target.exists(): + moved.append(name) + target.unlink() + continue + if target.exists() and target.read_bytes() == original: + continue + moved.append(name) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(original) + return ", ".join(sorted(moved)) + + #: How many failure lines a verdict carries. A suite with 2000 failures must not become the verdict, #: and after a dozen the operator has the shape of it — the rest is in `attempt_steps.output`. FAILURES_SHOWN = 12 @@ -524,6 +824,13 @@ def _restore_infrastructure(worktree: Path, pristine: dict[str, bytes]) -> str: The scope comes from `is_test_infrastructure`, which belongs to the instance. It deliberately is not the manifest's `test_path`: that field arrives from the watched repository, and it already pulls the other way — narrowing it tightens the reproduce-phase guard while loosening this one. + + **And configuration that was invented rather than edited is removed** (item 179). Iterating the + snapshot alone left a hole this guard's own sentence describes: a phase that *creates* a root + `conftest.py` where the project had none switches the suite off, is absent from `pristine`, and + was therefore never touched — leaving `restored` empty, so the second gate never ran and the + attempt published with the mechanism in its diff. Measured against this function before + `created_test_config` existed. """ restored: list[str] = [] for path, original in pristine.items(): @@ -535,6 +842,11 @@ def _restore_infrastructure(worktree: Path, pristine: dict[str, bytes]) -> str: restored.append(path) target.parent.mkdir(parents=True, exist_ok=True) target.write_bytes(original) + for path in created_test_config(worktree, pristine): + # Removed rather than restored: there is nothing to put back, and leaving it while + # reporting it would publish the file that decided the gate. + restored.append(path) + (worktree / path).unlink() return ", ".join(sorted(restored)) @@ -563,8 +875,24 @@ def build_brief_file(session: object, item: Item, contract_dir: Path) -> Path: Into the contract directory, not the worktree. The brief is Hullwork's input to the agent, not a change to the repository, and putting it in the tree made it look like one. """ + return write_brief(brief.build(session, item), contract_dir) # type: ignore[arg-type] + + +def write_brief(text: str, contract_dir: Path) -> Path: + """The same file, from text somebody else composed. Item 179. + + Split out because a refit's brief cannot come from `brief.build`: that one answers *what + Hullwork knows about this error* from the tracker and this instance's history, and a dependency + upgrade has no error, no fingerprint from a stranger and no occurrence count. Built from it, the + brief would open by saying the full event was never fetched — true of a tracker nobody asked, + and misleading about work whose evidence is better than any tracker's. + + What stays shared is everything about *where* it goes and how, because that half has been wrong + twice: in the worktree it looked like a change to the repository, and without the mode the + container could not read it. + """ contract_dir.chmod(CONTRACT_DIR_MODE) target = contract_dir / Path("brief.md") - target.write_text(brief.build(session, item), encoding="utf-8") # type: ignore[arg-type] + target.write_text(text, encoding="utf-8") target.chmod(0o644) return target diff --git a/hullwork/engine.py b/hullwork/engine.py index d81eaab..b2923ca 100644 --- a/hullwork/engine.py +++ b/hullwork/engine.py @@ -40,6 +40,13 @@ class Phase(StrEnum): REPRODUCE = "reproduce" FIX = "fix" + #: Item 179, DR-0018 step 4. The same job as `FIX` and a different thing to say, which is why + #: it is a phase rather than a flag on that one: `fix` opens by naming a reproducing test file + #: at `${TEST_PATH}/${TEST_FILE}`, and in a refit nobody wrote one — the failing tests are the + #: project's own, failing against an upgrade that is already applied. An agent told to look for + #: evidence that does not exist spends the item's one attempt on a misunderstanding, which is + #: what item 094 measured when the filename was wrong rather than absent. + REFIT = "refit" @dataclass(frozen=True) @@ -324,6 +331,23 @@ def parse(cls, text: str) -> "AgentReport": evidence, and a test the fix was allowed to edit proves nothing. Run the whole suite before you finish: every other test must still pass." ;; + refit) + # Item 179. The brief names the upgrade and the tests it broke; this says what the job is and + # what the one forbidden shortcut is. Stated here as well as enforced by the dispatcher, + # because an agent that reverts and is then reported for it has spent the item's one attempt + # on something it was never told not to do. + ASK="A dependency upgrade has ALREADY been applied to this checkout, and it broke tests that +were passing. The brief names the package, both versions, and the tests that now fail. + +Change this project's own source code so those tests pass with the new version. Run the whole +suite before you finish: every other test must still pass. + +Do NOT edit any dependency file — the pins, the manifest and the lock are read-only evidence +here. Putting the old version back would make the suite pass and is not a fix: Hullwork restores +those files, re-reads the version out of the tree, and reports that as a revert rather than as a +fix. If the upgrade genuinely cannot be made to work, change nothing and say so — that is a +correct and useful answer.${LINT_ASK}" + ;; *) echo "unknown phase $PHASE" >&2; exit 2 ;; esac diff --git a/hullwork/evidence.py b/hullwork/evidence.py index 8321a03..9af844b 100644 --- a/hullwork/evidence.py +++ b/hullwork/evidence.py @@ -20,7 +20,7 @@ import json import logging -from collections.abc import Mapping +from collections.abc import Mapping, Sequence from typing import TYPE_CHECKING, Any from hullwork import spend, testoutput @@ -28,6 +28,11 @@ from hullwork.scrub import Scrubber if TYPE_CHECKING: + # Imported for types only: `bump` and `osv` are about producing a verdict and this module is + # about rendering one, and a runtime import would make the renderer a dependency of the + # measurement rather than the other way round. + from hullwork.bump import Answer as BumpAnswer + from hullwork.osv import Advisory from hullwork.spend import Prices log = logging.getLogger(__name__) @@ -128,13 +133,21 @@ def _scrubber(secrets: list[str] | None = None) -> Scrubber: ) -def _claim(attempt: Attempt) -> str: +def _claim(attempt: Attempt, given: str = "") -> str: """The headline sentence, chosen from what the attempt actually did. `failed` at the red gate means the reproduction was refused; `failed` after it means the reproduction stood and the fix did not. The reader a comment is for cannot see the phase table first — the headline is what they act on, so it is the part that must not overstate. + + **`given` is a sequence that does not make this sequence's claim** (item 179). Every sentence + below is about *a bug* — reproduced, not reproduced, already fixed — and a refit is about a + dependency upgrade that was applied before the first gate ran. Its own claim travels on the + verdict rather than being inferred from the outcome here, because a headline chosen at this end + from an outcome that means something else is how an artefact comes to contradict its own table. """ + if given: + return given outcome = attempt.outcome or AttemptOutcome.ABANDONED if outcome is AttemptOutcome.FAILED: if attempt.phase_reached in _BEFORE_FIX: @@ -154,6 +167,10 @@ def pull_request_body( secrets: list[str] | None = None, #: What the operator pays, for the cost row. `None` prints tokens and no money (item 133). prices: "Prices | None" = None, + #: The headline, when the sequence that produced this does not make the ordinary claim + #: (item 179). Empty means it is chosen from the outcome, which is right for every attempt + #: that starts from a reproducing test somebody wrote. + claim: str = "", ) -> str: """The body of the draft pull request. @@ -163,7 +180,7 @@ def pull_request_body( """ scrub = _scrubber(secrets) lines = [ - _claim(attempt), + _claim(attempt, claim), "", ] if detail: @@ -201,6 +218,105 @@ def pull_request_body( return body +#: The sentence that keeps a green pull request honest, and the reason it is not a guarantee. +#: +#: **A green pull request is the easiest place in this product to overclaim.** The reviewer is being +#: asked to merge, the diff is one line, and every other tool that put it there was guessing — so +#: whatever this says will be read generously. +#: +#: **One author for the caveat, which is item 098's rule and was broken here first.** The first +#: version of this said *what was measured is your suite… not that the upgrade is safe or that it +#: fixes anything*, directly beneath `Answer.says`, which already ends *that is what was measured — +#: not that the upgrade is safe, and not that it fixes anything your suite does not exercise*. Two +#: paragraphs, same caveat, different words, at the top of the document a person acts on. Read +#: rather than tested: the assertions were all satisfied. Item 098 records what that costs — "the +#: first paragraph of `acme!9` said it twice and read like a program that had lost its place". +#: +#: So this carries only what the claim above does **not**: the mechanism. A suite that never touches +#: the dependency produces this exact document, and that is a thing a reviewer can check about their +#: own repository rather than a limit they are asked to hold in mind. +WHAT_WAS_MEASURED = ( + "The suite that was run is **yours**, which is what makes the sentence above worth something " + "and also what bounds it: if your tests never exercise this dependency, they go green without " + "ever loading the new version, and this page would read exactly the same. Nothing here " + "inspected the change itself." +) + + +def dependency_pull_request_body( + answer: "BumpAnswer", + advisories: "Sequence[Advisory]" = (), + *, + secrets: list[str] | None = None, +) -> str: + """The body of a pull request for an upgrade that passed. Item 178, DR-0018 step 3. + + **Three facts and no more**, because this is the artefact whose whole value is that it is + narrower than a competitor's: the advisory and where to read it, the claim in DR-0016's exact + wording, and the two runs with their exit codes and the runner's own summary lines. + + The claim is `Answer.says` rather than a sentence written here, and that is the point of it + being a property on the answer: the terminal report prints the same string, so the two surfaces + cannot come to disagree about what was measured. A second rendering that happens to match today + is a rendering that will not match after the first edit to either. + + **Scrubbed like everything else that leaves the instance** (item 027): the summary lines are + captured output of an arbitrary command, and a suite that prints an environment dump on failure + is not a rare event. + """ + scrub = _scrubber(secrets) + lines = [answer.says, "", WHAT_WAS_MEASURED, ""] + + if advisories: + lines += ["### What is published against the version you have", ""] + for advisory in advisories: + summary = f" — {scrub.text(advisory.summary)}" if advisory.summary else "" + lines.append(f"- [{advisory.id}]({advisory.url}){summary}") + lines.append("") + + runs = answer.runs + if runs is not None: + # Both runs in one table, because the *pair* is the evidence and a reader comparing two + # sections separated by a page is a reader who will read one of them. + lines += [ + "### The two runs", + "", + f"Command: `{scrub.text(runs.command)}`", + "", + "| | exit | what the runner said |", + "|---|---|---|", + f"| Before this change | `{runs.before_exit}` | " + f"{_summary_cell(runs.before_summary, scrub)} |", + f"| With `{answer.package} {answer.to}` | `{runs.after_exit}` | " + f"{_summary_cell(runs.after_summary, scrub)} |", + "", + ] + + lines += [ + "---", + "", + "Opened by Hullwork as a **draft**, after running the two commands above. Nobody merges " + "this but you.", + ] + body = "\n".join(lines) + if len(body) > MAX_BODY_CHARS: # pragma: no cover - three facts do not reach 60,000 characters + body = body[:MAX_BODY_CHARS] + "\n\n… [body truncated by Hullwork]" + return body + + +def _summary_cell(summary: str, scrub: Scrubber) -> str: + """The runner's own line, or the fact that it printed none. + + **Never silence**, which is `_what_was_checked`'s rule for the same reason: a cell left out + reads as *the suite said nothing*, and that is a claim nobody made. What happened is that this + runner does not print a line in a shape Hullwork reads, and the exit code beside it is + unaffected. + """ + if not summary.strip(): + return "this runner prints no summary line Hullwork reads — the exit code is the claim" + return f"`{scrub.text(summary)}`" + + def issue_comment( item: Item, attempt: Attempt, @@ -209,6 +325,8 @@ def issue_comment( secrets: list[str] | None = None, #: For the cost row (item 133). prices: "Prices | None" = None, + #: The headline, for a sequence with its own (item 179). See `pull_request_body`. + claim: str = "", ) -> str: """What goes on the issue when there is no pull request. @@ -217,7 +335,7 @@ def issue_comment( only exists in a database is one nobody acts on. """ scrub = _scrubber(secrets) - lines = [_claim(attempt), ""] + lines = [_claim(attempt, claim), ""] if detail: lines += [scrub.text(detail), ""] lines += [ diff --git a/hullwork/features.py b/hullwork/features.py new file mode 100644 index 0000000..3bbeda6 --- /dev/null +++ b/hullwork/features.py @@ -0,0 +1,369 @@ +"""What this can do for your project, and what it cannot. Item 186. + +**Hullwork is modular and never said so.** `hullwork.yml` is the switchboard and almost everything +is off by default — `autofix.agent: none`, no `lint`, `notify.channel: none`, +`autofix.unmatched: human`, `runtime.install: none`. A project that declares nothing gets filing and +nothing else, deliberately. + +What did not exist is the other half of the operator's framing: **each feature has its +limitations**. Nothing declared what it needed or what it could not do, so every limitation was +found by walking into it — and three items measured what that costs. Item 182 found a **false +verdict** produced for a project whose image Hullwork does not build; item 184 found a missing +credential reported after four container builds, as a traceback; item 185 found `propose` writing a +manifest under which nothing could be measured, silently. + +Each was fixed where it happened, and none of those is a place a person looks *before* deciding +whether this is for them. + +**The shape is `projects lanes --checkout .`**, which prints this instance's lane policy against +your tree with no credential of any kind, because *"a policy nobody has read is a policy nobody has +agreed to"*. Same rules here: a checkout, no credential, nothing executed, nothing written and no +socket opened. + +**Declaring a limitation is not accepting it**, and this module must never read as an excuse. That +dependency verification needs Hullwork to build the image is a real limit *and* a consequence of +building on the path DR-0007 demoted to sugar. Saying so is honest; leaving it there is a decision +nobody has taken. +""" + +from __future__ import annotations + +from collections.abc import Callable, Sequence +from dataclasses import dataclass, field + +from hullwork.manifest import Manifest + + +@dataclass(frozen=True) +class Need: + """One thing a feature requires, and what to do when it is missing. + + `met` is asked of the checkout and the manifest and **nothing else**: no network, no daemon, no + credential. A requirement this cannot answer from those two is not a requirement this module + states — it is `doctor`'s, and `INSTANCE_SHAPED` names those rather than guessing at them. + """ + + what: str + #: What to do about it, in the words a person would type. Never "configure it correctly". + fix: str + met: Callable[[Checkout], bool] + + +@dataclass(frozen=True) +class Feature: + """One thing Hullwork can do, what it needs, and what it cannot do even when it can. + + **Two fields and they must not blend.** `needs` is checkable and either met or not; `limits` is + true whatever the answer, and is stated either way. The second is the half the operator's + framing names and the half that had no home anywhere in this repository. + + Declared as data, like `resolve.RESOLVERS` and `image.INSTALL_COMMANDS`, so a feature is a row. + """ + + name: str + #: One line, for somebody deciding whether they want it. + does: str + needs: tuple[Need, ...] = () + #: What the **project** must have permitted, as opposed to what it must be able to do (DR-0019). + #: Unmet here is a third answer and not a fourth kind of missing part: *available, and this + #: project has not said yes*. Empty for every feature that writes nothing to a repository. + permits: tuple[Need, ...] = () + #: What it cannot do with everything in place. **Never empty**: a feature with no limits has to + #: say that in words, because an empty list reads as "nobody wrote them down" — which, until + #: this module, was true of all of them. + limits: tuple[str, ...] = () + + +@dataclass(frozen=True) +class Checkout: + """Everything this module is allowed to look at. Deliberately three fields. + + A checkout's tracked paths, its manifest if it has one, and **which instance variables are + set** — never their values. That last one is why this can say *"needs a model credential, and + none is configured"* without holding one, and why it can be run by somebody who has configured + nothing at all. + """ + + paths: tuple[str, ...] = () + manifest: Manifest | None = None + #: Names of environment variables that have a value. Names only, never values. + configured: frozenset[str] = field(default_factory=frozenset) + + def has(self, *names: str) -> bool: + """Whether the checkout tracks a file with one of these base names.""" + wanted = set(names) + return any(path.rsplit("/", 1)[-1] in wanted for path in self.paths) + + +def _manifest_says(check: Callable[[Manifest], bool]) -> Callable[[Checkout], bool]: + """A need that is about the manifest. False when there is none, which is the honest answer.""" + + def met(checkout: Checkout) -> bool: + return checkout.manifest is not None and check(checkout.manifest) + + return met + + +def _installs_from_a_pinned_file(manifest: Manifest) -> bool: + """Whether an upgrade could reach the environment the suite runs in. Item 182's finding. + + With `install: none` the image is `runtime.base` exactly as it comes and nothing is installed + from a lock file, so rewriting a pinned version changes nothing the suite would run against. + Measured: a checkout pinning `jinja2==2.4.1`, a base carrying 3.0.0, and a verdict saying the + suite passed before the change and after it — about a version never installed. + """ + runtime = manifest.runtime + return runtime is not None and runtime.install != "none" and bool(runtime.dependencies) + + +#: What a lock file is called, borrowed from the reader rather than restated — a second list is a +#: second thing to keep correct, and this one would go stale the day an ecosystem is added. +def _pins_anything(checkout: Checkout) -> bool: + from hullwork import dependencies + + return any( + path.rsplit("/", 1)[-1] in {"package-lock.json", "uv.lock", "poetry.lock"} + or dependencies.is_requirements(path) + for path in checkout.paths + ) + + +#: The variable each credential-shaped need looks for. Names, never values (`Checkout.configured`). +MODEL_KEY = "HULLWORK_MODEL_KEY" +CODE_TOKEN = "HULLWORK_FORGE_CODE_TOKEN" # noqa: S105 - a variable's name, never its value + +#: Features whose answer is about an **instance** rather than about a checkout, named rather than +#: guessed at. `doctor` owns these: whether the forge answers, whether the tracker is reachable, +#: whether the database has a schema, whether a dispatcher holds the lease. A checkout cannot know +#: any of it, and a report that pretended to would be worse than one that says whose question it is. +INSTANCE_SHAPED: tuple[str, ...] = ( + "filing a production error as an issue", + "the daily page", + "notifications", + "the recurrence watch", +) + +FEATURES: tuple[Feature, ...] = ( + Feature( + name="dependency report", + does="says which of your pinned versions have a published advisory, and what fixes each", + needs=( + Need( + what="a lock file or a pinned requirements file, committed", + fix="commit one, or pin with `==` — a declaration is a range and a range is not a " + "fact about what your build resolved to", + met=_pins_anything, + ), + ), + limits=( + "It reads what you pinned, so a dependency your build resolves at install time is " + "invisible to it.", + "It asks OSV, which is one database. An advisory nobody published is an advisory this " + "cannot know about.", + ), + ), + Feature( + name="dependency verification", + does="applies each published fix and runs your own suite against it, in a sandbox", + needs=( + Need( + what="a lock file or a pinned requirements file, committed", + fix="commit one, or pin with `==`", + met=_pins_anything, + ), + Need( + what="a hullwork.yml naming an image (`runtime.base`) and your test command", + fix="`hullwork propose --checkout .` writes one from your CI configuration", + met=_manifest_says( + lambda m: m.runtime is not None and bool(m.runtime.base) and bool(m.tests) + ), + ), + Need( + what="an installer that reads the file your versions are pinned in", + fix="name `runtime.install` and the file in `runtime.dependencies`. **Keep your " + "own image as the `base`** — this is one line on top of it, in your own words, not " + "a rebuild from scratch. Without it the image is your base exactly as it comes, so " + "changing a pin changes nothing your suite would run against", + met=_manifest_says(_installs_from_a_pinned_file), + ), + ), + limits=( + "What is measured is **your suite**. If it does not exercise the dependency it stays " + "green without ever loading the new version, and the verdict would read the same.", + "A phase has no network, so a suite that reaches the internet cannot be run here, and " + "is reported as a suite that does not pass rather than as a verdict about the upgrade.", + "It can only measure an upgrade it can **install**, so the image has to be refreshed " + "from the file that pins. That does not mean Hullwork must build your image: your own " + "image as the base, plus the one line that reinstalls your dependencies, is measured " + "the same way (item 188).", + ), + ), + Feature( + name="fixing an upgrade that breaks your suite", + does="asks an agent to change your code so the upgrade fits, then runs your suite again", + needs=( + Need( + what="everything dependency verification needs", + fix="see above — nothing is fixed until something has been measured breaking", + met=lambda c: _pins_anything(c) + and _manifest_says(_installs_from_a_pinned_file)(c), + ), + Need( + what="a model credential on the instance that runs it", + fix=f"set {MODEL_KEY} to an API key from any provider (DR-0004)", + met=lambda c: MODEL_KEY in c.configured, + ), + Need( + what="`autofix.agent` naming an engine this instance holds", + fix="set `autofix: {agent: claude-code}` — it is `none` by default, which is the " + "whole product for a project that wants nothing else", + met=_manifest_says(lambda m: m.autofix.agent != "none"), + ), + ), + limits=( + "One attempt per upgrade and then a person (DR-0003). A failure does not buy a second.", + "It may not touch your dependency files, so an upgrade that can only be made to fit by " + "changing the pin is reported as a revert rather than attempted.", + "It writes what it produced to disk and opens nothing anywhere.", + ), + ), + Feature( + name="opening the upgrades that pass", + does="opens one draft pull request per package whose suite passed, and never any other", + needs=( + Need( + what="everything dependency verification needs", + fix="see above — nothing is opened that was not run", + met=lambda c: _pins_anything(c) + and _manifest_says(_installs_from_a_pinned_file)(c), + ), + Need( + what="a credential able to write to your repository", + fix=f"set {CODE_TOKEN}. It is the one thing here that writes anything anywhere", + met=lambda c: CODE_TOKEN in c.configured, + ), + Need( + what="an `origin` remote, so the repository can be named", + fix="add one — a coordinate cannot be guessed from a directory name, and a wrong " + "guess opens a pull request somewhere else", + met=lambda c: "origin" in c.configured, + ), + ), + permits=( + Need( + what="`autofix.open_upgrades`, which this project has not set", + fix="set `autofix: {open_upgrades: true}` in hullwork.yml. It is false by default " + "because having the credential is not the same as having agreed (DR-0019), and " + "this is the only thing here that writes to your repository", + met=_manifest_says(lambda m: m.autofix.open_upgrades), + ), + ), + limits=( + "Only what passed. Nothing that broke, nothing blocked, no suite that was already red.", + "One pull request per package, never a batch — a grouped upgrade that breaks cannot be " + "bisected without undoing the work.", + "Every one is a draft. Nothing here merges anything, ever (constitution principle 1).", + ), + ), + Feature( + name="fixing a production error", + does="reproduces a reported error with a failing test, fixes it, and opens a draft pull " + "request carrying both", + needs=( + Need( + what="a hullwork.yml naming an image and your test command", + fix="`hullwork propose --checkout .` writes one from your CI configuration", + met=_manifest_says( + lambda m: m.runtime is not None and bool(m.runtime.base) and bool(m.tests) + ), + ), + Need( + what="`autofix.agent` naming an engine this instance holds", + fix="set `autofix: {agent: claude-code}` — `none` is the default", + met=_manifest_says(lambda m: m.autofix.agent != "none"), + ), + Need( + what="a model credential on the instance that runs it", + fix=f"set {MODEL_KEY} to an API key from any provider (DR-0004)", + met=lambda c: MODEL_KEY in c.configured, + ), + ), + limits=( + "It will not attempt anything in the red lane, and an error it cannot classify is red. " + "`hullwork projects lanes --checkout .` prints that policy against your own tree.", + "No fix without a test that fails first on untouched code (DR-0003). *I could not " + "reproduce this* is a result rather than a failure, and is the answer more often " + "than not.", + "One attempt per error, then a person.", + ), + ), +) + + +@dataclass(frozen=True) +class Answer: + """Whether one feature is available here, whether it is permitted, and what is in the way. + + **Three answers and not two** (DR-0019). *Available* is about what this project and instance + can do; *permitted* is about what the project has agreed to. Blending them would report a + decision somebody made as a part that is missing, which is the one way this report could + insult its reader. + """ + + feature: Feature + missing: tuple[Need, ...] + withheld: tuple[Need, ...] = () + + @property + def available(self) -> bool: + return not self.missing + + @property + def permitted(self) -> bool: + return not self.withheld + + +def examine(checkout: Checkout, features: Sequence[Feature] = FEATURES) -> list[Answer]: + """Which features this checkout can have, in the order they are declared. + + **Every need, not the first one that fails.** A reader who fixes one thing and runs this again + to find a second is a reader doing the work this command exists to save them. + """ + return [ + Answer( + feature, + tuple(need for need in feature.needs if not need.met(checkout)), + tuple(need for need in feature.permits if not need.met(checkout)), + ) + for feature in features + ] + + +def lines(answers: Sequence[Answer]) -> list[str]: + """The report, for a terminal. + + **The limits are printed whether or not the feature is available**, which is the whole of the + operator's framing: a feature you can have and a feature you cannot both have things they will + not do, and the first is the one where nobody thinks to look. + """ + said: list[str] = [] + for answer in answers: + if not answer.available: + mark = "no" + elif not answer.permitted: + # **A decision, spelled as one.** `no` here would read as a part that is missing, and + # somebody chose this. + mark = "not permitted here" + else: + mark = "yes" + said.append(f"[{mark}] {answer.feature.name} — {answer.feature.does}") + for need in answer.missing: + said.append(f" needs: {need.what}") + said.append(f" → {need.fix}") + for need in answer.withheld: + said.append(f" this project has not permitted it: {need.what}") + said.append(f" → {need.fix}") + for limit in answer.feature.limits: + said.append(f" limit: {limit}") + said.append("") + return said diff --git a/hullwork/forge/__init__.py b/hullwork/forge/__init__.py index 9bb56a2..6b70a06 100644 --- a/hullwork/forge/__init__.py +++ b/hullwork/forge/__init__.py @@ -627,9 +627,9 @@ def commit_files( head with an empty commit — so "the agent changed nothing" would otherwise become a branch and a pull request with no diff in it. - No sign-off trailer, ever, though the API adds one. CONTRIBUTING.md and the worker contract - both say the DCO sign-off is a human act performed at the merge gate; that a machine *can* - emit the trailer is exactly why it must not. + No sign-off trailer, ever, though the API adds one. CONTRIBUTING.md says the DCO + sign-off is a human act performed at the merge gate; that a machine *can* emit the + trailer is exactly why it must not. """ ... diff --git a/hullwork/forge/factory.py b/hullwork/forge/factory.py index 6b8eb51..55a0974 100644 --- a/hullwork/forge/factory.py +++ b/hullwork/forge/factory.py @@ -39,12 +39,16 @@ def make_forge(settings: Settings) -> Forge | None: def make_code_forge(settings: Settings) -> ForgeCode | None: - """The forge an agent pushes through. `None` until `HULLWORK_FORGE_CODE_TOKEN` is set. + """The forge Hullwork pushes verified work through. `None` until the code token is set. Separate function, separate setting, separate token — so that no request handler and no sweep can ever end up holding a credential that can write code. It falls back to nothing rather than to the ingest token: an accidental fallback is exactly how the boundary would be lost, quietly, on the day M2 lands. + + **Two callers now, and the second one runs no agent** (item 178). `hullwork deps --open` pushes + an upgrade the project's own suite passed, with no model and no gateway involved. This used to + say *the forge an agent pushes through*; `config.py` carries the decision and what it costs. """ if not settings.forge_url or not settings.forge_code_token: return None diff --git a/hullwork/forge/forgejo.py b/hullwork/forge/forgejo.py index 596a936..11d43b2 100644 --- a/hullwork/forge/forgejo.py +++ b/hullwork/forge/forgejo.py @@ -548,7 +548,7 @@ def commit_files( "author": identity, "committer": identity, # `signoff` is available and deliberately not set: the DCO sign-off is a human act - # performed at the merge gate (CONTRIBUTING.md, the worker contract). + # performed at the merge gate (CONTRIBUTING.md). "files": [_to_operation(change) for change in changes], } data = self._request("POST", f"/repos/{repo}/contents", json=payload) diff --git a/hullwork/manifest.py b/hullwork/manifest.py index fd9aae0..da6bd37 100644 --- a/hullwork/manifest.py +++ b/hullwork/manifest.py @@ -219,14 +219,37 @@ def _red_patterns_are_still_patterns(cls, value: list[str]) -> list[str]: class AutofixConfig(_Strict): - """How, and whether, an agent may attempt a fix. + """What Hullwork may do to this repository. **The permissions block**, and it is named after + one feature for historical reasons (DR-0019). `agent: none` is the default (DR-0002): the pipeline is fully useful with no external model call, and attempting fixes is what you opt into. + + **Every other block in this manifest describes what your project *is*** — the image, the test + command, the linter, where things live. This one describes what is *permitted*, and DR-0019 is + where that distinction was finally written down. The name stays `autofix` because a manifest + field is a public interface and every existing file would break for a better word. """ agent: str = NO_AGENT + #: Whether a verified-green dependency upgrade may be opened as a pull request here. DR-0019. + #: + #: **The first thing a project can refuse while Hullwork is perfectly able to do it**, and the + #: reason it exists is the gap item 186 could report and not close: until this field, declaring + #: an installer and a lock file *was* consenting to pull requests in your repository. Nobody + #: said so. That is the sentence this product already applies to lanes — *a policy nobody has + #: read is a policy nobody has agreed to* — pointed at itself. + #: + #: **False by default**, because every other default in this block is the refusing one and a + #: permission that arrives switched on is not a permission. + #: + #: DR-0019's rule for whether a feature gets a switch at all: *could a project have the + #: capability, understand the feature, and rationally not want it?* Applied to everything that + #: exists today it yields this field and nothing else — reading a lock file writes nothing, + #: verification runs on the operator's host, and both agent paths are already gated by `agent`. + open_upgrades: bool = False + @field_validator("agent") @classmethod def _an_engine_is_named_not_described(cls, value: str) -> str: diff --git a/hullwork/osv.py b/hullwork/osv.py new file mode 100644 index 0000000..34cb125 --- /dev/null +++ b/hullwork/osv.py @@ -0,0 +1,195 @@ +"""Asking OSV which of a project's pinned dependencies are known to be vulnerable. Item 172. + +**Why this database and not another**, recorded because DR-0016 had to check it before a line was +written: OSV is Apache-2.0, its API needs no key and no account, and it publishes no restriction on +commercial or hosted use. The same check removed CodeQL (free only on open-source code) and +Semgrep's own rule set (internal, non-competing, **non-SaaS**) from consideration on the same day — +either of those would have obliged this product's buyer to pay a competitor, or forbidden the +hosted edition DR-0015 plans. + +**It runs in the dispatcher and never in the sandbox.** The attempt has no network by design, and +nothing here is a reason to change that: the question *which versions are affected* is answered +from files on disk plus one host that is not the project's, before a container exists. +""" + +from __future__ import annotations + +import logging +from collections.abc import Sequence +from dataclasses import dataclass + +import httpx2 + +from hullwork.dependencies import Dependency + +log = logging.getLogger(__name__) + +#: The public instance. Addressed rather than configurable: a vulnerability database an operator +#: could point somewhere else is a supply-chain decision wearing a settings field. +OSV_URL = "https://api.osv.dev" + +#: What one batch may carry. The service's own documented limit, and the reason this never sends +#: one request per package — a project with a thousand pins would otherwise be a thousand requests. +BATCH = 1000 + + +@dataclass(frozen=True) +class Advisory: + """One published vulnerability, and the versions that end it. + + `fixed` is a tuple rather than a single value **on purpose**. An advisory that fixed a problem + on two release branches publishes two, and picking one would mean comparing versions across two + ecosystems' ordering rules — a wrong pick there is a bump that does not fix what it claims to. + All of them are reported and a person decides. + """ + + id: str + summary: str + fixed: tuple[str, ...] + + @property + def has_a_fix(self) -> bool: + """Whether there is any bump to attempt at all. + + Empty means the advisory publishes no fixed version — not that this failed to find one. + Proposing the next release and hoping is the guess this project does not make. + """ + return bool(self.fixed) + + @property + def url(self) -> str: + """Where a person reads it themselves, which is the point of naming the id.""" + return f"https://osv.dev/vulnerability/{self.id}" + + +@dataclass(frozen=True) +class Finding: + """A pinned dependency, and everything published against that exact version.""" + + dependency: Dependency + advisories: tuple[Advisory, ...] + + +class Osv: + """The vulnerability database, seen only as "which of these versions are affected?". + + Narrow on purpose, the way `TrackerInventory` is: an object that can only be asked one question + cannot accidentally be asked another. + """ + + def __init__(self, *, timeout: float = 20.0, transport: object | None = None) -> None: + self._client = httpx2.Client( + base_url=OSV_URL, + headers={"Accept": "application/json"}, + timeout=timeout, + follow_redirects=False, + **({"transport": transport} if transport is not None else {}), # type: ignore[arg-type] + ) + + def close(self) -> None: + self._client.close() + + def __enter__(self) -> Osv: + return self + + def __exit__(self, *exc: object) -> None: + self.close() + + def affected(self, deps: Sequence[Dependency]) -> list[Finding]: + """Every dependency with something published against its pinned version. + + Two round trips at most per batch: `querybatch` answers with ids only, so the detail — + which is where the fixing version lives — is fetched once per **id**, not once per package. + A clean project therefore costs exactly one request. + """ + findings: list[Finding] = [] + for start in range(0, len(deps), BATCH): + window = list(deps[start : start + BATCH]) + findings.extend(self._one_batch(window)) + return findings + + def _one_batch(self, window: Sequence[Dependency]) -> list[Finding]: + queries = [ + {"package": {"name": d.name, "ecosystem": d.ecosystem}, "version": d.version} + for d in window + ] + answered = self._post("/v1/querybatch", {"queries": queries}) + results = answered.get("results") if isinstance(answered, dict) else None + if not isinstance(results, list): + return [] + + # One fetch per distinct id: the same advisory routinely affects several packages, and + # asking for it once per package would multiply the requests by nothing gained. + detail: dict[str, dict[str, object]] = {} + findings: list[Finding] = [] + for dependency, result in zip(window, results, strict=False): + ids = _ids_in(result) + if not ids: + continue + advisories = [] + for vuln_id in ids: + if vuln_id not in detail: + detail[vuln_id] = self._get(f"/v1/vulns/{vuln_id}") + advisories.append(_advisory_for(dependency, vuln_id, detail[vuln_id])) + findings.append(Finding(dependency, tuple(advisories))) + return findings + + def _post(self, path: str, payload: dict[str, object]) -> dict[str, object]: + response = self._client.post(path, json=payload) + response.raise_for_status() + body = response.json() + return body if isinstance(body, dict) else {} + + def _get(self, path: str) -> dict[str, object]: + response = self._client.get(path) + response.raise_for_status() + body = response.json() + return body if isinstance(body, dict) else {} + + +def _ids_in(result: object) -> list[str]: + """The vulnerability ids in one `querybatch` result, which is `{}` when there are none.""" + if not isinstance(result, dict): + return [] + vulns = result.get("vulns") + if not isinstance(vulns, list): + return [] + return [v["id"] for v in vulns if isinstance(v, dict) and isinstance(v.get("id"), str)] + + +def _advisory_for( + dependency: Dependency, vuln_id: str, document: dict[str, object] +) -> Advisory: + """The fixing versions **for this package**, out of an advisory that may name several. + + One advisory routinely covers the same flaw across ecosystems. Reading every `fixed` event in + the document would hand a PyPI dependency npm's fixing version — a wrong answer that reads as + entirely plausible in a report, which makes it worse than an obvious one. + """ + fixed: list[str] = [] + affected = document.get("affected") + for entry in affected if isinstance(affected, list) else []: + if not isinstance(entry, dict): + continue + package = entry.get("package") + if not isinstance(package, dict): + continue + same = package.get("name") == dependency.name + if not same or package.get("ecosystem") != dependency.ecosystem: + continue + ranges = entry.get("ranges") + for one in ranges if isinstance(ranges, list) else []: + if not isinstance(one, dict): + continue + events = one.get("events") + for event in events if isinstance(events, list) else []: + if isinstance(event, dict) and isinstance(event.get("fixed"), str): + fixed.append(str(event["fixed"])) + + summary = document.get("summary") + return Advisory( + id=vuln_id, + summary=summary if isinstance(summary, str) else "", + # De-duplicated, order preserved: two ranges can name the same fixing version. + fixed=tuple(dict.fromkeys(fixed)), + ) diff --git a/hullwork/outcomes.py b/hullwork/outcomes.py index 4c92bd5..5ded31e 100644 --- a/hullwork/outcomes.py +++ b/hullwork/outcomes.py @@ -322,3 +322,174 @@ def review_lines(counted: Reviewed) -> list[str]: median = ordered[len(ordered) // 2] out.append(f"median time from first error to decision: {spoken(median)}") return out + + +#: Attempt outcomes that put a **verdict the gates produced** behind an item. Item 183, DR-0017. +#: +#: The two refusals belong here and that is the decision's second consequence, not a rounding: *"I +#: could not verify this" is a first-class result*, and a reasoned refusal with the runs attached is +#: what this product promises to deliver where nobody else does. What is excluded is everything that +#: says nothing about the claim — `abandoned` because the infrastructure got in the way, +#: `baseline-red` because the project's own suite stopped it before a model was called, +#: `already-fixed` because it is a fact about a deployment. +LEFT_WITH_EVIDENCE: frozenset[AttemptOutcome] = frozenset( + { + AttemptOutcome.PR_OPEN, + AttemptOutcome.PR_OPEN_LINT_FAILED, + AttemptOutcome.FAILED, + AttemptOutcome.NOT_REPRODUCIBLE, + } +) + +#: The two of those that carry a change somebody can merge. The rest are refusals, and the split is +#: printed rather than totalled — see `desk_lines`. +CARRIED_A_CHANGE: frozenset[AttemptOutcome] = frozenset( + {AttemptOutcome.PR_OPEN, AttemptOutcome.PR_OPEN_LINT_FAILED} +) + +#: States that mean Hullwork put this on somebody's desk rather than taking it off. Item 183. +#: +#: **The row `Funnel` cannot have**, and the reason this count exists. DR-0017's Context says the +#: first half of the pipeline is a *cost* to the buyer — *"a team of three to ten with Sentry has no +#: detection problem; it has more issues than it can serve. The opening move of the product adds to +#: the pile."* An item nobody may attempt is exactly that, and a number that cannot express it is a +#: number that flatters. +HANDED_OVER: frozenset[ItemState] = frozenset({ItemState.HUMAN_ONLY, ItemState.REJECTED}) + + +@dataclass +class Desk: + """How much of what arrived left a person's desk with evidence attached. Item 183, DR-0017. + + **The denominator is the whole point.** `Funnel`'s is `fair_try` — attempts that spent an item's + one try — so every question it can answer has the shape *of the attempts we made, how did they + go*. That is the "how many bugs did it fix" number in a more careful coat, and DR-0017's third + consequence names it as the one being replaced. This one counts **what arrived**, which is a + different question with a much worse available answer. + """ + + #: Claims that arrived. Every item is one, whatever became of it. + arrived: int = 0 + #: Items with a verdict the gates produced behind them. + left_with_evidence: int = 0 + #: …of those, the ones carrying a change. The remainder are refusals, and both are printed. + with_a_change: int = 0 + with_a_refusal: int = 0 + #: Still in the queue, including anything whose attempt was abandoned and went back. + still_waiting: int = 0 + #: **Put on** a person's desk: red lane, or a pull request a human read and refused. + handed_over: int = 0 + #: Attempted and not finished. Neither, and saying so costs one word (`Funnel`'s rule). + running: int = 0 + + def as_dict(self) -> dict[str, object]: + """For `--json`. No ratio and no percentage, for `Funnel.as_dict`'s reason: an operator who + wants one computes it from parts they can see, and six samples do not carry that precision. + """ + return { + "arrived": self.arrived, + "left_with_evidence": self.left_with_evidence, + "with_a_change": self.with_a_change, + "with_a_refusal": self.with_a_refusal, + "still_waiting": self.still_waiting, + "handed_over": self.handed_over, + "running": self.running, + } + + +def desk(session: Session) -> Desk: + """Count what became of every claim that arrived. Item 183. + + **By the attempt trail and never by the item's state**, which is the one thing here that could + be got wrong quietly. `done` is reached both by a merged pull request and by a person fixing + their own bug and closing the issue, and there is no state history to separate them — so an item + counted by state would have this product claiming credit for somebody else's afternoon. + + One pass over two tables rather than a query per bucket: this runs inside `status`, which an + operator types when something is already wrong. + """ + verdicts = session.execute( + select(Attempt.item_id, Attempt.outcome, Attempt.rehearsal).where( + Attempt.outcome.is_not(None) + ) + ).all() + + settled: dict[int, AttemptOutcome] = {} + started: set[int] = set() + for item_id, outcome, rehearsal in verdicts: + if rehearsal: + # It publishes nothing, so no forge state and nobody's queue moved. `Funnel` keeps + # rehearsals out of every number for the same reason. + continue + started.add(item_id) + # The best verdict an item ever got, so a second attempt that abandoned cannot take an + # earned one away. Items get one attempt (DR-0003), and this is what makes that assumption + # visible rather than relied on. + if outcome in LEFT_WITH_EVIDENCE and settled.get(item_id) not in LEFT_WITH_EVIDENCE: + settled[item_id] = outcome + + running = { + item_id + for (item_id,) in session.execute( + select(Attempt.item_id).where( + Attempt.outcome.is_(None), Attempt.rehearsal.is_(False) + ) + ).all() + } - started + + counted = Desk() + for item_id, state in session.execute(select(Item.id, Item.state)).all(): + counted.arrived += 1 + outcome = settled.get(item_id) + if outcome is not None: + counted.left_with_evidence += 1 + if outcome in CARRIED_A_CHANGE: + counted.with_a_change += 1 + else: + counted.with_a_refusal += 1 + elif item_id in running: + counted.running += 1 + elif state in HANDED_OVER: + counted.handed_over += 1 + else: + counted.still_waiting += 1 + return counted + + +def desk_lines(counted: Desk) -> list[str]: + """The number DR-0017 signed for, in words. Empty when nothing has arrived. + + **Nothing here is phrased as an achievement**, and the row for what was *added* to a desk least + of all: it is the one line in this product that can embarrass it, and rounding it into good news + is exactly how it would stop doing that. + + Zeros are not printed. An instance whose claims have all cleared says so with one line, and an + instance that has cleared none says *that* — which is not the same fact as a row of noughts, and + is the state every instance starts in. + """ + if not counted.arrived: + return [] + + said = [f"{counted.arrived} claim(s) have arrived"] + if counted.left_with_evidence: + how = [] + if counted.with_a_change: + how.append(f"{counted.with_a_change} with a change") + if counted.with_a_refusal: + how.append(f"{counted.with_a_refusal} with a reasoned refusal and the runs behind it") + said.append( + f"{counted.left_with_evidence} left your desk with evidence attached: " + ", ".join(how) + ) + else: + said.append("none of them has left your desk with evidence attached yet") + if counted.still_waiting: + said.append(f"{counted.still_waiting} are still in the queue") + if counted.running: + said.append(f"{counted.running} are being attempted now, which is neither") + if counted.handed_over: + # Said plainly, and this is the sentence DR-0017's Context is about. + said.append( + f"{counted.handed_over} went onto your desk rather than off it: red lane, or a pull " + f"request somebody read and refused" + ) + return said diff --git a/hullwork/page.py b/hullwork/page.py index 6f0790b..e644460 100644 --- a/hullwork/page.py +++ b/hullwork/page.py @@ -1154,9 +1154,9 @@ def instance( """What `hullwork status` says, for somebody who does not have a terminal on this host. **Every number comes from the function `status` calls**, never from a second query written for - this page: `readiness.check`, `outcomes.funnel`, `recurrence.counted` and `undecided`, - `lease.state` and `reporting_of`. A page that recomputed them would drift, and the first anybody - would know is a reader and an operator disagreeing about the same instance. + this page: `readiness.check`, `outcomes.desk`, `outcomes.funnel`, `recurrence.counted` and + `undecided`, `lease.state` and `reporting_of`. A page that recomputed them would drift, and the + first anybody would know is a reader and an operator disagreeing about the same instance. """ from hullwork import lease, outcomes, readiness, recurrence @@ -1195,6 +1195,11 @@ def instance( ), ] table = "".join(f"{_h(name)}{_h(value)}" for name, value in rows) + # **The number DR-0017 is measured by** (item 183), and it was in the terminal and not here — + # which is the same defect item 136 already found on this page once: a fact the instance knew, + # put where nobody reading would find it. The interface design says this surface exists + # to show what was verified and what was not; a count of attempts is not that, and this is. + desk = "".join(f"
  • {_h(line)}
    • " for line in outcomes.desk_lines(outcomes.desk(session))) attempts = "".join(f"
  • {_h(line)}
    • " for line in outcomes.lines(outcomes.funnel(session))) spent = "".join( f"
  • {_h(line.strip())}
    • " @@ -1245,6 +1250,10 @@ def instance( "How this instance is configured", f'
    {table}
    ', ) + # Before the attempts block, exactly as `status` orders them: this one has *what arrived* + # as its denominator and that one has *what was attempted*, so a reader who opens one + # should meet the wider question first. + + (_fold("What arrived, and how much left your desk", f"") if desk else "") + (_fold("What its attempts came to", f"") if attempts else "") + (_fold("What they cost", f"") if spent else "") + (_fold("What reviewers did", f"") if reviewed else "") diff --git a/hullwork/propose.py b/hullwork/propose.py index 1c60d7a..85bf1a0 100644 --- a/hullwork/propose.py +++ b/hullwork/propose.py @@ -45,7 +45,7 @@ import re from collections.abc import Callable from dataclasses import dataclass, field -from typing import Any +from typing import Any, NamedTuple import yaml @@ -61,6 +61,83 @@ ".gitlab-ci.yml", ) +#: CI locations that name the forge holding the repository, and the ones that do not. +#: +#: **`.github/workflows/` is deliberately absent, and that absence is the whole subject of +#: item 171.** Forgejo Actions and Gitea Actions both read that directory — this repository's +#: own deployment runs those workflows on a Forgejo instance — so treating it as evidence of +#: GitHub would be wrong for exactly the self-hosted projects this product is for. +CI_NAMES_THE_FORGE: tuple[tuple[str, str], ...] = ( + (".forgejo/workflows/", "forgejo"), + (".gitea/workflows/", "gitea"), + (".gitlab-ci.yml", "gitlab"), +) + +#: Hosts that name themselves. Everything else is self-hosted and unresolvable from here: +#: `git.example.com` may be Forgejo, Gitea, a private GitHub or a self-hosted GitLab, and no +#: request may be made to find out — `propose` reaches nothing and needs no credential. +HOSTS_THAT_NAME_THE_FORGE: dict[str, str] = { + "github.com": "github", + "gitlab.com": "gitlab", +} + +#: What `git.provider` says when nothing decided. +#: +#: A value rather than a placeholder, unlike `_coordinate_of`'s `owner/name`. The field is +#: required, so an unparseable proposal would cost every reader a fix to serve the undecidable +#: minority — and unlike a repository coordinate, a wrong forge name is something an operator +#: recognises on sight. What it must not do is look like a reading, so `render` says it is a +#: default and names what would have settled it. +PROVIDER_WHEN_UNDECIDED = "forgejo" + + +class ForgeGuess(NamedTuple): + """Which forge holds a repository, and what said so. + + `evidence` is `None` when nothing did. That is not a detail for the caller to ignore: the + difference between an observation and a default is what `render`'s contract is about. + """ + + provider: str + evidence: str | None + + +def host_of_remote(url: str | None) -> str | None: + """The host out of a git remote URL, in either spelling, or `None`. + + `_coordinate_of` has parsed this URL since item 107 and kept only the last two segments — + discarding the one part of it that says which forge this is (item 171). + """ + if not url: + return None + trimmed = url.strip().removesuffix(".git") + for scheme in ("https://", "http://", "ssh://", "git://"): + trimmed = trimmed.removeprefix(scheme) + # `git@host:owner/name` and `git@host/owner/name` after the scheme is gone. + _, _, after_user = trimmed.rpartition("@") + host = re.split(r"[:/]", after_user, maxsplit=1)[0] + # A host has a dot and no whitespace. Anything else was not a URL, and guessing from it + # would be the constant-in-a-costume this function exists to remove. + if not host or " " in host or "." not in host: + return None + return host.lower() + + +def forge_for(source: str | None, remote_host: str | None) -> ForgeGuess: + """Which forge holds this repository. Pure, and reaches nothing. + + **The host outranks the CI location**, because where a repository lives beats which runner + reads its workflows: a GitHub repository whose workflows sit in `.forgejo/workflows/` is a + mirror, and the coordinate a manifest needs is the one that answers requests. + """ + named = HOSTS_THAT_NAME_THE_FORGE.get(remote_host or "") + if named: + return ForgeGuess(named, f"the origin remote is on {remote_host}") + for prefix, provider in CI_NAMES_THE_FORGE: + if source and source.startswith(prefix): + return ForgeGuess(provider, f"the CI configuration is at {prefix}") + return ForgeGuess(PROVIDER_WHEN_UNDECIDED, None) + #: Package-manager invocations that mean "this step installs the dependencies". #: #: About a dozen, stable for years, and **being wrong here is free**: an unrecognised @@ -200,6 +277,9 @@ class Proposal: repo: str #: The CI file it read, or `None` when there was none. source: str | None = None + #: The host of the `origin` remote, when this was read from a checkout that has one. + #: Item 171 — the strongest signal for `git.provider`, and it was being thrown away. + remote_host: str | None = None base: str | None = None install: str | None = None tests: str | None = None @@ -904,6 +984,25 @@ def only_files_that_exist(proposal: Proposal, paths: object) -> Proposal: return proposal +def _the_git_lines(proposal: Proposal) -> list[str]: + """`git:`, and whether its provider was read or defaulted to. Item 171. + + Uncommented means observed, everywhere else in this output. A constant printed under that + rule is the failure this file exists to avoid, so an undecided provider is preceded by a + comment saying so rather than being quietly indistinguishable from a reading. + """ + guess = forge_for(proposal.source, proposal.remote_host) + line = f"git: {{provider: {guess.provider}, repo: {proposal.repo}}}" + if guess.evidence: + return [line] + return [ + "# `provider` below is a default, not a reading: neither the origin remote's host", + "# nor the CI path named a forge. `.github/workflows/` cannot name one — Forgejo", + "# and Gitea Actions read that directory too. Correct it if this is not a Forgejo.", + line, + ] + + def render(proposal: Proposal) -> str: """The proposal as manifest text: observed values live, everything else commented. @@ -920,7 +1019,7 @@ def render(proposal: Proposal) -> str: "# that looks finished is a proposal nobody checks.", "", f"project: {name}", - f"git: {{provider: forgejo, repo: {proposal.repo}}}", + *_the_git_lines(proposal), "", ] @@ -970,6 +1069,36 @@ def render(proposal: Proposal) -> str: "# install from — it is the cache key — and this reader could not name one", "# for that command. Add the file your project reads and uncomment both.", ] + elif proposal.base: + # **What no installer costs, said where the field is not** (item 185). Every other field + # this reader cannot fill carries a comment explaining what is missing; `install` carried + # none, because its absence produces a manifest that **parses and builds perfectly**. What + # it cannot do is measure a dependency upgrade: with no installer the image is the base + # exactly as it comes, so rewriting a pin changes nothing the suite runs against. + # + # Measured on 2026-08-09 (item 182) before this comment existed: a checkout pinning + # `jinja2==2.4.1`, a base image carrying 3.0.0, and a verdict reading *your suite passed + # before this change and passes after it* — about a version that was never installed. + # + # The command is deliberately not named: it is not in the published image, and naming + # something a reader cannot run invites them to type it and be told it does not exist. + lines += [ + f"{mark}# No `install:` — nothing in the CI file named one, and this reader does not", + f"{mark}# guess between pip, uv and poetry from a lock file. Your tests will run: the", + f"{mark}# image is `{proposal.base}` exactly as it comes.", + f"{mark}#", + f"{mark}# What it costs: dependency upgrades cannot be **measured** against this", + f"{mark}# manifest. Nothing is installed from a lock file, so changing a pinned", + f"{mark}# version changes nothing your suite would run against, and a green suite", + f"{mark}# would say nothing about the upgrade.", + f"{mark}#", + f"{mark}# To answer that, keep this base and add two lines — the command your CI", + f"{mark}# already uses, and the file it reads:", + f"{mark}# install: ", + f"{mark}# dependencies: []", + f"{mark}# That is one layer on top of the image you named, not a rebuild from", + f"{mark}# scratch, and it is why no list here has to grow for your ecosystem.", + ] if proposal.packages: lines.append(f"{mark}packages: [{', '.join(proposal.packages)}]") if proposal.dependencies: diff --git a/hullwork/refit.py b/hullwork/refit.py new file mode 100644 index 0000000..8fd0850 --- /dev/null +++ b/hullwork/refit.py @@ -0,0 +1,406 @@ +"""The fix for the ones that break. Item 179, DR-0018 step 4. + +`deps --verify` ends with a queue, and the middle of it is the interesting part: *six break, tests +named*. Renovate leaves those where they fell; DR-0018 says this is the one item on its list nobody +else could ship. Making a broken upgrade fit is a refactor, it is what everybody postpones, and the +loop this repository already has applies to it without modification. + +**Cheaper than it looks, because the expensive half is already paid for.** DR-0003's cost is *write +a test that reproduces the problem and show it failing first*, and item 174 produces exactly that as +a by-product of the verdict: the project's own tests, failing against the upgraded dependency, with +nobody having authored them for the occasion. No agent writes the oracle here, which is the property +`docs/what-hullwork-is.md` says every verdict rests on. + +**What this module is and is not.** The sequence and its gates belong to `dispatch.refit`; the world +those run in belongs to `work._attempt`. What is here is the part neither of them should know: what +a breakage *is*, what the agent is told about it, and how the version is read back out of the tree +afterwards. Nothing here starts a container or touches a forge. +""" + +from __future__ import annotations + +import logging +import re +import shutil +import tempfile +from collections.abc import Sequence +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING + +from sqlalchemy.orm import Session + +from hullwork import bump, dependencies, resolve +from hullwork.manifest import Manifest +from hullwork.models import Item, ItemKind, ItemState, Lane, Project +from hullwork.normalise import derive_fingerprint +from hullwork.states import transition + +if TYPE_CHECKING: # `work` imports this module inside `_attempt`, so the runtime import is lazy + from hullwork.config import Settings + from hullwork.work import Outcome + +log = logging.getLogger(__name__) + + +def _canonical(name: str) -> str: + """PEP 503 again, and for the third time in this repository deliberately rather than shared. + + `bump` and `resolve` each carry one because each is about a different file's spellings. This one + compares what OSV named against what a lock reader read back, and the two disagree the same way: + `Jinja2`, `jinja_2` and `jinja.2` are one distribution. npm names pass through unchanged, since + they contain none of the characters this collapses. + """ + return re.sub(r"[-_.]+", "-", name).lower() + + +@dataclass(frozen=True) +class Upgrade: + """One upgrade that broke a suite, and everything a refit needs to know about it. + + Frozen, and built once from the verdict rather than re-derived: the version this is about is + the one the gates ran against, and a second derivation is a second chance to disagree with it. + """ + + package: str + #: What the project pins today. Carried for the report, not for the run — nothing here ever + #: puts it back, which is the whole point of the item. + was: str + #: What the upgrade moves to, and what the tree must still pin when the green gate passes. + to: str + #: The dependency file that pins it, as a path relative to the checkout. + source: str + #: Every file moving this dependency can rewrite — read-only to the fix phase. Item 175 measured + #: that `npm install` rewrites `package.json` as well as the lock, so guarding only the file + #: that pins would leave the range widened back with the pin looking untouched. + guarded: tuple[str, ...] + #: The tests that failed with the upgrade applied, in the runner's own words. This is the + #: evidence, and it was written by the project rather than for the occasion. + failing: str = "" + #: The advisory that started this, when there is one. Empty is legal: an upgrade can be worth + #: making fit without anything published against the version it replaces. + advisory: str = "" + url: str = "" + + @property + def title(self) -> str: + """What a person reads in a queue. The pair of versions, because that is the work.""" + return f"{self.package} {self.was} → {self.to} breaks this project's suite" + + @property + def fingerprint(self) -> str: + """The identity of this work: the package **and both versions**, never the package alone. + + A fingerprint over the name would make next month's upgrade of the same library a repeat of + this one — `dedup` would increment a counter and no work would be created, which is the + failure mode that is invisible because it looks like deduplication working. + """ + return derive_fingerprint("deps", self.package, self.was, self.to, self.source) + + +def from_report( + report: bump.Report, + *, + source: str, + guarded: tuple[str, ...] = (), + advisory: str = "", + url: str = "", +) -> Upgrade | None: + """The upgrade worth handing to an agent, or `None` when this report is not that. + + **Only `needs work` reaches here.** A clean verdict is item 178's to deliver and needs no + agent; a red baseline is the project's own problem and nothing can be claimed against it; a + blocked one has nothing to try. Filtering by `needs_of` rather than by scanning for a `breaks` + answer is what keeps those three out — a report can carry a `breaks` answer *and* a later clean + one, and that is a package to take rather than work to do. + + Among the candidates that broke, the one that broke **fewest** tests. Same reasoning as + `bump.broke`'s ordering: the upgrade that breaks two tests is the one that can be closed this + afternoon, and starting with the twelve-test one buries the achievable under the daunting. + """ + if bump.needs_of(report) is not bump.Needs.NEEDS_WORK: + return None + broke = [a for a in report.answers if a.verdict is bump.Verdict.BREAKS] + if not broke: # pragma: no cover - `needs_of` returns NEEDS_WORK only when one exists + return None + chosen = min(broke, key=lambda a: (_failure_count(a.detail), a.to)) + return Upgrade( + package=report.package, + was=report.was, + to=chosen.to, + source=source, + guarded=guarded or (source,), + failing=chosen.detail, + advisory=advisory, + url=url, + ) + + +def _failure_count(detail: str) -> int: + """How many tests a `breaks` answer named. Blank lines are not failures.""" + return len([line for line in detail.splitlines() if line.strip()]) + + +def guarded_for(source: str) -> tuple[str, ...]: + """Every file that moving this dependency can rewrite, asked of the resolver that owns it. + + Read from `resolve.touches` rather than listed here, so an ecosystem added there is guarded here + without anybody remembering to. A file with no resolver is a list of versions and is the only + file its own move touches. + """ + resolver = resolve.resolver_for(source) + return resolve.touches(resolver) if resolver is not None else (source,) + + +def version_now(upgrade: Upgrade, worktree: Path) -> str | None: + """What this tree pins the package at **now**, read back after the gates have run. + + Item 172's readers rather than a fifth parser: they already know all four file shapes, they are + the ones `deps` used to find this dependency in the first place, and a second reader is a second + thing that can come to disagree about what a file says. + + `None` for a tree that no longer pins it at all — a deleted file, a removed line — which is a + different fact from pinning the old version and is reported as one. + """ + + def read(path: str) -> str | None: + try: + return (worktree / path).read_text(encoding="utf-8", errors="replace") + except OSError: + return None + + wanted = _canonical(upgrade.package) + for found in dependencies.read_lockfiles([upgrade.source], read): + if _canonical(found.name) == wanted: + return found.version + return None + + +#: How much of the runner's output goes into the brief. Enough for the failures and their messages, +#: bounded because a suite that fails 300 tests must not become most of the prompt. +MAX_FAILING_CHARS = 3_000 + + +def brief(upgrade: Upgrade) -> str: + """What the agent is told. **Not `brief.build`**, and the difference is the honesty of it. + + That one answers *what Hullwork knows about this error* from the tracker and this instance's + history, and for a refit every one of those fields is empty — there is no error, no fingerprint + from a stranger, no occurrence count. A brief built from it would open by saying the full event + was never fetched, which is true of a tracker nobody asked and misleading about work that has + better evidence than any tracker produces. + + Nothing here is untrusted in the sense `brief.build` fences against: the package name and the + versions came from a lock file and a vulnerability database, and the failing tests came from the + project's own runner. The runner's output is still bounded, because a suite can print for as + long as you let it. + """ + failing = upgrade.failing.strip()[:MAX_FAILING_CHARS] or "(the runner named none)" + lines = [ + "# What Hullwork knows about this upgrade", + "", + "This is context you cannot get by reading the repository: the upgrade below has already " + "been applied to the checkout you are working in, and these are the tests it broke when " + "Hullwork ran your own suite against it.", + "", + "## The upgrade", + "", + f"- Package: `{upgrade.package}`", + f"- Pinned at: `{upgrade.was}`", + f"- Applied here: `{upgrade.to}`", + f"- Pinned by: `{upgrade.source}`", + ] + if upgrade.advisory: + lines.append(f"- Advisory: {upgrade.advisory}{f' — {upgrade.url}' if upgrade.url else ''}") + lines += [ + "", + "## What it broke", + "", + "Your own tests, run by Hullwork with the upgrade applied and nothing else changed:", + "", + "```text", + failing, + "```", + "", + "## What you are being asked for", + "", + "Change this project's own source code so those tests pass with the new version. The whole " + "suite has to pass, not only the ones named above.", + "", + "## What you must not do", + "", + f"**The dependency files are read-only.** {', '.join(upgrade.guarded)} — do not edit, " + f"delete or replace any of them.", + "", + f"Putting `{upgrade.package}` back to `{upgrade.was}` would make the suite pass and is not " + f"a fix: it is a revert, and it undoes the upgrade this work exists to make possible. " + f"Hullwork restores those files before it runs the suite again and reads the version back " + f"out of the tree afterwards, so a revert is reported as a revert rather than published as " + f"a fix.", + "", + "If the upgrade genuinely cannot be made to work, change nothing and say so. That is a " + "correct and useful answer, and it is a better one than a change that only looks like a " + "fix.", + "", + ] + return "\n".join(lines) + + +def prepare( + checkout: Path, upgrade: Upgrade, *, present: Sequence[str], into: Path +) -> str | None: + """A copy of the checkout with the upgrade already in it, or the reason there is not one. + + **The upgrade goes in before the attempt starts, not during it**, and that is what makes the + first gate a red gate rather than a baseline. The agent then opens a tree where the new version + is simply what the project pins, which is also the tree a reviewer will see. + + Applied by the same two paths `bump` uses and not by a third: the ecosystem's own resolver for a + resolved graph (`resolve.upgrade`, which does not believe the tool's exit code either), and + `bump.editing` for a file that is a list of versions. A refit that moved a dependency its own + way would be a second opinion about what an upgrade is. + + **`.git` is not copied.** The agent gets history only if somebody chose to give it, and a + worktree with a repository in it can grow a hook that runs on the host — `prepare_worktree`'s + reasoning, and this directory is handed to the same machinery. + """ + shutil.copytree( + checkout, into, dirs_exist_ok=True, ignore=shutil.ignore_patterns(".git") + ) + resolver = resolve.resolver_for(upgrade.source) + if resolver is None: + try: + bump.can_rewrite(upgrade.source) + except bump.CannotRewriteError as refused: + return str(refused) + return bump.editing(upgrade.source, upgrade.package, upgrade.to)(into) + outcome = resolve.upgrade( + resolver=resolver, + worktree=into, + package=upgrade.package, + version=upgrade.to, + present=present, + run=resolve.in_a_container, + ) + return None if outcome.ok else f"{outcome.outcome.value}: {outcome.detail}" + + +def stage( + session: Session, manifest: Manifest, upgrade: Upgrade, *, repo: str +) -> tuple[Project, Item]: + """Put one project and one item where the dispatcher reads them from. Item 179. + + Shaped after `trial.stage`, and it differs in one place on purpose: that one goes through + `dedup.resolve` because a pasted stack trace has to be triaged, and the lane it lands in is the + product working. **There is nothing to triage here.** Triage matches lane rules against an + error's title and the code location that raised it, and this work has neither — the identity is + a package and two versions, both known exactly, and the code that will be changed is not known + until the agent has read the failures. + + So the lane is stated rather than derived, with the reason on the item where a person can + disagree with it. Green, because what an agent is being asked to touch is the project's own + source until its own suite passes again — the same territory `autofix` already covers — and + because the operator asked for this upgrade by name. + """ + project = session.query(Project).filter(Project.repo == repo).one_or_none() + if project is None: + project = Project( + slug=repo.rsplit("/", 1)[-1], + forge=manifest.git.provider, + repo=repo, + webhook_secret_hash="", # nothing listens: a refit has no webhook to authenticate + manifest=manifest.model_dump(mode="json"), + ) + session.add(project) + session.flush() + + item = Item( + project_id=project.id, + fingerprint=upgrade.fingerprint, + title=upgrade.title, + kind=ItemKind.OTHER, + lane=Lane.GREEN, + lane_reason=( + "a dependency upgrade the operator named, whose failing tests are the project's own — " + "there is no error to triage and no culprit to match a lane rule against, so the lane " + "is stated here rather than derived" + ), + permalink=upgrade.url or None, + ) + session.add(item) + session.flush() + # Through the state machine and never by assignment, which is item 042's single door. `new` is + # where a row starts and `triaged` is what it has to pass through, even when — as here — the + # triage was a decision rather than a match. + transition(item, ItemState.TRIAGED) + transition(item, ItemState.READY) + session.flush() + return project, item + + +class NotUpgradableError(Exception): + """The upgrade could not be put into the tree, so no attempt was started. Item 179. + + Its own exception because nothing failed: a manifest whose range forbids the version, a lock + file with no resolver, a registry that refused — each is a fact about the project, and the + attempt was never begun, so nothing was consumed and nothing is owed. `resolve.upgrade`'s own + refusals arrive here word for word rather than being summarised into "could not upgrade". + """ + + +def run( + settings: Settings, + checkout: Path, + manifest: Manifest, + upgrade: Upgrade, + *, + present: Sequence[str], + into: Path, + repo: str, +) -> Outcome: + """One refit, end to end. Composes what exists; decides nothing new. + + Shaped after `trial.run` and for the same reason: an ephemeral database, no forge anywhere in + the call path, and the artefact written to disk through `write_locally`. The forges are `None` + and `_attempt` is handed a checkout, so nothing here can reach one — which is stronger than not + configuring one, and it keeps this half of DR-0018 on the credential-free side of item 178. + + **What it does not remove is Docker and a model credential**, and saying so is part of the + honesty. The claim is that a project's own suite failed with an upgrade applied and passes with + a change, run in a sandbox, by a model whose identity was read off the wire. Faking either turns + this into a demonstration of itself. + """ + from hullwork import trial, work + from hullwork.scrub import instance_secrets + + session = trial.ephemeral_session() + project, item = stage(session, manifest, upgrade, repo=repo) + + # The sha of the tree the upgrade goes on top of, read **before** the copy: `prepare` leaves + # `.git` behind, so afterwards there is nothing to ask. Everything the artefact claims is a + # claim about one commit, and one that said `unknown` would be an artefact nobody could check. + base = trial.head_sha(checkout) + upgraded = Path(tempfile.mkdtemp(prefix="hullwork-refit-")) + try: + refused = prepare(checkout, upgrade, present=present, into=upgraded) + if refused is not None: + raise NotUpgradableError(refused) + log.info( + "refit starting", + extra={"package": upgrade.package, "to": upgrade.to, "sha": base}, + ) + return work._attempt( + session, + settings, + work.Eligible(item=item, project=project), + code_forge=None, + forge=None, + credential=work._model_credential(settings), + secrets=instance_secrets(settings), + rehearse_into=into, + local_checkout=work.Checkout(path=upgraded, sha=base), + upgrade=upgrade, + ) + finally: + shutil.rmtree(upgraded, ignore_errors=True) + session.close() diff --git a/hullwork/resolve.py b/hullwork/resolve.py new file mode 100644 index 0000000..440cf1c --- /dev/null +++ b/hullwork/resolve.py @@ -0,0 +1,280 @@ +"""Moving a resolved dependency graph, by running the tool that knows how. Item 175, DR-0016. + +**Why this is not a file edit.** A lock file is a *resolved graph*: moving `jinja2` can require +moving `markupsafe`, and moving that can require moving something else. Editing one version string +leaves a file that is internally incoherent — and the bad outcome is not that it fails to install, +it is that it **installs**, after which a green suite means nothing at all. That is a false verdict +of the exact kind DR-0017 says this product exists to prevent, produced by the product itself. + +Only the ecosystem's own resolver knows how to move that graph. So nothing here parses a lock file. +It runs `npm`, or `uv`, or `poetry`, in a container, and reads back what they wrote. + +**And it does not believe them either.** A manifest can forbid the upgrade — `"lodash": "^4.17.11"` +does not permit 5.x — and every one of these tools reports success after resolving to the highest +version the range allows, having not applied the fix. So the lock is re-read and the version +checked. The tool's exit code is not the verdict, which is the same rule the gates run on. +""" + +from __future__ import annotations + +import json +import logging +import tomllib +from collections.abc import Callable, Sequence +from dataclasses import dataclass +from enum import StrEnum +from pathlib import Path + +log = logging.getLogger(__name__) + +#: How long a resolver may take. Generous: it is a registry round trip plus a graph solve, and a +#: cold npm cache on a large tree is genuinely slow. +RESOLVE_TIMEOUT_SECONDS = 600 + + +class Outcome(StrEnum): + """What running the ecosystem's resolver did.""" + + #: The lock moved and the version in it is the one that was asked for. + RESOLVED = "resolved" + #: The tool succeeded and the package did not move: the manifest's range forbids it. + CONSTRAINED = "constrained-by-manifest" + #: The tool failed. Its own output is carried. + FAILED = "failed" + #: A file the resolver needs is not in the checkout. + MISSING = "missing-manifest" + + +@dataclass(frozen=True) +class Result: + outcome: Outcome + detail: str = "" + + @property + def ok(self) -> bool: + return self.outcome is Outcome.RESOLVED + + +@dataclass(frozen=True) +class Resolver: + """One ecosystem's way of moving its own graph, declared as data. + + A table entry rather than a code path, so a new ecosystem is a row: `needs` is what must be in + the context for the tool to work at all, `image` is where the tool lives, and `command` is a + template. Nothing here knows what any of these tools do. + """ + + #: Which lock file this resolver owns. + lock: str + #: Everything that must be present, lock included. **The manifest is not optional**: every + #: resolver reads it to know what versions are allowed, and without it they resolve nothing. + needs: tuple[str, ...] + image: str + #: `{package}` and `{version}` are substituted. Run through `sh -lc`. + command: str + + +#: The ecosystems whose graphs can be moved, and how. +#: +#: `--package-lock-only` on npm moves the graph without downloading `node_modules`, which is the +#: difference between seconds and minutes. The two Python entries are separate resolvers rather +#: than one because `uv` and `poetry` disagree about everything except the file they read. +RESOLVERS: tuple[Resolver, ...] = ( + Resolver( + lock="package-lock.json", + needs=("package.json", "package-lock.json"), + image="node:22-slim", + command="npm install {package}@{version} --package-lock-only --no-audit --no-fund", + ), + Resolver( + lock="uv.lock", + needs=("pyproject.toml", "uv.lock"), + image="ghcr.io/astral-sh/uv:python3.12-bookworm-slim", + command="uv lock --upgrade-package {package}=={version}", + ), + Resolver( + lock="poetry.lock", + needs=("pyproject.toml", "poetry.lock"), + image="python:3.12-slim", + command=( + "pip install --quiet poetry && " + "poetry add {package}=={version} --lock --no-interaction" + ), + ), +) + + +def resolver_for(source: str) -> Resolver | None: + """The resolver that owns this lock file, or `None` when nothing does. + + `None` is not a gap to fill silently: item 173's refusal still applies to it, by name, so a + lock file nobody can move is still declined rather than edited by hand. + """ + name = source.rsplit("/", 1)[-1] + return next((r for r in RESOLVERS if r.lock == name), None) + + +def version_in_lock(text: str, lock: str, package: str) -> str | None: + """What the lock says this package is pinned at now, or `None` if it does not carry it. + + Read back **after** the resolver has run, because a tool that resolved within a range the + manifest allows exits 0 having moved nothing — and taking that as success would publish a + `clean` verdict for an upgrade that never happened. + """ + name = lock.rsplit("/", 1)[-1] + if name == "package-lock.json": + try: + document = json.loads(text) + except ValueError: + return None + packages = document.get("packages") + if not isinstance(packages, dict): + return None + for path, entry in packages.items(): + if not path or not isinstance(entry, dict): + continue + if path.split("node_modules/")[-1] == package: + found = entry.get("version") + return found if isinstance(found, str) else None + return None + + try: + document = tomllib.loads(text) + except tomllib.TOMLDecodeError: + return None + entries = document.get("package") + if not isinstance(entries, list): + return None + wanted = _canonical(package) + for entry in entries: + if isinstance(entry, dict) and _canonical(str(entry.get("name", ""))) == wanted: + found = entry.get("version") + return found if isinstance(found, str) else None + return None + + +def _canonical(name: str) -> str: + """PEP 503 again — `Jinja2` and `jinja-2` are one package, and lock files disagree on which.""" + import re + + return re.sub(r"[-_.]+", "-", name).lower() + + +def command_for(resolver: Resolver, package: str, version: str) -> str: + """The template filled in. Separate so it can be asserted without a daemon.""" + return resolver.command.format(package=package, version=version) + + +def missing_from(resolver: Resolver, present: Sequence[str]) -> list[str]: + """Which of the files this resolver needs are not in the checkout. + + Checked before the container starts: a `uv.lock` with no `pyproject.toml` beside it cannot be + resolved by anything, and finding that out after pulling an image is a minute wasted on a fact + that was on disk. + """ + names = {path.rsplit("/", 1)[-1] for path in present} + return [needed for needed in resolver.needs if needed not in names] + + +def upgrade( + *, + resolver: Resolver, + worktree: Path, + package: str, + version: str, + present: Sequence[str], + run: Callable[[Resolver, Path, str], tuple[int, str]], +) -> Result: + """Move the graph, then check that it actually moved. Item 175. + + `run` takes the resolver, the directory to mount and the command, and returns an exit code and + the tool's output. Injected for the reason every other boundary here is: this stays testable + without a daemon, and nothing in this module knows Docker exists. + """ + absent = missing_from(resolver, present) + if absent: + return Result( + Outcome.MISSING, + f"{resolver.lock} cannot be resolved without {', '.join(absent)}: the resolver reads " + f"the manifest to know which versions are allowed, and there is none here.", + ) + + code, output = run(resolver, worktree, command_for(resolver, package, version)) + if code != 0: + return Result(Outcome.FAILED, output) + + lock_path = worktree / resolver.lock + landed = version_in_lock(lock_path.read_text(encoding="utf-8"), resolver.lock, package) + if landed != version: + # **The tool's exit code is not the verdict.** Every one of these resolves happily within + # whatever range the manifest allows and reports success, so `^4.17.11` answers 0 having + # never gone near 5.x. Believing it would publish `clean` for an upgrade that never + # happened — the worst artefact this repository can emit. + return Result( + Outcome.CONSTRAINED, + f"the resolver exited 0 and {package} is still {landed or 'absent'}, not {version}: " + f"the range in the manifest does not allow it. Widen it there, then run this again.", + ) + return Result(Outcome.RESOLVED) + + +def in_a_container( + resolver: Resolver, context: Path, command: str, *, docker: str = "docker" +) -> tuple[int, str]: + """Run one resolver's command in an ephemeral container. The only Docker in this module. + + **A bind mount rather than a volume**, unlike an attempt's worktree (item 055), and the + difference is worth stating so it does not later look like an oversight. An attempt's phases run + **the project's own untrusted code**, where a bind mount would let it write to the host as the + uid that started it. This runs one package manager's own command with no project code executing, + and the entire purpose is to get a regenerated file back — which a bind mount does and a volume + does not. + + **`--user` is not a detail.** Without it `npm` leaves root-owned files in the operator's + checkout, and the next ordinary command they run fails with a permission error nothing connects + back to us. + + **And this one has a network, deliberately.** Resolving *is* asking the registry what exists. It + is the trade `image.build` already makes, and it changes nothing about the phase that later runs + the suite, which still reaches nothing. + """ + import os + import subprocess + + argv = [ + docker, "run", "--rm", + "--user", f"{os.getuid()}:{os.getgid()}", + # A resolver that hangs must not hold the run: these are network calls to a registry. + "--stop-timeout", "10", + "-v", f"{context}:/w", + "-w", "/w", + # `HOME` so the tools have somewhere to write their caches; `/w` is the only writable path + # and a cache in the checkout would be left behind for the operator to find. + "-e", "HOME=/tmp", + resolver.image, + "sh", "-lc", command, + ] + log.info("resolving", extra={"image": resolver.image, "command": command}) + try: + done = subprocess.run( # noqa: S603 + argv, capture_output=True, text=True, check=False, timeout=RESOLVE_TIMEOUT_SECONDS + ) + except subprocess.TimeoutExpired: + return 1, f"the resolver did not finish within {RESOLVE_TIMEOUT_SECONDS}s" + return done.returncode, (done.stdout + done.stderr).strip() + + +def touches(resolver: Resolver) -> tuple[str, ...]: + """Every file this resolver may rewrite, which is **all** of them and not just the lock. + + **Measured, not assumed** (item 175's gate, 2026-08-09): `npm install lodash@4.17.21 + --package-lock-only` rewrote `package.json` as well, moving its range from `^4.17.11` to + `^4.17.21`. That is correct behaviour for an upgrade and it is not what the caller expected. + + Why it matters more than it looks: item 174 found that a candidate leaving its own pin behind + made the *next* candidate's baseline describe the previous one, and fixed it by restoring the + file it had rewritten. With a resolver in the path there is more than one such file, and + restoring only the lock leaves the manifest moved — the same defect, one file over, and + invisible in exactly the same way. + """ + return resolver.needs diff --git a/hullwork/sandbox/net.py b/hullwork/sandbox/net.py index 7bebea3..b0f85b9 100644 --- a/hullwork/sandbox/net.py +++ b/hullwork/sandbox/net.py @@ -573,3 +573,48 @@ def _quietly(docker: str, argv: list[str]) -> None: run_docker([docker, *argv], timeout=DOCKER_TIMEOUT_SECONDS) except SandboxError as exc: # docker missing, or not answering log.warning("could not tear down", extra={"argv": argv, "error": str(exc)}) + + +def why_the_gateway_cannot_start(*, docker: str = "docker") -> str | None: + """The sentence that refuses an agent run before anything is paid for, or `None`. Item 191. + + **Shaped after `image.why_it_cannot_host_a_phase`, and here for the same reason it exists + there**: two doors needed a refusal and only the expensive one had it. Every agent path starts a + gateway — `work`, `try` and `deps --fix` — so a missing image is a fact about the instance and + not about the command that happened to notice. + + Measured on 2026-08-09, running `deps --fix` against a real model for the first time. It died + with `could not start the gateway / Unable to find image 'hullwork:dev' locally` **after** OSV, + four image builds and two suite runs. Item 048's finding and item 184's, a third time: the + refusal existed and happened in the most expensive place available. + + **A daemon that cannot be reached is a different answer**, and answering it here would be + guessing at somebody else's problem: `doctor` owns that question and says it properly. This one + answers only *is the image there*, and says nothing at all when the client is absent. + """ + import shutil + import subprocess + + if shutil.which(docker) is None: + # Not this function's question. `doctor` reports a missing or unreachable daemon, with the + # three things it can mean; a second opinion here would be a worse copy of it. + return None + + found = subprocess.run( # noqa: S603 + [docker, "image", "inspect", GATEWAY_IMAGE], + capture_output=True, + timeout=DOCKER_TIMEOUT_SECONDS, + check=False, + ) + if found.returncode == 0: + return None + return ( + f"the gateway image `{GATEWAY_IMAGE}` is not on this Docker daemon, and every agent run " + f"needs one.\n" + f" The gateway is where your model credential lives, so that the sandbox running the " + f"project's own code never holds it (DR-0004). It runs Hullwork's own image because it is " + f"Hullwork's own code.\n" + f" Build it from a checkout: docker build --tag {GATEWAY_IMAGE} .\n" + f" `docker compose build` does **not** make it: the compose file pins a published image " + f"and has no build stage." + ) diff --git a/hullwork/sandbox/run.py b/hullwork/sandbox/run.py index 707c4b8..4f6765d 100644 --- a/hullwork/sandbox/run.py +++ b/hullwork/sandbox/run.py @@ -130,8 +130,8 @@ #: Refused **on purpose**, which is a different thing from the accident above. A workflow file is #: code that runs on the forge's runner with the repository's secrets — outside the sandbox, with -#: privileges the agent does not have and must not be able to grant itself. Changing CI is already -#: an amber decision for a human in the worker contract; here it is simply not on the table. +#: privileges the agent does not have and must not be able to grant itself. The derived lane policy +#: already keeps a human on CI definitions (`territory.py`); here it is simply not on the table. FORBIDDEN_DIRS = frozenset({GIT_DIR, ".github", ".forgejo", ".gitea"}) #: Things a toolchain writes by itself, which are **not** the agent's work and must never be @@ -773,6 +773,40 @@ def is_test_infrastructure(relative: str) -> bool: return bool(_TEST_FILE.match(parts[-1])) +def created_test_config(worktree: Path, before: dict[str, bytes]) -> list[str]: + """Test configuration a phase **created** where the checkout had none. Item 179. + + **The hole in item 046's guard, found by writing this item's tests and confirmed against the + existing sequence.** `_restore_infrastructure` iterates the before-image, so it restores + configuration that was *edited* and never sees configuration that was *invented*: a fix phase + that writes a root `conftest.py` where a project had none switches the suite off, is not + restored, produces an empty `restored` — so no second gate runs — and publishes as `pr-open` + with the mechanism inside its own diff. Measured against `dispatch.dispatch` before this + existed, on a `conftest.py` carrying `collect_ignore_glob = ['*']` and no fix at all. + + Twenty-one tests in `test_dispatch` did not cover it, and they could not have: every one of + them expresses tampering as an *edit*, which is how item 046 was written and therefore how it + was tested. + + **Only configuration, never a new test.** `is_test_infrastructure` is deliberately not the + predicate here: it also matches new files under `tests/`, and a fix that adds a test is welcome + — that distinction is the whole reason the original guard iterated the snapshot. What is refused + is a *new* file whose only job is to decide whether the suite runs at all, and there is no + legitimate version of that arriving from a fix phase: the sandbox has no network, so a + dependency such a file could configure cannot be installed anyway. + """ + found: list[str] = [] + for path in sorted(worktree.rglob("*")): + if not path.is_file() or path.is_symlink(): + continue + relative = path.relative_to(worktree).as_posix() + if _forbidden(relative) or relative in before: + continue + if relative.split("/")[-1] in TEST_CONFIG_FILES: + found.append(relative) + return found + + @dataclass(frozen=True) class Changes: """What a phase did to the tree: files written, and files removed. diff --git a/hullwork/trial.py b/hullwork/trial.py index 915795c..2f84a88 100644 --- a/hullwork/trial.py +++ b/hullwork/trial.py @@ -255,6 +255,17 @@ def run( if refusal: raise work.WiringError(refusal) + # **Before the ephemeral database and before any image** (item 191). Every agent path starts a + # gateway, and `try` is the door the README sends a stranger to first — so it must not be the + # one that discovers a missing image at the most expensive moment. Same placement as the base + # image refusal above, and for the same reason. + from hullwork.sandbox.net import why_the_gateway_cannot_start + + if manifest.autofix.agent != "none": + missing = why_the_gateway_cannot_start() + if missing: + raise work.WiringError(missing) + session = ephemeral_session() project, item = stage(session, manifest, trace, repo=checkout.name) if item.state is ItemState.WAITING_APPROVAL and approve: diff --git a/hullwork/upgrades.py b/hullwork/upgrades.py new file mode 100644 index 0000000..f7f3356 --- /dev/null +++ b/hullwork/upgrades.py @@ -0,0 +1,220 @@ +"""Open the ones that pass. Item 178, DR-0018 step 3. + +This is where DR-0018's claim stops being a report and becomes the thing a reviewer receives: +*Renovate opens forty pull requests; we open the thirty-one that pass and tell you what to do with +the nine that do not.* + +**What it may open, and what it may never.** Only reports whose `needs_of` is `ready to take`. Not +the ones that break, not the blocked ones, not the ones whose baseline was red. A pull request from +Hullwork means *this was run and it passed*, and the moment it can mean anything else the claim is +worth nothing — including the ones it makes correctly. + +**One pull request per package, never a batch.** A grouped upgrade that breaks cannot be bisected by +the reviewer without undoing our work for us, and the verdict was computed per package anyway. + +**Nothing is remembered between runs, and nothing needs to be.** There is no database on this path: +the branch name carries the package and both versions, so a second pass over an unchanged repository +asks the forge for a branch that already exists and is told so. That is the same answer +`work.publish` has relied on since item 048, and it is better than a table — a table can disagree +with the forge, and this cannot. +""" + +from __future__ import annotations + +import logging +import re +from collections.abc import Mapping, Sequence + +from hullwork import bump, evidence +from hullwork.forge import BranchExistsError, ForgeError +from hullwork.osv import Advisory + +log = logging.getLogger(__name__) + +#: Where these branches live. Namespaced under `hullwork/` like every other branch this product +#: creates, and under `deps/` beneath that so an operator can tell an upgrade from an agent's fix +#: without opening either. +BRANCH_PREFIX = "hullwork/deps" + +#: Everything git refuses in a ref name, plus the characters that merely make one awkward to type. +#: `@scope/pkg` is an ordinary npm name and `/` inside it would invent a directory level, so it goes +#: too — the branch has to be readable and it has to be creatable, and neither is negotiable. +_NOT_IN_A_REF = re.compile(r"[^A-Za-z0-9._-]+") + + +def branch_for(package: str, was: str, to: str) -> str: + """The branch this upgrade goes on, derived from the upgrade and nothing else. + + **Derived rather than allocated**, and that is what makes "nothing is opened twice" true without + any state: the same upgrade names the same branch on every run, for ever, so the forge is the + thing that remembers. An id from a counter would need a database, and a random suffix would open + the same pull request every hour. + + Both versions are in it because the pair is the work: next month's upgrade of the same package + is different work and gets its own branch. + """ + slug = "-".join(_NOT_IN_A_REF.sub("-", part).strip("-.") for part in (package, was, to)) + # A ref may not end in `.lock` nor contain `..`, and a name built from three sanitised parts + # cannot produce either — but it can produce a run of hyphens, which is legal and ugly. + return f"{BRANCH_PREFIX}/{re.sub('-{2,}', '-', slug)}" + + +def eligible(reports: Sequence[bump.Report]) -> list[bump.Report]: + """The reports that may be opened, which is one bucket of four. + + `needs_of` rather than a scan for a clean answer, for the reason item 179 found the hard way: a + report can carry a `breaks` answer *and* a later clean one, and asking what the report **needs** + is the only reading that puts that in the right bucket. + + **A clean answer with no files is refused rather than opened.** Those bytes are the diff, and + without them there would be a body making a claim about an empty commit — what was tested and + what is published have to be the same tree, which is what item 045 is named after. It can happen + honestly: an `Answer` built by hand, an older recording replayed. + """ + return [ + report + for report in reports + if bump.needs_of(report) is bump.Needs.JUST_TAKE_IT + and report.settled is not None + and report.settled.files + ] + + +def title_for(answer: bump.Answer) -> str: + """What the pull request is called. The upgrade, and that a suite was run — nothing else.""" + return f"deps: {answer.package} {answer.was} → {answer.to} (your suite passes)" + + +def commit_message_for(answer: bump.Answer, advisories: Sequence[Advisory]) -> str: + """One commit, saying what moved and why, without repeating the body. + + No DCO sign-off trailer, ever. `CONTRIBUTING.md` makes the sign-off a human act performed at the + merge gate, and that a machine *can* emit the trailer is exactly why it must not. + """ + named = ", ".join(a.id for a in advisories) + because = f"\n\nPublished against {answer.package} {answer.was}: {named}." if named else "" + return ( + f"deps: {answer.package} {answer.was} → {answer.to}\n\n" + f"This project's own test suite was run against this change in a sandbox and passed, " + f"having also passed before it. That is what was measured — not that the upgrade is safe." + f"{because}\n\n" + f"Opened by Hullwork." + ) + + +def open_them( + code_forge: object, + *, + repo: str, + reports: Sequence[bump.Report], + advisories: Mapping[str, Sequence[Advisory]], + base_sha: str, + permitted: bool, + secrets: list[str] | None = None, +) -> list[str]: + """Open one draft pull request per verified-green package. Returns where each one went. + + **`permitted` is required and has no default, which is the whole of DR-0019's guard.** It is the + project's `autofix.open_upgrades`, and item 017's rule is why it is a parameter of this function + rather than a check at the call site: *a guardrail that depends on every caller remembering it + is not a guardrail.* This is the only function in the product that opens anything, so a caller + who forgets gets a `TypeError` rather than an unguarded pull request in somebody's repository. + + **A failure on one does not cost the others.** A queue of five with one bad name has to be four + pull requests rather than a traceback, which is the same rule `publish` follows one layer up: + publishing is the last thing that happens and the only thing here that can fail after the + verdict already exists. + + `code_forge` is a parameter and is never built here, for the reason every other boundary in this + repository is: the credential belongs to whoever owns the process, and this module stays + testable against a double. + """ + if not permitted: + # **A decision somebody made, not a failure.** Logged rather than raised: the verification + # above is the valuable half and it already ran, so a refusal here ends the opening and + # nothing else. The caller says so in words; this is the record. + log.info( + "not opening: the project has not permitted it", + extra={"repo": repo, "eligible": len(eligible(reports))}, + ) + return [] + + opened: list[str] = [] + for report in eligible(reports): + answer = report.settled + assert answer is not None # noqa: S101 - `eligible` returns none without one + where = _open_one( + code_forge, + repo=repo, + answer=answer, + advisories=tuple(advisories.get(answer.package, ())), + base_sha=base_sha, + secrets=secrets, + ) + if where is not None: + opened.append(where) + return opened + + +def _open_one( + code_forge: object, + *, + repo: str, + answer: bump.Answer, + advisories: Sequence[Advisory], + base_sha: str, + secrets: list[str] | None, +) -> str | None: + """Branch, commit, draft pull request. `None` when there is nothing new to open.""" + from hullwork.work import _commit + + branch = branch_for(answer.package, answer.was, answer.to) + try: + # **Rooted at the sha the gates ran against**, never at whatever the default branch points + # at now. The base can move freely while a verification runs, and the pull request still + # contains precisely the tree the suite passed on. + code_forge.create_branch(repo, branch, base_sha) # type: ignore[attr-defined] + except BranchExistsError: + # The record of what was opened, kept by the forge rather than by us. A second pass over an + # unchanged repository lands here for every package it already dealt with, which is the + # whole of "nothing is opened twice". + log.info("already opened", extra={"branch": branch, "package": answer.package}) + return None + except ForgeError as exc: + log.warning( + "could not branch for an upgrade", + extra={"branch": branch, "package": answer.package, "error": str(exc)}, + ) + return None + + try: + _commit( + code_forge, repo, branch, + commit_message_for(answer, advisories), + dict(answer.files), + base_sha, + ) + pull = code_forge.open_draft_pull_request( # type: ignore[attr-defined] + repo, + head=branch, + base=code_forge.default_branch(repo), # type: ignore[attr-defined] + title=title_for(answer), + body=evidence.dependency_pull_request_body(answer, advisories, secrets=secrets), + ) + except ForgeError as exc: + # The branch exists and the pull request does not. Said rather than swallowed, because the + # next run will find the branch taken and open nothing — so this line is the only place + # anybody learns why that package never appeared. + log.warning( + "branched but could not open the pull request", + extra={"branch": branch, "package": answer.package, "error": str(exc)}, + ) + return None + + if not pull.draft: + # Forgejo derives draft from a title prefix an instance can reconfigure and no API exposes + # (spec §5.1), so the response is read back rather than assumed. A merge-ready pull request + # from a bot is the one artefact this product must never leave behind. + log.error("the forge did not mark it a draft", extra={"pull": pull.ref}) + log.info("opened", extra={"package": answer.package, "to": answer.to, "pull": pull.ref}) + return str(pull.html_url) diff --git a/hullwork/upstream.py b/hullwork/upstream.py index 489be5e..6fb360e 100644 --- a/hullwork/upstream.py +++ b/hullwork/upstream.py @@ -113,7 +113,7 @@ | { f"cli:{name}" for name in ( - "approve", "config", "doctor", "gateway", "init", "lease", + "approve", "config", "deps", "doctor", "features", "gateway", "init", "lease", "page-token", "password", "projects", "propose", "prune", "republish", "requeue", "status", "sweep", "try", "work", ) diff --git a/hullwork/webhooks.py b/hullwork/webhooks.py index 481b62e..d3eaed8 100644 --- a/hullwork/webhooks.py +++ b/hullwork/webhooks.py @@ -158,20 +158,36 @@ async def receive( def _authenticate(provider: str, token: str, raw: bytes, expected_hash: str) -> bool: - """Per provider, because they do not offer the same thing. - - GlitchTip cannot sign its webhooks — no header, no secret, no setting — so the token in the path - is the credential. Sentry does sign, and its route is not enabled yet: verifying an HMAC means - holding its client secret in a **reversible** form, which is a different storage decision from - the one-way hash used here and deserves to be made deliberately rather than in passing. + """**The token in the path is the credential, for both providers.** Item 189, DR-0009. + + GlitchTip cannot sign its webhooks — no header, no secret, no setting — so the token in the URL + has been the credential since M1, verified against a one-way hash. + + **Sentry does sign, and this deliberately does not check it.** Verifying its HMAC means holding + its client secret in a **reversible** form, which is a different storage decision from every + other credential here and has not been made. This route answered `501` for that reason, and the + reason was written as *reversible secret or nothing* — which left out the option the operator + took on 2026-08-09: give Sentry **the same credential GlitchTip has**, checked the same way. + The route was refusing to offer a guarantee better than the one the only working provider gets, + which is coherent only if a signature is mandatory — and if it were, GlitchTip could not be + enabled either. + + **What that does not protect against, said plainly.** An HMAC would authenticate the *body*, so + somebody who obtained the URL — from a log, a proxy, a referrer — but not Sentry's client secret + could not post. Here, knowing the URL is enough, because the URL contains the credential. That + is the exposure GlitchTip users have had all along; this extends it to a second provider rather + than creating it, and verifying the HMAC stays open as the upgrade. + + `raw` is unread on purpose, and stays in the signature: it is what an HMAC would be computed + over, and removing the parameter would make adding that verification a change to every caller + rather than a change to this function. """ - if provider == "glitchtip": + del raw + if provider in ("glitchtip", "sentry"): + # **One expression for both**, so the two refusals cannot come to differ. A wrong token has + # to look identical on either route: a difference is a way to confirm which provider a slug + # is registered with, by probing, from outside (item 122's rule for the page, here). return verify_token(token, expected_hash) - if provider == "sentry": - raise HTTPException( - status.HTTP_501_NOT_IMPLEMENTED, - "the Sentry route is not enabled in this version; use GlitchTip", - ) return False diff --git a/hullwork/work.py b/hullwork/work.py index 1e5efc6..384ddb9 100644 --- a/hullwork/work.py +++ b/hullwork/work.py @@ -920,6 +920,11 @@ def run_one( image_tag: str | None = None, base_sha: str | None = None, production_ref: str | None = None, + #: Which sequence to run in the box. `None` is `dispatch.dispatch`, the six steps. Item 179 + #: passes `dispatch.refit`, which is three — and everything around it here is the same, which + #: is the point: the claim, the seal, the ceiling checks, publication and release are about an + #: attempt rather than about what the attempt was for. + sequence: object = None, ) -> Outcome: """Claim, dispatch, publish, record. The order is the whole design. @@ -951,8 +956,9 @@ def run_one( try: box = box_factory(manifest) # type: ignore[operator] - verdict = dispatch_module.dispatch( - session, item, manifest, engine, # type: ignore[arg-type] + run = sequence or dispatch_module.dispatch + verdict = run( # type: ignore[operator] + session, item, manifest, engine, box=box, attempt=attempt, ) except dispatch_module.Abandoned as stop: @@ -1263,7 +1269,10 @@ def publish( outcome = attempt.outcome try: if outcome not in (AttemptOutcome.PR_OPEN, AttemptOutcome.PR_OPEN_LINT_FAILED): - return _comment(forge, repo=repo, item=item, attempt=attempt, secrets=secrets) + return _comment( + forge, repo=repo, item=item, attempt=attempt, secrets=secrets, + claim=str(getattr(verdict, "claim", "")), + ) branch = evidence.branch_name(item, attempt) try: @@ -1300,6 +1309,10 @@ def publish( item, attempt, detail=str(getattr(verdict, "detail", "")), brief_text=brief_text, brief_evidence=brief_evidence, secrets=secrets, prices=prices, + # Item 179: a sequence whose claim is not the ordinary one carries its own, and both + # publishers read it from the same place so the page and the pull request cannot come + # to disagree about what was measured. + claim=str(getattr(verdict, "claim", "")), ) pull = code_forge.open_draft_pull_request( # type: ignore[attr-defined] repo, @@ -1361,6 +1374,7 @@ def publisher(item: Item, attempt: Attempt, verdict: object) -> str | None: (into / "artefact.md").write_text( evidence.pull_request_body( item, attempt, detail=str(getattr(verdict, "detail", "")), + claim=str(getattr(verdict, "claim", "")), ), encoding="utf-8", ) @@ -1523,6 +1537,11 @@ def _attempt( secrets: list[str], rehearse_into: Path | None = None, local_checkout: "Checkout | None" = None, + #: The dependency upgrade this attempt is about, when it is one (item 179). It changes three + #: things and nothing else: which sequence runs, what the brief says, and what the artefact + #: can claim about the evidence the agent had. One parameter rather than three, because those + #: three have to agree and a caller that sets two of them has built a lie. + upgrade: object = None, ) -> Outcome: """Build one attempt's world, run the sequence in it, and take the world down again. @@ -1531,6 +1550,7 @@ def _attempt( variable are how the second item in a run ends up dispatched into the first one's container. """ from contextlib import ExitStack + from functools import partial from hullwork import dispatch as dispatch_module from hullwork import engine as engine_module @@ -1643,11 +1663,40 @@ def _attempt( contract_dir = Path(tempfile.mkdtemp(prefix="hullwork-contract-")) stack.callback(shutil.rmtree, contract_dir, ignore_errors=True) - dispatch_module.build_brief_file(session, item, contract_dir) - brief_text = build_brief(session, item) - # Read from the same event the brief was built from, before the attempt runs — enrichment - # can happen while it does, and the artefact has to say what the agent *had* (item 100). - brief_evidence = brief_evidence_level(session, item) + sequence: object = None + if upgrade is None: + dispatch_module.build_brief_file(session, item, contract_dir) + brief_text = build_brief(session, item) + # Read from the same event the brief was built from, before the attempt runs — + # enrichment can happen while it does, and the artefact has to say what the agent + # *had* (item 100). + brief_evidence = brief_evidence_level(session, item) + else: + # Item 179. Everything below this block is untouched: same image, same gates, same + # seal, same publisher. What a refit replaces is what the agent is told and which + # sequence reads its work — the two halves that are about a bug rather than about an + # attempt. + from hullwork import refit as refit_module + + brief_text = refit_module.brief(upgrade) # type: ignore[arg-type] + dispatch_module.write_brief(brief_text, contract_dir) + # **Not `brief.evidence_level`**, which reads a `FetchedEvent` this item does not have + # and would answer "the issue title only — the tracker was never asked". That sentence + # exists to warn a reviewer that an attempt ran on almost nothing; here it would + # understate the best evidence this product produces (item 100's rule, held to). + brief_evidence = ( + f"the upgrade, and the {len(brief_text.splitlines())}-line brief naming the tests " + f"your own suite failed on with it applied" + ) + sequence = partial( + dispatch_module.refit, + package=upgrade.package, # type: ignore[attr-defined] + to=upgrade.to, # type: ignore[attr-defined] + guarded=upgrade.guarded, # type: ignore[attr-defined] + version_now=lambda tree: refit_module.version_now( + upgrade, tree # type: ignore[arg-type] + ), + ) # The gateway runs **in** the attempt's own network, not on this host (item 054). A # container on an `--internal` network cannot reach a listener on the host — measured on a @@ -1750,6 +1799,7 @@ def publisher(published: Item, attempt: Attempt, verdict: object) -> str | None: base_sha=checked_out.sha, production_ref=_production_ref(session, item), rehearsal=rehearsal, + sequence=sequence, ) # The seal that was stored, not a third read of the journal. Two reads of a growing file are # two chances to print something the database does not say. @@ -1844,7 +1894,8 @@ def _production_ref(session: Session, item: Item) -> str | None: def _comment( - forge: object, *, repo: str, item: Item, attempt: Attempt, secrets: list[str] | None + forge: object, *, repo: str, item: Item, attempt: Attempt, secrets: list[str] | None, + claim: str = "", ) -> str | None: """Say on the issue what happened, with the ingest credential rather than the code one. @@ -1858,7 +1909,9 @@ def _comment( log.info("no issue to report to", extra={"item": item.id}) return None number = int(item.forge_issue_ref.lstrip("#")) - body = evidence.issue_comment(item, attempt, detail=attempt.error or "", secrets=secrets) + body = evidence.issue_comment( + item, attempt, detail=attempt.error or "", secrets=secrets, claim=claim + ) forge.comment(repo, number, body) # type: ignore[attr-defined] log.info("commented on the issue", extra={"item": item.id, "issue": number}) return item.forge_issue_ref diff --git a/images/banner.svg b/images/banner.svg index d1f3650..a00ff2e 100644 --- a/images/banner.svg +++ b/images/banner.svg @@ -1,11 +1,11 @@ - Hullwork — from production errors to reviewable draft pull requests + Hullwork — verifies which of your tools' claims are actually true The Hullwork wordmark beside its mark: two filled quadrants, diagonally opposed. Under - it, the promise — from production errors to reviewable draft pull requests, self-hosted, with your - forge, your error tracker, your model endpoint and a human gate on every merge. Along the bottom, - the three counts from one real attempt: 904 passed untouched, 2 failed with the new test and no - fix, 906 passed with the fix applied. + it, the claim — it verifies which of your tools' claims are actually true, before a person is + asked — and then: self-hosted, with your forge, your error tracker, your model endpoint and a + human gate on every merge. Along the bottom, the three counts from one real attempt: 904 passed + untouched, 2 failed with the new test and no fix, 906 passed with the fix applied. + Verifies which of your tools’ claims are actually true, before a person is asked. Self-hosted · your forge · your error tracker · your model endpoint · a human gate on every merge diff --git a/images/social-preview.png b/images/social-preview.png index f4c5bd7..5502835 100644 Binary files a/images/social-preview.png and b/images/social-preview.png differ diff --git a/images/social-preview.svg b/images/social-preview.svg new file mode 100644 index 0000000..6102365 --- /dev/null +++ b/images/social-preview.svg @@ -0,0 +1,77 @@ + + Hullwork — verifies which of your tools' claims are actually true + The Hullwork wordmark above the claim: it verifies which of your tools' claims are + actually true, before a person is asked — errors, dependencies, incidents. Below, the three + counts from one real attempt: the suite untouched, 904 passed; the new test with no fix, 2 + failed, exit 1, the required outcome; the fix applied, 906 passed. Along the bottom: self-hosted, + your forge, your error tracker, your model endpoint, a human gate on every merge — and the + project's state, pre-alpha, under FSL-1.1-ALv2. + + + + + + + + + + + + + hullwork + + Verifies which of your tools’ + claims are actually true. + Before a person is asked — errors, dependencies, incidents. + + + + + + the suite, untouched + 904 passed + green, so an attempt may start + + + the new test, no fix + 2 failed + exit 1 — the required outcome + + + the fix applied + 906 passed + and the 904 still pass + + + + Self-hosted. Your forge, your error tracker, your model endpoint. + A human gate on every merge. + pre-alpha · FSL-1.1-ALv2 + diff --git a/images/the-pipeline.svg b/images/the-pipeline.svg index 1209ad6..dbf8a18 100644 --- a/images/the-pipeline.svg +++ b/images/the-pipeline.svg @@ -1,7 +1,8 @@ - What Hullwork does, from a production error to a deploy - Two things arrive: a production error, from GlitchTip or anything Sentry-compatible, by + The path a production error takes, from the webhook to a deploy + One of Hullwork's three signals, end to end — the only one in a release. Two things + arrive: a production error, from GlitchTip or anything Sentry-compatible, by webhook; and a human report by email or chat, through a normaliser. Hullwork triages, deduplicates and assigns a risk lane. That becomes a work item with a lane. A green item goes to a coding agent in a sandbox, using your model key. The agent's work becomes a draft pull request. A person reviews diff --git a/pyproject.toml b/pyproject.toml index bb1a412..7080719 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,13 @@ build-backend = "hatchling.build" [project] name = "hullwork" -description = "From production errors to reviewable pull requests, on your own infrastructure." +# **Rewritten 2026-08-09 (item 181).** What it said before named one signal's two endpoints, offered +# in the first line PyPI shows — where a reader decides what this is, and answering a narrower +# question than the product does. `docs/what-hullwork-is.md` is the canonical statement and this is +# its one-line form. The previous wording is in item 181 and in the commit that changed it; it is +# deliberately not quoted here, because the guard that keeps it out of published documents is a +# plain text search and a guard that has to be clever is a guard nobody trusts. +description = "Verifies which of the things your tools claim are actually true, before a person is asked. Self-hosted." readme = "README.md" # **Narrowed 2026-08-04, because the wider range was a claim this project did not meet.** # 3.13 and 3.14 are red: `filterwarnings = ["error"]` turns unclosed SQLite connections into diff --git a/tests/test_bump.py b/tests/test_bump.py new file mode 100644 index 0000000..9fe04f0 --- /dev/null +++ b/tests/test_bump.py @@ -0,0 +1,563 @@ +"""Does the upgrade survive the project's own suite. Item 173, DR-0016. + +**No test here needs Docker**, and that is the same trade `dispatch` makes: the function is handed +a box and a directory, so a double serves it. The Docker path is measured once by hand and written +into the item, because a mocked container proves the wiring and not the claim. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from hullwork import bump +from hullwork.sandbox.run import RunResult + + +class FakeBox: + """A sandbox that answers with whatever the test queued, in order.""" + + def __init__(self, worktree: Path, *results: RunResult) -> None: + self.worktree = worktree + self._results = list(results) + self.commands: list[str] = [] + + def run(self, command: str, timeout: int = 0) -> RunResult: + del timeout + self.commands.append(command) + return self._results.pop(0) + + +def _ok(output: str = "12 passed") -> RunResult: + return RunResult(command="pytest", exit_code=0, output=output, duration_ms=10) + + +def _red(output: str = "FAILED tests/test_a.py::test_one\n1 failed") -> RunResult: + return RunResult(command="pytest", exit_code=1, output=output, duration_ms=10) + + +def _checkout(tmp_path: Path, text: str = "jinja2==2.4.1\n") -> Path: + (tmp_path / "requirements.txt").write_text(text, encoding="utf-8") + return tmp_path + + +# --- the edit ----------------------------------------------------------------------------- + + +def test_a_pin_is_rewritten_and_everything_else_on_the_line_survives() -> None: + """Extras, environment markers and trailing comments all outlive the upgrade. + + Only the version group is replaced, which is why this works at all — a line rebuilt from its + parsed parts would quietly drop the marker and change what gets installed on other platforms. + """ + line = 'httpx[http2]==0.27.0 ; python_version >= "3.8" # pinned by hand\n' + out = bump.rewrite_pin(line, "httpx", "0.28.1") + + assert out == 'httpx[http2]==0.28.1 ; python_version >= "3.8" # pinned by hand\n' + + +def test_only_the_named_package_moves() -> None: + text = "jinja2==2.4.1\nrequests==2.31.0\njinja2-time==0.2.0\n" + out = bump.rewrite_pin(text, "jinja2", "2.10.1") + + assert "jinja2==2.10.1" in out + assert "requests==2.31.0" in out + assert "jinja2-time==0.2.0" in out, "a longer name that starts the same is a different package" + + +def test_a_hashed_pin_is_refused_rather_than_broken() -> None: + """The hash describes the artefact pinned. Change the version and it will not install.""" + text = "jinja2==2.4.1 --hash=sha256:abc\n" + with pytest.raises(bump.CannotRewriteError, match="hash"): + bump.rewrite_pin(text, "jinja2", "2.10.1") + + +def test_a_package_that_is_not_pinned_here_is_not_silently_ignored() -> None: + with pytest.raises(bump.CannotRewriteError, match="no `django==…` line"): + bump.rewrite_pin("jinja2==2.4.1\n", "django", "5.0") + + +@pytest.mark.parametrize("name", ["Cargo.lock", "go.sum", "Gemfile.lock", "composer.lock"]) +def test_a_lock_with_no_resolver_is_still_refused_by_name(name: str) -> None: + """**The allow-list is the point.** + + Item 175 gave `package-lock.json`, `uv.lock` and `poetry.lock` a resolver, so those are no + longer refused. Everything else still is — and stating the rule as *only lists are editable* + rather than as three named refusals is what makes that true for a lock file nobody has taught + this about yet. The unsafe answer must never be the default. + """ + with pytest.raises(bump.CannotRewriteError) as caught: + bump.can_rewrite(f"path/to/{name}") + + assert name in str(caught.value) + assert "cannot install" in str(caught.value) + + +@pytest.mark.parametrize("name", ["package-lock.json", "uv.lock", "poetry.lock"]) +def test_a_lock_with_a_resolver_is_no_longer_refused(name: str) -> None: + """Item 175 lifted the refusal for these: their own tool can move the graph.""" + bump.can_rewrite(f"path/to/{name}") + + +def test_requirements_is_not_refused() -> None: + bump.can_rewrite("requirements.txt") + bump.can_rewrite("deep/nested/requirements.txt") + + +@pytest.mark.parametrize( + "name", + [ + "requirements/base.txt", + "requirements/prod.txt", + "requirements-dev.txt", + "dev-requirements.txt", + "backend/requirements/test.txt", + ], +) +def test_every_layout_the_reader_accepts_is_editable_by_hand(name: str) -> None: + """The consequence item 180 created, caught before it shipped. + + Widening the reader without widening this made `can_rewrite` refuse every one of these — and + refuse them with a sentence that is **false**: *"it is a resolved graph rather than a list of + versions"*. They are lists of versions; that is the whole reason the reader can read them. The + cost would have been the entire `--verify` / `--open` / `--fix` chain going quiet on any project + using a layout other than a root `requirements.txt`, with a wrong reason printed for each. + + One predicate for both, so a layout that becomes readable becomes editable in the same edit. + """ + bump.can_rewrite(name) + + +def test_a_hash_pinned_line_is_still_refused_by_name_in_any_layout() -> None: + """The refusal that existed and had never had the chance to fire. Item 180's last criterion. + + `requirements/build.txt` in this repository pins by hash, and until the reader read that file + nothing could reach this. Now that something can, the refusal has to be the one about hashes — + specific and actionable — rather than the generic one about resolved graphs. + """ + with pytest.raises(bump.CannotRewriteError, match="hash"): + bump.rewrite_pin("build==1.2.1 --hash=sha256:abc\n", "build", "1.3.0") + + +# --- the three phases --------------------------------------------------------------------- + + +def test_a_suite_that_is_already_red_stops_before_anything_is_rewritten(tmp_path: Path) -> None: + """**Before the edit and before a second build is paid for.** + + A suite already failing cannot support "passed before and passes after", and blaming the + upgrade for it is the error `dispatch` made until item 043. + """ + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _red()) + rebuilt: list[str] = [] + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=rebuilt.append, + ) + + assert answer.verdict is bump.Verdict.ALREADY_RED + assert rebuilt == [], "nothing may be rebuilt once the baseline is red" + assert (checkout / "requirements.txt").read_text() == "jinja2==2.4.1\n", "not rewritten" + assert "FAILED tests/test_a.py::test_one" in answer.detail + + +def test_a_clean_upgrade_says_exactly_what_it_measured(tmp_path: Path) -> None: + """**The wording is asserted so it cannot drift.** + + DR-0016 fixes it: *the suite passed before this change and passes after it*. Never "safe", and + never "fixes the vulnerability" — a suite that never exercised the library says so by staying + green, and widening the claim here is the defect item 171 removed. + """ + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok(), _ok()) + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda text: None, + ) + + assert answer.verdict is bump.Verdict.CLEAN + assert "passed before this change and passes after it" in answer.says + assert "not that the upgrade is safe" in answer.says + # **The tree is left as it was found, even on a clean verdict.** This measures; it does not + # apply. Leaving the rewrite in place would make one candidate's result describe the next + # one's baseline — which is what a real run did, reporting `already-red` about a suite that + # had been green a minute earlier. + assert (checkout / "requirements.txt").read_text() == "jinja2==2.4.1\n" + + +def test_a_clean_verdict_carries_the_file_the_passing_run_actually_saw(tmp_path: Path) -> None: + """The seam between measuring an upgrade and opening one. Item 178. + + **These bytes exist for about two lines.** The tree is restored on the way out — the test above + asserts that, and it has to stay true — so anything that wants to publish what passed has to be + handed it before the restore. Working the diff out afterwards would mean running the resolver a + second time, and a lock regenerated twice can differ: a version published in between, a + different ordering, a registry that answered differently. Publishing files that are not the ones + the suite passed against is the defect item 045 is named after. + + Verified by reintroducing the defect, and it is the worst kind: with these dropped, `upgrades` + finds nothing eligible and `deps --open` opens nothing at all — silently, with every other test + in both files still green. + """ + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok(), _ok("248 passed in 30.44s")) + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda text: None, + ) + + assert answer.files == {"requirements.txt": b"jinja2==2.10.1\n"}, "the upgraded file, not the " + # …and the tree it was read from is back to what it was, which is what makes the two facts + # different rather than redundant. + assert (checkout / "requirements.txt").read_text() == "jinja2==2.4.1\n" + assert answer.runs is not None + assert (answer.runs.before_exit, answer.runs.after_exit) == (0, 0) + assert answer.runs.after_summary == "248 passed in 30.44s" + assert answer.runs.command == "pytest" + + +def test_a_verdict_that_is_not_clean_carries_no_files_to_publish(tmp_path: Path) -> None: + """Nothing that broke has a tree anybody should be offered. Item 178's rule, at the source.""" + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok(), _red()) + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda text: None, + ) + + assert answer.verdict is bump.Verdict.BREAKS + assert answer.files == {} + # The runs are still carried: a reader of the report wants the exit codes either way. + assert answer.runs is not None and answer.runs.after_exit == 1 + + +def test_a_breaking_upgrade_names_the_tests_it_broke(tmp_path: Path) -> None: + """The finding, and the reason anybody would install this.""" + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok(), _red("FAILED tests/test_render.py::test_escape\n1 failed")) + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda text: None, + ) + + assert answer.verdict is bump.Verdict.BREAKS + assert "tests/test_render.py::test_escape" in answer.detail + assert "breaks your suite" in answer.says + + +def test_a_build_that_fails_is_not_the_same_as_a_suite_that_fails(tmp_path: Path) -> None: + """And the file is put back, so a refused upgrade leaves the checkout as it was found.""" + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok()) + + answer = bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", + rebuild=lambda text: "no matching distribution found for jinja2==2.10.1", + ) + + assert answer.verdict is bump.Verdict.WILL_NOT_INSTALL + assert "no matching distribution" in answer.detail + assert (checkout / "requirements.txt").read_text() == "jinja2==2.4.1\n", "put back" + + +def test_the_rebuild_is_handed_the_rewritten_text(tmp_path: Path) -> None: + """The caller builds the image, and it must build the file that will be tested. + + Reintroducing this defect — handing `rebuild` the original text — produced a green verdict for + an upgrade that had never been installed, which is the worst answer this module could give. + """ + checkout = _checkout(tmp_path) + box = FakeBox(checkout, _ok(), _ok()) + seen: list[str] = [] + + def rebuild(text: str) -> None: + seen.append(text) + + bump.attempt( + lambda: box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=rebuild, + ) + + assert seen == ["jinja2==2.10.1\n"], "the build must see the upgrade, not the old pin" + + +def test_a_lock_file_is_refused_before_the_baseline_is_paid_for(tmp_path: Path) -> None: + """No build, no suite run: a project whose only pins are locked is told at once.""" + box = FakeBox(tmp_path, _ok()) + + with pytest.raises(bump.CannotRewriteError): + bump.attempt( + lambda: box, tests="pytest", source="Cargo.lock", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda text: None, + ) + + assert box.commands == [], "nothing may run before the file is known to be rewritable" + + +def test_two_spellings_of_one_package_are_one_package() -> None: + """PEP 503: `Jinja2`, `jinja_2` and `jinja.2` all name the same distribution. + + OSV answers with the canonical name, and a requirements file carries whichever spelling its + author typed. Comparing raw strings refuses to rewrite a pin that is plainly there — which + reads as "no such dependency" about a line the reader can see. + """ + assert bump.rewrite_pin("Jinja2==2.4.1\n", "jinja2", "2.10.1") == "Jinja2==2.10.1\n" + assert bump.rewrite_pin("ruamel_yaml==0.1\n", "ruamel-yaml", "0.2") == "ruamel_yaml==0.2\n" + # And the spelling the project chose survives the rewrite: this edits a file a person owns. + assert "Jinja2" in bump.rewrite_pin("Jinja2==2.4.1\n", "jinja2", "2.10.1") + + +# --- trying the candidates ------------------------------------------------------------------ + + +class _Advisory: + def __init__(self, *fixed: str) -> None: + self.fixed = fixed + + +def test_the_candidates_are_every_published_fix_without_repeats() -> None: + """Item 172's deferred question, answered by execution rather than by comparison. + + That item prints every fixed version and chooses none, because choosing means comparing + versions under two ecosystems' rules. Here each is tried and the suite decides. + """ + assert bump.candidates([_Advisory("2.11.3", "3.1.3"), _Advisory("2.11.3")]) == [ + "2.11.3", + "3.1.3", + ] + + +def test_it_stops_at_the_first_candidate_that_leaves_the_suite_green(tmp_path: Path) -> None: + """**And does not keep going.** + + The remaining candidates are higher versions of the same fix, and upgrading further than the + advisory asks for is taking a larger change than the problem requires. + """ + tried: list[str] = [] + + def make_box(version: str) -> bump.Box: + # Called twice per candidate — once for the baseline, once for the rebuilt image — so each + # box answers exactly one run, which is what a real box does. + tried.append(version) + _checkout(tmp_path) + first = tried.count(version) == 1 + if first: + return FakeBox(tmp_path, _ok()) + return FakeBox(tmp_path, _red("FAILED tests/test_x.py::test_y") if version == "2.11.3" + else _ok()) + + report = bump.verify( + tests="pytest", source="requirements.txt", package="jinja2", was="2.4.1", + versions=["2.11.3", "3.1.3", "9.9.9"], make_box=make_box, rebuild=lambda text: None, + ) + + assert tried == ["2.11.3", "2.11.3", "3.1.3", "3.1.3"], "9.9.9 must never be attempted" + assert report.settled is not None + assert report.settled.to == "3.1.3" + assert [a.verdict for a in report.answers] == [bump.Verdict.BREAKS, bump.Verdict.CLEAN] + + +def test_a_red_baseline_does_not_burn_through_every_candidate(tmp_path: Path) -> None: + """The suite is broken, not the candidate. Asking it again gets the same answer. + + Reintroducing this ran a full build per published version against a suite that could never + answer — on lodash's seven advisories that is seven builds to learn nothing. + """ + tried: list[str] = [] + + def make_box(version: str) -> bump.Box: + tried.append(version) + _checkout(tmp_path) + return FakeBox(tmp_path, _red()) + + report = bump.verify( + tests="pytest", source="requirements.txt", package="jinja2", was="2.4.1", + versions=["2.11.3", "3.1.3"], make_box=make_box, rebuild=lambda text: None, + ) + + assert tried == ["2.11.3"], "a red baseline is about the project, not the candidate" + assert report.settled is None + assert report.answers[0].verdict is bump.Verdict.ALREADY_RED + + +def test_nothing_clean_means_nothing_settled(tmp_path: Path) -> None: + calls: list[str] = [] + + def make_box(version: str) -> bump.Box: + _checkout(tmp_path) + calls.append(version) + return FakeBox(tmp_path, _ok() if calls.count(version) == 1 else _red()) + + report = bump.verify( + tests="pytest", source="requirements.txt", package="jinja2", was="2.4.1", + versions=["2.11.3", "3.1.3"], make_box=make_box, rebuild=lambda text: None, + ) + + assert report.settled is None + assert len(report.answers) == 2, "every candidate is tried when none is clean" + + +def test_the_second_run_happens_in_a_box_the_rebuild_produced(tmp_path: Path) -> None: + """**The defect a real Docker run found and nineteen unit tests did not.** + + `attempt` used to take a box rather than a factory, so the green gate ran in the container + built *before* the upgrade: the suite of an upgraded project measured against the environment + it replaced. It reported `clean` for `jinja2 3.0.3 → 3.1.6` against a suite importing + `jinja2.Markup`, which 3.1 removed — a green verdict for a version that was never installed. + + A double has no image, which is exactly why nothing in this file could have noticed. What it + *can* assert is the shape that made it possible: the factory is called again after the + rebuild, and the second run happens somewhere else. + """ + _checkout(tmp_path) + handed: list[FakeBox] = [] + order: list[str] = [] + + def make_box() -> bump.Box: + box = FakeBox(tmp_path, _ok()) + handed.append(box) + return box + + def rebuild(text: str) -> None: + order.append("rebuild") + + bump.attempt( + make_box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=rebuild, + ) + + assert len(handed) == 2, "the box the baseline ran in cannot be the box the verdict runs in" + assert handed[0] is not handed[1] + assert [len(b.commands) for b in handed] == [1, 1], "one run each, in its own environment" + + +def test_no_second_box_is_built_when_the_rebuild_failed(tmp_path: Path) -> None: + """There is nothing to run in: the image the verdict needs does not exist.""" + _checkout(tmp_path) + handed: list[FakeBox] = [] + + def make_box() -> bump.Box: + box = FakeBox(tmp_path, _ok()) + handed.append(box) + return box + + answer = bump.attempt( + make_box, tests="pytest", source="requirements.txt", package="jinja2", + was="2.4.1", to="2.10.1", rebuild=lambda _text: "no matching distribution", + ) + + assert answer.verdict is bump.Verdict.WILL_NOT_INSTALL + assert len(handed) == 1 + + +def test_a_broken_candidate_does_not_describe_the_next_one(tmp_path: Path) -> None: + """**The second defect the real Docker run found.** + + A candidate that breaks the suite used to leave its own pin in the tree, so the next + candidate's baseline ran against it and reported `already-red` — about a suite that had been + green a minute before. One candidate must not be able to describe the next. + """ + _checkout(tmp_path) + seen: list[str] = [] + + def make_box(version: str) -> bump.Box: + seen.append((tmp_path / "requirements.txt").read_text()) + return FakeBox(tmp_path, _ok() if len(seen) % 2 else _red()) + + bump.verify( + tests="pytest", source="requirements.txt", package="jinja2", was="2.4.1", + versions=["2.11.3", "3.1.3"], make_box=make_box, rebuild=lambda _t: None, + ) + + baselines = [text for i, text in enumerate(seen) if i % 2 == 0] + assert all(t == "jinja2==2.4.1\n" for t in baselines), ( + f"every candidate must start from the original pin, saw {baselines}" + ) + + +# --- the ranked report. DR-0018 step 2. ------------------------------------------------------- + + +def _report(package: str, *answers: bump.Answer) -> bump.Report: + return bump.Report(package, "1.0.0", answers) + + +def _answer(verdict: bump.Verdict, detail: str = "") -> bump.Answer: + return bump.Answer(verdict, "p", "1.0.0", "2.0.0", detail=detail) + + +def test_what_a_verdict_asks_of_a_person() -> None: + """Ordered by what was established, not by a severity nobody here has read.""" + assert bump.needs_of(_report("a", _answer(bump.Verdict.ALREADY_RED))) is ( + bump.Needs.FIX_YOUR_SUITE + ) + assert bump.needs_of(_report("b", _answer(bump.Verdict.CLEAN))) is bump.Needs.JUST_TAKE_IT + assert bump.needs_of(_report("c", _answer(bump.Verdict.BREAKS, "FAILED x"))) is ( + bump.Needs.NEEDS_WORK + ) + assert bump.needs_of(_report("d", _answer(bump.Verdict.CANNOT_MOVE))) is bump.Needs.BLOCKED + + +def test_a_candidate_that_broke_does_not_outrank_one_that_later_passed() -> None: + """Trying three versions and settling on the third is `ready to take`, not `needs work`. + + Reintroducing this put every package that ever saw a red candidate into the section a person + has to work — which on lodash's seven advisories is most of them, and all of them wrongly. + """ + settled = _report( + "lodash", _answer(bump.Verdict.BREAKS, "FAILED a"), _answer(bump.Verdict.CLEAN) + ) + assert bump.needs_of(settled) is bump.Needs.JUST_TAKE_IT + + +def test_the_queue_is_ordered_worst_first_and_easiest_within_that() -> None: + """**The answer to the complaint Renovate cannot answer.** + + *"Here is every update, you decide"* is noise because nothing in it is ranked, and ranking + needs knowing what each one does. Within `needs work`, fewest broken tests first: the two-test + upgrade is the one somebody closes this afternoon, and burying it under a twelve-test one + hides the achievable behind the daunting. + """ + reports = [ + _report("easy", _answer(bump.Verdict.BREAKS, "FAILED a\nFAILED b")), + _report("clean", _answer(bump.Verdict.CLEAN)), + _report("stuck", _answer(bump.Verdict.CANNOT_MOVE, "constrained")), + _report("hard", _answer(bump.Verdict.BREAKS, "\n".join(f"FAILED {i}" for i in range(12)))), + _report("red", _answer(bump.Verdict.ALREADY_RED)), + ] + + assert [r.package for r in bump.ranked(reports)] == [ + "red", # nothing else can be decided until this is fixed + "easy", # needs a person, and is the cheapest of those + "hard", + "stuck", # nothing to try + "clean", # nothing to decide + ] + + +def test_the_summary_is_the_sentence_that_replaces_the_queue() -> None: + reports = [ + _report("a", _answer(bump.Verdict.CLEAN)), + _report("b", _answer(bump.Verdict.CLEAN)), + _report("c", _answer(bump.Verdict.BREAKS, "FAILED x")), + _report("d", _answer(bump.Verdict.WILL_NOT_INSTALL)), + ] + + counted = bump.summary(reports) + + assert counted[bump.Needs.JUST_TAKE_IT] == 2 + assert counted[bump.Needs.NEEDS_WORK] == 1 + assert counted[bump.Needs.BLOCKED] == 1 + assert counted[bump.Needs.FIX_YOUR_SUITE] == 0 + # Every bucket is present even at zero: a reader must be able to tell "none" from "not counted". + assert set(counted) == set(bump.Needs) diff --git a/tests/test_dependencies.py b/tests/test_dependencies.py new file mode 100644 index 0000000..cc91deb --- /dev/null +++ b/tests/test_dependencies.py @@ -0,0 +1,627 @@ +"""What the project pinned, and what OSV says about it. Item 172, DR-0016. + +**The suite never opens a socket.** httpx2's `MockTransport` serves the recorded shapes, exactly as +`test_forge_code.py` does — an OSV client tested against the real service would be a test whose +result depends on somebody else's database changing. + +**Lockfiles rather than declarations**, because a declaration is a range and a range is not a fact. +Two of the four readers below are checked against files that actually exist in this repository +rather than only against fixtures written for the test, which is the same rule `test_propose.py` +runs on: a reader tested only against its own fixtures is a reader tested against itself. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +import io +from collections.abc import Callable +from pathlib import Path + +import httpx2 + +from hullwork import dependencies, osv +from hullwork.manifest import Manifest, parse_manifest + +ROOT = Path(__file__).resolve().parent.parent + +PACKAGE_LOCK = """ +{ + "name": "thing", + "lockfileVersion": 3, + "packages": { + "": {"name": "thing", "version": "1.0.0"}, + "node_modules/lodash": {"version": "4.17.20", "resolved": "https://registry.npmjs.org/x"}, + "node_modules/@scope/pkg": {"version": "2.1.0"}, + "node_modules/no-version": {"resolved": "https://registry.npmjs.org/y"} + } +} +""" + +POETRY_LOCK = """ +[[package]] +name = "jinja2" +version = "2.4.1" +description = "A templating engine" + +[[package]] +name = "requests" +version = "2.31.0" +""" + +REQUIREMENTS = """ +# a comment +jinja2==2.4.1 +requests>=2.0 ; not a pin +django ~= 4.2 ; not a pin either +lodash==4.17.20 + +-e . +""" + + +def _read(files: dict[str, str]) -> Callable[[str], str | None]: + def read(path: str) -> str | None: + return files.get(path) + + return read + + +# --- the readers -------------------------------------------------------------------------- + + +def test_package_lock_gives_up_its_packages_without_the_root_entry() -> None: + """The `""` key is the project itself, not a dependency of it. + + Including it would have Hullwork ask OSV about the repository being scanned, which is both + wrong and slightly embarrassing. + """ + found = dependencies.read_lockfiles( + ["package-lock.json"], _read({"package-lock.json": PACKAGE_LOCK}) + ) + + assert [(d.name, d.version) for d in found] == [ + ("lodash", "4.17.20"), + ("@scope/pkg", "2.1.0"), + ] + assert {d.ecosystem for d in found} == {"npm"} + assert all(d.source == "package-lock.json" for d in found) + + +def test_a_package_lock_entry_with_no_version_is_not_a_pin() -> None: + """It appears in the file and pins nothing, so it is not something to ask about.""" + found = dependencies.read_lockfiles( + ["package-lock.json"], _read({"package-lock.json": PACKAGE_LOCK}) + ) + assert "no-version" not in {d.name for d in found} + + +def test_poetry_and_uv_are_the_same_two_keys() -> None: + """Both are TOML with `[[package]]`, `name` and `version`. One reader, not two.""" + found = dependencies.read_lockfiles(["poetry.lock"], _read({"poetry.lock": POETRY_LOCK})) + + assert [(d.name, d.version) for d in found] == [("jinja2", "2.4.1"), ("requests", "2.31.0")] + assert {d.ecosystem for d in found} == {"PyPI"} + + +def test_this_repositorys_own_uv_lock_reads(tmp_path: Path) -> None: + """**The measurement that stops this being tested against itself.** + + Not a fixture: `uv.lock` as this repository actually pins it. If the format moves under us, + this is what says so. + """ + del tmp_path + text = (ROOT / "uv.lock").read_text(encoding="utf-8") + found = dependencies.read_lockfiles(["uv.lock"], _read({"uv.lock": text})) + + names = {d.name for d in found} + assert {"alembic", "httpx2", "pydantic"} <= names + assert all(d.version for d in found), "a package with no version is not a pin" + assert all(d.ecosystem == "PyPI" for d in found) + + +def test_requirements_counts_what_it_could_not_pin() -> None: + """**The honest half of the weakest reader.** + + A line that is not `==` is not a pin and is skipped — but reporting four packages from a file + with six requirement lines, without saying so, would understate the answer silently. + """ + found = dependencies.read_lockfiles( + ["requirements.txt"], _read({"requirements.txt": REQUIREMENTS}) + ) + + assert [(d.name, d.version) for d in found] == [ + ("jinja2", "2.4.1"), + ("lodash", "4.17.20"), + ] + assert dependencies.unpinned(REQUIREMENTS) == 2, "requests and django are ranges, not pins" + + +def test_a_checkout_with_no_lockfile_is_told_what_was_looked_for() -> None: + """An empty list reads as "you have no dependencies", which is a different claim.""" + assert dependencies.read_lockfiles(["README.md"], _read({"README.md": "#"})) == [] + for name in ("package-lock.json", "uv.lock", "poetry.lock", "requirements.txt"): + assert name in dependencies.WHAT_IS_LOOKED_FOR + + +# --- OSV ---------------------------------------------------------------------------------- + + +def _osv(handler: Callable[[httpx2.Request], httpx2.Response]) -> osv.Osv: + return osv.Osv(transport=httpx2.MockTransport(handler)) + + +def test_it_batches_the_query_and_asks_for_detail_only_on_what_came_back() -> None: + """One batch for N packages, then one detail call per id — not per package.""" + calls: list[str] = [] + + def handler(request: httpx2.Request) -> httpx2.Response: + calls.append(str(request.url)) + if request.url.path.endswith("/querybatch"): + return httpx2.Response(200, json={"results": [{"vulns": [{"id": "GHSA-x"}]}, {}]}) + return httpx2.Response(200, json={ + "id": "GHSA-x", + "summary": "template injection", + "affected": [{ + "package": {"name": "jinja2", "ecosystem": "PyPI"}, + "ranges": [{"events": [{"introduced": "0"}, {"fixed": "2.11.3"}]}], + }], + }) + + deps = [ + dependencies.Dependency("PyPI", "jinja2", "2.4.1", "poetry.lock"), + dependencies.Dependency("PyPI", "requests", "2.31.0", "poetry.lock"), + ] + found = _osv(handler).affected(deps) + + assert len(calls) == 2, "one batch and one detail, never one call per package" + assert [f.dependency.name for f in found] == ["jinja2"] + assert found[0].advisories[0].fixed == ("2.11.3",) + + +def test_a_vulnerability_with_no_published_fix_says_so() -> None: + """There is no bump to attempt, and proposing the next release and hoping is not an answer.""" + def handler(request: httpx2.Request) -> httpx2.Response: + if request.url.path.endswith("/querybatch"): + return httpx2.Response(200, json={"results": [{"vulns": [{"id": "GHSA-y"}]}]}) + return httpx2.Response(200, json={ + "id": "GHSA-y", + "summary": "unfixed", + "affected": [{ + "package": {"name": "lodash", "ecosystem": "npm"}, + "ranges": [{"events": [{"introduced": "0"}]}], + }], + }) + + deps = [dependencies.Dependency("npm", "lodash", "4.17.20", "package-lock.json")] + found = _osv(handler).affected(deps) + + assert found[0].advisories[0].fixed == () + assert found[0].advisories[0].has_a_fix is False + + +def test_several_fixed_versions_are_all_reported_and_none_is_chosen() -> None: + """**Where honesty is cheap to lose.** + + Choosing would mean comparing versions across two ecosystems' ordering rules, and a wrong + choice is a bump that does not fix what it claims to. All of them, and a person decides. + """ + def handler(request: httpx2.Request) -> httpx2.Response: + if request.url.path.endswith("/querybatch"): + return httpx2.Response(200, json={"results": [{"vulns": [{"id": "GHSA-z"}]}]}) + return httpx2.Response(200, json={ + "id": "GHSA-z", + "summary": "two branches", + "affected": [{ + "package": {"name": "jinja2", "ecosystem": "PyPI"}, + "ranges": [ + {"events": [{"introduced": "0"}, {"fixed": "2.11.3"}]}, + {"events": [{"introduced": "3.0"}, {"fixed": "3.1.3"}]}, + ], + }], + }) + + deps = [dependencies.Dependency("PyPI", "jinja2", "2.4.1", "poetry.lock")] + found = _osv(handler).affected(deps) + + assert found[0].advisories[0].fixed == ("2.11.3", "3.1.3") + assert found[0].advisories[0].has_a_fix is True + + +def test_an_advisory_about_another_package_does_not_become_this_ones_fix() -> None: + """One advisory can name several packages; only the matching one decides this bump. + + Reintroducing this defect made a jinja2 finding claim npm's fixing version, which is the kind + of wrong answer that looks entirely plausible in a report. + """ + def handler(request: httpx2.Request) -> httpx2.Response: + if request.url.path.endswith("/querybatch"): + return httpx2.Response(200, json={"results": [{"vulns": [{"id": "GHSA-multi"}]}]}) + return httpx2.Response(200, json={ + "id": "GHSA-multi", + "summary": "affects two ecosystems", + "affected": [ + {"package": {"name": "lodash", "ecosystem": "npm"}, + "ranges": [{"events": [{"introduced": "0"}, {"fixed": "9.9.9"}]}]}, + {"package": {"name": "jinja2", "ecosystem": "PyPI"}, + "ranges": [{"events": [{"introduced": "0"}, {"fixed": "2.11.3"}]}]}, + ], + }) + + deps = [dependencies.Dependency("PyPI", "jinja2", "2.4.1", "poetry.lock")] + found = _osv(handler).affected(deps) + + assert found[0].advisories[0].fixed == ("2.11.3",) + + +def test_nothing_affected_asks_for_no_detail_at_all() -> None: + """A clean project costs one request, not one plus zero-length follow-ups.""" + calls: list[str] = [] + + def handler(request: httpx2.Request) -> httpx2.Response: + calls.append(str(request.url)) + return httpx2.Response(200, json={"results": [{}, {}]}) + + deps = [ + dependencies.Dependency("npm", "a", "1.0.0", "package-lock.json"), + dependencies.Dependency("npm", "b", "1.0.0", "package-lock.json"), + ] + assert _osv(handler).affected(deps) == [] + assert len(calls) == 1 + + +def test_one_advisory_shared_by_two_packages_is_fetched_once() -> None: + """**The test the call-counting one above did not make redundant.** + + That test has a single affected package, so the id cache is never asked the same question + twice and removing it left every assertion green. A lock file with two packages under the + same advisory is the case that pays for the cache — and it is the common one, since a flaw in + a library is published once and hits everything that pins it. + """ + calls: list[str] = [] + + def handler(request: httpx2.Request) -> httpx2.Response: + calls.append(str(request.url)) + if request.url.path.endswith("/querybatch"): + return httpx2.Response(200, json={ + "results": [{"vulns": [{"id": "GHSA-same"}]}, {"vulns": [{"id": "GHSA-same"}]}], + }) + return httpx2.Response(200, json={ + "id": "GHSA-same", + "summary": "one flaw, two pins", + "affected": [ + {"package": {"name": "a", "ecosystem": "npm"}, + "ranges": [{"events": [{"introduced": "0"}, {"fixed": "1.1.0"}]}]}, + {"package": {"name": "b", "ecosystem": "npm"}, + "ranges": [{"events": [{"introduced": "0"}, {"fixed": "2.2.0"}]}]}, + ], + }) + + deps = [ + dependencies.Dependency("npm", "a", "1.0.0", "package-lock.json"), + dependencies.Dependency("npm", "b", "2.0.0", "package-lock.json"), + ] + found = _osv(handler).affected(deps) + + detail_calls = [c for c in calls if "/vulns/" in c] + assert len(detail_calls) == 1, "the same advisory must not be fetched once per package" + # And each package still gets its own fixing version out of the one document. + assert found[0].advisories[0].fixed == ("1.1.0",) + assert found[1].advisories[0].fixed == ("2.2.0",) + + +# --- the requirements files nobody read (item 180) -------------------------------------------- + + +def test_the_layouts_python_projects_actually_use_are_all_read() -> None: + """Item 180, found by running `deps` against this repository. + + `requirements/build.txt` is tracked here, pins three packages, and was read as none of them — + with a confident count printed above the silence. Item 172 matched the exact basename, and its + own reasoning for matching on a name at all condemns that: *"a monorepo pins per package, and + only reading the root would report a fraction of the truth as the whole."* + """ + files = { + "requirements.txt": "a==1.0\n", + "requirements/base.txt": "b==2.0\n", + "requirements/prod.txt": "c==3.0\n", + "requirements-dev.txt": "d==4.0\n", + "dev-requirements.txt": "e==5.0\n", + "backend/requirements/test.txt": "f==6.0\n", + } + + found = dependencies.read_lockfiles(list(files), lambda p: files.get(p)) + + assert {d.name for d in found} == {"a", "b", "c", "d", "e", "f"} + # And each one says which file said so, because a report that cannot is a report nobody can act + # on — a project can pin the same name in two files at two versions. + assert {d.source for d in found} == set(files) + + +def test_a_text_file_that_is_not_a_requirements_list_is_not_read_at_all() -> None: + """The bound on item 180's widening, and the first version of this test did not test it. + + **It used prose containing no line that looks like a pin**, so it passed whether the matcher + was careful or `^.*\\.txt$` — verified by making it exactly that and watching nothing go red. + A changelog is the realistic hazard and it *does* carry pin-shaped lines at the start of a + line, which is where `_PINNED` looks. + + What it would cost is not a wrong count. `--verify` would try to rewrite that line, and + `--open` would offer somebody a pull request editing their release notes. + """ + changelog = ( + "2.10.1 — 2026-01-01\n" + "===================\n" + "jinja2==2.10.1 is now the minimum supported version.\n" + "requests==2.31.0 was dropped from the extras.\n" + ) + + read = dependencies.read_lockfiles(["docs/notes.txt", "CHANGELOG.txt"], lambda _p: changelog) + + assert read == [] + assert not dependencies.is_requirements("CHANGELOG.txt") + assert not dependencies.is_requirements("docs/notes.txt") + # And the near misses, which are the ones a looser pattern would take. + assert not dependencies.is_requirements("install-requirements-guide.txt") + assert not dependencies.is_requirements("requirements.md") + + # The same text under a name that *is* a requirements file is read, which is what makes the + # assertions above about the name rather than about the content. + assert len(dependencies.read_lockfiles(["requirements/prod.txt"], lambda _p: changelog)) == 2 + + +def test_one_package_pinned_twice_in_two_files_is_two_facts() -> None: + """A real thing, and merging them would hide the disagreement rather than report it.""" + files = { + "requirements/base.txt": "jinja2==2.4.1\n", + "requirements/dev.txt": "jinja2==3.1.4\n", + } + + found = dependencies.read_lockfiles(list(files), lambda p: files.get(p)) + + assert sorted((d.version, d.source) for d in found) == [ + ("2.4.1", "requirements/base.txt"), + ("3.1.4", "requirements/dev.txt"), + ] + + +def test_this_repositorys_own_requirements_directory_reads() -> None: + """The falsifiable gate, as a test. Item 180. + + Against the real file rather than a fixture, for `test_this_repositorys_own_uv_lock_reads`'s + reason: a reader tested only against its own fixtures is a reader tested against itself. This + one is also the exact file that produced the finding. + """ + path = ROOT / "requirements" / "build.txt" + found = dependencies.read_lockfiles( + ["requirements/build.txt"], lambda _p: path.read_text(encoding="utf-8") + ) + + assert len(found) >= 3 + assert all(d.source == "requirements/build.txt" for d in found) + assert "build" in {d.name for d in found} + + +# --- the build context a verification gets (item 182) ----------------------------------------- + + +def test_verify_hands_the_source_to_a_build_that_reads_it( + tmp_path: Path, monkeypatch: object +) -> None: + """Item 113's fix, on the path that never inherited it. Found on the first third-party tree. + + The build context holds the **declared dependency files and never the source**, and three + ordinary installers read the source anyway: a `requirements.txt` that begins `-e .`, a `Gemfile` + that says `gemspec`, and `mvn test`. Item 113 measured all three and fixed the attempt path; + `deps --verify` was written afterwards and passed `source=None` unconditionally. + + Measured on `encode/httpx`, whose first requirement is `-e .[brotli,cli,http2,socks,zstd]`: + + ERROR: file:///work does not appear to be a Python project: + neither 'setup.py' nor 'pyproject.toml' found. + + Reported to the reader as *your own environment does not build*, which was true of what we built + and false of the project. Ruby, Java and PHP are all on the roadmap as stacks whose attempts + work, and every one of them reaches this the same way. + + Driven through `image.build` rather than through Docker: what is under test is **what the + verification asks for**, and a daemon would prove the same call more slowly. + """ + import pytest + + from hullwork import cli + from hullwork.manifest import parse_manifest + + asked: list[dict[str, object]] = [] + + class Built: + tag = "img:1" + + def watch(runtime: object, files: object, engine: object, **kwargs: object) -> Built: + asked.append(kwargs) + return Built() + + (tmp_path / "requirements.txt").write_text("-e .\njinja2==2.4.1\n") + (tmp_path / "pyproject.toml").write_text("[project]\nname='x'\nversion='1'\n") + manifest = parse_manifest( + "project: p\ngit: {provider: github, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: pip, dependencies: [requirements.txt], " + "install_needs_source: true}\n" + ) + from hullwork.sandbox import image as image_module + + monkeypatch.setattr(image_module, "build", watch) # type: ignore[attr-defined] + # The first build is the baseline and is the only one this needs: it either carries the source + # or it does not, and everything after it is the same call. + with pytest.raises(Exception): # noqa: B017 - it stops at the sandbox, after the build + cli._verify_one( + tmp_path, ["requirements.txt"], lambda p: (tmp_path / p).read_text(), + manifest, + dependencies.Dependency("PyPI", "jinja2", "2.4.1", "requirements.txt"), + ["2.10.1"], io.StringIO(), + ) + + assert asked, "no image was built at all" + assert asked[0]["source"] is not None, "the build got no source to install from" + assert asked[0]["source_ref"] is not None, ( + "an image whose source is not in its tag is an image reused across commits" + ) + + +def test_a_project_whose_install_does_not_read_the_source_still_gets_none( + tmp_path: Path, monkeypatch: object +) -> None: + """The cheap path stays cheap: the tag does not move and the image is reused between runs.""" + import pytest + + from hullwork import cli + from hullwork.manifest import parse_manifest + from hullwork.sandbox import image as image_module + + asked: list[dict[str, object]] = [] + + class Built: + tag = "img:1" + + def watch(runtime: object, files: object, engine: object, **kwargs: object) -> Built: + asked.append(kwargs) + return Built() + + (tmp_path / "requirements.txt").write_text("jinja2==2.4.1\n") + manifest = parse_manifest( + "project: p\ngit: {provider: github, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: pip, dependencies: [requirements.txt]}\n" + ) + monkeypatch.setattr(image_module, "build", watch) # type: ignore[attr-defined] + with pytest.raises(Exception): # noqa: B017 + cli._verify_one( + tmp_path, ["requirements.txt"], lambda p: (tmp_path / p).read_text(), + manifest, + dependencies.Dependency("PyPI", "jinja2", "2.4.1", "requirements.txt"), + ["2.10.1"], io.StringIO(), + ) + + assert asked and asked[0]["source"] is None + assert asked[0]["source_ref"] is None + + +# --- what cannot be verified at all, said before anything is built (item 182) ------------------ + + +def _manifest(declared: str) -> Manifest: + return parse_manifest( + "project: p\ngit: {provider: github, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + f"runtime: {{base: python-3.12, install: pip, dependencies: {declared}}}\n" + ) + + +def test_a_pin_in_a_file_the_image_never_installs_cannot_be_verified() -> None: + """**The false artefact this whole product exists to prevent, produced by the product.** + + `pallets/flask` pins four of its five advisory-carrying packages in + `examples/celery/requirements.txt`, which its image is not built from. Rewriting a version + there changes no byte the build reads: `dependency_digest` does not move, the image is reused, + and the suite passes exactly as it passed before — so the verdict is `clean` and the queue says + *ready to take*. + + Measured against a real daemon on 2026-08-09, on a tree declaring `requirements.txt` and + carrying `extras/requirements.txt`: + + [ready to take] jinja2 2.4.1 → 2.10.1 + $ docker run --rm python -c "import jinja2" + ModuleNotFoundError: No module named 'jinja2' + + A package **not installed in the environment its suite ran in**, offered as ready to merge — + and with item 178's `--open`, as a pull request. Item 174 found the same false verdict by the + other route, where the box was reused; this one needs no box at all. + """ + from hullwork import cli + + outside = dependencies.Dependency("PyPI", "jinja2", "2.4.1", "extras/requirements.txt") + + refusal = cli._cannot_be_verified(outside, _manifest("[requirements.txt]"), ["2.10.1"]) + + assert refusal is not None + assert "not one of the files your image is built from" in refusal + assert "requirements.txt" in refusal + + +def test_a_pin_the_image_does_install_is_verified_as_before() -> None: + """The refusal must not swallow the ordinary case, which is every project with one file.""" + from hullwork import cli + + inside = dependencies.Dependency("PyPI", "jinja2", "2.4.1", "requirements.txt") + + assert cli._cannot_be_verified(inside, _manifest("[requirements.txt]"), ["2.10.1"]) is None + + +def test_a_project_that_installs_nothing_cannot_have_an_upgrade_measured() -> None: + """**The worse half of the same finding, and `install: none` is the default value.** + + The generated Dockerfile copies no dependency file and runs no installer when `install` is + `none`, so the environment is `runtime.base` exactly as it comes. DR-0007 makes *the project + brings its own image* the primary path, so this is most projects rather than an edge. + + Measured on 2026-08-09 against a base image carrying `jinja2 3.0.0`, on a checkout pinning + `jinja2==2.4.1`: + + [ready to take] jinja2 2.4.1 → 2.10.1 + $ docker run --rm python -c "import jinja2; print(__version__)" + 3.0.0 + 3.0.0 + + **Neither version in the claim was ever installed.** The verdict said the suite passed before + the change and after it, which was true and was about nothing. + """ + from hullwork import cli + from hullwork.manifest import parse_manifest + + own_image = parse_manifest( + "project: p\ngit: {provider: github, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: none, dependencies: []}\n" + ) + anywhere = dependencies.Dependency("PyPI", "jinja2", "2.4.1", "requirements.txt") + + refusal = cli._cannot_be_verified(anywhere, own_image, ["2.10.1"]) + + assert refusal is not None + assert "install: none" in refusal + assert "rebuilding it" in refusal, "it has to say what a person does instead" + + +def test_a_refusal_is_counted_rather_than_only_printed() -> None: + """"I could not verify this" is a first-class answer, so it has to be in the tally. + + **Driven through `_verify_upgrades`, because the first version of this was not.** That one + asserted `needs_of` and `summary` over a hand-built report — both already true — so it passed + with the `reports.append` deleted. What it claims to check is that the refusal *reaches* the + queue, and only the function that builds the queue can say. + + A refusal that only reached the terminal left the summary saying `0 blocked` for a run that + could verify none of them, and that count is the one number in this command a person acts on. + No Docker: every refusal here is answered before anything is built. + """ + from hullwork import bump, cli, osv + + outside = dependencies.Dependency("PyPI", "jinja2", "2.4.1", "extras/requirements.txt") + findings = [ + osv.Finding( + dependency=outside, + advisories=(osv.Advisory(id="GHSA-x", summary="", fixed=("2.10.1",)),), + ) + ] + printed = io.StringIO() + + reports = cli._verify_upgrades( + Path("/nowhere"), ["extras/requirements.txt"], lambda _p: None, + _manifest("[requirements.txt]"), findings, printed, + ) + + assert len(reports) == 1, "the refusal never reached the queue" + assert bump.needs_of(reports[0]) is bump.Needs.BLOCKED + assert bump.summary(reports)[bump.Needs.BLOCKED] == 1 + assert "not one of the files your image is built from" in printed.getvalue() diff --git a/tests/test_dispatch.py b/tests/test_dispatch.py index efaed66..9a16f66 100644 --- a/tests/test_dispatch.py +++ b/tests/test_dispatch.py @@ -449,6 +449,58 @@ def test_a_deleted_conftest_is_restored(session: Session, item: Item, tmp_path: assert "tests/conftest.py" in verdict.detail +def test_a_conftest_the_fix_invented_is_caught_too( + session: Session, item: Item, tmp_path: Path +) -> None: + """The hole in item 046, found while building item 179 and measured on this sequence. + + **The guard iterated the before-image**, so it restored configuration that was *edited* and + never saw configuration that was *invented*. A project with no `conftest.py` — which is most of + them — could have one written by the fix phase: absent from `pristine`, never restored, + `restored` empty so no second gate ran, and the attempt published as `pr-open` with the file + that decided the gate inside its own diff. Verified by removing `created_test_config` from + `_restore_infrastructure`, at which point this publishes. + + The third verb, after edited and deleted, and the one nothing here expressed: **created**. + """ + # Deliberately no `_with_conftest`: the premise is a project that has none. + verdict, _, _ = _go( + session, item, tmp_path, + # baseline green, red gate red, green gate green (nothing collected), restored gate red. + script={"pytest#1": 1, "pytest#3": 1, REPRO: 0, FIX: 0}, + writes={REPRO: GOOD_TEST, FIX: {"conftest.py": "collect_ignore_glob = ['*']\n"}}, + ) + + assert verdict.outcome is AttemptOutcome.FAILED + assert verdict.phase is AttemptPhase.GREEN_GATE_RESTORED + assert "conftest.py" in verdict.detail + assert "conftest.py" not in verdict.changes.written + assert not (tmp_path / "conftest.py").exists(), "there is nothing to put it back to" + + +def test_a_test_the_fix_invented_is_still_welcome( + session: Session, item: Item, tmp_path: Path +) -> None: + """The distinction the narrower predicate exists to keep. + + A new file under `tests/` is a test and a fix that adds one is welcome; a new `conftest.py` is + configuration and there is no legitimate version of a fix phase inventing one. Using + `is_test_infrastructure` for the created case would have deleted both. + """ + verdict, _, _ = _go( + session, item, tmp_path, + script={"pytest#1": 1, REPRO: 0, FIX: 0}, + writes={ + REPRO: GOOD_TEST, + FIX: {"src.py": "x = 2\n", "tests/test_new.py": "def test_new():\n pass\n"}, + }, + ) + + assert verdict.outcome is AttemptOutcome.PR_OPEN + assert verdict.restored == "" + assert "tests/test_new.py" in verdict.changes.written + + # --- the agent is told what its test will be judged by (item 064) --------------------------------- diff --git a/tests/test_forge_code.py b/tests/test_forge_code.py index 3aeff02..8876c3d 100644 --- a/tests/test_forge_code.py +++ b/tests/test_forge_code.py @@ -238,9 +238,8 @@ def handler(request: httpx2.Request) -> httpx2.Response: def test_the_sign_off_trailer_is_never_requested() -> None: """The API will add it, and that is exactly why we must not ask. - `CONTRIBUTING.md` and the worker contract both say the DCO sign-off is a human act, done at the - merge gate. A machine that can emit the trailer can certify provenance it has no standing to - certify. + `CONTRIBUTING.md` says the DCO sign-off is a human act, done at the merge gate. A machine + that can emit the trailer can certify provenance it has no standing to certify. """ seen: dict[str, Any] = {} diff --git a/tests/test_propose.py b/tests/test_propose.py index e5d0ced..b3e755f 100644 --- a/tests/test_propose.py +++ b/tests/test_propose.py @@ -19,6 +19,7 @@ from __future__ import annotations +import subprocess from pathlib import Path import pytest @@ -308,3 +309,199 @@ def test_a_proposal_never_names_an_agent() -> None: parse_manifest(rendered) except ManifestError as exc: # pragma: no cover - a failure here is the point pytest.fail(f"the proposal for {text[:24]!r} is not a manifest: {exc}") + + +# --- Which forge holds this. Item 171. ------------------------------------------------- +# +# The defect: `render` emitted `provider: forgejo` as a constant, uncommented, so it read +# as observed while no code path could produce any other value. Found by running the +# command against a checkout hosted on GitHub and reading the two lines it printed. + + +def test_the_remote_host_decides_when_it_is_one_that_can_be_recognised() -> None: + """`github.com` and `gitlab.com` are the only hosts that name themselves. + + Everything else is self-hosted and unresolvable without asking it, which `propose` + may never do — it reaches nothing and needs no credential. + """ + assert propose.forge_for(".github/workflows/ci.yml", "github.com").provider == "github" + assert propose.forge_for(".gitlab-ci.yml", "gitlab.com").provider == "gitlab" + + +def test_the_github_workflows_directory_is_not_evidence_of_github() -> None: + """**The finding that stops this being a one-line change.** + + Forgejo Actions and Gitea Actions both read `.github/workflows/`, and this repository's + own deployment is the proof — those workflows run on a Forgejo instance. Reading that + directory as GitHub would be wrong for exactly the self-hosted projects this is for. + + This test exists to fail if somebody later "improves" the mapping. + """ + guess = propose.forge_for(".github/workflows/ci.yml", "git.example.com") + assert guess.evidence is None, "that directory settles nothing on an unknown host" + assert guess.provider == propose.PROVIDER_WHEN_UNDECIDED + + +def test_the_ci_location_decides_when_the_host_cannot() -> None: + """A self-hosted host says nothing; the three unambiguous directories say plenty.""" + for source, expected in ( + (".forgejo/workflows/ci.yml", "forgejo"), + (".gitea/workflows/ci.yml", "gitea"), + (".gitlab-ci.yml", "gitlab"), + ): + guess = propose.forge_for(source, "git.example.com") + assert guess.provider == expected, source + assert guess.evidence is not None + + +def test_the_host_outranks_the_ci_location() -> None: + """Where the repository lives beats which runner reads its workflows. + + A GitHub repository whose workflows sit in `.forgejo/workflows/` is a mirror, and the + coordinate a manifest needs is the one that answers requests. + """ + assert propose.forge_for(".forgejo/workflows/ci.yml", "github.com").provider == "github" + + +def test_a_remote_url_gives_up_its_host_in_both_spellings() -> None: + """`https://` and the `git@host:owner/name` form, plus what is not a URL at all.""" + assert propose.host_of_remote("https://github.com/owner/repo.git") == "github.com" + assert propose.host_of_remote("git@github.com:owner/repo.git") == "github.com" + assert propose.host_of_remote("ssh://git@forge.example.com/o/r") == "forge.example.com" + assert propose.host_of_remote(None) is None + assert propose.host_of_remote("not a url") is None + + +def test_an_undecided_provider_says_so_instead_of_looking_read() -> None: + """The whole point of the item: a default must not wear the costume of a reading. + + `render`'s contract is that uncommented means observed. When nothing decided, the line + still has to carry a value — `git.provider` is required and an unparseable proposal is + worse — so it carries the default *and says which two things would have settled it*. + """ + rendered = propose.render(propose.read("a/b", ".github/workflows/ci.yml", WITH_A_CONTAINER)) + lines = rendered.splitlines() + git_at = next(i for i, line in enumerate(lines) if line.startswith("git:")) + + assert "provider: forgejo" in lines[git_at] + said_it = "\n".join(lines[max(0, git_at - 3) : git_at]) + assert "default, not a reading" in said_it, "an undecided value has to admit that it is one" + assert ".github/workflows/" in said_it, "and say why that directory settled nothing" + parse_manifest(rendered) # and it still parses, which is why the value stays + + +def test_a_decided_provider_does_not_apologise_for_itself() -> None: + """The mirror of the test above: an observation must not read as a guess.""" + proposal = propose.read("a/b", ".github/workflows/ci.yml", WITH_A_CONTAINER) + proposal.remote_host = "github.com" + rendered = propose.render(proposal) + + assert "provider: github" in rendered + git_line = next(line for line in rendered.splitlines() if line.startswith("git:")) + assert "default" not in git_line + parse_manifest(rendered) + + +def test_the_header_and_the_provider_can_never_contradict_each_other() -> None: + """The defect as a reader saw it: a header naming one forge's file, then another's name. + + Asserted for the three locations that decide, on a host that decides nothing — which is + where the contradiction was reachable. + """ + for source, text, expected in ( + (".forgejo/workflows/ci.yml", WITH_A_CONTAINER, "forgejo"), + (".gitea/workflows/ci.yml", WITH_A_CONTAINER, "gitea"), + (".gitlab-ci.yml", GITLAB, "gitlab"), + ): + proposal = propose.read("a/b", source, text) + proposal.remote_host = "git.example.com" + rendered = propose.render(proposal) + header = rendered.splitlines()[0] + assert source in header + assert f"provider: {expected}" in rendered, f"{header} then a different forge" + + +def test_the_command_itself_carries_the_host_to_the_proposal(tmp_path: Path) -> None: + """**The test the other eight did not make redundant**, and the reason it exists. + + Every assertion above exercises pure functions. Deleting the one line in `cli.py` that + reads the remote and hands its host to the proposal left all of them green — so the whole + repair would have been reachable for deletion without a single test noticing, which is the + shape of defect item 116 found (a test that passes with its own subject removed). + + This drives the command against a real checkout with a real `origin`, which is how the + defect was found in the first place. + """ + from hullwork.cli import propose_from_local_ci + + workflows = tmp_path / ".github" / "workflows" + workflows.mkdir(parents=True) + (workflows / "ci.yml").write_text(WITH_A_CONTAINER, encoding="utf-8") + + for argv in ( + ["init", "-q"], + ["add", "-A"], + ["remote", "add", "origin", "https://github.com/owner/thing.git"], + ): + done = subprocess.run( # noqa: S603 + ["git", "-C", str(tmp_path), *argv], # noqa: S607 + capture_output=True, text=True, check=False, + ) + assert done.returncode == 0, done.stderr + + rendered = propose_from_local_ci(tmp_path) + assert rendered is not None + assert "git: {provider: github, repo: owner/thing}" in rendered + parse_manifest(rendered) + + +def test_a_proposal_with_no_installer_says_what_that_costs() -> None: + """Item 185. Every field this reader cannot fill carries a comment saying what is missing; + `install` carried none, and its absence is the one that is invisible. + + **Because the manifest it produces parses and builds perfectly.** What it cannot do is measure a + dependency upgrade: with no installer the image is the base exactly as it comes, so rewriting a + pinned version changes nothing the suite runs against. Measured on 2026-08-09 (item 182) before + this comment existed — a checkout pinning `jinja2==2.4.1`, a base image carrying 3.0.0, and a + verdict reading *your suite passed before this change and passes after it*, about a version that + was never installed. + """ + proposal = propose.Proposal(repo="o/r", source="ci.yml", base="python-3.12", tests="pytest") + + text = propose.render(proposal) + + assert "cannot be **measured**" in text + assert "changes nothing your suite would run against" in text + # **And what to do about it, which item 188 added.** Saying only the cost sent a reader with an + # image of their own away from a feature that serves them: `base` takes any image and `install` + # takes their own command, so the answer is one layer on top of what they already named — not a + # rebuild from scratch, and not a stack anybody has to add. + assert "keep this base and add two lines" in text + assert "one layer on top of the image you named" in text + # The command is not named: it is not in the published image, and naming something a reader + # cannot run invites them to type it and be told it does not exist. + assert "hullwork deps" not in text + + +def test_a_proposal_that_names_an_installer_says_nothing_of_the_kind() -> None: + """The note is about an absence. A manifest that can measure must not carry a caveat it earned + nothing of — a warning printed for everybody is a warning nobody reads.""" + proposal = propose.Proposal( + repo="o/r", source="ci.yml", base="python-3.12", tests="pytest", + install="pip", dependencies=("requirements.txt",), + ) + + text = propose.render(proposal) + + assert "cannot be **measured**" not in text + assert "install: 'pip'" in text + + +def test_a_proposal_with_no_base_does_not_lecture_about_installers() -> None: + """Without a base the whole `runtime` block is commented out and refused anyway (item 111). + Adding a second paragraph about a third field there is noise on top of a refusal.""" + proposal = propose.Proposal(repo="o/r", source="ci.yml", tests="pytest") + + text = propose.render(proposal) + + assert "cannot be **measured**" not in text diff --git a/tests/test_refit.py b/tests/test_refit.py new file mode 100644 index 0000000..d423bc0 --- /dev/null +++ b/tests/test_refit.py @@ -0,0 +1,973 @@ +"""The fix for the ones that break. Item 179, DR-0018 step 4. + +Driven with a fake sandbox, like `test_dispatch`, because what is under test is the *sequence and +its guards* rather than Docker — and the guards are the whole of this item. The container is proved +by effect elsewhere; the real Docker run for this sequence is recorded in the item. + +**The defect every test here circles.** Reverting the dependency passes every gate: red before, +green after, and the upgrade gone. It is the most plausible-looking false artefact this product +could emit, so the dependency files are read-only to the fix phase and the version is read back out +of the tree after the green gate. Both, because they fail differently. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +import argparse +import io +import json +from collections.abc import Callable +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest +from sqlalchemy.orm import Session + +from hullwork import bump, dependencies, dispatch, evidence, osv, refit, work +from hullwork.attempts import finish, has_attempt_left, start +from hullwork.engine import Engine, Phase +from hullwork.manifest import parse_manifest +from hullwork.models import AttemptOutcome, AttemptPhase, Item, ItemState, Lane, Project +from hullwork.sandbox.run import RunResult +from hullwork.states import transition + +MANIFEST = """ +project: p +git: {provider: forgejo, repo: o/r} +autofix: {agent: claude-code, gates: [tests, human-merge]} +tests: "pytest" +test_path: tests +runtime: {base: python-3.12, install: none, dependencies: [requirements.txt]} +""" + +LINTED = MANIFEST.replace( + "gates: [tests, human-merge]", "gates: [tests, lint, human-merge]" +) + 'lint: "ruff check ."\n' + +ENGINE = Engine(name="fake", image="img", protocol="anthropic", command="agent {phase}") + +#: What the fix phase is invoked as, once `Phase.REFIT` exists. +REFIT = "agent refit" +TESTS = "pytest" + +WAS, TO = "4.17.11", "4.17.21" + + +class FakeBox: + """Stands in for the container. Scripted per command, and writes what an agent would write.""" + + def __init__( + self, + worktree: Path, + script: dict[str, int], + writes: dict[str, dict[str, str]], + outputs: dict[str, str] | None = None, + ) -> None: + self.worktree = worktree + self.contract_dir = worktree / "_contract" + self.contract_dir.mkdir(exist_ok=True) + self.script = script + self.writes = writes + self.outputs = outputs or {} + self.ran: list[str] = [] + self.envs: dict[str, dict[str, str]] = {} + # `pytest` runs twice with two different expected results — red first, green after — so a + # double keying only on the command string cannot express this sequence at all. + self.counts: dict[str, int] = {} + + def run(self, command: str, timeout: int, env: dict[str, str] | None = None) -> RunResult: + del timeout + self.ran.append(command) + self.envs[command] = dict(env or {}) + nth = self.counts.get(command, 0) + self.counts[command] = nth + 1 + for name, body in self.writes.get(command, {}).items(): + if name == "__delete__": + (self.worktree / body).unlink() + continue + target = self.worktree / name + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(body) + code = self.script.get(f"{command}#{nth}", self.script.get(command, 0)) + printed = self.outputs.get(f"{command}#{nth}", self.outputs.get(command, "out")) + return RunResult(command=command, exit_code=code, output=printed, duration_ms=1) + + # Item 058: the agent's phases go through the entry that has a route to the gateway. + run_with_model = run + + +@pytest.fixture +def item(session: Session) -> Item: + project = Project( + slug="p", forge="forgejo", repo="o/r", + webhook_secret_hash="x", # noqa: S106 + ) + session.add(project) + session.flush() + row = Item( + project_id=project.id, fingerprint="fp", + title=f"lodash {WAS} → {TO} breaks the suite", lane=Lane.GREEN, + ) + session.add(row) + session.flush() + return row + + +def _pin(worktree: Path) -> str | None: + """What `requirements.txt` in this tree pins lodash at, or `None`. + + A real reader rather than a canned answer, so the restore and the re-read are exercised + against the same file the double edits — which is the only way the two guards can be shown to + be different guards. + """ + text = (worktree / "requirements.txt").read_text(encoding="utf-8") + for line in text.splitlines(): + if line.startswith("lodash=="): + return line.split("==", 1)[1].strip() + return None + + +def _go( + session: Session, + item: Item, + tmp_path: Path, + script: dict[str, int], + writes: dict[str, dict[str, str]] | None = None, + outputs: dict[str, str] | None = None, + guarded: tuple[str, ...] = ("requirements.txt",), + version_now: Callable[[Path], str | None] | None = None, + manifest_text: str = MANIFEST, +) -> tuple[dispatch.Verdict, FakeBox, Any]: + (tmp_path / "src.py").write_text("x = 1\n") + (tmp_path / "tests").mkdir(exist_ok=True) + (tmp_path / "tests" / "test_a.py").write_text("def test_a():\n assert True\n") + # The tree arrives with the upgrade already applied: that is what makes the first gate red. + (tmp_path / "requirements.txt").write_text(f"lodash=={TO}\n") + box = FakeBox(tmp_path, script, writes or {}, outputs) + attempt = start(session, item) + verdict = dispatch.refit( + session, item, parse_manifest(manifest_text), ENGINE, + box=box, # type: ignore[arg-type] + attempt=attempt, + package="lodash", to=TO, guarded=guarded, + version_now=version_now or _pin, + ) + return verdict, box, attempt + + +REAL_FIX = {"src.py": "x = 2 # works with the new lodash\n"} +REVERT = {"requirements.txt": f"lodash=={WAS}\n"} + +#: The suite failing on the tests the upgrade broke, in the shape a runner prints them. +BROKEN = ( + "FAILED tests/test_a.py::test_shape - TypeError: lodash.merge is not a function\n" + "FAILED tests/test_a.py::test_deep\n" + "2 failed, 40 passed" +) + + +# --- the sequence ------------------------------------------------------------------------- + + +def test_the_red_gate_is_the_upgrade_and_no_agent_is_asked_to_write_a_test( + session: Session, item: Item, tmp_path: Path +) -> None: + """The expensive half of DR-0003 is already paid for, by evidence nobody authored. + + There is no reproduce phase here at all: the failing tests are the project's own, failing + against the upgraded dependency. An agent asked to write one would be authoring the oracle, + which is the one thing no oracle in this product may be. + """ + verdict, box, attempt = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert verdict.outcome is AttemptOutcome.PR_OPEN + assert box.ran == [TESTS, REFIT, TESTS] + assert "agent reproduce" not in box.ran + phases = [step.phase for step in attempt.steps] + assert AttemptPhase.REPRODUCE not in phases + # Recorded as the red gate rather than as a baseline, because that is what it is: the run that + # establishes the failure the fix is judged against. + assert phases == [AttemptPhase.RED_GATE, AttemptPhase.FIX, AttemptPhase.GREEN_GATE] + + +def test_a_suite_that_passes_with_the_upgrade_has_nothing_to_fix( + session: Session, item: Item, tmp_path: Path +) -> None: + """Before the model is called: the breakage did not reproduce, so there is no work here.""" + verdict, box, _ = _go(session, item, tmp_path, script={f"{TESTS}#0": 0}) + + assert verdict.outcome is AttemptOutcome.NOT_REPRODUCIBLE + assert box.ran == [TESTS], "the agent must not be paid for to fix a suite that passes" + assert "passes with" in verdict.detail + + +def test_a_fix_phase_that_changes_nothing_is_not_a_fix( + session: Session, item: Item, tmp_path: Path +) -> None: + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 1}, + writes={}, + ) + + assert verdict.outcome is AttemptOutcome.FAILED + assert verdict.phase is AttemptPhase.FIX + assert "changed nothing" in verdict.detail + + +def test_a_real_fix_publishes_and_the_lock_still_carries_the_upgrade( + session: Session, item: Item, tmp_path: Path +) -> None: + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert verdict.outcome is AttemptOutcome.PR_OPEN + assert verdict.changes.written["src.py"] == REAL_FIX["src.py"].encode() + assert not verdict.reverted + assert TO in verdict.detail, "the artefact has to say which upgrade this makes possible" + + +# --- the guard this item exists to get right ------------------------------------------------ + + +def test_the_dependency_file_is_put_back_when_the_fix_phase_edits_it( + session: Session, item: Item, tmp_path: Path +) -> None: + """Read-only means put back, not merely noticed. + + Restoring rather than detecting is what keeps the revert out of the published change: a + dependency file that reached `changes` would be a pull request that undoes the upgrade it + claims to make possible. + """ + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: {**REAL_FIX, **REVERT}}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert (tmp_path / "requirements.txt").read_text() == f"lodash=={TO}\n" + assert "requirements.txt" not in verdict.changes.written + assert verdict.reverted == "requirements.txt" + + +def test_a_green_gate_reached_by_reverting_is_reported_as_a_revert( + session: Session, item: Item, tmp_path: Path +) -> None: + """The whole item, in one test. + + The fix phase puts the old version back and does nothing else. The file is restored before + anything is collected, so there is no change left to gate — and no second gate is paid for, + which is the honest shape rather than a saving: within one attempt the revert could never have + bought a green gate anyway. The image was built before this ran and the phases have no network, + so the installed version does not move whatever the file says. What a revert buys is the + published diff, and that is what the restore takes away. + """ + verdict, box, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 1}, + writes={REFIT: REVERT}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert verdict.outcome is not AttemptOutcome.PR_OPEN + assert verdict.outcome is AttemptOutcome.FAILED + assert verdict.reverted == "requirements.txt" + assert "revert" in verdict.detail + assert not verdict.changes, "a revert that was put back leaves nothing to publish" + assert (tmp_path / "requirements.txt").read_text() == f"lodash=={TO}\n" + assert box.ran == [TESTS, REFIT] + + +def test_a_lock_that_no_longer_carries_the_upgrade_is_a_revert_however_it_got_there( + session: Session, item: Item, tmp_path: Path +) -> None: + """The backstop, and it is not the same guard as the restore. + + The restore covers the files this ecosystem's resolver is known to touch. The re-read covers + everything else — a file nobody taught `touches` about, a pin moved somewhere the guard does + not look. A green gate whose tree no longer carries the upgraded version is not a fix. + """ + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + version_now=lambda _worktree: WAS, + ) + + assert verdict.outcome is AttemptOutcome.FAILED + assert "revert" in verdict.detail + assert WAS in verdict.detail and TO in verdict.detail + + +def test_a_tree_that_lost_the_dependency_entirely_is_not_a_fix_either( + session: Session, item: Item, tmp_path: Path +) -> None: + """`None` is not `to`, and the message has to say which of the two happened.""" + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + version_now=lambda _worktree: None, + ) + + assert verdict.outcome is AttemptOutcome.FAILED + assert "no longer pins" in verdict.detail + + +def test_every_file_the_move_can_touch_is_guarded_not_only_the_one_that_pins( + session: Session, item: Item, tmp_path: Path +) -> None: + """Item 175's finding, one layer up. + + `npm install` rewrites `package.json` as well as the lock, so a guard that watched only the + lock would let a fix widen the range back and leave the pin looking untouched. + """ + (tmp_path / "package.json").write_text('{"dependencies": {"lodash": "^4.17.21"}}\n') + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: {**REAL_FIX, "package.json": '{"dependencies": {"lodash": "^4.17.11"}}\n'}}, + outputs={f"{TESTS}#0": BROKEN}, + guarded=("requirements.txt", "package.json"), + ) + + assert (tmp_path / "package.json").read_text() == '{"dependencies": {"lodash": "^4.17.21"}}\n' + assert "package.json" not in verdict.changes.written + assert verdict.reverted == "package.json" + + +def test_a_dependency_file_the_fix_deleted_comes_back( + session: Session, item: Item, tmp_path: Path +) -> None: + """Deleting the pin is a revert with extra steps, and `Changes` learned that lesson once.""" + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 1}, + writes={REFIT: {"__delete__": "requirements.txt"}}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert (tmp_path / "requirements.txt").read_text() == f"lodash=={TO}\n" + assert verdict.reverted == "requirements.txt" + assert "requirements.txt" not in verdict.changes.deleted + + +# --- the guards this sequence inherits rather than reimplements ------------------------------- + + +def test_test_infrastructure_the_fix_switched_off_is_still_put_back( + session: Session, item: Item, tmp_path: Path +) -> None: + """Item 046 applies unchanged: a suite that collects nothing passes trivially. + + And it is a `conftest.py` the phase **invented** rather than edited, because that is the shape + this item found the guard did not cover. An agent that cannot make an upgrade fit is exactly + the caller with a reason to switch off the tests it cannot satisfy, so this sequence needs the + fixed guard more than the one it was written for. + """ + verdict, box, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0, f"{TESTS}#2": 1}, + writes={REFIT: {**REAL_FIX, "conftest.py": "collect_ignore_glob = ['*']\n"}}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert verdict.outcome is AttemptOutcome.FAILED + assert verdict.phase is AttemptPhase.GREEN_GATE_RESTORED + assert "conftest.py" in verdict.restored + assert box.ran == [TESTS, REFIT, TESTS, TESTS] + + +def test_the_lint_gate_does_not_discard_a_verified_fix( + session: Session, item: Item, tmp_path: Path +) -> None: + """Item 067 applies unchanged: the lint gate contests the style, not the claim.""" + verdict, _, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0, "ruff check .": 1}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + manifest_text=LINTED, + ) + + assert verdict.outcome is AttemptOutcome.PR_OPEN_LINT_FAILED + + +def test_the_fix_phase_is_told_which_upgrade_and_which_tests( + session: Session, item: Item, tmp_path: Path +) -> None: + """The phase the agent is asked for is its own, not the bug-fixing one. + + `fix` tells the agent about a reproducing test file at a path that does not exist here. A + phase that names evidence nobody wrote is how an attempt spends its one try on a + misunderstanding, which is what item 094 measured. + """ + _, box, _ = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN}, + ) + + assert box.envs[REFIT]["HULLWORK_AGENT_PHASE"] == Phase.REFIT.value + + +# --- the accounting, which is the ordinary one ------------------------------------------------- + + +def test_the_attempt_is_spent_and_accounted_for_exactly_as_any_other( + session: Session, item: Item, tmp_path: Path +) -> None: + """Through `run_one`, because that is what a refit actually goes through in production. + + **The parameter this covers is `sequence`.** Everything `run_one` does — claiming the item, + opening the attempt, the seal, the ceiling checks, publishing, releasing — is about an attempt + rather than about what the attempt was for, so a refit gets all of it by passing a different + sequence and nothing else. Without this test that parameter is exercised nowhere. + + DR-0003's rule is then the ordinary one: this item has had its try, and `has_attempt_left` + says so to whoever asks next. + """ + from functools import partial + + (tmp_path / "src.py").write_text("x = 1\n") + (tmp_path / "tests").mkdir(exist_ok=True) + (tmp_path / "requirements.txt").write_text(f"lodash=={TO}\n") + box = FakeBox( + tmp_path, + {f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + {REFIT: REAL_FIX}, + {f"{TESTS}#0": BROKEN}, + ) + project = session.get(Project, item.project_id) + assert project is not None + project.manifest = parse_manifest(MANIFEST).model_dump(mode="json") + transition(item, ItemState.TRIAGED) + transition(item, ItemState.READY) + session.flush() + published: list[str] = [] + + def publisher(*_a: object) -> str: + published.append("written") + return "somewhere" + + outcome = work.run_one( + session, + work.Eligible(item=item, project=project), + engine=ENGINE, + box_factory=lambda _m: box, + publisher=publisher, + sequence=partial( + dispatch.refit, + package="lodash", to=TO, guarded=("requirements.txt",), version_now=_pin, + ), + ) + + assert outcome.outcome is AttemptOutcome.PR_OPEN + assert published == ["written"] + assert item.state is ItemState.PR_OPEN + assert not has_attempt_left(session, item), "DR-0003: one attempt, then a human" + + +# --- what the reviewer reads ------------------------------------------------------------------ + + +def test_the_artefact_names_the_upgrade_and_the_tests_and_does_not_overclaim( + session: Session, item: Item, tmp_path: Path +) -> None: + """The criterion this item is judged on, read off the document a person actually receives. + + And the sentence that must **not** be there: *a test that failed against unmodified code*. The + code was modified before the first gate ran — by the upgrade — so `evidence`'s ordinary + headline would be false in the one word carrying it, which is the overclaim item 171 removed + from this product once already. + """ + verdict, _, attempt = _go( + session, item, tmp_path, + script={f"{TESTS}#0": 1, f"{TESTS}#1": 0}, + writes={REFIT: REAL_FIX}, + outputs={f"{TESTS}#0": BROKEN, f"{TESTS}#1": "42 passed"}, + ) + finish(session, attempt, verdict.outcome) + + body = evidence.pull_request_body( + item, attempt, detail=verdict.detail, claim=verdict.claim + ) + + assert "unmodified code" not in body + assert "the upgrade is still" in body + # Which upgrade this makes possible. + assert f"lodash {TO}" in body + # And the tests that were failing, from the red gate's own output rather than from prose. + assert "test_shape" in body + assert "exit `1` as it must" in body + assert "exit `0` as it must" in body + + +# --- what a breakage is, and what the agent is told about it ---------------------------------- + + +def _breaks(package: str = "lodash") -> bump.Report: + return bump.Report( + package=package, + was=WAS, + answers=(bump.Answer(bump.Verdict.BREAKS, package, WAS, TO, detail=BROKEN),), + ) + + +def test_an_upgrade_is_built_from_a_breaks_verdict_and_carries_the_failing_tests() -> None: + upgrade = refit.from_report(_breaks(), source="requirements.txt") + + assert upgrade is not None + assert (upgrade.package, upgrade.was, upgrade.to) == ("lodash", WAS, TO) + assert "test_shape" in upgrade.failing + + +def test_a_report_that_did_not_break_is_not_work_for_an_agent() -> None: + """Only `needs work` reaches this. A clean verdict is item 178's to deliver, not this one's.""" + clean = bump.Report( + package="lodash", was=WAS, + answers=(bump.Answer(bump.Verdict.CLEAN, "lodash", WAS, TO),), + ) + red = bump.Report( + package="lodash", was=WAS, + answers=(bump.Answer(bump.Verdict.ALREADY_RED, "lodash", WAS, TO),), + ) + + assert refit.from_report(clean, source="requirements.txt") is None + assert refit.from_report(red, source="requirements.txt") is None + + +def test_a_package_that_broke_and_then_passed_is_not_work_for_an_agent() -> None: + """The case the `needs_of` filter actually exists for, and the two above do not reach. + + `bump.verify` tries candidates in order, so a report can carry a `breaks` answer **and** a + later clean one — the lower fix broke the suite and the higher one did not. That is a package + to take, which is item 178's to deliver, and scanning for a `breaks` answer instead of asking + what the report *needs* would hand it to an agent and pay a model to fix something already + fixed. + + Verified by reintroducing the defect: with the `needs_of` line removed the two tests above + still pass, because a report whose only answer is clean has no `breaks` answer to find. This + one is the only thing between that filter and being dead code. + """ + broke_then_passed = bump.Report( + package="lodash", was=WAS, + answers=( + bump.Answer(bump.Verdict.BREAKS, "lodash", WAS, "4.17.20", detail=BROKEN), + bump.Answer(bump.Verdict.CLEAN, "lodash", WAS, TO), + ), + ) + broke_then_the_suite_went_red = bump.Report( + package="lodash", was=WAS, + answers=( + bump.Answer(bump.Verdict.BREAKS, "lodash", WAS, "4.17.20", detail=BROKEN), + bump.Answer(bump.Verdict.ALREADY_RED, "lodash", WAS, TO), + ), + ) + + assert bump.needs_of(broke_then_passed) is bump.Needs.JUST_TAKE_IT + assert refit.from_report(broke_then_passed, source="requirements.txt") is None + assert refit.from_report(broke_then_the_suite_went_red, source="requirements.txt") is None + + +def test_what_a_move_can_touch_is_asked_of_the_resolver_that_owns_it() -> None: + """Item 175 measured that `npm install` rewrites `package.json` as well as the lock. + + Read from `resolve.touches` rather than listed here, so an ecosystem added there is guarded + without anybody remembering to — and asserted here, because every other test in this file + hands `guarded` in by hand and would pass with this function returning the lock alone. + """ + assert set(refit.guarded_for("package-lock.json")) == {"package.json", "package-lock.json"} + assert set(refit.guarded_for("uv.lock")) == {"pyproject.toml", "uv.lock"} + # A list of versions is the only file its own move touches. + assert refit.guarded_for("requirements.txt") == ("requirements.txt",) + # And a path in a subdirectory is still the file it is named after. + assert set(refit.guarded_for("frontend/package-lock.json")) == { + "package.json", "package-lock.json" + } + + +def test_the_brief_forbids_the_cheat_by_name_and_says_what_happens_if_it_is_tried() -> None: + """An agent that is not told is an agent that will try it, and be reported for it.""" + upgrade = refit.from_report(_breaks(), source="requirements.txt") + assert upgrade is not None + text = refit.brief(upgrade) + + assert "requirements.txt" in text + assert TO in text and WAS in text + assert "test_shape" in text, "the tests it has to make pass are the point of the brief" + lowered = text.lower() + assert "revert" in lowered + assert "read-only" in lowered or "do not" in lowered + + +@pytest.mark.parametrize( + ("name", "body", "expected"), + [ + ("requirements.txt", f"lodash=={TO}\n", TO), + ("requirements.txt", f"lodash=={WAS}\n", WAS), + ("requirements.txt", "something-else==1.0\n", None), + ], +) +def test_the_version_is_read_back_out_of_the_file_that_pins_it( + tmp_path: Path, name: str, body: str, expected: str | None +) -> None: + (tmp_path / name).write_text(body, encoding="utf-8") + upgrade = refit.from_report(_breaks(), source=name) + assert upgrade is not None + + assert refit.version_now(upgrade, tmp_path) == expected + + +def test_the_version_is_read_back_out_of_a_lock_too(tmp_path: Path) -> None: + """Whichever of the four file shapes pins it — item 172's readers, not a fifth parser.""" + (tmp_path / "package-lock.json").write_text( + json.dumps({"packages": {"": {}, "node_modules/lodash": {"version": TO}}}), + encoding="utf-8", + ) + upgrade = refit.from_report(_breaks(), source="package-lock.json") + assert upgrade is not None + + assert refit.version_now(upgrade, tmp_path) == TO + + +def test_a_file_that_is_no_longer_there_reads_as_no_version_rather_than_raising( + tmp_path: Path, +) -> None: + upgrade = refit.from_report(_breaks(), source="requirements.txt") + assert upgrade is not None + + assert refit.version_now(upgrade, tmp_path) is None + + +def test_the_item_carries_the_upgrade_where_a_person_reads_it(session: Session) -> None: + upgrade = refit.from_report(_breaks(), source="requirements.txt") + assert upgrade is not None + manifest = parse_manifest(MANIFEST) + + _, made = refit.stage(session, manifest, upgrade, repo="o/r") + + assert "lodash" in made.title + assert WAS in made.title and TO in made.title + assert made.lane is Lane.GREEN + assert made.lane_reason + + +# --- which of the queue is handed to an agent at all ------------------------------------------- + + +def _finding(name: str, was: str) -> osv.Finding: + return osv.Finding( + dependency=dependencies.Dependency("PyPI", name, was, "requirements.txt"), + advisories=( + osv.Advisory(id=f"GHSA-{name}", summary="something", fixed=(TO,)), + ), + ) + + +def _report(package: str, verdict: bump.Verdict, detail: str = "") -> bump.Report: + return bump.Report( + package=package, was=WAS, + answers=(bump.Answer(verdict, package, WAS, TO, detail=detail),), + ) + + +def _fix_run( + tmp_path: Path, reports: list[bump.Report], monkeypatch: pytest.MonkeyPatch, + fail: str = "", +) -> tuple[list[refit.Upgrade], str]: + """Run the `--fix` half of `deps` with the attempt itself stubbed out. + + What is under test here is the *selection*: which of a queue is worth a model at all, and in + what order. Running the attempt would be testing `_attempt`, which has its own tests and needs + a daemon and a credential. + """ + from hullwork import cli + + asked: list[refit.Upgrade] = [] + + def fake_run(_settings: object, *_a: object, **kwargs: object) -> object: + upgrade = kwargs["upgrade"] if "upgrade" in kwargs else _a[2] + asked.append(upgrade) # type: ignore[arg-type] + if fail: + raise refit.NotUpgradableError(fail) + return SimpleNamespace( + outcome=AttemptOutcome.PR_OPEN, detail="it fits now", pull_request="somewhere-on-disk" + ) + + monkeypatch.setattr(refit, "run", fake_run) + printed = io.StringIO() + code = cli._fix_the_ones_that_break( + argparse.Namespace(into=str(tmp_path / "out"), fix=True), + None, # type: ignore[arg-type] + tmp_path, + ["requirements.txt"], + parse_manifest(MANIFEST), + [_finding(r.package, r.was) for r in reports], + reports, + printed, + ) + assert code == 0 + return asked, printed.getvalue() + + +def test_only_the_ones_that_need_work_are_handed_to_an_agent( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A model is the expensive part of this command, so it is spent on one bucket of four. + + The verified-green ones need no agent and are item 178's to deliver; a red baseline is the + project's own suite and nothing can be claimed against it; a blocked one has nothing to try. + """ + reports = [ + _report("clean-one", bump.Verdict.CLEAN), + _report("red-one", bump.Verdict.ALREADY_RED), + _report("blocked-one", bump.Verdict.CANNOT_MOVE), + _report("broken-one", bump.Verdict.BREAKS, BROKEN), + ] + + asked, _ = _fix_run(tmp_path, reports, monkeypatch) + + assert [u.package for u in asked] == ["broken-one"] + + +def test_the_broken_ones_are_worked_easiest_first( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """`bump.ranked`'s order, not this command's own. + + A run that is interrupted has then spent its money on the ones a person would have started + with, and the ordering rule stays in the one module that already owns it. + """ + reports = [ + _report("twelve", bump.Verdict.BREAKS, "\n".join(f"FAILED t{i}" for i in range(12))), + _report("two", bump.Verdict.BREAKS, "FAILED a\nFAILED b"), + ] + + asked, _ = _fix_run(tmp_path, reports, monkeypatch) + + assert [u.package for u in asked] == ["two", "twelve"] + + +def test_the_advisory_travels_with_the_upgrade( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The agent's brief names why this upgrade exists, and a reader can go and check it.""" + asked, _ = _fix_run( + tmp_path, [_report("broken-one", bump.Verdict.BREAKS, BROKEN)], monkeypatch + ) + + assert asked[0].advisory == "GHSA-broken-one" + assert "osv.dev" in asked[0].url + + +def test_an_upgrade_that_cannot_be_applied_is_said_rather_than_raised( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """No attempt was begun, so nothing was spent — and the resolver's own words survive. + + A traceback here would end the whole run over one package, which on a queue of six is five + upgrades nobody was told about. + """ + _, printed = _fix_run( + tmp_path, + [_report("broken-one", bump.Verdict.BREAKS, BROKEN)], + monkeypatch, + fail="constrained-by-manifest: the range does not allow it", + ) + + assert "could not be applied" in printed + assert "the range does not allow it" in printed + assert "0 of 1" in printed + + +def test_a_queue_with_nothing_broken_says_so_rather_than_going_quiet( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + asked, printed = _fix_run(tmp_path, [_report("clean-one", bump.Verdict.CLEAN)], monkeypatch) + + assert asked == [] + assert "had nothing to do" in printed + + +def test_two_upgrades_of_one_package_are_two_items(session: Session) -> None: + """The identity of this work is the pair of versions, not the package. + + A fingerprint over the name alone would make the second upgrade a repeat of the first, and + `dedup` would increment a counter instead of creating work. + """ + manifest = parse_manifest(MANIFEST) + first = refit.from_report(_breaks(), source="requirements.txt") + second = bump.Report( + package="lodash", was=WAS, + answers=(bump.Answer(bump.Verdict.BREAKS, "lodash", WAS, "4.17.22", detail=BROKEN),), + ) + other = refit.from_report(second, source="requirements.txt") + assert first is not None and other is not None + + _, a = refit.stage(session, manifest, first, repo="o/r") + _, b = refit.stage(session, manifest, other, repo="o/r") + + assert a.fingerprint != b.fingerprint + + +# --- first contact, which is where item 048's lesson had not reached (item 184) ----------------- + + +def test_fix_without_a_model_refuses_before_a_container_is_built() -> None: + """**Item 048's finding, on the path that had not learned it.** + + Measured by running `--fix` for the first time on 2026-08-09: the refusal was raised inside + `refit.run`, so it arrived **after** every container had been built and every suite run — the + most expensive place available — and it arrived as a `WiringError` traceback rather than as a + refusal, which item 120 is about. The message itself was right; where and how it appeared were + not. + + Knowable from the settings and nothing else, so it costs nothing to answer this early. + """ + from hullwork import cli + from hullwork.config import Settings + + with pytest.raises(cli.CommandError) as refused: + cli._refuse_without_a_model(Settings()) + + assert "no model credential" in str(refused.value) + # And it names the half that needs nothing, because that is what the reader can still run. + assert "--verify" in str(refused.value) + + +def test_fix_asks_for_the_model_before_it_reads_a_single_lock_file( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """**The call site, which is where the defect was** — and the first version of this test did + not cover it. + + That one asserted `_refuse_without_a_model` raises, which it always did: the function was never + the problem. Deleting the *call* from `_cmd_deps` left every test green and put the refusal back + where it started, minutes and several containers later. Verified by doing exactly that. + + The stub raises something unmistakable, so what this asserts is the **order**: the refusal wins + over the lock-file refusal that the same checkout would otherwise produce. + """ + import argparse + import subprocess + + from hullwork import cli + from hullwork.config import Settings + + subprocess.run(["git", "init", "-q", str(tmp_path)], check=True) # noqa: S603, S607 + (tmp_path / "hullwork.yml").write_text( + "project: p\ngit: {provider: forgejo, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: none, dependencies: []}\n" + ) + + def refuse(*_a: object, **_k: object) -> None: + raise cli.CommandError("asked for the model first") + + monkeypatch.setattr(cli, "_refuse_without_a_model", refuse) + + with pytest.raises(cli.CommandError) as refused: + cli._cmd_deps( + argparse.Namespace( + checkout=str(tmp_path), verify=False, fix=True, open=False, into=str(tmp_path) + ), + Settings(), + io.StringIO(), + ) + + assert "asked for the model first" in str(refused.value) + + +def test_verify_alone_never_asks_for_a_model( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The property `deps` is sold on: `--verify` runs the suite against every published fix with + no credential of any kind. Asserted by making the lookup explode.""" + import argparse + import subprocess + + from hullwork import cli + from hullwork.config import Settings + + subprocess.run(["git", "init", "-q", str(tmp_path)], check=True) # noqa: S603, S607 + (tmp_path / "hullwork.yml").write_text( + "project: p\ngit: {provider: forgejo, repo: o/r}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: none, dependencies: []}\n" + ) + + def explode(*_a: object, **_k: object) -> None: + raise AssertionError("--verify asked for a model credential") + + monkeypatch.setattr(cli, "_refuse_without_a_model", explode) + + with pytest.raises(cli.CommandError) as refused: + cli._cmd_deps( + argparse.Namespace( + checkout=str(tmp_path), verify=True, fix=False, open=False, into=str(tmp_path) + ), + Settings(), + io.StringIO(), + ) + + assert "no lock file" in str(refused.value) + + +def test_fix_refuses_without_the_gateway_image_before_anything_is_built( + monkeypatch: pytest.MonkeyPatch +) -> None: + """**Item 048's finding for the third time**, found by running `--fix` against a real model. + + It died with `could not start the gateway / Unable to find image 'hullwork:dev' locally` — + after OSV, four image builds and two suite runs. The gateway is where the model credential + lives so the sandbox never holds it (DR-0004), and **every** agent path starts one, so this is + a fact about the instance rather than about the newest command. + + The refusal has to say what the image is for and how to build it: *image not found* is not + something anybody can act on. + """ + from hullwork import cli + from hullwork import work as work_module + from hullwork.config import Settings + + monkeypatch.setattr(work_module, "_model_credential", lambda _s: "a-key") + monkeypatch.setattr( + "hullwork.sandbox.net.why_the_gateway_cannot_start", + lambda **_k: "the gateway image `hullwork:dev` is not on this Docker daemon", + ) + + with pytest.raises(cli.CommandError, match="gateway image"): + cli._refuse_without_a_model(Settings()) + + +def test_the_gateway_refusal_names_what_to_run_and_what_not_to() -> None: + """`docker compose build` **does not make it**, and this repository said it did. + + The compose file pins a published image and has no build stage — its own comment says to add + one — so that instruction exits 0 and produces nothing. Measured on 2026-08-09 while chasing + exactly this skip, and the sentence a reader gets now says both halves. + """ + from hullwork.sandbox import net + + said = net.why_the_gateway_cannot_start.__doc__ or "" + assert "expensive place available" in said + + # The refusal itself, built without a daemon by asking for an image that cannot exist. + refusal = net.why_the_gateway_cannot_start(docker="a-binary-that-is-not-here") + assert refusal is None, "a missing docker client is doctor's question, not this one's" diff --git a/tests/test_resolve.py b/tests/test_resolve.py new file mode 100644 index 0000000..26dc10f --- /dev/null +++ b/tests/test_resolve.py @@ -0,0 +1,169 @@ +"""Moving a resolved graph by running the ecosystem's own tool. Item 175, DR-0016. + +**No test here starts a container**: `run` is injected, so what these assert is the decision logic +— which command, which files, and above all whether the tool is believed. The Docker path is +measured once by hand and written into the item. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +from pathlib import Path + +from hullwork import resolve + +NPM_LOCK = """ +{ + "lockfileVersion": 3, + "packages": { + "": {"name": "app", "version": "1.0.0"}, + "node_modules/lodash": {"version": "%s"} + } +} +""" + +UV_LOCK = """ +[[package]] +name = "jinja2" +version = "%s" +""" + + +def test_each_lock_file_has_exactly_one_owner() -> None: + assert resolve.resolver_for("package-lock.json").lock == "package-lock.json" # type: ignore[union-attr] + assert resolve.resolver_for("deep/uv.lock").lock == "uv.lock" # type: ignore[union-attr] + assert resolve.resolver_for("poetry.lock").lock == "poetry.lock" # type: ignore[union-attr] + + +def test_a_lock_nobody_can_move_has_no_resolver() -> None: + """`None` is not a gap to fill silently — item 173's refusal still applies, by name.""" + assert resolve.resolver_for("Cargo.lock") is None + assert resolve.resolver_for("requirements.txt") is None + + +def test_the_command_names_the_package_and_the_version() -> None: + npm = resolve.resolver_for("package-lock.json") + assert npm is not None + built = resolve.command_for(npm, "lodash", "4.17.21") + + assert "lodash@4.17.21" in built + # Moves the graph without downloading node_modules: seconds rather than minutes. + assert "--package-lock-only" in built + + +def test_the_manifest_is_required_and_named_when_absent() -> None: + """A `uv.lock` with no `pyproject.toml` cannot be resolved by anything. + + Checked before the container starts, because finding it out after pulling an image is a minute + spent on a fact that was on disk the whole time. + """ + uv = resolve.resolver_for("uv.lock") + assert uv is not None + assert resolve.missing_from(uv, ["uv.lock"]) == ["pyproject.toml"] + assert resolve.missing_from(uv, ["pyproject.toml", "uv.lock"]) == [] + # And from a subdirectory, because a monorepo pins per package. + assert resolve.missing_from(uv, ["svc/pyproject.toml", "svc/uv.lock"]) == [] + + +def test_the_version_is_read_back_out_of_each_lock_shape() -> None: + assert resolve.version_in_lock(NPM_LOCK % "4.17.21", "package-lock.json", "lodash") == "4.17.21" + assert resolve.version_in_lock(UV_LOCK % "3.1.6", "uv.lock", "jinja2") == "3.1.6" + # Spelling is not identity: lock files disagree about which one they write. + assert resolve.version_in_lock(UV_LOCK % "3.1.6", "uv.lock", "Jinja2") == "3.1.6" + assert resolve.version_in_lock(UV_LOCK % "3.1.6", "uv.lock", "absent") is None + + +def test_a_resolver_that_moved_nothing_is_not_a_success(tmp_path: Path) -> None: + """**The defect this whole module is careful about.** + + Every one of these tools resolves within whatever range the manifest allows and exits 0. A + `"^4.17.11"` answers success having never gone near 5.x. Believing the exit code publishes a + `clean` verdict for an upgrade that never happened — the worst artefact this can emit. + """ + (tmp_path / "package-lock.json").write_text(NPM_LOCK % "4.17.20", encoding="utf-8") + npm = resolve.resolver_for("package-lock.json") + assert npm is not None + + result = resolve.upgrade( + resolver=npm, worktree=tmp_path, package="lodash", version="5.0.0", + present=["package.json", "package-lock.json"], + run=lambda *_: (0, "up to date"), + ) + + assert result.outcome is resolve.Outcome.CONSTRAINED + assert result.ok is False + assert "still 4.17.20, not 5.0.0" in result.detail + assert "range in the manifest" in result.detail + + +def test_a_resolver_that_moved_the_graph_is_a_success(tmp_path: Path) -> None: + lock = tmp_path / "package-lock.json" + lock.write_text(NPM_LOCK % "4.17.11", encoding="utf-8") + npm = resolve.resolver_for("package-lock.json") + assert npm is not None + + def run(_resolver: resolve.Resolver, _dir: Path, _command: str) -> tuple[int, str]: + # What the real tool does: rewrites the file in place, through the bind mount. + lock.write_text(NPM_LOCK % "4.17.21", encoding="utf-8") + return 0, "" + + result = resolve.upgrade( + resolver=npm, worktree=tmp_path, package="lodash", version="4.17.21", + present=["package.json", "package-lock.json"], run=run, + ) + + assert result.outcome is resolve.Outcome.RESOLVED + assert result.ok is True + + +def test_a_failing_tool_carries_its_own_words(tmp_path: Path) -> None: + """`npm ERR! code ETARGET` is what the operator needs; "resolution failed" is not.""" + (tmp_path / "package-lock.json").write_text(NPM_LOCK % "4.17.11", encoding="utf-8") + npm = resolve.resolver_for("package-lock.json") + assert npm is not None + + result = resolve.upgrade( + resolver=npm, worktree=tmp_path, package="lodash", version="99.0.0", + present=["package.json", "package-lock.json"], + run=lambda *_: (1, "npm ERR! code ETARGET\nnpm ERR! notarget No matching version"), + ) + + assert result.outcome is resolve.Outcome.FAILED + assert "ETARGET" in result.detail + + +def test_nothing_runs_when_a_needed_file_is_absent(tmp_path: Path) -> None: + ran: list[str] = [] + npm = resolve.resolver_for("package-lock.json") + assert npm is not None + + def run(_resolver: resolve.Resolver, _dir: Path, _command: str) -> tuple[int, str]: + ran.append("ran") + return 0, "" + + result = resolve.upgrade( + resolver=npm, worktree=tmp_path, package="lodash", version="4.17.21", + present=["package-lock.json"], run=run, + ) + + assert result.outcome is resolve.Outcome.MISSING + assert "package.json" in result.detail + assert ran == [], "no container may start for a file that is missing on disk" + + +def test_a_resolver_may_rewrite_every_file_it_needs_not_only_the_lock() -> None: + """**Measured, and it surprised the design.** + + `npm install lodash@4.17.21 --package-lock-only` rewrites `package.json` too, moving its range + from `^4.17.11` to `^4.17.21`. Correct for an upgrade, and not what the caller assumed. + + The consequence is item 174's defect one file over: restoring only the lock between candidates + leaves the manifest moved, so the next candidate resolves against a range the previous one + widened — and nothing in the output would say so. + """ + for resolver in resolve.RESOLVERS: + assert resolve.touches(resolver) == resolver.needs + assert resolver.lock in resolve.touches(resolver) + # The manifest is in there too, which is the whole point of this test. + assert len(resolve.touches(resolver)) >= 2 diff --git a/tests/test_the_desk_it_cleared.py b/tests/test_the_desk_it_cleared.py new file mode 100644 index 0000000..654c65b --- /dev/null +++ b/tests/test_the_desk_it_cleared.py @@ -0,0 +1,294 @@ +"""What arrived, and how much of it left with evidence attached. Item 183, DR-0017. + +The decision signed on 2026-08-09 says what the product is measured by, and it is not `Funnel`: +that one's denominator is *attempts that spent an item's one try*, so every question it can answer +has the shape **of the attempts we made, how did they go**. The number signed for has *what arrived* +as its denominator, which is a different question and can produce a much worse answer. + +**Three cases here exist because they are the ones designed to be miscounted**, and each is a way +the product could claim credit it has not earned: an item a person fixed themselves, an item whose +attempt was abandoned by the infrastructure, and a red-lane item nobody was ever allowed to attempt. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +import pytest +from sqlalchemy.orm import Session + +from hullwork import outcomes +from hullwork.models import ( + Attempt, + AttemptOutcome, + AttemptPhase, + Item, + ItemState, + Lane, + Project, +) + + +@pytest.fixture +def project(session: Session) -> Project: + made = Project( + slug="p", forge="forgejo", repo="o/r", + webhook_secret_hash="x", # noqa: S106 + ) + session.add(made) + session.flush() + return made + + +def _item( + session: Session, project: Project, state: ItemState, *, lane: Lane = Lane.GREEN, n: int = 0 +) -> Item: + row = Item( + project_id=project.id, fingerprint=f"fp-{state.value}-{lane.value}-{n}", + title="something", lane=lane, state=state, + ) + session.add(row) + session.flush() + return row + + +def _attempt( + session: Session, + item: Item, + outcome: AttemptOutcome | None, + *, + consumed: bool = True, + rehearsal: bool = False, +) -> Attempt: + row = Attempt( + item_id=item.id, phase_reached=AttemptPhase.PUBLISH, + outcome=outcome, consumed=consumed, rehearsal=rehearsal, + ) + session.add(row) + session.flush() + return row + + +# --- the denominator, which is the whole change ------------------------------------------------ + + +def test_the_denominator_is_what_arrived_and_not_what_was_attempted( + session: Session, project: Project +) -> None: + """`Funnel` cannot express this, and that is why it is being added rather than extended. + + Nine claims arrive; one is attempted. `Funnel` reports a denominator of one and every ratio it + can form is about that one. The question DR-0017 signed for is what happened to the nine. + """ + settled = _item(session, project, ItemState.PR_OPEN) + _attempt(session, settled, AttemptOutcome.PR_OPEN) + for n in range(8): + _item(session, project, ItemState.TRIAGED, n=n) + + desk = outcomes.desk(session) + + assert desk.arrived == 9 + assert outcomes.funnel(session).fair_try == 1, "the old denominator, for contrast" + + +def test_the_buckets_sum_to_what_arrived_and_nothing_is_in_two( + session: Session, project: Project +) -> None: + """A report whose parts do not add up is a report nobody can act on.""" + _attempt(session, _item(session, project, ItemState.PR_OPEN), AttemptOutcome.PR_OPEN) + _attempt( + session, _item(session, project, ItemState.NOT_REPRODUCIBLE, n=1), + AttemptOutcome.NOT_REPRODUCIBLE, + ) + _item(session, project, ItemState.READY, n=2) + _item(session, project, ItemState.HUMAN_ONLY, lane=Lane.RED, n=3) + _attempt(session, _item(session, project, ItemState.IN_PROGRESS, n=4), None) + + desk = outcomes.desk(session) + + assert desk.arrived == 5 + assert ( + desk.left_with_evidence + desk.still_waiting + desk.handed_over + desk.running + == desk.arrived + ) + + +# --- the three that are designed to be miscounted ---------------------------------------------- + + +def test_an_item_a_person_fixed_themselves_is_not_ours( + session: Session, project: Project +) -> None: + """**`done` is reached two ways and there is no state history to tell them apart.** + + A merged pull request and a person closing their own issue both land here. The attempt trail is + what separates them, and counting by state would have the product claiming credit for somebody + else's afternoon — which is the single most dishonest thing this number could do. + """ + theirs = _item(session, project, ItemState.DONE) + ours = _item(session, project, ItemState.DONE, n=1) + _attempt(session, ours, AttemptOutcome.PR_OPEN) + + desk = outcomes.desk(session) + + assert desk.left_with_evidence == 1, "only the one with a verdict behind it" + assert desk.arrived == 2 + del theirs + + +def test_an_abandoned_attempt_left_nothing_on_anybodys_desk( + session: Session, project: Project +) -> None: + """The endpoint was unreachable, the sandbox would not start. No gate ran, so nothing is known. + + `abandoned` does not consume an item's attempt precisely because it says nothing about the + claim, and a number that counted it would be counting the infrastructure's bad days as work. + """ + stalled = _item(session, project, ItemState.READY) + _attempt(session, stalled, AttemptOutcome.ABANDONED, consumed=False) + + desk = outcomes.desk(session) + + assert desk.left_with_evidence == 0 + assert desk.still_waiting == 1, "it is back in the queue, which is where it is" + + +def test_a_red_lane_item_was_added_to_the_desk_rather_than_removed( + session: Session, project: Project +) -> None: + """The row `Funnel` cannot have, and the reason this number is worth building. + + DR-0017's own Context says the first half of the pipeline is a **cost**: a team with a tracker + has more issues than it can serve, and the opening move adds to the pile. An item nobody may + attempt is Hullwork putting work on somebody's desk, and the count has to say so. + """ + _item(session, project, ItemState.HUMAN_ONLY, lane=Lane.RED) + _item(session, project, ItemState.REJECTED, n=1) + + desk = outcomes.desk(session) + + assert desk.handed_over == 2 + assert desk.left_with_evidence == 0 + + +def test_a_rehearsal_is_not_a_desk_anybody_cleared( + session: Session, project: Project +) -> None: + """It publishes nothing, so no forge state and nobody's queue changed. `Funnel`'s rule, held.""" + rehearsed = _item(session, project, ItemState.READY) + _attempt(session, rehearsed, AttemptOutcome.PR_OPEN, rehearsal=True) + + desk = outcomes.desk(session) + + assert desk.left_with_evidence == 0 + assert desk.still_waiting == 1 + + +# --- what it says, which is the part a person reads -------------------------------------------- + + +def test_a_refusal_is_reported_beside_a_change_and_not_inside_a_total( + session: Session, project: Project +) -> None: + """The second consequence of DR-0017, made visible rather than averaged away. + + *"I could not verify this" is a first-class result*, so a total that hides how much of the + number it is would be the one place this product rounds its own honesty off. + """ + _attempt(session, _item(session, project, ItemState.PR_OPEN), AttemptOutcome.PR_OPEN) + _attempt( + session, _item(session, project, ItemState.NOT_REPRODUCIBLE, n=1), + AttemptOutcome.NOT_REPRODUCIBLE, + ) + _attempt(session, _item(session, project, ItemState.FAILED, n=2), AttemptOutcome.FAILED) + + desk = outcomes.desk(session) + + assert desk.left_with_evidence == 3 + assert desk.with_a_change == 1 + assert desk.with_a_refusal == 2 + said = " ".join(outcomes.desk_lines(desk)) + assert "1" in said and "2" in said + assert "refus" in said.lower() + + +def test_an_instance_that_attempted_nothing_says_so_in_words( + session: Session, project: Project +) -> None: + """Zeros read as *nothing happened*; this instance has claims and has cleared none of them. + + That is a different fact and the one worth printing on a first day — it is the state every + instance starts in, and the row of noughts that used to stand for it reads like a failure. + """ + for n in range(4): + _item(session, project, ItemState.TRIAGED, n=n) + + said = " ".join(outcomes.desk_lines(outcomes.desk(session))) + + assert "4" in said + assert "0 " not in said, "it says what is true rather than printing noughts" + + +def test_an_instance_with_nothing_at_all_says_nothing(session: Session) -> None: + """No claims have arrived, so there is no desk to report on. Silence, like `lines`.""" + assert outcomes.desk_lines(outcomes.desk(session)) == [] + + +def test_what_was_added_is_not_phrased_as_an_achievement( + session: Session, project: Project +) -> None: + """It is the row that can embarrass this product, and rounding it into good news is the way it + would stop doing that.""" + _attempt(session, _item(session, project, ItemState.PR_OPEN), AttemptOutcome.PR_OPEN) + _item(session, project, ItemState.HUMAN_ONLY, lane=Lane.RED, n=1) + + said = " ".join(outcomes.desk_lines(outcomes.desk(session))) + + assert "onto" in said or "added" in said or "put on" in said + for congratulation in ("successfully", "great", "achieved", "handled"): + assert congratulation not in said.lower() + + +def test_the_json_carries_the_parts_so_an_operator_computes_their_own_ratio( + session: Session, project: Project +) -> None: + """No percentage, for `Funnel`'s reason: six samples cannot carry that precision.""" + _attempt(session, _item(session, project, ItemState.PR_OPEN), AttemptOutcome.PR_OPEN) + + payload = outcomes.desk(session).as_dict() + + assert payload["arrived"] == 1 + assert payload["left_with_evidence"] == 1 + assert not any("percent" in key or "rate" in key for key in payload) + + +# --- and it has to be on the surface a person opens, not only in a terminal -------------------- + + +def test_the_number_is_on_the_page_and_not_only_in_the_terminal( + session: Session, project: Project +) -> None: + """**The defect item 136 already found on this page once**, reintroduced by item 183 and caught + the same day. + + That item's whole finding was three facts the instance knew and put where nobody reading would + find them. This number was added to `status` and to `--json` and not here — and the interface + design, rewritten the day before under the same decision, says this surface exists to show + **what was verified and what was not**. A count of attempts is not that; this is. + + Ordered before the attempts block for `status`'s reason: the wider denominator first. + """ + from hullwork import page + from hullwork.config import Settings + + _attempt(session, _item(session, project, ItemState.PR_OPEN), AttemptOutcome.PR_OPEN) + _item(session, project, ItemState.HUMAN_ONLY, lane=Lane.RED, n=1) + # Committed rather than flushed: this page reads through its own transaction and cannot see + # uncommitted state, which is invisible in production and is the whole of the difference here. + session.commit() + + body = page.instance(session, Settings(), error_reporting=False) + + assert "went onto your desk rather than off it" in body + assert "2 claim(s) have arrived" in body + assert body.index("how much left your desk") < body.index("What its attempts came to") diff --git a/tests/test_the_documentation_describes_the_published_artefact.py b/tests/test_the_documentation_describes_the_published_artefact.py index 2249307..9482165 100644 --- a/tests/test_the_documentation_describes_the_published_artefact.py +++ b/tests/test_the_documentation_describes_the_published_artefact.py @@ -54,6 +54,7 @@ "docs/install.md", "docs/releasing.md", "docs/status.md", + "docs/what-hullwork-is.md", ) def _withheld() -> set[str]: @@ -297,3 +298,57 @@ def test_every_environment_variable_named_in_the_documents_exists(document: str) } assert not (written - real), f"{document} names settings that do not exist: {written - real}" + + +#: The sentence `docs/what-hullwork-is.md` names as partial, and the reason it is. Item 181. +#: +#: **Matched case-insensitively, and the first version of this was not.** It missed the instance +#: that mattered most: line 1 of `README.md`, the banner's alt text, which begins with a lowercase +#: *from* because it continues a sentence. That is the first line a screen reader announces and what +#: renders when the image does not load — so the guard passed while the very first words of the page +#: still said the thing it exists to keep out. +#: +#: It is not wrong — it is one signal's endpoints, offered where a reader is deciding what this is. +#: The canonical page puts it plainly: *"None of them are wrong. All of them are partial."* A +#: description that answers a narrower question than the product does is a description that selects +#: the wrong readers, and this is the one sentence PyPI shows. +THE_PIPELINE_SENTENCE = "From production errors to reviewable" + +#: The hand-written image sources. Globbed rather than listed, because unlike a document an image is +#: never *deliberately* exempt from saying what the product is — there is no equivalent of the +#: withheld-document case here, so a new one should be bound the moment it is added. +IMAGES = tuple(sorted(str(p.relative_to(ROOT)) for p in (ROOT / "images").glob("*.svg"))) + + +def test_no_published_document_describes_the_product_by_its_plumbing() -> None: + """Item 181's falsifiable gate, the half that can be run. + + **Failing when it was written**, which is what made it a gate rather than a decoration: + `pyproject.toml` carried the sentence as its `description`, so it was the first thing PyPI + showed and the first thing a reader met. + + **The images are checked too, and the first version of this said they could not be.** It called + them "rendered assets" whose text a guard would trip over on a re-export. That was true of + `social-preview.png` and false of every SVG in `images/`: those are hand-written source, with + their own comments explaining their own layout, so a text search over them is exactly as stable + as one over Markdown. The PNG has no text a guard can read — it is covered instead by being + generated from `social-preview.svg` by `scripts/render-social-preview.sh`, both committed. + + Alt text counts, and it is where this guard first failed: line 1 of the README is the banner's + alternative text, which is what a screen reader announces and what renders when the image does + not load. + """ + carrying = [ + name + for name in (*PUBLISHED, "pyproject.toml", *IMAGES) + if THE_PIPELINE_SENTENCE.lower() in _text(name).lower() + # The canonical page *quotes* the sentence in order to name it as partial, which is the + # opposite of using it. Exempted by name rather than by a cleverer matcher: one exemption + # anybody can read beats a rule nobody can. + and name != "docs/what-hullwork-is.md" + ] + + assert carrying == [], ( + f"{carrying} describe Hullwork by one signal's endpoints. " + f"docs/what-hullwork-is.md says what it is; these have not caught up." + ) diff --git a/tests/test_upgrades.py b/tests/test_upgrades.py new file mode 100644 index 0000000..d015d18 --- /dev/null +++ b/tests/test_upgrades.py @@ -0,0 +1,616 @@ +"""Open the verified-green ones. Item 178, DR-0018 step 3. + +The forge is a double, because what is under test is *what may be opened and what it says* — and +the one thing this must never do is open something that was not run. A real forge would prove the +HTTP and not the rule. + +**The claim in a pull request body is the most dangerous sentence this product emits.** It arrives +under Hullwork's own account, in a place a human is meant to trust, next to a diff they are being +asked to merge. So the wording is asserted on the rendered text rather than on the function that +produces it. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from pathlib import Path + +import pytest + +from hullwork import bump, evidence, osv, upgrades +from hullwork.forge import BranchExistsError, ForgePullRequest + +WAS, TO = "2.4.1", "2.10.1" +BASE = "a" * 40 + + +@dataclass +class FakeForge: + """A code forge that records rather than pushes. Item 022's protocol, nothing more.""" + + branches: list[str] = field(default_factory=list) + commits: list[tuple[str, str, tuple[str, ...]]] = field(default_factory=list) + pulls: list[dict[str, object]] = field(default_factory=list) + #: Branches that already exist, so a second run can be expressed. + taken: tuple[str, ...] = () + known: dict[str, str] = field(default_factory=lambda: {"requirements.txt": "blob1"}) + + def default_branch(self, repo: str) -> str: + del repo + return "main" + + def create_branch(self, repo: str, name: str, from_ref: str) -> None: + del repo, from_ref + if name in self.taken: + raise BranchExistsError(name) + self.branches.append(name) + + def file_sha(self, repo: str, path: str, ref: str) -> str | None: + del repo, ref + return self.known.get(path) + + def commit_files( + self, repo: str, branch: str, message: str, changes: object, *, + author: str, email: str, + ) -> str: + del repo, author, email + paths = tuple(sorted(c.path for c in changes)) # type: ignore[attr-defined] + self.commits.append((branch, message, paths)) + return "c" * 40 + + def open_draft_pull_request( + self, repo: str, head: str, base: str, title: str, body: str, + label_ids: list[int] | None = None, + ) -> ForgePullRequest: + del repo, label_ids + self.pulls.append({"head": head, "base": base, "title": title, "body": body}) + return ForgePullRequest( + number=len(self.pulls), title=title, + html_url=f"https://forge/pull/{len(self.pulls)}", draft=True, + ) + + def close(self) -> None: + return None + + +def _answer( + verdict: bump.Verdict, package: str = "jinja2", to: str = TO, detail: str = "" +) -> bump.Answer: + return bump.Answer(verdict, package, WAS, to, detail=detail) + + +def _clean(package: str = "jinja2", to: str = TO) -> bump.Answer: + """A clean answer carrying what the gates actually ran against.""" + return bump.Answer( + bump.Verdict.CLEAN, package, WAS, to, + files={"requirements.txt": f"{package}=={to}\n".encode()}, + runs=bump.Runs( + command="pytest -q", + before_exit=0, after_exit=0, + before_summary="248 passed in 31.02s", + after_summary="248 passed in 30.44s", + ), + ) + + +def _report(package: str, *answers: bump.Answer) -> bump.Report: + return bump.Report(package=package, was=WAS, answers=answers) + + +def _every(text: str, needle: str) -> list[int]: + """Every index `needle` occurs at. One occurrence checked out of two is not a check.""" + found, at = [], text.find(needle) + while at != -1: + found.append(at) + at = text.find(needle, at + 1) + return found + + +ADVISORIES = ( + osv.Advisory( + id="GHSA-462w-v97r-4m45", + summary="Jinja2 sandbox escape via str.format", + fixed=(TO,), + ), +) + + +# --- what may be opened, and what may never --------------------------------------------------- + + +def test_only_the_verified_green_ones_are_eligible() -> None: + """A pull request from Hullwork means *this was run and it passed*. + + The moment it can mean anything else the claim is worth nothing, so every other bucket is + asserted out by name rather than by the absence of a test. + """ + reports = [ + _report("clean-one", _clean("clean-one")), + _report("broken-one", _answer(bump.Verdict.BREAKS, "broken-one", detail="FAILED a")), + _report("red-one", _answer(bump.Verdict.ALREADY_RED, "red-one")), + _report("stuck-one", _answer(bump.Verdict.CANNOT_MOVE, "stuck-one")), + _report("unbuildable", _answer(bump.Verdict.WILL_NOT_INSTALL, "unbuildable")), + ] + + assert [r.package for r in upgrades.eligible(reports)] == ["clean-one"] + + +def test_a_package_that_broke_before_it_passed_is_still_eligible() -> None: + """`verify` tries candidates in order, and the one that settled is the one that gets opened.""" + report = _report( + "jinja2", + _answer(bump.Verdict.BREAKS, "jinja2", to="2.9.0", detail="FAILED a"), + _clean("jinja2"), + ) + + assert upgrades.eligible([report]) == [report] + settled = report.settled + assert settled is not None and settled.to == TO + + +def test_a_clean_answer_with_nothing_to_commit_is_refused_rather_than_opened() -> None: + """The files are the diff. Without them there is a body making a claim about an empty commit. + + This is the shape item 045 is about, one product over: what was tested and what is published + have to be the same tree, and a pull request that carries no tree at all cannot be either. + """ + barren = _report("jinja2", bump.Answer(bump.Verdict.CLEAN, "jinja2", WAS, TO)) + + assert upgrades.eligible([barren]) == [] + + +# --- the opening ------------------------------------------------------------------------------ + + +def test_one_pull_request_per_package_and_a_run_of_three_opens_three() -> None: + """Never a batch: a grouped upgrade that breaks cannot be bisected without undoing our work.""" + forge = FakeForge() + reports = [_report(name, _clean(name)) for name in ("aaa", "bbb", "ccc")] + + opened = upgrades.open_them( + forge, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + + assert len(opened) == 3 + assert len(forge.pulls) == 3 + assert len({p["head"] for p in forge.pulls}) == 3, "three branches, not one" + for commit in forge.commits: + assert commit[2] == ("requirements.txt",) + + +def test_the_branch_names_the_upgrade_so_a_second_run_opens_nothing() -> None: + """No database in this path, so the branch name *is* the record of what was opened. + + A second pass over an unchanged repository asks for a branch that exists and is told so by the + forge, which is the same answer `work.publish` already relies on. Nothing is opened twice and + nothing has to be remembered between runs. + """ + reports = [_report("jinja2", _clean())] + first = FakeForge() + upgrades.open_them( + first, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + branch = first.branches[0] + + again = FakeForge(taken=(branch,)) + opened = upgrades.open_them( + again, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + + assert opened == [] + assert again.pulls == [] + assert branch == upgrades.branch_for("jinja2", WAS, TO) + + +def test_the_branch_survives_a_package_name_a_ref_cannot_carry() -> None: + """`@scope/pkg` is an ordinary npm name and not an ordinary git ref. + + **The character that matters is `/`, and the first version of this test did not check it.** + Verified by neutering the sanitiser: `@babel/core` came through whole, the name still began + with the prefix and still contained none of the characters git documents as forbidden, and this + passed. What it produces is `hullwork/deps/@babel/core-7.0.0-7.24.0` — a fourth path level, + which git will happily create and which collides with any branch named `hullwork/deps/@babel`. + + So the assertion is on the shape: three segments, and the last one is the whole upgrade. + """ + name = upgrades.branch_for("@babel/core", "7.0.0", "7.24.0") + + assert name.split("/")[:2] == ["hullwork", "deps"] + assert len(name.split("/")) == 3, f"the package invented a path level: {name}" + assert " " not in name + for forbidden in ("~", "^", ":", "?", "*", "[", "\\", "@{", ".."): + assert forbidden not in name + # And it still says what it is about, which is the other half of the name's job. + assert "babel" in name and "7.24.0" in name + + +def test_two_upgrades_of_one_package_are_two_branches() -> None: + """Otherwise next month's upgrade collides with this month's and opens nothing, silently. + + The failure mode is the bad one: the forge answers `BranchExistsError`, this treats it as + *already opened*, and a real upgrade never reaches anybody. Verified by dropping the versions + from the name, at which point the test above still passed because it computes what it expects + with the same function. + """ + first = upgrades.branch_for("jinja2", "2.4.1", "2.10.1") + second = upgrades.branch_for("jinja2", "2.10.1", "3.1.4") + + assert first != second + + +def test_a_branch_that_exists_and_a_forge_that_refused_are_told_apart( + caplog: pytest.LogCaptureFixture +) -> None: + """Both open nothing, and the operator is told they are different things. + + The terminal says *already open from an earlier run, or refused by the forge — the log says + which*, so the log has to actually say which. Nothing else can tell these apart: from the + caller's side both are an empty list. + """ + from hullwork.forge import ForgeError + + reports = [_report("jinja2", _clean())] + taken = FakeForge(taken=(upgrades.branch_for("jinja2", WAS, TO),)) + with caplog.at_level("INFO", logger="hullwork.upgrades"): + upgrades.open_them( + taken, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + already = caplog.text + caplog.clear() + + refusing = FakeForge() + + def refuse(repo: str, name: str, from_ref: str) -> None: + raise ForgeError("the forge said no") + + refusing.create_branch = refuse # type: ignore[method-assign] + with caplog.at_level("INFO", logger="hullwork.upgrades"): + upgrades.open_them( + refusing, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + + assert "already opened" in already + assert "already opened" not in caplog.text + assert "could not branch" in caplog.text + + +def test_the_pull_request_is_rooted_at_the_commit_the_gates_ran_against() -> None: + """Not at wherever the default branch points now — that is a tree nobody tested.""" + forge = FakeForge() + seen: list[str] = [] + original = forge.create_branch + + def watched(repo: str, name: str, from_ref: str) -> None: + seen.append(from_ref) + original(repo, name, from_ref) + + forge.create_branch = watched # type: ignore[method-assign] + upgrades.open_them( + forge, repo="o/r", reports=[_report("jinja2", _clean())], + advisories={}, base_sha=BASE, permitted=True, + ) + + assert seen == [BASE] + + +def test_a_forge_that_refuses_one_does_not_cost_the_others() -> None: + """A queue of five with one bad name is four pull requests, not a traceback.""" + from hullwork.forge import ForgeError + + forge = FakeForge() + calls = {"n": 0} + + def sometimes(repo: str, name: str, from_ref: str) -> None: + calls["n"] += 1 + if calls["n"] == 1: + raise ForgeError("the forge said no") + forge.branches.append(name) + + forge.create_branch = sometimes # type: ignore[method-assign] + reports = [_report(name, _clean(name)) for name in ("aaa", "bbb", "ccc")] + + opened = upgrades.open_them( + forge, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=True + ) + + assert len(opened) == 2 + + +# --- the credential, which is the whole reason this item is amber ----------------------------- + + +def test_open_refuses_before_anything_is_run_when_there_is_no_credential(tmp_path: Path) -> None: + """The refusal has to arrive before two container builds, not after them. + + Same lesson as `_manifest_for_verify` and with more at stake: this is the one flag in `deps` + that writes to somebody's repository, and a refusal that lands after the work it invalidates is + a refusal printed underneath its own contradiction. + """ + from hullwork.cli import CommandError, _forge_for_opening + from hullwork.config import Settings + + with pytest.raises(CommandError) as refused: + _forge_for_opening(Settings(), tmp_path) + + assert "HULLWORK_FORGE_CODE_TOKEN" in str(refused.value) + # And it names the thing that needs nothing, because that is the honest way to see what this + # would have opened. + assert "--verify" in str(refused.value) + + +def test_verify_alone_never_reaches_for_a_credential( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The claim `deps` is sold on: a stranger runs it in the first minute with no account. + + Asserted by making the credential lookup explode. `--verify` must reach the lock files without + ever touching it, so this fails loudly if the flag ever stops gating that call. + """ + import argparse + import io + import subprocess + + from hullwork import cli + from hullwork.config import Settings + + subprocess.run(["git", "init", "-q", str(tmp_path)], check=True) # noqa: S603, S607 + (tmp_path / "hullwork.yml").write_text( + "project: p\ngit: {provider: forgejo, repo: o/r}\n" + "autofix: {agent: none, gates: [tests, human-merge]}\n" + 'tests: "pytest"\ntest_path: tests\n' + "runtime: {base: python-3.12, install: none, dependencies: []}\n" + ) + + def explode(*_a: object, **_k: object) -> object: + raise AssertionError("--verify asked for a credential") + + monkeypatch.setattr(cli, "_forge_for_opening", explode) + + with pytest.raises(cli.CommandError) as refused: + cli._cmd_deps( + argparse.Namespace( + checkout=str(tmp_path), verify=True, fix=False, open=False, into=str(tmp_path) + ), + Settings(), + io.StringIO(), + ) + + # It got as far as looking for lock files, which is past every point a credential could have + # been wanted. + assert "no lock file" in str(refused.value) + + +def test_open_refuses_a_checkout_with_no_remote_to_name_it( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A coordinate cannot be guessed from a directory name, and a wrong one opens elsewhere.""" + from hullwork import cli + from hullwork.config import Settings + + monkeypatch.setattr(cli, "make_code_forge", lambda _s: FakeForge()) + + with pytest.raises(cli.CommandError) as refused: + cli._forge_for_opening(Settings(), tmp_path) + + assert "origin" in str(refused.value) + + +# --- what the body says, which is the part a person acts on ----------------------------------- + + +def _body(advisories: tuple[osv.Advisory, ...] = ADVISORIES) -> str: + return evidence.dependency_pull_request_body(_clean(), advisories) + + +def test_the_claim_is_dr_0016s_wording_and_never_the_word_safe() -> None: + """The one sentence that must not drift, asserted on the rendered text. + + A green pull request is the easiest place in this product to overclaim: the reviewer is being + asked to merge, and *safe* is the word they will read into anything vaguer. + """ + body = _body() + + assert "your suite passed before this change and passes after it" in body + # The word is allowed to appear — twice, in fact — and only ever inside a denial. Enumerating + # the two permitted sentences would be a test that has to be edited whenever either is + # reworded, which is how a guard comes to be maintained into uselessness. The rule is that + # nothing here *asserts* safety, so that is what is asserted. + for index in _every(body, "safe"): + before = body[max(0, index - 60):index] + assert "not" in before, f"'safe' claimed rather than denied, after: …{before}" + + +def test_the_claim_is_the_same_function_the_terminal_prints() -> None: + """The page and the pull request cannot come to disagree if there is one author. + + Not a second rendering that happens to match today: the body quotes `Answer.says`, so a change + to the wording changes both or neither. + """ + answer = _clean() + + assert answer.says in evidence.dependency_pull_request_body(answer, ADVISORIES) + + +def test_the_body_carries_the_advisory_its_id_and_where_to_read_it() -> None: + body = _body() + + assert "GHSA-462w-v97r-4m45" in body + assert "https://osv.dev/vulnerability/GHSA-462w-v97r-4m45" in body + assert "sandbox escape" in body + + +def test_the_body_carries_both_runs_with_the_suites_own_summary_lines() -> None: + """The command and the exit codes, and what the runner itself said. Not our paraphrase.""" + body = _body() + + assert "pytest -q" in body + assert "248 passed in 31.02s" in body + assert "248 passed in 30.44s" in body + + +def test_the_body_says_what_was_measured_was_your_suite() -> None: + """The sentence that keeps it honest, and the reason a green verdict is not a guarantee.""" + body = _body() + + assert "never exercise this dependency" in body + assert "Nothing here inspected the change itself" in body + + +def test_the_caveat_has_one_author() -> None: + """Item 098's rule, made checkable because this document broke it on its first reading. + + `Answer.says` ends by saying what was measured; the paragraph beneath it used to say the same + thing again in different words, and every assertion in this file passed. What catches that is + not a better assertion about content — it is counting. The phrase belongs to one sentence, and + a second author of it is a body that reads like a program that has lost its place. + """ + body = _body() + + assert len(_every(body, "what was measured")) == 1 + assert len(_every(body, "fixes anything")) == 1 + + +def test_an_upgrade_with_no_advisory_is_still_a_body_a_person_can_read() -> None: + """Not every upgrade worth taking has something published against the version it replaces.""" + body = evidence.dependency_pull_request_body(_clean(), ()) + + assert "your suite passed before this change and passes after it" in body + assert "GHSA" not in body + + +def test_a_suite_that_printed_a_secret_does_not_print_it_here() -> None: + """This text leaves the instance under our own account. Item 027's rule, one caller later.""" + answer = bump.Answer( + bump.Verdict.CLEAN, "jinja2", WAS, TO, + files={"requirements.txt": b"jinja2==2.10.1\n"}, + runs=bump.Runs( + command="pytest -q", + before_exit=0, after_exit=0, + before_summary="ok", + after_summary="248 passed; token=ghp_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", + ), + ) + + body = evidence.dependency_pull_request_body(answer, ()) + + assert "ghp_AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" not in body + + +def test_the_body_a_pull_request_gets_is_the_one_the_opener_sends() -> None: + """Rendered once, by the module that owns what a reviewer reads, and handed over whole.""" + forge = FakeForge() + + upgrades.open_them( + forge, repo="o/r", + reports=[_report("jinja2", _clean())], + advisories={"jinja2": ADVISORIES}, + base_sha=BASE, + permitted=True, + ) + + body = str(forge.pulls[0]["body"]) + assert "your suite passed before this change and passes after it" in body + assert "GHSA-462w-v97r-4m45" in body + assert "jinja2" in str(forge.pulls[0]["title"]) + + +def test_what_is_committed_is_what_the_gates_ran_against() -> None: + """Carried on the answer, never re-derived. + + Re-applying the upgrade to work out the diff would run the resolver a second time, and a lock + regenerated twice can differ — a version published in between, a different ordering. Publishing + files that are not the ones the suite passed against is the defect item 045 is named after. + """ + forge = FakeForge() + answer = _clean() + + upgrades.open_them( + forge, repo="o/r", reports=[_report("jinja2", answer)], + advisories={}, base_sha=BASE, permitted=True, + ) + + assert forge.commits[0][2] == ("requirements.txt",) + + +@pytest.mark.parametrize("draft", [True, False]) +def test_it_is_opened_as_a_draft_and_says_so_when_the_forge_disagrees(draft: bool) -> None: + """Constitution §1: nothing merges by itself, and a forge that un-drafts is a finding.""" + forge = FakeForge() + original = forge.open_draft_pull_request + + def answered(*args: object, **kwargs: object) -> ForgePullRequest: + made = original(*args, **kwargs) # type: ignore[arg-type] + return ForgePullRequest( + number=made.number, title=made.title, html_url=made.html_url, draft=draft + ) + + forge.open_draft_pull_request = answered # type: ignore[method-assign] + opened = upgrades.open_them( + forge, repo="o/r", reports=[_report("jinja2", _clean())], + advisories={}, base_sha=BASE, permitted=True, + ) + + assert len(opened) == 1 + + +# --- the first thing a project can refuse (item 187, DR-0019) ---------------------------------- + + +def test_a_project_that_has_not_permitted_it_gets_nothing_opened() -> None: + """**Having a capability is not consenting to the feature it enables.** + + Until DR-0019, declaring an installer and a lock file *was* agreeing to pull requests in your + repository — nobody said so. This is the sentence this product already applies to lanes, *a + policy nobody has read is a policy nobody has agreed to*, pointed at itself. + + Everything that would have been opened is verified green: the refusal is about consent and not + about the evidence, which is why the verification above it still ran. + """ + forge = FakeForge() + reports = [_report(name, _clean(name)) for name in ("aaa", "bbb")] + + opened = upgrades.open_them( + forge, repo="o/r", reports=reports, advisories={}, base_sha=BASE, permitted=False + ) + + assert opened == [] + assert forge.branches == [], "a branch is already a write to somebody's repository" + assert forge.pulls == [] + # And the eligible ones were eligible: this is consent, not a verdict. + assert len(upgrades.eligible(reports)) == 2 + + +def test_forgetting_the_permission_is_a_crash_and_never_an_open() -> None: + """Item 017's rule, and the reason this is a parameter rather than a check at the call site. + + *A guardrail that depends on every caller remembering it is not a guardrail.* This is the only + function in the product that opens anything, so a caller who forgets gets a `TypeError` — not + an unguarded pull request in somebody's repository. + """ + with pytest.raises(TypeError, match="permitted"): + upgrades.open_them( # type: ignore[call-arg] + FakeForge(), repo="o/r", reports=[_report("jinja2", _clean())], + advisories={}, base_sha=BASE, + ) + + +def test_the_permission_is_false_unless_a_project_wrote_it() -> None: + """A permission that arrives switched on is not a permission. + + Every other default in `autofix` is the refusing one — `agent: none`, `unmatched: human` — and + this one joins them. + """ + from hullwork.manifest import parse_manifest + + silent = parse_manifest( + "project: p\ngit: {provider: forgejo, repo: o/r}\ntests: pytest\n" + ) + asked = parse_manifest( + "project: p\ngit: {provider: forgejo, repo: o/r}\ntests: pytest\n" + "autofix: {open_upgrades: true}\n" + ) + + assert silent.autofix.open_upgrades is False + assert asked.autofix.open_upgrades is True diff --git a/tests/test_webhooks.py b/tests/test_webhooks.py index e8aac71..e20a859 100644 --- a/tests/test_webhooks.py +++ b/tests/test_webhooks.py @@ -19,7 +19,7 @@ from hullwork.config import get_settings from hullwork.db import make_engine, make_session_factory from hullwork.main import app -from hullwork.models import Delivery, Item, Project +from hullwork.models import Delivery, Item, Lane, Project from hullwork.security import generate_token, hash_token from hullwork.webhooks import MAX_BODY_BYTES, MAX_JSON_DEPTH, json_depth @@ -57,6 +57,15 @@ } +#: **The recorded envelope, not one written for this test** (item 191). It carries its own +#: provenance in a `_comment`: read from Sentry's `app_platform_event.py` on 2026-07-26, and it is +#: the same file `test_normalise` parses. Until this item it had only ever been handed to +#: `sentry.parse` directly — never posted at the door, which is the half that was never measured. +SENTRY_PAYLOAD = json.loads( + (ROOT / "tests/fixtures/webhook-sentry-event-alert.json").read_text(encoding="utf-8") +) + + @pytest.fixture def token() -> str: return generate_token() @@ -151,15 +160,79 @@ def test_an_unknown_provider_is_refused(client: TestClient, token: str) -> None: assert _post(client, token, provider="rollbar").status_code == 404 -def test_the_sentry_route_says_it_is_not_enabled_rather_than_failing_oddly( +def test_a_sentry_delivery_with_a_valid_token_becomes_an_item( + client: TestClient, token: str, tmp_path: Path +) -> None: + """**The route this refused for wanting a better guarantee** (item 189, operator's choice). + + It answered `501` because verifying Sentry's HMAC means holding its client secret reversibly — + correct, and it was written as *reversible secret or nothing*. There is a third option: + **GlitchTip cannot sign at all**, so the token in the URL has been the credential since M1, and + Sentry gets that same credential verified the same way. The route was declining to offer a + guarantee better than the one the only working provider gets. + """ + response = _post(client, token, payload=SENTRY_PAYLOAD, provider="sentry") + + assert response.status_code == 200 + assert response.json()["status"] == "accepted" + with _session(tmp_path) as db: + assert db.query(Delivery).count() == 1 + item = db.query(Item).one() + assert item.title.startswith("TypeError") + # **The fields that decide something**, not only the one that displays. + # + # This delivery is red, and the interesting part is *why*: the recorded culprit is + # `app.views.checkout in process_payment`, and this manifest declares `red: ["payment"]`. + # GlitchTip's fixture describes a different error (`app.cart in total`) and comes out green, + # so the two are not comparable and both are right. + # + # **The lane alone cannot say that**, which is the trap this assertion was nearly written + # into: red is *also* what an item whose culprit never arrived would get, because anything + # unclassified defaults to red. So the reason is what is asserted — it names the rule that + # fired, and a lost culprit would give the default's wording instead. + assert item.lane is Lane.RED + assert item.lane_reason and "payment" in item.lane_reason, ( + f"the culprit did not reach triage; the lane defaulted: {item.lane_reason!r}" + ) + assert item.permalink and "sentry.io" in item.permalink + + +def test_a_wrong_token_looks_the_same_on_both_routes( + client: TestClient, token: str +) -> None: + """**Status and body**, because a difference between them confirms a provider by probing. + + Item 122 already established that shape for the page: a wrong token's `404` is byte-identical + to an unknown path's. The same reasoning applies to two providers on one door — if Sentry's + refusal read differently from GlitchTip's, the door would answer a question nobody is entitled + to ask. + """ + wrong = "b" * 43 + + glitchtip = _post(client, wrong) + sentry = _post(client, wrong, payload=SENTRY_PAYLOAD, provider="sentry") + + assert glitchtip.status_code == sentry.status_code + assert glitchtip.text == sentry.text + + +def test_the_signature_header_is_never_read_as_authentication( client: TestClient, token: str ) -> None: - """Its adapter exists, but verifying its HMAC needs the secret in reversible form — a storage - decision not yet made. Saying so beats a confusing 401.""" - response = _post(client, token, provider="sentry") + """**(B) is weaker than (C) and must not pretend otherwise.** - assert response.status_code == 501 - assert "GlitchTip" in response.text + Sentry does sign, and this ignores it: what authenticates is the token in the path. A delivery + carrying a plainly bogus signature is accepted, because the signature is not what was checked — + and a reader of this test learns that the HMAC is unverified rather than inferring it from a + silence. + """ + response = client.post( + f"/webhooks/sentry/{SLUG}/{token}", + json=SENTRY_PAYLOAD, + headers={"sentry-hook-signature": "0" * 64}, + ) + + assert response.status_code == 200 # --- limits ---------------------------------------------------------------------------------- diff --git a/tests/test_what_it_can_do_for_you.py b/tests/test_what_it_can_do_for_you.py new file mode 100644 index 0000000..a93b10d --- /dev/null +++ b/tests/test_what_it_can_do_for_you.py @@ -0,0 +1,323 @@ +"""What this can do for your project, and what it cannot. Item 186. + +The operator's framing: **Hullwork is modular — a developer turns features on and off, and each +feature has its limitations.** The first half already existed in `hullwork.yml`; the second half +existed nowhere, so every limitation was found by walking into it. + +**The tests that matter here are about the two halves not blending.** A feature's *needs* are +checkable and either met or not; its *limits* are true whatever the answer, and both have to be +printed either way — because a limit you meet after adopting something is a limit you found the +expensive way. + +Every test here was verified by reintroducing the defect it covers. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from hullwork import features +from hullwork.manifest import Manifest, parse_manifest + +MANIFEST = """ +project: p +git: {provider: github, repo: o/r} +tests: "pytest" +test_path: tests +runtime: {base: python-3.12, install: pip, dependencies: [requirements.txt]} +""" + +#: The shape `propose` writes when a project's CI hides its install step, and the shape a project +#: that brings its own image has. Item 182 measured a **false verdict** produced under it. +OWN_IMAGE = """ +project: p +git: {provider: github, repo: o/r} +tests: "pytest" +test_path: tests +runtime: {base: python-3.12, install: none, dependencies: []} +""" + + +def _checkout( + manifest_text: str | None = MANIFEST, + *, + paths: tuple[str, ...] = ("requirements.txt", "src/app.py"), + configured: tuple[str, ...] = (), +) -> features.Checkout: + manifest: Manifest | None = parse_manifest(manifest_text) if manifest_text else None + return features.Checkout( + paths=paths, manifest=manifest, configured=frozenset(configured) + ) + + +def _named(answers: list[features.Answer], name: str) -> features.Answer: + return next(a for a in answers if a.feature.name == name) + + +# --- the half that had no home anywhere -------------------------------------------------------- + + +def test_every_feature_declares_what_it_cannot_do() -> None: + """**Never empty**, and this is the structural half of the operator's framing. + + A feature with no limits reads as one that has none. Until this module that was true of all of + them, and it was false of every single one — the limits existed and lived in docstrings, in + decision records, and in what a person found out by running it. + """ + for feature in features.FEATURES: + assert feature.limits, f"{feature.name} declares no limits, which is never true" + assert feature.needs, f"{feature.name} declares no needs" + for need in feature.needs: + assert need.fix, f"{feature.name}: a need with no way to satisfy it is a dead end" + + +def test_the_limits_are_printed_whether_or_not_the_feature_is_available() -> None: + """The whole point. A feature you *can* have is the one where nobody thinks to look.""" + available = features.examine(_checkout()) + unavailable = features.examine(_checkout(OWN_IMAGE)) + + said = " ".join(features.lines(available)) + also = " ".join(features.lines(unavailable)) + + for text in (said, also): + assert "What is measured is **your suite**" in text + assert "the image has to be refreshed" in text + + +# --- what it says about the case that produced a false verdict --------------------------------- + + +def test_a_project_that_brings_its_own_image_is_told_verification_cannot_serve_it() -> None: + """Item 182's finding, answered before anybody pays for it. + + With `install: none` the image is `runtime.base` exactly as it comes and nothing is installed + from a lock file, so rewriting a pinned version changes nothing the suite would run against. + Measured against a base carrying `jinja2 3.0.0` and a checkout pinning `2.4.1`: a verdict + reading *your suite passed before this change and passes after it*, about a version that was + never installed. + + `install: none` is the **default**, and DR-0007 makes *the project brings its own image* the + primary path — so this is most projects, and the answer has to arrive before the containers do. + """ + answer = _named(features.examine(_checkout(OWN_IMAGE)), "dependency verification") + + assert not answer.available + missing = " ".join(need.what for need in answer.missing) + assert "installer that reads the file your versions are pinned in" in missing + fix = " ".join(need.fix for need in answer.missing) + assert "runtime.install" in fix and "runtime.dependencies" in fix + + +def test_a_project_whose_image_hullwork_builds_is_told_it_can() -> None: + """The other side of the same answer: a report that says no to everything is not a report.""" + answer = _named(features.examine(_checkout()), "dependency verification") + + assert answer.available + assert answer.missing == () + + +def test_a_checkout_with_no_manifest_is_told_which_command_writes_one() -> None: + """The commonest first contact there is, and the one where a refusal has to end in a verb.""" + answer = _named(features.examine(_checkout(None)), "dependency verification") + + assert not answer.available + assert any("propose" in need.fix for need in answer.missing) + # And the report still works: a lock file is a fact about the checkout, not about the manifest. + assert _named(features.examine(_checkout(None)), "dependency report").available + + +def test_every_unmet_need_is_listed_and_not_just_the_first() -> None: + """A reader who fixes one thing and runs this again to find a second is doing the work this + command exists to save them.""" + answer = _named(features.examine(_checkout(None, paths=())), "dependency verification") + + assert len(answer.missing) >= 2 + + +# --- the rules it runs under ------------------------------------------------------------------- + + +def test_a_credential_is_read_for_whether_it_is_set_and_never_for_its_value() -> None: + """This is the command somebody runs before trusting the product with anything. + + It says *needs a model credential, and none is configured* while holding none — which is what + lets it be run by somebody who has configured nothing at all, and what stops it becoming a + place a secret can be printed. + """ + with_key = features.examine(_checkout(configured=(features.MODEL_KEY,))) + without = features.examine(_checkout()) + + assert _named(with_key, "fixing an upgrade that breaks your suite").missing != () + listed = " ".join( + need.what for need in _named(without, "fixing an upgrade that breaks your suite").missing + ) + assert "model credential" in listed + # The whole surface, checked for a value that was never handed to it. + assert "sk-" not in " ".join(features.lines(with_key)) + + +def test_nothing_it_reads_requires_a_daemon_a_socket_or_a_forge( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Asserted by making every route out explode. + + The precedent is `projects lanes --checkout .`, which prints this instance's policy against a + tree with no credential of any kind — *"a policy nobody has read is a policy nobody has agreed + to"*. A capability report that quietly opened a socket would be a different command wearing + this one's promise. + """ + import socket + import subprocess + + def forbidden(*_a: object, **_k: object) -> None: + raise AssertionError("features opened something it may not") + + monkeypatch.setattr(socket, "socket", forbidden) + monkeypatch.setattr(subprocess, "run", forbidden) + monkeypatch.setattr(subprocess, "Popen", forbidden) + + said = features.lines(features.examine(_checkout())) + + assert said + + +def test_the_features_a_checkout_cannot_answer_are_named_rather_than_guessed() -> None: + """Whether a forge answers, whether a tracker is reachable, whether a dispatcher holds the + lease — a checkout cannot know any of it, and a report that pretended to would be worse than + one that says whose question it is.""" + assert features.INSTANCE_SHAPED + reported = {feature.name for feature in features.FEATURES} + for named in features.INSTANCE_SHAPED: + assert named not in reported, f"{named} is claimed here and cannot be answered here" + + +def test_it_writes_nothing_at_all(tmp_path: Path) -> None: + """It is a reading of what you already have. A report that edited a manifest to make itself + truer would be the worst possible version of this command.""" + before = sorted(p.name for p in tmp_path.iterdir()) + + features.lines(features.examine(_checkout())) + + assert sorted(p.name for p in tmp_path.iterdir()) == before + + +# --- the third answer (item 187, DR-0019) ------------------------------------------------------ + + +def test_a_feature_a_project_has_not_permitted_reads_as_a_decision() -> None: + """**Not `no`.** That would report a choice somebody made as a part that is missing, which is + the one way this report could insult its reader. + + Everything the feature needs is here — the lock file, the manifest, the installer, the + credential, the remote. What is absent is the project's yes. + """ + everything = _checkout(configured=(features.CODE_TOKEN, "origin")) + + answer = _named(features.examine(everything), "opening the upgrades that pass") + + assert answer.available, "the capability is all there" + assert not answer.permitted + said = " ".join(features.lines([answer])) + assert "[not permitted here]" in said + assert "this project has not permitted it" in said + assert "open_upgrades: true" in said + + +def test_the_permission_granted_reads_as_yes() -> None: + """The other side: a project that said yes gets the plain answer and no lecture.""" + permitted = features.Checkout( + paths=("requirements.txt",), + manifest=parse_manifest(MANIFEST.rstrip() + "\nautofix: {open_upgrades: true}\n"), + configured=frozenset((features.CODE_TOKEN, "origin")), + ) + + answer = _named(features.examine(permitted), "opening the upgrades that pass") + + assert answer.available and answer.permitted + said = " ".join(features.lines([answer])) + assert "[yes]" in said + assert "not permitted" not in said + + +def test_a_missing_capability_still_reads_as_no_even_when_unpermitted() -> None: + """Two different absences, and the more fundamental one wins the headline. + + A project with no credential *and* no permission is told `no` — because *not permitted here* + would suggest that granting it would be enough, and it would not. + """ + neither = _checkout() + + answer = _named(features.examine(neither), "opening the upgrades that pass") + + assert not answer.available + assert "[no]" in " ".join(features.lines([answer])) + + +def test_only_what_writes_to_a_repository_asks_for_permission() -> None: + """DR-0019's bound, asserted structurally so the switchboard cannot grow quietly. + + *Could a project have the capability, understand the feature, and rationally not want it?* If + wanting is implied by having, there is no switch — and every feature that writes nothing to + somebody's repository is in that class. A second `permits` entry appearing here is Renovate's + complaint nº2 arriving as a feature, which DR-0018 refuses by name. + """ + asking = [f.name for f in features.FEATURES if f.permits] + + assert asking == ["opening the upgrades that pass"], ( + "a new permission was added; DR-0019's rule has to be applied to it in writing first" + ) + + +# --- the two paths compose (item 188, DR-0007) ------------------------------------------------- + +#: A project's own image, plus the one line that refreshes its dependencies on top of it. Legal +#: since DR-0007 was built and written down nowhere until item 188. +OWN_IMAGE_REFRESHED = """ +project: p +git: {provider: github, repo: o/r} +tests: "pytest" +test_path: tests +runtime: + base: ghcr.io/acme/ci-base:2026.7 + install: "pip install -r requirements.txt" + dependencies: [requirements.txt] +""" + + +def test_an_image_hullwork_did_not_build_is_served_when_it_is_refreshed() -> None: + """**The operator's directive answered without adding a stack** (item 188). + + Item 182 measured that verification served only projects whose image Hullwork builds — DR-0007's + path (A), which that decision demoted to *sugar*. Going to design a table of environment + strategies found none was needed: `base` takes any image and `install` takes the project's own + command, so *your image plus one line* was always legal. + + Measured against a real daemon on 2026-08-09, asking both containers rather than reading the + report: `before -> jinja2 3.0.0` (what the project's own image carries) and + `after -> jinja2 3.1.6` (the upgrade, actually installed). + """ + answer = _named(features.examine(_checkout(OWN_IMAGE_REFRESHED)), "dependency verification") + + assert answer.available, "a base Hullwork did not build is still measurable when refreshed" + + +def test_the_limit_no_longer_says_a_project_with_its_own_image_is_not_served() -> None: + """It said exactly that, and item 188 measured it false. + + A limit that overstates is worse than none: this one would have sent every project on DR-0007's + **primary** path away from a feature that serves them, and it would have read as honesty. + """ + said = " ".join(features.lines(features.examine(_checkout(OWN_IMAGE_REFRESHED)))) + + assert "not served" not in said + assert "does not mean Hullwork must build your image" in said + + +def test_the_remedy_tells_them_to_keep_their_own_base() -> None: + """The difference between one line and a rebuild from scratch, said where they will read it.""" + answer = _named(features.examine(_checkout(OWN_IMAGE)), "dependency verification") + + fix = " ".join(need.fix for need in answer.missing) + assert "Keep your own image" in fix diff --git a/tests/test_what_the_closed_items_owed.py b/tests/test_what_the_closed_items_owed.py index 1bad800..dba2a18 100644 --- a/tests/test_what_the_closed_items_owed.py +++ b/tests/test_what_the_closed_items_owed.py @@ -364,6 +364,10 @@ def test_the_doctor_does_not_claim_a_clean_host_it_could_not_look_at() -> None: #: `docker compose build` had made it in some earlier session and nothing recorded the dependency, #: so the test failed rather than skipped, with a message about the gateway rather than the image. #: +#: **And the instruction went stale under it** (item 191, 2026-08-09). `docker-compose.yml` now pins +#: a published image and has no build stage — its own comment says to *add* one — so +#: `docker compose build` exits 0 and produces nothing. Measured while chasing this exact skip. +#: #: A skip with a reason, not a build: building it here would put minutes into an unrelated test run #: and hide the same gap. Saying what is missing is the honest answer, and the error that exposed #: this now prints Docker's own words, which is how it was diagnosed in one read. @@ -385,7 +389,7 @@ def test_the_doctor_does_not_claim_a_clean_host_it_could_not_look_at() -> None: capture_output=True, timeout=30, check=False, ).returncode != 0, - reason="needs the hullwork:dev image; build it with `docker compose build`", + reason="needs the hullwork:dev image; build it with `docker build --tag hullwork:dev .`", )