fix: second audit pass — 16 more bugs found using applyr on real data - #50
Merged
Conversation
`plan` classified priority against absolute thresholds (200/100/40) while `gaps` classified it relative to the worst gap. On a real 207-offer database the weakest topic already scored 415, so `plan` called all six topics CRITICAL while `gaps` spread the same six across HIGH/MEDIUM/LOW. Two commands, one dataset, contradictory answers — and a plan that says "prioritize everything" says nothing. The earlier fix to gap priority changed `cmd_gaps` and left its sibling. - `cmd_plan` reuses `_gap_priority`, so both commands share one vocabulary and one scale - Rank by `total_gap` directly instead of recomputing frequency x avg_gap, which was the same number with a rounding step in the middle - Drop the priority ladder duplicated across the JSON and human branches - Remove PRIORITY_CRITICAL_SCORE / PRIORITY_HIGH_SCORE / PRIORITY_MEDIUM_SCORE, now unused 3 tests, one of which asserts the two commands return identical priorities. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule was "lowest partial, or else highest missing", which got it wrong twice over. Preferring the partial bucket meant a topic scored 30 and printed under "Missing" two lines above was passed over in favour of one scored 50 — the summary contradicted the breakdown it sits under. On a real offer it reported English (50) as the biggest weakness while Experience (30) sat in the Missing section directly above. And the missing branch sorted descending, so a profile weak across the board was told its *least* bad shortfall was the problem. It is now simply the lowest-scoring topic that is not already strong. A test pinned the old behaviour — asserting Experience (60) over Projects (30, "No relevant projects") — so it was asserting the bug. Corrected, plus two new cases: the worst of several missing topics wins, and a partial can still be the weakest when nothing is missing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cv keywords` matched the offer's required keywords against the whole generated file. That file opens with YAML frontmatter carrying the offer's own `tech_stack` and `summary` — so the offer's keywords were matched against a verbatim copy of themselves. Every keyword hit. Every generated CV scored 100% and reported KEYWORD STATUS: STRONG, whatever it actually said. Verified on a real CV that mentions neither AWS nor Redux nor Webpack: before, 11/11 matched and STRONG; after, 8/11 and ADEQUATE, with those three correctly listed as missing. The metric did not just drift — it was structurally incapable of returning anything but a perfect score, and it told the user their CV covered skills it never mentioned. A second fault kept it from ever running at all: the lookup globbed for `*offer_<id>*`, a naming convention nothing produces. `cv generate` names files `cv-<company-slug>.md` and records that stem in `cv_used`, which is the authoritative link. The command now reads `cv_used` first and keeps the glob as a fallback for hand-named files. - Extract `_strip_frontmatter`, and route `_parse_markdown_for_review` through it (that helper documented the frontmatter-stripping behaviour but had no callers at all) - 4 tests, including an unterminated frontmatter block that must not swallow the document Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two ways applyr put things where they did not belong. `export` wrote the whole database — every company, score, note and private assessment — into the current working directory. Run it from a checkout, as anyone working on applyr will, and it drops an untracked 200KB file of personal job-search data one `git add .` away from being published. Everything else applyr owns lives in APPLYR_DIR; this was the only command that wrote outside it. An explicit path still wins. The "database schema is newer than applyr" notice used a bare print(), putting a prose line above the payload of every `--json` command and breaking the parse at character one — for exactly the agent callers applyr exists to serve. `warn()` was already there, its docstring already said "warnings are not data", and nothing called it. Now it does. Also isolates `errors._json_mode` in the test suite. `cli.py` sets it on every `--json` run and nothing resets it, so one JSON test silenced stderr for every test collected after it and the suite's result depended on ordering. That is what surfaced this: the new stderr assertion passed alone and failed in the full run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The letter took the offer's first three technologies verbatim and wrote "With my background in <them>". The candidate's profile was never consulted, so the sentence asserted whatever the employer happened to ask for first. On a real vacancy it produced "my skills in React.js, Redux, Hooks" for a profile that contains no Redux anywhere. This is not a metric being wrong: a cover letter is sent to an employer, so the tool was writing a false claim on the candidate's behalf — against the first rule in its own agent contract, "never invent skills, projects, or experience". Key skills are now the intersection of what the offer asks for and what the profile actually contains, falling back to "relevant technologies" when there is no overlap rather than naming a stack the candidate cannot back up. - `_profile_haystack` flattens the profile for the check, preferring the raw cv-master text over the handful of parsed fields - `cmd_cv_cover_letter` now passes that raw text through - 4 tests, including an offer with zero overlap and one with no stack at all Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fallback branch printed a message and fell through, so the process exited 0. `applyr <typo> && next-step` therefore ran the next step after a command that did nothing — the exact failure an agent-driven CLI cannot afford. It also wrote the message to stdout, in `--json` mode too, where every other error path emits a structured object on stderr and leaves the payload stream parseable. The `cv` subcommand branch in the same file already used `die()` correctly; only the top level was left behind. Two tests pinned the old behaviour by asserting the message appeared in stdout, and a third read `assert code != 0 or "Unknown command" in out` — an `or` that passed either way, which is why nothing ever caught a branch that announced failure and exited 0. All three now assert the real contract. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen call sites printed a usage line and returned, so the process exited
0 having done nothing. `applyr show && next-step` ran the next step; an agent
gating on the exit code was told a command it never completed had succeeded.
The usage text went to stdout as well, contaminating the payload stream that
`--json` callers parse.
`compare` proved it was an oversight rather than a decision: with no arguments
it exited 0, with one argument it exited 1 through die(). That one site was
already right; the other sixteen now match it, sharing a `_usage` helper that
emits a stable `missing_arguments` code.
Asking for help is not a failure. Three branches conflated the two cases in a
single condition (`len(args) < 2 or args[1] in ("--help", "-h")`), which is
exactly how missing arguments inherited help's successful exit — they are now
separate, and `--help` still prints to stdout and exits 0.
Nine tests asserted the old contract (`assert code == 0`, usage in stdout) and
have been corrected; 32 new cases cover every affected command in both modes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`--sort` accepted only raw database column names and silently fell back to the default for anything else. `--sort score` and `--sort date` — the two a user would reach for first — returned an unsorted list with no indication, while `--sort compatibility_pct` worked. None of the working names appeared in the help text. The source already knew better: the status check three lines above exists because an unknown value "reads as 'no offers' rather than 'you mistyped it'". The sort field got the opposite treatment in the same function. - `SORT_FIELDS` maps what a user types to the column it means, so `score` and `date` work alongside the column names, and anything else is an error - The mapping is still an allowlist, so no user string reaches the ORDER BY - Help now names the accepted values `--limit -5` returned every offer in the database, because SQLite reads a negative LIMIT as unbounded. `--all` was built on that same accident, passing -1 deliberately; it now passes 0, which the existing code already treats as "emit no LIMIT clause", and a negative `--limit` is rejected with a pointer to `--all`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`follow_up_done` is checked by the followups query but written by nothing in
applyr — no command exposes it, so it stays 0 on every row forever and never
excluded anything. The query filtered on it anyway, which filtered out
nothing.
Verified on a real database: offer #142, rejected three weeks earlier, still
listed as an OVERDUE follow-up next to offers genuinely awaiting a first
reply. A follow-up only means something while an answer is still owed, and
that state already exists on the row — `cmd_update` encodes exactly this
reasoning when deciding whether to set a *new* follow-up date ("only while an
answer is still owed"), but nothing cleared the date already stored once a
reply arrived.
- Filter on `status IN ('applied', 'waiting')` instead of the dead flag —
the two statuses that mean a reply is still pending
- On the real database this dropped offer #142 out of the list and reduced
the overdue count from 16 to 15
Writing the test surfaced a second bug in the same function: the "nothing
pending" early return ignored `as_json` and always printed the human sentence,
so `followups --json` with an empty result handed the caller a JSONDecodeError
on exactly the case it most needs to handle cleanly — the same class of fault
already fixed in `response_rate`.
6 tests: one per reply status confirming exclusion, one confirming a
genuinely pending offer still shows, and two pinning the empty --json shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`.claude/CLAUDE.md` and `.cursor/rules` are both listed in `_AGENT_DETECT_ORDER` as valid signals — a project using either is correctly recognised as "claude" or "cursor". But the write target ignored which file was matched and always fell back to the first entry in `_AGENT_TARGETS` (top-level `CLAUDE.md`, or `.cursorrules`). A project already using the nested location therefore got a second, empty file created at the root, appended with instructions, while its real config file — the one the agent actually reads — was left untouched. The user's context ends up split across two files, one of which the tool created without being asked and the other of which it silently ignored. Verified end to end in a scratch project: before the fix, a `.claude/CLAUDE.md` already in place was left alone and a fresh top-level `CLAUDE.md` appeared; after, the existing `.claude/CLAUDE.md` is the one appended to, and no second file is created. - Track which path the auto-detect loop matched and prefer it over the default when writing - Explicit `--agent` still uses the default path, since no detection ran to prefer an alternative — unchanged behaviour, covered by a test - `detected_path` only ever holds a value from the curated detection table, never user input 3 tests: the nested claude path, the nested cursor path, and the explicit-agent case confirming detection bypass is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DeibyGS
added a commit
that referenced
this pull request
Aug 12, 2026
Patch release: every commit since 1.5.0 is a `fix:` — no new public API surface, no new flags. 25 bugs across two audit passes (PRs #47, #48, #49, #50), found by running applyr against a real 200+ offer database and a real CV/cover-letter pipeline end to end, repeatedly. Highlights: response tracking was structurally incapable of reporting a non-zero rate; the scoring rubric rewarded job postings for omitting requirements; `cv keywords` matched every generated CV against a copy of the offer's own tech stack, so it always reported 100%; cover letters could claim skills the candidate's profile never mentioned; seventeen call sites reported success on missing required arguments. Full list in CHANGELOG.md under 1.5.1. Co-authored-by: Deibyg <40776476+deiybg@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continuation of the audit that produced #47/#48/#49: this round came from running applyr against a real 200+ offer database and a real CV/cover-letter pipeline, end to end, repeatedly. 16 more bugs, 11 commits, each independently tested and passing before the next was started.
This PR is 992 lines (852 insertions, 140 deletions), over the 500-line budget. Per explicit user approval: kept as one PR rather than split into a chain, because each commit is already an atomic, independently-tested fix — splitting would distribute the same 11 commits across more reviews without reducing the risk of any individual change. Documented here per the budget rule's exception clause.
Changes
applyr/commands/analytics.pyplansharesgaps' priority ranking;followupsfilters by status instead of a dead flag;followups --jsonrespects--jsonon the empty-result pathapplyr/commands/core.pyBiggest weaknesspicks the true lowest score;--sort/--limitvalidated viaSORT_FIELDS;--alluses0instead of-1;setup-agentwrites to the detected config path, not always the defaultapplyr/cv.pycv keywordsfinds the filecv generateactually wrote (viacv_used), and matches keywords against the document body only (_strip_frontmatter), not the YAML frontmatter carrying a copy of the offer's owntech_stack; cover letters claim only skills present in the profileapplyr/db.pywarn()(stderr) instead of a bareprint()applyr/commands/workflow.pyexportdefaults intoAPPLYR_DIR, not the current working directoryapplyr/cli.pydie(), on stderr, instead of printing usage and exiting 0CHANGELOG.mdUnreleasedtests/*The two worth reading in full
cv keywordsalways reported 100%. It matched the offer's required keywords against the whole generated CV file, which opens with YAML frontmatter carrying a verbatim copy of the offer's owntech_stack. The offer's keywords were being matched against a copy of themselves — every generated CV, for every user, scored 100% STRONG regardless of what it actually said.The cover letter could invent skills.
generate_cover_lettertook the offer's first three technologies verbatim and wrote "With my background in<them>," without ever consulting the candidate's profile. On a real vacancy it produced "my skills in React.js, Redux, Hooks" for a profile with no Redux anywhere — a false claim, sent to an employer, on the candidate's behalf, against the tool's own first rule ("never invent skills, projects, or experience").Recurring pattern
Four of these bugs were functions or design decisions already written correctly, just not connected:
warn()existed and said "warnings are not data" in its own docstring;_parse_markdown_for_reviewalready stripped frontmatter and had no callers;comparealready useddie()correctly for missing arguments while sixteen sibling call sites didn't;cmd_update's own comment already stated "a rejected or closed offer needs no chasing" but nothing cleared the stale follow-up date it implied.Test Plan
git fsckrun after unrelated worktree cleanup earlier in the session — no corruption, only expected dangling objects from rebase/worktree-removalChecklist
🤖 Generated with Claude Code