feat(007)!: decommission the FueliX AI analyzer — the Cisco API check is the only mode (4.0.0) - #12
Merged
Conversation
… the FueliX AI analyzer Planning artifacts for retiring the advisory-driven AI mode so the inventory-driven Cisco PSIRT check becomes the tool's only behaviour. - brds/004: business requirements (31 items) with the decisions settled during review — options removed outright, product renamed, version 4.0.0 - constitution 2.2.1 -> 3.0.0 (MAJOR): removes every AI-provider rule. Principle I drops FueliX from permitted network access; Principle IV drops the AI-normalization clause and collapses to a single `Needs review` label; Principle V replaces AI-endpoint minimization with a stricter Cisco-bounded rule; the "AI provider" constraint and the dry-run quality gate are deleted. Product renamed to Cisco Advisory Impact Analyzer (command, distribution, and repository names unchanged). - specs/007: 6 prioritized user stories, 38 functional requirements, 12 success criteria, 10 edge cases, and the four /speckit-clarify decisions. No product code changes yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 0 and Phase 1 design output for the FueliX decommission. - plan.md: 12/12 constitution gates pass (v3.0.0), no violations; file-by-file DELETE/EDIT/KEEP map across caia/, tests/, docs, and packaging. - research.md: 11 decisions. D1 is load-bearing — argparse gives --help precedence over a merely-absent option (verified: `caia --api --help` exits 0 and prints help), so the removed-option guard must scan argv before the parser is built. Without it `caia --uninstall --dry-run` would really uninstall, which is the case FR-011 exists to prevent. - data-model.md: the 3 entities that change shape and the 7 that must not, since behavioural equivalence is what a deletion through shared code threatens. - contracts/: post-change CLI surface (options, refusal messages, precedence, exit codes), per-user config format, and the canonical product copy with the 13 surfaces each string appears on. - quickstart.md: 7 validation scenarios mapped to the user stories. Scenario 0 captures the equivalence baseline and must run before any deletion. No product code changes yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
46 tasks (T001-T046) grouped by user story: 2 setup, 4 foundational, 7 US1, 6 US2, 4 US3, 9 US4, 1 US5, 9 US6, 4 polish. 22 parallelizable. Two ordering decisions carry the risk of this change: - T001 captures the equivalence baseline before any source edit. Once the advisory path is deleted this working tree cannot produce it again, and it is the only evidence that proves FR-006 (the surviving check did not move) beyond the unit suite's property assertions. - The three doomed modules are deleted last, in US6, not in the foundational phase. Phase 2 makes them unreachable, which is enough to make every user-facing story true; keeping the files on disk until the end means the old code path is still available to compare against if the equivalence diff fails. MVP is US1, which is demonstrable but deliberately not releasable — after it alone the tool still prompts for an AI key and calls itself agentic. No product code changes yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… is the only mode (4.0.0) `caia` with no options now does what `caia --api` used to do: check every firewall in the inventory against Cisco's published advisory data and write one HTML impact report. The advisory-driven, AI-assisted mode is removed entirely. BREAKING CHANGE: five options are withdrawn (--api, --url, --dry-run, --keep-temp, --no-keep-temp), the Excel report is gone, and the FueliX API key is no longer read from any source. Passing a removed option refuses with a message naming it and what to run instead; leftover FUELIX_* values in an existing config file or the environment are inert and disappear the next time `--config` saves. Removed: caia/fuelix.py, caia/cisco.py, caia/report.py, tests/test_extraction.py, the advisory-run path and dry run in analyzer.py, the AI-only combo compaction in inventory.py, the now-dead validate_alias parameter in matching.py. Added: a removed-option guard in cli.py that runs before the parser is built. argparse gives --help precedence over a merely-absent option, so without this `caia --api --help` would print help and exit 0 and `caia --uninstall --dry-run` would really uninstall. Matching is exact, so an abbreviation such as --dry still gets argparse's ordinary error and the removal notice keeps its meaning. Renamed the product to Cisco Advisory Impact Analyzer (command, distribution, and repository names unchanged) and put every surface on one canonical tagline and description: banner, help, package metadata, README, landing page. Verification: - the post-change report is byte-identical to a pre-change baseline except the run timestamp — same devices, verdicts, reasons, request count, and exit code - a live run against Cisco: 70 firewalls, 27 checks, 55 Impacted / 0 Not impacted / 15 Needs review, matching the BRD-003 feasibility spike - 246 unit tests pass - clean-room install on python:3.9-slim passes, including that --uninstall --dry-run refuses and leaves the tool installed Note: tests/test_config.py deliberately names the FUELIX_* keys to prove they are inert; that behaviour cannot be tested without naming them. Implements spec 007 (specs/007-decommission-ai-analyzer/), BRD-004, constitution v3.0.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ali Bahaloo (alibahaloo)
added a commit
that referenced
this pull request
Jul 28, 2026
….0) (#16) * chore: ignore uv.lock — no install path consumes it `uv run` in a checkout writes a lockfile that nothing here reads. Every documented path is `uv tool install` (which resolves dependencies itself) or `python -m unittest`; no `uv sync` or `uv run` appears in CONTRIBUTING, README, or the workflows. Committing it would also work against the packaging intent: this package is both the `caia` command and an importable library, and pyproject pins loosely on purpose so consumers resolve their own versions. A committed lockfile would imply a pinned set neither install path honours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(008): BRD-005 and spec — bundle-scoped impact reporting Adds the requirements for scoping a run to a Cisco Event Response bundle and reporting one row per advisory in that bundle, rather than one row per device. Feasibility was verified against the live sources using ERP-75736 and the 70-row reference inventory, not read from documentation: - a bundle page yields its advisory identifiers from the links in its table; they are absent from the visible text, so reading the text alone fails - `advisoryId` and `firstPublished` are already in the per-device payload the tool retrieves, so filtering an assessment costs no extra requests - an advisory impacting nothing appears in no device's result, so its metadata needs an individual lookup (1 of 25 here; worst case the bundle size) - Cisco publishes no bundle endpoint (HTTP 404), hence reading the page - 24 of the 25 advisories touch at least one device, from 0 to 50 each Two decisions were escalated rather than defaulted, and are recorded with their reasoning in the spec's Clarifications: - the device is the unit of count, with a per-row rollup by affected release. A release-based denominator cannot close FR-025's accounting, because an undetermined device often has no release Cisco recognises. - rows are scoped to the families an advisory covers, with whole-run totals stated once. A single whole-inventory denominator would emit ~375 mostly meaningless undetermined markers across 25 rows, training readers to disregard the marker Principle IV depends on. The load-bearing constraint throughout: a device whose assessment reached no conclusion has no advisory list at all, so it can neither be asserted as impacted nor excluded as clean. FR-024 through FR-028 exist for that, and FR-027b fixes the direction of the guarantee — family scoping may only narrow which rows an undetermined device appears in, never suppress an impacted device or manufacture an unaffected one. Planning gate: fetching a bundle page adds sec.cloudapps.cisco.com as an endpoint the analyzer contacts, which Technology & Data Constraints designate a reviewed change requiring a constitution amendment before implementation. Docs only — no product code changes, so VERSION is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(008): plan, research, data model and contracts Phase 0 and Phase 1 output for bundle-scoped impact reporting. Fifteen research decisions, all three items the spec deferred to planning now resolved. What the research changed: - No new dependency. An earlier curl timeout suggested the bundle page might need a browser engine; it does not — urllib fetched it in 0.54s on first attempt. stdlib HTMLParser matching on href and a lexical scan both return exactly the 25 identifiers. The parser is chosen regardless: an advisory id mentioned in prose rather than linked would silently widen scope, and scope is the only thing the page is trusted for. - The publication-date cross-check floated in the spec is rejected. The date window returns 27 advisories for this bundle's date — the 25 plus two unrelated same-day publications — so it cannot distinguish a missed parse from a same-day publication the bundle legitimately excludes. A check that cannot fail cleanly is worse than none; the advisory count shown to the user before the run is the honest check. - Family derivation costs nothing: productNames is already in the per-device payload, so both derivation sources are free. Verified on two advisories. - Added cost is ~3s on a 72s run (0.54s page fetch plus one metadata lookup for the single advisory in this bundle that impacts nothing), inside SC-008. Module split mirrors the roles the package already has, so the Principle VI boundary needs no new reasoning: erp.py does I/O like psirt, bundle.py is pure like matching, bundle_report.py renders like html_report. erp and bundle join the shared surface, because the ERP parse and family derivation are exactly the fragile logic a second consumer would otherwise reimplement. The data model's worked example was verified arithmetically against the live assessment: the per-row invariant (impacted + unaffected + undetermined == population) closes on all four sampled rows. Constitution Check is a CONDITIONAL pass. Two gates depend on an amendment (3.1.0 -> 3.2.0, MINOR) carried as a blocking prerequisite of the first implementation task: admit sec.cloudapps.cisco.com; state the scope-selector boundary that keeps "no scraped or cached-as-authoritative alternatives" substantively true; and extend Principle VI's named module list from four to six. Complexity Tracking is empty — no violations. Product version will move to 4.2.0 (MINOR: the shared surface gains, nothing incompatible changes). Not bumped here — docs only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(008): task breakdown — 60 tasks across 9 phases Phase 2 output. Tasks organised by user story so each ships and tests independently; MVP is Phase 0-3 (US1). Three things differ from what the task template would produce by default: - Phase 0 is the constitution amendment, and it is the only phase that does not happen on this branch. Spec 006 recorded its amendment as already applied and explicitly not a task; ours is outstanding, so it becomes a blocking prerequisite on its own chore/ branch against main. Until it merges, an implementation that fetches a bundle page violates the very constraint that made the change reviewable. - Tests are marked not-optional. The template treats them as opt-in, but the Development Workflow gate makes them mandatory here on three counts: verdict-adjacent logic, analysis-flow changes needing credential-free coverage, and the shared surface's headless property. - FR-035 and FR-036 initially had no tasks — dropped because they are SHOULDs rather than MUSTs — and FR-019 was implemented but uncited. Added as T056/T057 and cited on T017. Coverage is now 46/46. Validated mechanically: 60 tasks, IDs sequential T001-T060, every task carrying a checkbox, id, optional labels and a concrete file path; all five user stories represented; 18 tasks parallelisable. The parallel win is Phase 2, where the erp.py and bundle.py chains share no files and can run concurrently, then the four test modules together. Each phase states its independent test criteria against a numbered quickstart scenario. T060 singles out Scenario 2 to re-run after any report change: a row reading "0 impacted" without its 11 undetermined devices is the failure mode the design exists to prevent. Docs only — VERSION untouched; its bump to 4.2.0 is T059. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(008): amend constitution to 3.2.0 — bundle page endpoint and scope-selector boundary Enacted as a prerequisite of spec 008. Cisco publishes no API equivalent of a bundle (a request for one returns HTTP 404), so a bundle's membership can only be learned from its published page. That made two existing constraints block the feature and left one principle's text stale. Four changes: - Networking admits sec.cloudapps.cisco.com, scoped to resolving a bundle to its advisory identifiers and its own identity. The request is explicitly unauthenticated and forbidden from carrying ANY inventory-derived data — a stricter bound than the advisory API's, which may carry the derived family/platform/release triple. - A new Scope selectors bullet, rather than a relaxation of "No scraped or cached-as-authoritative alternatives". Reading a published page could be argued to survive that prohibition on a technicality, which is the kind of reasoning a governance document should foreclose. So the prohibition stands and a narrow enumerated permission sits beside it: a bundle page may supply which advisories are in scope and the bundle's own identity, and may not supply any advisory fact — title, severity, score, CVE, affected product, fixing release, or an advisory's publication date, every one of which the page actually displays. Disagreement resolves in the API's favour without exception, and a page yielding no advisory is an error rather than an empty scope, because an empty result rendered as a report reads as an all-clear. This narrows the tool's latitude while unblocking the feature. - Principle VI's shared surface grows from four modules to six, adding erp (bundle resolution) and bundle (family derivation, filtering, row construction) — both fragile, safety-bearing logic a second consumer would otherwise reimplement, and a second divergent answer about which advisories are in scope is the harm the principle exists to prevent. bundle_report joins the terminal-side list beside html_report. - Consumers now enumerates all six modules of the released interface. A new constraint bullet plus materially expanded guidance, with no principle removed or redefined and no existing permission widened: MINOR. Applied on this feature branch rather than a standalone chore/ branch. Checking the history corrected an earlier assumption: both prior feature-driven amendments travelled in their feature's own PR — spec 006 in cb6938a (#9) and spec 007 in e043d41 (#12). The one standalone constitution commit, 73de536 (#15), was retroactive, documenting Principle VI after 4.1.0 had shipped, so it is not the precedent here. Keeping the amendment with the code it authorises also keeps its justification in front of the reviewer. Also realigns the spec-008 artifacts that asserted the amendment was still outstanding: the plan's Constitution Check moves from CONDITIONAL PASS to PASS, tasks Phase 0 is marked complete, and research Decision 9 records the applied date and the rejected standalone-branch alternative. The README and docs/index.html shared-surface updates remain tasks T051 and T052 — feature work due in this same PR, not amendment debt. Product VERSION untouched at 4.1.0; it moves to 4.2.0 as task T059. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(008): bundle-scoped impact reporting — one row per advisory (4.2.0) Scope a run to a Cisco bundled publication by pasting its Event Response link, and get back one row per advisory in that bundle rather than one row per firewall. Each row carries the advisory as a link, its publication date, and the products it impacts with the release that fixes them. Adds no data source. The advisory id and publication date were already in the payload retrieved per device, so filtering costs zero extra requests; only advisories that impact nothing need a lookup (1 of 25 for the reference bundle). Cisco publishes no bundle endpoint, so membership comes from the published page — which contributes scope only, never an advisory fact. New modules, taking the three roles the package already has: erp.py I/O, like psirt: bundle page -> advisory identifiers bundle.py pure, like matching: covered families, filtering, rows bundle_report.py rendering, like html_report: the per-advisory layout erp and bundle join the shared surface, because the page parse and the family derivation are the fragile parts a second consumer would otherwise reimplement. The load-bearing property, and the reason for most of the design: a device whose assessment reached no conclusion has NO advisory list at all, so it can neither be asserted as impacted nor excluded as clean. Two mechanisms enforce that rather than leaving it to review — the per-row accounting invariant (impacted + unaffected + undetermined == population), which makes a dropped device fail arithmetic instead of rendering as clean; and fail-open family scoping, where a failed derivation widens to every family so its worst outcome is a noisier report, never a suppressed device. One bug found while verifying, worth calling out because it would have misdirected patching. first_fix() returned one fixing release per row, while FR-023 requires it per impacted product. Invisible on the FMC row, where both affected releases share 7.4.4 — but cisco-sa-asaftd-ikev2-dos-eBueGdEG spans 13 releases needing 6 different fixes (7.0.9, 7.4.3, 7.7.11, 9.16.4.85, 9.18.4.71, 9.20.4.10). The old code would have shown one target for all 50 devices. Fixes are now per device, rendered per release group. Verified twice end to end against ERP-75736 and the 70-row reference inventory: 25 rows, zero accounting failures, identical row order across runs, 33 requests in ~75s (~3s over the whole-fleet baseline). Every number matches the design's worked examples — FMC auth bypass 5/0/0 fixed in 7.4.4; ASA TCP flood 0 impacted, 17 not impacted, 11 undetermined; FTD snort3 SSL 11/22/4; IKEv2 50/0/15. Tests: 272 -> 392, all without credentials or network, including the headless property of both new shared-surface modules. Page fixtures are hand-authored builders mirroring the verified live shape, following psirt_responses.py rather than vendoring ~180 KB of Cisco's page. Declining the prompt is byte-equivalent to today's behaviour, and no bundle page is fetched on that path. --erp makes the capability scriptable, as Principle III requires. VERSION 4.1.0 -> 4.2.0 (MINOR: the shared surface gains erp and bundle; nothing incompatible changes). Tag is pushed only after this PR merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(008): exercise the bundle flow in the container install test The clean-room harness installed the new code but never touched the new capability, so a --shell session would not have tested the thing it was spun up to test. - automated checks now assert --erp is advertised and requires a value - the --shell banner walks both report shapes and the three failure paths, and shows the two filenames side by side so the layouts are easy to compare Also records an ordering quirk that is confusing to hit unexplained: the credential gate fires before bundle validation, so --erp with a bad value exits 1 (no credentials) rather than 2 (not a bundle address) until --config has run. That order is deliberate — the tool refuses before making any request — so the banner explains it instead of the behaviour changing. Default mode installs the working tree, so this tests unreleased work without pushing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spec 006 shipped a check that gets its verdicts from Cisco directly. That made the older AI-driven mode the weaker of the two on every axis that matters for a security decision, while its cost was paid continuously — a second credential every user had to obtain, inventory context leaving the organisation, two report formats, two undecided labels, and a mode-comparison table a reader had to work through before running anything.
This removes it.
caiawith no options now does whatcaia --apiused to do.caiaanalysis_output_*.xlsxandimpact_report_*.htmlimpact_report_*.htmlIndeterminateandNeeds reviewNeeds reviewapi.fuelix.ai,sec.cloudapps.cisco.com,apix.cisco.com,id.cisco.comapix.cisco.com,id.cisco.comThe command, the
caia-clidistribution, and the repository name are unchanged. Only the descriptive product name moves, because "Agent" and "Agentic firewall impact analysis" now misdescribe the tool — and for a security tool, overstating how the answer is reached is a trust problem rather than a marketing one.Breaking change
Five options are withdrawn:
--api,--url,--dry-run,--keep-temp,--no-keep-temp.VERSIONgoes to 4.0.0 — documented options and a documented mode are removed, which is MAJOR under the repo's own semver rule.Passing a removed option refuses by name and says what to run instead, rather than falling through to argparse's
unrecognized arguments:The guard runs before the parser is built, which is the one piece of genuinely new behaviour here. Argparse gives
--helpprecedence over a merely-absent option, so with--apisimply deleted from the parser,caia --api --helpprints help and exits 0 — verified empirically. Worse,caia --uninstall --dry-runwould really uninstall, when a script author who wrote--dry-runthere plausibly meant "rehearse this". A withdrawn option therefore outranks everything, including--helpand--uninstall.Matching is exact (including the
--url=VALUEform), so--dry— a valid abbreviation before — still gets argparse's ordinary error. That keeps the removal notice meaningful: it appears only for options that genuinely existed.Saved credentials survive. Leftover
FUELIX_*values in an existing config file or in the environment are inert — no error, no warning, no effect — and disappear the next time--configsaves, becausesave()writes only the keys the product still manages. No migration step, no pruning pass on an ordinary run.--dry-runis removed as a capability, not renamed. It was not an AI feature — under the surviving check it previewed what would be asked of Cisco without contacting it — so this is a deliberate loss, accepted in review. Consequence: there is no longer any way to exercise the tool end to end without Cisco credentials, which narrows what the clean-room test and offline checks can prove.Changes
caia/fuelix.py,caia/cisco.py,caia/report.py,tests/test_extraction.pycaia/analyzer.py— the advisory-run path, URL validation/prompting, CSAF download, and the dry run go;run_api()becomesrun(). The shared helpers both flows used (die,resolve_inventory,ensure_dirs,default_output_dir,_result_line, …) are untouched — altering one in passing was the single most likely way to break equivalencecaia/cli.py— five options removed;REMOVED_OPTIONS+refuse_removed_options()added and invoked frommain()before the banner and the parser; dispatch collapses to one call; the update offer now keys on the TTY check alone, sinceargs.urlwas its proxy for "scripted run" and the TTY check already covers thatcaia/config.py— managed keys 5 → 2,DEFAULTSemptied,fueliximport dropped,run_config()prompts only for Cisco. Its verification behaviour — visible ID prompt, non-echoing secret with capture confirmation, skippable prompts, accepted/rejected/unreachable — is preserved exactlycaia/inventory.py—build_combos()/expand_combo_ids()removed (AI-only compaction)caia/matching.py— the now-deadvalidate_aliasparameter removed. Mechanical only; this module is verdict logic under Principle IV and nothing else in it changedcaia/ui.py,caia/__init__.py,caia/psirt.py,pyproject.toml— product name, canonical tagline, canonical descriptionREADME.md— restructured, not patched. The mode-comparison table, advisory-mode section, FueliX key section, and AI troubleshooting entries are removed as units and the remainder re-sequenced as one path. Deleting the 25 references sentence-by-sentence would have left a document shaped around a choice the reader no longer hasdocs/index.html— title, meta, hero, three feature cards, prerequisites, run steps; the FueliX key section removed. Both documents carry a 4.0.0 removal notice, per the Documentation gatetools/install-test.sh— advisory-mode and dry-run steps replaced with the removed-option guard, the precedence case, and the credential gate.env.example,.gitignore— Cisco variables only; ignore rules for artifacts the tool can no longer produce (analysis_output_*.xlsx,csaf_*/) removed.specify/memory/constitution.md— 2.2.1 → 3.0.0 (MAJOR), landed first as a prerequisiteRuntime dependencies re-justified rather than assumed:
openpyxlstays (it still reads the inventory; only the Excel writer is deleted),python-dotenvstays (.envsupport is retained for automation, with only its AI variables withdrawn).specs/andbrds/are left as written — frozen spec-kit records; rewriting them would falsify the record.Verification
Behavioural equivalence is the requirement most at risk here, because the deletion runs straight through code both flows shared. A unit test asserting "a report was written" would not catch a reworded reason or a reordered row, so a golden report was captured from the current
--apipath before anything was deleted and diffed afterwards:python:3.9-slim— the minimum supported Python, fresh container, no prior copy:caia-cli==4.0.0built from the tree, onlyopenpyxl+python-dotenvresolved, all five removed options refuse, the credential gate refuses, and--uninstall --dry-runrefused while leaving the tool installed — the footgun that motivated the precedence rule, proven outside a mockConstitution compliance (v3.0.0)
2, the existing argument-error status;130on interrupt unchanged). The principle's clause that a withdrawn option must be refused by name is what the guard implementsNeeds reviewis now the sole undecided label; positive confirmation beforeNot impactedintact. Equivalence is proven, not assertedOne documented exception:
tests/test_config.pydeliberately writesFUELIX_API_KEY/FUELIX_MODEL/FUELIX_BASE_URLinto a fixture, to prove those values are inert and vanish on save. That behaviour cannot be tested without naming the real keys users have on disk. The only other mentions anywhere are the two 4.0.0 removal notices the Documentation requirement asks for.After merge
python tools/release.py) — the Release workflow refuses to publish when the tag does not matchVERSION🤖 Generated with Claude Code