feat(competitor): competitor analysis domain — spine + collector reframe + structured feeds - #17
Merged
Merged
Conversation
Adds 4 new tables for the competitor analysis spine: - competitors, competitor_sources, competitor_snapshots, competitor_changes All company-scoped with cascade FKs and appropriate indexes. Generated additive migration via drizzle-kit (0015_shocking_lord_tyger.sql). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Company-scoped CRUD for competitors, sources, snapshots, and changes. getDueSources uses a JS-side filter (PGlite-safe interval arithmetic alternative). 7 tests, all green alongside the existing 250. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the Collector interface (types.ts + httpFetch), pricingCollector (JSON-LD → DOM heuristic fallback), collector registry (index.ts), HTML fixtures, and tests. Social collector registration is left as a commented breadcrumb for Task 6. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- `getCompanyNotifyUserIds(companyId)` in packages/db: returns owner + admin userIds for a company; exported from queries/index.ts - `apps/web/src/lib/competitor/notify.ts`: `buildDigest` reduces alerts into a single notification payload (highest severity wins) - `apps/web/src/lib/competitor/pipeline.ts`: `runSource` (fetch → parse → hash → store-on-change → diff → evaluateRules → insertChanges → notify) and `runDueCompetitorSyncs` for use by scheduler + sync route - Integration test with stub collector + real PGlite (|db| project): proves first-run snapshot creation, store-on-change dedup, and price_increase alert Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…or_changes)
Add two deterministic, read-only tools wired through the existing tool
registry (toolSchemas + toolHandlers in index.ts, mirroring calculate):
- list_competitors — queries listCompetitors(companyId), returns text
- list_competitor_changes — queries listChanges(companyId, {limit}), returns text
Both have no `mutates`, require no AI provider to execute, and are
MCP-exposed by default. Zod schemas added to toolSchemas so
validateToolInput passes. All 27 ai-tools test files green; all
@burnless/ai guard tests (tools-naming, registry-derivation) still pass.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create domains/competitor.ts with competitorDomainModule (non-core, tools: competitorTools gates LLM visibility), competitorContributor (recent-changes context, order 40, graceful-degradation), and competitorNavEntries (Swords icon). Register in registerDomains(); add one Competitors entry to coreNavItems in nav-config.ts. TDD: 4 shape assertions GREEN; all other domain tests unaffected. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GET/POST /api/competitors and GET/PATCH/DELETE /api/competitors/[id] mirroring the transactions/accounts sibling conventions exactly. requireDomainEnabled gates every handler; POST creates optional sources in the same transaction. Route test asserts 403 DOMAIN_DISABLED path. Guard allowlist extended: competitors cache is non-financial so trackDataMutation must not fire. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add COMPETITOR_DOMAINS set to mutation-bus (separate from FINANCIAL_DOMAINS so competitor syncs don't reset the AI-insight stale countdown) - Map /competitors URLs to "competitor" domain in domainFromUrl - Update apiFetch to publish competitor-domain mutations on the bus - Add KEYS.competitors + KEYS.competitorChanges to the key registry - New apps/web/src/lib/swr/competitor.ts: useCompetitors + useCompetitorChanges both subscribe to subscribeMutation, mirroring useTransactions (WS2 pattern) - Re-export hooks + DTO types from @/lib/swr index Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ drop dead status field
parseInt("abc", 10) returns NaN; Math.max(1, NaN) propagates NaN
down to listChanges. Replace with Number.isFinite guard, defaulting
to 50 when the param is absent or non-numeric.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ard Intl RangeError)
…e-positive parses JSON-LD path: filter Products to only those with a numeric offer price; zero priced plans → confidence 0.1. DOM-heuristic path: keep only well-formed blocks (real name ≠ "Plan" + numeric price); 0 or > 8 well-formed plans → confidence 0.1 (noise/empty). Adds two smoke fixtures (pricing-jsonld-noprice, pricing-dom-noise) and three tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…loor) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tput) Adds two pure engine functions to packages/engine/src/competitor/text-diff.ts: - diffText: multiset line-difference over normalized visible text (order-insensitive, O(n), lines capped at 20, truncated flag) - contentChangeAlert: converts a TextDiff into an info Alert or null Re-exported via competitor/index.ts. 6 tests added, all pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
before/after each derive truncated from their own line count vs full count, instead of the OR-combined diff.truncated which misreported the smaller side under asymmetric changes. Adds a gap-closing test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ditive) Adds two nullable columns to competitor_snapshots: `normalized` (visible text after HTML stripping) and `normalized_hash` (for store-on-change keying in Task 4 pipeline). Migration is purely additive — two ALTER TABLE ADD COLUMN statements with no NOT NULL/default; existing rows keep NULL and re-baseline on next run. Extends insertSnapshot input with optional normalized/normalizedHash fields; getLatestSnapshot surfaces them automatically via select *. TDD: round-trip test written first (RED), then schema+migration brought it GREEN alongside all 7 existing tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tightens insertSnapshot input from optional normalized?/normalizedHash? to required normalized: string / normalizedHash: string, restoring the compile-time guarantee that every stored snapshot carries a non-null normalizedHash — the field the Task-4 pipeline keys store-on-change on. Updates the 2 pre-existing snapshot test call sites to pass the fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-gated Tier-2 Tier 1 (always): fetch → normalizeHtml → store-on-change keyed on normalizedHash (nonce-churn fix) → diffText → one content_changed alert. Tier 2 (only when a collector exists and parse confidence >= 0.4): diffStructured → typed changes. page-type sources have no collector (Tier 1 only); low-confidence/no-collector on a successful fetch is content_only, not broken. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hemas Widens the source-type Zod enum in both the POST /api/competitors body schema and the POST /api/competitors/[id]/sources schema from ["pricing","social"] to ["pricing","social","page"], allowing callers to register a generic page-watcher source (Tier-1-only, no collector needed). Adds a TDD test that POST with a page-type source returns 201. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y guard
The Tier-1/Tier-2 reframe tests embed $29/$39/$${amount} inside sample
competitor HTML (fetched+normalized+diffed) and expected normalized-text
assertions — third-party scraped-price INPUT, not app display code. Same
rationale as the already-allowlisted pricing.ts collector. Adds the three
competitor test dirs so pnpm check's currency guard stays green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t.ts Review Minor: file-scope the db __tests__ allowlist entry to the single file that holds scraped-price fixtures rather than the whole directory. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…atcher source
Move the competitor CRUD list from /competitors (now the analysis dashboard)
to a nested /competitors/manage route, mirroring /transactions/accounts. Adds a
"Back to Competitors" back-link header and a "Watch a page" URL field that
appends a { type: "page" } source (Task 5 POST already accepts it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Review Minor: /competitors (dashboard) and /competitors/manage both showed an identical 'Competitors' h1. The manage surface now reads 'Manage competitors' with add/configure sub-copy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nt Date.now() DashboardContent is an async Server Component where Date.now() is safe (each render is a fresh server request, not a pure-functional client render). Follow the existing pattern for react-hooks/set-state-in-effect and react-hooks/refs — both already demoted to warn for the same reason. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-baseline A snapshot created before the reframe migration has normalized == null. The Tier-1 content alert diffed fresh content against "" (via ?? ""), emitting an all-lines-added content_changed alert + admin notification on the one-time re-baseline. Guard the alert with `latest.normalized != null`; the re-baseline snapshot is still stored (store-on-change keys on normalizedHash), and the Tier-2 structured diff still runs. Adds a legacy-row regression test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Final review Minor: the type doc still read "pricing | social"; page is now a valid source type. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds feedAlert/sitemapAlert/dropNull/capAlerts + MAX_ALERTS_PER_RUN=25 to the engine rule layer. feed drops removed items; sitemap drops count mods; both types are capped at 25 individual + 1 summary per changeType. pricing/social/unknown dispatch path is byte-identical to before. TDD: 3 new tests red → green; full engine suite 778/778 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds feedCollector (type:"feed") that regex-parses RSS 2.0 and Atom XML
into a structured { items: { [id]: { title, link, publishedAt } } }
payload (id = guid ‖ id ‖ link). Confidence 0.9 for ≥1 well-formed
item, 0.1 otherwise. CDATA + HTML-entity decode; 50-item cap; toIso
never throws. Registered in COLLECTORS; SOURCE_LABELS gets "Feed".
TDD: 2 XML fixtures + 3 tests (RED → GREEN verified).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…y guard feed.ts strips CDATA via String.replace(…, "$1") — a regex backreference, not a currency amount. Same false-positive class as the existing camelCase- splitter backreference entries. Keeps pnpm check's currency guard green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add sitemapCollector: parses <urlset> into url→lastmod map with 2000-url cap + truncated marker; confidence 0.9 for urlsets, 0.1 for sitemapindex or non-sitemap documents. Register in COLLECTORS + add SOURCE_LABELS entry. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds discoverFeeds(siteUrl) — zero-AI, best-effort: parses homepage
<link rel=alternate> for RSS/Atom feeds, reads robots.txt Sitemap: lines,
and falls back to probing /sitemap.xml. Every fetch is try/catch-guarded;
returns absolute, de-duped {type,url}[] candidates. Task 5 will wire this
into competitor-add + /detect route.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen both source-type enums to accept feed/sitemap. Add
POST /api/competitors/[id]/detect (same gate as the sibling sources
route: requireCompanyWrite + requireDomainEnabled("competitor") +
revalidateTag("competitors")) that runs discoverFeeds and creates
newly-found sources, de-duped against listSources for idempotency.
Wire best-effort discovery into POST /api/competitors, wrapped in
try/catch so a discovery failure never blocks competitor creation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…over noUncheckedIndexedAccess flags regex capture-group and Record index access as string|undefined. All sites use non-optional capture groups or fixed-fixture lookups, so non-null assertions are correct and sound. No logic or assertion changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds two end-to-end integration tests that prove the reframed Tier-2 pipeline already handles feed/sitemap sources without any production-code change: feed-gains-item → post_published, and sitemap-gains-url → page_added both pass by exercising the real collectors + evaluateRules through runSource. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen SnapshotDto.structured to carry feed items and sitemap urls/count, and derive two new summaries in the shared CompetitorAnalysisBody: Recent posts (newest ~10 feed items, title linked + date) and Site structure (Pages tracked count). Both mirror the existing pricing derivation and render only when a feed/sitemap source has a latest snapshot with data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add optional RSS/Atom feed + sitemap URL inputs to the add-competitor modal (mirroring the existing page input, appended as feed/sitemap sources), and a per-row "Detect feeds" action that POSTs to the detect route and revalidates so newly-found sources appear. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…(cluster A) CodeQL security: - feed.ts: strip unterminated trailing tags (<[^>]*>?) so a bare "<script" at end-of-input is removed (incomplete HTML sanitization). - sitemap.ts: decode & LAST to avoid double-unescaping. SonarCloud (feed/sitemap/diff/rules/normalize): - diff.ts S2871: localeCompare sort comparator. - feed.ts S8786/S7780: linear tag strip + String.raw regex. - sitemap.ts S8786: drop ambiguous \s* around LOC/LASTMOD captures. - rules.ts S3776/S7755/S3358/S6582: extract priceAmountAlert + overflowNoun helpers, .at(-1), optional-chain dispatch. - normalize.ts S5843/S8786: composed BLOCK_CLOSE + linear tag strip. Behavior unchanged; all competitor engine/web tests + currency guard green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…scover (cluster B) pricing.ts: merge PRICE_RE constant (S8786+S5843 regex backtracking+complexity via [$€£] char-class + month|mo/year|yr ordering + \d shorthands), replaceAll (S7781), exec() (S6594 ×3), char-class for alteration (S6035), extract resolvePeriod helper (S3358), extract jsonLdNodeToPlan helper reducing fromJsonLd complexity 19→~10 (S3776), optional-chain addressed by extraction (S6582), H1_RE/PRICE_BLOCK_RE constants. social.ts: FOLLOWERS_RE constant with \d+\.\d+|\d+ (S8786), [km]+/i removing duplicate K/M (S5869), \d shorthands ×2 (S6353), exec() (S6594), replaceAll (S7781), extract getMultiplier helper eliminating nested ternary (S3358). pipeline.ts: merge two @burnless/db value imports into one (S3863); extract persistAlertsAndNotify helper removing the alert-fan-out block from runSource, reducing cognitive complexity 19→~12 (S3776). discover.ts: decompose discoverFeeds into resolveCandidate + probeHomeFeeds + probeRobotsSitemap + probeFallbackSitemap helpers; discoverFeeds complexity 18→~1 (S3776). All best-effort/never-throws semantics preserved. Verified: competitor suite 55/55 ✓, type-check 0 errors ✓, no-hardcoded-currency 5/5 ✓ Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er C)
- S6759: mark props read-only on all 10 components/sub-components (Readonly<{…}> wrapper or readonly interface fields)
- S3735: drop `void` from 5 mutate() call-sites (handleDelete, handleDetect, onAdded, SWR subscription ×2)
- S3358: extract nested ternaries in SeverityBadge (nonCriticalCls const) and price render (planCurrency const + block body)
- S6479: composite keys in ContentSnippet removed/added line maps (`r${i}-${l}` / `a${i}-${l}`)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- S7773 changes/route.ts:19: Number.parseInt over parseInt (same NaN guard) - S5976 pricing.test.ts:26: collapse 3 identical-shape low-confidence tests into one it.each parameterized test; all 3 cases + assertions preserved - Web:DoctypePresenceCheck / Web:PageWithoutTitleCheck / Web:S5254 on all 6 HTML fixtures: add <!DOCTYPE html>, lang="en" on <html>, and <title> in <head>; body content unchanged byte-for-byte Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sanitization The prior fix stripped tags BEFORE decoding entities, so `<script>` in a feed title decoded to a literal `<script>` after the strip (CodeQL: 'may still contain <script'). Move the tag-strip to the LAST step so any `<>` produced by entity-decoding is removed; `&` stays last among the entity decodes to avoid double-unescaping. Output is plain text either way (React escapes on render); tests unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…on remediation) CodeQL flags single-pass regex tag removal as incomplete sanitization. Replace it with the documented fixpoint remediation: strip <...> repeatedly until the string is stable (also removes nested/reconstructed and unterminated tags). Same output for all real inputs; feed text is plain React-escaped display text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Competitor Analysis domain — deterministic, in-house, AI-optional
Adds an automated, continuous competitor analysis capability as a new Foundation domain (
competitor, non-core, per-company gated). The whole pipeline runs deterministically with zero AI configured; AI is a reserved optional superpower layer, never a dependency. No external/paid data vendors — every signal is fetched-and-parsed in-house.This branch lands three reviewed sub-projects (49 commits):
1. Change-detection spine
Store-on-change snapshots + structured diff + rule/alert engine + per-competitor change timeline, collector-agnostic,
companyId-scoped, with minimal competitor management. Proven with pricing + social collectors. Read-only AI tools (list_competitors,list_competitor_changes). Schedulercompetitor-syncjob. Additive migrations0015(spine tables).2. Collector reframe (detection vs extraction)
Splits detection from extraction: a Tier-1 raw-diff floor normalizes HTML → visible text → hashes → store-on-change keyed on the normalized hash (kills nonce/CSRF churn) → line-diff → an honest
contentchange on any page. Tier-2 structured parsers (pricing/social) are demoted to confidence-gated, additive contributors (silent on low confidence, not "broken"). Adds a genericpagewatcher. UI reframed:/competitors= analysis dashboard (expandable cards),/competitors/manage= CRUD (nested),/competitors/[id]= permalink. Additive migration0016(snapshotnormalized/normalized_hashcolumns).3. Structured-feed collectors
Two new Tier-2 collectors —
feed(RSS 2.0 + Atom → new-post detection) andsitemap(page appear/disappear + count) — emitting id-keyed structured payloads so the reframe pipeline handles them with zero production change. Deterministic feed/sitemap auto-discovery (discoverFeedsvia<link rel="alternate">+robots.txt), wired best-effort into competitor-add plus a manual/detectaction. Analysis summaries (Recent posts / Pages tracked). Regex parsing, no new dependency, no migration.Design principles honored
0015,0016); user data sacred.WidgetCard/DataTablecompositions, no new primitives.requireCompanyAccess/requireCompanyWrite+requireDomainEnabled("competitor"); both self-host and cloud editions verified.Testing
pnpm checkgreen (0 type/lint errors). Per-package: engine competitor suites, DB competitor tests (PGlite), web competitor lib/API/UI suites — all green.next buildcompiles;/competitors,/competitors/manage,/competitors/[id]all dynamic;/api/competitors/[id]/detectpresent. Both-mode gating verified.0015and0016(both additive;0016adds two nullable snapshot columns — one-time re-baseline per source, guarded so it emits no spurious alert).🤖 Generated with Claude Code