crawl returns a single page when the seed host redirects (apex→www): same-origin scope uses the pre-redirect origin

[divya0795]

Describe the bug

Crawler.crawl() derives the same-origin scope from the pre-redirect seed URL and never
reconciles it with the seed page's final URL, so crawling any site whose apex redirects to www
(or http → https) discovers zero links and returns a single page.

src/crawl/crawler.ts:38:

const seedOrigin = new URL(input.url).origin;   // pre-redirect

that value is threaded into crawlTraversal(input, seedOrigin, …) and used by filterLinks
(:193):

const parsed = new URL(link);
if (parsed.origin !== seedOrigin) return false;

The fetch layer follows redirects and returns the post-redirect URL — crawlTraversal already uses
it for the result item at :136 (canonicalForOutput(fetchResult.url)) — but seedOrigin still
holds the original. Every link on the fetched page carries the final host, so every link is
rejected as off-origin and the frontier never grows past the seed.

To reproduce

Crawl any host that redirects apex → www, e.g.:

wigolo crawl https://example.com --max-depth 2 --max-pages 20

where https://example.com 301s to https://www.example.com.

Expected behavior

The crawl stays on the seed site and walks it to max_depth / max_pages.

Output / logs

crawled: 1, total_found: 1, pages containing only the seed. The one page returned has
url: https://www.example.com — the final host — which is exactly the mismatch: the output already
reflects the redirect while the scope check does not.

Environment

• wigolo version: main @ 56da0c8
• Node.js version: 20
• OS: any
• Which tool/command: crawl (strategy: bfs / dfs, and the auto fallback path when no sitemap
  is found)

Additional context

The fix I have in mind is to re-anchor the scope once, inside crawlTraversal, after the seed fetch
succeeds — if (depth === 0) scopeOrigin = new URL(fetchResult.url).origin — and pass that to
filterLinks. Depth 0 is only ever the seed, so it fires exactly once and only on a successful
fetch; a malformed final URL falls back to the original origin.

Two adjacent things I deliberately left out of that change, happy to file separately if you'd like
them fixed:

• crawl() fetches robots.txt from the pre-redirect origin (:44), so a redirecting seed reads
  the wrong robots file. respectRobotsTxt is off by default.
• The visited set is keyed on requested URLs only, so two links that redirect to the same target
  are both fetched and indexed.

tests/unit/crawl/crawler.test.ts already has the harness for a regression test — its
makeFetchOutput(url, …) takes the final URL, so a seed whose fetch returns a different host
reproduces this directly.