Generalize site search for all registered sources and normalize result URLs - #261
Generalize site search for all registered sources and normalize result URLs#261kmatsunami wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c934b5a5db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if normalized.startswith("/"): | ||
| normalized = urljoin(search_url, normalized) |
There was a problem hiding this comment.
_resolve_result_url now only applies urljoin when href starts with /, so links like works/123 or ./series/abc are treated as scheme-less URLs and dropped. The previous implementation always resolved relative links against the page base URL, so this change can silently remove valid search hits (or all hits) for sources whose markup uses non-leading-slash relative paths.
Useful? React with 👍 / 👎.
| title_attr_match = re.search(r'title="([^"]+)"', match.group(0), re.I | re.S) | ||
| title = _normalize_anchor_text(title_attr_match.group(1) if title_attr_match else match.group(2)) | ||
| if not title: | ||
| continue | ||
| seed_url = _normalize_result_url(urljoin(base_url, href)) | ||
| if seed_url in seen_urls: | ||
| continue | ||
| seen_urls.add(seed_url) | ||
| title = canonical_seed_url |
There was a problem hiding this comment.
Title extraction now only checks the anchor title attribute or stripped inner text, but image-only cards (e.g. <a ...><img alt="..."/></a>) produce an empty string after tag stripping. In that case the code falls back to the URL, which regresses user-facing labels compared with the previous champion-cross logic that read img alt text; search results become hard to identify when sites render cards without text nodes.
Useful? React with 👍 / 👎.
1 participant
Motivation
Description
SUPPORTED_SEARCH_SOURCESwithREGISTERED_SOURCESand importednormalize_seed_urlfrommanga_watch.sourcesand switchedquotetoquote_plusfor query encoding._SOURCE_SEARCH_CONFIGto declare each source'ssearch_urlandallowed_domainsand implemented_search_via_public_siteto drive searches for any configured source._extract_anchor_resultsand added helpers_resolve_result_url,_is_allowed_domain, and_canonical_seed_url_for_sourceto resolve relative links, enforce allowed domains, and produce canonical seed URLs vianormalize_seed_url._normalize_result_url, and built_SEARCHERSdynamically so each registered source uses the generic search path.Testing
tests/test_source_search.pywhich assert supported sources and seed URL normalization and they passed.tests/test_discord_command_registration_search.pywhich validates the expanded searchchoicesand it passed.tests/test_discord_supertwins.pywhich checks search invocation behavior and it passed.Codex Task