Generalize site search for all registered sources and normalize result URLs - #264
Generalize site search for all registered sources and normalize result URLs#264kmatsunami wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f061644f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| title_attr_match = re.search(r'title="([^"]+)"', match.group(0), re.I | re.S) | ||
| title = _normalize_anchor_text(title_attr_match.group(1) if title_attr_match else match.group(2)) | ||
| if not title: | ||
| continue | ||
| seed_url = _normalize_result_url(urljoin(base_url, href)) | ||
| if seed_url in seen_urls: | ||
| continue | ||
| seen_urls.add(seed_url) | ||
| title = canonical_seed_url |
There was a problem hiding this comment.
Handle image-only anchors when extracting titles
Search result cards that render as <a><img alt="..."></a> now lose their human-readable title because _extract_anchor_results only looks at the anchor title attribute or text body, then falls back to the URL when both are empty. This is a regression from the previous champion-cross parser (which read img alt), and it will surface in production as URL-looking option labels that make search results hard to distinguish.
Useful? React with 👍 / 👎.
Motivation
Description
SUPPORTED_SEARCH_SOURCESwithREGISTERED_SOURCESand importednormalize_seed_urlfrommanga_watch.sourcesand switchedquotetoquote_plusfor query encoding._SOURCE_SEARCH_CONFIGto declare each source'ssearch_urlandallowed_domainsand implemented_search_via_public_siteto drive searches for any configured source._extract_anchor_resultsand added helpers_resolve_result_url,_is_allowed_domain, and_canonical_seed_url_for_sourceto resolve relative links, enforce allowed domains, and produce canonical seed URLs vianormalize_seed_url._normalize_result_url, and built_SEARCHERSdynamically so each registered source uses the generic search path.Testing
tests/test_source_search.pywhich assert supported sources and seed URL normalization and they passed.tests/test_discord_command_registration_search.pywhich validates the expanded searchchoicesand it passed.tests/test_discord_supertwins.pywhich checks search invocation behavior and it passed.Codex Task