Generalize site search for all registered sources and normalize result URLs - #262
Generalize site search for all registered sources and normalize result URLs#262kmatsunami wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f228197b4d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| title_attr_match = re.search(r'title="([^"]+)"', match.group(0), re.I | re.S) | ||
| title = _normalize_anchor_text(title_attr_match.group(1) if title_attr_match else match.group(2)) | ||
| if not title: | ||
| continue | ||
| seed_url = _normalize_result_url(urljoin(base_url, href)) | ||
| if seed_url in seen_urls: | ||
| continue | ||
| seen_urls.add(seed_url) | ||
| title = canonical_seed_url |
There was a problem hiding this comment.
In _extract_anchor_results, title extraction now only checks the anchor title attribute or stripped inner HTML, and falls back to the URL when that is empty. For card-style search markup where <a> contains only an <img alt="..."> (the case the old champion-cross parser explicitly handled), _normalize_anchor_text returns empty and users see raw URLs instead of work titles in search UI. This degrades result accuracy/usability and is a regression from the previous per-source logic.
Useful? React with 👍 / 👎.
| if normalized.startswith("/"): | ||
| normalized = urljoin(search_url, normalized) |
There was a problem hiding this comment.
_resolve_result_url only calls urljoin when href starts with /, so valid relative links like ./work/123 or work/123 are passed to urlsplit with no scheme and then discarded. Since this module is now generic across many sources, dropping non-root-relative anchors can silently remove legitimate results from otherwise successful searches.
Useful? React with 👍 / 👎.
1 participant
Motivation
Description
SUPPORTED_SEARCH_SOURCESwithREGISTERED_SOURCESand importednormalize_seed_urlfrommanga_watch.sourcesand switchedquotetoquote_plusfor query encoding._SOURCE_SEARCH_CONFIGto declare each source'ssearch_urlandallowed_domainsand implemented_search_via_public_siteto drive searches for any configured source._extract_anchor_resultsand added helpers_resolve_result_url,_is_allowed_domain, and_canonical_seed_url_for_sourceto resolve relative links, enforce allowed domains, and produce canonical seed URLs vianormalize_seed_url._normalize_result_url, and built_SEARCHERSdynamically so each registered source uses the generic search path.Testing
tests/test_source_search.pywhich assert supported sources and seed URL normalization and they passed.tests/test_discord_command_registration_search.pywhich validates the expanded searchchoicesand it passed.tests/test_discord_supertwins.pywhich checks search invocation behavior and it passed.Codex Task