Generalize site search for all registered sources and normalize result URLs - #263
Generalize site search for all registered sources and normalize result URLs#263kmatsunami wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 23f45b635e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from manga_watch.sources.base import HttpClient, RequestsHttpClient | ||
|
|
||
| SUPPORTED_SEARCH_SOURCES: tuple[str, ...] = ("champion-cross", "kakuyomu") | ||
| SUPPORTED_SEARCH_SOURCES: tuple[str, ...] = tuple(REGISTERED_SOURCES) |
There was a problem hiding this comment.
Bound synchronous source fan-out to avoid command timeouts
Making SUPPORTED_SEARCH_SOURCES equal to every registered adapter means supertwins search now fans out synchronously across up to 13 non-root sources (_search_results loops sources one-by-one and each call does blocking HTTP). With the current HTTP defaults (25s timeout and retries), one slow/unreachable site can stall a source search for a long time, so a single interaction can exceed Discord response windows by minutes. This regression is introduced by expanding the source list without adding parallelism or tighter fail-fast behavior.
Useful? React with 👍 / 👎.
| if normalized.startswith("/"): | ||
| normalized = urljoin(search_url, normalized) |
There was a problem hiding this comment.
Resolve all relative href forms before URL validation
_resolve_result_url currently resolves only root-relative links (/path) and drops document-relative links like works/123, ./episode/..., or ../series/... because they reach urlsplit without a scheme and are rejected. That means valid anchors can silently disappear from results on sites that emit non-root relative URLs. The previous implementation used urljoin for all hrefs, so this is a behavior regression in link extraction.
Useful? React with 👍 / 👎.
Motivation
Description
SUPPORTED_SEARCH_SOURCESwithREGISTERED_SOURCESand importednormalize_seed_urlfrommanga_watch.sourcesand switchedquotetoquote_plusfor query encoding._SOURCE_SEARCH_CONFIGto declare each source'ssearch_urlandallowed_domainsand implemented_search_via_public_siteto drive searches for any configured source._extract_anchor_resultsand added helpers_resolve_result_url,_is_allowed_domain, and_canonical_seed_url_for_sourceto resolve relative links, enforce allowed domains, and produce canonical seed URLs vianormalize_seed_url._normalize_result_url, and built_SEARCHERSdynamically so each registered source uses the generic search path.Testing
tests/test_source_search.pywhich assert supported sources and seed URL normalization and they passed.tests/test_discord_command_registration_search.pywhich validates the expanded searchchoicesand it passed.tests/test_discord_supertwins.pywhich checks search invocation behavior and it passed.Codex Task