feat: integrated live scraper for anime-sama.to - #28
Conversation
Replace the dependency on the external ani-data JSON repository with an integrated scraper for anime-sama.to. The catalog is now fetched live at search time instead of relying on a static file that goes stale. Changes: - Add Source trait (src/sources/mod.rs) for extensible multi-site support - Implement anime-sama.to scraper with search, seasons and episodes - Multi-source video host selection (sibnet > vidmoly > sendvid) - Auto-detect VF/VA languages via HEAD probing - Switch UX from full catalog listing to search-first flow - Add SCRAPER.md documentation for contributors - Remove unused deps (serde, serde_json, chrono, directories) - Remove obsolete data.rs and anime.rs
- Per-episode host selection: instead of picking one source array, build a hybrid list with the best supported host for each episode position across all sources. Fixes cases where anime-sama mixes hosts within a single source array (e.g. sibnet with vidmoly gaps). - Browser fallback: when no yt-dlp-supported host is available, open the embed URL in the system default browser instead of crashing with cryptic yt-dlp errors. Cross-platform (xdg-open, open, start). - Remove vidmoly.to from supported hosts list (yt-dlp has no extractor for it).
|
Hello 👋 Your live scraper is really great, but it ran into the exact problem I was worried about. The anime-sama.to domain is currently down, so ani-dl no longer works 😭😭😭 |
anime-sama rotates its domain regularly (.to, .xyz, .org, .fr...), so a hardcoded host breaks the tool every time the domain moves — exactly the fragility flagged on the PR. The base URL is now resolved at startup (src/config.rs), by precedence: 1. ANIME_SAMA_BASE_URL env -> explicit override, no probing 2. ANIME_SAMA_MIRRORS env / ~/.config/ani-dl/mirrors.txt -> probed, first alive wins 3. built-in default mirror list mirrors.txt is auto-created (self-documented) on first run, so a domain change is fixed by editing a text file instead of recompiling. AnimeSama now holds a base_url field instead of a const. No new dependencies.
|
Hey @FeuGuillaume186, thanks a lot for testing it and for the kind words! 🙏 You're completely right — and that single-domain dependency was the real weak point. I just pushed a fix for exactly that: the
It won't prevent anime-sama from going down (we're a bit at their mercy 😅), but it makes the tool resilient to domain rotation and trivial to keep alive. |
|
Hey @Tetardtek, I tested your latest update for the live scraper, and everything works perfectly 🎉🎉 I've been using it for several days now, and it's been working really well. For now, the anime-sama.to domain still seems to be up, and I noticed that you've left the possibility to add scrapers for other websites, which is quite interesting. In any case, everything is working great on my end. Thanks 😊✨ |
3 participants
Closes #27
What this does
Replaces the static
ani-dataJSON dependency with a live scraper for anime-sama.to. No more stale links, no more manual catalog updates — episodes are fetched fresh at search time.How it works
Simple 3-step pipeline:
POST /template-php/defaut/fetch.php→ matching animespanneauAnime()calls + HEAD probe for VF/VA availabilityepisodes.js, pick best video host (sibnet > vidmoly > sendvid)The scraper is built on a
Sourcetrait (src/sources/mod.rs) so adding support for other catalog sites is straightforward — just implement the 3 methods. SeeSCRAPER.mdfor details.Changes
src/sources/mod.rs—Sourcetrait + shared types (AnimeResult,SeasonInfo)src/sources/anime_sama.rs— anime-sama.to scraper implementationsrc/main.rs— search-first UX flow (replaces full catalog listing)SCRAPER.md— documentation for contributors wanting to add new sourcessrc/data.rs,src/anime.rs(no longer needed)serde,serde_json,chrono,directoriesscraper,regexKey features
epsNarrays fromepisodes.js, picks hosts that yt-dlp supports (skips unsupported ones like embed4me)panneauAnime()calls, so the scraper probes for them with HEAD requestsSourcetrait makes it easy to add new sites (ES, EN, etc.)Testing
I've been testing locally and shared the fork with a few people. Works on the animes I've tried so far. Would appreciate more testing across the catalog before merging.
Happy to iterate on feedback!