feat: deletion detection + fast scraper intervals + scraper reliability fixes#32
Merged
Conversation
When crawler scans a source listing, any CrawlUrl no longer present is bumped to next_check_at=now so the scraper picks it up in the next batch, fetches it, gets 404, and archives the linked offer. Manually created offers (no CrawlUrl) are HEAD-checked directly by the crawler: filtered by organization, source domain, and crawl_urls__isnull. Published offers → ARCHIVED, drafts → deleted on 404/410. Guard: if live_url_set is empty (network flake), both deletion steps are skipped to prevent mass false-archiving. Scheduler intervals switched from minutes to seconds via env vars CRAWLER_INTERVAL_SECONDS / SCRAPER_INTERVAL_SECONDS (default 1s), updated in docker-compose.yml, backend/.env, and backend/.env.example.
…ype-aware mock template - Remove empty-live-set guard: network errors throw exceptions (caught in run()); a 200 response with 0 URLs is a genuine empty listing, not a flake - Clear crawl_url.offer_id after offer.delete() to avoid Django's 'save() prohibited to prevent data loss due to unsaved related object' ValueError - Update mock_website_detail.html meta description to use type-specific keywords so TF-IDF classifier reliably scores above 0.30 threshold (tested: 0.59-0.67)
…ng UI A 404/410 during deletion detection is expected success — the scraper correctly archives the offer. Previously it emitted event=url_failed (shown as red "error" chip) while errors_count stayed 0, causing a confusing mismatch. Now 404/410 emits event=url_archived, the URL table shows a grey "archived" chip, and only genuine errors (5xx, network timeouts) increment errors_count and show as red.
Draft and published offers both move to ARCHIVED when their source URL returns 404/410. Hard-deleting drafts lost audit trail and any manual enrichment done before publishing.
…for 5xx Any 4xx client error means the resource is gone/inaccessible — archive immediately instead of only 404/410. 5xx server errors keep the existing exponential backoff since they indicate transient server issues. 200 responses returning a generic/empty page also archive the linked offer (if one exists); new URLs with no linked offer still just neglect.
Meta description was enriched with type keywords causing those keywords to be stored as the offer description. Reverted meta to real description only. Type keywords now come from body list items (offer_type field) which feed extracted.details. Classifier now uses title+summary+details so it picks up the body signal.
…scraper APScheduler skipped every tick when crawl took longer than the interval (1s) with multiple sources, flooding logs with 'maximum instances' warnings. Thread loops run fn → sleep N seconds → repeat, guaranteeing no overlap and no skip warnings regardless of how long a crawl takes. APScheduler kept only for infrequent cron jobs (translation, link-checker, archive).
…erministic immediately When Ollama is unreachable (connection refused), models were put in 60s cooldown and _wait_for_available_model slept for ~58s before retrying — blocking the scraper thread indefinitely. Now connection errors (non-429) are tracked separately. If all cooled-down models failed due to connection errors, the wait is skipped entirely and the caller falls through to deterministic classification immediately. Rate-limit (429) cooldowns still wait as before. LLM enabled/disabled per-source via admin panel llm_fallback_enabled flag.
Empty batches (all neglected, nothing logged) no longer create a ScrapingRun DB row — they are deleted before saving. Only runs with actual log entries (created/updated/archived/errors) appear in the UI tracking tab.
…archiving behaviour
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
CRAWLER_INTERVAL_SECONDS/SCRAPER_INTERVAL_SECONDS)_loop(fn, interval, stop)) — crawler/scraper never overlapSCRAPER_TIMEOUT_SECONDS=1(was 30s) for fast failure detectionextract_deterministicstores<meta name="description">as summary; TF-IDF signal from body<li>elements, not injected into descriptionerrors_countreflects only genuine failuresFiles Changed
backend/content/scrapers/queue_service.pybackend/content/scrapers/service.pybackend/content/scrapers/ollama_client.pybackend/content/management/commands/run_scraper_worker.pybackend/content/templates/content/mock_website_detail.htmlbackend/ui/src/app/shared/run-log.parser.tsurl_archivedevent → archived flagbackend/ui/src/app/pages/scrapper-admin-page.component.*docker-compose.ymlbackend/.env/.env.exampleTest Plan
python manage.py test content)tsc --noEmit)