Headless browser crawler with a dark-themed PyQt6 desktop GUI for discovering, cataloging, and downloading stock video clips from multiple sites — with full metadata extraction, FTS5 keyword search, and a concurrent download manager.
git clone https://github.com/SysAdminDoc/Stock-Video-Collector.git
cd Stock-Video-Collector
py -3 -m venv .venv
.\.venv\Scripts\python -m pip install -r requirements.txt
.\.venv\Scripts\python -m playwright install chromium
.\.venv\Scripts\python artlist_scraper.pyPortable mode keeps config, database, thumbnails, and default output beside the app under portable-data:
.\.venv\Scripts\python artlist_scraper.py --portableCreating a portable.flag file beside the script or EXE enables the same mode without command-line arguments.
YouTube Creative Commons ingest is optional and uses yt-dlp for metadata-only collection without a browser:
.\.venv\Scripts\python -m pip install yt-dlpThe setup installs:
- Python packages (
PyQt6,playwright,imageio-ffmpeg) - Chromium for Playwright crawling
- The dark PyQt6 desktop GUI
Requirements: Python 3.9+ — no other prerequisites. Works on Windows, Linux, and macOS.
.\.venv\Scripts\python -m pip install pyinstaller
.\.venv\Scripts\python -m pip install --upgrade -r requirements-lock.txt
.\.venv\Scripts\python tools\build_release.py --verify-only
.\.venv\Scripts\python tools\build_release.pyRelease verification checks the app version surfaces, exact requirements-lock.txt pins, the active venv package set, and a local pip-audit vulnerability report before building.
| Site | Video Types | Metadata | Pagination |
|---|---|---|---|
| Artlist | M3U8 HLS streams | Clip ID, resolution, duration, FPS, camera, formats, creator, collection, tags | Infinite scroll |
| Adobe Stock | Watermarked MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, asset cards, preview metadata | Search/video grids + infinite scroll |
| Shutterstock | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, clip cards, preview metadata | Video/search grids + infinite scroll |
| Envato Elements | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, item cards, preview metadata | Stock-video grids + infinite scroll |
| Motion Array | MP4/WebM/HLS/DASH previews | OpenGraph, JSON-LD, product cards, preview metadata | Stock-video grids + infinite scroll |
| Vimeo | HLS, MP4, WebM, DASH previews | OpenGraph, JSON-LD, channel/card metadata | Channel/group/showcase grids + infinite scroll |
| YouTube CC-BY | Metadata-only YouTube Creative Commons ingest via yt-dlp |
Title, uploader, channel/playlist, duration, resolution, tags, license/attribution | Browser-free URL/playlist/channel ingest |
| Coverr | MP4/WebM/HLS/MOV royalty-free previews | OpenGraph, JSON-LD, slug IDs, license defaults | Video categories, collections, scenes |
| Mazwai | MP4/MOV/WebM/HLS via current Magnific/Freepik redirect | Free video license defaults, numeric IDs, attribution state | Legacy Mazwai URL + Magnific video catalog |
| Videvo | MP4/MOV/WebM/HLS via current Magnific/Freepik redirect | Free video license defaults, numeric IDs, attribution state | Legacy Videvo URL + Magnific video catalog |
| Mixkit | MP4/WebM/HLS/MOV free stock video | Item duration, frame rate, tags, license defaults | Free-stock-video categories |
| Videezy | MP4/WebM/MOV/HLS free stock footage | Category/id pages, tag metadata, per-item license defaults | Newest/popular/category grids |
| RSS/Atom Feed | MP4/WebM/MOV/HLS/DASH video enclosures | Feed title, entry title/link, author, categories, media thumbnail, license URL | Browser-free feed URL ingest |
| Pexels | MP4 direct (SD/HD/UHD via Canva CDN) | OpenGraph + JSON-LD, URL slug titles | Load More button (up to 15 clicks) |
| Pixabay | MP4, WebM | OpenGraph + JSON-LD | Infinite scroll |
| Storyblocks | M3U8, MP4, WebM | OpenGraph + JSON-LD | Infinite scroll |
| Generic | M3U8, MP4, WebM, DASH, MOV | Auto-detect (OG, JSON-LD, DOM) | Infinite scroll |
The Generic profile works on any site — it intercepts all video network requests and extracts whatever metadata is available.
| Feature | Description |
|---|---|
| Stealth mode | Hides navigator.webdriver flag, spoofs plugin array and WebGL vendor/renderer |
| Challenge detection | Auto-detects Cloudflare, CAPTCHA, and challenge pages |
| Manual solve mode | Switches to visible browser for CAPTCHA solving, resumes automatically on clearance |
| Challenge notifications | Optional one-time Discord/Telegram ping when a CAPTCHA or bot challenge appears |
| Persistent profile | Browser session cookies, localStorage, and tokens persist across runs |
| Profile slot rotation | Optional browser_profiles/slot-N rotation for Playwright persistent contexts |
| Proxy pool | Optional proxies.txt support for Playwright browser launches; supports host:port, HTTP(S), SOCKS4, and SOCKS5 entries |
| Residential sticky proxy session | Optional sticky mode keeps crawl, API discovery, and bootstrap browser launches on one selected proxy for the worker session |
| Crawl-budget guardrails | Optional robots.txt crawl-delay/request-rate enforcement plus Retry-After and X-RateLimit cooldowns |
| Request interception | Blocks heavy HLS .ts segments during crawl to save bandwidth |
| Configurable delays | Page delay, scroll delay, M3U8 wait, timeout — all adjustable per-run |
| Secret-safe config | API keys, auth headers, cookies, and proxy credentials are stored through the OS keyring when available, with encrypted local fallback |
| Official API connectors | Optional API-first crawling for Pexels, Pixabay, Vimeo, and Adobe Stock when keys/tokens are configured, with browser fallback |
| yt-dlp ingest mode | Optional browser-free YouTube CC-BY metadata ingest for Creative Commons URLs, playlists, and search results |
| RSS/Atom feed ingest | Browser-free XML feed ingest for feeds that expose video enclosures or media:content items |
| Fetch safety policy | App-initiated HTTP(S) fetches block localhost, private/link-local networks, metadata services, ambiguous IP literals, and redirect-to-private targets |
| App-data migration | Legacy ArtlistScraper config, vault, database, and backup files are copied into StockVideoCollector without overwriting current data |
| Portable mode | --portable, STOCK_VIDEO_COLLECTOR_PORTABLE=1, or portable.flag stores config, database, thumbnails, and default output under the app directory |
| Failed-crawl diagnostics | Browser crawls can save redacted trace, HTML snapshot, screenshot, and network log bundles for replay/debugging |
The crawler uses four complementary strategies to find video URLs on every page:
┌───────────────────────────────────────────────────────────────────┐
│ Page Load │
├───────────────┬─────────────────┬─────────────┬───────────────────┤
│ XHR/Fetch │ DOM Observer │ Response │ HTML Regex │
│ Intercept │ (MutationObs) │ Body Scan │ Fallback │
│ │ │ │ │
│ Hooks into │ Watches for │ Scans all │ Regex sweep for │
│ XMLHttpReq & │ <video src> │ HTTP resp │ M3U8/MP4/WebM │
│ fetch() API │ injections │ bodies │ + Canva partner │
│ │ │ │ links (Pexels) │
└───────┬───────┴────────┬────────┴──────┬──────┴────────┬──────────┘
│ │ │ │
└────────────────┴───────┬───────┴───────────────┘
▼
┌─────────────────────┐
│ Quality Comparison │
│ UHD > HD > SD │
│ Dedup by clip ID │
└──────────┬──────────┘
▼
┌─────────────────────┐
│ SQLite Database │
│ + FTS5 Index │
└─────────────────────┘
| Feature | Description |
|---|---|
| SQLite with WAL mode | Concurrent reads, crash-safe writes |
| FTS5 full-text search | Search across title, creator, collection, tags, resolution, camera, duration |
| AND/OR search modes | Toggle between inclusive and exclusive multi-term search |
| Column filters | Filter by source site, resolution, creator, collection — all combinable with text search |
| Duration filter | Quick filter by clip length range |
| Saved searches | Save and recall frequent search + filter combos |
| FTS index rebuild | One-click repair if search results drift out of sync |
| Backup catalog | Clearing the database creates verified SQLite backups with SHA-256 catalog, selected restore, and retention pruning |
| License provenance | Source license, embedded rights tags, attribution requirement, terms URL, and preview/watermark status are stored with clips when known |
| Feature | Description |
|---|---|
| Star ratings | 1–5 star rating per clip |
| Favorites | Quick-toggle favorite flag for any clip |
| Notes | Free-text notes per clip |
| User tags | Custom tag system independent of source tags |
| Collections | Organize clips into named collections with color coding and lock/unlock for curation |
| Tag editor | Bulk rename, merge, and split user tags across all clips |
| Bulk operations | Context menu actions on any card in the grid |
| Duplicate review | Exact SHA-256 and near visual-hash groups with Library filter and keep/ignore/review actions |
| Feature | Description |
|---|---|
| Concurrent downloads | Configurable parallel download workers (default: 2) |
| ffmpeg HLS→MP4 | Automatic M3U8-to-MP4 conversion via ffmpeg |
| yt-dlp fallback | Automatic fallback to yt-dlp when ffmpeg download fails |
| Retry with backoff | Exponential backoff retry (configurable max attempts) |
| Atomic writes & validation | Downloads write to .part, validate the video stream, then atomically promote to MP4 |
| Accurate HLS resume | Validates partial downloads on resume and promotes or re-downloads |
| Priority queue | High/Normal/Low download priorities with priority queue ordering |
| Per-site concurrency caps | Configurable max concurrent downloads per source site |
| Bandwidth schedule | Time-based speed profiles (throttle daytime, business hours, nights only) |
| Post-download transcode | Automatic transcode presets: H.264 1080p proxy, HEVC 4K archive, ProRes master |
| Speed & ETA tracking | Real-time download speed and estimated completion time |
| Bandwidth limiting | Optional download speed cap |
| Filename templates | Customizable output filenames: {title}, {clip_id}, {creator}, {collection}, {resolution} |
| Sidecar metadata | JSON metadata file with stock-video-collector.provenance.v1 sidecar-only provenance, written alongside each downloaded MP4 |
| Thumbnail extraction | Auto-extracts thumbnails and records retryable HTTP/ffmpeg failure diagnostics |
| Format | Contents |
|---|---|
.txt |
Plain list of M3U8/MP4 URLs |
.json |
Full metadata for all clips (title, creator, tags, URLs, timestamps, license/provenance fields, embedded rights report) |
.m3u |
Media player playlist — uses local path if downloaded, M3U8 URL otherwise |
.csv |
Spreadsheet-ready with clip, source, license/provenance, embedded rights, and duplicate-review metadata columns |
.xml |
Adobe Premiere / DaVinci Resolve bin (xmeml v5) with resolution metadata and local file paths |
.fcpxml |
Final Cut Pro (FCPXML 1.9) with asset resources and spine clips |
| Batch | Export all four formats at once |
| Handoff package | Portable folder + ZIP with copied media, thumbnails, sidecars, provenance manifest, attribution file, and SHA-256 checksums |
| Feature | Description |
|---|---|
| Dark theme | Catppuccin-inspired deep dark palette |
| Card grid view | Visual thumbnail grid with configurable card sizes (S/M/L) |
| Hover video preview | Mouse-over any card to preview the video inline |
| Detail panel | Always-visible side panel with full metadata, ratings, notes, tags, collections |
| System tray | Minimize to tray, continue crawling/downloading in background |
| Toast notifications | Non-blocking status notifications |
| Live crawl log | Real-time scrolling log with verbose/quiet toggle |
| Clipboard monitor | Opt-in URL detection from clipboard (auto-fills crawl URL input) |
| Scheduled crawls | Automatic re-crawl at configurable intervals (1–168 hours) |
| Watch folder | QFileSystemWatcher auto-imports new video files with ffprobe metadata |
| Saved-search feeds | New-match notifications when saved search results grow |
| Watermark stripper | Batch ffmpeg delogo filter for user-owned watermarked clips |
| User site plugins | Drop-in user_profiles/*.py modules auto-register without touching core |
| Accessibility metadata | Major controls, status surfaces, and media cards expose accessible names/descriptions |
- Select a site profile or feed mode — check one or more profiles in the Crawl tab, or select RSS/Atom Feed mode and paste a feed URL with video enclosures
- Set the start URL — auto-populated per profile, or paste any URL for Generic mode
- Configure crawl settings — batch size, depth, delays, headless mode
- Start crawling — the crawler discovers pages, extracts metadata, and intercepts video URLs
- Browse results — switch to the Library tab to search, filter, rate, tag, and organize clips
- Download — select clips and download with the built-in manager, or export URL lists for external tools
All settings persist automatically in a JSON config file. Key options:
| Setting | Default | Description |
|---|---|---|
| Batch size | 50 | Pages per crawl batch |
| Page delay | 2s | Wait between page loads |
| Scroll delay | 1s | Wait between scroll steps |
| M3U8 wait | 5s | Time to wait for video URLs to appear |
| Scroll steps | 10 | Number of scroll-down actions per page |
| Timeout | 30s | Page load timeout |
| Respect robots/rate limits | On | Fetch robots.txt per host, enforce crawl-delay/request-rate, and cool down from Retry-After/X-RateLimit headers |
| Min host gap | 0 ms | Optional minimum delay between browser navigations to the same host |
| Max pages | 0 (unlimited) | Stop after N pages |
| Max depth | 3 | Link-following depth |
| Headless | On | Run browser without visible window |
| Challenge notifications | Off | Send a Discord webhook or Telegram bot message when a browser challenge is detected |
| Rotate browser profiles / slots | Off / 4 | Use separate persistent browser-profile directories between crawl sessions |
| Proxy pool / path | Off / app-data proxies.txt |
Route Playwright browser launches through one proxy entry from a user-supplied proxy file |
| Residential sticky proxy session | Off | Keep crawl, API discovery, and bootstrap browser launches on one selected proxy during the same worker session |
| Use official APIs | On | Use configured official API connectors before browser crawling |
| API search query | Blank | Optional query for API-backed Pexels/Pixabay/Vimeo/Adobe Stock searches |
| API pages / per page | 3 / 40 | Quota-aware pagination limits for official API connector crawls |
| Concurrent DLs | 2 | Parallel download workers |
| Max retries | 3 | Download retry attempts |
| Bandwidth limit | 0 (unlimited) | Download speed cap in KB/s |
| Clipboard monitor | Off | Auto-detect URLs from clipboard |
Customize download filenames using template variables:
{title} → Beautiful_Sunset.mp4
{clip_id}_{title} → abc123_Beautiful_Sunset.mp4
{creator}/{collection}/{title} → JohnDoe/Nature/Beautiful_Sunset.mp4
Available variables: {title}, {clip_id}, {creator}, {collection}, {resolution}
┌─────────────────────────────────────────────────────────────────────────┐
│ PyQt6 GUI │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Crawl │ │ Library │ │ Detail │ │ Download │ │ Export │ │
│ │ Tab │ │ Tab │ │ Panel │ │ Tab │ │ Tab │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬────┘ │
└───────┼──────────────┼──────────────┼──────────────┼──────────────┼─────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌──────────────┐ ┌──────────────────────────┐ ┌──────────────────────┐
│ Crawler │ │ SQLite + FTS5 │ │ Download Worker │
│ Worker │──│ │──│ │
│ (QThread) │ │ clips, crawl_queue, │ │ ThreadPoolExecutor │
│ │ │ crawled_pages, │ │ + ffmpeg HLS→MP4 │
│ Playwright │ │ collections, │ │ + retry backoff │
│ Chromium │ │ saved_searches │ │ + speed tracking │
└──────────────┘ └──────────────────────────┘ └──────────────────────┘
Crawler Worker — Runs Playwright in an async event loop on a dedicated QThread. Navigates pages, injects JavaScript hooks for XHR/fetch/DOM video interception, extracts metadata via regex selectors + OpenGraph + JSON-LD, and manages the crawl queue with depth/priority.
Database Layer — Thread-safe SQLite with WAL mode and a dedicated threading.Lock. FTS5 external content table indexes title, creator, collection, tags, resolution, camera, and duration. Quality-aware M3U8 URL upgrades prefer UHD over HD over SD.
Download Worker — Persistent queue on a QThread with a ThreadPoolExecutor for concurrent downloads. Handles M3U8→MP4 conversion via ffmpeg, exponential backoff retry, real-time speed/ETA calculation, sidecar JSON metadata, and thumbnail extraction.
"Chromium not found" — Click the "Install Browser" button on the Crawl tab. This runs playwright install chromium automatically. Direct HTTP mode can still start without Chromium.
YouTube CC-BY ingest says yt-dlp is missing — Install the optional CLI in the active environment with .\.venv\Scripts\python -m pip install yt-dlp, then select "yt-dlp Ingest" as the crawl mode.
Search results seem wrong or incomplete — Click the "🔄 Rebuild Index" button on the Crawl tab to rebuild the FTS5 search index from scratch.
Bot challenge / CAPTCHA detected — Uncheck "Headless" mode and restart the crawl. The browser will open visibly so you can solve the challenge manually. Optional Challenge Notifications can ping a configured Discord webhook or Telegram bot/chat when the challenge appears.
Downloads fail repeatedly — Check that ffmpeg is installed and on your PATH. The scraper auto-detects ffmpeg in common locations, but if it can't find it, downloads that require HLS→MP4 conversion will fail.
Downloaded files show as invalid — Archive verification validates local videos with ffprobe when available and falls back to an ffmpeg stream check. Invalid files are reset to pending from the Archive tab so they can be re-queued.
A crawl page fails repeatedly — Enable "Save failed-crawl diagnostics" in Configure. Failed browser pages write redacted diagnostic bundles under the app data crawl_traces folder with metadata.json, snapshot.html, screenshot.png, network.har, and Playwright trace.zip.
Clipboard monitor not working — The clipboard monitor is opt-in. Enable it in your config by adding "clipboard_monitor": true, or toggle it programmatically. On Linux/Wayland, clipboard access may require additional permissions.
Config contains __secret_ref__ values — Sensitive settings are stored through the OS keyring when available, or the local encrypted vault when not available, and are hydrated automatically at runtime.
Where is app data stored? — Profile mode stores config, vault, database, backups, browser profile, and thumbnail cache under StockVideoCollector. Portable mode stores them under portable-data beside the script/EXE. Existing ArtlistScraper config, vault, database, and backups are copied forward only in profile mode without deleting or overwriting legacy data. The Archive tab shows the active config mode and paths.
MIT License — see LICENSE for details.
Issues and PRs welcome. If you add support for a new site, add a module under profiles/ with a build(SiteProfile) factory and CONTRACT fixtures for URL matching, clip IDs, and video URL filtering.