scraper/.hermes/plans/scraper-downloader-fix-spec.md

Scraper Downloader API Fix Specification

Goal

Fix all 15 broken downloader endpoints that return HTTP 500, replace dead third-party APIs with working alternatives, and re-test with real URLs from real platforms.

Current State Analysis

Working Endpoints (HTTP 200):

Platform	Provider	Notes
TikTok	tikwm.com	✓ Working (tiktok.com API)
MediaFire	mediafire	✓ Direct page scrape
Mega	mega.co.nz	✓ Valid error messages for bad URLs
GDrive	gdrive	✓ Valid error for missing file ID
Pinterest	pinterestdownloader.io	Returns 200 with {success:false, error:"Unknown error occurred."}
Threads	downr.org	Returns 500 (via fetch_snapsave → fetch_all_in_one)

Broken Endpoints (HTTP 500):

Platform	Upstream API	Error
YouTube	media.savetube.me	DNS resolution failure
Instagram	downr.org	403 action_forbidden (Cloudflare)
Facebook	downr.org	403 action_forbidden (Cloudflare)
Twitter	savetwitter.net	API returns 404 "Video not found" but code crashes on missing data field
Spotify	ytdlpyton.nvlgroup.my.id	DNS resolution failure
SoundCloud	ytdlpyton.nvlgroup.my.id	DNS resolution failure
PixelDrain	pixeldrain.com	Code crashes parsing viewer_data (wrong regex?)
KrakenFiles	krakenfiles.com	API returns 403/empty
Danbooru	danbooru.donmai.us	Returns 404
DoodStream	d000d.com	Proxy worker dead
TeraBox	tera2.sylyt93.workers.dev	DNS resolution failure
Bilibili	cobalt instances	Cobalt DNS failures

Root Cause

1. Error Handling Bug: ScrapingError::Http maps to AppError::Internal (HTTP 500) via the From<ScrapingError> impl in error.rs. This was already patched to AppError::ScraperError → HTTP 502, but the running binary hasn't been rebuilt.
2. Dead Upstream APIs: Many third-party APIs have moved/changed/blocked requests from VPS IPs.
3. Error Propagation: Fetch functions use .unwrap() on JSON paths (resp["data"], resp["download_url"]) which panics when the upstream returns an unexpected response structure, causing HTTP 500.

Fix Plan

Fix 1: Robust error handling (no panics on missing JSON fields)

Replace all .unwrap() and direct indexing (resp["key"]) with proper .get() + error handling. When upstream returns an error response, return DownloadResult::error("...") instead of panicking.

Fix 2: Replace dead upstream APIs

YouTube (media.savetube.me → multiple alternatives)

• Primary: Use yt-dlp subprocess via std::process::Command (industry standard, handles all platforms)
• Fallback: https://co.wuk.sh/api/json (Cobalt instance)
• Fallback: https://api.tikmate.app/api/lookup for TikTok

Instagram/Facebook (downr.org → alternatives)

• Primary: Use yt-dlp subprocess
• Fallback chain: snapsave.app/action.php, ddownr.com, qurls.app

Twitter (savetwitter.net → alternatives)

• Primary: Use twitsave.com (which returned 200 but no MP4 for that test URL)
• Add proper handling for 404 status code from savetwitter.net API
• Fallback: https://twitsave.com/info?url=...

Spotify (ytdlpyton → alternatives)

• Use Spotify API for metadata + alternative audio source
• Replace with direct Spotify metadata API (already has a fallback to spotify.com API for metadata)

Pinterest (pinterestdownloader.io → alternatives)

• Use yt-dlp subprocess
• Fallback: https://savepin.app, https://pin-downloader.com

PixelDrain (fix existing code)

• The regex for extracting viewer_data is wrong - test with real URLs

KrakenFiles

• Fix the POST request (the API likely changed)

Danbooru

• The API key may be required now

DoodStream

• The proxy worker (rv.lil-hacker.workers.dev) is dead - need new proxy

TeraBox

• The workers.dev endpoint is dead - find new endpoint

Bilibili

• Use working Cobalt instance or the direct API

Fix 3: Add yt-dlp as universal fallback

Install yt-dlp binary on the VPS. Add a generic download function that shells out to yt-dlp for any platform that fails with its primary API. yt-dlp handles YouTube, IG, FB, Twitter, TikTok, SoundCloud, etc.

Fix 4: Improve DownloadResult error messages

Ensure every error path returns a descriptive DownloadResult::error("...") with platform-specific failure info, not a generic 500.

Files to Modify

1. /home/code/scraper/src/presentation/error.rs - Already patched (ScrapingError → 502)
2. /home/code/scraper/src/infrastructure/repository/downloader.rs - Main fix target
3. /home/code/scraper/Cargo.toml - Add tokio-process for yt-dlp subprocess
4. tests/ - Add integration tests with real URLs (non-flaky)

Verification

• Rebuild and restart the scraper binary
• Test each endpoint with REAL platform URLs
• Verify HTTP status codes are 200 (with success=false data) or 502 (Bad Gateway), never 500