Compare commits

..
37 Commits
Author SHA1 Message Date
asepharyana 7d68ff5620 fix(downloader): terabox proxy as primary link + accept threads.net URLs
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- Terabox: return the working cookie-authenticated proxy URL as the
  FIRST media entry; keep the raw dlink second with an honest note
  (it 403s without cookies).
- Threads: accept threads.net URLs in the snapsave/IG validator so
  /download/threads no longer rejects them.
2026-09-07 13:40:32 +07:00
asepharyana 8b9bdeb56a fix(downloader): real FB media for all URL shapes + public terabox proxy
Facebook:
- Broaden URL validation to accept ALL real FB shapes: /watch/?v=,
  /reel/:id, /videos/:id (user vids), /video.php, /story.php, fb.watch,
  and no-www facebook.com. Old regex rejected /reel, /videos, fb.watch.
- Add yt-dlp as the PRIMARY provider (authoritative, returns direct
  fbcdn progressive URLs with real title/format metadata, no junk).
  Only falls back to downr.org then Playwright. This kills the
  'link exists but 404s' bug where Playwright harvested broken
  bytestart byte-range chunks alongside the one good URL.
- Filter Playwright-harvested bytestart= chunk URLs + dedupe.

TeraBox:
- Add /proxy/terabox streaming endpoint that forwards the resolver's
  cookie-authenticated stream through the public API.
- Rewrite the returned download URL to the public proxy (raw dlink
  403s without cookies; TERABOX_PUBLIC_BASE_URL env override).
- Fix: server_port env is APP_SERVER_PORT (single underscore); the
  OLD stale 4091 bind in tests came from APP__SERVER_PORT not mapping.

flake: bump scraper-0.1.0 -> 0.1.1
2026-09-07 13:20:11 +07:00
asepharyana 029837d0f1 feat(downloader): youtube progressive-first + server-side merge
- /download/youtube now PREFERS a progressive MP4 (single file with
  video+audio) using the android player client — full-speed from VPS,
  no datacenter rate-limit (verified: itag 18 downloads 28MB in ~3s).
- Falls back to adaptive bestvideo+bestaudio (2 URLs) when the requested
  quality has no progressive format.
- New param merge=true triggers server-side ffmpeg merge into one MP4,
  served from GET /file/yt_merge/{name} (temp, 2h TTL cleanup).
- /download/youtube/mp3: report the REAL container ext (webm/opus)
  instead of hardcoded 'mp3'.
2026-09-05 20:11:16 +07:00
asepharyana 8a55fd91a4 fix(youtube): filter m3u8/hls protocol streams + check requested_formats
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- Filter by protocol (m3u8_native) not just ext (m3u8) since yt-dlp
  returns ext=mp4 with protocol=m3u8_native for HLS streams
- Check requested_formats before formats[]: when yt-dlp uses a merged
  format selector, the actual URLs are in requested_formats, while
  formats[] is the full list of all individual format IDs
- In requested_formats branch, use vcodec/acodec to classify formats
  correctly (video-only, audio-only, or muxed)
2026-09-02 18:46:13 +07:00
asepharyana 5cec284cd6 fix(youtube): filter broken formats, use bestvideo+bestaudio for merged output
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- Change format string from bestvideo (video-only) to bestvideo+bestaudio
  so the primary result has both video and audio streams
- Add --merge-output-format mp4 for clean container output
- Skip m3u8/hls_native, m4a, mhtml formats (unusable individual streams)
- Only expose direct HTTPS URLs (skip non-http protocol URLs)
- Remove Audio media type variant from YouTube path (video+audio merged)

Root cause: YouTube DASH provides video-only and audio-only separate formats.
Old code used bestvideo format selector (video-only) and returned ALL format
URLs including broken ones. Fix: use merged format selector + filter list.
2026-09-02 18:31:13 +07:00
asepharyana adfbbfcea7 feat(downloader): route terabox via local Playwright resolver
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Replace dead tera2.sylyt93.workers.dev (NXDOMAIN) with a local
terabox-resolver service (systemd, :4092) that uses Playwright +
residential proxy + user cookies to resolve TeraBox share links
end-to-end to real signed dlink. Proven path: navigate terabox.app
directly, duplicate cookies across .terabox.app domains, call
/share/list from browser context. Supports directories (child files).
2026-09-02 16:20:34 +07:00
asepharyana 09c706cc61 feat(stalk): remove fake genshin stalk
Deploy Scraper / build-and-deploy (push) Canceled after 0s
live-probed: /stalk/genshin was NOT real — it just pass-through the raw
SvelteKit __data.json (only nickname/level/nameCardId/profilePicture),
not a full Genshin profile like the Shirokami source (which parses
characters/element/costume/stats via enka-network-api -> get_profile_for_uid).
That proper API is now 404 without an enka token. Removed.

stalk now has 3 real endpoints: github, youtube, twitter.
2026-09-02 13:39:54 +07:00
asepharyana 93828c7253 feat(stalk): remove dead mobile-legends, free-fire, tiktok, instagram
Audit live-probed all 8 stalk upstreams today. 4 alive (github, youtube,
genshin enka, twitter fxtwitter). 4 dead with no free alternative reachable
from VPS: mobile-legends (gempaytopup /stalk-ml now 405), free-fire
(discordbot.freefirecommunity dead 404), tiktok (WAF -> login page, no
rehydration data), instagram (mollygram captcha; i.instagram requires
login/rate-limited 401). All free alternates probed dead (onrender FF 404,
freefireapi.me 404, i.instagram 401 require_login, m.tiktok 403).

Keep 4 alive: github, youtube, genshin, twitter. stalk.rs now clippy-clean.
2026-09-02 13:29:39 +07:00
asepharyana 5470d74921 feat: remove ai, uploader, and weebs modules
Remove all AI chat/gemini/image endpoints, ryzencdn uploader/file-serve
endpoints, and weebs anime-info/manga-info/whatanime/waifu endpoints.
14 files changed, 1032 lines deleted. cargo check clean, all tests pass.
2026-09-02 13:22:00 +07:00
asepharyana bbb8bed389 docs: add utoipa OpenAPI annotations to all scraper endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Add #[utoipa::path] + IntoParams to all new category handlers:
- downloader (25 routes): /download, /detect-platform, 23 platform handlers
- misc (4): currency-converter, harga-emas, kurs-bca, server-info
- stalk (8): github, youtube, ml, free-fire, genshin, twitter, tiktok, instagram
- weebs (5): anime-info, manga-info, whatanime, waifu (sfw/nsfw)
- search (5): bmkg, jadwal-sholat, weather, google, yt
- tool (6): whois, ip-location, tinyurl, check-hosting, hargapangan, cek-resi
- image (2): brat, brat-animated
- uploader (2): ryzencdn, serve-file
- ai (3): chat, gemini, image

Register all in ModuleApiDoc paths() with per-category tags.
Fix clippy warnings in touched handlers (expect/unwrap -> map_err, needless lifetimes).

/docs now documents the full scraper API surface (was only anime/komik).
2026-09-01 22:39:02 +07:00
asepharyana e6f543078b feat(ai): add pollinations chat, gemini, image endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Ported from Shirokami-API scraper/ai/pollinationsai-{text,gemini,image}.js:
- /ai/chat - OpenAI-compatible chat via text.pollinations.ai (default model 'openai'; source's gpt-5-nano is dead upstream, 404 now triggers fallback)
- /ai/gemini - gemini model via same endpoint
- /ai/image - flux image gen via image.pollinations.ai (returns raw bytes, sniffs JPEG/PNG mime)

Note: pollinations is a free shared API - intermittently 402/502/empty from VPS IP (upstream flakiness, not code)
2026-09-01 20:57:50 +07:00
asepharyana 2769b11918 feat(uploader): add ryzencdn local-disk uploader (POST multipart + GET serve)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Shirokami-API source uploads to Ryzumi S3 (s3.ryzumi.vip) which is DNS-dead
from this VPS. Implemented a local-disk uploader keeping the source's response
shape {success, url, fileName, size}:
- POST /uploader/ryzencdn (multipart 'file' field, magic-byte ext detection)
- GET /uploader/file/{name} (path-traversal safe, MIME by extension)
Files stored under /var/lib/scraper/uploads (owned by scraper user).
2026-09-01 20:39:09 +07:00
asepharyana 8d74d04b43 feat(stalk): add mobile-legends, free-fire, genshin, twitter, tiktok, instagram
Ported from Shirokami-API scraper/stalk/*.js:
- /stalk/mobile-legends (gempaytopup CSRF+POST)
- /stalk/free-fire (freefirecommunity API + field transform)
- /stalk/genshin-impact (enka.network __data.json)
- /stalk/twitter (fxtwitter API)
- /stalk/tiktok (rehydration JSON parse)
- /stalk/instagram (mollygram HTML scrape)

Upstream status: twitter/genshin verified working live (elonmusk 241M followers).
github+youtube verified earlier. tiktok WAF-blocked, instagram mollygram-captcha
blocked, mobile-legends gempaytopup changed POST->GET+JS (all upstream dead for
VPS IP; code faithful to source).
2026-09-01 20:32:10 +07:00
asepharyana 4efcf1f0ea feat(image): add brat + animated brat image generation
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Ported from Shirokami-API scraper/image/brat.js (v2 path):
- /image/brat - static brat PNG via brat.siputzx.my.id API
- /image/brat/animated - animated brat GIF
Both return raw image bytes with proper Content-Type (image/png, image/gif)
2026-09-01 20:17:32 +07:00
asepharyana 278d4fe32a feat(tools): add whois(RDAP), iplocation, tinyurl, check-hosting, cek-resi, hargapangan
Ported from Shirokami-API scraper/tool/*.js:
- /tool/whois - RDAP lookup (whois.com HTML .df-raw is dead/JS-rendered since 2026; RDAP returns structured register/expiry/registrar data)
- /tool/iplocation - ipapi.co JSON
- /tool/tinyurl - tinyurl.com API
- /tool/check-hosting - hosting-checker.net API
- /tool/cek-resi - cekresi.com AES-128-CBC encrypted form POST
- /tool/hargapangan - Bapanas government API (upstream unreachable from VPS)

Fixes: scraper crate has no :has-text() pseudo-class -> replaced history-table scan with 2-cell-row detection; Html held across .await makes future non-Send -> scoped doc parsing
2026-09-01 20:12:52 +07:00
asepharyana 7ffc1bdd81 feat(search): add bmkg, jadwal-sholat, weather, yt-search, google endpoints 2026-09-01 19:51:34 +07:00
asepharyana 091a68bd55 feat(weebs): add anime/manga info, whatanime, waifu endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Ported from Shirokami-API scraper/weebs/*.js:
- /weebs/anime-info (jikan.moe anime search)
- /weebs/manga-info (jikan.moe manga search)
- /weebs/whatanime (trace.moe reverse image search)
- /weebs/sfw-waifu + /weebs/nsfw-waifu (waifu.pics)

Tested: anime-info (Naruto real data) + whatanime (real match). manga-info
+ waifu currently report upstream 504/DNS-down (jikan manga endpoint &
waifu.pics unreachable from VPS right now) - proxy logic verified via the
working anime/whatanime siblings.
2026-09-01 19:35:32 +07:00
asepharyana 242e3498fe feat(stalk): add github + youtube profile stalk endpoints
Ported from Shirokami-API scraper/stalk/{github,youtube}.js:
- /stalk/github (GitHub public API profile)
- /stalk/youtube (ytInitialData parse from /videos subpage:
  handles lockupViewModel (new), videoRenderer, gridVideoRenderer,
  shortsLockupViewModel; channel metadata + top-5 videos)

Also fix misc harga-emas main price to smallest-gram (1gr).
2026-09-01 19:31:37 +07:00
asepharyana 6df8af54eb feat(misc): add misc endpoints (currency-converter, harga-emas, kurs-bca, server-info)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Ported from Shirokami-API scraper/misc/*.js:
- /misc/currency-converter (wise.com scrape, amount/from/to)
- /misc/harga-emas (harga-emas.org Antam table)
- /misc/kurs-bca (bca.co.id jual/beli)
- /misc/server-info (OS/CPU/RAM/disk from /proc)

Also: remove /download/tiktok/v2 + /download/twitter/v2 as separate routes;
douyin.wtf + twitsave now internal fallback steps in the primary
fetch_tiktok/fetch_twitter chains (v2 = fallback per user directive).
2026-09-01 19:05:42 +07:00
asepharyana dc44085cad feat: wire videy + tiktok/v2 + twitter/v2 downloader endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Ported all missing downloader endpoints from Shirokami-API source
(verified against /home/code/Shirokami-API/scraper/downloader/):

- /download/videy: direct CDN link build (cdn.videy.co/{id}.mp4|.mov)
  matches videy.js exactly (id.len()==9 && id[8]=='2' -> .mov)
- /download/tiktok/v2: douyin.wtf hybrid API w/ fallback to embed scrape
- /download/twitter/v2: twitsave.com HTML scrape w/ fallback to Syndication API
- detect_platform now recognizes videy.co
- download_all_in_one dispatches videy

fetch_videy already existed as dead code (never wired); connected it.
== all others (bilibili/danbooru/dood/gdrive/instagram/kfiles/mediafire/
== mega/pinterest/pixeldrain/savetube/soundcloud/spotify/terabox/threads/
== tiktok/twitter/youtubeV2) already mapped to their Rust equivalents
2026-09-01 18:36:54 +07:00
asepharyana dfa37f86e3 feat: add Streamable downloader endpoint + fix detect false-positive
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- /download/streamable: real MP4 via yt-dlp (verified ftypisom)
- Fix detect_platform: t.co substring greedily matched reddit.com
  ('.reddit.com' contains 't.co') - now requires t.co/ path
- Removed duplicate reddit branch in detect chain
2026-09-01 18:23:49 +07:00
asepharyana 7850b5fa03 feat: add Dailymotion + Reddit downloader endpoints via yt-dlp
Deploy Scraper / build-and-deploy (push) Canceled after 0s
New platforms from Shirokami-style downloadter set:
- /download/dailymotion: real MP4/HLS streams (288p up to 2160p)
- /download/reddit: real v.redd.it HLS media for public posts
- detect + all-in-one dispatch updated

Both verified working from VPS with real URLs.
2026-09-01 18:05:35 +07:00
asepharyana 4dee99cc43 feat: Spotify via oEmbed+ytsearch, Pinterest via pinterest-dl CLI scraping
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Spotify (was returning DRM metadata stub):
- Resolve track title via Spotify oEmbed API (no auth, works from VPS)
- Search YouTube Music via yt-dlp ytsearch: -> return real audio stream URL
- Provider: spotify-oembed+ytsearch

Pinterest (was returning 'No download URLs found'):
- Replace dead pinterestdownloader.io API with pinterest-dl CLI scraping
- pinterest-dl handles guest-token/cookie dance, returns real media for
  public pins: HLS video stream (v1.pinimg.com/videos/*.m3u8) + poster image
- Provider: pinterest-dl, with Playwright fallback retained

Both verified live: Spotify returns real googlevideo audio URL (itag=251,
3.4MB); Pinterest returns real v1.pinimg.com HLS streams (200 verified).
2026-09-01 17:14:38 +07:00
asepharyana 6b3fad1ad4 feat: Pinterest Playwright fallback + Spotify graceful DRM message
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- fetch_pinterest: pinterestdownloader.io API failures now non-fatal, fall
  through to Playwright scraping (captures v.pinimg.com video URLs); returns
  graceful 200 'No download URLs found' instead of 502 on dead API.
- fetch_spotify: catch yt-dlp DRM error and return 200 with track metadata +
  clear message (direct audio needs Spotify Premium). No more raw 502.
2026-09-01 16:36:36 +07:00
asepharyana 563bce5211 fix: Twitter/X via Syndication API returns real MP4s (twitter-syndication provider)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Use cargo tokenless https://cdn.syndication.twimg.com/tweet-result?id={id}&token=0
which returns tweet JSON with video variants (MP4 at multiple bitrates).
Works server-side with no auth, wired as primary before savetwitter/yt-dlp.
2026-09-01 16:22:43 +07:00
asepharyana 134fde4ac2 fix: TikTok via embed-page scraping returns real MP4 (tiktok-embed provider)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Scrape https://www.tiktok.com/embed/v2/{id} HTML and extract the direct
v16m.tiktokcdn.com MP4 from the <video data-testid=play-video> tag.
Works server-side (no auth) while main site/API are Cloudflare-blocked.
Verimplemented before tikwm/yt-dlp/Playwright fallbacks.
2026-09-01 16:17:26 +07:00
asepharyana 4684a6a75c fix: make scrape_media.py path resolution robust + include it in Nix build
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- Nix flake now installs scrape_media.py to $out/bin so CI deploys it with the binary
- run_playwright_scraper locates the script alongside the running binary (Nix store),
  the Cargo manifest dir (dev), /home/code/scraper, or $SCRAPER_SCRIPT_DIR
- Fix var_os() Option match
2026-09-01 15:40:57 +07:00
asepharyana 4c5dfdb156 feat: add Playwright browser scraping fallback for Instagram/Facebook/Twitter
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- Add scrape_media.py using Playwright+Chromium to bypass Cloudflare/anti-bot
- Add run_playwright_scraper() Rust helper (spawns Python, probes venv)
- Add playwright_to_download_result() JSON->DownloadResult converter
- Instagram/Facebook: try downr.org, fall back to Playwright (returns real cdninstagram/fbcdn URLs)
- Twitter: scope scraper::Html/Selector parsing in a block so the future stays Send, then Playwright fallback
- Revert --impersonate chrome (unsupported on Linux) to --user-agent
- Install Chromium browsers to /usr/local/share/ms-playwright for all users
2026-09-01 15:35:20 +07:00
asepharyana dd490535fc fix: integrate yt-dlp for Bilibili/SoundCloud/TikTok fallbacks; fix Bilibili BV ID regex; fix stdout pipe buffer truncation (Stdio::piped); fix duplicate comment lines
Deploy Scraper / build-and-deploy (push) Canceled after 0s
2026-09-01 13:47:36 +07:00
asepharyana fb33b67a29 fix: replace dead YouTube APIs with yt-dlp subprocess; fix all JSON panic crashes; add yt-dlp fallback for TikTok/Bilibili/SoundCloud; fix Bilibili BV ID regex; fix stdout pipe buffer truncation (Stdio::piped)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
2026-09-01 13:34:47 +07:00
asepharyana a41425a887 fix: replace dead YouTube APIs with yt-dlp subprocess; fix all JSON panic crashes; add yt-dlp helper functions
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- YouTube MP3/MP4: replaced dead savetube.media + ytdlpyton.nvlgroup.my.id with yt-dlp subprocess
- Added find_ytdlp() + run_ytdlp_json() helpers with spawn_blocking for >64KB stdout
- Fixed all resp['key'] -> resp.get('key') panics in Twitter, Spotify, KrakenFiles, Danbooru
- Removed dead fetch_youtube_v2_mp3/mp4 + decrypt_savetube functions
- ScraperError::Http -> AppError::ScraperError -> HTTP 502 instead of 500
- All endpoints now return graceful 502 with error message instead of crashing 500

Tested with real URLs:
- YouTube MP3: 200, success, real audio URL extracted
- YouTube MP4: 200, success, 48 format URLs extracted
- TikTok: 200 (tikwm provider)
- MediaFire: 200 (success)
2026-09-01 12:51:52 +07:00
asepharyana c3e6537a63 fix: map ScrapingError to 502 Bad Gateway instead of 500 Internal Server Error; fix downr.org Cloudflare block; fix YouTube savetube DNS failure; fix twitter/pinterest error handling 2026-09-01 11:26:29 +07:00
asepharyana 763a641593 feat(downloader): add 26 downloader APIs + detect endpoint
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Port downloaders from Shirokami-API (Node.js):
- Instagram/Facebook (snapsave.app, downr.org fallback)
- TikTok (tikwm.com)
- YouTube MP4/MP3 (savetube.media, ydlp.yard.id)
- Spotify, Twitter/X (api.lrm.tube)
- Bilibili, Pinterest, MediaFire, Mega, TeraBox
- PixelDrain, Threads, DoodStream, KrakenFiles
- Danbooru, SoundCloud, Google Drive

Domain layer: DownloadResult + MediaItem entities
Application layer: 7 use-cases + detect_platform auto-detect
Infrastructure: DownloaderRepository (22 fetch functions)
Presentation: 21 handlers + DTO + OpenAPI schemas

Also: upgrade OpenTelemetry to 0.28 (axum 0.7 conflict resolution)
2026-09-01 00:54:06 +07:00
asepharyana e6d75edf33 fix(anime): repair genre & latest endpoints for the new otakudesu layout
Deploy Scraper / build-and-deploy (push) Canceled after 0s
The current otakudesu site changed paths and card layout, breaking two APIs:

1. genre pages: singular /genre/{slug}/page/{n}/ 301-redirects to a dead
   otakudesu.io placeholder → 0 items. Fix: plural /genres/{slug}/page/{n}/.
   Also re-target parse_genre_anime_document from the retired .venz ul li /
   .thumbz h2.jdlflm layout to the new .col-anime card layout
   (.col-anime-title a, .col-anime-eps, .col-anime-rating, .col-anime-cover img).

2. latest: /latest-anime/ was removed (301 → otakudesu.io). The homepage IS
   the latest-episodes feed (.venz ul li with .thumbz h2.jdlflm/.epz), so
   fetch_latest_anime_page now uses base_url() for page 1 and /page/N/ for
   later pages instead of the dead path.
2026-08-31 20:59:20 +07:00
asepharyana 80af7b9460 fix(komik): keep api.komiku.org for lists, source genre-list from komiku.org root
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Blanket-switching the whole komik module to komiku.org broke the manga
list: komiku.org/manga/?tipe=manga 301-redirects to /pustaka/ (the new
library layout, 0 .bge items), while api.komiku.org/manga/?tipe=manga
serves the .bge grid the parser expects unredirected.

Correct split:
- api_url() = api.komiku.org  -> manga/manhua/manhwa/genre/search lists (.bge layout)
- genre_list fetches base_url() = komiku.org  root, which is POPULATED
  (#Genre .ls3 .ls3p h4 + /genre/<slug>/ links) — api.komiku.org root is empty

Reverts the api_url() half of a46adf1; keeps the underlying insight that
the genre-list needs the populated komiku.org root.
2026-08-31 20:42:12 +07:00
asepharyana a46adf1ce9 fix(komik): use komiku.org instead of dead api.komiku.org root
Deploy Scraper / build-and-deploy (push) Canceled after 0s
api.komiku.org returns an empty body for its root path, which broke the
komik genre-list endpoint (it fetches the api root and parsed 0 genres).
komiku.org serves identical sub-path content (manga lists, genre, search)
plus a populated root. Consolidate the komik module onto the canonical
working domain. KOMIK2_API_URL env override still honoured.
2026-08-31 20:25:56 +07:00
asepharyana 6bfffa5364 fix(anime): parse otakudesu search results from .chivsrc li
The search parser targeted the index layout (.venz ul li, .thumbz h2.jdlflm,
img, .epz, genre-tag/status/rating classes) but otakudesu search results live
in <ul class=chivsrc><li> with <h2><a>Title</a></h2> plus .set label/value
rows. Search silently returned 0 items.

Rewrote to parse .chivsrc li, extracting title + url from h2 a and
Status/Rating/Genre from .set rows. No poster on the search page (empty).
2026-08-31 20:22:39 +07:00
40 changed files with 8266 additions and 244 deletions

No files matched your search

@@ -0,0 +1,101 @@
# Scraper Downloader API Fix Specification
## Goal
Fix all 15 broken downloader endpoints that return HTTP 500, replace dead third-party APIs with working alternatives, and re-test with real URLs from real platforms.
## Current State Analysis
### Working Endpoints (HTTP 200):
| Platform | Provider | Notes |
|----------|----------|-------|
| TikTok | tikwm.com | ✓ Working (tiktok.com API) |
| MediaFire | mediafire | ✓ Direct page scrape |
| Mega | mega.co.nz | ✓ Valid error messages for bad URLs |
| GDrive | gdrive | ✓ Valid error for missing file ID |
| Pinterest | pinterestdownloader.io | Returns 200 with `{success:false, error:"Unknown error occurred."}` |
| Threads | downr.org | Returns 500 (via fetch_snapsave → fetch_all_in_one) |
### Broken Endpoints (HTTP 500):
| Platform | Upstream API | Error |
|----------|-------------|-------|
| YouTube | media.savetube.me | DNS resolution failure |
| Instagram | downr.org | 403 action_forbidden (Cloudflare) |
| Facebook | downr.org | 403 action_forbidden (Cloudflare) |
| Twitter | savetwitter.net | API returns 404 "Video not found" but code crashes on missing data field |
| Spotify | ytdlpyton.nvlgroup.my.id | DNS resolution failure |
| SoundCloud | ytdlpyton.nvlgroup.my.id | DNS resolution failure |
| PixelDrain | pixeldrain.com | Code crashes parsing viewer_data (wrong regex?) |
| KrakenFiles | krakenfiles.com | API returns 403/empty |
| Danbooru | danbooru.donmai.us | Returns 404 |
| DoodStream | d000d.com | Proxy worker dead |
| TeraBox | tera2.sylyt93.workers.dev | DNS resolution failure |
| Bilibili | cobalt instances | Cobalt DNS failures |
## Root Cause
1. **Error Handling Bug**: `ScrapingError::Http` maps to `AppError::Internal` (HTTP 500) via the `From<ScrapingError>` impl in error.rs. This was already patched to `AppError::ScraperError` → HTTP 502, but **the running binary hasn't been rebuilt**.
2. **Dead Upstream APIs**: Many third-party APIs have moved/changed/blocked requests from VPS IPs.
3. **Error Propagation**: Fetch functions use `.unwrap()` on JSON paths (`resp["data"]`, `resp["download_url"]`) which panics when the upstream returns an unexpected response structure, causing HTTP 500.
## Fix Plan
### Fix 1: Robust error handling (no panics on missing JSON fields)
Replace all `.unwrap()` and direct indexing (`resp["key"]`) with proper `.get()` + error handling. When upstream returns an error response, return `DownloadResult::error("...")` instead of panicking.
### Fix 2: Replace dead upstream APIs
#### YouTube (media.savetube.me → multiple alternatives)
- Primary: Use yt-dlp subprocess via `std::process::Command` (industry standard, handles all platforms)
- Fallback: `https://co.wuk.sh/api/json` (Cobalt instance)
- Fallback: `https://api.tikmate.app/api/lookup` for TikTok
#### Instagram/Facebook (downr.org → alternatives)
- Primary: Use yt-dlp subprocess
- Fallback chain: `snapsave.app/action.php`, `ddownr.com`, `qurls.app`
#### Twitter (savetwitter.net → alternatives)
- Primary: Use twitsave.com (which returned 200 but no MP4 for that test URL)
- Add proper handling for 404 status code from savetwitter.net API
- Fallback: `https://twitsave.com/info?url=...`
#### Spotify (ytdlpyton → alternatives)
- Use Spotify API for metadata + alternative audio source
- Replace with direct Spotify metadata API (already has a fallback to spotify.com API for metadata)
#### Pinterest (pinterestdownloader.io → alternatives)
- Use yt-dlp subprocess
- Fallback: `https://savepin.app`, `https://pin-downloader.com`
#### PixelDrain (fix existing code)
- The regex for extracting viewer_data is wrong - test with real URLs
#### KrakenFiles
- Fix the POST request (the API likely changed)
#### Danbooru
- The API key may be required now
#### DoodStream
- The proxy worker (rv.lil-hacker.workers.dev) is dead - need new proxy
#### TeraBox
- The workers.dev endpoint is dead - find new endpoint
#### Bilibili
- Use working Cobalt instance or the direct API
### Fix 3: Add yt-dlp as universal fallback
Install yt-dlp binary on the VPS. Add a generic download function that shells out to yt-dlp for any platform that fails with its primary API. yt-dlp handles YouTube, IG, FB, Twitter, TikTok, SoundCloud, etc.
### Fix 4: Improve DownloadResult error messages
Ensure every error path returns a descriptive `DownloadResult::error("...")` with platform-specific failure info, not a generic 500.
## Files to Modify
1. `/home/code/scraper/src/presentation/error.rs` - Already patched (ScrapingError → 502)
2. `/home/code/scraper/src/infrastructure/repository/downloader.rs` - Main fix target
3. `/home/code/scraper/Cargo.toml` - Add tokio-process for yt-dlp subprocess
4. `tests/` - Add integration tests with real URLs (non-flaky)
## Verification
- Rebuild and restart the scraper binary
- Test each endpoint with REAL platform URLs
- Verify HTTP status codes are 200 (with success=false data) or 502 (Bad Gateway), never 500
Generated
+110 -174
View File
@@ -8,6 +8,17 @@ version = "2.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa"
[[package]]
name = "aes"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b169f7a6d4742236a0a00c541b845991d0ac43e546831af1249753ab4c3aa3a0"
dependencies = [
"cfg-if",
"cipher",
"cpufeatures 0.2.17",
]
[[package]] [[package]]
name = "ahash" name = "ahash"
version = "0.7.8" version = "0.7.8"
@@ -151,40 +162,13 @@ version = "1.5.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
[[package]]
name = "axum"
version = "0.7.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f"
dependencies = [
"async-trait",
"axum-core 0.4.5",
"bytes",
"futures-util",
"http",
"http-body",
"http-body-util",
"itoa",
"matchit 0.7.3",
"memchr",
"mime",
"percent-encoding",
"pin-project-lite",
"rustversion",
"serde",
"sync_wrapper",
"tower 0.5.3",
"tower-layer",
"tower-service",
]
[[package]] [[package]]
name = "axum" name = "axum"
version = "0.8.9" version = "0.8.9"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90"
dependencies = [ dependencies = [
"axum-core 0.5.6", "axum-core",
"axum-macros", "axum-macros",
"base64", "base64",
"bytes", "bytes",
@@ -196,7 +180,7 @@ dependencies = [
"hyper", "hyper",
"hyper-util", "hyper-util",
"itoa", "itoa",
"matchit 0.8.4", "matchit",
"memchr", "memchr",
"mime", "mime",
"multer", "multer",
@@ -210,32 +194,12 @@ dependencies = [
"sync_wrapper", "sync_wrapper",
"tokio", "tokio",
"tokio-tungstenite", "tokio-tungstenite",
"tower 0.5.3", "tower",
"tower-layer", "tower-layer",
"tower-service", "tower-service",
"tracing", "tracing",
] ]
[[package]]
name = "axum-core"
version = "0.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09f2bd6146b97ae3359fa0cc6d6b376d9539582c7b4220f041a33ec24c226199"
dependencies = [
"async-trait",
"bytes",
"futures-util",
"http",
"http-body",
"http-body-util",
"mime",
"pin-project-lite",
"rustversion",
"sync_wrapper",
"tower-layer",
"tower-service",
]
[[package]] [[package]]
name = "axum-core" name = "axum-core"
version = "0.5.6" version = "0.5.6"
@@ -337,6 +301,15 @@ dependencies = [
"hybrid-array", "hybrid-array",
] ]
[[package]]
name = "block-padding"
version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93"
dependencies = [
"generic-array",
]
[[package]] [[package]]
name = "borsh" name = "borsh"
version = "1.8.1" version = "1.8.1"
@@ -422,6 +395,15 @@ version = "1.12.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04" checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
[[package]]
name = "cbc"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26b52a9543ae338f279b96b0b9fed9c8093744685043739079ce85cd58f289a6"
dependencies = [
"cipher",
]
[[package]] [[package]]
name = "cc" name = "cc"
version = "1.4.4" version = "1.4.4"
@@ -460,6 +442,16 @@ dependencies = [
"windows-link", "windows-link",
] ]
[[package]]
name = "cipher"
version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common 0.1.7",
"inout",
]
[[package]] [[package]]
name = "combine" name = "combine"
version = "4.6.8" version = "4.6.8"
@@ -1156,7 +1148,7 @@ dependencies = [
"futures-core", "futures-core",
"futures-sink", "futures-sink",
"http", "http",
"indexmap 2.14.1", "indexmap",
"slab", "slab",
"tokio", "tokio",
"tokio-util", "tokio-util",
@@ -1362,19 +1354,6 @@ dependencies = [
"tower-service", "tower-service",
] ]
[[package]]
name = "hyper-timeout"
version = "0.5.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0"
dependencies = [
"hyper",
"hyper-util",
"pin-project-lite",
"tokio",
"tower-service",
]
[[package]] [[package]]
name = "hyper-tls" name = "hyper-tls"
version = "0.6.0" version = "0.6.0"
@@ -1408,7 +1387,7 @@ dependencies = [
"libc", "libc",
"percent-encoding", "percent-encoding",
"pin-project-lite", "pin-project-lite",
"socket2 0.6.5", "socket2",
"system-configuration", "system-configuration",
"tokio", "tokio",
"tower-service", "tower-service",
@@ -1544,16 +1523,6 @@ dependencies = [
"icu_properties", "icu_properties",
] ]
[[package]]
name = "indexmap"
version = "1.9.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bd070e393353796e801d209ad339e89596eb4c8d430d18ede6a1cced8fafbd99"
dependencies = [
"autocfg",
"hashbrown 0.12.3",
]
[[package]] [[package]]
name = "indexmap" name = "indexmap"
version = "2.14.1" version = "2.14.1"
@@ -1597,6 +1566,16 @@ dependencies = [
"libc", "libc",
] ]
[[package]]
name = "inout"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"block-padding",
"generic-array",
]
[[package]] [[package]]
name = "ipnet" name = "ipnet"
version = "2.12.1" version = "2.12.1"
@@ -1766,12 +1745,6 @@ dependencies = [
"regex-automata", "regex-automata",
] ]
[[package]]
name = "matchit"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e7465ac9959cc2b1404e8e2367b43684a6d13790fe23056cc8c6c5a6b7bcb94"
[[package]] [[package]]
name = "matchit" name = "matchit"
version = "0.8.4" version = "0.8.4"
@@ -1879,7 +1852,7 @@ dependencies = [
"num_cpus", "num_cpus",
"rand 0.8.8", "rand 0.8.8",
"rayon", "rayon",
"thiserror 2.0.20", "thiserror",
"tokio", "tokio",
"tracing", "tracing",
] ]
@@ -2137,42 +2110,56 @@ dependencies = [
[[package]] [[package]]
name = "opentelemetry" name = "opentelemetry"
version = "0.27.1" version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab70038c28ed37b97d8ed414b6429d343a8bbf44c9f79ec854f3a643029ba6d7" checksum = "236e667b670a5cdf90c258f5a55794ec5ac5027e960c224bff8367a59e1e6426"
dependencies = [ dependencies = [
"futures-core", "futures-core",
"futures-sink", "futures-sink",
"js-sys", "js-sys",
"pin-project-lite", "pin-project-lite",
"thiserror 1.0.69", "thiserror",
"tracing",
]
[[package]]
name = "opentelemetry-http"
version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8863faf2910030d139fb48715ad5ff2f35029fc5f244f6d5f689ddcf4d26253"
dependencies = [
"async-trait",
"bytes",
"http",
"opentelemetry",
"reqwest",
"tracing", "tracing",
] ]
[[package]] [[package]]
name = "opentelemetry-otlp" name = "opentelemetry-otlp"
version = "0.27.0" version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "91cf61a1868dacc576bf2b2a1c3e9ab150af7272909e80085c3173384fe11f76" checksum = "5bef114c6d41bea83d6dc60eb41720eedd0261a67af57b66dd2b84ac46c01d91"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"futures-core", "futures-core",
"http", "http",
"opentelemetry", "opentelemetry",
"opentelemetry-http",
"opentelemetry-proto", "opentelemetry-proto",
"opentelemetry_sdk", "opentelemetry_sdk",
"prost", "prost",
"thiserror 1.0.69", "reqwest",
"tokio", "thiserror",
"tonic",
"tracing", "tracing",
] ]
[[package]] [[package]]
name = "opentelemetry-proto" name = "opentelemetry-proto"
version = "0.27.0" version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a6e05acbfada5ec79023c85368af14abd0b307c015e9064d249b2a950ef459a6" checksum = "56f8870d3024727e99212eb3bb1762ec16e255e3e6f58eeb3dc8db1aa226746d"
dependencies = [ dependencies = [
"opentelemetry", "opentelemetry",
"opentelemetry_sdk", "opentelemetry_sdk",
@@ -2182,9 +2169,9 @@ dependencies = [
[[package]] [[package]]
name = "opentelemetry_sdk" name = "opentelemetry_sdk"
version = "0.27.1" version = "0.28.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "231e9d6ceef9b0b2546ddf52335785ce41252bc7474ee8ba05bfad277be13ab8" checksum = "84dfad6042089c7fc1f6118b7040dc2eb4ab520abbf410b79dc481032af39570"
dependencies = [ dependencies = [
"async-trait", "async-trait",
"futures-channel", "futures-channel",
@@ -2195,7 +2182,7 @@ dependencies = [
"percent-encoding", "percent-encoding",
"rand 0.8.8", "rand 0.8.8",
"serde_json", "serde_json",
"thiserror 1.0.69", "thiserror",
"tokio", "tokio",
"tokio-stream", "tokio-stream",
"tracing", "tracing",
@@ -2627,7 +2614,7 @@ dependencies = [
"rustls-native-certs", "rustls-native-certs",
"ryu", "ryu",
"sha1_smol", "sha1_smol",
"socket2 0.6.5", "socket2",
"tokio", "tokio",
"tokio-rustls", "tokio-rustls",
"tokio-util", "tokio-util",
@@ -2699,6 +2686,7 @@ dependencies = [
"base64", "base64",
"bytes", "bytes",
"encoding_rs", "encoding_rs",
"futures-channel",
"futures-core", "futures-core",
"futures-util", "futures-util",
"h2", "h2",
@@ -2724,7 +2712,7 @@ dependencies = [
"tokio", "tokio",
"tokio-native-tls", "tokio-native-tls",
"tokio-util", "tokio-util",
"tower 0.5.3", "tower",
"tower-http", "tower-http",
"tower-service", "tower-service",
"url", "url",
@@ -2978,15 +2966,22 @@ dependencies = [
name = "scraper-service" name = "scraper-service"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"aes",
"anyhow", "anyhow",
"async-trait", "async-trait",
"axum 0.8.9", "axum",
"base64",
"block-padding",
"cbc",
"chrono", "chrono",
"cipher",
"dashmap", "dashmap",
"deadpool-redis", "deadpool-redis",
"dotenvy", "dotenvy",
"fastrand",
"flate2", "flate2",
"futures", "futures",
"hex",
"http", "http",
"mytheclipse", "mytheclipse",
"mytheclipse-cache", "mytheclipse-cache",
@@ -3005,7 +3000,7 @@ dependencies = [
"sea-orm", "sea-orm",
"serde", "serde",
"serde_json", "serde_json",
"thiserror 2.0.20", "thiserror",
"tokio", "tokio",
"tower-http", "tower-http",
"tracing", "tracing",
@@ -3053,7 +3048,7 @@ dependencies = [
"serde_json", "serde_json",
"sqlx", "sqlx",
"strum", "strum",
"thiserror 2.0.20", "thiserror",
"time", "time",
"tracing", "tracing",
"url", "url",
@@ -3241,7 +3236,7 @@ version = "0.9.34+deprecated"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47"
dependencies = [ dependencies = [
"indexmap 2.14.1", "indexmap",
"itoa", "itoa",
"ryu", "ryu",
"serde", "serde",
@@ -3364,16 +3359,6 @@ dependencies = [
"serde", "serde",
] ]
[[package]]
name = "socket2"
version = "0.5.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678"
dependencies = [
"libc",
"windows-sys 0.52.0",
]
[[package]] [[package]]
name = "socket2" name = "socket2"
version = "0.6.5" version = "0.6.5"
@@ -3436,7 +3421,7 @@ dependencies = [
"futures-util", "futures-util",
"hashbrown 0.15.5", "hashbrown 0.15.5",
"hashlink", "hashlink",
"indexmap 2.14.1", "indexmap",
"log", "log",
"memchr", "memchr",
"once_cell", "once_cell",
@@ -3447,7 +3432,7 @@ dependencies = [
"serde_json", "serde_json",
"sha2 0.10.9", "sha2 0.10.9",
"smallvec", "smallvec",
"thiserror 2.0.20", "thiserror",
"time", "time",
"tokio", "tokio",
"tokio-stream", "tokio-stream",
@@ -3535,7 +3520,7 @@ dependencies = [
"smallvec", "smallvec",
"sqlx-core", "sqlx-core",
"stringprep", "stringprep",
"thiserror 2.0.20", "thiserror",
"time", "time",
"tracing", "tracing",
"uuid", "uuid",
@@ -3578,7 +3563,7 @@ dependencies = [
"smallvec", "smallvec",
"sqlx-core", "sqlx-core",
"stringprep", "stringprep",
"thiserror 2.0.20", "thiserror",
"time", "time",
"tracing", "tracing",
"uuid", "uuid",
@@ -3605,7 +3590,7 @@ dependencies = [
"serde", "serde",
"serde_urlencoded", "serde_urlencoded",
"sqlx-core", "sqlx-core",
"thiserror 2.0.20", "thiserror",
"time", "time",
"tracing", "tracing",
"url", "url",
@@ -3775,33 +3760,13 @@ dependencies = [
"utf-8", "utf-8",
] ]
[[package]]
name = "thiserror"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
dependencies = [
"thiserror-impl 1.0.69",
]
[[package]] [[package]]
name = "thiserror" name = "thiserror"
version = "2.0.20" version = "2.0.20"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f" checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f"
dependencies = [ dependencies = [
"thiserror-impl 2.0.20", "thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "1.0.69"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
] ]
[[package]] [[package]]
@@ -3891,7 +3856,7 @@ dependencies = [
"parking_lot", "parking_lot",
"pin-project-lite", "pin-project-lite",
"signal-hook-registry", "signal-hook-registry",
"socket2 0.6.5", "socket2",
"tokio-macros", "tokio-macros",
"windows-sys 0.61.2", "windows-sys 0.61.2",
] ]
@@ -4000,7 +3965,7 @@ version = "0.22.27"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a" checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [ dependencies = [
"indexmap 2.14.1", "indexmap",
"serde", "serde",
"serde_spanned", "serde_spanned",
"toml_datetime 0.6.11", "toml_datetime 0.6.11",
@@ -4014,7 +3979,7 @@ version = "0.25.13+spec-1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6975367e4d2ef766d86af01ffad14b622fecc8d4357a998fbc4deb6e9bacaf9b" checksum = "6975367e4d2ef766d86af01ffad14b622fecc8d4357a998fbc4deb6e9bacaf9b"
dependencies = [ dependencies = [
"indexmap 2.14.1", "indexmap",
"toml_datetime 1.1.1+spec-1.1.0", "toml_datetime 1.1.1+spec-1.1.0",
"toml_parser", "toml_parser",
"winnow 1.0.4", "winnow 1.0.4",
@@ -4041,45 +4006,16 @@ version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52" checksum = "877c5b330756d856ffcc4553ab34a5684481ade925ecc54bcd1bf02b1d0d4d52"
dependencies = [ dependencies = [
"async-stream",
"async-trait", "async-trait",
"axum 0.7.9",
"base64", "base64",
"bytes", "bytes",
"h2",
"http", "http",
"http-body", "http-body",
"http-body-util", "http-body-util",
"hyper",
"hyper-timeout",
"hyper-util",
"percent-encoding", "percent-encoding",
"pin-project", "pin-project",
"prost", "prost",
"socket2 0.5.10",
"tokio",
"tokio-stream", "tokio-stream",
"tower 0.4.13",
"tower-layer",
"tower-service",
"tracing",
]
[[package]]
name = "tower"
version = "0.4.13"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c"
dependencies = [
"futures-core",
"futures-util",
"indexmap 1.9.3",
"pin-project",
"pin-project-lite",
"rand 0.8.8",
"slab",
"tokio",
"tokio-util",
"tower-layer", "tower-layer",
"tower-service", "tower-service",
"tracing", "tracing",
@@ -4123,7 +4059,7 @@ dependencies = [
"pin-project-lite", "pin-project-lite",
"tokio", "tokio",
"tokio-util", "tokio-util",
"tower 0.5.3", "tower",
"tower-layer", "tower-layer",
"tower-service", "tower-service",
"url", "url",
@@ -4222,7 +4158,7 @@ dependencies = [
"log", "log",
"rand 0.9.5", "rand 0.9.5",
"sha1", "sha1",
"thiserror 2.0.20", "thiserror",
] ]
[[package]] [[package]]
@@ -4324,7 +4260,7 @@ version = "5.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8bde15df68e80b16c7d16b9616e80770ad158988daa56a27dccd1e55558b0160" checksum = "8bde15df68e80b16c7d16b9616e80770ad158988daa56a27dccd1e55558b0160"
dependencies = [ dependencies = [
"indexmap 2.14.1", "indexmap",
"serde", "serde",
"serde_json", "serde_json",
"utoipa-gen", "utoipa-gen",
@@ -4348,7 +4284,7 @@ version = "9.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d047458f1b5b65237c2f6dc6db136945667f40a7668627b3490b9513a3d43a55" checksum = "d047458f1b5b65237c2f6dc6db136945667f40a7668627b3490b9513a3d43a55"
dependencies = [ dependencies = [
"axum 0.8.9", "axum",
"base64", "base64",
"mime_guess", "mime_guess",
"regex", "regex",
@@ -4955,7 +4891,7 @@ dependencies = [
"arbitrary", "arbitrary",
"crc32fast", "crc32fast",
"flate2", "flate2",
"indexmap 2.14.1", "indexmap",
"memchr", "memchr",
"zopfli", "zopfli",
] ]
+11 -4
View File
@@ -1,4 +1,3 @@
[package] [package]
name = "scraper-service" name = "scraper-service"
version = "0.1.0" version = "0.1.0"
@@ -45,6 +44,13 @@ regex = "1.12.2"
urlencoding = "2.1" urlencoding = "2.1"
url = "2.5.8" url = "2.5.8"
base64 = "0.22"
cipher = "0.4"
aes = "0.8"
cbc = "0.1"
block-padding = "0.3"
hex = "0.4"
fastrand = "2.0"
tower-http = { version = "0.6.8", features = ["fs", "cors", "compression-gzip", "compression-br", "compression-zstd"] } tower-http = { version = "0.6.8", features = ["fs", "cors", "compression-gzip", "compression-br", "compression-zstd"] }
dashmap = "6.1" dashmap = "6.1"
@@ -59,9 +65,10 @@ utoipa = { version = "5.0", features = ["axum_extras"] }
utoipa-swagger-ui = { version = "9.0", features = ["axum"] } utoipa-swagger-ui = { version = "9.0", features = ["axum"] }
# OpenTelemetry metrics # OpenTelemetry metrics
opentelemetry = { version = "0.27", features = ["metrics"] } opentelemetry = { version = "0.28", features = ["metrics"] }
opentelemetry_sdk = { version = "0.27", features = ["metrics", "rt-tokio"] } opentelemetry_sdk = { version = "0.28", features = ["metrics", "rt-tokio"] }
opentelemetry-otlp = { version = "0.27", features = ["metrics"] } opentelemetry-otlp = { version = "0.28", features = ["metrics"] }
# Dependensi yang hanya dibutuhkan untuk build script (build.rs) # Dependensi yang hanya dibutuhkan untuk build script (build.rs)
Generated
+61
View File
@@ -0,0 +1,61 @@
{
"nodes": {
"flake-utils": {
"inputs": {
"systems": "systems"
},
"locked": {
"lastModified": 1731533236,
"narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
"owner": "numtide",
"repo": "flake-utils",
"rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "flake-utils",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1787964612,
"narHash": "sha256-0N9nghg3nwzX6b6qc77EzjR9cu/Z+UR66FlfsCqiURs=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "e8be7818e19ada32105a8af937a6a473b38167ca",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixpkgs-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"flake-utils": "flake-utils",
"nixpkgs": "nixpkgs"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
"owner": "nix-systems",
"repo": "default",
"rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
"type": "github"
},
"original": {
"owner": "nix-systems",
"repo": "default",
"type": "github"
}
}
},
"root": "root",
"version": 7
}
+3 -1
View File
@@ -15,7 +15,7 @@
}; };
scraper = pkgs.stdenv.mkDerivation { scraper = pkgs.stdenv.mkDerivation {
name = "scraper-0.1.0"; name = "scraper-0.1.1";
src = ./.; src = ./.;
nativeBuildInputs = with pkgs; [ nativeBuildInputs = with pkgs; [
@@ -39,6 +39,8 @@
installPhase = '' installPhase = ''
mkdir -p $out/bin mkdir -p $out/bin
cp target/release/scraper $out/bin/scraper cp target/release/scraper $out/bin/scraper
cp $src/scrape_media.py $out/bin/scrape_media.py
chmod +x $out/bin/scrape_media.py
''; '';
}; };
in in
+161
View File
@@ -0,0 +1,161 @@
#!/usr/bin/env python3
"""
Universal scraper for social media video URLs using Playwright + Chromium.
Used as fallback when yt-dlp is blocked by anti-bot protection.
Usage: scrape_media.py <url> <platform>
Output: JSON with title, media items, etc.
Platforms supported:
- instagram: Scrape reel/post video URLs from cdninstagram.com
- facebook: Scrape video URLs from fbcdn.net
- tiktok: Try scraping, but may be blocked by anti-bot
- twitter: Scrape video URLs from video.twimg.com
- pinterest: Scrape video URLs from v.pinimg.com
"""
import sys
import asyncio
import re
import json
import os
# Point Playwright at system-wide browser cache so it works regardless of
# the service user's HOME permissions.
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/usr/local/share/ms-playwright")
from playwright.async_api import async_playwright
async def scrape(url, platform):
async with async_playwright() as p:
browser = await p.chromium.launch(
headless=True,
args=["--no-sandbox", "--disable-bypass"]
)
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
viewport={"width": 1920, "height": 1080},
java_script_enabled=True,
)
page = await context.new_page()
# Track network responses
media_responses = []
def handle_response(response):
resp_url = response.url
# Look for video/audio URLs
if any(ext in resp_url for ext in ['.mp4', '.m3u8', '.mp3']):
media_responses.append({
"url": resp_url,
"status": response.status,
"content_type": response.headers.get('content-type', ''),
})
page.on("response", handle_response)
try:
await page.goto(url, timeout=60000)
await page.wait_for_timeout(15000)
except Exception as e:
await browser.close()
print(json.dumps({"success": False, "error": f"Navigation error: {e}"}))
return
# Get page title
title = await page.title()
# Get video element info
video_info = await page.evaluate("""() => {
const video = document.querySelector('video');
if (!video) return null;
return {
src: video.src,
duration: video.duration,
poster: video.poster,
};
}""")
# Filter for real video URLs (not static assets, not ttwstatic)
seen = set()
media = []
for resp in media_responses:
url_val = resp["url"]
if url_val in seen:
continue
if len(url_val) < 50:
continue
# Skip static assets
if 'ttwstatic' in url_val or 'rsrc.php' in url_val:
continue
if 'cdninstagram' not in url_val and 'fbcdn' not in url_val and 'video.twimg' not in url_val and 'pinimg' not in url_val:
# For non-Instagram/facebook, still accept
if 'static' in url_val:
continue
seen.add(url_val)
# Determine content type
ext = "mp4"
if '.m3u8' in url_val:
ext = "m3u8"
elif '.mp3' in url_val:
ext = "mp3"
elif '.mp4' in url_val:
ext = "mp4"
media.append({
"url": url_val,
"ext": ext,
"status": resp["status"],
"content_type": resp["content_type"],
})
# Also check for video element src (blob URLs won't work, but worth checking)
if video_info and video_info.get('src'):
video_src = video_info['src']
if not video_src.startswith('blob:'):
media.append({
"url": video_src,
"ext": "mp4",
"status": 200,
"content_type": "video/mp4",
})
# If no media found, check page content for URLs
if not media:
content = await page.content()
# Look for video URLs in page source
content_urls = re.findall(r'https://[^\s"\'<>]+', content)
for url_val in content_urls:
if any(ext in url_val for ext in ['.mp4', '.m3u8']) and len(url_val) > 50:
if 'ttwstatic' not in url_val and 'rsrc.php' not in url_val:
if url_val not in seen:
seen.add(url_val)
ext = "m3u8" if '.m3u8' in url_val else "mp4"
media.append({
"url": url_val,
"ext": ext,
"status": 200,
"content_type": f"video/{ext}" if ext != 'm3u8' else 'application/x-mpegURL',
})
result = {
"success": len(media) > 0,
"title": title if title else "Unknown",
"platform": platform,
"media": media,
"media_count": len(media),
"provider": f"playwright-{platform}",
}
await browser.close()
print(json.dumps(result, indent=2))
if __name__ == "__main__":
if len(sys.argv) < 3:
print(json.dumps({"success": False, "error": "Usage: scrape_media.py <url> <platform>"}))
sys.exit(1)
url = sys.argv[1]
platform = sys.argv[2]
asyncio.run(scrape(url, platform))
+186
View File
@@ -0,0 +1,186 @@
//! Application-layer use cases for media downloading.
//!
//! Thin coordinators that delegate to `DownloaderRepository` and return
//! `DownloadResult` domain entities. Each use case corresponds to one
//! downloader endpoint defined in the spec (Shirokami-API reference).
use crate::domain::entity::downloader::DownloadResult;
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::DownloaderRepository;
/// Use case: unified dispatcher — auto-detects platform from URL pattern
/// and delegates to the appropriate downloader.
pub async fn download_all_in_one(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_all_in_one(url, cookies).await
}
pub async fn download_instagram(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_instagram(url, cookies).await
}
pub async fn download_facebook(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_facebook(url, cookies).await
}
pub async fn download_tiktok(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_tiktok(url, cookies).await
}
pub async fn download_youtube(url: &str, quality: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_youtube(url, quality).await
}
pub async fn download_youtube_merge(
url: &str,
quality: &str,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::fetch_youtube_merge(url, quality).await
}
pub async fn download_youtube_mp3(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_youtube_mp3(url).await
}
pub async fn download_spotify(
url: &str,
api_key: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_spotify(url, api_key).await
}
pub async fn download_twitter(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_twitter(url, cookies).await
}
pub async fn download_pinterest(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_pinterest(url).await
}
pub async fn download_mega(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_mega(url).await
}
pub async fn download_terabox(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_terabox(url).await
}
pub async fn download_gdrive(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_gdrive(url).await
}
pub async fn download_mediafire(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_mediafire(url).await
}
pub async fn download_pixeldrain(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_pixeldrain(url).await
}
pub async fn download_threads(
url: &str,
cookies: Option<&str>,
) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_threads(url, cookies).await
}
pub async fn download_doodstream(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_doodstream(url).await
}
pub async fn download_krakenfiles(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_krakenfiles(url).await
}
pub async fn download_danbooru(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_danbooru(url).await
}
pub async fn download_soundcloud(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_soundcloud(url).await
}
pub async fn download_dailymotion(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_dailymotion(url).await
}
pub async fn download_reddit(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_reddit(url).await
}
pub async fn download_streamable(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_streamable(url).await
}
pub async fn download_videy(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_videy(url).await
}
pub async fn download_bilibili(url: &str) -> Result<DownloadResult, ScrapingError> {
DownloaderRepository::download_bilibili(url).await
}
/// Use case: detect platform from URL pattern.
pub fn detect_platform(url: &str) -> String {
if url.contains("instagram.com") || url.contains("instagr.am") {
"instagram".to_string()
} else if url.contains("facebook.com") || url.contains("fb.watch") {
"facebook".to_string()
} else if url.contains("tiktok.com") || url.contains("vm.tiktok.com") {
"tiktok".to_string()
} else if url.contains("youtube.com") || url.contains("youtu.be") {
"youtube".to_string()
} else if url.contains("open.spotify.com") || url.contains("spotify.link") {
"spotify".to_string()
} else if url.contains("twitter.com") || url.contains("x.com") || url.contains("t.co/") {
"twitter".to_string()
} else if url.contains("pinterest") {
"pinterest".to_string()
} else if url.contains("reddit.com") || url.contains("redd.it") {
"reddit".to_string()
} else if url.contains("mega.nz") || url.contains("mega.io") {
"mega".to_string()
} else if url.contains("terabox") || url.contains("nfile") {
"terabox".to_string()
} else if url.contains("drive.google.com") || url.contains("docs.google.com") {
"gdrive".to_string()
} else if url.contains("mediafire.com") {
"mediafire".to_string()
} else if url.contains("pixeldrain.com") {
"pixeldrain".to_string()
} else if url.contains("threads.net") || url.contains("threads.com") {
"threads".to_string()
} else if url.contains("dood.") || url.contains("doodstream") || url.contains("dood.so") {
"doodstream".to_string()
} else if url.contains("krakenfiles.com") {
"krakenfiles".to_string()
} else if url.contains("danbooru") || url.contains("safebooru") || url.contains("rule34") {
"danbooru".to_string()
} else if url.contains("soundcloud.com") {
"soundcloud".to_string()
} else if url.contains("dailymotion.com") {
"dailymotion".to_string()
} else if url.contains("streamable.com") {
"streamable".to_string()
} else if url.contains("videy.co") {
"videy".to_string()
} else if url.contains("bilibili.com") || url.contains("b23.tv") {
"bilibili".to_string()
} else {
"unknown".to_string()
}
}
+14
View File
@@ -0,0 +1,14 @@
//! Application use-cases for image generation.
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::image as Repo;
pub async fn brat(text: &str) -> Result<Vec<u8>, ScrapingError> {
Repo::fetch_brat(text).await.map_err(ScrapingError::Http)
}
pub async fn brat_animated(text: &str) -> Result<Vec<u8>, ScrapingError> {
Repo::fetch_brat_animated(text)
.await
.map_err(ScrapingError::Http)
}
+1 -1
View File
@@ -49,7 +49,7 @@ impl KomikUseCases {
.get_or_set("komik:genres:list:v3", GENRE_LIST_CACHE_TTL, || async { .get_or_set("komik:genres:list:v3", GENRE_LIST_CACHE_TTL, || async {
let html = self let html = self
.repository .repository
.fetch_html(&self.repository.api_url()) .fetch_html(&self.repository.base_url())
.await .await
.map_err(|e| e.to_string())?; .map_err(|e| e.to_string())?;
let genres = tokio::task::spawn_blocking(move || parser::parse_genres(&html)) let genres = tokio::task::spawn_blocking(move || parser::parse_genres(&html))
+27
View File
@@ -0,0 +1,27 @@
//! Application use-cases for misc utilities.
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::misc as Repo;
use serde_json::Value;
/// Currency converter.
pub async fn currency_converter(amount: f64, from: &str, to: &str) -> Result<Value, ScrapingError> {
Repo::fetch_currency_converter(amount, from, to)
.await
.map_err(ScrapingError::Http)
}
/// Harga emas Antam.
pub async fn harga_emas() -> Result<Value, ScrapingError> {
Repo::fetch_harga_emas().await.map_err(ScrapingError::Http)
}
/// Kurs BCA (jual/beli).
pub async fn kurs_bca() -> Result<Value, ScrapingError> {
Repo::fetch_kurs_bca().await.map_err(ScrapingError::Http)
}
/// Server info (OS/CPU/RAM/disk).
pub async fn server_info() -> Result<Value, ScrapingError> {
Repo::fetch_server_info().await.map_err(ScrapingError::Http)
}
+6
View File
@@ -1,4 +1,10 @@
pub mod anime; pub mod anime;
pub mod anime2; pub mod anime2;
pub mod downloader;
pub mod image;
pub mod komik; pub mod komik;
pub mod misc;
pub mod proxy; pub mod proxy;
pub mod search;
pub mod stalk;
pub mod tools;
+29
View File
@@ -0,0 +1,29 @@
//! Application use-cases for search utilities.
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::search as Repo;
use serde_json::Value;
pub async fn bmkg() -> Result<Value, ScrapingError> {
Repo::fetch_bmkg().await.map_err(ScrapingError::Http)
}
pub async fn jadwal_sholat(kota: &str) -> Result<Value, ScrapingError> {
Repo::fetch_jadwal_sholat(kota)
.await
.map_err(ScrapingError::Http)
}
pub async fn weather(city: &str) -> Result<Value, ScrapingError> {
Repo::fetch_weather(city).await.map_err(ScrapingError::Http)
}
pub async fn google(query: &str) -> Result<Value, ScrapingError> {
Repo::fetch_google(query).await.map_err(ScrapingError::Http)
}
pub async fn yt_search(query: &str) -> Result<Value, ScrapingError> {
Repo::fetch_yt_search(query)
.await
.map_err(ScrapingError::Http)
}
+26
View File
@@ -0,0 +1,26 @@
//! Application use-cases for stalk (profile/user lookups).
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::stalk as Repo;
use serde_json::Value;
/// GitHub user profile stalk.
pub async fn github(username: &str) -> Result<Value, ScrapingError> {
Repo::fetch_github_stalk(username)
.await
.map_err(ScrapingError::Http)
}
/// YouTube channel/profile stalk.
pub async fn youtube(username: &str) -> Result<Value, ScrapingError> {
Repo::fetch_youtube_stalk(username)
.await
.map_err(ScrapingError::Http)
}
/// Twitter/X user stalk.
pub async fn twitter(username: &str) -> Result<Value, ScrapingError> {
Repo::fetch_twitter_stalk(username)
.await
.map_err(ScrapingError::Http)
}
+35
View File
@@ -0,0 +1,35 @@
//! Application use-cases for tools.
use crate::domain::error::ScrapingError;
use crate::infrastructure::repository::tools as Repo;
use serde_json::Value;
pub async fn whois(domain: &str) -> Result<Value, ScrapingError> {
Repo::fetch_whois(domain).await.map_err(ScrapingError::Http)
}
pub async fn ip_location(ip: &str) -> Result<Value, ScrapingError> {
Repo::fetch_ip_location(ip)
.await
.map_err(ScrapingError::Http)
}
pub async fn tinyurl(url: &str) -> Result<Value, ScrapingError> {
Repo::fetch_tinyurl(url).await.map_err(ScrapingError::Http)
}
pub async fn check_hosting(domain: &str) -> Result<Value, ScrapingError> {
Repo::fetch_check_hosting(domain)
.await
.map_err(ScrapingError::Http)
}
pub async fn hargapangan() -> Result<Value, ScrapingError> {
Repo::fetch_hargapangan().await.map_err(ScrapingError::Http)
}
pub async fn cek_resi(resi: String, ekspedisi: Option<String>) -> Result<Value, ScrapingError> {
Repo::fetch_cek_resi(&resi, ekspedisi.as_deref())
.await
.map_err(ScrapingError::Http)
}
+112
View File
@@ -0,0 +1,112 @@
//! Domain entities for media downloader data.
//!
//! Pure domain structs representing download results from various social
//! platforms and file hosts. No framework dependencies beyond serde + utoipa.
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
/// Media type classification for a download link.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, ToSchema)]
pub enum MediaType {
Video,
Audio,
Image,
File,
}
/// The status of a download attempt.
#[derive(Serialize, Deserialize, Debug, Clone, PartialEq, Eq, ToSchema)]
#[serde(rename_all = "lowercase")]
pub enum DownloadStatus {
Success,
/// Provider returned a known error (bad URL, rate limited, etc.)
Error,
/// All providers failed
Failed,
}
/// A single downloadable media item (a video/audio/image/file link).
#[derive(Serialize, Deserialize, Debug, Clone, ToSchema)]
pub struct MediaItem {
pub url: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub quality: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub file_type: Option<MediaType>,
#[serde(skip_serializing_if = "Option::is_none")]
pub extension: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thumbnail: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub file_size: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub size_bytes: Option<u64>,
#[serde(skip_serializing_if = "Option::is_none")]
pub frame_width: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub frame_height: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub note: Option<String>,
}
/// Unified download result returned by every downloader endpoint.
///
/// This is intentionally flexible — different platforms return different
/// metadata (some have duration, some don't), so most fields are `Option`.
#[derive(Serialize, Deserialize, Debug, Clone, ToSchema)]
pub struct DownloadResult {
pub status: DownloadStatus,
#[serde(skip_serializing_if = "Option::is_none")]
pub message: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub title: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub author: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub thumbnail: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub duration: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub description: Option<String>,
#[serde(skip_serializing_if = "Vec::is_empty")]
pub media: Vec<MediaItem>,
/// Provider that supplied the result (e.g. "snapsave", "tikwm", "savetube")
#[serde(skip_serializing_if = "Option::is_none")]
pub provider: Option<String>,
}
impl DownloadResult {
pub fn success(title: Option<String>) -> Self {
Self {
status: DownloadStatus::Success,
message: None,
title,
author: None,
thumbnail: None,
duration: None,
description: None,
media: Vec::new(),
provider: None,
}
}
pub fn error(message: impl Into<String>) -> Self {
Self {
status: DownloadStatus::Error,
message: Some(message.into()),
title: None,
author: None,
thumbnail: None,
duration: None,
description: None,
media: Vec::new(),
provider: None,
}
}
pub fn add_media(mut self, item: MediaItem) -> Self {
self.media.push(item);
self
}
}
+1
View File
@@ -1,2 +1,3 @@
pub mod anime; pub mod anime;
pub mod downloader;
pub mod komik; pub mod komik;
File diff suppressed because it is too large. Load diff
+46
View File
@@ -0,0 +1,46 @@
//! Infrastructure — Image generation endpoints (brat).
//!
//! Ported from Shirokami-API `scraper/image/brat.js`.
//! brat v2 uses the external brat.siputzx.my.id API (returns image bytes).
use crate::infrastructure::utils::http_client::http_client;
use reqwest::header::USER_AGENT;
const BRAT_UA: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0 Safari/537.36";
fn urlencode(s: &str) -> String {
s.replace('%', "%25")
.replace('&', "%26")
.replace('?', "%3F")
}
/// Fetch a brat image (static) for the given text.
pub async fn fetch_brat(text: &str) -> Result<Vec<u8>, String> {
let url = format!("https://brat.siputzx.my.id/image?text={}", urlencode(text));
fetch_bytes(&url).await
}
/// Fetch an animated brat GIF for the given text.
pub async fn fetch_brat_animated(text: &str) -> Result<Vec<u8>, String> {
let url = format!("https://brat.siputzx.my.id/gif?text={}", urlencode(text));
fetch_bytes(&url).await
}
async fn fetch_bytes(url: &str) -> Result<Vec<u8>, String> {
let resp = http_client()
.client()
.get(url)
.header(USER_AGENT, BRAT_UA)
.header("X-Forwarded-For", "104.28.209.35")
.header("X-Real-IP", "104.28.209.35")
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {}", resp.status()));
}
resp.bytes()
.await
.map(|b| b.to_vec())
.map_err(|e| format!("Body: {}", e))
}
+352
View File
@@ -0,0 +1,352 @@
//! Infrastructure — Misc utility scrapers (currency-converter, harga-emas, kurs-bca, server-info).
//!
//! Ported from Shirokami-API `scraper/misc/*.js`.
//! HTML scraping via reqwest + scraper crate.
use crate::infrastructure::utils::http_client::http_client;
use regex::Regex;
use reqwest::header::USER_AGENT;
use scraper::{Html, Selector};
use serde_json::{json, Value};
/// Currency converter: scrape wise.com exchange rate page.
/// `amount`: amount to convert, `from`/`to`: currency codes (e.g. USD→IDR).
pub async fn fetch_currency_converter(amount: f64, from: &str, to: &str) -> Result<Value, String> {
let from_lower = from.to_lowercase();
let to_lower = to.to_lowercase();
let url = format!(
"https://wise.com/gb/currency-converter/{}-to-{}-rate?amount={}",
from_lower, to_lower, amount
);
let resp = http_client()
.client()
.get(&url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
)
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {} from wise.com", resp.status()));
}
let body = resp
.text()
.await
.map_err(|e| format!("Failed to read body: {}", e))?;
// Try regex first: "1 USD = 15,885.00 IDR"
let re = Regex::new(r"1 [A-Z]{3} = ([0-9,.]+) [A-Z]{3}")
.map_err(|e| format!("Regex compile: {}", e))?;
if let Some(caps) = re.captures(&body) {
let rate_str = caps.get(1).map(|m| m.as_str()).unwrap_or("0");
let rate = rate_str.replace(',', "").parse::<f64>().unwrap_or(0.0);
if rate > 0.0 {
let result = amount * rate;
return Ok(json!({
"from": from.to_uppercase(),
"to": to.to_uppercase(),
"amount": amount,
"rate": rate,
"result": result,
"updateTime": chrono::Utc::now().to_rfc3339(),
}));
}
}
// Fallback: scrape the page for any rate info
let document = Html::parse_document(&body);
let selector = Selector::parse("body").unwrap();
let text = document
.select(&selector)
.flat_map(|e| e.text())
.collect::<String>();
// Look for patterns like "15,885.00 IDR" in body text
let re2 = Regex::new(&format!(r"([0-9,.]+)\s+{}", to.to_uppercase()))
.map_err(|e| format!("Regex compile: {}", e))?;
if let Some(caps) = re2.captures(&text) {
let rate_str = caps.get(1).map(|m| m.as_str()).unwrap_or("0");
let rate = rate_str.replace(',', "").parse::<f64>().unwrap_or(0.0);
if rate > 0.0 {
let result = amount * rate;
return Ok(json!({
"from": from.to_uppercase(),
"to": to.to_uppercase(),
"amount": amount,
"rate": rate,
"result": result,
"updateTime": chrono::Utc::now().to_rfc3339(),
}));
}
}
Err("Gagal mendapatkan kurs dari wise.com".to_string())
}
/// Scrape harga emas Antam from harga-emas.org.
pub async fn fetch_harga_emas() -> Result<Value, String> {
let url = "https://harga-emas.org/";
let resp = http_client()
.client()
.get(url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
)
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {} from harga-emas.org", resp.status()));
}
let body = resp
.text()
.await
.map_err(|e| format!("Failed to read body: {}", e))?;
let document = Html::parse_document(&body);
let mut result = json!({
"title": "Harga Emas Antam",
"hargaUtama": "N/A",
"perubahan": "0",
"detail": [],
});
// Look for tables containing "Antam" + "Pegadaian"
let table_sel = Selector::parse("table").unwrap();
let tr_sel = Selector::parse("tbody tr").unwrap();
let td_sel = Selector::parse("td").unwrap();
let mut detail: Vec<Value> = Vec::new();
for table in document.select(&table_sel) {
let table_text = table.text().collect::<String>();
if table_text.contains("Antam") && table_text.contains("Pegadaian") {
for tr in table.select(&tr_sel) {
let cols: Vec<String> = tr
.select(&td_sel)
.map(|td| td.text().collect::<String>())
.map(|s| s.trim().to_string())
.collect();
if cols.len() >= 2 {
let label = cols[0].trim();
let price = cols[1].trim();
if !label.is_empty() && !price.is_empty() {
// Check if label is a numeric gram value
let numeric_label: String =
label.chars().filter(|c| c.is_ascii_digit()).collect();
if !numeric_label.is_empty() {
detail.push(json!({
"ukuran": format!("{} gr", label.replace(" gr", "").replace("gr", "").trim()),
"harga": format!("Rp {}", price),
}));
}
}
}
}
break;
}
}
if !detail.is_empty() {
// Find the 1gr price as main price (fallback: smallest gram size).
let one_gr = detail.iter().find(|d| {
d["ukuran"]
.as_str()
.map(|s| {
let s = s.trim();
s == "1 gr" || s.starts_with("1 gr") || s.starts_with("1gr")
})
.unwrap_or(false)
});
let main_price = one_gr
.map(|d| d["harga"].as_str().unwrap_or("N/A"))
.unwrap_or_else(|| detail[0]["harga"].as_str().unwrap_or("N/A"));
result["hargaUtama"] = json!(main_price);
result["detail"] = json!(detail);
}
Ok(result)
}
/// Scrape kurs BCA (jual/beli) from bca.co.id.
pub async fn fetch_kurs_bca() -> Result<Value, String> {
let url = "https://www.bca.co.id/id/informasi/kurs";
let resp = http_client()
.client()
.get(url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
)
.header("Referer", "https://www.google.com/")
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {} from bca.co.id", resp.status()));
}
let body = resp
.text()
.await
.map_err(|e| format!("Failed to read body: {}", e))?;
let document = Html::parse_document(&body);
let table_sel = Selector::parse("table").unwrap();
let tr_sel = Selector::parse("tbody tr").unwrap();
let td_sel = Selector::parse("td").unwrap();
let mut kurs_data: Vec<Value> = Vec::new();
for table in document.select(&table_sel) {
let table_text = table.text().collect::<String>();
if table_text.contains("USD") || table_text.contains("SGD") || table_text.contains("kurs") {
for tr in table.select(&tr_sel) {
let cols: Vec<String> = tr
.select(&td_sel)
.map(|td| td.text().collect::<String>().trim().to_string())
.collect();
if cols.len() >= 3
&& !cols[0].is_empty()
&& !cols[1].is_empty()
&& !cols[2].is_empty()
{
kurs_data.push(json!({
"currency": cols[0],
"jual": cols[1],
"beli": cols[2],
}));
}
}
if !kurs_data.is_empty() {
break;
}
}
}
if kurs_data.is_empty() {
return Err("Gagal mendapatkan data kurs BCA".to_string());
}
Ok(json!({
"source": "BCA",
"url": url,
"data": kurs_data,
}))
}
/// Local server info: OS, CPU, RAM, storage.
pub async fn fetch_server_info() -> Result<Value, String> {
let os_info = json!({
"platform": std::env::consts::OS,
"arch": std::env::consts::ARCH,
"family": std::env::consts::FAMILY,
});
// CPU info from /proc/cpuinfo (Linux)
let cpu_model = tokio::fs::read_to_string("/proc/cpuinfo")
.await
.ok()
.and_then(|s| {
s.lines()
.find(|l| l.starts_with("model name"))
.and_then(|l| l.split(':').nth(1))
.map(|s| s.trim().to_string())
})
.unwrap_or_else(|| "unknown".to_string());
let cpu_cores = std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(1);
// RAM from /proc/meminfo
let mem_info = tokio::fs::read_to_string("/proc/meminfo")
.await
.ok()
.unwrap_or_default();
let total_mem_kb: u64 = mem_info
.lines()
.find(|l| l.starts_with("MemTotal"))
.and_then(|l| l.split_whitespace().nth(1))
.and_then(|s| s.parse().ok())
.unwrap_or(0);
let free_mem_kb: u64 = mem_info
.lines()
.find(|l| l.starts_with("MemAvailable"))
.and_then(|l| l.split_whitespace().nth(1))
.and_then(|s| s.parse().ok())
.unwrap_or(0);
let total_mem_gb = total_mem_kb as f64 / 1048576.0;
let free_mem_gb = free_mem_kb as f64 / 1048576.0;
// Disk usage from `df -B1 /`
let disk_usage = tokio::process::Command::new("df")
.args(["-B1", "/"])
.output()
.await
.ok()
.and_then(|o| String::from_utf8(o.stdout).ok())
.and_then(|out| {
let lines: Vec<&str> = out.lines().collect();
if lines.len() >= 2 {
let parts: Vec<&str> = lines[1].split_whitespace().collect();
if parts.len() >= 4 {
let total: u64 = parts[1].parse().unwrap_or(0);
let used: u64 = parts[2].parse().unwrap_or(0);
let free: u64 = parts[3].parse().unwrap_or(0);
return Some(json!({
"drive": parts[5],
"total": format_gb(total),
"used": format_gb(used),
"free": format_gb(free),
}));
}
}
None
})
.unwrap_or(json!({
"drive": "/",
"total": "N/A",
"used": "N/A",
"free": "N/A",
}));
Ok(json!({
"os": os_info,
"cpu": {
"model": cpu_model,
"cores": cpu_cores,
},
"ram": {
"total": format!("{:.2} GB", total_mem_gb),
"free": format!("{:.2} GB", free_mem_gb),
"used": format!("{:.2} GB", total_mem_gb - free_mem_gb),
},
"storage": disk_usage,
}))
}
fn format_gb(bytes: u64) -> String {
let gb = bytes as f64 / 1_073_741_824.0;
format!("{:.2} GB", gb)
}
+7
View File
@@ -1,10 +1,17 @@
pub mod alqanime; pub mod alqanime;
pub mod downloader;
pub mod image;
pub mod komik; pub mod komik;
pub mod misc;
pub mod otakudesu; pub mod otakudesu;
pub mod parsers; pub mod parsers;
pub mod proxy; pub mod proxy;
pub mod search;
pub mod stalk;
pub mod tools;
pub use alqanime::AlqanimeRepository; pub use alqanime::AlqanimeRepository;
pub use downloader::DownloaderRepository;
pub use komik::KomikRepository; pub use komik::KomikRepository;
pub use otakudesu::OtakudesuRepository; pub use otakudesu::OtakudesuRepository;
pub use proxy::ProxyRepository; pub use proxy::ProxyRepository;
+12 -2
View File
@@ -56,7 +56,10 @@ impl OtakudesuRepository {
} }
pub fn genre_page_url(&self, genre_slug: &str, page: &str) -> String { pub fn genre_page_url(&self, genre_slug: &str, page: &str) -> String {
format!("{}/genre/{}/page/{}/", self.base_url(), genre_slug, page) // Current otakudesu uses the plural `/genres/{slug}/` path (archive
// layout). The singular `/genre/{slug}/page/{page}/` path 301-redirects
// to otakudesu.io (a 404 placeholder), which made genre pages return 0.
format!("{}/genres/{}/page/{}/", self.base_url(), genre_slug, page)
} }
pub fn full_episode_url(&self, slug: &str) -> String { pub fn full_episode_url(&self, slug: &str) -> String {
@@ -144,7 +147,14 @@ impl OtakudesuRepository {
&self, &self,
slug: &str, slug: &str,
) -> Result<(Vec<LatestAnimeItem>, Pagination), ScrapingError> { ) -> Result<(Vec<LatestAnimeItem>, Pagination), ScrapingError> {
let url = self.page_url("latest-anime", slug); // The site's `/latest-anime/` path was removed (301 → otakudesu.io
// placeholder). The homepage IS the latest-episodes page (`.venz ul li`
// with `.thumbz h2.jdlflm`, `.epz`), paginated via WordPress `/page/N/`.
let url = if slug == "1" {
format!("{}/", self.base_url())
} else {
format!("{}/page/{}/", self.base_url(), slug)
};
let html = self.fetch_html(&url).await?; let html = self.fetch_html(&url).await?;
let slug_owned = slug.to_string(); let slug_owned = slug.to_string();
tokio::task::spawn_blocking(move || { tokio::task::spawn_blocking(move || {
@@ -410,6 +410,16 @@ pub fn parse_latest_anime_document(
// SEARCH ANIME PAGE // SEARCH ANIME PAGE
// ============================================================================ // ============================================================================
/// Normalise scraped text: collapse newlines/tabs into single spaces and trim.
fn clean_text(text: String) -> String {
text.replace(['\n', '\t'], " ")
.split_whitespace()
.collect::<Vec<_>>()
.join(" ")
.trim()
.to_string()
}
pub fn parse_search_anime_document( pub fn parse_search_anime_document(
html: &str, html: &str,
page: &str, page: &str,
@@ -417,53 +427,82 @@ pub fn parse_search_anime_document(
let document = parse_html(html); let document = parse_html(html);
let mut items = Vec::new(); let mut items = Vec::new();
let venz_sel = selector(".venz ul li") // Otakudesu search results live under `<ul class="chivsrc"><li>...`, one
.ok_or_else(|| ScrapingError::Parse("Failed to parse .venz ul li selector".into()))?; // result per `<li>` with `<h2><a href="...">Title</a></h2>` plus a series
let title_sel = selector(".thumbz h2.jdlflm") // of `<div class="set"><b>Label</b> : value</div>` rows (Status, Rating,
.ok_or_else(|| ScrapingError::Parse("Failed to parse title selector".into()))?; // Genre...). The old parser targeted the index layout (`.venz ul li`,
let link_sel = selector("a") // `.thumbz h2.jdlflm`, `img`, `.epz`) which does NOT appear on the search
.ok_or_else(|| ScrapingError::Parse("Failed to parse link selector".into()))?; // page, so search silently returned 0 items.
let img_sel = selector("img") let item_sel = selector(".chivsrc li")
.ok_or_else(|| ScrapingError::Parse("Failed to parse img selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse .chivsrc li selector".into()))?;
let ep_sel = selector(".epz") let title_link_sel = selector("h2 a")
.ok_or_else(|| ScrapingError::Parse("Failed to parse epz selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse h2 a selector".into()))?;
let genre_sel = selector(".genre-tag") let set_row_sel = selector(".set")
.ok_or_else(|| ScrapingError::Parse("Failed to parse genre-tag selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse .set selector".into()))?;
let status_sel = selector(".status")
.ok_or_else(|| ScrapingError::Parse("Failed to parse status selector".into()))?;
let rating_sel = selector(".rating")
.ok_or_else(|| ScrapingError::Parse("Failed to parse rating selector".into()))?;
for element in document.select(&venz_sel) { let status_sel = selector(".set b:first-child, .set b")
let title = text_from_or(&element, &title_sel, ""); .ok_or_else(|| ScrapingError::Parse("Failed to parse status label selector".into()))?;
let anime_url = attr_from_or(&element, &link_sel, "href", "");
for element in document.select(&item_sel) {
let title = text_from_or(&element, &title_link_sel, "");
let anime_url = attr_from_or(&element, &title_link_sel, "href", "");
let slug = extract_slug(&anime_url); let slug = extract_slug(&anime_url);
let poster = attr_from_or(&element, &img_sel, "src", "");
let episode = text_from_or(&element, &ep_sel, "N/A");
if title.is_empty() {
continue;
}
// Extract label → value pairs from each `.set` row (e.g. "Status : Ongoing",
// "Rating : 8.0", "Genre : Action, Drama"). Falls back gracefully when the
// value is blank or the row label is absent.
let mut status = String::new();
let mut rating = String::new();
let mut genres = Vec::new(); let mut genres = Vec::new();
for genre_elem in element.select(&genre_sel) { for row in element.select(&set_row_sel) {
genres.push(text(&genre_elem)); let row_text = text(&row);
let lower = row_text.to_lowercase();
if lower.contains("status") {
if let Some(label) = row.select(&status_sel).next() {
let label_len = text(&label).len();
status = clean_text(row_text[label_len..].to_string());
} else {
status = clean_text(row_text.trim_start_matches("Status").to_string());
}
} else if lower.contains("rating") {
let label_len = row
.select(&status_sel)
.next()
.map(|label| text(&label).len())
.unwrap_or("Rating".len());
rating = clean_text(row_text[label_len..].to_string());
} else if lower.contains("genre") {
let label_len = row
.select(&status_sel)
.next()
.map(|label| text(&label).len())
.unwrap_or("Genre".len());
let raw = clean_text(row_text[label_len..].to_string());
genres = raw
.split(',')
.map(|g| g.trim().to_string())
.filter(|g| !g.is_empty())
.collect();
}
} }
let status = text_from_or(&element, &status_sel, ""); items.push(SearchAnimeItem {
let rating = text_from_or(&element, &rating_sel, ""); title,
slug,
if !title.is_empty() { poster: String::new(),
items.push(SearchAnimeItem { episode: String::new(),
title, anime_url,
slug, genres,
poster, status,
episode, rating,
anime_url, description: String::new(),
genres, r#type: String::new(),
status, season: String::new(),
rating, });
description: String::new(),
r#type: String::new(),
season: String::new(),
});
}
} }
let pagination = parse_pagination(page, &document)?; let pagination = parse_pagination(page, &document)?;
@@ -481,23 +520,29 @@ pub fn parse_genre_anime_document(
let document = parse_html(html); let document = parse_html(html);
let mut items = Vec::new(); let mut items = Vec::new();
let venz_sel = selector(".venz ul li") // Current otakudesu genre archive pages use the `.col-anime` card layout
.ok_or_else(|| ScrapingError::Parse("Failed to parse .venz ul li selector".into()))?; // (`.col-md-4 .col-anime`), each with `.col-anime-title a`, `.col-anime-eps`,
let title_sel = selector(".thumbz h2.jdlflm") // `.col-anime-rating`, `.col-anime-cover img` and a synopsis. The old
// `.venz ul li` / `.thumbz h2.jdlflm` / `.epz` layout no longer appears on
// these pages, which made genre filtering silently return 0 items.
let card_sel = selector(".col-anime")
.ok_or_else(|| ScrapingError::Parse("Failed to parse .col-anime selector".into()))?;
let title_sel = selector(".col-anime-title a")
.ok_or_else(|| ScrapingError::Parse("Failed to parse title selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse title selector".into()))?;
let link_sel = selector("a") let eps_sel = selector(".col-anime-eps")
.ok_or_else(|| ScrapingError::Parse("Failed to parse link selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse eps selector".into()))?;
let img_sel = selector("img") let rating_sel = selector(".col-anime-rating")
.ok_or_else(|| ScrapingError::Parse("Failed to parse rating selector".into()))?;
let img_sel = selector(".col-anime-cover img")
.ok_or_else(|| ScrapingError::Parse("Failed to parse img selector".into()))?; .ok_or_else(|| ScrapingError::Parse("Failed to parse img selector".into()))?;
let ep_sel = selector(".epz")
.ok_or_else(|| ScrapingError::Parse("Failed to parse epz selector".into()))?;
for element in document.select(&venz_sel) { for element in document.select(&card_sel) {
let title = text_from_or(&element, &title_sel, ""); let title = text_from_or(&element, &title_sel, "");
let anime_url = attr_from_or(&element, &link_sel, "href", ""); let anime_url = attr_from_or(&element, &title_sel, "href", "");
let slug = extract_slug(&anime_url); let slug = extract_slug(&anime_url);
let poster = attr_from_or(&element, &img_sel, "src", ""); let poster = attr_from_or(&element, &img_sel, "src", "");
let episode = text_from_or(&element, &ep_sel, "N/A"); let episode = text_from_or(&element, &eps_sel, "N/A");
let score = text_from_or(&element, &rating_sel, "");
if !title.is_empty() { if !title.is_empty() {
items.push(GenreAnimeItem { items.push(GenreAnimeItem {
@@ -505,7 +550,7 @@ pub fn parse_genre_anime_document(
slug, slug,
poster, poster,
episode, episode,
score: String::new(), score,
status: String::new(), status: String::new(),
anime_url, anime_url,
}); });
+289
View File
@@ -0,0 +1,289 @@
//! Infrastructure — Search utilities.
//!
//! Ported from Shirokami-API `scraper/search/*.js`:
//! bmkg (BMKG earthquake), jadwal-sholat (myquran.com), weather (OpenWeather),
//! google (HTML scrape), yt (via yt-dlp flat-playlist).
use crate::infrastructure::utils::http_client::http_client;
use reqwest::header::USER_AGENT;
use scraper::{Html, Selector};
use serde_json::{json, Value};
use std::process::Stdio;
/// BMKG Indonesia earthquake info (auto-gempa, terkini, dirasakan).
pub async fn fetch_bmkg() -> Result<Value, String> {
let urls = [
"https://data.bmkg.go.id/DataMKG/TEWS/autogempa.json",
"https://data.bmkg.go.id/DataMKG/TEWS/gempaterkini.json",
"https://data.bmkg.go.id/DataMKG/TEWS/gempadirasakan.json",
];
let mut auto = Value::Null;
let mut terkini: Vec<Value> = Vec::new();
let mut dirasakan: Vec<Value> = Vec::new();
for url in urls {
let resp = http_client()
.client()
.get(url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36",
)
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("BMKG returned HTTP {}", resp.status()));
}
let data: Value = resp
.json()
.await
.map_err(|e| format!("JSON parse: {}", e))?;
if url.contains("autogempa") {
auto = data
.pointer("/Infogempa/gempa")
.cloned()
.unwrap_or(Value::Null);
} else if url.contains("gempaterkini") {
terkini = data
.pointer("/Infogempa/gempa")
.and_then(|g| g.as_array())
.cloned()
.unwrap_or_default();
} else {
dirasakan = data
.pointer("/Infogempa/gempa")
.and_then(|g| g.as_array())
.cloned()
.unwrap_or_default();
}
}
Ok(json!({
"autogempa": auto,
"gempaterkini": terkini,
"gempadirasakan": dirasakan,
}))
}
/// Jadwal sholat from myquran.com.
pub async fn fetch_jadwal_sholat(kota: &str) -> Result<Value, String> {
// 1. Find city id
let search_url = format!(
"https://api.myquran.com/v2/sholat/kota/cari/{}",
urlencode(kota)
);
let city_data = get_json(&search_url).await?;
if city_data
.get("status")
.and_then(|s| s.as_bool())
.unwrap_or(false)
!= true
{
return Ok(json!({ "error": "Kota tidak ditemukan" }));
}
let id_list = city_data
.get("data")
.and_then(|d| d.as_array())
.cloned()
.unwrap_or_default();
if id_list.is_empty() {
return Ok(json!({ "error": "Kota tidak ditemukan" }));
}
// 2. Today's date (Asia/Jakarta). Use chrono with a fixed offset approximation.
let now = chrono::Local::now();
let (year, month, day) = (
now.format("%Y").to_string(),
now.format("%m").to_string(),
now.format("%d").to_string(),
);
let mut results: Vec<Value> = Vec::new();
for city in id_list.iter().take(5) {
let cid = city.get("id").and_then(|i| i.as_str()).unwrap_or("");
let jadwal_url = format!(
"https://api.myquran.com/v2/sholat/jadwal/{}/{}/{}/{}",
cid, year, month, day
);
if let Ok(sched) = get_json(&jadwal_url).await {
if let Some(d) = sched.get("data") {
results.push(json!({
"lokasi": d.get("lokasi").cloned().unwrap_or(Value::Null),
"daerah": d.get("daerah").cloned().unwrap_or(Value::Null),
"jadwal": d.get("jadwal").cloned().unwrap_or(Value::Null),
}));
}
}
}
if results.is_empty() {
return Ok(json!({ "error": "Tidak dapat mengambil jadwal" }));
}
Ok(json!({ "total": results.len(), "schedules": results }))
}
/// OpenWeather current weather (uses env OPENWEATHER_API_KEY, falls back to source's key).
pub async fn fetch_weather(city: &str) -> Result<Value, String> {
let key = std::env::var("OPENWEATHER_API_KEY")
.unwrap_or_else(|_| "060a6bcfa19809c2cd4d97a212b19273".to_string());
let url = format!(
"https://api.openweathermap.org/data/2.5/weather?q={}&units=metric&appid={}",
urlencode(city),
key
);
get_json(&url).await
}
/// Google web search via HTML scraping.
pub async fn fetch_google(query: &str) -> Result<Value, String> {
let url = format!(
"https://www.google.com/search?q={}&safe=off&hl=en&gl=us",
urlencode(query)
);
let resp = http_client()
.client()
.get(&url)
.header(USER_AGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36")
.header("Accept-Language", "en-US")
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
let body = resp
.text()
.await
.map_err(|e| format!("Failed to read body: {}", e))?;
let document = Html::parse_document(&body);
let result_sel =
Selector::parse("div.g, div[data-hveid]").unwrap_or_else(|_| Selector::parse("a").unwrap());
let link_sel = Selector::parse("a").unwrap();
let h3_sel = Selector::parse("h3").unwrap();
let mut results: Vec<Value> = Vec::new();
for node in document.select(&result_sel) {
let link = node
.select(&link_sel)
.find_map(|a| a.value().attr("href"))
.and_then(|h| h.strip_prefix("/url?q="))
.map(|h| h.split('&').next().unwrap_or(h).to_string());
let title = node
.select(&h3_sel)
.next()
.map(|h| h.text().collect::<String>());
if title.is_some() && link.is_some() {
results.push(json!({
"title": title.unwrap_or_default(),
"description": "",
"link": link.unwrap_or_default(),
}));
}
if results.len() >= 10 {
break;
}
}
Ok(json!(results))
}
/// YouTube search via yt-dlp flat playlist (real results, scraping).
pub async fn fetch_yt_search(query: &str) -> Result<Value, String> {
let q = format!("ytsearch15:{}", query);
let output = run_ytdlp(&q).await?;
let mut videos: Vec<Value> = Vec::new();
let arr: Vec<Value> = serde_json::from_str(&output).unwrap_or_default();
for v in arr.iter().take(15) {
videos.push(json!({
"title": v.get("title").cloned().unwrap_or(Value::Null),
"id": v.get("id").cloned().unwrap_or(Value::Null),
"url": v.get("webpage_url").cloned().unwrap_or(Value::Null),
"thumbnail": v.get("thumbnail").cloned().unwrap_or(Value::Null),
"duration": v.get("duration").cloned().unwrap_or(Value::Null),
"views": v.get("view_count").cloned().unwrap_or(Value::Null),
"author": {
"name": v.pointer("/channel").cloned().unwrap_or(Value::Null),
"url": v.get("channel_url").cloned().unwrap_or(Value::Null),
},
}));
}
Ok(json!({ "total": videos.len(), "videos": videos }))
}
// ============================================================================
// Helpers
// ============================================================================
async fn get_json(url: &str) -> Result<Value, String> {
let resp = http_client()
.client()
.get(url)
.header(USER_AGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36")
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {} from upstream", resp.status()));
}
resp.json::<Value>()
.await
.map_err(|e| format!("JSON parse: {}", e))
}
/// Run yt-dlp with `--dump-json --flat-playlist` for the given search/query arg.
async fn run_ytdlp(query_arg: &str) -> Result<String, String> {
let bins = [
"/usr/local/bin/yt-dlp-native",
"/home/code/hermes-agent/.venv/bin/yt-dlp",
"yt-dlp",
];
let bin = bins
.iter()
.find(|b| std::path::Path::new(b).exists() || **b == "yt-dlp")
.copied()
.unwrap_or("yt-dlp");
let child = tokio::process::Command::new(bin)
.args(["--dump-json", "--flat-playlist", "--no-warnings", query_arg])
.stdout(Stdio::piped())
.stderr(Stdio::null())
.spawn()
.map_err(|e| format!("Failed to spawn yt-dlp: {}", e))?;
let output = child
.wait_with_output()
.await
.map_err(|e| format!("yt-dlp failed: {}", e))?;
let stdout = String::from_utf8(output.stdout).map_err(|e| format!("Invalid UTF-8: {}", e))?;
// `--flat-playlist` outputs multiple JSON lines (one per video).
let mut joined = String::new();
joined.push('[');
let mut first = true;
for line in stdout.lines() {
let line = line.trim();
if line.is_empty() {
continue;
}
if !first {
joined.push(',');
}
joined.push_str(line);
first = false;
}
joined.push(']');
Ok(joined)
}
fn urlencode(s: &str) -> String {
let mut out = String::new();
for b in s.bytes() {
match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => {
out.push(b as char)
}
b' ' => out.push_str("%20"),
_ => out.push_str(&format!("%{:02X}", b)),
}
}
out
}
+458
View File
@@ -0,0 +1,458 @@
//! Infrastructure — Stalk utilities (github, youtube, twitter).
//!
//! Ported from Shirokami-API `scraper/stalk/*.js`.
use crate::infrastructure::utils::http_client::http_client;
use regex::Regex;
use reqwest::header::USER_AGENT;
use serde_json::{json, Value};
/// GitHub user stalk via public API.
pub async fn fetch_github_stalk(username: &str) -> Result<Value, String> {
let url = format!("https://api.github.com/users/{}", username);
let resp = http_client()
.client()
.get(&url)
.header(USER_AGENT, "Scraper/1.0")
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
let status = resp.status().as_u16();
return Err(format!("GitHub user not found (HTTP {})", status));
}
let data: Value = resp
.json()
.await
.map_err(|e| format!("JSON parse: {}", e))?;
let pick = |k: &str| -> Value { data.get(k).cloned().unwrap_or(Value::Null) };
Ok(json!({
"username": username,
"login": pick("login"),
"id": pick("id"),
"node_id": pick("node_id"),
"avatar_url": pick("avatar_url"),
"html_url": pick("html_url"),
"type": pick("type"),
"site_admin": pick("site_admin"),
"name": pick("name"),
"company": pick("company"),
"blog": pick("blog"),
"location": pick("location"),
"email": pick("email"),
"hireable": pick("hireable"),
"bio": pick("bio"),
"twitter_username": pick("twitter_username"),
"public_repos": pick("public_repos"),
"public_gists": pick("public_gists"),
"followers": pick("followers"),
"following": pick("following"),
"created_at": pick("created_at"),
"updated_at": pick("updated_at"),
}))
}
/// YouTube channel stalk: parse ytInitialData JSON from channel page.
/// Fetches the `/<handle>/videos` subpage (richGridRenderer with lockupViewModel).
pub async fn fetch_youtube_stalk(username: &str) -> Result<Value, String> {
let url = format!("https://youtube.com/@{username}/videos?hl=en&gl=US");
let resp = http_client()
.client()
.get(&url)
.header(USER_AGENT, "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36")
.header("Accept-Language", "en-US")
.send()
.await
.map_err(|e| format!("HTTP error: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {} from youtube", resp.status()));
}
let body = resp
.text()
.await
.map_err(|e| format!("Failed to read body: {}", e))?;
// Extract var ytInitialData = {...} via brace matching (JSON may contain ";").
let start = body
.find("var ytInitialData = ")
.ok_or_else(|| "ytInitialData not found".to_string())?;
let brace = body[start..]
.find('{')
.ok_or_else(|| "ytInitialData parse".to_string())?
+ start;
let json_str = extract_balanced_json(&body, brace)?;
let parsed: Value =
serde_json::from_str(&json_str).map_err(|e| format!("JSON parse: {}", e))?;
// Channel metadata from metadata.channelMetadataRenderer
let mut channel = json!({
"username": username,
"subscriberCount": Value::Null,
"videoCount": Value::Null,
"avatarUrl": Value::Null,
"channelUrl": Value::Null,
"externalId": Value::Null,
"description": Value::Null,
"isFamilySafe": Value::Null,
});
if let Some(meta) = parsed
.get("metadata")
.and_then(|m| m.get("channelMetadataRenderer"))
{
let mut set = |dst: &str, src: &str| {
if let Some(v) = meta.get(src) {
channel[dst] = v.clone();
}
};
set("description", "description");
set("externalId", "externalId");
set("channelUrl", "channelUrl");
set("isFamilySafe", "isFamilySafe");
}
// Subscriber/video count from header.pageHeaderRenderer
if let Some(header) = parsed
.get("header")
.and_then(|h| h.get("pageHeaderRenderer"))
{
if let Some(rows) = header
.pointer("/content/pageHeaderViewModel/metadata/contentMetadataViewModel/metadataRows")
{
if let Some(arr) = rows.as_array() {
for row in arr {
if let Some(parts) = row.get("metadataParts").and_then(|p| p.as_array()) {
for part in parts {
let text = part
.pointer("/text/content")
.and_then(|t| t.as_str())
.unwrap_or("");
if text.contains("subscriber") {
channel["subscriberCount"] = json!(text.trim());
} else if text.contains(" video") || text.ends_with(" videos") {
channel["videoCount"] = json!(text.trim());
}
}
}
}
}
}
// Avatar
if let Some(sources) = header
.pointer("/content/pageHeaderViewModel/image/decoratedAvatarViewModel/avatar/avatarViewModel/image/sources")
.and_then(|s| s.as_array())
{
if !sources.is_empty() {
channel["avatarUrl"] = sources[0].get("url").cloned().unwrap_or(Value::Null);
}
}
}
// Videos list (up to 5). The `/videos` subpage uses richGridRenderer with
// lockupViewModel (new) or videoRenderer (legacy); we handle both.
let mut videos: Vec<Value> = Vec::new();
if let Some(tabs) = parsed
.pointer("/contents/twoColumnBrowseResultsRenderer/tabs")
.and_then(|t| t.as_array())
{
for tab in tabs {
walk_video_grid(tab, &mut videos);
if videos.len() >= 5 {
break;
}
}
}
// Fallback: source's shelfRenderer/horizontalListRenderer approach.
if videos.is_empty() {
if let Some(tabs) = parsed
.pointer("/contents/twoColumnBrowseResultsRenderer/tabs")
.and_then(|t| t.as_array())
{
if let Some(contents) = tabs
.first()
.and_then(|t| t.pointer("/tabRenderer/content/sectionListRenderer/contents"))
.and_then(|c| c.as_array())
{
for item in contents {
if let Some(section) = item.get("itemSectionRenderer") {
if let Some(inner) = section.get("contents").and_then(|c| c.as_array()) {
for content in inner {
if let Some(shelf) = content.get("shelfRenderer") {
if let Some(items) = shelf
.pointer("/content/horizontalListRenderer/items")
.and_then(|i| i.as_array())
{
for video in items {
if let Some(gv) = video.get("gridVideoRenderer") {
videos.push(video_from_grid_video(gv));
}
}
}
}
}
}
}
if videos.len() >= 5 {
break;
}
}
}
}
}
Ok(json!({
"channelMetadata": channel,
"videoDataList": videos.iter().take(5).cloned().collect::<Vec<_>>(),
}))
}
/// Extract a balanced JSON object string starting at `open_brace` (index of '{').
fn extract_balanced_json(body: &str, open_brace: usize) -> Result<String, String> {
let bytes = body.as_bytes();
let mut depth = 0i32;
let mut in_str = false;
let mut escaped = false;
let mut close = None;
for (i, b) in bytes.iter().enumerate().skip(open_brace) {
let c = *b as char;
if in_str {
if escaped {
escaped = false;
} else if c == '\\' {
escaped = true;
} else if c == '"' {
in_str = false;
}
} else {
match c {
'"' => in_str = true,
'{' => depth += 1,
'}' => {
depth -= 1;
if depth == 0 {
close = Some(i);
break;
}
}
_ => {}
}
}
}
let close = close.ok_or_else(|| "unbalanced JSON in ytInitialData".to_string())?;
Ok(body[open_brace..=close].to_string())
}
/// Recursively find `richItemRenderer` entries and convert their content to video objects.
fn walk_video_grid(node: &Value, out: &mut Vec<Value>) {
match node {
Value::Object(map) => {
if let Some(item) = map.get("richItemRenderer") {
if let Some(content) = item.get("content") {
if let Some(v) = content.get("videoRenderer") {
out.push(video_from_video_renderer(v));
} else if let Some(lv) = content.get("lockupViewModel") {
out.push(video_from_lockup(lv));
} else if let Some(sh) = content.get("shortsLockupViewModel") {
out.push(video_from_shorts(sh));
}
}
}
for v in map.values() {
if out.len() >= 5 {
return;
}
walk_video_grid(v, out);
}
}
Value::Array(arr) => {
for v in arr {
if out.len() >= 5 {
return;
}
walk_video_grid(v, out);
}
}
_ => {}
}
}
/// Parse a legacy `videoRenderer` into our video shape.
fn video_from_video_renderer(vd: &Value) -> Value {
let mut v = json!({
"videoId": vd.get("videoId").cloned().unwrap_or(Value::Null),
"title": vd.pointer("/title/runs/0/text").cloned().unwrap_or(Value::Null),
"thumbnail": vd.pointer("/thumbnail/thumbnails/0/url").cloned().unwrap_or(Value::Null),
"publishedTime": vd.pointer("/publishedTimeText/simpleText").cloned().unwrap_or(Value::Null),
"viewCount": vd.pointer("/viewCountText/simpleText").cloned().unwrap_or(Value::Null),
"navigationUrl": Value::Null,
"duration": Value::Null,
});
if let Some(url) = vd.pointer("/navigationEndpoint/commandMetadata/webCommandMetadata/url") {
v["navigationUrl"] = url.clone();
}
if let Some(dur) = vd
.pointer("/thumbnailOverlays")
.and_then(|o| o.as_array())
.and_then(|arr| {
arr.iter().find_map(|ov| {
ov.pointer("/thumbnailOverlayTimeStatusRenderer/text/simpleText")
.cloned()
})
})
{
v["duration"] = dur;
}
v
}
/// Parse a `lockupViewModel` (newer YouTube renderer) into our video shape.
fn video_from_lockup(lv: &Value) -> Value {
// videoId from thumbnail URL `i.ytimg.com/vi/<ID>/...`
let thumb = lv
.pointer("/contentImage/thumbnailViewModel/image/sources/0/url")
.and_then(|u| u.as_str())
.unwrap_or("");
let video_id = Regex::new(r"vi/([A-Za-z0-9_-]{11})/")
.ok()
.and_then(|re| re.captures(thumb))
.and_then(|c| c.get(1))
.map(|m| m.as_str().to_string())
.unwrap_or_default();
let title = lv
.pointer("/metadata/lockupMetadataViewModel/title/content")
.and_then(|t| t.as_str())
.unwrap_or("")
.to_string();
let mut view_count = Value::Null;
let mut published = Value::Null;
if let Some(parts) = lv
.pointer("/metadata/lockupMetadataViewModel/metadata/contentMetadataViewModel/metadataRows/0/metadataParts")
.and_then(|p| p.as_array())
{
if let Some(p0) = parts.first() {
view_count = p0
.pointer("/text/content")
.cloned()
.unwrap_or(Value::Null);
}
if let Some(p1) = parts.get(1) {
published = p1
.pointer("/text/content")
.cloned()
.unwrap_or(Value::Null);
}
}
let duration = lv
.pointer("/contentImage/thumbnailViewModel/image/overlays/0/thumbnailBottomOverlayViewModel/badges/0/thumbnailBadgeViewModel/text")
.cloned()
.unwrap_or(Value::Null);
json!({
"videoId": json!(video_id),
"title": json!(title),
"thumbnail": json!(thumb),
"publishedTime": published,
"viewCount": view_count,
"navigationUrl": Value::Null,
"duration": duration,
})
}
/// Parse a `shortsLockupViewModel` (Shorts) into our video shape (best-effort).
fn video_from_shorts(sh: &Value) -> Value {
let video_id = sh
.pointer("/overlayMetadata/primaryText/content")
.and_then(|c| c.as_str())
.unwrap_or("");
json!({
"videoId": json!(video_id),
"title": Value::Null,
"thumbnail": Value::Null,
"publishedTime": Value::Null,
"viewCount": Value::Null,
"navigationUrl": Value::Null,
"duration": Value::Null,
})
}
/// Parse a legacy `gridVideoRenderer` into our video shape.
fn video_from_grid_video(gv: &Value) -> Value {
let mut v = json!({
"videoId": gv.get("videoId").cloned().unwrap_or(Value::Null),
"title": gv.pointer("/title/simpleText").cloned().unwrap_or(Value::Null),
"thumbnail": gv.pointer("/thumbnail/thumbnails/0/url").cloned().unwrap_or(Value::Null),
"publishedTime": gv.pointer("/publishedTimeText/simpleText").cloned().unwrap_or(Value::Null),
"viewCount": gv.pointer("/viewCountText/simpleText").cloned().unwrap_or(Value::Null),
"navigationUrl": Value::Null,
"duration": Value::Null,
});
if let Some(nav) = gv.pointer("/navigationEndpoint/commandMetadata/webCommandMetadata/url") {
v["navigationUrl"] = nav.clone();
}
if let Some(dur) = gv
.pointer("/thumbnailOverlays")
.and_then(|o| o.as_array())
.and_then(|arr| {
arr.iter().find_map(|ov| {
ov.pointer("/thumbnailOverlayTimeStatusRenderer/text/simpleText")
.cloned()
})
})
{
v["duration"] = dur;
}
v
}
// ---------------------------------------------------------------------------
// Twitter — fxtwitter API
// ---------------------------------------------------------------------------
pub async fn fetch_twitter_stalk(username: &str) -> Result<Value, String> {
let url = format!("https://api.fxtwitter.com/{}", username);
let resp = http_client()
.client()
.get(&url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/130.0 Safari/537.36",
)
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
let data: Value = resp.json().await.map_err(|e| format!("JSON: {}", e))?;
// Transform to the source's shape
let u = &data["user"];
Ok(json!({
"message": data.get("message").cloned().unwrap_or(Value::Null),
"user": {
"id": u.get("id").cloned().unwrap_or(Value::Null),
"url": u.get("url").cloned().unwrap_or(Value::Null),
"screen_name": u.get("screen_name").cloned().unwrap_or(Value::Null),
"name": u.get("name").cloned().unwrap_or(Value::Null),
"location": u.get("location").cloned().unwrap_or(Value::Null),
"description": u.get("description").cloned().unwrap_or(Value::Null),
"followers": u.get("followers").cloned().unwrap_or(Value::Null),
"following": u.get("following").cloned().unwrap_or(Value::Null),
"likes": u.get("likes").cloned().unwrap_or(Value::Null),
"banner_url": u.get("banner_url").cloned().unwrap_or(Value::Null),
"avatar_url": u.get("avatar_url").cloned().unwrap_or(Value::Null),
"joined_at": u.get("joined").cloned().unwrap_or(Value::Null),
"website": u.get("website").cloned().unwrap_or(Value::Null),
}
}))
}
+438
View File
@@ -0,0 +1,438 @@
//! Infrastructure — Tool utilities.
//!
//! Ported from Shirokami-API `scraper/tool/*.js` and `scraper/tool/*.js`:
//! whois, ip-location, tinyurl, check-hosting, cek-resi, hargapangan.
use crate::infrastructure::utils::http_client::http_client;
use regex::Regex;
use reqwest::header::USER_AGENT;
use scraper::{Html, Selector};
use serde_json::{json, Value};
use urlencoding::encode as urlencode;
const CHROME_UA: &str = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36";
// ---------------------------------------------------------------------------
// Helpers
// ---------------------------------------------------------------------------
// ---------------------------------------------------------------------------
// WHOIS — RDAP lookup (whois.com HTML `.df-raw` is dead/JS-rendered since 2026;
// RDAP is the structured, reliable server-side source for the same data).
// ---------------------------------------------------------------------------
pub async fn fetch_whois(domain: &str) -> Result<Value, String> {
let domain = domain.to_lowercase();
let tld = domain.rsplit('.').next().unwrap_or("com");
let base = match tld {
"com" | "net" => "https://rdap.verisign.com",
"org" => "https://rdap.org",
"io" => "https://rdap.identitydigital.services",
"co" => "https://rdap.nic.co",
"id" => "https://rdap.idnnic.id",
_ => "https://rdap.org", // generic bootstrap
};
let url = if base == "https://rdap.verisign.com" {
format!("{}/{}/v1/domain/{}", base, tld, domain)
} else {
format!("{}/domain/{}", base, domain)
};
let resp = http_client()
.client()
.get(&url)
.header(
USER_AGENT,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0 Safari/537.36",
)
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
if resp.status() == reqwest::StatusCode::NOT_FOUND {
return Ok(
json!({"domain": domain, "available": true, "valid": true, "data": "No data", "rawData": "No data"}),
);
}
if !resp.status().is_success() {
return Ok(
json!({"domain": domain, "valid": false, "data": "RDAP unavailable", "rawData": "No data"}),
);
}
let data: Value = resp.json().await.map_err(|e| format!("JSON: {}", e))?;
let event_date = |action: &str| -> String {
data.get("events")
.and_then(|e| e.as_array())
.map(|events| {
events
.iter()
.find(|ev| ev.get("eventAction").and_then(|a| a.as_str()) == Some(action))
.and_then(|ev| ev.get("eventDate").and_then(|d| d.as_str()))
.unwrap_or("No data")
.to_string()
})
.unwrap_or_else(|| "No data".to_string())
};
let domain_name = data
.get("ldhName")
.and_then(|v| v.as_str())
.unwrap_or(&domain)
.to_string();
let domain_id = data
.get("handle")
.and_then(|v| v.as_str())
.unwrap_or("No data")
.to_string();
let creation = event_date("registration");
let expiration = event_date("expiration");
let updated = event_date("last changed");
// Registrar name from the registrar entity's vcardArray
let registrar = data
.get("entities")
.and_then(|e| e.as_array())
.map(|entities| {
entities
.iter()
.find(|en| {
en.get("roles")
.and_then(|r| r.as_array())
.map(|roles| roles.iter().any(|ro| ro.as_str() == Some("registrar")))
.unwrap_or(false)
})
.and_then(|reg| {
reg.get("vcardArray")
.and_then(|v| v.as_array())
.and_then(|arr| arr.get(1))
.and_then(|props| props.as_array())
.and_then(|props| {
props.iter().find_map(|p| {
let pa = p.as_array()?;
if pa.get(0)?.as_str() == Some("fn") {
pa.get(3).and_then(|v| v.as_str()).map(|s| s.to_string())
} else {
None
}
})
})
})
.unwrap_or_else(|| "No data".to_string())
})
.unwrap_or_else(|| "No data".to_string());
Ok(json!({
"domain": domain,
"available": false,
"valid": true,
"data": {
"domainName": domain_name,
"registryDomainId": domain_id,
"creationDate": creation,
"expirationDate": expiration,
"updatedDate": updated,
"country": "No data",
"registrar": registrar,
},
"rawData": serde_json::to_string(&data).unwrap_or_default(),
}))
}
// ---------------------------------------------------------------------------
// IP Location — ipapi.co JSON
// ---------------------------------------------------------------------------
pub async fn fetch_ip_location(ip: &str) -> Result<Value, String> {
let url = format!("https://ipapi.co/{}/json/", ip);
let resp = http_client()
.client()
.get(&url)
.header(USER_AGENT, "nodejs-ipapi-v1.02")
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
resp.json().await.map_err(|e| format!("JSON: {}", e))
}
// ---------------------------------------------------------------------------
// TinyURL — tinyurl.com API
// ---------------------------------------------------------------------------
pub async fn fetch_tinyurl(url: &str) -> Result<Value, String> {
let api_url = format!("https://tinyurl.com/api-create.php?url={}", urlencode(url));
let resp = http_client()
.client()
.get(&api_url)
.header(USER_AGENT, CHROME_UA)
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
let short_url = resp.text().await.map_err(|e| format!("Body: {}", e))?;
Ok(json!({"shortUrl": short_url.trim()}))
}
// ---------------------------------------------------------------------------
// Check Hosting — hosting-checker.net API
// ---------------------------------------------------------------------------
pub async fn fetch_check_hosting(domain: &str) -> Result<Value, String> {
let cd = domain.to_lowercase().trim().to_string();
// Strip scheme and www
let cd = cd
.trim_start_matches("http://")
.trim_start_matches("https://")
.trim_start_matches("www.")
.to_string();
let domain_re =
Regex::new(r"^(?:[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,}$").unwrap();
if !domain_re.is_match(&cd) {
return Err("Invalid domain format".into());
}
let url = format!("https://hosting-checker.net/api/hosting/{}", cd);
let resp = http_client()
.client()
.get(&url)
.header("authority", "hosting-checker.net")
.header(USER_AGENT, "Postify/1.0.0")
.header(
"referer",
format!("https://hosting-checker.net/websites/{}", cd),
)
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {}", resp.status()));
}
let data: Value = resp.json().await.map_err(|e| format!("JSON: {}", e))?;
// Pass through the relevant fields (same structure as JS version)
Ok(json!({
"domain": {
"name": data["web"]["domain"],
"original": data["web"]["originalDomain"],
"ipv6_support": data["web"]["ipV6Support"],
},
"web": {
"ips": data["web"]["lookups"],
"providers": data["web"]["providers"],
},
"nameserver": data["nameserver"],
"mail": {"incoming": data["incomingMail"], "outgoing": data["outgoingMail"]},
}))
}
// ---------------------------------------------------------------------------
// Harga Pangan (BAPANAS) — Indonesian government food prices
// ---------------------------------------------------------------------------
const BAPANAS_KEY: &str = "zHWbt7U2qvPoUDkiUgvnOqYrtj3zClR7unnH2G4apE7HcMV4QyNC6BSD0yV3uvSHqS91TxwE8aMDTiCznmGceEX3zQmO1Xwq7TJblotIt2CpwvK6YjRKDJwcgMJwav9p4RshM3nfuFyurSQQv9BhueMJ0HJ778oD";
pub async fn fetch_hargapangan() -> Result<Value, String> {
let url = "https://api-panelhargav2.badanpangan.go.id/api/front/harga-pangan-informasi";
let resp = http_client()
.client()
.get(url)
.header(USER_AGENT, CHROME_UA)
.header("accept", "application/json")
.header("x-api-key", BAPANAS_KEY)
.header("origin", "https://panelharga.badanpangan.go.id")
.header("referer", "https://panelharga.badanpangan.go.id/")
.query(&[("level_harga_id", "1")])
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?;
if !resp.status().is_success() {
return Err(format!("HTTP {}", resp.status()));
}
let data: Value = resp.json().await.map_err(|e| format!("JSON: {}", e))?;
if data.get("status").and_then(|s| s.as_str()) == Some("success") {
Ok(data.get("data").cloned().unwrap_or(Value::Null))
} else {
Ok(Value::Null)
}
}
// ---------------------------------------------------------------------------
// Cek Resi (cekresi.com) — Indonesian package tracking
// ---------------------------------------------------------------------------
fn aes_encrypt_resi(resi: &str) -> String {
use aes::cipher::generic_array::GenericArray;
use aes::cipher::{block_padding::Pkcs7, BlockEncryptMut, KeyIvInit};
type Aes128CbcEnc = cbc::Encryptor<aes::Aes128>;
let key_hex = "79540e250fdb16afac03e19c46dbdeb3";
let iv_hex = "eb2bb9425e81ffa942522e4414e95bd0";
let key_bytes = hex::decode(key_hex).expect("bad key hex");
let iv_bytes = hex::decode(iv_hex).expect("bad iv hex");
let key = GenericArray::from_slice(&key_bytes);
let iv = GenericArray::from_slice(&iv_bytes);
let data = resi.as_bytes();
let mut buf = vec![0u8; data.len() + 16];
buf[..data.len()].copy_from_slice(data);
let ct = Aes128CbcEnc::new(key, iv)
.encrypt_padded_mut::<Pkcs7>(&mut buf, data.len())
.expect("encrypt");
base64::Engine::encode(&base64::engine::general_purpose::STANDARD, ct)
}
fn detect_ekspedisi(resi: &str) -> Option<&'static str> {
let upper = resi.to_uppercase();
let map = [
("jne", "JNE"),
("pos", "POS"),
("tiki", "TIKI"),
("jnt", "J&T"),
("sicepat", "SICEPAT"),
("anteraja", "ANTERAJA"),
("ninja", "NINJA"),
("lionparcel", "LIONPARCEL"),
];
for (prefix, code) in map {
if upper.starts_with(code) {
return Some(prefix);
}
}
None
}
pub async fn fetch_cek_resi(noresi: &str, ekspedisi: Option<&str>) -> Result<Value, String> {
let noresi = noresi.trim().to_uppercase();
let ekspedisi = match ekspedisi {
Some(e) => e.to_string(),
None => detect_ekspedisi(&noresi)
.ok_or_else(|| "Ekspedisi tidak ditemukan dari nomor resi".to_string())?
.to_string(),
};
let timers = aes_encrypt_resi(&noresi);
// 1. GET the form to extract viewstate + secret_key
let client = http_client().client();
let html = client
.get("https://cekresi.com/")
.header(USER_AGENT, CHROME_UA)
.send()
.await
.map_err(|e| format!("HTTP: {}", e))?
.text()
.await
.map_err(|e| format!("Body: {}", e))?;
let (viewstate, secret_key) = {
let doc = Html::parse_document(&html);
let input_sel = Selector::parse(r#"input[name="viewstate"]"#).unwrap();
let secret_sel = Selector::parse(r#"input[name="secret_key"]"#).unwrap();
let vs = doc
.select(&input_sel)
.next()
.and_then(|e| e.value().attr("value"))
.unwrap_or("")
.to_string();
let sk = doc
.select(&secret_sel)
.next()
.and_then(|e| e.value().attr("value"))
.unwrap_or("")
.to_string();
(vs, sk)
};
// 2. POST the tracking request
let w = &format!("{:x}", fastrand::u32(..));
let post_url = format!(
"https://apa2.cekresi.com/cekresi/resi/initialize.php?ui=e0ad7e971ce77822056ba7a155f85c11&p=1&w={}",
w
);
let form = [
("viewstate", viewstate.as_str()),
("secret_key", secret_key.as_str()),
("e", &ekspedisi),
("noresi", &noresi),
("timers", &timers),
];
let resp = client.post(&post_url)
.header(USER_AGENT, "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36")
.header("referer", "https://cekresi.com/")
.header("origin", "https://cekresi.com")
.form(&form)
.send().await.map_err(|e| format!("HTTP: {}", e))?;
let body = resp.text().await.map_err(|e| format!("Body: {}", e))?;
let doc2 = Html::parse_document(&body);
// 3. Parse results
let alert_sel = Selector::parse(".alert.alert-success").unwrap();
let success = doc2.select(&alert_sel).next().is_some();
if success {
let message = doc2
.select(&alert_sel)
.next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_default();
let table_sel = Selector::parse("table.table-striped tbody tr").unwrap();
let mut status = String::new();
let mut tanggal = String::new();
for row in doc2.select(&table_sel) {
let tds: Vec<_> = row.select(&Selector::parse("td").unwrap()).collect();
if tds.len() >= 3 {
let label = tds[0].text().collect::<String>().trim().to_string();
let value = tds[2].text().collect::<String>().trim().to_string();
if label == "Tanggal Pengiriman" {
tanggal = value;
} else if label == "Status" {
status = value;
}
}
}
let last_pos = Selector::parse("#last_position").unwrap();
let last_position = doc2
.select(&last_pos)
.next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_default();
// History: scan all tables for the one whose rows have exactly 2 cells
// (tanggal, keterangan). The status table above uses 3 cells, so the
// 2-cell table after the "History" heading is unambiguous.
let mut history: Vec<Value> = Vec::new();
let table_sel_all = Selector::parse("table").unwrap();
let td_sel = Selector::parse("td").unwrap();
for table in doc2.select(&table_sel_all) {
let mut rows: Vec<Value> = Vec::new();
for tr in table.select(&Selector::parse("tbody tr").unwrap()) {
let tds: Vec<_> = tr.select(&td_sel).collect();
if tds.len() == 2 {
rows.push(json!({
"tanggal": tds[0].text().collect::<String>().trim(),
"keterangan": tds[1].text().collect::<String>().trim(),
}));
}
}
// A table with 2-cell rows is the history table; skip its header row.
if rows.len() > 1 {
history = rows.into_iter().skip(1).collect();
break;
}
}
Ok(json!({
"success": true,
"message": message,
"data": {
"resi": noresi,
"ekspedisi": doc2.select(&Selector::parse("#nama_expedisi").unwrap()).next()
.map(|e| e.text().collect::<String>().trim().to_string()).unwrap_or_default(),
"status": status,
"tanggalKirim": tanggal,
"lastPosition": last_position,
"history": history,
}
}))
} else {
let alert_err = Selector::parse(".alert.alert-danger, .alert.alert-warning").unwrap();
let message = doc2
.select(&alert_err)
.next()
.map(|e| e.text().collect::<String>().trim().to_string())
.unwrap_or_else(|| "Tidak dapat mengambil informasi resi".to_string());
Ok(json!({"success": false, "message": message}))
}
}
@@ -20,6 +20,13 @@ pub fn get_production_url() -> String {
} }
/// Get Komik API URL from environment config. /// Get Komik API URL from environment config.
///
/// Defaults to `https://api.komiku.org`. This is the API mirror whose
/// sub-paths (`/manga/?tipe=manga`, `/genre/<slug>/`, `/search/<q>/`) serve
/// the `.bge` grid layout the parsers expect. NOTE: `api.komiku.org`'s ROOT
/// path returns an empty body — the komik genre-list endpoint must therefore
/// fetch from `komiku.org` (base URL) instead of this root (see
/// `KomikUseCase::genre_list`, which uses `base_url()`).
pub fn get_komik_api_url() -> String { pub fn get_komik_api_url() -> String {
env::var("KOMIK2_API_URL").unwrap_or_else(|_| "https://api.komiku.org".to_string()) env::var("KOMIK2_API_URL").unwrap_or_else(|_| "https://api.komiku.org".to_string())
} }
+7 -3
View File
@@ -4,6 +4,8 @@
//! - Global `MeterProvider` connected via OTLP gRPC to the metrics backend //! - Global `MeterProvider` connected via OTLP gRPC to the metrics backend
//! - Standard HTTP server metrics middleware //! - Standard HTTP server metrics middleware
//! //!
//! #![allow(clippy::all)]
//!
//! Environment: //! Environment:
//! OTEL_EXPORTER_OTLP_ENDPOINT — default: http://localhost:4317 //! OTEL_EXPORTER_OTLP_ENDPOINT — default: http://localhost:4317
//! OTEL_SERVICE_NAME — default: scraper-api //! OTEL_SERVICE_NAME — default: scraper-api
@@ -47,16 +49,18 @@ pub fn init_otel_metrics() {
// Build the gRPC OTLP exporter // Build the gRPC OTLP exporter
let exporter = opentelemetry_otlp::MetricExporter::builder() let exporter = opentelemetry_otlp::MetricExporter::builder()
.with_tonic() .with_http()
.with_endpoint(endpoint.clone()) .with_endpoint(endpoint.clone())
.build() .build()
.expect("Failed to create OTLP metric exporter"); .expect("Failed to create OTLP metric exporter");
let reader = PeriodicReader::builder(exporter, opentelemetry_sdk::runtime::Tokio) let reader = PeriodicReader::builder(exporter)
.with_interval(std::time::Duration::from_millis(export_interval_ms)) .with_interval(std::time::Duration::from_millis(export_interval_ms))
.build(); .build();
let resource = Resource::new(vec![KeyValue::new("service.name", service_name.clone())]); let resource = Resource::builder()
.with_service_name(service_name.clone())
.build();
let provider = MeterProviderBuilder::default() let provider = MeterProviderBuilder::default()
.with_resource(resource) .with_resource(resource)
+62 -6
View File
@@ -4,7 +4,7 @@ use utoipa::OpenApi;
#[derive(OpenApi)] #[derive(OpenApi)]
#[openapi( #[openapi(
paths( paths(
// Anime module handlers // ── Anime module ──────────────────────────────────────────
crate::presentation::handler::anime::anime_index, crate::presentation::handler::anime::anime_index,
crate::presentation::handler::anime::genres, crate::presentation::handler::anime::genres,
crate::presentation::handler::anime::detail_slug, crate::presentation::handler::anime::detail_slug,
@@ -16,7 +16,7 @@ use utoipa::OpenApi;
crate::presentation::handler::anime::search_slug_page, crate::presentation::handler::anime::search_slug_page,
crate::presentation::handler::anime::genre_slug_index, crate::presentation::handler::anime::genre_slug_index,
crate::presentation::handler::anime::genre_slug_page, crate::presentation::handler::anime::genre_slug_page,
// Anime2 module handlers // ── Anime2 module ─────────────────────────────────────────
crate::presentation::handler::anime2::index, crate::presentation::handler::anime2::index,
crate::presentation::handler::anime2::genre_list, crate::presentation::handler::anime2::genre_list,
crate::presentation::handler::anime2::filter, crate::presentation::handler::anime2::filter,
@@ -28,7 +28,7 @@ use utoipa::OpenApi;
crate::presentation::handler::anime2::latest_slug, crate::presentation::handler::anime2::latest_slug,
crate::presentation::handler::anime2::ongoing_anime_slug, crate::presentation::handler::anime2::ongoing_anime_slug,
crate::presentation::handler::anime2::complete_anime_slug, crate::presentation::handler::anime2::complete_anime_slug,
// Komik module handlers // ── Komik module ──────────────────────────────────────────
crate::presentation::handler::komik::genre_list, crate::presentation::handler::komik::genre_list,
crate::presentation::handler::komik::chapter_slug, crate::presentation::handler::komik::chapter_slug,
crate::presentation::handler::komik::detail_slug, crate::presentation::handler::komik::detail_slug,
@@ -40,19 +40,75 @@ use utoipa::OpenApi;
crate::presentation::handler::komik::popular_slug, crate::presentation::handler::komik::popular_slug,
crate::presentation::handler::komik::search_slug, crate::presentation::handler::komik::search_slug,
crate::presentation::handler::komik::search_slug_page, crate::presentation::handler::komik::search_slug_page,
// Proxy module handlers // ── Download module ────────────────────────────────────────
crate::presentation::handler::downloader::download,
crate::presentation::handler::downloader::detect_platform_handler,
crate::presentation::handler::downloader::download_instagram,
crate::presentation::handler::downloader::download_facebook,
crate::presentation::handler::downloader::download_tiktok,
crate::presentation::handler::downloader::download_youtube,
crate::presentation::handler::downloader::download_youtube_mp3,
crate::presentation::handler::downloader::download_spotify,
crate::presentation::handler::downloader::download_twitter,
crate::presentation::handler::downloader::download_pinterest,
crate::presentation::handler::downloader::download_mega,
crate::presentation::handler::downloader::download_terabox,
crate::presentation::handler::downloader::download_gdrive,
crate::presentation::handler::downloader::download_mediafire,
crate::presentation::handler::downloader::download_pixeldrain,
crate::presentation::handler::downloader::download_threads,
crate::presentation::handler::downloader::download_doodstream,
crate::presentation::handler::downloader::download_krakenfiles,
crate::presentation::handler::downloader::download_danbooru,
crate::presentation::handler::downloader::download_soundcloud,
crate::presentation::handler::downloader::download_dailymotion,
crate::presentation::handler::downloader::download_reddit,
crate::presentation::handler::downloader::download_streamable,
crate::presentation::handler::downloader::download_videy,
crate::presentation::handler::downloader::download_bilibili,
crate::presentation::handler::downloader::serve_merged_file,
// ── Misc module ────────────────────────────────────────────
crate::presentation::handler::misc::currency_converter_handler,
crate::presentation::handler::misc::harga_emas_handler,
crate::presentation::handler::misc::kurs_bca_handler,
crate::presentation::handler::misc::server_info_handler,
// ── Stalk module ───────────────────────────────────────────
crate::presentation::handler::stalk::github_handler,
crate::presentation::handler::stalk::youtube_handler,
crate::presentation::handler::stalk::twitter_handler,
// ── Search module ──────────────────────────────────────────
crate::presentation::handler::search::bmkg_handler,
crate::presentation::handler::search::jadwal_sholat_handler,
crate::presentation::handler::search::weather_handler,
crate::presentation::handler::search::google_handler,
crate::presentation::handler::search::yt_handler,
// ── Tool module ────────────────────────────────────────────
crate::presentation::handler::tools::whois_handler,
crate::presentation::handler::tools::ip_location_handler,
crate::presentation::handler::tools::tinyurl_handler,
crate::presentation::handler::tools::check_hosting_handler,
crate::presentation::handler::tools::hargapangan_handler,
crate::presentation::handler::tools::cek_resi_handler,
// ── Image module ───────────────────────────────────────────
crate::presentation::handler::image::brat_handler,
crate::presentation::handler::image::brat_animated_handler,
), ),
components( components(
schemas( schemas(
// Application response wrapper
crate::presentation::dto::common::ApiResponse<String>, crate::presentation::dto::common::ApiResponse<String>,
crate::presentation::dto::downloader::DownloadResponse,
) )
), ),
tags( tags(
(name = "anime", description = "Anime endpoints"), (name = "anime", description = "Anime endpoints"),
(name = "anime2", description = "Anime2 endpoints"), (name = "anime2", description = "Anime2 endpoints"),
(name = "komik", description = "Komik endpoints"), (name = "komik", description = "Komik endpoints"),
(name = "proxy", description = "Proxy endpoints"), (name = "download", description = "Media download endpoints (23 platforms)"),
(name = "misc", description = "Miscellaneous utilities (currency, gold, server)"),
(name = "stalk", description = "User profile stalking (GitHub, YT, Twitter)"),
(name = "search", description = "Search endpoints (BMKG, prayer times, weather, YouTube, Google)"),
(name = "tool", description = "Utility tools (WHOIS, IP lookup, URL shortener, tracking, hosting)"),
(name = "image", description = "Image generation (brat static & animated)"),
) )
)] )]
pub struct ModuleApiDoc; pub struct ModuleApiDoc;
+53
View File
@@ -0,0 +1,53 @@
//! Presentation DTOs for media downloader endpoints.
use serde::Serialize;
use utoipa::ToSchema;
use crate::domain::entity::downloader::DownloadResult;
/// Standard response wrapper for all downloader endpoints.
///
/// Mirrors the Shirokami-API pattern of `{ success, ...data }` but
/// adds `status` and `message` for consistency with the anime/komik modules.
#[derive(Serialize, Debug, Clone, ToSchema)]
pub struct DownloadResponse {
pub success: bool,
pub status: i16,
#[serde(skip_serializing_if = "Option::is_none")]
pub message: Option<String>,
pub data: Option<DownloadResult>,
}
impl From<DownloadResult> for DownloadResponse {
fn from(data: DownloadResult) -> Self {
let success = data.status == crate::domain::entity::downloader::DownloadStatus::Success;
Self {
success,
status: if success { 200 } else { 400 },
message: data.message.clone(),
data: Some(data),
}
}
}
impl DownloadResponse {
pub fn ok(data: DownloadResult) -> Self {
let success = data.status == crate::domain::entity::downloader::DownloadStatus::Success;
let msg = data.message.clone();
Self {
success,
status: 200,
message: msg,
data: Some(data),
}
}
pub fn error(status: i16, message: impl Into<String>) -> Self {
Self {
success: false,
status,
message: Some(message.into()),
data: None,
}
}
}
+1
View File
@@ -1,2 +1,3 @@
pub mod common; pub mod common;
pub mod downloader;
pub mod komik; pub mod komik;
+10
View File
@@ -132,3 +132,13 @@ impl IntoResponse for AppError {
(status, body).into_response() (status, body).into_response()
} }
} }
impl From<ScrapingError> for AppError {
fn from(e: ScrapingError) -> Self {
// Map ScrapingError::Http to BAD_GATEWAY (502) — the scraper code
// correctly returns DownloadResult::error for client-side issues
// (invalid URL, bad format), so any ScraperError that escapes is
// an upstream provider failure, not a local bug.
AppError::ScraperError(e.to_string())
}
}
+741
View File
@@ -0,0 +1,741 @@
//! HTTP handlers for media download endpoints.
//!
//! Each handler wraps an application-layer use case, parses request params,
//! and returns a JSON response. Platform is auto-detected from the URL
//! via the all-in-one dispatcher.
use axum::extract::Query;
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use utoipa::IntoParams;
use crate::application::downloader as use_cases;
use crate::presentation::dto::downloader::DownloadResponse;
use crate::presentation::error::AppError;
/// Request params for downloader endpoints.
#[derive(Debug, Deserialize, IntoParams)]
pub struct DownloadParams {
pub url: String,
pub cookies: Option<String>,
pub api_key: Option<String>,
pub quality: Option<String>,
/// When `true`, YouTube downloads are merged server-side (video + audio
/// into a single MP4) instead of returning separate video/audio URLs.
pub merge: Option<String>,
}
// ========================================================================
// Handlers
// ========================================================================
/// All-in-one auto-detect downloader.
/// `/download?url=...`
#[utoipa::path(
get,
path = "/download",
tag = "download",
operation_id = "dl_download",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let merge = params
.merge
.as_deref()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
let result = if merge {
// All-in-one with merge=true → force merge for YouTube URLs.
if crate::application::downloader::detect_platform(&params.url) == "youtube" {
use_cases::download_youtube_merge(
&params.url,
params.quality.as_deref().unwrap_or("720"),
)
.await?
} else {
use_cases::download_all_in_one(&params.url, params.cookies.as_deref()).await?
}
} else {
use_cases::download_all_in_one(&params.url, params.cookies.as_deref()).await?
};
Ok(Json(DownloadResponse::ok(result)))
}
/// Detect platform from URL.
/// `/detect-platform?url=...`
#[utoipa::path(
get,
path = "/detect-platform",
tag = "download",
operation_id = "dl_detect_platform_handler",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn detect_platform_handler(
Query(params): Query<DownloadParams>,
) -> Result<Json<Value>, AppError> {
let platform = use_cases::detect_platform(&params.url);
Ok(Json(serde_json::json!({
"platform": platform,
"url": params.url,
})))
}
/// Download Instagram media.
/// `/download/instagram?url=...`
#[utoipa::path(
get,
path = "/download/instagram",
tag = "download",
operation_id = "dl_download_instagram",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_instagram(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_instagram(&params.url, params.cookies.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Facebook video.
/// `/download/facebook?url=...`
#[utoipa::path(
get,
path = "/download/facebook",
tag = "download",
operation_id = "dl_download_facebook",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_facebook(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_facebook(&params.url, params.cookies.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download TikTok video.
/// `/download/tiktok?url=...`
#[utoipa::path(
get,
path = "/download/tiktok",
tag = "download",
operation_id = "dl_download_tiktok",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_tiktok(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_tiktok(&params.url, params.cookies.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download YouTube video.
/// `/download/youtube?url=...`
#[utoipa::path(
get,
path = "/download/youtube",
tag = "download",
operation_id = "dl_download_youtube",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_youtube(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let quality = params.quality.as_deref().unwrap_or("720");
let merge = params
.merge
.as_deref()
.map(|v| v == "true" || v == "1")
.unwrap_or(false);
let result = if merge {
use_cases::download_youtube_merge(&params.url, quality).await?
} else {
use_cases::download_youtube(&params.url, quality).await?
};
Ok(Json(DownloadResponse::ok(result)))
}
/// Convert YouTube to MP3.
/// `/download/youtube/mp3?url=...`
#[utoipa::path(
get,
path = "/download/youtube/mp3",
tag = "download",
operation_id = "dl_download_youtube_mp3",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_youtube_mp3(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_youtube_mp3(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Spotify track.
/// `/download/spotify?url=...`
#[utoipa::path(
get,
path = "/download/spotify",
tag = "download",
operation_id = "dl_download_spotify",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_spotify(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_spotify(&params.url, params.api_key.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Twitter/X media.
/// `/download/twitter?url=...`
#[utoipa::path(
get,
path = "/download/twitter",
tag = "download",
operation_id = "dl_download_twitter",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_twitter(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_twitter(&params.url, params.cookies.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Pinterest pin.
/// `/download/pinterest?url=...`
#[utoipa::path(
get,
path = "/download/pinterest",
tag = "download",
operation_id = "dl_download_pinterest",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_pinterest(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_pinterest(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from Mega.
/// `/download/mega?url=...`
#[utoipa::path(
get,
path = "/download/mega",
tag = "download",
operation_id = "dl_download_mega",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_mega(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_mega(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from Terabox.
/// `/download/terabox?url=...`
#[utoipa::path(
get,
path = "/download/terabox",
tag = "download",
operation_id = "dl_download_terabox",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_terabox(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_terabox(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Stream a TeraBox file through this API.
/// `/proxy/terabox?surl=...`
///
/// The resolver's `/download` endpoint replays the full 302 chain WITH
/// cookies and streams real bytes — this handler forwards those bytes to the
/// caller so the returned URL is a public, downloadable link (not a 403
/// dlink, not a private 127.0.0.1 URL).
#[utoipa::path(
get,
path = "/proxy/terabox",
tag = "download",
operation_id = "dl_proxy_terabox",
params(
("surl" = String, Query, description = "TeraBox share short URL ID"),
("filename" = Option<String>, Query, description = "Download file name hint"),
),
responses(
(status = 200, description = "File stream"),
(status = 400, description = "Missing surl"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn proxy_terabox(
Query(params): Query<TeraboxProxyParams>,
) -> Result<axum::response::Response, AppError> {
let surl = params.surl;
if surl.is_empty() {
return Err(AppError::BadRequest("missing surl parameter".into()));
}
let resolver_url = std::env::var("TERABOX_RESOLVER_URL")
.unwrap_or_else(|_| "http://127.0.0.1:4092".to_string());
let fetch_url = format!("{}/download?surl={}", resolver_url, surl);
// Ask the resolver for the redirect chain (it follows 302s internally with
// cookies) — but we only want the final CDN stream URL. The resolver
// already streams; simplest reliable path is to just proxy the resolver's
// response body as-is.
let client = crate::infrastructure::utils::http_client::http_client();
let resp = client
.client()
.get(&fetch_url)
.header("User-Agent", "Mozilla/5.0")
.timeout(std::time::Duration::from_secs(120))
.send()
.await
.map_err(|e| AppError::ScraperError(format!("terabox proxy fetch failed: {e}")))?;
if !resp.status().is_success() {
let status = resp.status();
let body = resp.text().await.unwrap_or_default();
return Err(AppError::ScraperError(format!(
"terabox proxy returned HTTP {status}: {body}"
)));
}
let content_type = resp
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.to_string();
let content_disposition = resp
.headers()
.get(reqwest::header::CONTENT_DISPOSITION)
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string())
.unwrap_or_else(|| {
let fname = params
.filename
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "terabox-download.bin".to_string());
format!("attachment; filename=\"{fname}\"")
});
let stream = resp.bytes_stream();
Ok(axum::response::Response::builder()
.header("Content-Type", content_type)
.header("Content-Disposition", content_disposition)
.header("X-Accel-Buffering", "no")
.header("Cache-Control", "no-store")
.body(axum::body::Body::from_stream(stream))
.map_err(|e| AppError::Internal(format!("build stream response: {e}")))?)
}
/// Params for the TeraBox streaming proxy.
#[derive(Debug, Deserialize, IntoParams)]
pub struct TeraboxProxyParams {
pub surl: String,
pub filename: Option<String>,
}
/// Download from Google Drive.
/// `/download/gdrive?url=...`
#[utoipa::path(
get,
path = "/download/gdrive",
tag = "download",
operation_id = "dl_download_gdrive",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_gdrive(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_gdrive(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from MediaFire.
/// `/download/mediafire?url=...`
#[utoipa::path(
get,
path = "/download/mediafire",
tag = "download",
operation_id = "dl_download_mediafire",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_mediafire(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_mediafire(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from Pixeldrain.
/// `/download/pixeldrain?url=...`
#[utoipa::path(
get,
path = "/download/pixeldrain",
tag = "download",
operation_id = "dl_download_pixeldrain",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_pixeldrain(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_pixeldrain(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Threads media.
/// `/download/threads?url=...`
#[utoipa::path(
get,
path = "/download/threads",
tag = "download",
operation_id = "dl_download_threads",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_threads(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_threads(&params.url, params.cookies.as_deref()).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from DoodStream.
/// `/download/doodstream?url=...`
#[utoipa::path(
get,
path = "/download/doodstream",
tag = "download",
operation_id = "dl_download_doodstream",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_doodstream(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_doodstream(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from KrakenFiles.
/// `/download/krakenfiles?url=...`
#[utoipa::path(
get,
path = "/download/krakenfiles",
tag = "download",
operation_id = "dl_download_krakenfiles",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_krakenfiles(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_krakenfiles(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download from Danbooru.
/// `/download/danbooru?url=...`
#[utoipa::path(
get,
path = "/download/danbooru",
tag = "download",
operation_id = "dl_download_danbooru",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_danbooru(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_danbooru(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download SoundCloud track.
/// `/download/soundcloud?url=...`
#[utoipa::path(
get,
path = "/download/soundcloud",
tag = "download",
operation_id = "dl_download_soundcloud",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_soundcloud(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_soundcloud(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Dailymotion video.
/// `/download/dailymotion?url=...`
#[utoipa::path(
get,
path = "/download/dailymotion",
tag = "download",
operation_id = "dl_download_dailymotion",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_dailymotion(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_dailymotion(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Reddit media.
/// `/download/reddit?url=...`
#[utoipa::path(
get,
path = "/download/reddit",
tag = "download",
operation_id = "dl_download_reddit",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_reddit(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_reddit(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Streamable video.
/// `/download/streamable?url=...`
#[utoipa::path(
get,
path = "/download/streamable",
tag = "download",
operation_id = "dl_download_streamable",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_streamable(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_streamable(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Videy video.
/// `/download/videy?url=...`
#[utoipa::path(
get,
path = "/download/videy",
tag = "download",
operation_id = "dl_download_videy",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_videy(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_videy(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
/// Download Bilibili video.
/// `/download/bilibili?url=...`
#[utoipa::path(
get,
path = "/download/bilibili",
tag = "download",
operation_id = "dl_download_bilibili",
params(DownloadParams),
responses(
(status = 200, description = "Download result", body = DownloadResponse),
(status = 400, description = "Bad URL or platform"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn download_bilibili(
Query(params): Query<DownloadParams>,
) -> Result<Json<DownloadResponse>, AppError> {
let result = use_cases::download_bilibili(&params.url).await?;
Ok(Json(DownloadResponse::ok(result)))
}
// ========================================================================
// Merged-file static serving
// ========================================================================
/// Serve a merged YouTube MP4 produced by `/download/youtube?merge=true`.
/// Path is sanitised: only a plain filename (no `/`, `..`) inside the
/// `yt_merge` uploads dir is allowed.
///
/// `/file/yt_merge/{filename}`
#[utoipa::path(
get,
path = "/file/yt_merge/{filename}",
tag = "download",
operation_id = "dl_serve_merged_file",
params(
("filename" = String, Path, description = "Merged MP4 file name")
),
responses(
(status = 200, description = "Merged MP4 file", content_type = "video/mp4"),
(status = 404, description = "File not found or invalid name")
)
)]
pub async fn serve_merged_file(
axum::extract::Path(filename): axum::extract::Path<String>,
) -> Result<axum::response::Response, AppError> {
// Reject any path traversal or nested segments.
if filename.contains('/')
|| filename.contains('\\')
|| filename.contains("..")
|| filename.is_empty()
{
return Err(AppError::NotFound(format!("invalid file name: {filename}")));
}
let dir = std::path::PathBuf::from("/var/lib/scraper/uploads/yt_merge");
let path = dir.join(&filename);
let bytes = tokio::fs::read(&path)
.await
.map_err(|_| AppError::NotFound(format!("file not found: {filename}")))?;
let content_type = if filename.ends_with(".mp4") {
"video/mp4"
} else if filename.ends_with(".webm") {
"video/webm"
} else {
"application/octet-stream"
};
Ok(axum::response::Response::builder()
.header("Content-Type", content_type)
.header("Content-Length", bytes.len().to_string())
.header(
"Content-Disposition",
format!("attachment; filename=\"{filename}\""),
)
.body(axum::body::Body::from(bytes))
.map_err(|e| AppError::Internal(format!("build response: {e}")))?)
}
+68
View File
@@ -0,0 +1,68 @@
//! Axum handlers for image generation endpoints.
//!
//! Returns raw image bytes with the proper Content-Type.
use axum::body::Body;
use axum::extract::Query;
use axum::http::{header, StatusCode};
use axum::response::Response;
use serde::Deserialize;
use utoipa::IntoParams;
use crate::application::image as use_cases;
use crate::presentation::error::AppError;
#[derive(Deserialize, IntoParams)]
pub struct BratParams {
pub text: Option<String>,
}
/// GET /image/brat?text=Hello
#[utoipa::path(
get,
path = "/image/brat",
tag = "image",
operation_id = "image_brat",
params(BratParams),
responses(
(status = 200, description = "Static brat image (PNG)", body = Vec<u8>),
(status = 400, description = "Missing text"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn brat_handler(Query(params): Query<BratParams>) -> Result<Response, AppError> {
let text = params
.text
.ok_or_else(|| AppError::BadRequest("text parameter is required".into()))?;
let bytes = use_cases::brat(&text).await?;
Response::builder()
.status(StatusCode::OK)
.header(header::CONTENT_TYPE, "image/png")
.body(Body::from(bytes))
.map_err(|e| AppError::Internal(format!("build response: {e}")))
}
/// GET /image/brat/animated?text=Hello
#[utoipa::path(
get,
path = "/image/brat/animated",
tag = "image",
operation_id = "image_brat_animated",
params(BratParams),
responses(
(status = 200, description = "Animated brat image (GIF)", body = Vec<u8>),
(status = 400, description = "Missing text"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn brat_animated_handler(Query(params): Query<BratParams>) -> Result<Response, AppError> {
let text = params
.text
.ok_or_else(|| AppError::BadRequest("text parameter is required".into()))?;
let bytes = use_cases::brat_animated(&text).await?;
Response::builder()
.status(StatusCode::OK)
.header(header::CONTENT_TYPE, "image/gif")
.body(Body::from(bytes))
.map_err(|e| AppError::Internal(format!("build response: {e}")))
}
+92
View File
@@ -0,0 +1,92 @@
//! Axum handlers for misc utility endpoints.
//!
//! Ported from Shirokami-API `scraper/misc/*.js`.
use axum::extract::Query;
use axum::http::StatusCode;
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use utoipa::IntoParams;
use crate::application::misc as use_cases;
use crate::presentation::error::AppError;
/// Query params for currency converter.
#[derive(Deserialize, IntoParams)]
pub struct CurrencyParams {
pub amount: Option<f64>,
pub from: Option<String>,
pub to: Option<String>,
}
/// GET /misc/currency-converter?amount=100&from=USD&to=IDR
#[utoipa::path(
get,
path = "/misc/currency-converter",
tag = "misc",
operation_id = "currency_converter",
params(CurrencyParams),
responses(
(status = 200, description = "Currency conversion result", body = Value),
(status = 502, description = "Upstream error"),
)
)]
pub async fn currency_converter_handler(
Query(params): Query<CurrencyParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let amount = params.amount.unwrap_or(1.0);
let from = params.from.unwrap_or_else(|| "USD".to_string());
let to = params.to.unwrap_or_else(|| "IDR".to_string());
let result = use_cases::currency_converter(amount, &from, &to).await?;
Ok((StatusCode::OK, Json(result)))
}
/// GET /misc/harga-emas
#[utoipa::path(
get,
path = "/misc/harga-emas",
tag = "misc",
operation_id = "harga_emas",
responses(
(status = 200, description = "Gold prices (Antam)", body = Value),
(status = 502, description = "Upstream error"),
)
)]
pub async fn harga_emas_handler() -> Result<(StatusCode, Json<Value>), AppError> {
let result = use_cases::harga_emas().await?;
Ok((StatusCode::OK, Json(result)))
}
/// GET /misc/kurs-bca
#[utoipa::path(
get,
path = "/misc/kurs-bca",
tag = "misc",
operation_id = "kurs_bca",
responses(
(status = 200, description = "BCA exchange rates", body = Value),
(status = 502, description = "Upstream error"),
)
)]
pub async fn kurs_bca_handler() -> Result<(StatusCode, Json<Value>), AppError> {
let result = use_cases::kurs_bca().await?;
Ok((StatusCode::OK, Json(result)))
}
/// GET /misc/server-info
#[utoipa::path(
get,
path = "/misc/server-info",
tag = "misc",
operation_id = "server_info",
responses(
(status = 200, description = "Server info", body = Value),
(status = 502, description = "Upstream error"),
)
)]
pub async fn server_info_handler() -> Result<(StatusCode, Json<Value>), AppError> {
let result = use_cases::server_info().await?;
Ok((StatusCode::OK, Json(result)))
}
+6
View File
@@ -1,5 +1,11 @@
pub mod anime; pub mod anime;
pub mod anime2; pub mod anime2;
pub mod downloader;
pub mod health; pub mod health;
pub mod image;
pub mod komik; pub mod komik;
pub mod misc;
pub mod proxy; pub mod proxy;
pub mod search;
pub mod stalk;
pub mod tools;
+122
View File
@@ -0,0 +1,122 @@
//! Axum handlers for search utilities.
//!
//! Ported from Shirokami-API `scraper/search/*.js`.
use axum::extract::Query;
use axum::http::StatusCode;
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use crate::application::search as use_cases;
use crate::presentation::error::AppError;
#[derive(Deserialize)]
pub struct SearchParams {
pub query: Option<String>,
pub q: Option<String>,
pub kota: Option<String>,
pub city: Option<String>,
}
fn require(v: Option<&String>, name: &str) -> Result<String, AppError> {
v.cloned()
.ok_or_else(|| AppError::BadRequest(format!("Missing '{}' parameter", name)))
}
/// GET /search/bmkg
#[utoipa::path(
get,
path = "/search/bmkg",
tag = "search",
operation_id = "search_bmkg_handler",
responses(
(status = 200, description = "OK", body = serde_json::Value),
(status = 400, description = "Bad Request"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn bmkg_handler() -> Result<(StatusCode, Json<Value>), AppError> {
Ok((StatusCode::OK, Json(use_cases::bmkg().await?)))
}
/// GET /search/jadwal-sholat?kota=jakarta
#[utoipa::path(
get,
path = "/search/jadwal-sholat",
tag = "search",
operation_id = "search_jadwal_sholat_handler",
responses(
(status = 200, description = "OK", body = serde_json::Value),
(status = 400, description = "Bad Request"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn jadwal_sholat_handler(
Query(p): Query<SearchParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let kota = require(p.kota.as_ref(), "kota")?;
Ok((StatusCode::OK, Json(use_cases::jadwal_sholat(&kota).await?)))
}
/// GET /search/weather?city=jakarta
#[utoipa::path(
get,
path = "/search/weather",
tag = "search",
operation_id = "search_weather_handler",
responses(
(status = 200, description = "OK", body = serde_json::Value),
(status = 400, description = "Bad Request"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn weather_handler(
Query(p): Query<SearchParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let city = require(p.city.as_ref().or(p.q.as_ref()), "city")?;
Ok((StatusCode::OK, Json(use_cases::weather(&city).await?)))
}
/// GET /search/google?query=rust
#[utoipa::path(
get,
path = "/search/google",
tag = "search",
operation_id = "search_google_handler",
responses(
(status = 200, description = "OK", body = serde_json::Value),
(status = 400, description = "Bad Request"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn google_handler(
Query(p): Query<SearchParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let query = require(p.query.as_ref().or(p.q.as_ref()), "query")?;
Ok((StatusCode::OK, Json(use_cases::google(&query).await?)))
}
/// GET /search/yt?query=rickroll
#[utoipa::path(
get,
path = "/search/yt",
tag = "search",
operation_id = "search_yt_handler",
responses(
(status = 200, description = "OK", body = serde_json::Value),
(status = 400, description = "Bad Request"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn yt_handler(
Query(p): Query<SearchParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let query = require(p.query.as_ref().or(p.q.as_ref()), "query")?;
Ok((StatusCode::OK, Json(use_cases::yt_search(&query).await?)))
}
+92
View File
@@ -0,0 +1,92 @@
//! Axum handlers for stalk (profile/user lookups).
//!
//! Ported from Shirokami-API `scraper/stalk/*.js`.
use axum::extract::Query;
use axum::http::StatusCode;
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use utoipa::IntoParams;
use crate::application::stalk as use_cases;
use crate::presentation::error::AppError;
/// Query params for stalk endpoints.
#[derive(Deserialize, IntoParams)]
pub struct StalkParams {
pub username: Option<String>,
pub user_id: Option<String>,
pub userid: Option<String>,
pub zone_id: Option<String>,
pub zoneid: Option<String>,
}
/// GET /stalk/github?username=asepharyana
#[utoipa::path(
get,
path = "/stalk/github",
tag = "stalk",
operation_id = "stalk_github",
params(StalkParams),
responses(
(status = 200, description = "GitHub user profile", body = Value),
(status = 400, description = "Missing username"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn github_handler(
Query(params): Query<StalkParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let username = params
.username
.ok_or_else(|| AppError::BadRequest("Missing 'username' parameter".to_string()))?;
let result = use_cases::github(&username).await?;
Ok((StatusCode::OK, Json(result)))
}
/// GET /stalk/youtube?username=<handle>
#[utoipa::path(
get,
path = "/stalk/youtube",
tag = "stalk",
operation_id = "stalk_youtube",
params(StalkParams),
responses(
(status = 200, description = "YouTube channel profile", body = Value),
(status = 400, description = "Missing username"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn youtube_handler(
Query(params): Query<StalkParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let username = params
.username
.ok_or_else(|| AppError::BadRequest("Missing 'username' parameter".to_string()))?;
let result = use_cases::youtube(&username).await?;
Ok((StatusCode::OK, Json(result)))
}
/// GET /stalk/twitter?username=...
#[utoipa::path(
get,
path = "/stalk/twitter",
tag = "stalk",
operation_id = "stalk_twitter",
params(StalkParams),
responses(
(status = 200, description = "Twitter/X user profile", body = Value),
(status = 400, description = "Missing username"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn twitter_handler(
Query(params): Query<StalkParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let username = params
.username
.ok_or_else(|| AppError::BadRequest("Missing 'username' parameter".to_string()))?;
let result = use_cases::twitter(&username).await?;
Ok((StatusCode::OK, Json(result)))
}
+162
View File
@@ -0,0 +1,162 @@
//! Axum handlers for tool endpoints.
use axum::extract::Query;
use axum::http::StatusCode;
use axum::Json;
use serde::Deserialize;
use serde_json::Value;
use utoipa::IntoParams;
use crate::application::tools as use_cases;
use crate::presentation::error::AppError;
#[derive(Deserialize, IntoParams)]
pub struct DomainParams {
pub domain: Option<String>,
}
#[derive(Deserialize, IntoParams)]
pub struct IpParams {
pub ip: Option<String>,
}
#[derive(Deserialize, IntoParams)]
pub struct UrlParams {
pub url: Option<String>,
}
#[derive(Deserialize, IntoParams)]
pub struct ResiParams {
pub resi: Option<String>,
pub ekspedisi: Option<String>,
}
/// GET /tool/whois?domain=example.com
#[utoipa::path(
get,
path = "/tool/whois",
tag = "tool",
operation_id = "tool_whois",
params(DomainParams),
responses(
(status = 200, description = "WHOIS info", body = Value),
(status = 400, description = "Missing domain"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn whois_handler(
Query(params): Query<DomainParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let domain = params
.domain
.ok_or_else(|| AppError::BadRequest("domain parameter is required".into()))?;
Ok((StatusCode::OK, Json(use_cases::whois(&domain).await?)))
}
/// GET /tool/ip-location?ip=1.1.1.1
#[utoipa::path(
get,
path = "/tool/ip-location",
tag = "tool",
operation_id = "tool_ip_location",
params(IpParams),
responses(
(status = 200, description = "IP geolocation", body = Value),
(status = 400, description = "Missing IP"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn ip_location_handler(
Query(params): Query<IpParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let ip = params
.ip
.ok_or_else(|| AppError::BadRequest("ip parameter is required".into()))?;
Ok((StatusCode::OK, Json(use_cases::ip_location(&ip).await?)))
}
/// GET /tool/tinyurl?url=https://example.com
#[utoipa::path(
get,
path = "/tool/tinyurl",
tag = "tool",
operation_id = "tool_tinyurl",
params(UrlParams),
responses(
(status = 200, description = "Shortened URL", body = Value),
(status = 400, description = "Missing URL"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn tinyurl_handler(
Query(params): Query<UrlParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let url = params
.url
.ok_or_else(|| AppError::BadRequest("url parameter is required".into()))?;
Ok((StatusCode::OK, Json(use_cases::tinyurl(&url).await?)))
}
/// GET /tool/check-hosting?domain=example.com
#[utoipa::path(
get,
path = "/tool/check-hosting",
tag = "tool",
operation_id = "tool_check_hosting",
params(DomainParams),
responses(
(status = 200, description = "Hosting info", body = Value),
(status = 400, description = "Missing domain"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn check_hosting_handler(
Query(params): Query<DomainParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let domain = params
.domain
.ok_or_else(|| AppError::BadRequest("domain parameter is required".into()))?;
Ok((
StatusCode::OK,
Json(use_cases::check_hosting(&domain).await?),
))
}
/// GET /tool/hargapangan
#[utoipa::path(
get,
path = "/tool/hargapangan",
tag = "tool",
operation_id = "tool_hargapangan",
responses(
(status = 200, description = "Food prices", body = Value),
(status = 502, description = "Upstream error"),
)
)]
pub async fn hargapangan_handler() -> Result<(StatusCode, Json<Value>), AppError> {
Ok((StatusCode::OK, Json(use_cases::hargapangan().await?)))
}
/// GET /tool/cek-resi?resi=...&ekspedisi=...
#[utoipa::path(
get,
path = "/tool/cek-resi",
tag = "tool",
operation_id = "tool_cek_resi",
params(ResiParams),
responses(
(status = 200, description = "Shipment tracking", body = Value),
(status = 400, description = "Missing resi"),
(status = 502, description = "Upstream error"),
)
)]
pub async fn cek_resi_handler(
Query(params): Query<ResiParams>,
) -> Result<(StatusCode, Json<Value>), AppError> {
let ResiParams { resi, ekspedisi } = params;
let resi = resi.ok_or_else(|| AppError::BadRequest("resi parameter is required".into()))?;
Ok((
StatusCode::OK,
Json(use_cases::cek_resi(resi, ekspedisi).await?),
))
}
+194
View File
@@ -155,6 +155,200 @@ pub fn build_router(app_state: Arc<AppState>) -> anyhow::Result<Router> {
) )
// Proxy routes // Proxy routes
// (all proxy routes removed) // (all proxy routes removed)
// Downloader routes
.route(
"/download",
axum::routing::get(crate::presentation::handler::downloader::download),
)
.route(
"/download/detect",
axum::routing::get(crate::presentation::handler::downloader::detect_platform_handler),
)
.route(
"/download/instagram",
axum::routing::get(crate::presentation::handler::downloader::download_instagram),
)
.route(
"/download/facebook",
axum::routing::get(crate::presentation::handler::downloader::download_facebook),
)
.route(
"/download/tiktok",
axum::routing::get(crate::presentation::handler::downloader::download_tiktok),
)
.route(
"/download/youtube",
axum::routing::get(crate::presentation::handler::downloader::download_youtube),
)
.route(
"/download/youtube/mp3",
axum::routing::get(crate::presentation::handler::downloader::download_youtube_mp3),
)
.route(
"/file/yt_merge/{filename}",
axum::routing::get(crate::presentation::handler::downloader::serve_merged_file),
)
.route(
"/download/spotify",
axum::routing::get(crate::presentation::handler::downloader::download_spotify),
)
.route(
"/download/twitter",
axum::routing::get(crate::presentation::handler::downloader::download_twitter),
)
.route(
"/download/pinterest",
axum::routing::get(crate::presentation::handler::downloader::download_pinterest),
)
.route(
"/download/mega",
axum::routing::get(crate::presentation::handler::downloader::download_mega),
)
.route(
"/download/terabox",
axum::routing::get(crate::presentation::handler::downloader::download_terabox),
)
.route(
"/proxy/terabox",
axum::routing::get(crate::presentation::handler::downloader::proxy_terabox),
)
.route(
"/download/gdrive",
axum::routing::get(crate::presentation::handler::downloader::download_gdrive),
)
.route(
"/download/mediafire",
axum::routing::get(crate::presentation::handler::downloader::download_mediafire),
)
.route(
"/download/pixeldrain",
axum::routing::get(crate::presentation::handler::downloader::download_pixeldrain),
)
.route(
"/download/threads",
axum::routing::get(crate::presentation::handler::downloader::download_threads),
)
.route(
"/download/dood",
axum::routing::get(crate::presentation::handler::downloader::download_doodstream),
)
.route(
"/download/kraken",
axum::routing::get(crate::presentation::handler::downloader::download_krakenfiles),
)
.route(
"/download/danbooru",
axum::routing::get(crate::presentation::handler::downloader::download_danbooru),
)
.route(
"/download/soundcloud",
axum::routing::get(crate::presentation::handler::downloader::download_soundcloud),
)
.route(
"/download/dailymotion",
axum::routing::get(crate::presentation::handler::downloader::download_dailymotion),
)
.route(
"/download/reddit",
axum::routing::get(crate::presentation::handler::downloader::download_reddit),
)
.route(
"/download/streamable",
axum::routing::get(crate::presentation::handler::downloader::download_streamable),
)
.route(
"/download/videy",
axum::routing::get(crate::presentation::handler::downloader::download_videy),
)
.route(
"/download/bilibili",
axum::routing::get(crate::presentation::handler::downloader::download_bilibili),
)
// Misc routes
.route(
"/misc/currency-converter",
axum::routing::get(crate::presentation::handler::misc::currency_converter_handler),
)
.route(
"/misc/harga-emas",
axum::routing::get(crate::presentation::handler::misc::harga_emas_handler),
)
.route(
"/misc/kurs-bca",
axum::routing::get(crate::presentation::handler::misc::kurs_bca_handler),
)
.route(
"/misc/server-info",
axum::routing::get(crate::presentation::handler::misc::server_info_handler),
)
// Stalk routes
.route(
"/stalk/github",
axum::routing::get(crate::presentation::handler::stalk::github_handler),
)
.route(
"/stalk/youtube",
axum::routing::get(crate::presentation::handler::stalk::youtube_handler),
)
.route(
"/stalk/twitter",
axum::routing::get(crate::presentation::handler::stalk::twitter_handler),
)
// Search routes
.route(
"/search/bmkg",
axum::routing::get(crate::presentation::handler::search::bmkg_handler),
)
.route(
"/search/jadwal-sholat",
axum::routing::get(crate::presentation::handler::search::jadwal_sholat_handler),
)
.route(
"/search/weather",
axum::routing::get(crate::presentation::handler::search::weather_handler),
)
.route(
"/search/google",
axum::routing::get(crate::presentation::handler::search::google_handler),
)
.route(
"/search/yt",
axum::routing::get(crate::presentation::handler::search::yt_handler),
)
// Tool routes
.route(
"/tool/whois",
axum::routing::get(crate::presentation::handler::tools::whois_handler),
)
.route(
"/tool/iplocation",
axum::routing::get(crate::presentation::handler::tools::ip_location_handler),
)
.route(
"/tool/tinyurl",
axum::routing::get(crate::presentation::handler::tools::tinyurl_handler),
)
.route(
"/tool/check-hosting",
axum::routing::get(crate::presentation::handler::tools::check_hosting_handler),
)
.route(
"/tool/hargapangan",
axum::routing::get(crate::presentation::handler::tools::hargapangan_handler),
)
.route(
"/tool/cek-resi",
axum::routing::get(crate::presentation::handler::tools::cek_resi_handler),
)
// Image routes
.route(
"/image/brat",
axum::routing::get(crate::presentation::handler::image::brat_handler),
)
.route(
"/image/brat/animated",
axum::routing::get(crate::presentation::handler::image::brat_animated_handler),
)
// Health // Health
.route( .route(
"/health", "/health",
+64
View File
@@ -0,0 +1,64 @@
"""Extract TikTok user-detail JSON (__UNIVERSAL_DATA_FOR_REHYDRATION__) via Playwright.
The direct HTTP path gets a SlardarWAF challenge page; a real browser with JS
executes the challenge and exposes the __UNIVERSAL_DATA_FOR_REHYDRATION__ script.
Usage: python3 stalk_tiktok.py <username>
"""
import sys
import asyncio
import json
import os
os.environ.setdefault("PLAYWRIGHT_BROWSERS_PATH", "/usr/local/share/ms-playwright")
from playwright.async_api import async_playwright
async def main(username):
url = f"https://www.tiktok.com/@{username}?_t=ZS-8tHANz7ieoS&_r=1"
async with async_playwright() as p:
browser = await p.chromium.launch(
headless=True,
args=["--no-sandbox", "--disable-bypass", "--disable-blink-features=AutomationControlled"],
)
context = await browser.new_context(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
viewport={"width": 1366, "height": 768},
java_script_enabled=True,
locale="en-US",
)
page = await context.new_page()
try:
await page.goto(url, timeout=60000, wait_until="domcontentloaded")
except Exception as e:
await browser.close()
print(json.dumps({"success": False, "error": f"Navigation: {e}"}))
return
# Wait for the rehydration script to appear (WAF challenge may take a few seconds)
found = None
for _ in range(30):
try:
found = await page.evaluate(
"""() => {
const s = document.querySelector('script#__UNIVERSAL_DATA_FOR_REHYDRATION__');
return s ? s.textContent : null;
}"""
)
if found:
break
except Exception:
pass
await page.wait_for_timeout(1000)
await browser.close()
if not found:
print(json.dumps({"success": False, "error": "Rehydration data not found (WAF challenge)"}))
return
print(json.dumps({"success": True, "data": found}))
if __name__ == "__main__":
asyncio.run(main(sys.argv[1]))