MythEclipse/scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
48 Commits 1 Branch 0 Tags
6df8af54eb6ade25dae66d919d3694dc01e34d59
Commit Graph
9 Commits
Author SHA1 Message Date
asepharyana 6df8af54eb feat(misc): add misc endpoints (currency-converter, harga-emas, kurs-bca, server-info)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Ported from Shirokami-API scraper/misc/*.js:
- /misc/currency-converter (wise.com scrape, amount/from/to)
- /misc/harga-emas (harga-emas.org Antam table)
- /misc/kurs-bca (bca.co.id jual/beli)
- /misc/server-info (OS/CPU/RAM/disk from /proc)

Also: remove /download/tiktok/v2 + /download/twitter/v2 as separate routes;
douyin.wtf + twitsave now internal fallback steps in the primary
fetch_tiktok/fetch_twitter chains (v2 = fallback per user directive).
 2026-09-01 19:05:42 +07:00
asepharyana dc44085cad feat: wire videy + tiktok/v2 + twitter/v2 downloader endpoints
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Ported all missing downloader endpoints from Shirokami-API source
(verified against /home/code/Shirokami-API/scraper/downloader/):

- /download/videy: direct CDN link build (cdn.videy.co/{id}.mp4|.mov)
  matches videy.js exactly (id.len()==9 && id[8]=='2' -> .mov)
- /download/tiktok/v2: douyin.wtf hybrid API w/ fallback to embed scrape
- /download/twitter/v2: twitsave.com HTML scrape w/ fallback to Syndication API
- detect_platform now recognizes videy.co
- download_all_in_one dispatches videy

fetch_videy already existed as dead code (never wired); connected it.
== all others (bilibili/danbooru/dood/gdrive/instagram/kfiles/mediafire/
== mega/pinterest/pixeldrain/savetube/soundcloud/spotify/terabox/threads/
== tiktok/twitter/youtubeV2) already mapped to their Rust equivalents
 2026-09-01 18:36:54 +07:00
asepharyana dfa37f86e3 feat: add Streamable downloader endpoint + fix detect false-positive
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
- /download/streamable: real MP4 via yt-dlp (verified ftypisom)
- Fix detect_platform: t.co substring greedily matched reddit.com
  ('.reddit.com' contains 't.co') - now requires t.co/ path
- Removed duplicate reddit branch in detect chain
 2026-09-01 18:23:49 +07:00
asepharyana 7850b5fa03 feat: add Dailymotion + Reddit downloader endpoints via yt-dlp
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
New platforms from Shirokami-style downloadter set:
- /download/dailymotion: real MP4/HLS streams (288p up to 2160p)
- /download/reddit: real v.redd.it HLS media for public posts
- detect + all-in-one dispatch updated

Both verified working from VPS with real URLs.
 2026-09-01 18:05:35 +07:00
asepharyana c3e6537a63 fix: map ScrapingError to 502 Bad Gateway instead of 500 Internal Server Error; fix downr.org Cloudflare block; fix YouTube savetube DNS failure; fix twitter/pinterest error handling 2026-09-01 11:26:29 +07:00
asepharyana 763a641593 feat(downloader): add 26 downloader APIs + detect endpoint
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Port downloaders from Shirokami-API (Node.js):
- Instagram/Facebook (snapsave.app, downr.org fallback)
- TikTok (tikwm.com)
- YouTube MP4/MP3 (savetube.media, ydlp.yard.id)
- Spotify, Twitter/X (api.lrm.tube)
- Bilibili, Pinterest, MediaFire, Mega, TeraBox
- PixelDrain, Threads, DoodStream, KrakenFiles
- Danbooru, SoundCloud, Google Drive

Domain layer: DownloadResult + MediaItem entities
Application layer: 7 use-cases + detect_platform auto-detect
Infrastructure: DownloaderRepository (22 fetch functions)
Presentation: 21 handlers + DTO + OpenAPI schemas

Also: upgrade OpenTelemetry to 0.28 (axum 0.7 conflict resolution)
 2026-09-01 00:54:06 +07:00
asepharyana c7ffd29e79 refactor: migrate infra to mytheclipse crates (retry, cache, event, ratelimit, config)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Replace hand-rolled infrastructure with the custom mytheclipse library:

- retry (src/infrastructure/scraping/retry.rs): backoff crate -> mytheclipse
  RetryConfig + retry() with retry_all predicate; same call-site helpers.
- cache (src/infrastructure/cache): deadpool raw AsyncCommands ->
  mytheclipse_cache::RedisCache bridge (src/infrastructure/cache/mytheclipse.rs),
  typed JSON wrapper keeps the Cache<'a> API used by use cases.
- events (src/events/bus.rs): custom broadcast pub/sub ->
  mytheclipse_event InMemoryEventBus + TypedEventBus alias.
- ratelimit (src/presentation/middleware/ratelimit.rs): hand-rolled window ->
  mytheclipse::RateLimiter token bucket.
- config (src/config/mod.rs): config crate -> mytheclipse_config ConfigLoader,
  preserving env-only fallback (config files optional).

No route/API/Redis-key changes. mytheclipse-cache path-dep points at local
source (redis 0.32 aligned with deadpool). Removed backoff/config deps.
 2026-08-30 18:53:00 +07:00
asepharyana 5c51ec895f feat: remove croxy proxy and image cache features 2026-07-22 14:51:08 +07:00
asepharyanaandClaude Opus 4.8 776fa828d8 feat: full clean architecture refactor — domain → application → infrastructure → presentation 
- Restructure from Modular MVC to Clean Architecture with 4 strict layers
- Domain: entities (anime/komik/proxy), Repository traits, typed errors
- Application: use case classes with proper error propagation
- Infrastructure: repository impls, parsers, Redis cache, HTTP/scraping, browser
- Presentation: Axum handlers, DTOs, AppState, router, AppError+IntoResponse
- Migrate all parsers (otakudesu, alqanime, komik) to native infra implementations
- Replace once_cell::sync::Lazy/OnceCell with std::sync::LazyLock/OnceLock
- Remove 150+ old files in modules/ and shared/ directories
- Remove once_cell from Cargo.toml dependencies
- Fix test/debug binaries to use new import paths
- Zero new clippy warnings

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 12:54:23 +07:00