MythEclipse/scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
45 Commits 1 Branch 0 Tags
7850b5fa03b2e94eef3c7a9d7e71b0799d8d2091
Commit Graph
11 Commits
Author SHA1 Message Date
asepharyana 763a641593 feat(downloader): add 26 downloader APIs + detect endpoint
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Port downloaders from Shirokami-API (Node.js):
- Instagram/Facebook (snapsave.app, downr.org fallback)
- TikTok (tikwm.com)
- YouTube MP4/MP3 (savetube.media, ydlp.yard.id)
- Spotify, Twitter/X (api.lrm.tube)
- Bilibili, Pinterest, MediaFire, Mega, TeraBox
- PixelDrain, Threads, DoodStream, KrakenFiles
- Danbooru, SoundCloud, Google Drive

Domain layer: DownloadResult + MediaItem entities
Application layer: 7 use-cases + detect_platform auto-detect
Infrastructure: DownloaderRepository (22 fetch functions)
Presentation: 21 handlers + DTO + OpenAPI schemas

Also: upgrade OpenTelemetry to 0.28 (axum 0.7 conflict resolution)
 2026-09-01 00:54:06 +07:00
asepharyana 3d38008af3 build: use published mytheclipse crates for queue + tracing (1.21.2)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
mytheclipse-queue and mytheclipse-tracing are now published on crates.io
(v1.21.2). Switch both from local path deps to version deps so the whole
mytheclipse family resolves from crates.io — no more path deps in the
dependency graph.
 2026-08-30 20:02:01 +07:00
asepharyana e681bbbe51 build: use published mytheclipse crates (1.21.2) via version deps
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
The user published all mytheclipse crates to crates.io at 1.21.2
(including the mytheclipse-cache redis-0.32 fix). Switch the scraper to
version deps for the released crates (mytheclipse, -cache, -config,
-event) so it tracks the published releases instead of local path deps.
mytheclipse-queue + mytheclipse-tracing are not yet on crates.io, so they
remain path deps to the local workspace.
 2026-08-30 19:45:05 +07:00
asepharyana 966d86ff67 refactor: migrate thread/async/queue/scheduler infra to mytheclipse
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
- bootstrap: TracingLayer from mytheclipse-tracing (composed with scraper
  env filter), RuntimeConfig::auto() thread logging, init job queue + cron.
- proxy_fetch: leader task via ::mytheclipse::spawn_io, gzip decompression
  offloaded to ::mytheclipse::compute (sized rayon pool, panic-isolated),
  bounded by new async fetch limiter (tokio Semaphore bridge).
- queue: new infrastructure/queue module over mytheclipse-queue
  (InMemoryQueue + WorkerPool + BackpressureEnforcer) for repair jobs.
- scheduler: new infrastructure/scheduler.rs using mytheclipse::cron for
  the daily 02:00 UTC image-cache cleanup.
- deps: add mytheclipse-queue + mytheclipse-tracing path deps; mytheclipse
  -> full feature; rayon 1.12; keep path deps for unpublished crates.
- tests: infra_round2 runtime smoke tests (compute panic isolation,
  spawn_io, backpressure admission, cron parse, queue roundtrip).
- Fix: local cache::mytheclipse bridge module shadowed the mytheclipse
  crate name; use leading :: at spawn_io/compute call sites.
 2026-08-30 19:40:01 +07:00
asepharyana c7ffd29e79 refactor: migrate infra to mytheclipse crates (retry, cache, event, ratelimit, config)
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Details 
Replace hand-rolled infrastructure with the custom mytheclipse library:

- retry (src/infrastructure/scraping/retry.rs): backoff crate -> mytheclipse
  RetryConfig + retry() with retry_all predicate; same call-site helpers.
- cache (src/infrastructure/cache): deadpool raw AsyncCommands ->
  mytheclipse_cache::RedisCache bridge (src/infrastructure/cache/mytheclipse.rs),
  typed JSON wrapper keeps the Cache<'a> API used by use cases.
- events (src/events/bus.rs): custom broadcast pub/sub ->
  mytheclipse_event InMemoryEventBus + TypedEventBus alias.
- ratelimit (src/presentation/middleware/ratelimit.rs): hand-rolled window ->
  mytheclipse::RateLimiter token bucket.
- config (src/config/mod.rs): config crate -> mytheclipse_config ConfigLoader,
  preserving env-only fallback (config files optional).

No route/API/Redis-key changes. mytheclipse-cache path-dep points at local
source (redis 0.32 aligned with deadpool). Removed backoff/config deps.
 2026-08-30 18:53:00 +07:00
asepharyana 7640e66c76 chore: deny unused_mut, unreachable_code, trivial_casts, trivial_numeric_casts, explicit_outlives_requirements, unused_labels, unused_braces, noop_method_call, clippy::unnecessary_to_owned 2026-07-22 15:03:13 +07:00
asepharyana 96ce29b0f3 chore: set dead_code lint to deny 2026-07-22 14:52:19 +07:00
asepharyana 5c51ec895f feat: remove croxy proxy and image cache features 2026-07-22 14:51:08 +07:00
asepharyana 4cd04410cf chore: remove 17 unused Rust dependencies from scraper 2026-07-22 14:12:58 +07:00
asepharyanaandClaude Opus 4.8 776fa828d8 feat: full clean architecture refactor — domain → application → infrastructure → presentation 
- Restructure from Modular MVC to Clean Architecture with 4 strict layers
- Domain: entities (anime/komik/proxy), Repository traits, typed errors
- Application: use case classes with proper error propagation
- Infrastructure: repository impls, parsers, Redis cache, HTTP/scraping, browser
- Presentation: Axum handlers, DTOs, AppState, router, AppError+IntoResponse
- Migrate all parsers (otakudesu, alqanime, komik) to native infra implementations
- Replace once_cell::sync::Lazy/OnceCell with std::sync::LazyLock/OnceLock
- Remove 150+ old files in modules/ and shared/ directories
- Remove once_cell from Cargo.toml dependencies
- Fix test/debug binaries to use new import paths
- Zero new clippy warnings

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-21 12:54:23 +07:00
asepharyana 80c96eaa42 chore: initial commit for asepharyana-hub-scraper 2026-07-09 22:07:03 +07:00