scraper/.hermes/plans/scraper-mytheclipse-round2-spec.md

Spec Round 2 — Scraper → mytheclipse Deep Infra Migration (thread/async/queue)

Date: 2026-08-30 Repo: /home/code/scraper (scraper-service) Library: /home/code/mytheclipse (mytheclipse crates v1.20+ local path)

Scope

Round 1 (committed c7ffd29) migrated retry, redis cache, event bus, ratelimit, config. Round 2 targets the remaining runtime/concurrency/queue/observability infrastructure — the parts the user explicitly asked to push "sampai tingkat thread, async, queue dan lainnya".

1. Runtime thread sizing — mytheclipse::runtime_auto::RuntimeConfig

Current: bootstrap logs std::thread::available_parallelism() manually; tokio runtime is #[tokio::main(flavor = "multi_thread")] with default worker counts.

Change:

• Add mytheclipse = { features = ["lifecycle", ...] } (add lifecycle feature).
• In bootstrap/mod.rs, compute RuntimeConfig::auto() once at build, log worker/blocking/compute/io counts, and keep the tokio runtime default (already multi_thread). Optionally pass worker_threads via a #[tokio::main] alternative — but since main uses the macro, we keep the macro form and simply surface the auto-derived counts in logs + store them in AppState for future pool sizing.
• Replace the manual available_parallelism() log with RuntimeConfig::auto().

2. Async I/O + background task spawns — mytheclipse::spawn_io / spawn_bg

Current:

• proxy_fetch.rs:114 uses tokio::spawn(...) for the request-coalescing leader
• 30+ tokio::task::spawn_blocking(...) sites across otakudesu.rs, anime2 use_cases, komik use_cases, proxy_fetch (gzip decompress)
• No bounded background task pool

Change:

• proxy_fetch.rs leader task → mytheclipse::spawn_io(...) (same semantics, adds tracing span instrumentation). No behavior change.
• Add mytheclipse = { features = ["io", "bg"] }.
• The spawn_blocking sites stay as-is this round (they're CPU-bound parser calls and spawn_blocking is the correct tokio primitive; mytheclipse compute() would replace them but that's a 2k+ LOC parser sweep — deferred as before). Document this in the spec's residual section.
• Add bg feature and expose a bounded background executor via mytheclipse::spawn_bg for the cache-write / relay-fallback background tasks that currently fire-and-forget (if any are found in use_cases).

3. Worker queue — mytheclipse-queue InMemoryQueue + WorkerPool

Current: no queue abstraction; the only "background" work is the cache batch writes (cache_image_urls_batch_lazy in the proxy image-cache path — but proxy routes were removed, so that path may be dead). Event bus has no worker pool.

Change:

• Add mytheclipse-queue = { path = ..., features = ["in-memory"] }.
• Wire a small application-level JobQueue in src/infrastructure/queue/:
  • InMemoryQueue + WorkerPool consuming a scrape:repair topic for background image-cache repair/re-sync jobs.
  • Graceful shutdown: on app shutdown, stop workers.
  • This is additive infra (no existing behavior replaced), so it's low-risk and demonstrates the queue crate in the scraper.
• Actually use it: the image_cache repair path (cache cleanup at 2 AM) is currently absent (scheduler dir missing). Introduce a small src/infrastructure/scheduler.rs using mytheclipse::cron (lifecycle feature) OR tokio::time::interval for the daily 02:00 cleanup that enqueues repair jobs onto the queue. Given the original CLAUDE.md mentions "scheduler — Cron jobs (daily cache cleanup at 2 AM)", recreating it on mytheclipse-queue is a faithful, additive migration.

4. Backpressure / concurrency — mytheclipse::ConcurrencyLimiter / BackpressureQueue

Current: proxy_fetch has request coalescing via DashMap + broadcast (kept — it's application logic, not infra). No app-level backpressure.

Change:

• Add mytheclipse = { features = ["traffic"] } (already enabled for RateLimiter in round 1).
• Wrap the fetch-with-proxy path with a ConcurrencyLimiter (bound concurrent outbound scrapes, default e.g. 16) so burst traffic can't saturate the upstream. This is a real, central hot path (every scrape goes through it).
• Keep the RateLimiter middleware (round 1) for inbound.

5. Observability — mytheclipse-tracing TracingLayer (+ keep OTel metrics)

Current: tracing_subscriber::fmt().with_env_filter(...) in bootstrap; opentelemetry metrics in observability/metrics.rs (kept — mytheclipse-tracing has no metrics exporter, round-1 decision).

Change:

• Add mytheclipse-tracing = { path = ..., features = ["env"] }.
• Replace tracing_subscriber::fmt() in bootstrap with mytheclipse_tracing::TracingLayer::install() (same RUST_LOG env-filter semantics, adds tracing span layer compatibility).
• Keep init_otel_metrics() untouched (OTel exporter stays).

6. HTTP client — mytheclipse-http HttpClient (optional, re-evaluate)

Round 1 deferred this because mytheclipse-http's client "loses header/UA/status control". Re-check: mytheclipse-http::client::HttpClient — if it exposes with_timeout, get_text, get_json, post_json, and header injection, use it for the non-critical relay fetches (fetch_via_relays). If not header-capable, skip (kept as residual).

Files touched

• Cargo.toml (features: add lifecycle, io, bg; add mytheclipse-queue, mytheclipse-tracing)
• src/bootstrap/mod.rs (RuntimeConfig log + TracingLayer + scheduler start)
• src/infrastructure/scraping/proxy_fetch.rs (spawn_io + ConcurrencyLimiter)
• src/infrastructure/queue/mod.rs (NEW — InMemoryQueue + WorkerPool + repair topic)
• src/infrastructure/scheduler.rs (NEW — daily 02:00 cleanup → queue enqueue)
• src/infrastructure/cache/mytheclipse.rs (maybe wire limiter through cache path)
• src/presentation/state.rs (add queue handle / limiter if needed)

Verification

1. cargo check — green after each batch
2. cargo test — all pass
3. cargo build --release — green
4. cargo fmt --check on touched files
5. Runtime smoke: boot server, hit /health, confirm logs show RuntimeConfig and scheduler started; queue jobs process (test harness)

Residual (deliberately deferred)

• spawn_blocking → mytheclipse::compute() sweep (2k+ LOC parser files)
• OTel metrics → mytheclipse-tracing (no metrics exporter in library yet)
• rayon in komik_parser
• mytheclipse-http client if header control is insufficient

Commit

Single commit at end: refactor: migrate runtime/queue/concurrency infra to mytheclipse (round 2)