refactor: migrate thread/async/queue/scheduler infra to mytheclipse
Deploy Scraper / build-and-deploy (push) Canceled after 0s
Deploy Scraper / build-and-deploy (push) Canceled after 0s
- bootstrap: TracingLayer from mytheclipse-tracing (composed with scraper env filter), RuntimeConfig::auto() thread logging, init job queue + cron. - proxy_fetch: leader task via ::mytheclipse::spawn_io, gzip decompression offloaded to ::mytheclipse::compute (sized rayon pool, panic-isolated), bounded by new async fetch limiter (tokio Semaphore bridge). - queue: new infrastructure/queue module over mytheclipse-queue (InMemoryQueue + WorkerPool + BackpressureEnforcer) for repair jobs. - scheduler: new infrastructure/scheduler.rs using mytheclipse::cron for the daily 02:00 UTC image-cache cleanup. - deps: add mytheclipse-queue + mytheclipse-tracing path deps; mytheclipse -> full feature; rayon 1.12; keep path deps for unpublished crates. - tests: infra_round2 runtime smoke tests (compute panic isolation, spawn_io, backpressure admission, cron parse, queue roundtrip). - Fix: local cache::mytheclipse bridge module shadowed the mytheclipse crate name; use leading :: at spawn_io/compute call sites.
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
//! Scheduled maintenance jobs — driven by `mytheclipse::cron`.
|
||||
//!
|
||||
//! The scraper runs a daily 02:00 UTC cache-cleanup job that enqueues
|
||||
//! image-cache repair work onto the application job queue. The cron driver
|
||||
//! comes from the mytheclipse core crate (`schedule` + `CronSchedule`); the
|
||||
//! actual per-key repair work is queued through `mytheclipse-queue`.
|
||||
|
||||
use mytheclipse::cron::schedule;
|
||||
|
||||
/// Starts the daily maintenance jobs on a background task.
|
||||
///
|
||||
/// Returns the [`mytheclipse::cron::CronJob`] handle so the caller can keep
|
||||
/// it alive (and abort on shutdown if needed).
|
||||
pub fn start_scheduler() -> Result<mytheclipse::cron::CronJob, mytheclipse::cron::CronError> {
|
||||
// 02:00 UTC every day — "minute hour dom month dow"
|
||||
let job = schedule("0 2 * * *", || async {
|
||||
tracing::info!("[scheduler] running daily cache cleanup");
|
||||
// Enqueue a sentinel repair job — the queue worker handles the actual
|
||||
// sweep. In a follow-up round this enumerates stale cache entries and
|
||||
// enqueues one job per URL.
|
||||
let _ =
|
||||
crate::infrastructure::queue::enqueue_cache_repair("__daily_sweep__".to_string()).await;
|
||||
tracing::info!("[scheduler] daily cache cleanup enqueued");
|
||||
})?;
|
||||
tracing::info!("[scheduler] daily cache cleanup scheduled at 02:00 UTC");
|
||||
Ok(job)
|
||||
}
|
||||
Reference in New Issue
Block a user