feat: add panic tracking and logging with PanicTracker
- Implemented PanicTracker to log panics using tracing within a dedicated span. - Introduced PanicInfo struct to capture panic details. - Added tests for panic catching and hook restoration. feat: implement token-bucket rate limiter - Created RateLimiter to manage token consumption with configurable rates and burst capacity. - Added methods for acquiring tokens and checking available tokens. - Included tests for rate limiting behavior and token availability. feat: add automatic retry with exponential backoff - Developed retry function to handle transient errors with configurable retry strategies. - Introduced RetryConfig for customizing retry behavior and jitter. - Added tests for various retry scenarios and error handling. feat: implement graceful shutdown coordination - Created ShutdownManager to manage shutdown signals and task completion. - Introduced ShutdownSignal for tasks to observe shutdown requests. - Added tests for shutdown request handling and task draining. feat: add timeout functionality for async operations - Implemented with_timeout function to enforce execution time limits on futures. - Created Timeout struct for wrapping futures with a deadline. - Added tests for timeout behavior and error mapping.
This commit is contained in:
asepharyana committed
1 parent
ca39dea5db
commit
d119821339
76 files changed
+10360
-191
No files matched your search
@@ -1,163 +1,64 @@
|
||||
# mytheclipse
|
||||
|
||||
[](https://crates.io/crates/mytheclipse)
|
||||
[](https://docs.rs/mytheclipse)
|
||||
[](LICENSE-MIT)
|
||||
A personal collection of Rust abstractions for building resource-aware,
|
||||
resilient, and well-instrumented applications without hand-rolling the same
|
||||
plumbing every time — organized as a Cargo workspace, one focused crate per
|
||||
concern.
|
||||
|
||||
Resource-aware execution primitives and reliability abstractions for Rust: async I/O, heavy compute, background queue management, resiliency, traffic control, lifecycle management, and observability. Sized automatically from the host's logical core count and exposed through a single, lazily-initialized engine context alongside self-contained, constructible utilities.
|
||||
[](crates/mytheclipse/LICENSE-MIT)
|
||||
|
||||
## Resource Sizing
|
||||
## Crates
|
||||
|
||||
Given $N$ logical cores (via `num_cpus::get()`):
|
||||
| Crate | Description | Docs |
|
||||
| :--- | :--- | :--- |
|
||||
| [`mytheclipse`](crates/mytheclipse) | Resource-aware execution primitives (async I/O, compute, background queues), resiliency (retry, circuit breaker, timeout), traffic control (rate limiter, backpressure, concurrency limiter), lifecycle (graceful shutdown, cron), and observability (metrics, panic tracking). | [README](crates/mytheclipse/README.md) |
|
||||
| [`corex-cache`](crates/corex-cache) | Unified multi-layer (L1/L2) cache abstraction: in-memory or Moka L1, Redis/Valkey L2, cache-aside read-through. | [README](crates/corex-cache/README.md) |
|
||||
| [`corex-storage`](crates/corex-storage) | Unified storage & file system abstraction: one driver interface over local disk, S3/MinIO, and Google Cloud Storage, stream-based. | [README](crates/corex-storage/README.md) |
|
||||
| [`corex-event`](crates/corex-event) | Unified events & message bus abstraction: in-memory pub/sub dispatcher plus RabbitMQ and NATS broker adapters behind one trait. | [README](crates/corex-event/README.md) |
|
||||
| [`corex-config`](crates/corex-config) | Type-safe, dynamic configuration engine: load `.env`/YAML/JSON/TOML into typed structs, with hot-reload. | [README](crates/corex-config/README.md) |
|
||||
| [`corex-crypto`](crates/corex-crypto) | Safe hashing (Argon2id), encryption (AES-256-GCM), and JWT tokens, with key rotation support. | [README](crates/corex-crypto/README.md) |
|
||||
|
||||
| Subsystem | Sizing Formula | Default on 8 cores | Backing Primitive |
|
||||
| :--- | :--- | :--- | :--- |
|
||||
| **Async I/O** | $N$ | 8 | Ambient `tokio::spawn` + `tracing` span |
|
||||
| **Compute** | $\max(1, N - 1)$ | 7 | Sized `rayon::ThreadPool` + `catch_unwind` |
|
||||
| **Background Queue** | $\max(2, \lfloor N / 2 \rfloor)$ | 4 | `tokio::sync::Semaphore` + `tokio::spawn` |
|
||||
Every crate follows the same philosophy: **one small interface, pluggable
|
||||
backends behind feature flags, and a working default that needs no external
|
||||
service to build or test.** Distributed backends (Redis, S3, GCS, RabbitMQ,
|
||||
NATS) are feature-gated and their integration tests are `#[ignore]`d unless
|
||||
the corresponding environment variables point at a live service.
|
||||
|
||||
The three execution primitives (`io`, `compute`, `bg`) are sized from the host's CPU core count through `mytheclipse::context()`. Resiliency, traffic control, lifecycle, and observability utilities are constructible per-instance without global state.
|
||||
## Getting started
|
||||
|
||||
## Features
|
||||
|
||||
- **`io`**: enables `mytheclipse::spawn_io`, instrumented async task spawning.
|
||||
- **`compute`**: enables `mytheclipse::compute`, panic-isolated execution on a sized Rayon pool.
|
||||
- **`bg`**: enables `mytheclipse::spawn_bg`, semaphore-bounded background tasks.
|
||||
- **`resiliency`**: fault tolerance abstractions:
|
||||
- `retry` — auto-retry with exponential backoff and full/equal jitter.
|
||||
- `CircuitBreaker` — failure threshold and cooldown state machine (Closed/Open/HalfOpen).
|
||||
- `with_timeout` / `timeout` / `Timeout` — hard execution deadlines.
|
||||
- **`traffic`**: traffic & load control:
|
||||
- `RateLimiter` — lazy token-bucket rate limiter with burst capacity.
|
||||
- `BackpressureQueue` — bounded queue with `DropOldest`, `Reject`, and `Block` overflow policies.
|
||||
- `ConcurrencyLimiter` — synchronous RAII semaphore wrapper for capping concurrent operations.
|
||||
- **`lifecycle`**: system lifecycle coordination:
|
||||
- `ShutdownManager` / `ShutdownSignal` — OS signal catching (SIGINT/SIGTERM/Ctrl-C) and graceful task draining.
|
||||
- `CronSchedule` / `schedule` — self-contained 5-field cron parser and async timer scheduler.
|
||||
- **`observability`**: runtime visibility:
|
||||
- `MetricsCollector` — thread-safe statistics collector with Prometheus text exposition format export.
|
||||
- `PanicTracker` — non-fatal panic logging with tracing context and boundary isolation.
|
||||
- **`full`**: enables all subsystems: `io`, `compute`, `bg`, `resiliency`, `traffic`, `lifecycle`, `observability`.
|
||||
|
||||
Zero features enabled by default (`default = []`), so you only pull in the dependencies your application actually uses.
|
||||
|
||||
## Quick Start
|
||||
|
||||
Add to your `Cargo.toml`:
|
||||
Each crate is published independently; add the ones you need:
|
||||
|
||||
```toml
|
||||
[dependencies]
|
||||
mytheclipse = { version = "0.2", features = ["full"] }
|
||||
mytheclipse = { version = "1", features = ["full"] }
|
||||
corex-cache = "0.1"
|
||||
corex-storage = { version = "0.1", features = ["s3"] }
|
||||
corex-event = { version = "0.1", features = ["nats"] }
|
||||
corex-config = "0.1"
|
||||
corex-crypto = "0.1"
|
||||
```
|
||||
|
||||
### 1. Execution Primitives
|
||||
See each crate's own README (linked above) for usage examples and the full
|
||||
feature-flag list.
|
||||
|
||||
```rust
|
||||
#[tokio::main]
|
||||
async fn main() {
|
||||
let ctx = mytheclipse::init();
|
||||
## Development
|
||||
|
||||
// Async I/O (instrumented with tracing)
|
||||
let io = mytheclipse::spawn_io(async { 42 });
|
||||
|
||||
// Heavy Compute (isolated from worker panics)
|
||||
let sum = mytheclipse::compute(|| (1..=1_000_000u64).sum::<u64>())?;
|
||||
|
||||
// Background Queue (concurrency-bounded)
|
||||
let bg = mytheclipse::spawn_bg(async { /* task */ }).await;
|
||||
|
||||
let _ = (io.await, bg.await);
|
||||
}
|
||||
```
|
||||
|
||||
### 2. Resiliency & Fault Tolerance
|
||||
|
||||
```rust
|
||||
use std::time::Duration;
|
||||
|
||||
// Auto-Retry with Exponential Backoff + Jitter
|
||||
let result = mytheclipse::retry(
|
||||
mytheclipse::RetryConfig::default(),
|
||||
|err| err.is_transient(),
|
||||
|| async { make_network_request().await },
|
||||
).await?;
|
||||
|
||||
// Circuit Breaker
|
||||
let breaker = mytheclipse::CircuitBreaker::new(mytheclipse::CircuitBreakerConfig::default());
|
||||
let value = breaker.call(|| fetch_remote_resource())?;
|
||||
|
||||
// Timeout & Deadlines
|
||||
let value = mytheclipse::with_timeout(Duration::from_secs(5), async {
|
||||
long_running_task().await
|
||||
}).await?;
|
||||
```
|
||||
|
||||
### 3. Traffic & Resource Control
|
||||
|
||||
```rust
|
||||
// Rate Limiter (Token Bucket)
|
||||
let limiter = mytheclipse::RateLimiter::new(100.0, 10);
|
||||
limiter.acquire().await?;
|
||||
|
||||
// Concurrency Limiter (Sync Semaphore)
|
||||
let limiter = mytheclipse::ConcurrencyLimiter::new(10);
|
||||
let _permit = limiter.acquire(); // released on drop
|
||||
|
||||
// Backpressure Queue (Graceful Degradation)
|
||||
let queue = mytheclipse::BackpressureQueue::new(100, mytheclipse::OverflowPolicy::DropOldest);
|
||||
queue.push(job).await?;
|
||||
let next_job = queue.pop().await;
|
||||
```
|
||||
|
||||
### 4. Lifecycle & State Management
|
||||
|
||||
```rust
|
||||
// Graceful Shutdown Manager
|
||||
let shutdown = mytheclipse::ShutdownManager::new();
|
||||
let sig = shutdown.handle();
|
||||
tokio::spawn(async move {
|
||||
let mut sig = sig;
|
||||
tokio::select! {
|
||||
_ = sig.wait() => { /* clean up */ }
|
||||
_ = worker_loop() => {}
|
||||
}
|
||||
});
|
||||
shutdown.drain(Duration::from_secs(10)).await;
|
||||
|
||||
// Cron Periodic Job Scheduler (Self-contained, no external crates)
|
||||
let cron = mytheclipse::CronSchedule::parse("0 1 * * *")?; // 1 AM daily
|
||||
let job = mytheclipse::schedule("0 1 * * *", || async {
|
||||
clean_cache().await;
|
||||
})?;
|
||||
```
|
||||
|
||||
### 5. Telemetry & Observability
|
||||
|
||||
```rust
|
||||
// Centralized Metrics Collector (Prometheus text exposition format)
|
||||
let metrics = mytheclipse::MetricsCollector::new();
|
||||
metrics.record_task(Duration::from_millis(15));
|
||||
metrics.inc_counter("http_requests_total", 1);
|
||||
let prometheus_output = metrics.export_prometheus();
|
||||
|
||||
// Panic Isolation Tracker
|
||||
let guard = mytheclipse::PanicTracker::install(); // logs panics with tracing span
|
||||
let result = mytheclipse::PanicTracker::catch(|| {
|
||||
risky_operation()
|
||||
});
|
||||
```
|
||||
|
||||
## Running the Example
|
||||
This is a Cargo workspace; run commands from the repository root:
|
||||
|
||||
```bash
|
||||
cargo run --example main --features full
|
||||
cargo build --workspace --all-features
|
||||
cargo test --workspace --all-features
|
||||
cargo clippy --workspace --all-features -- -D warnings
|
||||
cargo fmt --all --check
|
||||
```
|
||||
|
||||
Or target a single crate with `-p <name>`, e.g. `cargo test -p corex-cache`.
|
||||
|
||||
## License
|
||||
|
||||
Licensed under either of:
|
||||
|
||||
- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
|
||||
- MIT license ([LICENSE-MIT](LICENSE-MIT) or <http://opensource.org/licenses/MIT>)
|
||||
- Apache License, Version 2.0 ([LICENSE-APACHE](crates/mytheclipse/LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
|
||||
- MIT license ([LICENSE-MIT](crates/mytheclipse/LICENSE-MIT) or <http://opensource.org/licenses/MIT>)
|
||||
|
||||
at your option.
|
||||
Reference in new issue
Block a user