asepharyana-hub-guide/skills/logging-observability/SKILL.md

---
name: logging-observability
description: Best practices for logging, metrics, tracing, alerting, and observability — structured logging, correlation IDs, Prometheus metrics, distributed tracing, and dashboard design. Use when adding logging, designing monitoring, setting up dashboards, debugging production issues, or whenever the user mentions "logging," "observability," "metrics," "tracing," "Prometheus," "Grafana," "Jaeger," "OpenTelemetry," "structured logging," or "alerting."
---

# Logging & Observability

## The Three Pillars

```
Logs — discrete events: "user logged in at 10:32:14"
Metrics — aggregatable numbers: 50 req/s, 200ms p99 latency
Traces — request lifecycle across services: "order #123 took 400ms total"
```

All three are needed. None replaces the others.

## Structured Logging

**Log in JSON format** — machine-readable, parsable, searchable.

```json
// ❌ Unstructured (text search grey area)
"User 123 created order 456 for $50.99"

// ✅ Structured (parseable, filterable)
{
  "level": "info",
  "time": "2025-07-25T10:32:14Z",
  "message": "order created",
  "service": "order-service",
  "trace_id": "abc123def456",
  "user_id": "123",
  "order_id": "456",
  "amount": 50.99,
  "currency": "USD",
  "duration_ms": 45
}
```

### Log Levels — Use Consistently

| Level | When | Example |
|-------|------|---------|
| **ERROR** | Something is broken. Needs human attention. | DB connection failed, payment declined |
| **WARN** | Something unexpected but recoverable. | Retry succeeded, rate limit approaching |
| **INFO** | Notable lifecycle events. User-triggered actions. | Order created, user signed up, scheduled job ran |
| **DEBUG** | Detailed context for debugging. Off in prod. | SQL query, request body, iteration details |
| **TRACE** | Very fine-grained. For deep debugging only. | Function entry/exit, loop iterations |

### What to Log

- **Every error** with stack trace, context, and correlation ID.
- **Every request** at INFO — method, path, status, duration.
- **Business events** — state transitions (order created → paid → shipped).
- **Auth events** — login success/failure, token refresh, privilege change.
- **Third-party calls** — external API call duration, status.

### What NOT to Log (Security)

- ❌ Passwords, tokens, secrets, API keys
- ❌ PII beyond necessity (email, phone, SSN, address)
- ❌ Full request/response bodies (except DEBUG, and even then — sanitize)
- ❌ Database connection strings
- ❌ Internal IPs in public-facing systems

## Metrics (Prometheus)

### RED Method (for services)

| Metric | What | Good |
|--------|------|------|
| **Rate** | Requests per second | Flat line — load |
| **Errors** | Failed requests / total | <1% (target), <0.1% (excellent) |
| **Duration** | Latency distribution | p50 < 100ms, p99 < 500ms |

### USE Method (for infrastructure)

| Metric | What |
|--------|------|
| **Utilization** | CPU, RAM, disk, network bandwidth |
| **Saturation** | Queue depth, swap usage, load average |
| **Errors** | Disk IO errors, packet drops, OOM kills |

### Four Golden Signals (Google SRE)

1. **Latency** — time to serve a request
2. **Traffic** — demand on the system (RPS, active users)
3. **Errors** — explicit failures + implicit (200 with wrong data)
4. **Saturation** — how "full" the system is

### Key Metrics to Expose

```prometheus
# Service
http_requests_total{method, path, status}
http_request_duration_seconds{quantile="0.99"}
errors_total{type}
active_users

# Business
orders_created_total{currency}
revenue_total
jobs_failed_total

# System
process_cpu_seconds_total
process_resident_memory_bytes
process_open_fds
```

## Distributed Tracing (Jaeger / OpenTelemetry)

- **Every request gets a `trace_id`** — propagate through headers (`x-trace-id`, `traceparent`).
- **Span per operation** — HTTP handler, DB query, external API call.
- **Annotate spans** — with relevant metadata (user ID, order ID, error details).
- **Sampling:** sample 100% of errors, 1-10% of successful requests.
- **Dapr/Jaeger:** auto-injects trace headers across sidecars.

```typescript
// OpenTelemetry example
const tracer = opentelemetry.trace.getTracer('order-service');
const span = tracer.startSpan('createOrder', { attributes: { orderId } });
try {
  const result = await createOrder(input);
  span.setStatus({ code: SpanStatusCode.OK });
  return result;
} catch (e) {
  span.recordException(e);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw e;
} finally {
  span.end();
}
```

## Alerting

### Alert Design

- **Alert on symptoms, not causes** — "API error rate >1%" not "server CPU >90%"
- **Alert fatigue kills alerts** — every alert should need human action. If nobody acts, delete it.
- **Define SLOs** — 99.9% uptime, 95% requests <500ms. Alert when approaching breach.
- **Alert structure:**
  ```
  Title: [P1] High error rate on order-service
  Summary: Error rate = 5.2% (threshold: 1%) over last 5 minutes
  Runbook: /runbooks/high-error-rate.md
  Severity: P1 (critical, paging) / P2 (urgent, business hours) / P3 (warning, ticket)
  ```

### Common Alert Rules

| Rule | Severity | Threshold |
|------|----------|-----------|
| High error rate | P1 | >1% over 5min |
| High latency | P2 | p99 >1s over 5min |
| Service down | P1 | No metrics for 5min |
| Disk space | P2 | <10% free |
| Certificate expiring | P2 | <30 days |
| Rate limit hit rate | P3 | >10% of requests rate-limited |

## Observability Stack (for this repo)

| Component | Role |
|-----------|------|
| **Prometheus** | Metrics collection + alerting |
| **Jaeger** | Distributed tracing (all-in-one, OTLP receiver) |
| **Traefik** | Metrics endpoint (`--metrics.prometheus=true`) |
| **Docker labels** | Auto-discovery: `prometheus.io/scrape=true` |
| **Dashboard** | `/dashboard` — auto-refresh 15s, shows containers + traces + metrics |

## Anti-patterns

- ❌ **String interpolation in logs** — `log.info("User " + id + " logged in")` instead of structured fields
- ❌ **Logging in every function** — no, log at service boundaries and business events
- ❌ **No correlation IDs** — cannot trace request across services
- ❌ **Silent catch** — `catch (e) {}` swallows errors with no log
- ❌ **Alert fatigue** — 50 alerts daily means none are actionable
- ❌ **Dashboards without action** — beautiful dashboard but nobody knows what to do when a metric goes red