asepharyana-hub-guide/skills/logging-observability/SKILL.md

name, description

name	description
logging-observability	Best practices for logging, metrics, tracing, alerting, and observability — structured logging, correlation IDs, Prometheus metrics, distributed tracing, and dashboard design. Use when adding logging, designing monitoring, setting up dashboards, debugging production issues. Detects from code context and project files — not dependent on specific language keywords."

Logging & Observability

The Three Pillars

Logs — discrete events: "user logged in at 10:32:14"
Metrics — aggregatable numbers: 50 req/s, 200ms p99 latency
Traces — request lifecycle across services: "order #123 took 400ms total"

All three are needed. None replaces the others.

Structured Logging

Log in JSON format — machine-readable, parsable, searchable.

// ❌ Unstructured (text search grey area)
"User 123 created order 456 for $50.99"

// ✅ Structured (parseable, filterable)
{
  "level": "info",
  "time": "2025-07-25T10:32:14Z",
  "message": "order created",
  "service": "order-service",
  "trace_id": "abc123def456",
  "user_id": "123",
  "order_id": "456",
  "amount": 50.99,
  "currency": "USD",
  "duration_ms": 45
}

Log Levels — Use Consistently

Level	When	Example
ERROR	Something is broken. Needs human attention.	DB connection failed, payment declined
WARN	Something unexpected but recoverable.	Retry succeeded, rate limit approaching
INFO	Notable lifecycle events. User-triggered actions.	Order created, user signed up, scheduled job ran
DEBUG	Detailed context for debugging. Off in prod.	SQL query, request body, iteration details
TRACE	Very fine-grained. For deep debugging only.	Function entry/exit, loop iterations

What to Log

• Every error with stack trace, context, and correlation ID.
• Every request at INFO — method, path, status, duration.
• Business events — state transitions (order created → paid → shipped).
• Auth events — login success/failure, token refresh, privilege change.
• Third-party calls — external API call duration, status.

What NOT to Log (Security)

• ❌ Passwords, tokens, secrets, API keys
• ❌ PII beyond necessity (email, phone, SSN, address)
• ❌ Full request/response bodies (except DEBUG, and even then — sanitize)
• ❌ Database connection strings
• ❌ Internal IPs in public-facing systems

Metrics (Prometheus)

RED Method (for services)

Metric	What	Good
Rate	Requests per second	Flat line — load
Errors	Failed requests / total	<1% (target), <0.1% (excellent)
Duration	Latency distribution	p50 < 100ms, p99 < 500ms

USE Method (for infrastructure)

Metric	What
Utilization	CPU, RAM, disk, network bandwidth
Saturation	Queue depth, swap usage, load average
Errors	Disk IO errors, packet drops, OOM kills

Four Golden Signals (Google SRE)

1. Latency — time to serve a request
2. Traffic — demand on the system (RPS, active users)
3. Errors — explicit failures + implicit (200 with wrong data)
4. Saturation — how "full" the system is

Key Metrics to Expose

# Service
http_requests_total{method, path, status}
http_request_duration_seconds{quantile="0.99"}
errors_total{type}
active_users

# Business
orders_created_total{currency}
revenue_total
jobs_failed_total

# System
process_cpu_seconds_total
process_resident_memory_bytes
process_open_fds

Distributed Tracing (Jaeger / OpenTelemetry)

• Every request gets a trace_id — propagate through headers (x-trace-id, traceparent).
• Span per operation — HTTP handler, DB query, external API call.
• Annotate spans — with relevant metadata (user ID, order ID, error details).
• Sampling: sample 100% of errors, 1-10% of successful requests.
• Dapr/Jaeger: auto-injects trace headers across sidecars.

// OpenTelemetry example
const tracer = opentelemetry.trace.getTracer('order-service');
const span = tracer.startSpan('createOrder', { attributes: { orderId } });
try {
  const result = await createOrder(input);
  span.setStatus({ code: SpanStatusCode.OK });
  return result;
} catch (e) {
  span.recordException(e);
  span.setStatus({ code: SpanStatusCode.ERROR });
  throw e;
} finally {
  span.end();
}

Alerting

Alert Design

• Alert on symptoms, not causes — "API error rate >1%" not "server CPU >90%"
• Alert fatigue kills alerts — every alert should need human action. If nobody acts, delete it.
• Define SLOs — 99.9% uptime, 95% requests <500ms. Alert when approaching breach.
• Alert structure:

  Title: [P1] High error rate on order-service
  Summary: Error rate = 5.2% (threshold: 1%) over last 5 minutes
  Runbook: /runbooks/high-error-rate.md
  Severity: P1 (critical, paging) / P2 (urgent, business hours) / P3 (warning, ticket)
  

Common Alert Rules

Rule	Severity	Threshold
High error rate	P1	>1% over 5min
High latency	P2	p99 >1s over 5min
Service down	P1	No metrics for 5min
Disk space	P2	<10% free
Certificate expiring	P2	<30 days
Rate limit hit rate	P3	>10% of requests rate-limited

Observability Stack (for this repo)

Component	Role
Prometheus	Metrics collection + alerting
Jaeger	Distributed tracing (all-in-one, OTLP receiver)
Traefik	Metrics endpoint (--metrics.prometheus=true)
Docker labels	Auto-discovery: prometheus.io/scrape=true
Dashboard	/dashboard — auto-refresh 15s, shows containers + traces + metrics

Anti-patterns

• ❌ String interpolation in logs — log.info("User " + id + " logged in") instead of structured fields
• ❌ Logging in every function — no, log at service boundaries and business events
• ❌ No correlation IDs — cannot trace request across services
• ❌ Silent catch — catch (e) {} swallows errors with no log
• ❌ Alert fatigue — 50 alerts daily means none are actionable
• ❌ Dashboards without action — beautiful dashboard but nobody knows what to do when a metric goes red