asepharyana
53d636af0e
- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
2.1 KiB
id, title, type, tags, status, author, created_at, updated_at
| id | title | type | tags | status | author | created_at | updated_at | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bullmq-workers | BullMQ Background Workers | documentation |
|
published | asep | 2026-08-20 | 2026-08-20 |
BullMQ Background Workers
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the shared Redis instance.
Connection requirements
BullMQ requires an ioredis connection with maxRetriesPerRequest: null. A finite
retry count causes the cryptic Connection in key mode error on blocking commands.
The shared client in packages/queue sets this correctly:
const opts: RedisOptions = {
maxRetriesPerRequest: null,
lazyConnect: true,
enableOfflineQueue: true,
};
Job model
Three job types flow through the mcpedia-index queue (prefix mcpedia: on Redis):
| name | data | action |
|---|---|---|
index-doc |
{ relPath, reason } |
index one content file |
index-all |
{ reason } |
full corpus reindex |
reindex |
(legacy) | alias of full |
Job IDs use a __ separator (doc__<slug>, full__<ts>) — BullMQ reserves : for
repeatable jobs, so a literal : in a custom jobId is rejected.
Worker lifecycle
The worker is a Worker with concurrency: 4. Each job calls the shared
indexContentFile / runFullIndex entry points in @mcpedia/core — the same code
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
logs the reason and the job is retried per BullMQ defaults.
Graceful shutdown: worker.close() on SIGINT/SIGTERM. systemd sends SIGTERM on
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
Inspecting state
bun run enqueue --all # enqueue a full reindex
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
A stuck queue (waiting > 0, active = 0) means the worker died — check
systemctl status mcpedia-worker and the journal.