asepharyana
53d636af0e
CI / typecheck + build (turbo) (push) Canceled after 0s
- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
69 lines
2.1 KiB
Markdown
69 lines
2.1 KiB
Markdown
---
|
|
id: bullmq-workers
|
|
title: BullMQ Background Workers
|
|
type: documentation
|
|
tags:
|
|
- bullmq
|
|
- redis
|
|
- jobs
|
|
- queue
|
|
- infra
|
|
status: published
|
|
author: asep
|
|
created_at: 2026-08-20
|
|
updated_at: 2026-08-20
|
|
---
|
|
|
|
# BullMQ Background Workers
|
|
|
|
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
|
|
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
|
|
shared Redis instance.
|
|
|
|
## Connection requirements
|
|
|
|
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
|
|
retry count causes the cryptic `Connection in key mode` error on blocking commands.
|
|
The shared client in `packages/queue` sets this correctly:
|
|
|
|
```ts
|
|
const opts: RedisOptions = {
|
|
maxRetriesPerRequest: null,
|
|
lazyConnect: true,
|
|
enableOfflineQueue: true,
|
|
};
|
|
```
|
|
|
|
## Job model
|
|
|
|
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
|
|
|
|
| name | data | action |
|
|
|------------|-------------------|---------------------------------|
|
|
| `index-doc`| `{ relPath, reason }` | index one content file |
|
|
| `index-all`| `{ reason }` | full corpus reindex |
|
|
| `reindex` | (legacy) | alias of full |
|
|
|
|
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
|
|
repeatable jobs, so a literal `:` in a custom jobId is rejected.
|
|
|
|
## Worker lifecycle
|
|
|
|
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
|
|
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
|
|
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
|
|
logs the reason and the job is retried per BullMQ defaults.
|
|
|
|
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
|
|
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
|
|
|
|
## Inspecting state
|
|
|
|
```bash
|
|
bun run enqueue --all # enqueue a full reindex
|
|
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
|
|
```
|
|
|
|
A stuck queue (waiting > 0, active = 0) means the worker died — check
|
|
`systemctl status mcpedia-worker` and the journal.
|