feat(phase7): grow corpus, MCP write-tools+auth, /metrics observability
CI / typecheck + build (turbo) (push) Canceled after 0s
CI / typecheck + build (turbo) (push) Canceled after 0s
- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
This commit is contained in:
1 parent
76ed91c468
commit
53d636af0e
12 files changed
+524
-4
No files matched your search
@@ -0,0 +1,68 @@
|
||||
---
|
||||
id: bullmq-workers
|
||||
title: BullMQ Background Workers
|
||||
type: documentation
|
||||
tags:
|
||||
- bullmq
|
||||
- redis
|
||||
- jobs
|
||||
- queue
|
||||
- infra
|
||||
status: published
|
||||
author: asep
|
||||
created_at: 2026-08-20
|
||||
updated_at: 2026-08-20
|
||||
---
|
||||
|
||||
# BullMQ Background Workers
|
||||
|
||||
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
|
||||
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
|
||||
shared Redis instance.
|
||||
|
||||
## Connection requirements
|
||||
|
||||
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
|
||||
retry count causes the cryptic `Connection in key mode` error on blocking commands.
|
||||
The shared client in `packages/queue` sets this correctly:
|
||||
|
||||
```ts
|
||||
const opts: RedisOptions = {
|
||||
maxRetriesPerRequest: null,
|
||||
lazyConnect: true,
|
||||
enableOfflineQueue: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Job model
|
||||
|
||||
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
|
||||
|
||||
| name | data | action |
|
||||
|------------|-------------------|---------------------------------|
|
||||
| `index-doc`| `{ relPath, reason }` | index one content file |
|
||||
| `index-all`| `{ reason }` | full corpus reindex |
|
||||
| `reindex` | (legacy) | alias of full |
|
||||
|
||||
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
|
||||
repeatable jobs, so a literal `:` in a custom jobId is rejected.
|
||||
|
||||
## Worker lifecycle
|
||||
|
||||
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
|
||||
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
|
||||
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
|
||||
logs the reason and the job is retried per BullMQ defaults.
|
||||
|
||||
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
|
||||
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
|
||||
|
||||
## Inspecting state
|
||||
|
||||
```bash
|
||||
bun run enqueue --all # enqueue a full reindex
|
||||
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
|
||||
```
|
||||
|
||||
A stuck queue (waiting > 0, active = 0) means the worker died — check
|
||||
`systemctl status mcpedia-worker` and the journal.
|
||||
Reference in new issue
Block a user