feat(phase7): grow corpus, MCP write-tools+auth, /metrics observability
CI / typecheck + build (turbo) (push) Canceled after 0s

- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts,
  cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17
  chunks, 5 revisions (was 4 docs).
- apps/mcp: add write-tools index_document/reindex_all/restore_revision (require
  x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads
  the HTTP header; stdio keeps writes open (trusted local).
- apps/api: GET /metrics (Prometheus text: uptime + queue job gauges).
- Caddy: expose /metrics on wiki. domain -> :4020.
Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error,
auth -> enqueues + worker drains; typecheck green.
This commit is contained in:
asepharyana
2026-08-20 10:04:16 +07:00
parent 76ed91c468
commit 53d636af0e
12 changed files with 524 additions and 4 deletions
+68
View File
@@ -0,0 +1,68 @@
---
id: bullmq-workers
title: BullMQ Background Workers
type: documentation
tags:
- bullmq
- redis
- jobs
- queue
- infra
status: published
author: asep
created_at: 2026-08-20
updated_at: 2026-08-20
---
# BullMQ Background Workers
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
shared Redis instance.
## Connection requirements
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
retry count causes the cryptic `Connection in key mode` error on blocking commands.
The shared client in `packages/queue` sets this correctly:
```ts
const opts: RedisOptions = {
maxRetriesPerRequest: null,
lazyConnect: true,
enableOfflineQueue: true,
};
```
## Job model
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
| name | data | action |
|------------|-------------------|---------------------------------|
| `index-doc`| `{ relPath, reason }` | index one content file |
| `index-all`| `{ reason }` | full corpus reindex |
| `reindex` | (legacy) | alias of full |
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
repeatable jobs, so a literal `:` in a custom jobId is rejected.
## Worker lifecycle
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
logs the reason and the job is retried per BullMQ defaults.
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
## Inspecting state
```bash
bun run enqueue --all # enqueue a full reindex
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
```
A stuck queue (waiting > 0, active = 0) means the worker died — check
`systemctl status mcpedia-worker` and the journal.
+85
View File
@@ -0,0 +1,85 @@
---
id: caddy-reverse-proxy
title: Caddy Reverse Proxy
type: documentation
tags:
- caddy
- reverse-proxy
- tls
- infra
status: published
author: asep
created_at: 2026-08-20
updated_at: 2026-08-20
---
# Caddy Reverse Proxy
Caddy is the single reverse proxy on the host. Every public service sits behind it
and terminates TLS with automatic Let's Encrypt certificates. Understanding its
config model prevents the two recurring failure modes: **525 (origin TLS)** and
**404 (path routing)**.
## Config model
The live config is `/etc/caddy/Caddyfile`. It is a manual, tuned file — CI does not
deploy it, so the live file is the source of truth and must be kept in sync with any
repo reference.
A reusable snippet handles the common case:
```
(proxy) {
encode zstd gzip
header {
-Server
X-Content-Type-Options "nosniff"
}
reverse_proxy 127.0.0.1:{args[0]} {
transport http {
keepalive 120s
dial_timeout 3s
}
}
}
```
A site block wires a domain to a backend port:
```
wiki.asepharyana.my.id {
import proxy 4016
}
```
## Path routing: `handle` vs `handle_path`
`handle /trpc/*` forwards the request **with** the `/trpc` prefix preserved.
`handle_path /trpc/*` **strips** it before proxying. Stripping is wrong when the
upstream already mounts the route at `/trpc` — the upstream then receives `/` and 404s.
Rule: when the upstream already serves the path (e.g. Hono `app.all("/trpc/*")`),
use `handle`, not `handle_path`.
## 525 — origin TLS handshake failed
Cloudflare proxies every `*.asepharyana.my.id` record. If a subdomain has **no**
Caddy site block, Caddy has no certificate for that SNI and the TLS handshake dies →
Cloudflare returns 525. Fix: add the block, `caddy validate`, `systemctl reload caddy`.
The first request after adding a block triggers ACME certificate issuance; until it
completes the origin may briefly 525. That is expected and self-heals in ~10s.
## Slow upstreams
LLM gateways (9router) have time-to-first-token of 30–40s. The default
`response_header_timeout 30s` yields false 504s. Lengthen it for those blocks:
```
reverse_proxy 127.0.0.1:4014 {
transport http {
response_header_timeout 120s
read_timeout 300s
write_timeout 300s
}
}
```
+60
View File
@@ -0,0 +1,60 @@
---
id: mcp-streamable-http
title: MCP Streamable HTTP Transport
type: documentation
tags:
- mcp
- protocol
- http
- infra
status: published
author: asep
created_at: 2026-08-20
updated_at: 2026-08-20
---
# MCP Streamable HTTP Transport
The MCPedia MCP server is served over **Streamable HTTP** (the MCP 2025-03-26
transport) so remote clients — Claude, a Discord bot, a web frontend — can call its
tools and read its resources without spawning a stdio subprocess.
## Why stateless
The server uses `StreamableHTTPServerTransport` in **stateless mode**
(`sessionIdGenerator: undefined`):
- One `McpServer` + transport is created **per request**.
- No session affinity, no shared-transport `connect()` race, no session-map memory
leak under burst traffic.
- Re-registering the 6 tools + 4 resources per request is negligible for a KB-sized
corpus.
Stateful mode (a `sessionIdGenerator` returning a UUID) would require holding a
transport map keyed by session id and cleaning it up on `onclose`. For this read-mostly
knowledge base, stateless is simpler and equally correct.
## Endpoint
```
POST https://mcp.asepharyana.my.id/mcp
Content-Type: application/json
Accept: application/json, text/event-stream
```
Responses use SSE framing (`event: message` / `data: {...}`) even for unary results.
Clients must send `Accept: application/json, text/event-stream` or the server returns
406. The MCP `initialize` handshake sets `protocolVersion: "2025-03-26"`.
## CORS
`/mcp` returns permissive CORS headers (`Access-Control-Allow-Origin: *`) so browser
clients can call it directly. Preflight `OPTIONS` is answered with 204.
## Auth for write tools
Read tools (`search_documents`, `get_document`, ...) are open. Write tools
(`index_document`, `reindex_all`, `restore_revision`) require the
`x-webhook-secret` header to match `WEBHOOK_SECRET` — the same shared secret used by
the git-sync webhook. A missing/invalid header makes the tool return an error before
any mutation.