feat(phase7): grow corpus, MCP write-tools+auth, /metrics observability
CI / typecheck + build (turbo) (push) Canceled after 0s
CI / typecheck + build (turbo) (push) Canceled after 0s
- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
---
|
||||
id: bullmq-workers
|
||||
title: BullMQ Background Workers
|
||||
type: documentation
|
||||
tags:
|
||||
- bullmq
|
||||
- redis
|
||||
- jobs
|
||||
- queue
|
||||
- infra
|
||||
status: published
|
||||
author: asep
|
||||
created_at: 2026-08-20
|
||||
updated_at: 2026-08-20
|
||||
---
|
||||
|
||||
# BullMQ Background Workers
|
||||
|
||||
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
|
||||
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
|
||||
shared Redis instance.
|
||||
|
||||
## Connection requirements
|
||||
|
||||
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
|
||||
retry count causes the cryptic `Connection in key mode` error on blocking commands.
|
||||
The shared client in `packages/queue` sets this correctly:
|
||||
|
||||
```ts
|
||||
const opts: RedisOptions = {
|
||||
maxRetriesPerRequest: null,
|
||||
lazyConnect: true,
|
||||
enableOfflineQueue: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Job model
|
||||
|
||||
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
|
||||
|
||||
| name | data | action |
|
||||
|------------|-------------------|---------------------------------|
|
||||
| `index-doc`| `{ relPath, reason }` | index one content file |
|
||||
| `index-all`| `{ reason }` | full corpus reindex |
|
||||
| `reindex` | (legacy) | alias of full |
|
||||
|
||||
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
|
||||
repeatable jobs, so a literal `:` in a custom jobId is rejected.
|
||||
|
||||
## Worker lifecycle
|
||||
|
||||
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
|
||||
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
|
||||
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
|
||||
logs the reason and the job is retried per BullMQ defaults.
|
||||
|
||||
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
|
||||
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
|
||||
|
||||
## Inspecting state
|
||||
|
||||
```bash
|
||||
bun run enqueue --all # enqueue a full reindex
|
||||
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
|
||||
```
|
||||
|
||||
A stuck queue (waiting > 0, active = 0) means the worker died — check
|
||||
`systemctl status mcpedia-worker` and the journal.
|
||||
@@ -0,0 +1,85 @@
|
||||
---
|
||||
id: caddy-reverse-proxy
|
||||
title: Caddy Reverse Proxy
|
||||
type: documentation
|
||||
tags:
|
||||
- caddy
|
||||
- reverse-proxy
|
||||
- tls
|
||||
- infra
|
||||
status: published
|
||||
author: asep
|
||||
created_at: 2026-08-20
|
||||
updated_at: 2026-08-20
|
||||
---
|
||||
|
||||
# Caddy Reverse Proxy
|
||||
|
||||
Caddy is the single reverse proxy on the host. Every public service sits behind it
|
||||
and terminates TLS with automatic Let's Encrypt certificates. Understanding its
|
||||
config model prevents the two recurring failure modes: **525 (origin TLS)** and
|
||||
**404 (path routing)**.
|
||||
|
||||
## Config model
|
||||
|
||||
The live config is `/etc/caddy/Caddyfile`. It is a manual, tuned file — CI does not
|
||||
deploy it, so the live file is the source of truth and must be kept in sync with any
|
||||
repo reference.
|
||||
|
||||
A reusable snippet handles the common case:
|
||||
|
||||
```
|
||||
(proxy) {
|
||||
encode zstd gzip
|
||||
header {
|
||||
-Server
|
||||
X-Content-Type-Options "nosniff"
|
||||
}
|
||||
reverse_proxy 127.0.0.1:{args[0]} {
|
||||
transport http {
|
||||
keepalive 120s
|
||||
dial_timeout 3s
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A site block wires a domain to a backend port:
|
||||
|
||||
```
|
||||
wiki.asepharyana.my.id {
|
||||
import proxy 4016
|
||||
}
|
||||
```
|
||||
|
||||
## Path routing: `handle` vs `handle_path`
|
||||
|
||||
`handle /trpc/*` forwards the request **with** the `/trpc` prefix preserved.
|
||||
`handle_path /trpc/*` **strips** it before proxying. Stripping is wrong when the
|
||||
upstream already mounts the route at `/trpc` — the upstream then receives `/` and 404s.
|
||||
|
||||
Rule: when the upstream already serves the path (e.g. Hono `app.all("/trpc/*")`),
|
||||
use `handle`, not `handle_path`.
|
||||
|
||||
## 525 — origin TLS handshake failed
|
||||
|
||||
Cloudflare proxies every `*.asepharyana.my.id` record. If a subdomain has **no**
|
||||
Caddy site block, Caddy has no certificate for that SNI and the TLS handshake dies →
|
||||
Cloudflare returns 525. Fix: add the block, `caddy validate`, `systemctl reload caddy`.
|
||||
The first request after adding a block triggers ACME certificate issuance; until it
|
||||
completes the origin may briefly 525. That is expected and self-heals in ~10s.
|
||||
|
||||
## Slow upstreams
|
||||
|
||||
LLM gateways (9router) have time-to-first-token of 30–40s. The default
|
||||
`response_header_timeout 30s` yields false 504s. Lengthen it for those blocks:
|
||||
|
||||
```
|
||||
reverse_proxy 127.0.0.1:4014 {
|
||||
transport http {
|
||||
response_header_timeout 120s
|
||||
read_timeout 300s
|
||||
write_timeout 300s
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
id: mcp-streamable-http
|
||||
title: MCP Streamable HTTP Transport
|
||||
type: documentation
|
||||
tags:
|
||||
- mcp
|
||||
- protocol
|
||||
- http
|
||||
- infra
|
||||
status: published
|
||||
author: asep
|
||||
created_at: 2026-08-20
|
||||
updated_at: 2026-08-20
|
||||
---
|
||||
|
||||
# MCP Streamable HTTP Transport
|
||||
|
||||
The MCPedia MCP server is served over **Streamable HTTP** (the MCP 2025-03-26
|
||||
transport) so remote clients — Claude, a Discord bot, a web frontend — can call its
|
||||
tools and read its resources without spawning a stdio subprocess.
|
||||
|
||||
## Why stateless
|
||||
|
||||
The server uses `StreamableHTTPServerTransport` in **stateless mode**
|
||||
(`sessionIdGenerator: undefined`):
|
||||
|
||||
- One `McpServer` + transport is created **per request**.
|
||||
- No session affinity, no shared-transport `connect()` race, no session-map memory
|
||||
leak under burst traffic.
|
||||
- Re-registering the 6 tools + 4 resources per request is negligible for a KB-sized
|
||||
corpus.
|
||||
|
||||
Stateful mode (a `sessionIdGenerator` returning a UUID) would require holding a
|
||||
transport map keyed by session id and cleaning it up on `onclose`. For this read-mostly
|
||||
knowledge base, stateless is simpler and equally correct.
|
||||
|
||||
## Endpoint
|
||||
|
||||
```
|
||||
POST https://mcp.asepharyana.my.id/mcp
|
||||
Content-Type: application/json
|
||||
Accept: application/json, text/event-stream
|
||||
```
|
||||
|
||||
Responses use SSE framing (`event: message` / `data: {...}`) even for unary results.
|
||||
Clients must send `Accept: application/json, text/event-stream` or the server returns
|
||||
406. The MCP `initialize` handshake sets `protocolVersion: "2025-03-26"`.
|
||||
|
||||
## CORS
|
||||
|
||||
`/mcp` returns permissive CORS headers (`Access-Control-Allow-Origin: *`) so browser
|
||||
clients can call it directly. Preflight `OPTIONS` is answered with 204.
|
||||
|
||||
## Auth for write tools
|
||||
|
||||
Read tools (`search_documents`, `get_document`, ...) are open. Write tools
|
||||
(`index_document`, `reindex_all`, `restore_revision`) require the
|
||||
`x-webhook-secret` header to match `WEBHOOK_SECRET` — the same shared secret used by
|
||||
the git-sync webhook. A missing/invalid header makes the tool return an error before
|
||||
any mutation.
|
||||
Reference in New Issue
Block a user