feat(phase7): grow corpus, MCP write-tools+auth, /metrics observability

- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
2026-08-20 10:04:16 +07:00
parent 76ed91c468
commit 53d636af0e
12 changed files with 524 additions and 4 deletions
@@ -0,0 +1,41 @@
+# MCPedia — "lanjut semua" workstream
+
+Three real gaps remain (from review): tiny corpus (4 docs), MCP read-only (no write/auth),
+no observability. This plan closes all three.
+
+## 1. Content corpus (grow the KB)
+Author real, useful docs so search/semantic/revisions have something to operate on.
+Frontmatter schema (from packages/parser): id,title,type,tags,status,author,created_at,updated_at.
+Sections: docs|writeups|research|notes. Files under content/<section>/...
+New docs to add:
+- content/docs/caddy/reverse-proxy.md        (ops reference, tags: caddy, reverse-proxy, tls)
+- content/docs/bullmq/workers.md             (queue/worker reference, tags: bullmq, redis, jobs)
+- content/docs/mcp/streamable-http.md        (MCP transport reference, tags: mcp, protocol, http)
+- content/notes/postgres/full-text-search.md (PG FTS notes, tags: postgres, fts, tsvector)
+- content/writeups/infra/cloudflare-525.md   (debugging writeup, tags: cloudflare, tls, 525)
+After adding: `bun run index` to reindex (writes revisions + chunks), verify counts.
+
+## 2. MCP write-tools + auth
+Add mutating + admin tools to the MCP server (currently read-only):
+- `index_document(slug)`  -> enqueueIndexDoc (requires MCP auth header)
+- `reindex_all()`         -> enqueueFullIndex (requires MCP auth header)
+- `queue_status()`        -> getQueue counts (read, public)
+- `restore_revision(id)`  -> restoreRevision (requires MCP auth header)
+Auth: MCP client must send header `x-webhook-secret` (reuse WEBHOOK_SECRET). StreamableHTTP
+transport: read the Authorization/header in http.ts, pass to server via a factory closure
+capturing the request; tools check it. Stateless per-request server already created fresh,
+so threading the header is clean. Guard write-tools with the same requireWriteAuth logic.
+Verify: unauthenticated call to index_document -> error; authenticated -> enqueues job.
+
+## 3. Observability
+- `GET /metrics` on the API (Prometheus text format): queue counts (waiting/active/
+  completed/failed/delayed), uptime, service name. Public (safe to expose).
+- Caddy: expose /metrics on wiki. domain -> :4020 (add to handle list).
+- tRPC `queueStatus` already exists; /metrics reuses getQueue.
+Verify: curl /metrics -> text exposition with mcpedia_queue_* gauges.
+
+## Verification
+- typecheck green (turbo run typecheck).
+- MCP smoke extended: tools/list shows new tools; authenticated index_document enqueues.
+- /metrics returns 200 text; queue drains.
+- commit + push.
@@ -154,6 +154,30 @@ bun run api              # Hono+tRPC API on :4020 (added /hooks/* webhooks)
 - GitHub push webhook delivers 200; `queueStatus` shows completed:N, failed:0.
 - `turbo run typecheck` green across all 4 apps.

+## Phase 7 — Corpus, MCP write-tools + auth, observability ✅ DONE
+
+> Closed the remaining review gaps: tiny corpus (4 docs), MCP read-only (no write/auth),
+> no observability.
+
+- [x] **Content corpus grown** — added 5 real docs (Caddy reverse proxy, BullMQ workers,
+  MCP Streamable HTTP, Postgres FTS, Cloudflare-525 debugging writeup) across
+  docs/writeups/notes. `bun run index` reindexed: **9 documents, 17 chunks, 5 new
+  revisions** (was 4 docs). Search/semantic/revisions now operate on a real corpus.
+- [x] **MCP write-tools + auth** — added `index_document`, `reindex_all`,
+  `restore_revision` (write, require `x-webhook-secret`) and `queue_status` (public).
+  `createMcpServer(authSecret?)` threads the request header; stdio keeps write tools
+  open (trusted local). Verified: unauthenticated `index_document` → `isError` +
+  "unauthorized"; authenticated → enqueues job, worker drains.
+- [x] **Observability** — `GET /metrics` on the API (Prometheus text exposition:
+  `mcpedia_uptime_seconds`, `mcpedia_queue_jobs{state=...}`). Exposed on the domain at
+  `https://wiki.asepharyana.my.id/metrics`. Public, safe to scrape.
+
+### Verification done (real, against live services)
+- `https://wiki.asepharyana.my.id/metrics` → 200 Prometheus text (uptime + queue gauges).
+- `tools/list` over MCP → 10 tools (6 read + 4 new). `index_document` auth gate works.
+- Worker drained the MCP-enqueued job (completed count incremented, failed:0).
+- `turbo run typecheck` green across all 4 apps.
+
 ## Decisions locked (from initial planning)

 - **Tooling:** bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).
@@ -6,7 +6,7 @@ import { fetchRequestHandler } from "@trpc/server/adapters/fetch";
 import { db } from "@mcpedia/db";
 import { appRouter } from "./router";
 import type { Context } from "./trpc";
-import { enqueueIndexDoc, enqueueFullIndex } from "@mcpedia/queue";
+import { enqueueIndexDoc, enqueueFullIndex, getQueue, INDEX_QUEUE } from "@mcpedia/queue";
 import { WEBHOOK_SECRET } from "@mcpedia/config";

 // Fail fast: never expose an open git-sync endpoint. If the operator hasn't
@@ -22,6 +22,34 @@ const app = new Hono();
 // Health check (no auth — safe to expose).
 app.get("/health", (c) => c.json({ ok: true }));

+// --- Phase 7: Prometheus metrics (public, safe to scrape) ---
+const startedAt = Date.now();
+app.get("/metrics", async (c) => {
+  const queue = getQueue();
+  const [waiting, active, completed, failed, delayed] = await Promise.all([
+    queue.getWaitingCount(),
+    queue.getActiveCount(),
+    queue.getCompletedCount(),
+    queue.getFailedCount(),
+    queue.getDelayedCount(),
+  ]);
+  const lines = [
+    "# HELP mcpedia_uptime_seconds seconds since process start",
+    "# TYPE mcpedia_uptime_seconds gauge",
+    `mcpedia_uptime_seconds ${((Date.now() - startedAt) / 1000).toFixed(1)}`,
+    `# HELP mcpedia_queue_jobs queue job counts for "${INDEX_QUEUE}"`,
+    "# TYPE mcpedia_queue_jobs gauge",
+    `mcpedia_queue_jobs{state="waiting"} ${waiting}`,
+    `mcpedia_queue_jobs{state="active"} ${active}`,
+    `mcpedia_queue_jobs{state="completed"} ${completed}`,
+    `mcpedia_queue_jobs{state="failed"} ${failed}`,
+    `mcpedia_queue_jobs{state="delayed"} ${delayed}`,
+  ];
+  return c.text(lines.join("\n") + "\n", 200, {
+    "Content-Type": "text/plain; version=0.0.4; charset=utf-8",
+  });
+});
+
 // Shared guard for the git-sync webhooks: require `x-webhook-secret` header to
 // match the configured secret. Reject anything else with 401.
 // Verify a git-provider webhook. Supports GitHub's native HMAC signature
@@ -16,6 +16,7 @@
  "dependencies": {
    "@mcpedia/config": "workspace:*",
    "@mcpedia/core": "workspace:*",
+    "@mcpedia/queue": "workspace:*",
    "@mcpedia/search": "workspace:*",
    "@modelcontextprotocol/sdk": "^1.29.0",
    "zod": "^4.0.0"
@@ -32,9 +32,12 @@ const httpServer = createServer(async (req: IncomingMessage, res: ServerResponse
    return;
  }

-  // Stateless: fresh server + transport per request.
+  // Stateless: fresh server + transport per request. The x-webhook-secret header
+  // (if present) is threaded into the server so write tools can require it.
+  const rawSecret = req.headers["x-webhook-secret"];
+  const authSecret = Array.isArray(rawSecret) ? rawSecret[0] : rawSecret;
  const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
-  const server = createMcpServer();
+  const server = createMcpServer(authSecret);
  await server.connect(transport);
  await transport.handleRequest(req, res);
 });
@@ -10,12 +10,24 @@ import {
  hybridSearch,
  keywordSearch,
  listRevisions,
+  getRevision,
+  restoreRevision,
  readContentFile,
 } from "@mcpedia/core";
+import { enqueueIndexDoc, enqueueFullIndex, getQueue, INDEX_QUEUE } from "@mcpedia/queue";
 import { CONTENT_ROOT } from "@mcpedia/config";
 import { join } from "node:path";

-export function createMcpServer(): McpServer {
+// Write tools require the caller to supply `x-webhook-secret` matching the
+// configured WEBHOOK_SECRET. Read tools are open. `authSecret` is threaded from
+// the HTTP transport (the request header); for stdio it is undefined (local use).
+function requireMcpAuth(authSecret?: string) {
+  if (!authSecret) {
+    throw new Error("unauthorized: this tool requires the x-webhook-secret header");
+  }
+}
+
+export function createMcpServer(authSecret?: string): McpServer {
  const server = new McpServer({
    name: "mcpedia",
    version: "0.1.0",
@@ -131,6 +143,88 @@ export function createMcpServer(): McpServer {
    },
  );

+  // --- Phase 7: mutating + admin tools (require x-webhook-secret) ---
+  server.registerTool(
+    "index_document",
+    {
+      description:
+        "Enqueue a single-document reindex job (parses, upserts, re-embeds). Requires the x-webhook-secret header. slug is the content path without extension, e.g. 'docs/caddy/reverse-proxy'.",
+      inputSchema: z.object({
+        slug: z.string().describe("Document slug, e.g. 'docs/caddy/reverse-proxy'"),
+      }),
+    },
+    async ({ slug }) => {
+      requireMcpAuth(authSecret);
+      const relPath = slug.endsWith(".md") || slug.endsWith(".mdx") ? slug : `${slug}.md`;
+      const job = await enqueueIndexDoc(relPath, "mcp");
+      return {
+        content: [{ type: "text", text: JSON.stringify({ ok: true, jobId: job.id, slug }) }],
+      };
+    },
+  );
+
+  server.registerTool(
+    "reindex_all",
+    {
+      description:
+        "Enqueue a full-corpus reindex (walks every content file). Requires the x-webhook-secret header.",
+      inputSchema: z.object({}),
+    },
+    async () => {
+      requireMcpAuth(authSecret);
+      const job = await enqueueFullIndex("mcp");
+      return {
+        content: [{ type: "text", text: JSON.stringify({ ok: true, jobId: job.id, kind: "full" }) }],
+      };
+    },
+  );
+
+  server.registerTool(
+    "restore_revision",
+    {
+      description:
+        "Restore a document revision by id (writes its body back into the live row + rebuilds chunks). Requires the x-webhook-secret header.",
+      inputSchema: z.object({ id: z.string().describe("Revision UUID") }),
+    },
+    async ({ id }) => {
+      requireMcpAuth(authSecret);
+      const result = await restoreRevision(id);
+      return {
+        content: [{ type: "text", text: JSON.stringify(result ?? { ok: false, error: "not found" }) }],
+      };
+    },
+  );
+
+  server.registerTool(
+    "queue_status",
+    {
+      description: "Current BullMQ index-queue counts (waiting/active/completed/failed/delayed).",
+      inputSchema: z.object({}),
+    },
+    async () => {
+      const queue = getQueue();
+      const [waiting, active, completed, failed, delayed] = await Promise.all([
+        queue.getWaitingCount(),
+        queue.getActiveCount(),
+        queue.getCompletedCount(),
+        queue.getFailedCount(),
+        queue.getDelayedCount(),
+      ]);
+      return {
+        content: [
+          {
+            type: "text",
+            text: JSON.stringify(
+              { queue: INDEX_QUEUE, counts: { waiting, active, completed, failed, delayed } },
+              null,
+              2,
+            ),
+          },
+        ],
+      };
+    },
+  );
+
  // --- Phase 3: MCP Resources (read-only knowledge base surfaced via URIs) ---
  // mcpedia://docs                       -> list all published documents
  // mcpedia://docs/{slug}                -> full markdown body (from disk)
@@ -38,6 +38,7 @@
      "dependencies": {
        "@mcpedia/config": "workspace:*",
        "@mcpedia/core": "workspace:*",
+        "@mcpedia/queue": "workspace:*",
        "@mcpedia/search": "workspace:*",
        "@modelcontextprotocol/sdk": "^1.29.0",
        "zod": "^4.0.0",
@@ -0,0 +1,68 @@
+---
+id: bullmq-workers
+title: BullMQ Background Workers
+type: documentation
+tags:
+  - bullmq
+  - redis
+  - jobs
+  - queue
+  - infra
+status: published
+author: asep
+created_at: 2026-08-20
+updated_at: 2026-08-20
+---
+
+# BullMQ Background Workers
+
+BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
+git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
+shared Redis instance.
+
+## Connection requirements
+
+BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
+retry count causes the cryptic `Connection in key mode` error on blocking commands.
+The shared client in `packages/queue` sets this correctly:
+
+```ts
+const opts: RedisOptions = {
+  maxRetriesPerRequest: null,
+  lazyConnect: true,
+  enableOfflineQueue: true,
+};
+```
+
+## Job model
+
+Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
+
+| name       | data              | action                          |
+|------------|-------------------|---------------------------------|
+| `index-doc`| `{ relPath, reason }` | index one content file      |
+| `index-all`| `{ reason }`      | full corpus reindex            |
+| `reindex`  | (legacy)          | alias of full                  |
+
+Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
+repeatable jobs, so a literal `:` in a custom jobId is rejected.
+
+## Worker lifecycle
+
+The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
+`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
+path the CLI uses, so behavior never diverges. On completion it logs; on failure it
+logs the reason and the job is retried per BullMQ defaults.
+
+Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
+stop, so the process exits cleanly and in-flight jobs are returned to the queue.
+
+## Inspecting state
+
+```bash
+bun run enqueue --all          # enqueue a full reindex
+curl localhost:4020/trpc/queueStatus   # waiting/active/completed/failed
+```
+
+A stuck queue (waiting > 0, active = 0) means the worker died — check
+`systemctl status mcpedia-worker` and the journal.
@@ -0,0 +1,85 @@
+---
+id: caddy-reverse-proxy
+title: Caddy Reverse Proxy
+type: documentation
+tags:
+  - caddy
+  - reverse-proxy
+  - tls
+  - infra
+status: published
+author: asep
+created_at: 2026-08-20
+updated_at: 2026-08-20
+---
+
+# Caddy Reverse Proxy
+
+Caddy is the single reverse proxy on the host. Every public service sits behind it
+and terminates TLS with automatic Let's Encrypt certificates. Understanding its
+config model prevents the two recurring failure modes: **525 (origin TLS)** and
+**404 (path routing)**.
+
+## Config model
+
+The live config is `/etc/caddy/Caddyfile`. It is a manual, tuned file — CI does not
+deploy it, so the live file is the source of truth and must be kept in sync with any
+repo reference.
+
+A reusable snippet handles the common case:
+
+```
+(proxy) {
+    encode zstd gzip
+    header {
+        -Server
+        X-Content-Type-Options "nosniff"
+    }
+    reverse_proxy 127.0.0.1:{args[0]} {
+        transport http {
+            keepalive 120s
+            dial_timeout 3s
+        }
+    }
+}
+```
+
+A site block wires a domain to a backend port:
+
+```
+wiki.asepharyana.my.id {
+    import proxy 4016
+}
+```
+
+## Path routing: `handle` vs `handle_path`
+
+`handle /trpc/*` forwards the request **with** the `/trpc` prefix preserved.
+`handle_path /trpc/*` **strips** it before proxying. Stripping is wrong when the
+upstream already mounts the route at `/trpc` — the upstream then receives `/` and 404s.
+
+Rule: when the upstream already serves the path (e.g. Hono `app.all("/trpc/*")`),
+use `handle`, not `handle_path`.
+
+## 525 — origin TLS handshake failed
+
+Cloudflare proxies every `*.asepharyana.my.id` record. If a subdomain has **no**
+Caddy site block, Caddy has no certificate for that SNI and the TLS handshake dies →
+Cloudflare returns 525. Fix: add the block, `caddy validate`, `systemctl reload caddy`.
+The first request after adding a block triggers ACME certificate issuance; until it
+completes the origin may briefly 525. That is expected and self-heals in ~10s.
+
+## Slow upstreams
+
+LLM gateways (9router) have time-to-first-token of 30–40s. The default
+`response_header_timeout 30s` yields false 504s. Lengthen it for those blocks:
+
+```
+reverse_proxy 127.0.0.1:4014 {
+    transport http {
+        response_header_timeout 120s
+        read_timeout 300s
+        write_timeout 300s
+    }
+}
+```
@@ -0,0 +1,60 @@
+---
+id: mcp-streamable-http
+title: MCP Streamable HTTP Transport
+type: documentation
+tags:
+  - mcp
+  - protocol
+  - http
+  - infra
+status: published
+author: asep
+created_at: 2026-08-20
+updated_at: 2026-08-20
+---
+
+# MCP Streamable HTTP Transport
+
+The MCPedia MCP server is served over **Streamable HTTP** (the MCP 2025-03-26
+transport) so remote clients — Claude, a Discord bot, a web frontend — can call its
+tools and read its resources without spawning a stdio subprocess.
+
+## Why stateless
+
+The server uses `StreamableHTTPServerTransport` in **stateless mode**
+(`sessionIdGenerator: undefined`):
+
+- One `McpServer` + transport is created **per request**.
+- No session affinity, no shared-transport `connect()` race, no session-map memory
+  leak under burst traffic.
+- Re-registering the 6 tools + 4 resources per request is negligible for a KB-sized
+  corpus.
+
+Stateful mode (a `sessionIdGenerator` returning a UUID) would require holding a
+transport map keyed by session id and cleaning it up on `onclose`. For this read-mostly
+knowledge base, stateless is simpler and equally correct.
+
+## Endpoint
+
+```
+POST https://mcp.asepharyana.my.id/mcp
+Content-Type: application/json
+Accept: application/json, text/event-stream
+```
+
+Responses use SSE framing (`event: message` / `data: {...}`) even for unary results.
+Clients must send `Accept: application/json, text/event-stream` or the server returns
+406. The MCP `initialize` handshake sets `protocolVersion: "2025-03-26"`.
+
+## CORS
+
+`/mcp` returns permissive CORS headers (`Access-Control-Allow-Origin: *`) so browser
+clients can call it directly. Preflight `OPTIONS` is answered with 204.
+
+## Auth for write tools
+
+Read tools (`search_documents`, `get_document`, ...) are open. Write tools
+(`index_document`, `reindex_all`, `restore_revision`) require the
+`x-webhook-secret` header to match `WEBHOOK_SECRET` — the same shared secret used by
+the git-sync webhook. A missing/invalid header makes the tool return an error before
+any mutation.
@@ -0,0 +1,59 @@
+---
+id: postgres-full-text-search
+title: PostgreSQL Full-Text Search
+type: documentation
+tags:
+  - postgres
+  - fts
+  - tsvector
+  - search
+status: published
+author: asep
+created_at: 2026-08-20
+updated_at: 2026-08-20
+---
+
+# PostgreSQL Full-Text Search
+
+MCPedia's keyword search is backed by PostgreSQL's native full-text search (FTS), not
+an external engine. The `documents` table carries a generated `tsvector` column that
+combines the title (weight `A`) and body (weight `B`).
+
+## Generated search vector
+
+The column is `generatedAlwaysAs`, so it is always consistent with the row and needs
+no trigger:
+
+```ts
+searchVector: tsvector("search_vector")
+  .notNull()
+  .generatedAlwaysAs(
+    sql`setweight(to_tsvector('simple', coalesce(${documents.title}, '')), 'A') ||
+        setweight(to_tsvector('simple', coalesce(${documents.body}, '')), 'B')`,
+  ),
+```
+
+The `'simple'` config disables stemming, so mixed identifier/English queries (e.g.
+`websocket`, `tsvector`) match literally. A GIN index on `searchVector` keeps lookups
+fast.
+
+## Query + ranking
+
+Search parses the user query with `websearch_to_tsquery` (or `plainto_tsquery`), then
+ranks with `ts_rank`:
+
+```sql
+SELECT *, ts_rank(search_vector, q) AS rank
+FROM documents, websearch_to_tsquery('simple', $1) q
+WHERE search_vector @@ q
+ORDER BY rank DESC;
+```
+
+A headline snippet for the UI comes from `ts_headline`, which bolds the matched lexemes.
+
+## Hybrid fusion
+
+Semantic search (embedding cosine) and FTS are fused with **Reciprocal Rank Fusion**
+(RRF) in `packages/search`. Each result set is ranked, scored `1/(k + rank)`, and the
+summed scores re-rank the union — no cross-score normalization needed, which is robust
+when the two signals live on different scales.
@@ -0,0 +1,56 @@
+---
+id: cloudflare-525-writeup
+title: Diagnosing Cloudflare 525 (Origin TLS Handshake Failed)
+type: writeup
+tags:
+  - cloudflare
+  - tls
+  - 525
+  - caddy
+  - debugging
+status: published
+author: asep
+created_at: 2026-08-20
+updated_at: 2026-08-20
+---
+
+# Diagnosing Cloudflare 525 (Origin TLS Handshake Failed)
+
+A 525 appears between the user and the origin when Cloudflare (Strict TLS mode) cannot
+complete the TLS handshake to the origin server. This writeup captures the debugging
+loop that recurred while wiring new subdomains.
+
+## Symptom
+
+`curl https://<sub>.asepharyana.my.id` → `HTTP/2 525`. Browser shows Cloudflare's
+"SSL handshake failed" page.
+
+## Root causes (in order of likelihood)
+
+1. **No Caddy site block for the SNI.** Cloudflare proxies every `*.asepharyana.my.id`
+   record. A host without a matching Caddy `site` block has no certificate, so the
+   handshake dies. This is the #1 cause and the one that bit `wiki` and `mcp`.
+2. **Certificate still provisioning.** The first request after adding a block triggers
+   ACME `http-01` issuance. Until the cert lands (~10s), the origin 525s. Self-heals.
+3. **Wrong cert presented.** Rare here — Caddy serves the SNI-matched cert; a mismatch
+   means the block points at the wrong backend or the cert store is stale.
+
+## The debugging loop
+
+```
+curl -sI https://<sub>/            # 525?
+grep -n "<sub>" /etc/caddy/Caddyfile   # block present?
+sudo journalctl -u caddy | grep -i "tls\|acme\|<sub>"   # cert issued?
+openssl s_client -connect 127.0.0.1:443 -servername <sub>   # origin cert valid?
+```
+
+If the block is missing: add `import proxy <port>`, `caddy validate`, `systemctl
+reload caddy`. If the cert is mid-issuance: wait and re-test. Do **not** point Cloudflare
+at a non-existent origin or set the SSL mode to Flexible — Flexible mode breaks already-
+working Strict setups.
+
+## Lesson
+
+Every new subdomain needs (a) a Caddy site block and (b) a Cloudflare DNS record that
+proxies to the origin. Omit either and you get a 525. The wildcard DNS means you only
+add the Caddy side.