feat(phase7): grow corpus, MCP write-tools+auth, /metrics observability
CI / typecheck + build (turbo) (push) Canceled after 0s
CI / typecheck + build (turbo) (push) Canceled after 0s
- content/: +5 real docs (caddy, bullmq, mcp-streamable-http, postgres-fts, cloudflare-525 writeup) across docs/writeups/notes. Reindexed: 9 docs, 17 chunks, 5 revisions (was 4 docs). - apps/mcp: add write-tools index_document/reindex_all/restore_revision (require x-webhook-secret) + queue_status (public). createMcpServer(authSecret?) threads the HTTP header; stdio keeps writes open (trusted local). - apps/api: GET /metrics (Prometheus text: uptime + queue job gauges). - Caddy: expose /metrics on wiki. domain -> :4020. Verified live: /metrics 200; MCP tools/list -> 10; index_document unauth -> error, auth -> enqueues + worker drains; typecheck green.
This commit is contained in:
@@ -0,0 +1,41 @@
|
|||||||
|
# MCPedia — "lanjut semua" workstream
|
||||||
|
|
||||||
|
Three real gaps remain (from review): tiny corpus (4 docs), MCP read-only (no write/auth),
|
||||||
|
no observability. This plan closes all three.
|
||||||
|
|
||||||
|
## 1. Content corpus (grow the KB)
|
||||||
|
Author real, useful docs so search/semantic/revisions have something to operate on.
|
||||||
|
Frontmatter schema (from packages/parser): id,title,type,tags,status,author,created_at,updated_at.
|
||||||
|
Sections: docs|writeups|research|notes. Files under content/<section>/...
|
||||||
|
New docs to add:
|
||||||
|
- content/docs/caddy/reverse-proxy.md (ops reference, tags: caddy, reverse-proxy, tls)
|
||||||
|
- content/docs/bullmq/workers.md (queue/worker reference, tags: bullmq, redis, jobs)
|
||||||
|
- content/docs/mcp/streamable-http.md (MCP transport reference, tags: mcp, protocol, http)
|
||||||
|
- content/notes/postgres/full-text-search.md (PG FTS notes, tags: postgres, fts, tsvector)
|
||||||
|
- content/writeups/infra/cloudflare-525.md (debugging writeup, tags: cloudflare, tls, 525)
|
||||||
|
After adding: `bun run index` to reindex (writes revisions + chunks), verify counts.
|
||||||
|
|
||||||
|
## 2. MCP write-tools + auth
|
||||||
|
Add mutating + admin tools to the MCP server (currently read-only):
|
||||||
|
- `index_document(slug)` -> enqueueIndexDoc (requires MCP auth header)
|
||||||
|
- `reindex_all()` -> enqueueFullIndex (requires MCP auth header)
|
||||||
|
- `queue_status()` -> getQueue counts (read, public)
|
||||||
|
- `restore_revision(id)` -> restoreRevision (requires MCP auth header)
|
||||||
|
Auth: MCP client must send header `x-webhook-secret` (reuse WEBHOOK_SECRET). StreamableHTTP
|
||||||
|
transport: read the Authorization/header in http.ts, pass to server via a factory closure
|
||||||
|
capturing the request; tools check it. Stateless per-request server already created fresh,
|
||||||
|
so threading the header is clean. Guard write-tools with the same requireWriteAuth logic.
|
||||||
|
Verify: unauthenticated call to index_document -> error; authenticated -> enqueues job.
|
||||||
|
|
||||||
|
## 3. Observability
|
||||||
|
- `GET /metrics` on the API (Prometheus text format): queue counts (waiting/active/
|
||||||
|
completed/failed/delayed), uptime, service name. Public (safe to expose).
|
||||||
|
- Caddy: expose /metrics on wiki. domain -> :4020 (add to handle list).
|
||||||
|
- tRPC `queueStatus` already exists; /metrics reuses getQueue.
|
||||||
|
Verify: curl /metrics -> text exposition with mcpedia_queue_* gauges.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
- typecheck green (turbo run typecheck).
|
||||||
|
- MCP smoke extended: tools/list shows new tools; authenticated index_document enqueues.
|
||||||
|
- /metrics returns 200 text; queue drains.
|
||||||
|
- commit + push.
|
||||||
@@ -154,6 +154,30 @@ bun run api # Hono+tRPC API on :4020 (added /hooks/* webhooks)
|
|||||||
- GitHub push webhook delivers 200; `queueStatus` shows completed:N, failed:0.
|
- GitHub push webhook delivers 200; `queueStatus` shows completed:N, failed:0.
|
||||||
- `turbo run typecheck` green across all 4 apps.
|
- `turbo run typecheck` green across all 4 apps.
|
||||||
|
|
||||||
|
## Phase 7 — Corpus, MCP write-tools + auth, observability ✅ DONE
|
||||||
|
|
||||||
|
> Closed the remaining review gaps: tiny corpus (4 docs), MCP read-only (no write/auth),
|
||||||
|
> no observability.
|
||||||
|
|
||||||
|
- [x] **Content corpus grown** — added 5 real docs (Caddy reverse proxy, BullMQ workers,
|
||||||
|
MCP Streamable HTTP, Postgres FTS, Cloudflare-525 debugging writeup) across
|
||||||
|
docs/writeups/notes. `bun run index` reindexed: **9 documents, 17 chunks, 5 new
|
||||||
|
revisions** (was 4 docs). Search/semantic/revisions now operate on a real corpus.
|
||||||
|
- [x] **MCP write-tools + auth** — added `index_document`, `reindex_all`,
|
||||||
|
`restore_revision` (write, require `x-webhook-secret`) and `queue_status` (public).
|
||||||
|
`createMcpServer(authSecret?)` threads the request header; stdio keeps write tools
|
||||||
|
open (trusted local). Verified: unauthenticated `index_document` → `isError` +
|
||||||
|
"unauthorized"; authenticated → enqueues job, worker drains.
|
||||||
|
- [x] **Observability** — `GET /metrics` on the API (Prometheus text exposition:
|
||||||
|
`mcpedia_uptime_seconds`, `mcpedia_queue_jobs{state=...}`). Exposed on the domain at
|
||||||
|
`https://wiki.asepharyana.my.id/metrics`. Public, safe to scrape.
|
||||||
|
|
||||||
|
### Verification done (real, against live services)
|
||||||
|
- `https://wiki.asepharyana.my.id/metrics` → 200 Prometheus text (uptime + queue gauges).
|
||||||
|
- `tools/list` over MCP → 10 tools (6 read + 4 new). `index_document` auth gate works.
|
||||||
|
- Worker drained the MCP-enqueued job (completed count incremented, failed:0).
|
||||||
|
- `turbo run typecheck` green across all 4 apps.
|
||||||
|
|
||||||
## Decisions locked (from initial planning)
|
## Decisions locked (from initial planning)
|
||||||
|
|
||||||
- **Tooling:** bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).
|
- **Tooling:** bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).
|
||||||
|
|||||||
+29
-1
@@ -6,7 +6,7 @@ import { fetchRequestHandler } from "@trpc/server/adapters/fetch";
|
|||||||
import { db } from "@mcpedia/db";
|
import { db } from "@mcpedia/db";
|
||||||
import { appRouter } from "./router";
|
import { appRouter } from "./router";
|
||||||
import type { Context } from "./trpc";
|
import type { Context } from "./trpc";
|
||||||
import { enqueueIndexDoc, enqueueFullIndex } from "@mcpedia/queue";
|
import { enqueueIndexDoc, enqueueFullIndex, getQueue, INDEX_QUEUE } from "@mcpedia/queue";
|
||||||
import { WEBHOOK_SECRET } from "@mcpedia/config";
|
import { WEBHOOK_SECRET } from "@mcpedia/config";
|
||||||
|
|
||||||
// Fail fast: never expose an open git-sync endpoint. If the operator hasn't
|
// Fail fast: never expose an open git-sync endpoint. If the operator hasn't
|
||||||
@@ -22,6 +22,34 @@ const app = new Hono();
|
|||||||
// Health check (no auth — safe to expose).
|
// Health check (no auth — safe to expose).
|
||||||
app.get("/health", (c) => c.json({ ok: true }));
|
app.get("/health", (c) => c.json({ ok: true }));
|
||||||
|
|
||||||
|
// --- Phase 7: Prometheus metrics (public, safe to scrape) ---
|
||||||
|
const startedAt = Date.now();
|
||||||
|
app.get("/metrics", async (c) => {
|
||||||
|
const queue = getQueue();
|
||||||
|
const [waiting, active, completed, failed, delayed] = await Promise.all([
|
||||||
|
queue.getWaitingCount(),
|
||||||
|
queue.getActiveCount(),
|
||||||
|
queue.getCompletedCount(),
|
||||||
|
queue.getFailedCount(),
|
||||||
|
queue.getDelayedCount(),
|
||||||
|
]);
|
||||||
|
const lines = [
|
||||||
|
"# HELP mcpedia_uptime_seconds seconds since process start",
|
||||||
|
"# TYPE mcpedia_uptime_seconds gauge",
|
||||||
|
`mcpedia_uptime_seconds ${((Date.now() - startedAt) / 1000).toFixed(1)}`,
|
||||||
|
`# HELP mcpedia_queue_jobs queue job counts for "${INDEX_QUEUE}"`,
|
||||||
|
"# TYPE mcpedia_queue_jobs gauge",
|
||||||
|
`mcpedia_queue_jobs{state="waiting"} ${waiting}`,
|
||||||
|
`mcpedia_queue_jobs{state="active"} ${active}`,
|
||||||
|
`mcpedia_queue_jobs{state="completed"} ${completed}`,
|
||||||
|
`mcpedia_queue_jobs{state="failed"} ${failed}`,
|
||||||
|
`mcpedia_queue_jobs{state="delayed"} ${delayed}`,
|
||||||
|
];
|
||||||
|
return c.text(lines.join("\n") + "\n", 200, {
|
||||||
|
"Content-Type": "text/plain; version=0.0.4; charset=utf-8",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
// Shared guard for the git-sync webhooks: require `x-webhook-secret` header to
|
// Shared guard for the git-sync webhooks: require `x-webhook-secret` header to
|
||||||
// match the configured secret. Reject anything else with 401.
|
// match the configured secret. Reject anything else with 401.
|
||||||
// Verify a git-provider webhook. Supports GitHub's native HMAC signature
|
// Verify a git-provider webhook. Supports GitHub's native HMAC signature
|
||||||
|
|||||||
@@ -16,6 +16,7 @@
|
|||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@mcpedia/config": "workspace:*",
|
"@mcpedia/config": "workspace:*",
|
||||||
"@mcpedia/core": "workspace:*",
|
"@mcpedia/core": "workspace:*",
|
||||||
|
"@mcpedia/queue": "workspace:*",
|
||||||
"@mcpedia/search": "workspace:*",
|
"@mcpedia/search": "workspace:*",
|
||||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||||
"zod": "^4.0.0"
|
"zod": "^4.0.0"
|
||||||
|
|||||||
@@ -32,9 +32,12 @@ const httpServer = createServer(async (req: IncomingMessage, res: ServerResponse
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stateless: fresh server + transport per request.
|
// Stateless: fresh server + transport per request. The x-webhook-secret header
|
||||||
|
// (if present) is threaded into the server so write tools can require it.
|
||||||
|
const rawSecret = req.headers["x-webhook-secret"];
|
||||||
|
const authSecret = Array.isArray(rawSecret) ? rawSecret[0] : rawSecret;
|
||||||
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
|
const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: undefined });
|
||||||
const server = createMcpServer();
|
const server = createMcpServer(authSecret);
|
||||||
await server.connect(transport);
|
await server.connect(transport);
|
||||||
await transport.handleRequest(req, res);
|
await transport.handleRequest(req, res);
|
||||||
});
|
});
|
||||||
|
|||||||
+95
-1
@@ -10,12 +10,24 @@ import {
|
|||||||
hybridSearch,
|
hybridSearch,
|
||||||
keywordSearch,
|
keywordSearch,
|
||||||
listRevisions,
|
listRevisions,
|
||||||
|
getRevision,
|
||||||
|
restoreRevision,
|
||||||
readContentFile,
|
readContentFile,
|
||||||
} from "@mcpedia/core";
|
} from "@mcpedia/core";
|
||||||
|
import { enqueueIndexDoc, enqueueFullIndex, getQueue, INDEX_QUEUE } from "@mcpedia/queue";
|
||||||
import { CONTENT_ROOT } from "@mcpedia/config";
|
import { CONTENT_ROOT } from "@mcpedia/config";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
|
||||||
export function createMcpServer(): McpServer {
|
// Write tools require the caller to supply `x-webhook-secret` matching the
|
||||||
|
// configured WEBHOOK_SECRET. Read tools are open. `authSecret` is threaded from
|
||||||
|
// the HTTP transport (the request header); for stdio it is undefined (local use).
|
||||||
|
function requireMcpAuth(authSecret?: string) {
|
||||||
|
if (!authSecret) {
|
||||||
|
throw new Error("unauthorized: this tool requires the x-webhook-secret header");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createMcpServer(authSecret?: string): McpServer {
|
||||||
const server = new McpServer({
|
const server = new McpServer({
|
||||||
name: "mcpedia",
|
name: "mcpedia",
|
||||||
version: "0.1.0",
|
version: "0.1.0",
|
||||||
@@ -131,6 +143,88 @@ export function createMcpServer(): McpServer {
|
|||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// --- Phase 7: mutating + admin tools (require x-webhook-secret) ---
|
||||||
|
server.registerTool(
|
||||||
|
"index_document",
|
||||||
|
{
|
||||||
|
description:
|
||||||
|
"Enqueue a single-document reindex job (parses, upserts, re-embeds). Requires the x-webhook-secret header. slug is the content path without extension, e.g. 'docs/caddy/reverse-proxy'.",
|
||||||
|
inputSchema: z.object({
|
||||||
|
slug: z.string().describe("Document slug, e.g. 'docs/caddy/reverse-proxy'"),
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
async ({ slug }) => {
|
||||||
|
requireMcpAuth(authSecret);
|
||||||
|
const relPath = slug.endsWith(".md") || slug.endsWith(".mdx") ? slug : `${slug}.md`;
|
||||||
|
const job = await enqueueIndexDoc(relPath, "mcp");
|
||||||
|
return {
|
||||||
|
content: [{ type: "text", text: JSON.stringify({ ok: true, jobId: job.id, slug }) }],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
"reindex_all",
|
||||||
|
{
|
||||||
|
description:
|
||||||
|
"Enqueue a full-corpus reindex (walks every content file). Requires the x-webhook-secret header.",
|
||||||
|
inputSchema: z.object({}),
|
||||||
|
},
|
||||||
|
async () => {
|
||||||
|
requireMcpAuth(authSecret);
|
||||||
|
const job = await enqueueFullIndex("mcp");
|
||||||
|
return {
|
||||||
|
content: [{ type: "text", text: JSON.stringify({ ok: true, jobId: job.id, kind: "full" }) }],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
"restore_revision",
|
||||||
|
{
|
||||||
|
description:
|
||||||
|
"Restore a document revision by id (writes its body back into the live row + rebuilds chunks). Requires the x-webhook-secret header.",
|
||||||
|
inputSchema: z.object({ id: z.string().describe("Revision UUID") }),
|
||||||
|
},
|
||||||
|
async ({ id }) => {
|
||||||
|
requireMcpAuth(authSecret);
|
||||||
|
const result = await restoreRevision(id);
|
||||||
|
return {
|
||||||
|
content: [{ type: "text", text: JSON.stringify(result ?? { ok: false, error: "not found" }) }],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
server.registerTool(
|
||||||
|
"queue_status",
|
||||||
|
{
|
||||||
|
description: "Current BullMQ index-queue counts (waiting/active/completed/failed/delayed).",
|
||||||
|
inputSchema: z.object({}),
|
||||||
|
},
|
||||||
|
async () => {
|
||||||
|
const queue = getQueue();
|
||||||
|
const [waiting, active, completed, failed, delayed] = await Promise.all([
|
||||||
|
queue.getWaitingCount(),
|
||||||
|
queue.getActiveCount(),
|
||||||
|
queue.getCompletedCount(),
|
||||||
|
queue.getFailedCount(),
|
||||||
|
queue.getDelayedCount(),
|
||||||
|
]);
|
||||||
|
return {
|
||||||
|
content: [
|
||||||
|
{
|
||||||
|
type: "text",
|
||||||
|
text: JSON.stringify(
|
||||||
|
{ queue: INDEX_QUEUE, counts: { waiting, active, completed, failed, delayed } },
|
||||||
|
null,
|
||||||
|
2,
|
||||||
|
),
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
// --- Phase 3: MCP Resources (read-only knowledge base surfaced via URIs) ---
|
// --- Phase 3: MCP Resources (read-only knowledge base surfaced via URIs) ---
|
||||||
// mcpedia://docs -> list all published documents
|
// mcpedia://docs -> list all published documents
|
||||||
// mcpedia://docs/{slug} -> full markdown body (from disk)
|
// mcpedia://docs/{slug} -> full markdown body (from disk)
|
||||||
|
|||||||
@@ -38,6 +38,7 @@
|
|||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@mcpedia/config": "workspace:*",
|
"@mcpedia/config": "workspace:*",
|
||||||
"@mcpedia/core": "workspace:*",
|
"@mcpedia/core": "workspace:*",
|
||||||
|
"@mcpedia/queue": "workspace:*",
|
||||||
"@mcpedia/search": "workspace:*",
|
"@mcpedia/search": "workspace:*",
|
||||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||||
"zod": "^4.0.0",
|
"zod": "^4.0.0",
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
---
|
||||||
|
id: bullmq-workers
|
||||||
|
title: BullMQ Background Workers
|
||||||
|
type: documentation
|
||||||
|
tags:
|
||||||
|
- bullmq
|
||||||
|
- redis
|
||||||
|
- jobs
|
||||||
|
- queue
|
||||||
|
- infra
|
||||||
|
status: published
|
||||||
|
author: asep
|
||||||
|
created_at: 2026-08-20
|
||||||
|
updated_at: 2026-08-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# BullMQ Background Workers
|
||||||
|
|
||||||
|
BullMQ runs the async indexing/embedding pipeline. Jobs are enqueued by the CLI, the
|
||||||
|
git-sync webhook, or an MCP tool, and drained by a long-running worker connected to the
|
||||||
|
shared Redis instance.
|
||||||
|
|
||||||
|
## Connection requirements
|
||||||
|
|
||||||
|
BullMQ requires an **ioredis** connection with `maxRetriesPerRequest: null`. A finite
|
||||||
|
retry count causes the cryptic `Connection in key mode` error on blocking commands.
|
||||||
|
The shared client in `packages/queue` sets this correctly:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const opts: RedisOptions = {
|
||||||
|
maxRetriesPerRequest: null,
|
||||||
|
lazyConnect: true,
|
||||||
|
enableOfflineQueue: true,
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
## Job model
|
||||||
|
|
||||||
|
Three job types flow through the `mcpedia-index` queue (prefix `mcpedia:` on Redis):
|
||||||
|
|
||||||
|
| name | data | action |
|
||||||
|
|------------|-------------------|---------------------------------|
|
||||||
|
| `index-doc`| `{ relPath, reason }` | index one content file |
|
||||||
|
| `index-all`| `{ reason }` | full corpus reindex |
|
||||||
|
| `reindex` | (legacy) | alias of full |
|
||||||
|
|
||||||
|
Job IDs use a `__` separator (`doc__<slug>`, `full__<ts>`) — BullMQ reserves `:` for
|
||||||
|
repeatable jobs, so a literal `:` in a custom jobId is rejected.
|
||||||
|
|
||||||
|
## Worker lifecycle
|
||||||
|
|
||||||
|
The worker is a `Worker` with `concurrency: 4`. Each job calls the shared
|
||||||
|
`indexContentFile` / `runFullIndex` entry points in `@mcpedia/core` — the same code
|
||||||
|
path the CLI uses, so behavior never diverges. On completion it logs; on failure it
|
||||||
|
logs the reason and the job is retried per BullMQ defaults.
|
||||||
|
|
||||||
|
Graceful shutdown: `worker.close()` on `SIGINT`/`SIGTERM`. systemd sends SIGTERM on
|
||||||
|
stop, so the process exits cleanly and in-flight jobs are returned to the queue.
|
||||||
|
|
||||||
|
## Inspecting state
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bun run enqueue --all # enqueue a full reindex
|
||||||
|
curl localhost:4020/trpc/queueStatus # waiting/active/completed/failed
|
||||||
|
```
|
||||||
|
|
||||||
|
A stuck queue (waiting > 0, active = 0) means the worker died — check
|
||||||
|
`systemctl status mcpedia-worker` and the journal.
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
---
|
||||||
|
id: caddy-reverse-proxy
|
||||||
|
title: Caddy Reverse Proxy
|
||||||
|
type: documentation
|
||||||
|
tags:
|
||||||
|
- caddy
|
||||||
|
- reverse-proxy
|
||||||
|
- tls
|
||||||
|
- infra
|
||||||
|
status: published
|
||||||
|
author: asep
|
||||||
|
created_at: 2026-08-20
|
||||||
|
updated_at: 2026-08-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# Caddy Reverse Proxy
|
||||||
|
|
||||||
|
Caddy is the single reverse proxy on the host. Every public service sits behind it
|
||||||
|
and terminates TLS with automatic Let's Encrypt certificates. Understanding its
|
||||||
|
config model prevents the two recurring failure modes: **525 (origin TLS)** and
|
||||||
|
**404 (path routing)**.
|
||||||
|
|
||||||
|
## Config model
|
||||||
|
|
||||||
|
The live config is `/etc/caddy/Caddyfile`. It is a manual, tuned file — CI does not
|
||||||
|
deploy it, so the live file is the source of truth and must be kept in sync with any
|
||||||
|
repo reference.
|
||||||
|
|
||||||
|
A reusable snippet handles the common case:
|
||||||
|
|
||||||
|
```
|
||||||
|
(proxy) {
|
||||||
|
encode zstd gzip
|
||||||
|
header {
|
||||||
|
-Server
|
||||||
|
X-Content-Type-Options "nosniff"
|
||||||
|
}
|
||||||
|
reverse_proxy 127.0.0.1:{args[0]} {
|
||||||
|
transport http {
|
||||||
|
keepalive 120s
|
||||||
|
dial_timeout 3s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
A site block wires a domain to a backend port:
|
||||||
|
|
||||||
|
```
|
||||||
|
wiki.asepharyana.my.id {
|
||||||
|
import proxy 4016
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Path routing: `handle` vs `handle_path`
|
||||||
|
|
||||||
|
`handle /trpc/*` forwards the request **with** the `/trpc` prefix preserved.
|
||||||
|
`handle_path /trpc/*` **strips** it before proxying. Stripping is wrong when the
|
||||||
|
upstream already mounts the route at `/trpc` — the upstream then receives `/` and 404s.
|
||||||
|
|
||||||
|
Rule: when the upstream already serves the path (e.g. Hono `app.all("/trpc/*")`),
|
||||||
|
use `handle`, not `handle_path`.
|
||||||
|
|
||||||
|
## 525 — origin TLS handshake failed
|
||||||
|
|
||||||
|
Cloudflare proxies every `*.asepharyana.my.id` record. If a subdomain has **no**
|
||||||
|
Caddy site block, Caddy has no certificate for that SNI and the TLS handshake dies →
|
||||||
|
Cloudflare returns 525. Fix: add the block, `caddy validate`, `systemctl reload caddy`.
|
||||||
|
The first request after adding a block triggers ACME certificate issuance; until it
|
||||||
|
completes the origin may briefly 525. That is expected and self-heals in ~10s.
|
||||||
|
|
||||||
|
## Slow upstreams
|
||||||
|
|
||||||
|
LLM gateways (9router) have time-to-first-token of 30–40s. The default
|
||||||
|
`response_header_timeout 30s` yields false 504s. Lengthen it for those blocks:
|
||||||
|
|
||||||
|
```
|
||||||
|
reverse_proxy 127.0.0.1:4014 {
|
||||||
|
transport http {
|
||||||
|
response_header_timeout 120s
|
||||||
|
read_timeout 300s
|
||||||
|
write_timeout 300s
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
---
|
||||||
|
id: mcp-streamable-http
|
||||||
|
title: MCP Streamable HTTP Transport
|
||||||
|
type: documentation
|
||||||
|
tags:
|
||||||
|
- mcp
|
||||||
|
- protocol
|
||||||
|
- http
|
||||||
|
- infra
|
||||||
|
status: published
|
||||||
|
author: asep
|
||||||
|
created_at: 2026-08-20
|
||||||
|
updated_at: 2026-08-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# MCP Streamable HTTP Transport
|
||||||
|
|
||||||
|
The MCPedia MCP server is served over **Streamable HTTP** (the MCP 2025-03-26
|
||||||
|
transport) so remote clients — Claude, a Discord bot, a web frontend — can call its
|
||||||
|
tools and read its resources without spawning a stdio subprocess.
|
||||||
|
|
||||||
|
## Why stateless
|
||||||
|
|
||||||
|
The server uses `StreamableHTTPServerTransport` in **stateless mode**
|
||||||
|
(`sessionIdGenerator: undefined`):
|
||||||
|
|
||||||
|
- One `McpServer` + transport is created **per request**.
|
||||||
|
- No session affinity, no shared-transport `connect()` race, no session-map memory
|
||||||
|
leak under burst traffic.
|
||||||
|
- Re-registering the 6 tools + 4 resources per request is negligible for a KB-sized
|
||||||
|
corpus.
|
||||||
|
|
||||||
|
Stateful mode (a `sessionIdGenerator` returning a UUID) would require holding a
|
||||||
|
transport map keyed by session id and cleaning it up on `onclose`. For this read-mostly
|
||||||
|
knowledge base, stateless is simpler and equally correct.
|
||||||
|
|
||||||
|
## Endpoint
|
||||||
|
|
||||||
|
```
|
||||||
|
POST https://mcp.asepharyana.my.id/mcp
|
||||||
|
Content-Type: application/json
|
||||||
|
Accept: application/json, text/event-stream
|
||||||
|
```
|
||||||
|
|
||||||
|
Responses use SSE framing (`event: message` / `data: {...}`) even for unary results.
|
||||||
|
Clients must send `Accept: application/json, text/event-stream` or the server returns
|
||||||
|
406. The MCP `initialize` handshake sets `protocolVersion: "2025-03-26"`.
|
||||||
|
|
||||||
|
## CORS
|
||||||
|
|
||||||
|
`/mcp` returns permissive CORS headers (`Access-Control-Allow-Origin: *`) so browser
|
||||||
|
clients can call it directly. Preflight `OPTIONS` is answered with 204.
|
||||||
|
|
||||||
|
## Auth for write tools
|
||||||
|
|
||||||
|
Read tools (`search_documents`, `get_document`, ...) are open. Write tools
|
||||||
|
(`index_document`, `reindex_all`, `restore_revision`) require the
|
||||||
|
`x-webhook-secret` header to match `WEBHOOK_SECRET` — the same shared secret used by
|
||||||
|
the git-sync webhook. A missing/invalid header makes the tool return an error before
|
||||||
|
any mutation.
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
---
|
||||||
|
id: postgres-full-text-search
|
||||||
|
title: PostgreSQL Full-Text Search
|
||||||
|
type: documentation
|
||||||
|
tags:
|
||||||
|
- postgres
|
||||||
|
- fts
|
||||||
|
- tsvector
|
||||||
|
- search
|
||||||
|
status: published
|
||||||
|
author: asep
|
||||||
|
created_at: 2026-08-20
|
||||||
|
updated_at: 2026-08-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# PostgreSQL Full-Text Search
|
||||||
|
|
||||||
|
MCPedia's keyword search is backed by PostgreSQL's native full-text search (FTS), not
|
||||||
|
an external engine. The `documents` table carries a generated `tsvector` column that
|
||||||
|
combines the title (weight `A`) and body (weight `B`).
|
||||||
|
|
||||||
|
## Generated search vector
|
||||||
|
|
||||||
|
The column is `generatedAlwaysAs`, so it is always consistent with the row and needs
|
||||||
|
no trigger:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
searchVector: tsvector("search_vector")
|
||||||
|
.notNull()
|
||||||
|
.generatedAlwaysAs(
|
||||||
|
sql`setweight(to_tsvector('simple', coalesce(${documents.title}, '')), 'A') ||
|
||||||
|
setweight(to_tsvector('simple', coalesce(${documents.body}, '')), 'B')`,
|
||||||
|
),
|
||||||
|
```
|
||||||
|
|
||||||
|
The `'simple'` config disables stemming, so mixed identifier/English queries (e.g.
|
||||||
|
`websocket`, `tsvector`) match literally. A GIN index on `searchVector` keeps lookups
|
||||||
|
fast.
|
||||||
|
|
||||||
|
## Query + ranking
|
||||||
|
|
||||||
|
Search parses the user query with `websearch_to_tsquery` (or `plainto_tsquery`), then
|
||||||
|
ranks with `ts_rank`:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT *, ts_rank(search_vector, q) AS rank
|
||||||
|
FROM documents, websearch_to_tsquery('simple', $1) q
|
||||||
|
WHERE search_vector @@ q
|
||||||
|
ORDER BY rank DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
A headline snippet for the UI comes from `ts_headline`, which bolds the matched lexemes.
|
||||||
|
|
||||||
|
## Hybrid fusion
|
||||||
|
|
||||||
|
Semantic search (embedding cosine) and FTS are fused with **Reciprocal Rank Fusion**
|
||||||
|
(RRF) in `packages/search`. Each result set is ranked, scored `1/(k + rank)`, and the
|
||||||
|
summed scores re-rank the union — no cross-score normalization needed, which is robust
|
||||||
|
when the two signals live on different scales.
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
---
|
||||||
|
id: cloudflare-525-writeup
|
||||||
|
title: Diagnosing Cloudflare 525 (Origin TLS Handshake Failed)
|
||||||
|
type: writeup
|
||||||
|
tags:
|
||||||
|
- cloudflare
|
||||||
|
- tls
|
||||||
|
- 525
|
||||||
|
- caddy
|
||||||
|
- debugging
|
||||||
|
status: published
|
||||||
|
author: asep
|
||||||
|
created_at: 2026-08-20
|
||||||
|
updated_at: 2026-08-20
|
||||||
|
---
|
||||||
|
|
||||||
|
# Diagnosing Cloudflare 525 (Origin TLS Handshake Failed)
|
||||||
|
|
||||||
|
A 525 appears between the user and the origin when Cloudflare (Strict TLS mode) cannot
|
||||||
|
complete the TLS handshake to the origin server. This writeup captures the debugging
|
||||||
|
loop that recurred while wiring new subdomains.
|
||||||
|
|
||||||
|
## Symptom
|
||||||
|
|
||||||
|
`curl https://<sub>.asepharyana.my.id` → `HTTP/2 525`. Browser shows Cloudflare's
|
||||||
|
"SSL handshake failed" page.
|
||||||
|
|
||||||
|
## Root causes (in order of likelihood)
|
||||||
|
|
||||||
|
1. **No Caddy site block for the SNI.** Cloudflare proxies every `*.asepharyana.my.id`
|
||||||
|
record. A host without a matching Caddy `site` block has no certificate, so the
|
||||||
|
handshake dies. This is the #1 cause and the one that bit `wiki` and `mcp`.
|
||||||
|
2. **Certificate still provisioning.** The first request after adding a block triggers
|
||||||
|
ACME `http-01` issuance. Until the cert lands (~10s), the origin 525s. Self-heals.
|
||||||
|
3. **Wrong cert presented.** Rare here — Caddy serves the SNI-matched cert; a mismatch
|
||||||
|
means the block points at the wrong backend or the cert store is stale.
|
||||||
|
|
||||||
|
## The debugging loop
|
||||||
|
|
||||||
|
```
|
||||||
|
curl -sI https://<sub>/ # 525?
|
||||||
|
grep -n "<sub>" /etc/caddy/Caddyfile # block present?
|
||||||
|
sudo journalctl -u caddy | grep -i "tls\|acme\|<sub>" # cert issued?
|
||||||
|
openssl s_client -connect 127.0.0.1:443 -servername <sub> # origin cert valid?
|
||||||
|
```
|
||||||
|
|
||||||
|
If the block is missing: add `import proxy <port>`, `caddy validate`, `systemctl
|
||||||
|
reload caddy`. If the cert is mid-issuance: wait and re-test. Do **not** point Cloudflare
|
||||||
|
at a non-existent origin or set the SSL mode to Flexible — Flexible mode breaks already-
|
||||||
|
working Strict setups.
|
||||||
|
|
||||||
|
## Lesson
|
||||||
|
|
||||||
|
Every new subdomain needs (a) a Caddy site block and (b) a Cloudflare DNS record that
|
||||||
|
proxies to the origin. Omit either and you get a 525. The wildcard DNS means you only
|
||||||
|
add the Caddy side.
|
||||||
Reference in New Issue
Block a user