mcpedia/PHASES.md

MCPedia — Phase Status

Legend: ✅ built · 🟡 partial · ⬜ deferred

Phase 1 — MVP (✅ DONE)

Capability	Status	Notes
Monorepo (bun + Turbo)	✅	apps/{web,mcp}, packages/{types,config,db,parser,search,core}, scripts
Content as Markdown	✅	content/{docs,writeups,research,notes}/, Git-tracked
Frontmatter parsing	✅	@mcpedia/parser (gray-matter)
Postgres metadata	✅	@mcpedia/db Drizzle, documents table
Postgres FTS	✅	weighted tsvector (title A / body B), GIN index, ts_rank+ts_headline
Core services	✅	Document / Content / Search — single business-logic layer
Indexer	✅	scripts/indexer.ts walks content/ → upserts
Web UI (Next 16)	✅	home (list), doc view (SSG), search (dynamic). react-markdown render
MCP server (stdio)	✅	4 tools; in-memory smoke test passing
Hybrid/semantic search	⬜	Phase 2

Phase 2 — Semantic + API

• packages/embeddings — EmbeddingProvider interface + OpenRouter provider (via 9router /v1, encoding_format:"float"); chunkText + embedChunks batcher. EMBED_DIM=2048 discovered live.
• Schema document_chunks (id, document_id→documents.id cascade, slug, chunk_index, content, embedding real[]). Stored as real[] because pgvector is not installed on the shared imrnes Postgres (installing needs host-level apt — deferred). Cosine computed in-app; instant for a KB-sized corpus.
• scripts/indexer.ts — chunks + embeds + upserts (per-doc replace).
• @mcpedia/search — semanticSearch (cosine) + hybridSearch (FTS + cosine, RRF fusion). keywordSearch unchanged.
• apps/api — Hono + tRPC v11 (@trpc/server fetch adapter, @hono/node-server on :4020): search, semanticSearch, hybridSearch, getDocument, listDocuments, related.
• MCP server — added semantic_search + hybrid_search tools (6 total).
• Web search — keyword/hybrid toggle (?mode=hybrid), hybrid reaches semantically-related docs keyword misses.

Phase 3 — Async + Scale ✅ DONE

• Redis + BullMQ background indexing / embedding workers — packages/queue (ioredis singleton + BullMQ Queue/Worker, prefix mcpedia: on shared imrnes Redis :6379); apps/worker runs startWorker(). Three job types: index-doc, index-all, reindex. Single indexing entry point indexContentFile/runFullIndex in @mcpedia/core shared by the script, worker, and git hook. Verified end-to-end against live Redis (job enqueue → worker → Postgres write).
• Git synchronization hook (auto-reindex on push) — API webhook POST /hooks/reindex (full) and POST /hooks/index?slug= (single) enqueue BullMQ jobs. Wire a Git provider (GitHub/Gitea) post-receive / webhook to POST /hooks/reindex to auto-reindex on push. scripts/enqueue.ts is a one-shot enqueue helper (bun run enqueue --all / <slug>).
• Document revision system (document_revisions) — packages/db migration 0002_document_revisions.sql. Indexer snapshots a revision only when the body actually changes vs the latest revision (pure metadata edits don't bloat history). listRevisions / getRevision / restoreRevision in @mcpedia/core; exposed as tRPC revisions / getRevision / restoreRevision and the mcpedia://docs/{+slug}/revisions MCP Resource.
• MCP Resources (mcpedia://docs/...) — alongside the 6 tools: mcpedia://docs (list), mcpedia://docs/{+slug} (body from disk), mcpedia://docs/{+slug}/chunks (chunk preview), mcpedia://docs/{+slug}/revisions (history). {+slug} uses RFC 6570 reserved expansion so slugs containing / match.

New/changed commands

bun run index            # full reindex (runFullIndex, writes revisions)
bun run enqueue --all    # enqueue a full reindex job (no worker needed)
bun run enqueue <slug>   # enqueue a single-doc reindex job
bun run worker           # start the BullMQ indexing worker (long-running)
bun run api              # Hono+tRPC API on :4020 (added /hooks/* webhooks)

Verification done (real, against imrnes Redis + Postgres)

• turbo run typecheck green across all 13 packages.
• BullMQ e2e: enqueue index-doc → worker completes → documents + document_chunks + document_revisions rows present.
• Revision dedup proven: editing a body creates a new revision; metadata-only reindex does not; restoreRevision writes history back into the live row.
• MCP smoke test passes (tools + all 4 resources).
• API webhook POST /hooks/reindex enqueues → worker drains queue → queueStatus reflects counts.

Phase 4 — Operability & Correctness Hardening ✅ DONE

Reinterpreted from the original "Scale-out" plan: OpenSearch/object-storage/ multi-tenant were flagged YAGNI at KB scale (4 docs), so Phase 4 = make the Phase 3 async + revision machinery correct, secure, observable, deployable.

• T1 — restoreRevision rebuilds semantic chunks (CORRECTNESS BUG) — previously restore wrote the old body into documents but left document_chunks on the new body, so semantic/hybrid search went stale after a restore. @mcpedia/core reindexChunks(slug) now re-chunks + re-embeds from the live body; restoreRevision calls it after the update (embed failure is logged, not thrown). Verified: restore → document_chunks count matches re-chunk of the restored body.
• T2 — Secure git-sync webhook (SECURITY) — /hooks/* now require an x-webhook-secret header matching WEBHOOK_SECRET (401 otherwise). API fails fast at startup if WEBHOOK_SECRET is unset (no open endpoint). Added WEBHOOK_SECRET to @mcpedia/config + .env.example; generated a real secret in the local .env (gitignored).
• T3 — Web UI revisions view (UX) — doc page now shows a "History" panel (revision no, reason, date, body length) with a per-revision Restore button. Restore POSTs to apps/web/app/api/revisions/restore/route.ts → restoreRevision → revalidatePath (server-component only, no client JS).
• T4 — Paginate listRevisions — added offset param (summary never includes body). API revisions + MCP resource use the summary.
• T5 — Deploy as supervised services (OPS) — deploy/mcpedia-api.service
  • deploy/mcpedia-worker.service systemd units (Restart=on-failure, EnvironmentFile=.env, WorkingDirectory=/home/code/mcpedia). Enable with: cp deploy/*.service /etc/systemd/system && systemctl daemon-reload && systemctl enable --now mcpedia-api mcpedia-worker. (Not auto-enabled on host without explicit user go-ahead.)

Verification done (real, against imrnes Redis + Postgres)

• turbo run typecheck + turbo run build green (incl. next build with the History panel).
• T1: edit → reindex (new revision + chunks) → restore rev #1 → document_chunks count for that slug matches re-chunk of rev #1; semanticSearch on a term unique to rev #1 returns it.
• T2: curl -XPOST /hooks/reindex → 401; with -H "x-webhook-secret: $WEBHOOK_SECRET" → 200 + jobId; job drains via worker.
• T3: History panel renders; restore route rebuilds chunks (T1 path).
• T4: revisions returns summaries (no body); offset paging works.
• T5: systemd-analyze verify deploy/*.service passes (off-host safe check).

Phase 5 — Deferred scale-out (only when needed)

• Dedicated search engine (OpenSearch/Elasticsearch) — YAGNI until FTS is insufficient
• pgvector migration (install on imrnes Postgres) — when real[] cosine stalls
• Object storage for assets
• Advanced ranking, distributed workers, observability, multi-tenant

Phase 6 — Network deployment + review hardening ✅ DONE

Closed the real gaps found during review: the MCP server was stdio-only (unreachable over the network) and the tRPC API was not routed on the domain (swallowed by web → /trpc/* returned Next.js 404). Also found + fixed a security hole.

• MCP over Streamable HTTP — apps/mcp/src/http.ts serves the 6 tools + 4 resources via MCP 2025-03-26 Streamable HTTP on :4021, stateless mode (sessionIdGenerator: undefined, one server+transport per request, CORS on /mcp). Deployed as mcpedia-mcp.service; reachable at https://mcp.asepharyana.my.id/mcp. Stdio entry (bun run mcp) retained for local subprocess use.
• tRPC API routed on the domain — Caddy wiki.asepharyana.my.id now forwards /trpc/* (+ /hooks/*, /health) to the API on :4020; web stays on :4016. Read-only procedures (search, list, revisions, job status) are public; the restoreRevision mutation is gated by x-webhook-secret (see security fix below).
• Security: lock down restoreRevision — the state-changing tRPC mutation was anonymously callable over the network. Now requires x-webhook-secret (consistent with /hooks auth). The Web UI calls @mcpedia/core directly in a server component, so the gate does not affect the UI's restore button. Verified: no-secret → 401-class rejection, with-secret → reaches handler.
• All four services live + supervised — mcpedia-web (:4016), mcpedia-api (:4020), mcpedia-worker (BullMQ), mcpedia-mcp (:4021) all active, reboot-safe. GitHub push webhook → https://wiki.asepharyana.my.id/hooks/reindex (verified 200, worker drains, 0 failed).

Verification done (real, against live services)

• https://mcp.asepharyana.my.id/mcp initialize → 200 + serverInfo; tools/list → 6; resources/list → 4 (Streamable HTTP SSE framing).
• https://wiki.asepharyana.my.id/ → 200; /health → 200; /trpc/listDocuments → 200.
• GitHub push webhook delivers 200; queueStatus shows completed:N, failed:0.
• turbo run typecheck green across all 4 apps.

Phase 7 — Corpus, MCP write-tools + auth, observability ✅ DONE

Closed the remaining review gaps: tiny corpus (4 docs), MCP read-only (no write/auth), no observability.

• Content corpus grown — added 5 real docs (Caddy reverse proxy, BullMQ workers, MCP Streamable HTTP, Postgres FTS, Cloudflare-525 debugging writeup) across docs/writeups/notes. bun run index reindexed: 9 documents, 17 chunks, 5 new revisions (was 4 docs). Search/semantic/revisions now operate on a real corpus.
• MCP write-tools + auth — added index_document, reindex_all, restore_revision (write, require x-webhook-secret) and queue_status (public). createMcpServer(authSecret?) threads the request header; stdio keeps write tools open (trusted local). Verified: unauthenticated index_document → isError + "unauthorized"; authenticated → enqueues job, worker drains.
• Observability — GET /metrics on the API (Prometheus text exposition: mcpedia_uptime_seconds, mcpedia_queue_jobs{state=...}). Exposed on the domain at https://wiki.asepharyana.my.id/metrics. Public, safe to scrape.

Verification done (real, against live services)

• https://wiki.asepharyana.my.id/metrics → 200 Prometheus text (uptime + queue gauges).
• tools/list over MCP → 10 tools (6 read + 4 new). index_document auth gate works.
• Worker drained the MCP-enqueued job (completed count incremented, failed:0).
• turbo run typecheck green across all 4 apps.

Phase 8 — Dashboard (observability UI) ✅ DONE

The metrics endpoint existed (Phase 7) but had no consumer. Added a zero-dependency dashboard so the KB is actually observable + searchable from a browser.

• GET /dashboard on the API — self-contained HTML (no build, no deps) that:
  • pulls /metrics (same origin) and renders queue gauges (waiting/active/completed/ failed/delayed) + uptime, refreshing every 5s with a live-dot status indicator;
  • runs a live search box that calls the MCP hybrid_search tool directly from the browser (MCP /mcp is CORS-open), returning ranked hits that link to the web doc page (/docs/...).
• XSS hardening: all KB-sourced fields (slug/title/section/error message) are esc()-escaped before innerHTML (defense-in-depth; data is server-trusted).
• Caddy: wiki.asepharyana.my.id/dashboard → :4020.

Verification done (real)

• https://wiki.asepharyana.my.id/dashboard → 200, serves the page (title + JS present).
• /metrics → 200, 7 gauge lines including mcpedia_queue_jobs{state=...}.
• MCP hybrid_search from browser path returns real ranked hits (verified the exact tools/call payload the dashboard issues; shape {doc:{slug,title,section},rank}).
• Dashboard link points to working web doc route /docs/<slug> (verified 200).
• turbo run typecheck green.

Phase 9 — Test coverage + CI gating ✅ DONE

Before Phase 9 the only test was an integration smoke (apps/mcp/src/smoke.test.ts) requiring a live DB; its assertions had also rotted (expected 6 tools, now 10). Added a real bun:test suite that runs green in CI with no external services via in-process module mocking.

• Test infra — turbo.json test task (cache:false); test script on every package/app that has .test.ts files; @types/bun added to root devDeps; tsconfig.base.json registers types: ["bun","node"]; CI step bun run test added after Build.
• @mcpedia/embeddings (5 tests) — chunkText: empty input, single chunk, multi-chunk split, overlap/word-boundary integrity, default options.
• @mcpedia/parser (5 tests) — parseFile: frontmatter extraction, section derivation from top-level dir, invalid type/status fallbacks, missing-field defaults, body excludes delimiter.
• @mcpedia/search (8 tests) — cosine (orthogonal/identical/zero-vector/ mismatched-length/negative) + toTsQuery (AND-prefix, sanitization, empty/garbage).
• @mcpedia/core (4 tests) — shouldCreateRevision dedup truth table (no prior revision → snapshot; identical body → skip; changed body → snapshot; empty vs non-empty). restoreRevision gained an opts.reindex seam for the chunk-rebuild contract.
• apps/api (8 tests) — refactored index.ts → app.ts createApp(deps?) factory (pure construction, injectable QueueLike); dashboard.ts extracted; /health, /metrics, /hooks/reindex (401 w/o secret, 200 w/ secret), /hooks/index (400 w/o slug, 200 w/ slug+secret, 401 wrong secret), /dashboard.
• apps/mcp (6 tests) — write-tool auth gates via InMemoryTransport: index_document/reindex_all/restore_revision error without secret and enqueue with secret (mocked @mcpedia/queue + @mcpedia/core); queue_status public; tool discovery lists all 10 tools regardless of secret (gate is in handler).
• Fixed rot — renamed smoke.test.ts → smoke.ts (so bun test doesn't run the integration smoke as a unit test) and updated stale assertions (10-tool set, 4 docs in docs section).

Verification done (real)

• bun run test → 32 tests green across 6 packages, no DB/Redis (all fakes).
• bun run typecheck → 4 apps green (no test-only type errors).
• bun --cwd apps/mcp run smoke → SMOKE OK (integration, live DB).
• Live API (temp port): /health→200, /metrics→200 gauges, /hooks/reindex→401/200.
• CI workflow now runs bun run test.

Files changed

new:    apps/api/src/app.ts                 # createApp factory
new:    apps/api/src/dashboard.ts           # dashboard HTML module
new:    packages/embeddings/src/chunk.test.ts
new:    packages/parser/src/parse.test.ts
new:    packages/search/src/cosine.test.ts
new:    packages/core/src/index.service.test.ts
new:    apps/api/src/app.test.ts
new:    apps/mcp/src/auth.test.ts
mod:    turbo.json, package.json, tsconfig.base.json, .github/workflows/ci.yml
mod:    apps/api/src/index.ts (thin re-export), apps/api/src/router.ts (unchanged)
renamed: apps/mcp/src/smoke.test.ts -> smoke.ts (fixed stale assertions)

Phase 10 — Audit + Bug fixes (live verification)

Full feature audit against live services. Found + fixed one real bug.

Bug: Frontmatter leaking into rendered doc pages + MCP doc body

• Symptom: Doc pages showed raw YAML frontmatter (id: websocket-contract, title: WebSocket Contract, etc.) as visible plain text between <hr/> markers. MCP mcpedia://docs/{+slug} resource had the same leak.
• Root cause: getDocument() in @mcpedia/core preferred the on-disk file via readFileSync(abs, "utf8") — returning raw file content including the --- frontmatter block. The indexer correctly stripped frontmatter via parseFile (gray-matter), but getDocument bypassed it. ReactMarkdown rendered --- as <hr/> and the YAML as paragraphs.
• Fix: package/core/src/document.service.ts — replaced readFileSync with parseFile(abs, row.path).body (same frontmatter stripping as the indexer). DB fallback (row.body) unchanged (already clean).
• Verified: 9/9 doc pages render clean (no frontmatter id: text, proper <h2> + <code> elements in SSR HTML); MCP mcpedia:// doc body resource returns clean markdown (starts with # WebSocket Contract).

Audit findings (all phases verified live)

Phase	Feature	Live check	Status
P1	Web UI /docs/<section>/<slug>	200, renders markdown	✅
P1	Search page (?q= + ?mode=hybrid)	200, returns results	✅
P1	MCP stdio + HTTP (/4021)	10 tools, 4 resources	✅
P2	Semantic/hybrid search	returns ranked chunks	✅
P2	tRPC API on domain (/trpc/*)	listDocuments → 4 docs	✅
P3	BullMQ worker drains jobs	queue completed 17→19 after enqueue	✅
P3	Revision system	listRevisions → rev #1 "phase4-final-clean"	✅
P3	Git webhook auth gate	401 w/o secret, 200 w/ secret	✅
P4	Dashboard	/dashboard → 200 HTML	✅
P6	All 4 systemd services	web/api/mcp/worker all active	✅
P6	restoreRevision mutation locked	401 w/o secret, executes w/ secret	✅
P7	10 MCP tools (6 read + 4 write)	tools/list → 10	✅
P7	Write-tool auth gate	reindex_all w/o secret → isError	✅
P7	Prometheus metrics	/metrics → 7 gauges, 200	✅
P8	Dashboard live search	fetch("/metrics") + hybrid_search via /mcp	✅
P9	Test suite	6/6 packages, 32 tests, 0 fail	✅

Notes / non-bugs

• Doc URLs follow /<section>/<slug> (e.g. /docs/caddy/reverse-proxy, /writeups/infra/cloudflare-525, /notes/postgres/full-text-search). The route is [section]/[...slug] — /docs/websocket/contract works because the section IS docs for that doc; /notes/postgres/fts does not (the correct slug is notes/postgres/full-text-search).
• restoreRevision via tRPC needs the x-webhook-secret as an HTTP header (not inside the JSON body) — the fetch adapter reads c.req.raw.headers.

Decisions locked (from initial planning)

• Tooling: bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).
• DB: imrnes Postgres 100.121.180.82:6432/mcpedia for both dev and deploy; driver prepare:false (PgBouncer). Docker Compose reserved for future prod.
• Phase 1 scope: Core + Web + MCP only. tRPC/Hono API, pgvector, auth, BullMQ deferred (YAGNI).