mcpedia/PHASES.md

MCPedia — Phase Status

Legend: ✅ built · 🟡 partial · ⬜ deferred

Phase 1 — MVP (✅ DONE)

Capability	Status	Notes
Monorepo (bun + Turbo)	✅	apps/{web,mcp}, packages/{types,config,db,parser,search,core}, scripts
Content as Markdown	✅	content/{docs,writeups,research,notes}/, Git-tracked
Frontmatter parsing	✅	@mcpedia/parser (gray-matter)
Postgres metadata	✅	@mcpedia/db Drizzle, documents table
Postgres FTS	✅	weighted tsvector (title A / body B), GIN index, ts_rank+ts_headline
Core services	✅	Document / Content / Search — single business-logic layer
Indexer	✅	scripts/indexer.ts walks content/ → upserts
Web UI (Next 16)	✅	home (list), doc view (SSG), search (dynamic). react-markdown render
MCP server (stdio)	✅	4 tools; in-memory smoke test passing
Hybrid/semantic search	⬜	Phase 2

Phase 2 — Semantic + API

• packages/embeddings — EmbeddingProvider interface + OpenRouter provider (via 9router /v1, encoding_format:"float"); chunkText + embedChunks batcher. EMBED_DIM=2048 discovered live.
• Schema document_chunks (id, document_id→documents.id cascade, slug, chunk_index, content, embedding real[]). Stored as real[] because pgvector is not installed on the shared imrnes Postgres (installing needs host-level apt — deferred). Cosine computed in-app; instant for a KB-sized corpus.
• scripts/indexer.ts — chunks + embeds + upserts (per-doc replace).
• @mcpedia/search — semanticSearch (cosine) + hybridSearch (FTS + cosine, RRF fusion). keywordSearch unchanged.
• apps/api — Hono + tRPC v11 (@trpc/server fetch adapter, @hono/node-server on :4020): search, semanticSearch, hybridSearch, getDocument, listDocuments, related.
• MCP server — added semantic_search + hybrid_search tools (6 total).
• Web search — keyword/hybrid toggle (?mode=hybrid), hybrid reaches semantically-related docs keyword misses.

Phase 3 — Async + Scale ✅ DONE

• Redis + BullMQ background indexing / embedding workers — packages/queue (ioredis singleton + BullMQ Queue/Worker, prefix mcpedia: on shared imrnes Redis :6379); apps/worker runs startWorker(). Three job types: index-doc, index-all, reindex. Single indexing entry point indexContentFile/runFullIndex in @mcpedia/core shared by the script, worker, and git hook. Verified end-to-end against live Redis (job enqueue → worker → Postgres write).
• Git synchronization hook (auto-reindex on push) — API webhook POST /hooks/reindex (full) and POST /hooks/index?slug= (single) enqueue BullMQ jobs. Wire a Git provider (GitHub/Gitea) post-receive / webhook to POST /hooks/reindex to auto-reindex on push. scripts/enqueue.ts is a one-shot enqueue helper (bun run enqueue --all / <slug>).
• Document revision system (document_revisions) — packages/db migration 0002_document_revisions.sql. Indexer snapshots a revision only when the body actually changes vs the latest revision (pure metadata edits don't bloat history). listRevisions / getRevision / restoreRevision in @mcpedia/core; exposed as tRPC revisions / getRevision / restoreRevision and the mcpedia://docs/{+slug}/revisions MCP Resource.
• MCP Resources (mcpedia://docs/...) — alongside the 6 tools: mcpedia://docs (list), mcpedia://docs/{+slug} (body from disk), mcpedia://docs/{+slug}/chunks (chunk preview), mcpedia://docs/{+slug}/revisions (history). {+slug} uses RFC 6570 reserved expansion so slugs containing / match.

New/changed commands

bun run index            # full reindex (runFullIndex, writes revisions)
bun run enqueue --all    # enqueue a full reindex job (no worker needed)
bun run enqueue <slug>   # enqueue a single-doc reindex job
bun run worker           # start the BullMQ indexing worker (long-running)
bun run api              # Hono+tRPC API on :4020 (added /hooks/* webhooks)

Verification done (real, against imrnes Redis + Postgres)

• turbo run typecheck green across all 13 packages.
• BullMQ e2e: enqueue index-doc → worker completes → documents + document_chunks + document_revisions rows present.
• Revision dedup proven: editing a body creates a new revision; metadata-only reindex does not; restoreRevision writes history back into the live row.
• MCP smoke test passes (tools + all 4 resources).
• API webhook POST /hooks/reindex enqueues → worker drains queue → queueStatus reflects counts.

Phase 4 — Operability & Correctness Hardening ✅ DONE

Reinterpreted from the original "Scale-out" plan: OpenSearch/object-storage/ multi-tenant were flagged YAGNI at KB scale (4 docs), so Phase 4 = make the Phase 3 async + revision machinery correct, secure, observable, deployable.

• T1 — restoreRevision rebuilds semantic chunks (CORRECTNESS BUG) — previously restore wrote the old body into documents but left document_chunks on the new body, so semantic/hybrid search went stale after a restore. @mcpedia/core reindexChunks(slug) now re-chunks + re-embeds from the live body; restoreRevision calls it after the update (embed failure is logged, not thrown). Verified: restore → document_chunks count matches re-chunk of the restored body.
• T2 — Secure git-sync webhook (SECURITY) — /hooks/* now require an x-webhook-secret header matching WEBHOOK_SECRET (401 otherwise). API fails fast at startup if WEBHOOK_SECRET is unset (no open endpoint). Added WEBHOOK_SECRET to @mcpedia/config + .env.example; generated a real secret in the local .env (gitignored).
• T3 — Web UI revisions view (UX) — doc page now shows a "History" panel (revision no, reason, date, body length) with a per-revision Restore button. Restore POSTs to apps/web/app/api/revisions/restore/route.ts → restoreRevision → revalidatePath (server-component only, no client JS).
• T4 — Paginate listRevisions — added offset param (summary never includes body). API revisions + MCP resource use the summary.
• T5 — Deploy as supervised services (OPS) — deploy/mcpedia-api.service
  • deploy/mcpedia-worker.service systemd units (Restart=on-failure, EnvironmentFile=.env, WorkingDirectory=/home/code/mcpedia). Enable with: cp deploy/*.service /etc/systemd/system && systemctl daemon-reload && systemctl enable --now mcpedia-api mcpedia-worker. (Not auto-enabled on host without explicit user go-ahead.)

Verification done (real, against imrnes Redis + Postgres)

• turbo run typecheck + turbo run build green (incl. next build with the History panel).
• T1: edit → reindex (new revision + chunks) → restore rev #1 → document_chunks count for that slug matches re-chunk of rev #1; semanticSearch on a term unique to rev #1 returns it.
• T2: curl -XPOST /hooks/reindex → 401; with -H "x-webhook-secret: $WEBHOOK_SECRET" → 200 + jobId; job drains via worker.
• T3: History panel renders; restore route rebuilds chunks (T1 path).
• T4: revisions returns summaries (no body); offset paging works.
• T5: systemd-analyze verify deploy/*.service passes (off-host safe check).

Phase 5 — Deferred scale-out (only when needed)

• Dedicated search engine (OpenSearch/Elasticsearch) — YAGNI until FTS is insufficient
• pgvector migration (install on imrnes Postgres) — when real[] cosine stalls
• Object storage for assets
• Advanced ranking, distributed workers, observability, multi-tenant

Phase 6 — Network deployment + review hardening ✅ DONE

Closed the real gaps found during review: the MCP server was stdio-only (unreachable over the network) and the tRPC API was not routed on the domain (swallowed by web → /trpc/* returned Next.js 404). Also found + fixed a security hole.

• MCP over Streamable HTTP — apps/mcp/src/http.ts serves the 6 tools + 4 resources via MCP 2025-03-26 Streamable HTTP on :4021, stateless mode (sessionIdGenerator: undefined, one server+transport per request, CORS on /mcp). Deployed as mcpedia-mcp.service; reachable at https://mcp.asepharyana.my.id/mcp. Stdio entry (bun run mcp) retained for local subprocess use.
• tRPC API routed on the domain — Caddy wiki.asepharyana.my.id now forwards /trpc/* (+ /hooks/*, /health) to the API on :4020; web stays on :4016. Read-only procedures (search, list, revisions, job status) are public; the restoreRevision mutation is gated by x-webhook-secret (see security fix below).
• Security: lock down restoreRevision — the state-changing tRPC mutation was anonymously callable over the network. Now requires x-webhook-secret (consistent with /hooks auth). The Web UI calls @mcpedia/core directly in a server component, so the gate does not affect the UI's restore button. Verified: no-secret → 401-class rejection, with-secret → reaches handler.
• All four services live + supervised — mcpedia-web (:4016), mcpedia-api (:4020), mcpedia-worker (BullMQ), mcpedia-mcp (:4021) all active, reboot-safe. GitHub push webhook → https://wiki.asepharyana.my.id/hooks/reindex (verified 200, worker drains, 0 failed).

Verification done (real, against live services)

• https://mcp.asepharyana.my.id/mcp initialize → 200 + serverInfo; tools/list → 6; resources/list → 4 (Streamable HTTP SSE framing).
• https://wiki.asepharyana.my.id/ → 200; /health → 200; /trpc/listDocuments → 200.
• GitHub push webhook delivers 200; queueStatus shows completed:N, failed:0.
• turbo run typecheck green across all 4 apps.

Phase 7 — Corpus, MCP write-tools + auth, observability ✅ DONE

Closed the remaining review gaps: tiny corpus (4 docs), MCP read-only (no write/auth), no observability.

• Content corpus grown — added 5 real docs (Caddy reverse proxy, BullMQ workers, MCP Streamable HTTP, Postgres FTS, Cloudflare-525 debugging writeup) across docs/writeups/notes. bun run index reindexed: 9 documents, 17 chunks, 5 new revisions (was 4 docs). Search/semantic/revisions now operate on a real corpus.
• MCP write-tools + auth — added index_document, reindex_all, restore_revision (write, require x-webhook-secret) and queue_status (public). createMcpServer(authSecret?) threads the request header; stdio keeps write tools open (trusted local). Verified: unauthenticated index_document → isError + "unauthorized"; authenticated → enqueues job, worker drains.
• Observability — GET /metrics on the API (Prometheus text exposition: mcpedia_uptime_seconds, mcpedia_queue_jobs{state=...}). Exposed on the domain at https://wiki.asepharyana.my.id/metrics. Public, safe to scrape.

Verification done (real, against live services)

• https://wiki.asepharyana.my.id/metrics → 200 Prometheus text (uptime + queue gauges).
• tools/list over MCP → 10 tools (6 read + 4 new). index_document auth gate works.
• Worker drained the MCP-enqueued job (completed count incremented, failed:0).
• turbo run typecheck green across all 4 apps.

Phase 8 — Dashboard (observability UI) ✅ DONE

The metrics endpoint existed (Phase 7) but had no consumer. Added a zero-dependency dashboard so the KB is actually observable + searchable from a browser.

• GET /dashboard on the API — self-contained HTML (no build, no deps) that:
  • pulls /metrics (same origin) and renders queue gauges (waiting/active/completed/ failed/delayed) + uptime, refreshing every 5s with a live-dot status indicator;
  • runs a live search box that calls the MCP hybrid_search tool directly from the browser (MCP /mcp is CORS-open), returning ranked hits that link to the web doc page (/docs/...).
• XSS hardening: all KB-sourced fields (slug/title/section/error message) are esc()-escaped before innerHTML (defense-in-depth; data is server-trusted).
• Caddy: wiki.asepharyana.my.id/dashboard → :4020.

Verification done (real)

• https://wiki.asepharyana.my.id/dashboard → 200, serves the page (title + JS present).
• /metrics → 200, 7 gauge lines including mcpedia_queue_jobs{state=...}.
• MCP hybrid_search from browser path returns real ranked hits (verified the exact tools/call payload the dashboard issues; shape {doc:{slug,title,section},rank}).
• Dashboard link points to working web doc route /docs/<slug> (verified 200).
• turbo run typecheck green.

Phase 9 — Test coverage + CI gating ✅ DONE

Before Phase 9 the only test was an integration smoke (apps/mcp/src/smoke.test.ts) requiring a live DB; its assertions had also rotted (expected 6 tools, now 10). Added a real bun:test suite that runs green in CI with no external services via in-process module mocking.

• Test infra — turbo.json test task (cache:false); test script on every package/app that has .test.ts files; @types/bun added to root devDeps; tsconfig.base.json registers types: ["bun","node"]; CI step bun run test added after Build.
• @mcpedia/embeddings (5 tests) — chunkText: empty input, single chunk, multi-chunk split, overlap/word-boundary integrity, default options.
• @mcpedia/parser (5 tests) — parseFile: frontmatter extraction, section derivation from top-level dir, invalid type/status fallbacks, missing-field defaults, body excludes delimiter.
• @mcpedia/search (8 tests) — cosine (orthogonal/identical/zero-vector/ mismatched-length/negative) + toTsQuery (AND-prefix, sanitization, empty/garbage).
• @mcpedia/core (4 tests) — shouldCreateRevision dedup truth table (no prior revision → snapshot; identical body → skip; changed body → snapshot; empty vs non-empty). restoreRevision gained an opts.reindex seam for the chunk-rebuild contract.
• apps/api (8 tests) — refactored index.ts → app.ts createApp(deps?) factory (pure construction, injectable QueueLike); dashboard.ts extracted; /health, /metrics, /hooks/reindex (401 w/o secret, 200 w/ secret), /hooks/index (400 w/o slug, 200 w/ slug+secret, 401 wrong secret), /dashboard.
• apps/mcp (6 tests) — write-tool auth gates via InMemoryTransport: index_document/reindex_all/restore_revision error without secret and enqueue with secret (mocked @mcpedia/queue + @mcpedia/core); queue_status public; tool discovery lists all 10 tools regardless of secret (gate is in handler).
• Fixed rot — renamed smoke.test.ts → smoke.ts (so bun test doesn't run the integration smoke as a unit test) and updated stale assertions (10-tool set, 4 docs in docs section).

Verification done (real)

• bun run test → 32 tests green across 6 packages, no DB/Redis (all fakes).
• bun run typecheck → 4 apps green (no test-only type errors).
• bun --cwd apps/mcp run smoke → SMOKE OK (integration, live DB).
• Live API (temp port): /health→200, /metrics→200 gauges, /hooks/reindex→401/200.
• CI workflow now runs bun run test.

Files changed

new:    apps/api/src/app.ts                 # createApp factory
new:    apps/api/src/dashboard.ts           # dashboard HTML module
new:    packages/embeddings/src/chunk.test.ts
new:    packages/parser/src/parse.test.ts
new:    packages/search/src/cosine.test.ts
new:    packages/core/src/index.service.test.ts
new:    apps/api/src/app.test.ts
new:    apps/mcp/src/auth.test.ts
mod:    turbo.json, package.json, tsconfig.base.json, .github/workflows/ci.yml
mod:    apps/api/src/index.ts (thin re-export), apps/api/src/router.ts (unchanged)
renamed: apps/mcp/src/smoke.test.ts -> smoke.ts (fixed stale assertions)

Phase 10 — Audit + Bug fixes + CI/CD deploy (live verification)

Full feature audit against live services. Found + fixed one real bug. Added the missing CI/CD deploy pipeline.

Bug: Frontmatter leaking into rendered doc pages + MCP doc body

• Symptom: Doc pages showed raw YAML frontmatter (id: websocket-contract, title: WebSocket Contract, etc.) as visible plain text between <hr/> markers. MCP mcpedia://docs/{+slug} resource had the same leak.
• Root cause: getDocument() in @mcpedia/core preferred the on-disk file via readFileSync(abs, "utf8") — returning raw file content including the --- frontmatter block. The indexer correctly stripped frontmatter via parseFile (gray-matter), but getDocument bypassed it. ReactMarkdown rendered --- as <hr/> and the YAML as paragraphs.
• Fix: packages/core/src/document.service.ts — replaced readFileSync with parseFile(abs, row.path).body (same frontmatter stripping as the indexer). DB fallback (row.body) unchanged (already clean).
• Verified: 9/9 doc pages render clean (no frontmatter id: text, proper <h2> + <code> elements in SSR HTML); MCP mcpedia:// doc body resource returns clean markdown (starts with # WebSocket Contract).

Audit findings (all phases verified live)

Phase	Feature	Live check	Status
P1	Web UI /docs/<section>/<slug>	200, renders markdown	✅
P1	Search page (?q= + ?mode=hybrid)	200, returns results	✅
P1	MCP stdio + HTTP (/4021)	10 tools, 4 resources	✅
P2	Semantic/hybrid search	returns ranked chunks	✅
P2	tRPC API on domain (/trpc/*)	listDocuments → 4 docs	✅
P3	BullMQ worker drains jobs	queue completed 17→19 after enqueue	✅
P3	Revision system	listRevisions → rev #1 "phase4-final-clean"	✅
P3	Git webhook auth gate	401 w/o secret, 200 w/ secret	✅
P4	Dashboard	/dashboard → 200 HTML	✅
P6	All 4 systemd services	web/api/mcp/worker all active	✅
P6	restoreRevision mutation locked	401 w/o secret, executes w/ secret	✅
P7	10 MCP tools (6 read + 4 write)	tools/list → 10	✅
P7	Write-tool auth gate	reindex_all w/o secret → isError	✅
P7	Prometheus metrics	/metrics → 7 gauges, 200	✅
P8	Dashboard live search	fetch("/metrics") + hybrid_search via /mcp	✅
P9	Test suite	6/6 packages, 32 tests, 0 fail	✅

Notes / non-bugs

• Doc URLs follow /<section>/<slug> (e.g. /docs/caddy/reverse-proxy, /writeups/infra/cloudflare-525, /notes/postgres/full-text-search). The route is [section]/[...slug] — /docs/websocket/contract works because the section IS docs for that doc; /notes/postgres/fts does not (the correct slug is notes/postgres/full-text-search).
• restoreRevision via tRPC needs the x-webhook-secret as an HTTP header (not inside the JSON body) — the fetch adapter reads c.req.raw.headers.

CI/CD deploy pipeline (Phase 10 addition)

Before this audit the CI workflow only built+tested — it did not deploy. The VPS services were configured manually (systemd units in deploy/). Added a deploy.yml workflow per the nix-ci-deploy pattern (CI builds, deploy is separate):

• Trigger: workflow_run on CI completion (only runs if CI passes).
• Build: same as CI (bun install + typecheck + web build) on the GitHub runner — fails fast if the build is broken.
• Deploy: SSHes to the VPS over the public IP (45.127.35.244, not the Tailscale 100.79.111.61 which GitHub runners can't reach), pulls from git, reinstalls deps, rebuilds the web app, and restarts all 4 services via sudo systemctl restart (NOPASSWD already configured for code user).
• Secrets (GitHub repo secrets, not files): SSH_DEPLOY_HOST, SSH_DEPLOY_PORT, SSH_DEPLOY_USER, SSH_DEPLOY_KEY (ed25519 deploy key). Deploy key's public half is in /home/code/.ssh/authorized_keys on the VPS.

Gotchas discovered + fixed during setup

1. SSH host = public IP, not Tailscale IP. 100.79.111.61 is a Tailscale tailscale0 interface IP (CGNAT 100.64.0.0/10); GitHub Actions runners can't route to it. Use the real public IP 45.127.35.244 (port 22 open in iptables).
2. SSH key storage. Storing the key via shell variable (gh secret set --body "$VAR") mangles newlines → ssh.ParsePrivateKey: no key found. Store directly from file: cat keyfile | gh secret set SSH_DEPLOY_KEY --repo ... Use appleboy/ssh-action@v1 (not @v1.1.0) which correctly parses the key. Add -o IdentitiesOnly=yes to prevent "too many authentication failures".
3. Key rotation. Force-pushing amended commits changes the SHA but CI triggers on push: branches: [main] (CI) + workflow_run (deploy) — both fire correctly.

Verification done (real)

• CI workflow_run → Deploy triggers after CI success; all 4 services active.
• All public URLs return 200: web /, /docs/..., /search, /dashboard, /metrics, /trpc/*, mcp.asepharyana.my.id/mcp.
• Doc pages render clean markdown (no frontmatter); History panel + Restore work.

Phase 11 — CRUD + Auth + Web UI ✅ DONE

User requested: "perbagus agar jadi CRUD, pastikan ada autentikasi dan bisa manual dari web atau lewat agent melalui MCP, dan perbaui UI/UXnya."

Backend (Core + API + MCP)

• packages/parser — stringifyFile() — serialize DocumentMeta + body back to a markdown file with YAML frontmatter (gray-matter). Round-trip stable with parseFile.
• @mcpedia/core — CRUD functions:
  • createDocument({slug, title, section, body, type?, status?, author?, tags?}) — writes file to content/{section}/{slug}.md, upserts documents row, snapshots revision, indexes chunks.
  • updateDocument(slug, {...}) — writes file, updates DB row, snapshots revision (if body changed), reindexes chunks.
  • deleteDocument(slug) — removes file + documents/document_chunks/ document_revisions rows.
  • Slug validation: [a-z0-9][a-z0-9/_-]*, no //, no .. traversal.
• apps/api — tRPC CRUD routers — createDocument, updateDocument, deleteDocument (all .use(requireWriteAuth)). Fixed requireWriteAuth to compare against ctx.expectedSecret (injected from deps) instead of the module-level WEBHOOK_SECRET env constant — latent bug that made the middleware untestable without env manipulation.
• apps/mcp — 3 new write tools — create_document, update_document, delete_document (all require x-webhook-secret). Tools: 10 → 13.
• Auth — MCP/API writes reuse the existing WEBHOOK_SECRET / x-webhook-secret pattern. Web CRUD adds cookie-based auth: ADMIN_PASSWORD env + /api/auth/login (HMAC-signed mcpedia_admin cookie, HttpOnly).

Web UI

• /create page — form (section/type/status/title/slug/tags/author/body), POSTs to /api/docs with x-webhook-secret.
• ?edit=1 on doc pages — inline edit form (DocForm component), PUTs to /api/docs/{slug}.
• /login page — password → /api/auth/login → cookie → redirect /create.
• Edit buttons — homepage "+ Create Document" + per-doc "✎" (auth-gated); doc page "Edit" button (auth-gated).
• TOC — doc page auto-generates a table of contents from h2 headings.
• Dark mode — toggle persisted in localStorage, defaults to system.
• /api/docs REST routes — POST (create), PUT (update), DELETE (delete), all x-webhook-secret gated.

Files changed

new: apps/web/app/api/auth/login/route.ts  # cookie-based login + verify
new: apps/web/app/api/docs/route.ts        # GET (list) + POST (create)
new: apps/web/app/api/docs/[...slug]/route.ts  # PUT (update) + DELETE (delete)
new: apps/web/app/components/DocForm.tsx   # shared create/edit form
new: apps/web/app/components/Sidebar.tsx   # client-side doc navigation tree
new: apps/web/app/components/TOC.tsx       # auto-generated TOC (github-slugger)
new: apps/web/app/components/ThemeToggle.tsx  # dark mode toggle
new: apps/web/app/create/page.tsx          # create UI
new: apps/web/app/login/page.tsx           # login UI
new: apps/web/app/docs/page.tsx            # docs index listing
mod: apps/web/app/layout.tsx               # Linear design: sticky header + sidebar + dark canvas
mod: apps/web/app/page.tsx                 # editorial-style homepage w/ section doc listings
mod: apps/web/app/[section]/[...slug]/page.tsx # Linear doc layout (breadcrumb, TOC, metadata)
mod: apps/web/app/search/page.tsx          # dark-themed search w/ result cards
mod: apps/web/app/components/Markdown.tsx  # Linear typography + rehype-slug
mod: apps/web/app/globals.css              # Inter font, Linear dark-mode-first palette
mod: apps/web/app/app/api/docs/route.ts    # dual auth: cookie OR x-webhook-secret
mod: apps/web/app/api/docs/[...slug]/route.ts
mod: packages/core/src/document.service.ts # createDocument/updateDocument/deleteDocument
mod: packages/core/src/index.service.ts    # export snapshotRevision
mod: packages/core/src/index.ts            # re-export CRUD + types
mod: packages/parser/src/index.ts          # stringifyFile
mod: packages/config/src/index.ts          # ADMIN_PASSWORD
mod: apps/api/src/router.ts                # CRUD routers + fix requireWriteAuth
mod: apps/api/src/app.ts                   # createContext passes expectedSecret
mod: apps/api/src/trpc.ts                  # Context.expectedSecret
mod: apps/mcp/src/index.ts                 # 3 new CRUD write tools
mod: apps/mcp/src/auth.test.ts             # +4 CRUD auth tests
mod: apps/api/src/app.test.ts              # +5 tRPC CRUD auth tests
mod: .env.example                          # ADMIN_PASSWORD
new dep: rehype-slug                       # heading anchors for TOC links
new dep: github-slugger                    # matching slug algorithm for client-side TOC

Gotchas / lessons

1. tRPC fetch adapter expects input directly as JSON body, NOT JSON-RPC envelope ({"slug":...} not {"jsonrpc":"2.0","method":...,"params":{...}}).
2. requireWriteAuth env-constant bug — comparing ctx.webhookSecret !== WEBHOOK_SECRET (module-level env constant) is untestable. Fix: thread expectedSecret through Context from createApp(deps).
3. Next.js catch-all routes — [...slug]/edit/ is invalid (catch-all must be last). Used ?edit=1 query param instead. Also, Next.js App Router won't match PUT/DELETE on /api/docs/route.ts for nested paths — need a dynamic segment /api/docs/[...slug]/route.ts.
4. stringifyFile YAML — quote string values with JSON.stringify for special-char safety; arrays use [...] syntax.
5. Next.js SSG + DB — client components ("use client") don't block SSG during next build even if they fetch at runtime. Used for Sidebar (fetches /api/docs at runtime) to avoid ECONNREFUSED on CI.
6. MCP SDK zod-v4 skew — @modelcontextprotocol/sdk@1.30 compiled .d.ts references zod-4 internal types. Pin zod: ^4.0.0 in the MCP app (not workspace-wide) to match the SDK.
7. StreamableHTTP transport headers — use requestInit: { headers: {...} } (not a top-level headers option) on StreamableHTTPClientTransport.
8. SDK type noise — use type-cast helpers (as AnyContent, as CallToolResult) for the SDK's content union types which don't expose .content[0].text cleanly.

Phase 12 — MCP Client + Full Layout Overhaul ✅ DONE

User: "buat mcp untuk client nya" (create the MCP client) + "fokus ke web nya" (the user also said the UI was "boring, only style changed despite requesting a full layout overhaul").

MCP Client (apps/mcp-client/)

New independent bun workspace package that connects to the MCPedia MCP server over Streamable HTTP and provides both a programmatic API + CLI interface.

• src/client.ts — McpediaClient class wrapping the SDK's StreamableHTTPClientTransport + Client. Typed methods for all 13 tools: listDocuments, getDocument, search, semanticSearch, hybridSearch, getRelated, indexDocument, reindexAll, queueStatus, createDocument, updateDocument, deleteDocument, listTools, callTool, listResources, readResource, disconnect. Accepts custom headers (x-webhook-secret for write tools).
• src/index.ts — Interactive REPL (bun run chat): /tools, /resources, /search, /ss, /hybrid, /doc, /related, /create (prompts), /update, /delete, /index, /status, /help, /quit.
• src/ask.ts — One-shot CLI (bun run ask <cmd> [args]) for scripting.
• src/client.test.ts — 7 tests (mock SDK, no network/DB).

Layout Overhaul (Linear design system)

Complete web UI redesign, not just style changes:

• Dark-mode-first — near-black canvas (#08090a), white-opacity borders (rgba(255,255,255,0.05–0.08)), Inter font with cv01/ss03 features.
• Sticky header — MCPedia brand + Docs/Search/Login + theme toggle.
• Sticky sidebar (xl+) — hierarchical doc tree with indented children.
• Editorial homepage — H1 + description, "Create Document" button, section-organized doc listings with tag previews.
• Doc page — breadcrumb nav links, title + metadata bar (author/date/tags), TOC (CONTENTS), clean prose rendering, Related + History sections.
• Create/Edit — /create page, ?edit=1 inline form (DocForm).
• Login — /login with dark-themed form + brand-indigo CTA.
• Search — /search?q= with dark-themed results + snippets.
• Docs index — /docs listing all documents by section.

Gotchas

• Next.js catch-all [...slug]/edit/ is invalid — used ?edit=1 query param.

• Next.js App Router: PUT/DELETE need /api/docs/[...slug]/route.ts.

• tRPC fetch adapter expects JSON body directly (not JSON-RPC envelope).

• requireWriteAuth env-constant bug: use ctx.expectedSecret from deps.

• Next.js SSG + DB: Sidebar as "use client" to avoid DB connection during build.

• MCP SDK zod-v4 skew: pin zod: ^4.0.0 in the MCP app.

• StreamableHTTPClientTransport: use requestInit: { headers } not top-level headers.

• Tooling: bun workspaces + Turborepo (repo already used bun; pnpm rejected to minimize churn).

• DB: imrnes Postgres 100.121.180.82:6432/mcpedia for both dev and deploy; driver prepare:false (PgBouncer). Docker Compose reserved for future prod.

• Phase 1 scope: Core + Web + MCP only. tRPC/Hono API, pgvector, auth, BullMQ deferred (YAGNI).

Phase 13 — CTF Writeup Template System + Dynamic Custom Fields ✅ DONE

User: "pastikan semua dinamis dan rapih untuk banyak situasi jadi tergantung user bukan hardcode" + "jadikan dinamis field nya jangan static begini, jadi yg membuat yg menentukan isinya"

Problem

CTF writeups need per-event organization (event → many challenges). Initial approach hardcoded CTF fields (event, challenge, category, difficulty, points) at the key-name level — if (key === "points") styling, etc. User rejected this: "field ditentukan user, bukan hardcode." Also, tables weren't rendering — two bugs: (1) remark-gfm was missing (tables rendered as pipe text, not HTML), (2) @tailwindcss/typography plugin was not installed in the web app (no CSS for prose classes, so markdown had zero styling).

Table rendering fix

• Installed remark-gfm@4 — enables GFM table/strikethrough/task-list parsing in ReactMarkdown. Tables now render as proper <table>/<thead>/<tbody>.
• Installed @tailwindcss/typography@0.5.20 — @plugin "@tailwindcss/typography" directive in globals.css. Generates prose CSS including table styling.

Dynamic Custom Fields (fully dynamic, user-controlled)

The system is now 100% dynamic — no field names or patterns are hardcoded. The content creator adds any frontmatter key with any value type, and the system auto-discovers + auto-styles:

1. DB layer — documents.extra_fields JSONB column (migration 0003_document_extra_fields.sql). Stores any key-value pairs as native JSONB (numbers, booleans, strings, arrays, objects) — types preserved.

2. Parser (parseFile) — any frontmatter key not in the standard set (title, type, section, status, author, tags, created_at, updated_at) is extracted as an extraField → returned in DocumentMeta.extraFields.

3. Parser (stringifyFile) — writes extraFields back to YAML frontmatter for round-trip stability (parse → stringify → parse yields same result).

4. Core — createDocument/updateDocument accept extraFields?: Record<string, unknown>, merge with existing (update), pass through to DB + file.

5. toMeta (DB → meta) — spreads extra_fields JSONB from DB row into DocumentMeta, making custom fields available to the UI.

6. API routes — splitPayload() separates standard CRUD fields from custom fields. Custom fields passed through with preserved types (no string coercion).

7. DocForm — "+ Add Field" UI lets content creators add ANY key-value pair. Help text is generic (no hardcoded field-name examples).

8. CustomFieldBadges — auto-styles based on VALUE TYPE + VALUE CONTENT, not key name:

   Value type	Badge style	Example
   Number	purple, value shown	5 → purple 5
   Boolean	green (true) / red (no)	true → green Yes
   Array	purple, joined	["a","b"] → purple a, b
   Object	gray, truncated JSON	{timeout:30} → gray {"timeout":30}
   String: difficulty-like	color-coded	easy→green, medium→yellow, hard→red
   String: event-like (contains ctf/def con/hack)	purple	DEF CON CTF Quals 2024
   String: points-like (100 pts)	purple	100 pts
   String: category-like (pwn/web/crypto)	orange	Pwn
   String: status-like (solved/wip/pending)	color-coded	Solved→green
   Any other string	default gray	linux → gray linux

   The same value "medium" gets the same yellow badge whether the key is difficulty, complexity, tier, or level. Content creators control the appearance via values, not by using specific key names.

CTF Writeup Template + Sample

• content/writeups/ctf/template/writeup-template.md — template with: Challenge Info table, Initial Recon, Approach, Step-by-Step Solve, Flag, Summary.
• content/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment.md — sample writeup demonstrating the template with arbitrary frontmatter fields.

Deploy workflow fix

• Added bun run scripts/indexer.ts (explicit path) to deploy workflow instead of bun run index (resolved to wrong package.json in SSH context).
• Removed db:push from deploy (column applied manually via ALTER TABLE; dribble push is interactive/non-blocking). DB migration 0003_document_extra_fields.sql is tracked for future reference.

Verification done (real, against live services)

Endpoints

Path	Status
/	✅ 200
/docs	✅ 200
/docs/mcp/streamable-http	✅ 200
/notes/postgres/full-text-search	✅ 200 (table renders via GFM)
/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment	✅ 200 (dynamic badges + table + TOC)
/writeups/ctf/template/writeup-template	✅ 200
/search	✅ 200
/login, /create, /dashboard	✅ 200

Dynamic badges verified

Created a test doc with fields of ALL value types and verified rendering:

Field	Type	Rendered badge	Color
os	string "linux"	linux	gray (default)
priority	number 5	5	purple (numeric)
resolved	boolean true	Yes	green (boolean)
complexity	string "medium"	medium	yellow (difficulty value match)
team_members	array ["alice","bob"]	alice, bob	purple (array)
config	object {timeout:30,retry:3}	{"retry":3,"timeout":30}	gray (truncated JSON)

Tests

• bun run test → 7 task groups, all pass
• turbo run typecheck → green across all packages
• DB extra_fields column verified: ALTER TABLE documents ADD COLUMN extra_fields jsonb DEFAULT '{}'::jsonb NOT NULL

Files changed

new:    packages/db/drizzle/0003_document_extra_fields.sql  # migration
mod:    packages/types/src/index.ts                        # extraFields: Record<string, unknown>
mod:    packages/parser/src/index.ts                       # parseFile extracts; stringifyFile writes
mod:    packages/core/src/document.service.ts              # accept extraFields
mod:    packages/search/src/index.ts                       # toMeta spreads extra_fields from DB
mod:    apps/web/app/[section]/[...slug]/page.tsx          # CustomFieldBadges value-based styling
mod:    apps/web/app/components/DocForm.tsx                # generic help text (no hardcode)
mod:    apps/web/app/api/docs/route.ts                     # splitPayload preserves types
mod:    apps/web/app/api/docs/[...slug]/route.ts           # splitPayload preserves types
mod:    .github/workflows/deploy.yml                       # explicit indexer path
new:    content/writeups/ctf/template/writeup-template.md # template
new:    content/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment.md # sample

Phase 14 — Hierarchical Folder Structure ✅ DONE

User: "gk ada bedanya, maksud saya inginnya itu bisa yg bertingkat seperti github yg memiliki folder dalam folder" (I want it to be hierarchical like GitHub, with folders inside folders)

Problem

URLs were flat: /<section>/<slug> where slug could contain / but there were no folder index pages — navigating to a folder path (e.g. /writeups/ctf) returned 404. The sidebar showed a flat list with indentation based on slug depth, but no actual folder-node entries or folder navigation.

Solution

1. Section index pages (apps/web/app/[section]/page.tsx) — new generic route for every section. Shows a folder tree (built from doc paths) + a flat list of all docs in that section with "View all (N)" links. Previously only /docs/page.tsx existed; now /writeups, /research, /notes all have index pages.

2. Folder index pages (apps/web/app/[section]/[...slug]/page.tsx) — the doc page now classifies the incoming slug path using classifyPath(docPaths, path):

   • "doc" → leaf document (existing doc page behavior)
   • "folder" → renders FolderIndexPage component listing subfolders 📁 + immediate docs 📄
   • "none" → 404

   This is fully content-driven — no config maps. If a path has child docs, it's a folder. If it matches a doc exactly, it's a leaf.

3. Hierarchical sidebar (apps/web/app/components/Sidebar.tsx) — rebuilds the tree from flat doc slugs. Folder nodes (📁) have collapsible children; leaf docs (📄) link directly. Indentation scales with depth.

4. DocForm v2 (apps/web/app/components/DocForm.tsx) — now has a parent folder dropdown populated from existing folders in the selected section. The slug input is for the leaf name only; the resolved slug (with folder prefix) is displayed below. Create/edit modes handled separately (edit keeps slug read-only).

5. Core helpers (packages/core/src/document.service.ts) — exported extractFoldersForSection(docPaths, section) and classifyPath(docPaths, path) from @mcpedia/core so both web app and future MCP tools can use them.

6. Example hierarchy — reorganized the CTF writeup into proper nested folders:

writeups/
  ctf/
    _index.md              ← folder intro
    defcon-quals-2024/
      _index.md            ← event intro
      pwn/
        pwn-100-ret2win-alignment.md  ← the actual writeup
    template/
      writeup-template.md

Verification done (real, against live services)

Path	Status	Type
/	✅ 200	home
/docs	✅ 200	section index
/docs/caddy/reverse-proxy	✅ 200	doc page (nested slug)
/writeups	✅ 200	section index (tree)
/writeups/ctf	✅ 200	folder index (subfolder: defcon-quals-2024)
/writeups/ctf/defcon-quals-2024	✅ 200	folder index (subfolder: pwn)
/writeups/ctf/defcon-quals-2024/pwn	✅ 200	folder index (docs: pwn-100)
/writeups/ctf/defcon-quals-2024/pwn/pwn-100-ret2win-alignment	✅ 200	doc page + dynamic badges
/writeups/ctf/template/writeup-template	✅ 200	doc page
/writeups/infra/cloudflare-525	✅ 200	flat doc (still works)
/notes/postgres/full-text-search	✅ 200	nested doc (still works)

• bun run test → 7 task groups, all pass
• turbo run typecheck → green across all packages
• next build → success (17 routes registered)
• Folder index pages render 📁 subfolders + 📄 docs with correct hrefs
• Section index pages render 📁 folder tree with indentation
• DocForm parent folder dropdown populated from real folder structure

Files changed

new:    apps/web/app/[section]/page.tsx          # generic section index with tree
mod:    apps/web/app/[section]/[...slug]/page.tsx # folder index detection
mod:    apps/web/app/components/Sidebar.tsx       # hierarchical tree
mod:    apps/web/app/components/DocForm.tsx       # parent folder dropdown
mod:    apps/web/app/create/page.tsx              # pass existingFolders
mod:    packages/core/src/document.service.ts     # extractFoldersForSection + classifyPath
new:    content/writeups/ctf/_index.md            # folder intro
new:    content/writeups/ctf/defcon-quals-2024/_index.md  # event intro
moved:  content/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment.md
     → content/writeups/ctf/defcon-quals-2024/pwn/pwn-100-ret2win-alignment.md