feat: revamp to dynamic database-first architecture and cleanup phase docs
CI / typecheck + build (turbo) (push) Canceled after 0s

- Migrate document fetching and CRUD to be PostgreSQL-authoritative
- Remove static section enums and add dynamic listSections query
- Support custom sections and metadata across API, MCP, and Web UI
- Add /api/sections endpoint and update Header, Sidebar, and Forms
- Remove obsolete phase planning docs and modernize README/AGENTS
This commit is contained in:
asepharyana
2026-08-21 11:35:42 +07:00
parent a937e8f51b
commit a9b67385c3
37 changed files with 720 additions and 2515 deletions
File diff suppressed because one or more lines are too long
@@ -1,161 +0,0 @@
# MCPedia Phase 2 — Semantic Search + tRPC/Hono API
> **For Hermes:** implement task-by-task. Spec-first (user rule 2026-08-19).
**Goal:** Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so
MCPedia is queryable by embeddings, not just keyword FTS — and expose the
corpus over a programmatic HTTP API.
**Architecture:** Content (Markdown) → chunk → embed (OpenRouter) → store
`document_chunks` with `vector(N)` in Postgres → `semanticSearch` (cosine) and
`hybridSearch` (FTS + cosine, reciprocal-rank fusion) in `@mcpedia/search` →
exposed via Core, the MCP server (new tools), and a new `apps/api` (Hono +
tRPC v11).
**Embedding provider:** OpenRouter (`openrouter/llama-nemotron-embed-vl-1b-v2:free`)
via `9router_ai_llm_api_key` + `9router_ai_llm_base_url` (BWS). Dimension is
discovered at first live call (see Step 1.3) and pinned in schema/migration.
**Tech stack:** drizzle-orm `vector` column + pgvector extension, HNSW index,
`@trpc/server` v11 (fetch adapter), `hono` + `@hono/node-server`.
---
## Task P2.1 — `packages/embeddings` (provider + abstraction)
**Files:** `packages/embeddings/package.json`, `src/index.ts`, `src/provider.ts`,
`src/openrouter.ts`
- `EmbeddingProvider` interface: `embed(texts: string[]): Promise<number[][]>`, `readonly model`, `readonly dimensions`.
- `OpenRouterEmbeddingProvider`: POST `${baseUrl}/embeddings` with `{ model, input }`,
`Authorization: Bearer ${key}`. Returns `data[].embedding`. Validate length === dimensions.
- Read `EMBED_BASE_URL`, `EMBED_API_KEY`, `EMBED_MODEL` from `@mcpedia/config`
(with `.env` fallback). Dimensions discovered live (Step 1.3) → export `EMBED_DIM`.
- Chunk helper `chunkText(text, { size=1000, overlap=150 })` in `src/chunk.ts`.
**Step 1.3 (discover dim):** live call `embed(["test"])`, read `embedding.length`,
pin `EMBED_DIM`, assert mismatch throws.
**Verify:** `bun run` a temp script: `embed(["hello world"])` prints a vector of
length N (e.g. 1024). Confirm no key is logged.
---
## Task P2.2 — Schema: `document_chunks` + vector extension
**Files:** `packages/db/src/schema.ts` (add), `packages/db/drizzle.config.ts`
(unchanged), new migration.
- `CREATE EXTENSION IF NOT EXISTS vector;` (idempotent; run once via psql).
- `document_chunks` table:
- `id` uuid pk default gen_random_uuid()
- `document_id` text → `documents.id` on delete cascade
- `slug` text (denormalized for convenience)
- `chunk_index` integer
- `content` text
- `embedding` vector(EMBED_DIM)
- `created_at` timestamp default now()
- index `chunk_embedding_idx` using hnsw (`embedding` op `vector_cosine_ops`)
- Generate migration with `drizzle-kit generate`, apply via `psql` (drizzle-kit
push is unreliable here — known).
**Verify:** `\d document_chunks` shows `embedding vector(N)` + HNSW index;
`select count(*) from document_chunks` = 0.
---
## Task P2.3 — Indexer: chunk + embed + upsert
**Files:** `scripts/indexer.ts` (extend), `packages/core/src/document.service.ts`
(add `indexChunks`).
- For each published doc: read body (already on disk), `chunkText`, `embed` in
batches (≤ 16), delete existing chunks for slug, insert new rows.
- Guard: if embedding provider fails, log + skip (don't crash the whole index).
- Add `bun run index:embed` (or extend `bun run index` to also embed).
**Verify:** after running, `select count(*) from document_chunks` > 0; a sample
row has non-null `embedding`.
---
## Task P2.4 — `packages/search`: semantic + hybrid
**Files:** `packages/search/src/index.ts` (add `semanticSearch`, `hybridSearch`).
- `semanticSearch(vec, limit)`: order by `embedding <=> ${vec}` asc, filter published.
- `hybridSearch(q, limit)`: run FTS (`ts_rank`) + semantic (cosine) in parallel;
fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
- Keep `keywordSearch` unchanged (Phase 1).
**Verify:** unit-ish script: embed a query, `semanticSearch` returns relevant
chunks; `hybridSearch("websocket")` returns ≥ keyword results.
---
## Task P2.5 — `packages/core` expose semantic/hybrid
**Files:** `packages/core/src/search.service.ts`, `index.ts`.
- Re-export `semanticSearch`, `hybridSearch` from Core.
---
## Task P2.6 — `apps/api` (Hono + tRPC v11)
**Files:** `apps/api/package.json`, `tsconfig.json`, `src/index.ts`,
`src/router.ts`, `src/trpc.ts`.
- `initTRPC.create()` router with procedures: `search`, `semanticSearch`,
`hybridSearch`, `getDocument`, `listDocuments` (mirrors MCP tools).
- Mount `fetchRequestHandler` on a Hono app at `/trpc/*`; serve via
`@hono/node-server` `serve({ fetch: app.fetch, port: 4020 })`.
- `createContext` returns `{ db }`.
**Verify:** `bun run dev` → `curl -X POST localhost:4020/trpc/search`
with JSON body returns hits.
---
## Task P2.7 — MCP server: semantic + hybrid tools
**Files:** `apps/mcp/src/index.ts` (add `semantic_search`, `hybrid_search`),
extend `smoke.test.ts`.
- `semantic_search`: embed query → `semanticSearch`.
- `hybrid_search`: embed query → `hybridSearch`.
- Smoke: assert both return ≥1 hit for "websocket".
---
## Task P2.8 — Web: semantic toggle on search
**Files:** `apps/web/app/search/page.tsx`.
- Add `mode=keyword|hybrid` query param; server component calls Core
`hybridSearch` when `mode=hybrid`. Minimal UI toggle (link/buttons).
- Keep keyword as default.
**Verify:** `bun run build`; `curl '/search?q=websocket&mode=hybrid'` returns hits.
---
## Task P2.9 — Verify all + commit
- `bunx turbo run build` (web + api + mcp), `bun run apps/mcp smoke`,
live API curl, live web hybrid search.
- Update `README.md` + `PHASES.md` (mark Phase 2 ✅).
- `git add -A` (exclude `.env`), commit as asepharyana (no Co-Authored-By).
---
## Risks / decisions
- **Dimension unknown until live call** → P2.1.3 discovers it; pinned EMBED_DIM=2048.
- **pgvector NOT available on shared imrnes Postgres** (extension not installed;
installing needs host-level apt on a managed/shared DB — deferred). PIVOT:
store `embedding` as `real[]` and compute cosine similarity in the app layer.
Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds
of chunks). pgvector+HNSW is the Phase-4 scale-out path.
- **PgBouncer + real[]**: fine; simple queries, no extension needed.
- **API port 4020** (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
- **YAGNI**: no auth/revisions this phase (Phase 3).
-151
View File
@@ -1,151 +0,0 @@
# Phase 14 — Hierarchical Folder Structure
> User: "gk ada bedanya, maksud saya inginnya itu bisa yg bertingkat seperti github yg memiliki folder dalam folder"
> Context: after the full dynamic-custom-fields overhaul (Phase 13), the user
> wants document URLs/content organized in **nested folders** like GitHub —
> `writeups/ctf/defcon-quals-2024/pwn-100/...` with subfolders under subfolders,
> not just one level deep.
## Problem
The current URL scheme is `/<section>/<slug>` where `slug` can contain `/`
(e.g. `writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` →
`/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment`). This works for
**files** but there are no **folder-level index pages** — navigating to
`/writeups/ctf/defcon-quals-2024/` returns 404 because Next.js catch-all
`[section]/[...slug]/page.tsx` requires at least one slug segment beyond the
section, and the sidebar only shows flat doc titles (no folder tree).
GitHub's model: `github.com/org/repo/tree/main/path/to/folder/file` — every
folder has an index page (`/path/to/folder/`) listing its contents.
## Solution
### 1. Folder Index Pages
**Create `apps/web/app/[section]/[...slug]/folder.tsx`** (or a parallel route).
Actually — cleaner approach per Next.js App Router: the catch-all
`[section]/[...slug]/page.tsx` handles both. Add logic: if the slug resolves to
an actual markdown file → doc page (existing behavior). If the slug resolves to
a **directory** (folder of docs) → render a folder index listing all docs whose
`path` starts with that prefix.
**Mechanism:**
- Call `listDocuments()` to get all docs.
- The incoming URL path is `{section}/{...slug}`.
- Build the "folder prefix" = `${section}/${slug.join("/")}/` (with trailing `/`,
or just `${section}/${slug.join("/")}` if no slug segments).
- Filter docs whose `doc.path` starts with that prefix.
- If exactly one doc matches AND its path === prefix (trimmed .md) → it's a
doc page (existing). If zero or multiple match and they all start with the
prefix → it's a folder index.
- Edge: a folder with exactly one doc whose path matches exactly — still a doc
page. A folder is when there are docs at `prefix/sub/...`.
**Better heuristic:** A slug path is a "folder" if there exist docs whose `path`
is `prefix/deep/...` (i.e., the slug is a parent of other doc paths, not a
leaf itself). A slug is a "leaf doc" if `path === prefix + ".md"`.
### 2. Sidebar Tree
**Update `Sidebar.tsx`:**
- `listDocuments()` already returns all docs with their full `slug` and `path`.
- Build a **tree** from the flat list: split each slug by `/`, create nested
folder nodes.
- Render nested `<ul>` with indentation (already done via `marginLeft` based on
depth).
- **Folder nodes** (collapsed/expanded) get a folder icon 📁 and a CSS class.
- Clicking a folder → navigates to the folder index page `/{section}/{path}`.
- **Leaf doc nodes** → link to `/{doc.slug}` (existing behavior).
- Group by the first segment after section too (e.g. `ctf/defcon-quals-2024/`
is a folder, then `pwn-100-...` are children).
Tree-building algorithm (from flat slugs):
```
For slug "writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment":
parts = ["writeups", "ctf", "defcon-quals-2024", "pwn-100-ret2win-alignment"]
→ tree: writeups → ctf → defcon-quals-2024 → pwn-100-ret2win-alignment (leaf)
```
### 3. DocForm / Folder Selection
**Update `DocForm.tsx`:**
- Add a "Parent folder" input (autocomplete or text) that shows existing folders
for the selected section. The slug field already supports `/` but the user
experience is better with folder picker.
- When creating, the slug becomes `{parentFolder}/{slug}` automatically.
- Show existing folder structure as `<select>` or tree picker.
### 4. Example Hierarchy
Create a real hierarchical structure to demonstrate:
```
writeups/
ctf/
defcon-quals-2024/
pwn/
pwn-100-ret2win-alignment.md
pwn-200-bof-heap.md
crypto/
crypto-100-xor.md
crypto-200-rsa.md
template/
writeup-template.md
_index.md ← folder index (optional intro)
hackthebox/
machine-name/
walkthrough.md
```
For now, reorganize the existing Defcon writeup into proper subfolders + add a
folder index page. The existing `content/writeups/ctf/defcon-quals-2024/` is
already a folder — just need the folder index route to work.
### 5. Route Changes
**Current:** `[section]/[...slug]/page.tsx` — catch-all requires ≥1 slug segment.
- `/writeups` → `notFound()` (no index for bare section unless we add one)
- `/writeups/ctf` → catch-all gets `slug=["ctf"]` → currently treated as a doc
(looks up `writeups/ctf` doc, 404 if none)
- `/writeups/ctf/defcon-quals-2024/` → catch-all `slug=["ctf","defcon-quals-2024"]`
**Plan:**
1. Add `/writeups/page.tsx` (section index) — lists top-level folders + root
docs in that section. (Currently `/docs/page.tsx` exists but `/writeups/page.tsx`
doesn't.)
2. In `[section]/[...slug]/page.tsx`: at the top of the page component, check if
the slug path is a folder (has child docs). If so, render folder index instead
of doc page.
**Section index pages:** Create `[section]/page.tsx` for all 4 sections, or a
generic one. Currently only `/docs/page.tsx` exists. Add a shared
`SectionIndex` component.
### 6. Files to Change
```
new: apps/web/app/[section]/page.tsx # generic section index (folder + doc listing)
mod: apps/web/app/[section]/[...slug]/page.tsx # add folder-index detection
mod: apps/web/app/components/Sidebar.tsx # tree from flat slugs
mod: apps/web/app/components/DocForm.tsx # parent folder picker
new: content/writeups/ctf/defcon-quals-2024/_index.md # folder intro (optional)
new: content/writeups/ctf/_index.md # CTF section intro
mod: apps/web/app/docs/page.tsx # may need generic version
```
### 7. Verification
- `/writeups` → 200, shows folders: ctf/, template/
- `/writeups/ctf` → 200, folder index listing defcon-quals-2024/
- `/writeups/ctf/defcon-quals-2024` → 200, folder index listing pwn-100-...
- `/writeups/ctf/defcon-quals-2024/pwn-100-ret2win-alignment` → 200, doc page
- Sidebar shows nested tree with folder icons
- DocForm parent folder picker works
- `bun run test` green, `turbo run typecheck` green
- Live verification via curl
## Constraints
- User: "pastikan semua dinamis dan rapih untuk banyak situasi jadi tergantung
user bukan hardcode" — folder detection must be content-driven, not config.
- No breaking existing flat slugs.
- Section icons/labels stay the same.
-42
View File
@@ -1,42 +0,0 @@
# MCP HTTP transport + deploy + review actions
## Goal
Make the MCPedia MCP server reachable over the network (not just stdio subprocess), so
remote MCP clients (Claude, a Discord bot, a web client) can call its 6 tools + 4 resources.
Serve via Streamable HTTP (MCP 2025-03-26 spec), deploy as a supervised systemd service,
expose through Caddy on a dedicated subdomain.
## Design decisions
- **StreamableHTTPServerTransport, stateless mode** (`sessionIdGenerator: undefined`).
One McpServer + transport per request. No session map, no shared-transport connect race,
no memory leak. Re-registering 6 tools + 4 resources per request is negligible for a KB.
- **New entry `apps/mcp/src/http.ts`** served by Node `http` (built-in), NOT mounted on the
API app — keeps the MCP app's zod-4 isolation intact (api app is zod 3).
- **Port 4021** (next free in the 4000s range; 4020 is the API).
- **Subdomain `mcp.asepharyana.my.id`** -> 4021 (Cloudflare `*` wildcard already proxies it;
Caddy auto-issues LE cert, no extra DNS work).
- **CORS** allow on `/mcp` (remote web clients need it).
- MCP tools are read-only (search/get/list/related + read-only resources) => open MCP is
low-risk. No auth on MCP itself.
## Files
- `apps/mcp/src/http.ts` (NEW) — Node http server, `/mcp` route, stateless transport.
- `apps/mcp/package.json` — add `serve:http` script.
- `package.json` (root) — add `mcp:http` script (absolute bun path).
- `deploy/mcpedia-mcp.service` (NEW) — systemd unit, MCP_PORT=4021.
- `/etc/caddy/Caddyfile` — add `mcp.asepharyana.my.id { import proxy 4021 }`.
## Security finding (review, flagged not silently built)
The tRPC `restoreRevision` mutation is exposed UNauthenticated at
`https://wiki.asepharyana.my.id/trpc/restoreRevision` — anyone can revert a live doc.
The web UI's restore path calls `@mcpedia/core` directly (server component), so the tRPC
mutation is dead surface. Fix: guard the mutation with the existing WEBHOOK_SECRET header,
or drop it from the router. Will apply the guard (consistent with /hooks auth) unless user
prefers removal.
## Verification
- `bun --cwd apps/mcp run typecheck` green.
- Live: `curl -XPOST https://mcp.asepharyana.my.id/mcp` initialize -> 200 + serverInfo;
tools/list -> 6 tools; resources/list -> 4 resources.
- `systemctl is-active mcpedia-mcp` == active.
- Commit + push.
-67
View File
@@ -1,67 +0,0 @@
# MCPedia Phase 11 — CRUD + Auth + Web UI
## STATUS: ✅ ALL DONE (committed f8b525d, CI+Deploy success)
## Plan (spec BEFORE implementation, per user rule)
### Scope: 4 areas
1. **CRUD**: Create/Read/Update/Delete documents from web UI + MCP
2. **Authentication**: MCP/API writes (x-webhook-secret), Web CRUD (cookie-based ADMIN_PASSWORD)
3. **Web + Agent access**: Web forms + MCP write tools
4. **UI/UX**: Edit forms, TOC, dark mode
### Requirements:
1. Source of truth = filesystem (markdown files in content/{section}/{slug}.md)
2. DB mirrors disk (documents, document_chunks, document_revisions tables)
3. Single indexing path: indexContentFile in @mcpedia/core
4. Auth: MCP/API writes use WEBHOOK_SECRET; Web uses ADMIN_PASSWORD cookie
5. Slug rules: [a-z0-9][a-z0-9/_-]*, no //, no .. traversal
6. No breaking existing features (32 original tests still green)
7. UI/UX: edit button on doc pages, ?edit=1 inline form, /create page, login, TOC, dark mode
### Implementation
#### Backend
- `packages/parser`: added `stringifyFile()` (writes markdown with frontmatter)
- `packages/core`: `createDocument`, `updateDocument`, `deleteDocument` (file I/O + DB + revision + chunks)
- `apps/api`: tRPC CRUD routers (`requireWriteAuth`), fixed `requireWriteAuth` env-constant bug (now uses `ctx.expectedSecret` injected from `createApp(deps)`)
- `apps/mcp`: 3 new write tools (`create_document`, `update_document`, `delete_document`) gated by `x-webhook-secret`
#### Web UI
- `apps/web/app/api/auth/login/route.ts`: POST login → verify ADMIN_PASSWORD, set `mcpedia_admin` cookie
- `apps/web/app/api/docs/route.ts`: POST (create)
- `apps/web/app/api/docs/[...slug]/route.ts`: PUT (update), DELETE (delete)
- `apps/web/app/components/DocForm.tsx`: shared create/edit form
- `apps/web/app/create/page.tsx`: create form
- `apps/web/app/login/page.tsx`: login form
- `apps/web/app/[section]/[...slug]/page.tsx`: `?edit=1` inline edit, TOC, dark mode, Edit button
- `apps/web/app/docs/page.tsx`: docs index listing
- `apps/web/app/components/TOC.tsx`: auto-generated TOC from h2/h3 headings
- `apps/web/app/components/ThemeToggle.tsx`: dark mode toggle (localStorage + system default)
#### New deps (minimal — only for UX):
- `rehype-slug` (heading anchors for TOC links)
- `github-slugger` (matching slug algorithm for TOC client-side)
### Gotchas (learned the hard way)
1. tRPC fetch adapter expects input directly as JSON body, NOT JSON-RPC envelope
2. requireWriteAuth compared `ctx.webhookSecret !== WEBHOOK_SECRET` (module-level env constant) — untestable. Fixed: `ctx.webhookSecret !== ctx.expectedSecret` (injected per-app via deps).
3. Next.js catch-all routes: `[...slug]/edit/` is INVALID (catch-all must be last). Used `?edit=1` query param instead.
4. Next.js App Router: PUT/DELETE on `/api/docs/route.ts` doesn't match `/api/docs/{slug}` — need dynamic route `/api/docs/[...slug]/route.ts`.
5. `@env.example` should be updated.
6. `ADMIN_PASSWORD` must be set in VPS `.env` (deployed separately).
### Verification
- Typecheck: ✅ 4/4 apps green
- Tests: ✅ 40 tests green (32 original + 8 new), no DB/Redis
- Build: ✅ web compiled
- CI: ✅ success → Deploy: ✅ success
- Live: all 9 endpoints 200, 13 MCP tools live, CRUD e2e verified (login → create → view → delete via cookie auth), MCP create_document verified via header auth
- Test docs cleaned up (404 confirmed)
### Commits
1. `57f9001` feat: Phase 11 — CRUD + auth + web UI
2. `1dd16eb` feat(web): Phase 11 UI/UX — TOC, dark mode toggle, /docs index
3. `98437cb` fix(web): /api/docs accepts cookie OR header (not both required)
4. `8ed8c67` fix(web): split PUT/DELETE into /api/docs/[...slug]/route.ts
5. `f8b525d` chore: remove test docs
-24
View File
@@ -1,24 +0,0 @@
# Phase 3 — Deploy + git-sync wiring (remaining work)
Status: Phase 3/4 code is DONE and e2e-verified (webhook enqueue -> worker drain, 0 failed).
What was missing on the host: API + worker never ran as systemd services, and the GitHub
push webhook was never created. Also a real integration bug: `assertWebhookAuth` only
accepts a plain `x-webhook-secret` header, which GitHub does NOT send (GitHub delivers
`X-Hub-Signature-256` = HMAC-SHA256 of raw body). So a real GitHub webhook would 401.
## Changes
1. `apps/api/src/index.ts` — `assertWebhookAuth` now verifies GitHub `X-Hub-Signature-256`
(HMAC-SHA256 of raw body w/ WEBHOOK_SECRET) and still accepts `x-webhook-secret` for manual tests.
2. root `package.json` scripts — `api`: `bun --cwd apps/api run dev` -> `bun --cwd apps/api src/index.ts`
(the `run dev` form errors in bun 1.3.14; direct-file form verified booting + health). `worker` -> same form for consistency.
3. systemd — `cp deploy/*.service /etc/systemd/system`, `daemon-reload`, `enable --now mcpedia-api mcpedia-worker`.
4. Caddy — expose `/hooks/*` on `wiki.asepharyana.my.id` -> :4020 (no new DNS). Keep web on :4016.
5. GitHub webhook — `gh api repos/asepharyana/mcpedia/hooks` POST:
`https://wiki.asepharyana.my.id/hooks/reindex`, content_type json, secret=WEBHOOK_SECRET, events=push.
## Verification
- `systemctl is-active mcpedia-api mcpedia-worker` == active.
- `curl /health` on :4020 -> ok.
- `curl -X POST https://wiki.asepharyana.my.id/hooks/reindex -H "X-Hub-Signature-256: ..."` (or x-webhook-secret) -> 200 + jobId; worker drains.
- `gh api .../hooks` lists the webhook.
- `turbo run typecheck` green; commit + push.
-114
View File
@@ -1,114 +0,0 @@
# MCPedia — Phase 3 "Async + Scale" Implementation Plan
Status: Phase 1 (MVP) + Phase 2 (Semantic+API) DONE. Phase 3 adds async
background work, git-driven reindex, document revision history, and MCP
Resources. All logic stays in `@mcpedia/core`; new `packages/queue` wires
BullMQ; `apps/worker` runs the worker process; the existing API gets a git-sync
webhook + job-status procedures; the MCP server gains Resources.
## Scope (4 features from PHASES.md)
1. **Redis + BullMQ background indexing/embedding workers**
2. **Git synchronization hook** (auto-reindex on push via webhook)
3. **Document revision system** (`document_revisions`)
4. **MCP Resources** (`mcpedia://docs/...`) alongside existing tools
## Architecture decisions (locked)
- **Redis**: shared imrnes Redis `100.121.180.82:6379`, no auth (verified
`+PONG`). `REDIS_URL` env (default `redis://100.121.180.82:6379`), optional
`REDIS_PASSWORD`. BullMQ key prefix `mcpedia:` to avoid collisions on the
shared instance.
- **Queue lib**: `bullmq@6.1.2` + `ioredis@6.0.0` (BullMQ peer dep). Pass an
ioredis instance; BullMQ duplicates it for blocking commands.
- **Single source of truth preserved**: per-doc indexing logic moves into
`@mcpedia/core` as `indexContentFile(relPath, reason?)`. The script, the
worker, and the git hook ALL call this. Revisions are snapshotted inside it.
- **Revisions**: created only when body actually changes vs the latest revision
(avoids bloat on every sync). Stored in `document_revisions`.
## Files touched
### packages/config
- `src/index.ts`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
### packages/db
- `src/schema.ts`: add `documentRevisions` table
(id, documentId→documents.id cascade, slug, revisionNo int, title, body,
meta jsonb, reason text, createdAt). Index (document_id, revision_no DESC),
(slug).
- `drizzle/0002_document_revisions.sql`: migration (applied via psql).
- `drizzle/meta/0002_snapshot.json` + `_journal.json` entry (keeps drizzle-kit
consistent even though we apply manually).
### packages/core (new)
- `src/index.service.ts`:
- `indexContentFile(relPath: string, reason = "index")` — parse → upsert
`documents` → `indexChunks` → snapshot revision (if changed).
- `runFullIndex(reason?)` — walk content, index each, return counts.
- `src/revision.service.ts`:
- `createRevision(...)`, `listRevisions(slug, limit)`,
`getRevision(id)`, `latestRevisionBody(slug)`, `restoreRevision(id)`.
- `src/index.ts`: export both.
### packages/queue (NEW)
- `package.json` (@mcpedia/queue): deps bullmq, ioredis, @mcpedia/core,
@mcpedia/db, @mcpedia/config.
- `src/client.ts`: ioredis instance factory from config.
- `src/queue.ts`:
- `INDEX_QUEUE = "mcpedia-index"`.
- `enqueueIndexDoc(slug, absPath, reason)`, `enqueueFullIndex(reason)`.
- `getQueue()` lazy singleton.
- `src/worker.ts`: `startWorker()` — BullMQ Worker with 3 job types:
`index-doc` (single), `index-all` (full), `reindex` (full, reason=git-push).
Graceful shutdown on SIGINT/SIGTERM. Job progress + error handling.
### apps/worker (NEW)
- `package.json` (@mcpedia/worker): script `start: bun src/index.ts`.
- `src/index.ts`: `startWorker()` + heartbeat log.
### apps/api
- `src/index.ts`: add `POST /hooks/reindex` (full) and
`POST /hooks/index?slug=` (single) webhook routes → enqueue jobs. Mount
AFTER /trpc.
- `src/router.ts`: add `jobStatus` (id→state/prev/failedData),
`queueStatus` (waiting/active/completed/failed counts),
`revisions` (slug→list), `restoreRevision` (id→new slug/doc).
- `package.json`: add `@mcpedia/queue` dep, `hooks` reused.
### apps/mcp
- `src/index.ts`: register Resources:
- `mcpedia://docs` (list all metas)
- `mcpedia://docs/{slug}` (full body from disk)
- `mcpedia://docs/{slug}/chunks` (chunk previews)
- `mcpedia://docs/{slug}/revisions` (revision list)
- `src/smoke.test.ts`: add `listResources` + read `mcpedia://docs` assertion.
### scripts
- `scripts/indexer.ts`: refactor `main()` to call `runFullIndex()`.
### Root
- `package.json`: add `"worker": "bun --cwd apps/worker run start"`,
`"reindex": "bun run scripts/worker.ts"`? No — `worker` runs the listener;
triggering reindex = `bun run api` webhook or `enqueueFullIndex` helper.
Add `"enqueue-index": "bun run scripts/enqueue.ts"` (one-shot enqueue).
- `.env.example`: add `REDIS_URL`, `REDIS_PASSWORD`, `QUEUE_PREFIX`.
### Docs
- `PHASES.md`: mark Phase 3 items DONE with notes.
- `README.md`: document worker, webhook, revisions, MCP resources.
## Verification (real, not claimed)
1. `bun install` picks up new deps.
2. `bunx turbo run build` + `typecheck` green across workspace.
3. **Real BullMQ e2e against imrnes Redis**: script that enqueues an
`index-doc` job, starts a Worker, asserts the job completes and the doc row
+ chunks + a revision row appear in Postgres. Verifies Redis+ioredis+bullmq
+ db + core all wired correctly.
4. `bun --cwd apps/mcp run smoke` passes (incl. new resources).
5. `bun run index` (runFullIndex) green; verify `documents`,
`document_chunks`, `document_revisions` row counts via psql.
6. API webhook: `curl -XPOST localhost:4020/hooks/reindex` enqueues; worker
processes; `curl localhost:4020/trpc/queueStatus` reflects counts.
7. MCP resource read returns real content.
-105
View File
@@ -1,105 +0,0 @@
# MCPedia Phase 4 — Operability & Correctness Hardening
> Reinterpretation: the PHASES.md "Scale-out" items (OpenSearch, object storage,
> multi-tenant, distributed workers) are YAGNI at KB scale (4 docs). Phase 4 =
> make the Phase 3 async + revision machinery **correct, secure, observable, and
> deployable** — not speculative infra. Each task below fixes a real gap found
> by reading the code, not a hypothetical need.
## Tasks
### T1 — `restoreRevision` must rebuild semantic chunks (CORRECTNESS BUG)
**Root cause:** `packages/core/src/revision.service.ts` `restoreRevision` writes
the old body back into `documents` but never calls `indexChunks(slug, body)`.
So after a restore, keyword search (FTS on `documents.body`) is correct but
`document_chunks`/embeddings stay on the *new* body → semantic + hybrid search
return stale/ghost chunks.
**Fix:**
- Add `reindexChunks(slug)` to `@mcpedia/core` that re-runs `indexChunks(slug, body)`
using the live `documents.body` (the new body after the update).
- Call it inside `restoreRevision` after the `documents` update (wrap in try/catch
like `indexContentFile` so embed failure doesn't abort the restore).
- Add a unit-style assertion to the MCP smoke test or a small script: restore →
`document_chunks` count matches re-chunked body.
**Files:** `packages/core/src/revision.service.ts`, `packages/core/src/index.ts`,
`packages/core/src/document.service.ts` (export existing `indexChunks` if needed).
### T2 — Secure the git-sync webhook (SECURITY)
**Root cause:** `apps/api/src/index.ts` `/hooks/reindex` and `/hooks/index` accept
any request with no `WEBHOOK_SECRET` check — `.env.example` defines `WEBHOOK_SECRET`
but the router never reads it.
**Fix:**
- In `apps/api/src/index.ts`, compare `c.req.header("x-webhook-secret")` (or
`?secret=`) against `WEBHOOK_SECRET` (from `@mcpedia/config`). If unset/mismatch →
`401`. If `WEBHOOK_SECRET` env is empty, reject at startup with a clear log
(fail-fast, don't run an open endpoint).
- Add `WEBHOOK_SECRET` to `packages/config/src/index.ts` export.
- Document the header in README + verify with curl (401 without secret, 200 with).
**Files:** `apps/api/src/index.ts`, `packages/config/src/index.ts`, README.
### T3 — Web UI revisions view (UX)
**Root cause:** Web UI (server components) calls `@mcpedia/core` directly; there is
no revisions surface even though `revisions`/`restoreRevision` tRPC + MCP resource
exist.
**Fix (server-component only, no client JS):**
- On the doc page (`apps/web/app/[section]/[...slug]/page.tsx`), fetch
`listRevisions(fullSlug, 10)` and render a "History" panel: revision number,
reason, createdAt, body length, and a `/api/revisions/restore` link/POST that
calls the tRPC `restoreRevision` mutation via a server action or a form POST to
a small route handler. Simplest: a `<form method="post" action="/api/revisions/restore">`
with hidden `id` + a route handler in `apps/web` calling `restoreRevision`.
Keep it read-mostly; restore is a deliberate action.
- Add `apps/web/app/api/revisions/restore/route.ts` (POST) → `restoreRevision(id)`
→ `revalidatePath` the doc.
**Files:** `apps/web/app/[section]/[...slug]/page.tsx`,
`apps/web/app/api/revisions/restore/route.ts`.
### T4 — Paginate `listRevisions` / `revisions` API (PERF)
**Root cause:** `revision.service.ts` `listRevisions` does `select length(body)`
(fine) but the tRPC `revisions` and MCP resource return *full* revision rows
including the body in some callers; list endpoints should never carry bodies.
**Fix:**
- Ensure `listRevisions` summary excludes `body` (it already does — `bodyLength`
only). Add `offset` param for paging. Confirm MCP resource uses the summary.
- No behavior change for the doc page (uses summary).
**Files:** `packages/core/src/revision.service.ts` (add `offset`), router unchanged.
### T5 — Deployable as supervised services (OPS)
**Root cause:** `apps/api` and `apps/worker` run only ad-hoc; the host already runs
`zeavis` via Nix/systemd. Phase-4 operability = provide a systemd unit (or Nix
service) so `mcpedia-api` + `mcpedia-worker` start on boot and restart on failure.
**Fix (Nix-first, per MEMORY):**
- Write `mcpedia-api.service` + `mcpedia-worker.service` systemd unit files under
`deploy/` (bun run api / bun run worker, `WorkingDirectory`, `Restart=on-failure`,
`EnvironmentFile` pointing at `.env`, `After=network-online.target`).
- README section "Run as a service" with `cp deploy/*.service /etc/systemd/system && systemctl daemon-reload && systemctl enable --now mcpedia-api mcpedia-worker`.
- **Do NOT** `systemctl` on the host without user confirmation (changing live
services). Provide the files + instructions only; user runs enable.
**Files:** `deploy/mcpedia-api.service`, `deploy/mcpedia-worker.service`, README.
## Verification (all real, against imrnes Redis + Postgres)
1. `turbo run typecheck` + `turbo run build` green.
2. T1: script — edit a doc, reindex (new revision + new chunks), restore rev #1,
assert `document_chunks` count for that slug now matches re-chunk of rev #1 body
and `semanticSearch` on a term unique to rev #1 returns it.
3. T2: `curl -XPOST localhost:4020/hooks/reindex` → 401; with
`-H "x-webhook-secret: $WEBHOOK_SECRET"` → 200 + jobId.
4. T3: `next build` includes the History panel; restore form rebuilds chunks
(verified via T1 path through the route handler).
5. T4: `revisions` API returns summaries without body; offset paging works.
6. T5: `systemd-analyze verify deploy/*.service` passes (off-host safe check);
README documents enable steps.
## Out of scope (YAGNI, keep deferred per PHASES.md)
OpenSearch/Elasticsearch, object storage, multi-tenant, distributed workers,
pgvector migration. Revisit only when corpus > ~10k docs or query latency bites.
-41
View File
@@ -1,41 +0,0 @@
# MCPedia — "lanjut semua" workstream
Three real gaps remain (from review): tiny corpus (4 docs), MCP read-only (no write/auth),
no observability. This plan closes all three.
## 1. Content corpus (grow the KB)
Author real, useful docs so search/semantic/revisions have something to operate on.
Frontmatter schema (from packages/parser): id,title,type,tags,status,author,created_at,updated_at.
Sections: docs|writeups|research|notes. Files under content/<section>/...
New docs to add:
- content/docs/caddy/reverse-proxy.md (ops reference, tags: caddy, reverse-proxy, tls)
- content/docs/bullmq/workers.md (queue/worker reference, tags: bullmq, redis, jobs)
- content/docs/mcp/streamable-http.md (MCP transport reference, tags: mcp, protocol, http)
- content/notes/postgres/full-text-search.md (PG FTS notes, tags: postgres, fts, tsvector)
- content/writeups/infra/cloudflare-525.md (debugging writeup, tags: cloudflare, tls, 525)
After adding: `bun run index` to reindex (writes revisions + chunks), verify counts.
## 2. MCP write-tools + auth
Add mutating + admin tools to the MCP server (currently read-only):
- `index_document(slug)` -> enqueueIndexDoc (requires MCP auth header)
- `reindex_all()` -> enqueueFullIndex (requires MCP auth header)
- `queue_status()` -> getQueue counts (read, public)
- `restore_revision(id)` -> restoreRevision (requires MCP auth header)
Auth: MCP client must send header `x-webhook-secret` (reuse WEBHOOK_SECRET). StreamableHTTP
transport: read the Authorization/header in http.ts, pass to server via a factory closure
capturing the request; tools check it. Stateless per-request server already created fresh,
so threading the header is clean. Guard write-tools with the same requireWriteAuth logic.
Verify: unauthenticated call to index_document -> error; authenticated -> enqueues job.
## 3. Observability
- `GET /metrics` on the API (Prometheus text format): queue counts (waiting/active/
completed/failed/delayed), uptime, service name. Public (safe to expose).
- Caddy: expose /metrics on wiki. domain -> :4020 (add to handle list).
- tRPC `queueStatus` already exists; /metrics reuses getQueue.
Verify: curl /metrics -> text exposition with mcpedia_queue_* gauges.
## Verification
- typecheck green (turbo run typecheck).
- MCP smoke extended: tools/list shows new tools; authenticated index_document enqueues.
- /metrics returns 200 text; queue drains.
- commit + push.
-157
View File
@@ -1,157 +0,0 @@
# MCPedia Phase 9 — Test Coverage + Observability Hardening
**Goal:** Add a real test suite (CI-gated) covering every layer of MCPedia — pure
logic, Core services, the API surface, the MCP server + auth gates, and the Web
UI — so regressions are caught before deploy. The suite must run green in CI
with **no external services** (no Postgres, no Redis) by using in-process fakes.
## Constraints recap
- Tooling: bun workspaces + Turborepo, bun 1.3.14 has a built-in `bun:test` runner.
- DB is `imrnes` Postgres at `:6432` (no DB in CI) — tests must NOT touch it.
- pgvector is NOT installed (vectors are `real[]`, cosine in-app) — confirmed.
- Existing smoke test (`apps/mcp/src/smoke.test.ts`) is a script with `main()`
run via `bun run smoke`, NOT a `bun:test` file. It hits the DB → cannot run in CI.
- Secrets (`WEBHOOK_SECRET`, `DATABASE_URL`, `EMBED_*`) live in `.env` (gitignored)
or BWS for deploy — never in tests or committed config.
## Decisions
1. **Runner:** `bun:test` — zero-config, built into the bun 1.3.14 toolchain
already used. No extra deps. Add a `test` task to `turbo.json` and `package.json`
scripts; add a `Test` step to CI.
2. **No live DB in CI.** Tests that would need Postgres/Redis/Embeddings use
**in-process fakes** (memory stores + a stub embedder returning fixed vectors).
This means Core service tests cannot use the real `@mcpedia/db` singleton —
they must accept an injected DB (drizzle-pg mem or a hand-rolled fake). We will
**refactor the Core services' DB access behind injectable handles** where cheap,
and for the MCP/HTTP auth-layer tests we stub `@mcpedia/queue` + `@mcpedia/core`
at the module boundary (the transport/auth logic does not need a real queue).
3. **Test boundaries by package:**
- `packages/embeddings` — pure: `chunkText`, `cosine`. Real assertions, no I/O.
- `packages/search` — pure: `toTsQuery`, `cosine`. SQL-bearing functions
(`keywordSearch`/`semanticSearch`/`hybridSearch`) tested via a **fake db**
injected into `@mcpedia/db`, OR via the `cosine`/fusion helpers in isolation.
- `packages/core` — `snapshotRevision` dedup logic (refactor to accept an inject
fn or test the public `indexContentFile`/`restoreRevision` with fakes).
Focus: revision-dedup correctness + `restoreRevision` triggers `reindexChunks`.
- `apps/api` — Hono app: `/health`, `/metrics` shape, `/hooks/*` auth (401 w/o
secret, 200 + enqueue w/ secret using a fake queue), tRPC `restoreRevision`
mutation auth gate (401 w/o secret).
- `apps/mcp` — auth gates on write tools: `index_document`/`reindex_all`/
`restore_revision` error without secret, enqueue with secret (fake queue).
Read tools + resources via `InMemoryTransport` (reuse smoke style but without
DB).
- `apps/web` — render correctness of home (lists sections), doc page
(renders title + markdown + history panel when revisions exist), search page
(keyword/hybrid toggle, empty state). These need a fake Core.
## Approach per test (minimal, high-signal)
### embeddings: `packages/embeddings/src/chunk.test.ts`
- `chunkText("hello world")` with short size → single chunk.
- `chunkText` long text → multiple chunks, overlap honored, no word splits past boundary.
- `chunkText("")` / `" "` → `[]`.
### search: `packages/search/src/cosine.test.ts` (new tiny file) + refactor
- `cosine([1,0],[0,1])` ≈ 0; `cosine([1,1],[1,1])` = 1; `cosine([],[1])` = 0.
- `toTsQuery("a b c")` → `"a:* & b:* & c:*"`; empty/garbage → `""`.
### core: `packages/core/src/index.service.test.ts`
The hard part: `indexContentFile`/`restoreRevision`/`snapshotRevision` call `db`
directly. Two options:
- **Option A (chosen):** extract `snapshotRevision`'s "latest body" + "insert"
steps behind the existing `db` but make `indexContentFile` test the revision
*decision* by inserting a doc + revision directly via `db` in a test Postgres
(too heavy for CI).
- **Option B (chosen):** test the **pure decision logic** by refactoring
`snapshotRevision` to export a pure helper
`shouldCreateRevision(latestBody, body): boolean` — `true` when latest is null
or latest.body !== body. Then a unit test asserts the dedup truth table;
`indexContentFile` is verified by the existing e2e (manual `bun run index`).
This is the CI-safe win.
- `restoreRevision` correctness: assert it calls `reindexChunks(slug)` — we can
test by spying. Since `reindexChunks` is in the same module, we'll export a
seam: `restoreRevision(id, { reindexChunks: spy })` — keep backward compat by
defaulting. (Or test the public contract via the API layer instead.)
### api: `apps/api/src/index.test.ts`
- Build the Hono `app` from a testable factory that accepts a fake queue + fake
webhook secret. Current `index.ts` throws at import if `WEBHOOK_SECRET` unset —
that breaks import in CI. **Refactor:** move the fail-fast check into
`listen()`/serve start, so the app is constructable without a secret for
testing. Export `createApp(opts?)` returning the Hono instance.
- `/health` → 200 `{ok:true}`.
- `/metrics` → 200, text/plain, contains `mcpedia_uptime_seconds` +
`mcpedia_queue_jobs` for each state (fake queue returns 0/1).
- `POST /hooks/reindex` w/o `x-webhook-secret` → 401; with matching secret →
200 + `{ok:true, jobId}` (fake queue records the enqueue).
- tRPC: build a client against the app, call `restoreRevision` without secret →
error; the public `listDocuments` returns from a fake DB.
### mcp: `apps/mcp/src/auth.test.ts`
- `createMcpServer()` (no secret) → `index_document`/`reindex_all`/`restore_revision`
throw "unauthorized".
- `createMcpServer("secret")` → same tools reach the enqueue call (fake queue).
- Read tools still work without secret (server loads, resources list).
### web: `apps/web/app/search/page.test.tsx` (or a lighter harness)
- This is the hardest to test without a browser. **Decision:** keep web tests
minimal — assert that `toTsQuery`/render helpers exist; full DOM tests deferred
(needs playwright + a running server). We'll instead add a **contract test**
that the search page's `dynamic = "force-dynamic"` export exists (static-
generation guard, the kind of thing that broke CI before).
## File layout (new files)
```
packages/embeddings/src/chunk.test.ts
packages/search/src/cosine.test.ts
packages/core/src/index.service.test.ts # snapshotRevision + restoreRevision seam
apps/api/src/index.test.ts # Hono /health /metrics /hooks + tRPC gate
apps/mcp/src/auth.test.ts # write-tool auth gates
```
## turbo.json
Add a `test` task (like `typecheck`, no dependsOn, cache false so it always runs):
```jsonc
"test": { "cache": false }
```
Each app/pkg gets `"test": "bun test"` in its package.json.
## CI (`.github/workflows/ci.yml`)
After `Build`, add:
```yaml
- name: Test
run: bun run test # -> turbo run test
```
## Source changes required (enablers)
1. `apps/api/src/app.ts` (NEW) — extracted `createApp(deps?)` factory returning a
`Promise<Hono>`. Pure construction (no process exit, no fail-fast). Accepts
injected `ApiDeps` (`{ queue, webhookSecret }`); when omitted, lazily
imports the real queue + uses `WEBHOOK_SECRET` (production path). `/dashboard`
now renders the HTML from a new `apps/api/src/dashboard.ts` module.
2. `apps/api/src/dashboard.ts` (NEW) — the self-contained dashboard HTML, split
out of the original `index.ts` so the route is testable + the const is
importable. XSS-safe (esc() on all KB-sourced fields; documented in comment).
3. `apps/api/src/index.ts` — now a thin re-export of `createApp`/`start` + the
`isMain` bootstrap. Systemd unit still runs `bun --cwd apps/api src/index.ts`.
4. `packages/core/src/index.service.ts` — exported `shouldCreateRevision` (pure
predicate for the revision dedup invariant). `restoreRevision` in
`revision.service.ts` now accepts an optional `opts.reindex` seam (defaults
to the real `reindexChunks`), making the chunk-rebuild contract testable.
5. `apps/mcp/src/smoke.ts` — renamed from `smoke.test.ts` (so `bun test` doesn't
treat the integration smoke as a unit run), and fixed stale assertions:
expected tool set updated to all 10 tools (Phase 7 additions + write tools),
`list_documents(section=docs)` count updated to 4 (post-Phase-7 corpus).
## Verification
- `bun run test` (local) → 32 tests green across 6 packages (embeddings 5,
search 8, core 4, parser 5, mcp 6, api 8), **no live DB needed** (mocks
stub `@mcpedia/db`, `@mcpedia/queue`, `@mcpedia/core`).
- `bun run typecheck` → green (4 apps, no test-only type errors).
- `bun --cwd apps/mcp run smoke` → green (integration, needs live DB — runs in
CI on the deploy host, not in CI's no-services job).
- Live API check: `/health`, `/metrics`, `/dashboard`, `/hooks/*` auth gate
all 200/401-verified against a temp-port server.
- Commit + push; worker redeploy not needed (code + tests only).
-80
View File
@@ -1,80 +0,0 @@
# Phase 15 — UI/UX Polish Pass
> User: "perbagus ui ux nya"
## Audit
### Current issues
1. **Section index** (`[section]/page.tsx`):
- `buildFolderTree` uses path segments instead of doc titles for leaf nodes
- Folder tree + flat list at bottom is redundant
- No folder icons (📁), no doc type indicators, no dates
- Tree rendering is very basic (no visual hierarchy)
2. **Doc page** (`[section]/[...slug]/page.tsx`):
- CustomFieldBadges shows key=value but labels all badges with key name as title
- No structured metadata card layout
- Related docs cards are functional but visually flat
3. **Folder index** (`FolderIndexPage` in `[...slug]/page.tsx`):
- Shows doc slug parts (e.g. "pwn-100-ret2win-alignment") instead of titles
- No date, no tags, no description
- Subfolders are plain text links, no folder count or doc count
4. **Homepage** (`page.tsx`):
- Folder tree inside cards is cramped (text-xs, no spacing)
- Section cards show "View all (N)" but folder tree duplicates that
5. **Sidebar** (`Sidebar.tsx`):
- Tree uses indentation via inline style but no visual depth cues
- No hover expand for folders
- Active state only on exact match, not parent folders
## Plan
### 1. Section index — enhanced tree
- Use doc titles for leaf nodes (already have `doc` reference in tree)
- Add 📁 for folders, 📄 for docs
- Show doc count per folder
- Remove flat list (redundant with tree)
- Add "View all" link per section
- Better visual hierarchy: folder headers with counts, doc titles with dates
### 2. Doc page — metadata card
- Replace inline badge row with a structured metadata card
- Show custom fields as labeled badges (key → value with color)
- Keep CustomFieldBadges for backward compat but improve layout
- Add metadata card with: author, date, tags, and all custom fields
### 3. Folder index — rich listing
- Show doc titles (fetch from listDocuments, match by path)
- Show update date per doc
- Show tags per doc
- Add doc count for subfolders
- Better visual separation between subfolders and docs
### 4. Homepage — cleaner tree
- Increase font size for tree items
- Show section icon + label in tree header
- Better spacing between sections
### 5. Sidebar — depth cues
- Use ml-4 per level instead of inline style
- Add section divider lines
- Highlight active section + parent folders
## Files to change
```
mod: apps/web/app/[section]/page.tsx # enhanced tree, use doc titles
mod: apps/web/app/[section]/[...slug]/page.tsx # FolderIndexPage + metadata card
mod: apps/web/app/page.tsx # cleaner section tree cards
mod: apps/web/app/components/Sidebar.tsx # depth + active parent highlight
mod: apps/web/app/globals.css # additional CSS vars if needed
```
## Verification
- All endpoints still 200
- `bun run test` green
- `turbo run typecheck` green
- `next build` succeeds
- Live visual check of folder index, section index, doc page