MythEclipse/mcpedia
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: revamp to dynamic database-first architecture and cleanup phase docs
CI / typecheck + build (turbo) (push) Canceled after 0s
Details

Browse Source 
- Migrate document fetching and CRUD to be PostgreSQL-authoritative
- Remove static section enums and add dynamic listSections query
- Support custom sections and metadata across API, MCP, and Web UI
- Add /api/sections endpoint and update Header, Sidebar, and Forms
- Remove obsolete phase planning docs and modernize README/AGENTS
This commit is contained in:
asepharyana
2026-08-21 11:35:42 +07:00
parent a937e8f51b
commit a9b67385c3
37 changed files with 720 additions and 2515 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+260 -136
View File
@@ -1,186 +1,310 @@
# MCPedia
> A content-first knowledge base — readable as Markdown/MDX in Git, queryable by
> humans via a Web UI and by AI agents via the Model Context Protocol (MCP).
> A content-first knowledge base — readable as Markdown in Git, queryable by humans via a modern Web UI, and accessible to AI agents via the Model Context Protocol (MCP).
MCPedia keeps content as plain Markdown files under `content/`. A Git-tracked
source of truth, indexed into PostgreSQL (metadata + a `tsvector` full-text
column) and served through a single **Core** layer that every interface
(Web, MCP) shares — no business logic duplicated per surface.
MCPedia keeps content as plain Markdown files under `content/` as a Git-tracked source of truth. Content is indexed into PostgreSQL (with a weighted `tsvector` full-text search column and chunked embeddings) and served through a unified **Core** layer that every interface (Web, MCP, API, Worker, CLI) shares.
## Monorepo layout
---
## Architecture Overview
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Web UI │ │ MCP Server │ │ tRPC / Hono │ │ BullMQ Worker │
│ (Next.js 16) │ │ (Stdio + HTTP) │ │ API (:4020) │ │ (Async Queue) │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │ │
└──────────────────────┴──────────┬───────────┴──────────────────────┘
▼
┌─────────────────────────┐
│ @mcpedia/core │
│ (Unified Business Logic)│
└────────────┬────────────┘
│
┌─────────────────────────────────┼─────────────────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Markdown Files │ │ @mcpedia/db │ │ @mcpedia/search │
│ (content/ tree) │ │ (PostgreSQL) │ │ (FTS + Cosine) │
└──────────────────┘ └──────────────────┘ └──────────────────┘
```
### Core Principles
1. **Content as Source of Truth**: Markdown files with YAML frontmatter under `content/` are primary. The database holds metadata, search indices, chunk embeddings, and revision history.
2. **Single Core Layer**: All business logic (CRUD, indexing, search, revisions, path classification) is encapsulated in `@mcpedia/core`. Interfaces never touch the database directly.
3. **Multi-Modal Search**: Keyword search (Postgres FTS), semantic search (vector cosine similarity), and hybrid search (Reciprocal Rank Fusion / RRF) work out of the box.
4. **Resilient Revision System**: Content edits snapshot revisions automatically; metadata-only edits are deduplicated. Restoring a revision automatically rebuilds semantic search chunks.
5. **Dual Interface**: Full human-friendly web experience + first-class AI agent integration via MCP.
---
## Monorepo Layout
```
mcpedia/
├── apps/
│ ├── web/ # Next.js 16 (Turbopack) — human-facing docs UI + search
│ ├── mcp/ # MCP server (stdio) — AI-agent interface (tools + resources)
│ └── api/ # Hono + tRPC v11 API on :4020 (+ /hooks/* git-sync webhooks)
│ ├── web/ # Next.js 16 (Turbopack) — dark-mode UI, hierarchical doc tree, TOC, CRUD
│ ├── mcp/ # MCP server (stdio + Streamable HTTP on :4021) — 13 tools + 4 resources
│ ├── mcp-client/ # Interactive MCP CLI REPL, one-shot command runner, and TypeScript client SDK
│ ├── api/ # Hono + tRPC v11 API on :4020, git-sync webhooks, Prometheus metrics, dashboard
│ └── worker/ # Long-running BullMQ worker for async indexing and embedding jobs
├── packages/
│ ├── types/ # shared domain types (DocSection, Document, SearchHit, ...)
│ ├── config/ # loads .env (repo root) as authoritative dev config
│ ├── db/ # Drizzle ORM schema + client + drizzle-kit config
│ ├── parser/ # frontmatter (gray-matter) parsing
│ ├── search/ # Postgres FTS query (ts_rank + ts_headline)
│ ├── embeddings/ # embedding provider + chunker
│ ├── queue/ # Redis (ioredis) + BullMQ worker/queue (Phase 3)
│ └── core/ # Document/Content/Search/Index/Revision — the only business logic
├── content/ # docs/ writeups/ research/ notes/ (the knowledge base)
└── scripts/ # indexer.ts (full reindex), enqueue.ts (one-shot job enqueue)
│ ├── core/ # Document, Content, Search, Index, Revision, and Path services (the business logic)
│ ├── db/ # Drizzle ORM schema, client, and migrations
│ ├── parser/ # Frontmatter (gray-matter) parsing & stringification with dynamic field support
│ ├── search/ # Postgres FTS query (ts_rank + ts_headline), vector cosine, and RRF hybrid fusion
│ ├── embeddings/ # Chunking algorithms & embedding provider integrations
│ ├── queue/ # Redis (ioredis) client & BullMQ queue/worker definitions
│ ├── types/ # Shared TypeScript domain types and interfaces
│ └── config/ # Authoritative environment configuration loader (.env)
├── content/ # Markdown knowledge base organized by sections and nested folders
│ ├── docs/ # System and architectural documentation
│ ├── writeups/ # Technical writeups, CTF solutions, and debugging reports
│ ├── research/ # Research notes and evaluations
│ └── notes/ # Engineering notes and quick references
└── scripts/ # indexer.ts (full corpus reindexing), enqueue.ts (one-shot job enqueue)
```
## Architecture principle
---
```
Web ─┐
├──► Core ──► Repository (@mcpedia/db) ──► PostgreSQL
MCP ─┘
```
## Key Features
All interfaces go through `@mcpedia/core`. Nothing outside `packages/db` and
`packages/core` touches the database directly.
### 1. Multi-Modal Search
## Quick start
| Search Mode | Mechanism | Best Used For |
|---|---|---|
| **Keyword Search** | PostgreSQL `tsvector` (`simple` config, weighted A/B) + GIN index + `ts_rank` + `ts_headline` | Exact terms, code symbols, error codes, identifiers |
| **Semantic Search** | Text chunking (1000 chars, 150 overlap) + vector embeddings + cosine similarity ranking | Conceptual questions, paraphrased queries, intent matching |
| **Hybrid Search** | Reciprocal Rank Fusion ($RRF = \sum \frac{1}{k + rank}$) fusing keyword and semantic signals | General-purpose search with optimal relevance |
### 2. Hierarchical Folder Navigation
MCPedia supports nested folder structures (like GitHub repositories):
- **Dynamic Path Classification**: The router inspects paths and automatically distinguishes between leaf documents and folder nodes containing subfolders or child documents.
- **Folder Index Pages**: Navigating to any folder (e.g. `/writeups/ctf/defcon-quals-2024`) renders subfolders and documents within that path.
- **Collapsible Sidebar**: Hierarchical navigation tree reflecting the on-disk directory structure.
- **Section Indexes**: Dedicated overview pages for each section (`/docs`, `/writeups`, `/research`, `/notes`).
### 3. Dynamic Custom Fields
Content frontmatter supports arbitrary custom key-value pairs without schema modifications:
- **Automatic Storage**: Custom fields are persisted into a JSONB `extra_fields` column in PostgreSQL.
- **Type Preservation**: Numbers, booleans, arrays, objects, and strings maintain native types across parser, database, and API.
- **Value-Aware UI Badges**: The Web UI automatically styles badges based on value types and semantic patterns (difficulty levels, categories, tags, status) rather than hardcoded field names.
### 4. Revision History & Rollback
- **Smart Snapshotting**: Whenever a document body changes, a revision snapshot is created in `document_revisions`.
- **Deduplication**: Metadata-only updates do not produce duplicate body snapshots.
- **One-Click Restore**: Restoring any past revision writes the historic content back to disk and database, and automatically triggers semantic chunk re-indexing to ensure search consistency.
---
## Model Context Protocol (MCP)
MCPedia runs an MCP server accessible via **Stdio** (for local subagents) and **Streamable HTTP** (for remote agents over the network at `:4021`).
### MCP Tools (13 Total)
#### Read Tools (Public)
| Tool | Description |
|---|---|
| `search_documents` | Full-text keyword search over the knowledge base with headline snippets |
| `semantic_search` | Embedding-based cosine search across chunked content |
| `hybrid_search` | Fused full-text and semantic search via Reciprocal Rank Fusion |
| `get_document` | Fetches the full Markdown body of a document by slug |
| `list_documents` | Lists documents, optionally filtered by section or status |
| `get_related_documents` | Finds documents sharing tags with a given slug |
| `queue_status` | Returns current BullMQ indexing queue metrics |
#### Mutating & Admin Tools (Requires `x-webhook-secret`)
| Tool | Description |
|---|---|
| `create_document` | Creates a new document (writes Markdown file, database row, revision, and chunks) |
| `update_document` | Updates an existing document's body, metadata, or dynamic custom fields |
| `delete_document` | Removes a document file, database rows, revisions, and chunks |
| `index_document` | Enqueues a single-document background reindex job |
| `reindex_all` | Enqueues a full-corpus background reindexing job |
| `restore_revision` | Restores a document to a specific revision and rebuilds search chunks |
### MCP Resources
| Resource URI | Description | MIME Type |
|---|---|---|
| `mcpedia://docs` | List of all published documents in the knowledge base | `application/json` |
| `mcpedia://docs/{+slug}` | Full Markdown body of a single document from disk | `text/markdown` |
| `mcpedia://docs/{+slug}/chunks` | Preview of embedded semantic chunks for a document | `application/json` |
| `mcpedia://docs/{+slug}/revisions` | Summary of revision history for a document | `application/json` |
---
## HTTP, tRPC & Webhook Endpoints
The API application (`apps/api`) runs on **:4020** with Hono and tRPC v11:
### tRPC Procedures
- **Queries**: `search`, `semanticSearch`, `hybridSearch`, `getDocument`, `listDocuments`, `related`, `revisions`, `getRevision`, `jobStatus`, `queueStatus`.
- **Mutations** (Protected by `x-webhook-secret`): `createDocument`, `updateDocument`, `deleteDocument`, `restoreRevision`.
### Webhooks & Observability
- `POST /hooks/reindex` — Triggers a full corpus reindex (ideal for Git push webhooks).
- `POST /hooks/index?slug=<slug>` — Reindexes a single document.
- `GET /metrics` — Prometheus metrics exposition (`mcpedia_uptime_seconds`, `mcpedia_queue_jobs{state=...}`).
- `GET /dashboard` — Standalone real-time web dashboard for queue monitoring and live search.
- `GET /health` — Health check endpoint.
---
## Authentication & Security
MCPedia utilizes dual-tier security:
1. **Web UI Authentication**:
- Gated via `ADMIN_PASSWORD` in `.env`.
- Generates an HMAC-signed `mcpedia_admin` HttpOnly cookie upon login at `/login`.
- Protects document creation (`/create`), inline editing (`?edit=1`), and deletion.
2. **API & MCP Write Authentication**:
- Gated via `WEBHOOK_SECRET` in `.env`.
- Requires the `x-webhook-secret` HTTP header on mutating endpoints and tools.
- The API will refuse to start if `WEBHOOK_SECRET` is unset, preventing unsecured deployments.
---
## Quick Start
### Prerequisites
- [Bun](https://bun.sh/) (v1.1+)
- [PostgreSQL](https://www.postgresql.org/) (with `tsvector` support)
- [Redis](https://redis.io/) (for BullMQ async workers)
### 1. Installation & Environment Setup
```bash
bun install # install workspace deps
cp .env.example .env # set DATABASE_URL (dev uses imrnes Postgres :6432)
bunx turbo run build # typecheck + build every package
# Clone repository
git clone https://github.com/asepharyana/mcpedia.git
cd mcpedia
bun run index # walk content/ -> upsert into Postgres
bun --cwd apps/web run dev # Web UI on :3000
bun run mcp # MCP server on stdio (pipe to an MCP client)
# Install workspace dependencies
bun install
# Configure environment variables
cp .env.example .env
```
### Database
Ensure `.env` contains your database connection, Redis configuration, webhook secret, and admin password.
Schema is defined in `packages/db/src/schema.ts` (`documents` with a weighted
`search_vector` tsvector + GIN index, and `document_chunks` with an `embedding real[]`).
The `pgvector` extension is **not available** on the shared imrnes Postgres, so
semantic search stores vectors as `real[]` and ranks by in-app cosine similarity.
### 2. Database Setup
Migrations live in `packages/db/drizzle/`. They were applied manually via `psql`
(`drizzle-kit push` is unreliable under PgBouncer transaction pooling); to
re-apply on a fresh DB:
Apply migrations to initialize tables (`documents`, `document_chunks`, `document_revisions`):
```bash
psql $DATABASE_URL -f packages/db/drizzle/0000_grey_toro.sql
psql $DATABASE_URL -f packages/db/drizzle/0001_document_chunks.sql
psql $DATABASE_URL -f packages/db/drizzle/0002_document_revisions.sql
psql $DATABASE_URL -f packages/db/drizzle/0003_document_extra_fields.sql
```
> Note: on imrnes (PgBouncer `:6432`) a leaked `DATABASE_URL` shell var can
> shadow `.env`. `@mcpedia/config` loads `.env` **last** so the repo config
> always wins for local/dev.
### 3. Build & Index
## Content
```bash
# Typecheck and build all packages
bunx turbo run build
Each Markdown file carries YAML frontmatter:
# Run initial full indexation (parses content/, embeds chunks, writes DB)
bun run index
```
### 4. Running Services
```bash
# Start all development services (Web, API, MCP, Worker) concurrently
bun run dev
# Or run services individually:
bun --cwd apps/web run dev # Web UI on :3000 (or :4016 in production)
bun run api # Hono + tRPC API on :4020
bun run worker # BullMQ background worker
bun run mcp # MCP server (stdio mode)
bun run mcp:http # MCP server (Streamable HTTP mode on :4021)
```
---
## Scripts & CLI Reference
| Command | Action |
|---|---|
| `bun run index` | Run full synchronous reindexer over `content/` |
| `bun run enqueue --all` | Enqueue a full reindex job to Redis/BullMQ |
| `bun run enqueue <slug>` | Enqueue a single-document reindex job |
| `bun run worker` | Start the BullMQ background worker |
| `bun run api` | Start the Hono + tRPC API server |
| `bun run mcp` | Run MCP server in stdio mode |
| `bun run mcp:http` | Run MCP server in Streamable HTTP mode |
| `bun run chat` | Launch interactive MCP Client REPL |
| `bun run ask <cmd> [args]` | Run one-shot MCP client command |
| `bun run test` | Run the comprehensive test suite across all packages |
| `bun run typecheck` | Run TypeScript type checking across all workspaces |
---
## Content Format
Content files reside in `content/<section>/` as Markdown files with YAML frontmatter:
```yaml
---
id: websocket-contract
title: WebSocket Contract
id: sample-document
title: Sample Architecture Document
type: documentation
tags: [typescript, websocket, rpc]
section: docs
tags: [architecture, typescript, postgres]
status: published
author: asep
created_at: 2026-08-19
updated_at: 2026-08-19
created_at: 2026-08-20
updated_at: 2026-08-20
# Arbitrary custom fields (auto-discovered & styled):
difficulty: intermediate
points: 100
category: system-design
verified: true
---
# Sample Architecture Document
Your document content in standard GitHub Flavored Markdown (GFM).
Tables, code snippets, and custom sections are fully supported.
```
`slug` = relative path under `content/` (e.g. `docs/websocket/contract`). The
`body` shown in the UI is always read from the on-disk file (source of truth);
the DB stores metadata + the search vector.
---
## MCP tools
## Production Deployment
| Tool | Purpose |
| --------------------- | ------------------------------------------------ |
| `search_documents` | Postgres FTS over the corpus (ranked + snippet) |
| `semantic_search` | Embedding/cosine search over chunked content |
| `hybrid_search` | FTS + semantic fused via RRF |
| `get_document` | Full markdown body by slug |
| `list_documents` | List, optionally filtered by section |
| `get_related_documents` | Docs sharing tags with a given slug |
### MCP Resources
| URI | Purpose |
| -------------------------------- | ---------------------------------------- |
| `mcpedia://docs` | List all published documents |
| `mcpedia://docs/{+slug}` | Full markdown body (read from disk) |
| `mcpedia://docs/{+slug}/chunks` | Preview of embedded semantic chunks |
| `mcpedia://docs/{+slug}/revisions` | Revision history summary |
(`{+slug}` uses RFC 6570 reserved expansion so a slug like
`docs/websocket/contract` matches the template.)
Smoke test (in-memory transport, real JSON-RPC):
```bash
bun --cwd apps/mcp run smoke
```
## API (Phase 2 + Phase 3)
A tRPC v11 API is exposed via Hono on **:4020** (all procedures mirror the
MCP tools). Phase 3 adds async job + revision procedures and git-sync webhooks:
```bash
bun run api # http://localhost:4020 (GET /health, POST/GET /trpc/*)
```
tRPC procedures: `search`, `semanticSearch`, `hybridSearch`, `getDocument`,
`listDocuments`, `related` (Phase 2); plus `revisions`, `getRevision`,
`restoreRevision`, `jobStatus`, `queueStatus` (Phase 3).
Git-sync webhooks (enqueue BullMQ jobs; the worker processes them):
- `POST /hooks/reindex` — full-corpus reindex (point your Git provider's
push webhook here to auto-reindex on push).
- `POST /hooks/index?slug=<slug>` — reindex a single document.
> **Security:** both webhooks require an `x-webhook-secret` header that matches
> `WEBHOOK_SECRET` (set in `.env`). The API refuses to start if `WEBHOOK_SECRET`
> is unset, so the hooks are never left open.
`bun run index` now also chunks + embeds (Phase 2 indexer) and snapshots a
revision whenever the body changes (Phase 3). See `.env.example` for
`EMBED_*` / `REDIS_*` / `QUEUE_PREFIX` / `WEBHOOK_SECRET` vars.
### Run as a supervised service (Phase 4)
`deploy/mcpedia-api.service` + `deploy/mcpedia-worker.service` are systemd units
(`Restart=on-failure`, `EnvironmentFile=.env`, `WorkingDirectory=/home/code/mcpedia`).
Enable them with:
Production services run as supervised systemd units behind a Caddy reverse proxy:
```bash
# Copy systemd unit templates
sudo cp deploy/*.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now mcpedia-api mcpedia-worker
# tail logs
journalctl -u mcpedia-api -u mcpedia-worker -f
# Enable and start all supervised services
sudo systemctl enable --now mcpedia-web mcpedia-api mcpedia-worker mcpedia-mcp
# Inspect service logs
journalctl -u mcpedia-web -u mcpedia-api -u mcpedia-worker -u mcpedia-mcp -f
```
The API should sit behind Caddy (or your reverse proxy) for TLS; expose only
`:4020` internally and the web app publicly.
---
## Status
## Testing
**Phase 1 — MVP (DONE):** monorepo, Core, Web UI (home/doc/search), MCP server,
Postgres FTS keyword search, content indexing.
The test suite runs with in-memory mocks without requiring an active database or Redis instance:
**Phase 2 — Semantic + API (DONE):** embeddings provider (OpenRouter via 9router),
chunked `document_chunks`, `semanticSearch` + `hybridSearch` (RRF), tRPC/Hono API
(`apps/api`, :4020), MCP `semantic_search`/`hybrid_search` tools, web hybrid toggle.
```bash
bun run test
```
**Phase 3 — Async + Scale (DONE):** Redis + BullMQ background indexing/embedding
workers (`packages/queue`, `apps/worker`), git-sync webhooks (`POST /hooks/*`),
document revision system (`document_revisions` + restore), and MCP Resources
(`mcpedia://docs/...`). See `PHASES.md`.
> pgvector is **not installed** on the shared imrnes Postgres, so vector storage is
> a `real[]` column with in-app cosine similarity (instant at KB scale). pgvector is
> the Phase-4 scale-out path. See `PHASES.md`.
See `PHASES.md` for Phase 3–4 (Redis/BullMQ, auth, revisions, scale-out).
All 32+ unit and integration tests across packages and apps validate chunking, frontmatter parsing, cosine similarity, revision deduplication, auth gates, and route handlers.