A content-first knowledge base — readable as Markdown in Git, queryable by humans via a modern Web UI, and accessible to AI agents via the Model Context Protocol (MCP).

MCPedia keeps content as plain Markdown files under content/ as a Git-tracked source of truth. Content is indexed into PostgreSQL (with a weighted tsvector full-text search column and chunked embeddings) and served through a unified Core layer that every interface (Web, MCP, API, Worker, CLI) shares.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│     Web UI      │    │   MCP Server    │    │  tRPC / Hono    │    │  BullMQ Worker  │
│   (Next.js 16)  │    │  (Stdio + HTTP) │    │   API (:4020)   │    │ (Async Queue)   │
└────────┬────────┘    └────────┬────────┘    └────────┬────────┘    └────────┬────────┘
         │                      │                      │                      │
         └──────────────────────┴──────────┬───────────┴──────────────────────┘
                                           ▼
                              ┌─────────────────────────┐
                              │     @mcpedia/core       │
                              │ (Unified Business Logic)│
                              └────────────┬────────────┘
                                           │
         ┌─────────────────────────────────┼─────────────────────────────────┐
         ▼                                 ▼                                 ▼
┌──────────────────┐             ┌──────────────────┐             ┌──────────────────┐
│ Markdown Files   │             │   @mcpedia/db    │             │ @mcpedia/search  │
│ (content/ tree)  │             │  (PostgreSQL)    │             │ (FTS + Cosine)   │
└──────────────────┘             └──────────────────┘             └──────────────────┘

Core Principles

Content as Source of Truth: Markdown files with YAML frontmatter under content/ are primary. The database holds metadata, search indices, chunk embeddings, and revision history.
Single Core Layer: All business logic (CRUD, indexing, search, revisions, path classification) is encapsulated in @mcpedia/core. Interfaces never touch the database directly.
Multi-Modal Search: Keyword search (Postgres FTS), semantic search (vector cosine similarity), and hybrid search (Reciprocal Rank Fusion / RRF) work out of the box.
Resilient Revision System: Content edits snapshot revisions automatically; metadata-only edits are deduplicated. Restoring a revision automatically rebuilds semantic search chunks.
Dual Interface: Full human-friendly web experience + first-class AI agent integration via MCP.

Monorepo Layout

mcpedia/
├── apps/
│   ├── web/          # Next.js 16 (Turbopack) — dark-mode UI, hierarchical doc tree, TOC, CRUD
│   ├── mcp/          # MCP server (stdio + Streamable HTTP on :4021) — 13 tools + 4 resources
│   ├── mcp-client/   # Interactive MCP CLI REPL, one-shot command runner, and TypeScript client SDK
│   ├── api/          # Hono + tRPC v11 API on :4020, git-sync webhooks, Prometheus metrics, dashboard
│   └── worker/       # Long-running BullMQ worker for async indexing and embedding jobs
├── packages/
│   ├── core/         # Document, Content, Search, Index, Revision, and Path services (the business logic)
│   ├── db/           # Drizzle ORM schema, client, and migrations
│   ├── parser/       # Frontmatter (gray-matter) parsing & stringification with dynamic field support
│   ├── search/       # Postgres FTS query (ts_rank + ts_headline), vector cosine, and RRF hybrid fusion
│   ├── embeddings/   # Chunking algorithms & embedding provider integrations
│   ├── queue/        # Redis (ioredis) client & BullMQ queue/worker definitions
│   ├── types/        # Shared TypeScript domain types and interfaces
│   └── config/       # Authoritative environment configuration loader (.env)
├── content/          # Markdown knowledge base organized by sections and nested folders
│   ├── docs/         # System and architectural documentation
│   ├── writeups/     # Technical writeups, CTF solutions, and debugging reports
│   ├── research/     # Research notes and evaluations
│   └── notes/        # Engineering notes and quick references
└── scripts/          # indexer.ts (full corpus reindexing), enqueue.ts (one-shot job enqueue)

Key Features

Search Mode	Mechanism	Best Used For
Keyword Search	PostgreSQL `tsvector` (`simple` config, weighted A/B) + GIN index + `ts_rank` + `ts_headline`	Exact terms, code symbols, error codes, identifiers
Semantic Search	Text chunking (1000 chars, 150 overlap) + vector embeddings + cosine similarity ranking	Conceptual questions, paraphrased queries, intent matching
Hybrid Search	Reciprocal Rank Fusion (`RRF = \sum \frac{1}{k + rank}`) fusing keyword and semantic signals	General-purpose search with optimal relevance

MCPedia supports nested folder structures (like GitHub repositories):

Dynamic Path Classification: The router inspects paths and automatically distinguishes between leaf documents and folder nodes containing subfolders or child documents.
Folder Index Pages: Navigating to any folder (e.g. /writeups/ctf/defcon-quals-2024) renders subfolders and documents within that path.
Collapsible Sidebar: Hierarchical navigation tree reflecting the on-disk directory structure.
Section Indexes: Dedicated overview pages for each section (/docs, /writeups, /research, /notes).

3. Dynamic Custom Fields

Content frontmatter supports arbitrary custom key-value pairs without schema modifications:

Automatic Storage: Custom fields are persisted into a JSONB extra_fields column in PostgreSQL.
Type Preservation: Numbers, booleans, arrays, objects, and strings maintain native types across parser, database, and API.
Value-Aware UI Badges: The Web UI automatically styles badges based on value types and semantic patterns (difficulty levels, categories, tags, status) rather than hardcoded field names.

4. Revision History & Rollback

Smart Snapshotting: Whenever a document body changes, a revision snapshot is created in document_revisions.
Deduplication: Metadata-only updates do not produce duplicate body snapshots.
One-Click Restore: Restoring any past revision writes the historic content back to disk and database, and automatically triggers semantic chunk re-indexing to ensure search consistency.

Model Context Protocol (MCP)

MCPedia runs an MCP server accessible via Stdio (for local subagents) and Streamable HTTP (for remote agents over the network at :4021).

MCP Tools (13 Total)

Read Tools (Public)

Tool	Description
`search_documents`	Full-text keyword search over the knowledge base with headline snippets
`semantic_search`	Embedding-based cosine search across chunked content
`hybrid_search`	Fused full-text and semantic search via Reciprocal Rank Fusion
`get_document`	Fetches the full Markdown body of a document by slug
`list_documents`	Lists documents, optionally filtered by section or status
`get_related_documents`	Finds documents sharing tags with a given slug
`queue_status`	Returns current BullMQ indexing queue metrics

Mutating & Admin Tools (Requires `x-webhook-secret`)

Tool	Description
`create_document`	Creates a new document (writes Markdown file, database row, revision, and chunks)
`update_document`	Updates an existing document's body, metadata, or dynamic custom fields
`delete_document`	Removes a document file, database rows, revisions, and chunks
`index_document`	Enqueues a single-document background reindex job
`reindex_all`	Enqueues a full-corpus background reindexing job
`restore_revision`	Restores a document to a specific revision and rebuilds search chunks

MCP Resources

Resource URI	Description	MIME Type
`mcpedia://docs`	List of all published documents in the knowledge base	`application/json`
`mcpedia://docs/{+slug}`	Full Markdown body of a single document from disk	`text/markdown`
`mcpedia://docs/{+slug}/chunks`	Preview of embedded semantic chunks for a document	`application/json`
`mcpedia://docs/{+slug}/revisions`	Summary of revision history for a document	`application/json`

HTTP, tRPC & Webhook Endpoints

The API application (apps/api) runs on :4020 with Hono and tRPC v11:

tRPC Procedures

Queries: search, semanticSearch, hybridSearch, getDocument, listDocuments, related, revisions, getRevision, jobStatus, queueStatus.
Mutations (Protected by x-webhook-secret): createDocument, updateDocument, deleteDocument, restoreRevision.

Webhooks & Observability

POST /hooks/reindex — Triggers a full corpus reindex (ideal for Git push webhooks).
POST /hooks/index?slug=<slug> — Reindexes a single document.
GET /metrics — Prometheus metrics exposition (mcpedia_uptime_seconds, mcpedia_queue_jobs{state=...}).
GET /dashboard — Standalone real-time web dashboard for queue monitoring and live search.
GET /health — Health check endpoint.

Authentication & Security

MCPedia utilizes dual-tier security:

Web UI Authentication:
- Gated via ADMIN_PASSWORD in .env.
- Generates an HMAC-signed mcpedia_admin HttpOnly cookie upon login at /login.
- Protects document creation (/create), inline editing (?edit=1), and deletion.
API & MCP Write Authentication:
- Gated via WEBHOOK_SECRET in .env.
- Requires the x-webhook-secret HTTP header on mutating endpoints and tools.
- The API will refuse to start if WEBHOOK_SECRET is unset, preventing unsecured deployments.

Quick Start

Prerequisites

Bun (v1.1+)
PostgreSQL (with tsvector support)
Redis (for BullMQ async workers)

1. Installation & Environment Setup

# Clone repository
git clone https://github.com/asepharyana/mcpedia.git
cd mcpedia

# Install workspace dependencies
bun install

# Configure environment variables
cp .env.example .env

Ensure .env contains your database connection, Redis configuration, webhook secret, and admin password.

2. Database Setup

Apply migrations to initialize tables (documents, document_chunks, document_revisions):

psql $DATABASE_URL -f packages/db/drizzle/0000_grey_toro.sql
psql $DATABASE_URL -f packages/db/drizzle/0001_document_chunks.sql
psql $DATABASE_URL -f packages/db/drizzle/0002_document_revisions.sql
psql $DATABASE_URL -f packages/db/drizzle/0003_document_extra_fields.sql

3. Build & Index

# Typecheck and build all packages
bunx turbo run build

# Run initial full indexation (parses content/, embeds chunks, writes DB)
bun run index

4. Running Services

# Start all development services (Web, API, MCP, Worker) concurrently
bun run dev

# Or run services individually:
bun --cwd apps/web run dev     # Web UI on :3000 (or :4016 in production)
bun run api                    # Hono + tRPC API on :4020
bun run worker                 # BullMQ background worker
bun run mcp                    # MCP server (stdio mode)
bun run mcp:http               # MCP server (Streamable HTTP mode on :4021)

Scripts & CLI Reference

Command	Action
`bun run index`	Run full synchronous reindexer over `content/`
`bun run enqueue --all`	Enqueue a full reindex job to Redis/BullMQ
`bun run enqueue <slug>`	Enqueue a single-document reindex job
`bun run worker`	Start the BullMQ background worker
`bun run api`	Start the Hono + tRPC API server
`bun run mcp`	Run MCP server in stdio mode
`bun run mcp:http`	Run MCP server in Streamable HTTP mode
`bun run chat`	Launch interactive MCP Client REPL
`bun run ask <cmd> [args]`	Run one-shot MCP client command
`bun run test`	Run the comprehensive test suite across all packages
`bun run typecheck`	Run TypeScript type checking across all workspaces

Content Format

Content files reside in content/<section>/ as Markdown files with YAML frontmatter:

---
id: sample-document
title: Sample Architecture Document
type: documentation
section: docs
tags: [architecture, typescript, postgres]
status: published
author: asep
created_at: 2026-08-20
updated_at: 2026-08-20
# Arbitrary custom fields (auto-discovered & styled):
difficulty: intermediate
points: 100
category: system-design
verified: true
---

# Sample Architecture Document

Your document content in standard GitHub Flavored Markdown (GFM).
Tables, code snippets, and custom sections are fully supported.

Production Deployment

Production services run as supervised systemd units behind a Caddy reverse proxy:

# Copy systemd unit templates
sudo cp deploy/*.service /etc/systemd/system/
sudo systemctl daemon-reload

# Enable and start all supervised services
sudo systemctl enable --now mcpedia-web mcpedia-api mcpedia-worker mcpedia-mcp

# Inspect service logs
journalctl -u mcpedia-web -u mcpedia-api -u mcpedia-worker -u mcpedia-mcp -f

Testing

The test suite runs with in-memory mocks without requiring an active database or Redis instance:

bun run test

All 32+ unit and integration tests across packages and apps validate chunking, frontmatter parsing, cosine similarity, revision deduplication, auth gates, and route handlers.

README.md

MCPedia