feat(mcpedia): Phase 2 — semantic + hybrid search, tRPC/Hono API

- @mcpedia/embeddings: OpenRouter provider (9router /v1, encoding_format float),
  chunkText + embedChunks; EMBED_DIM=2048
- document_chunks table (real[] embedding) — pgvector NOT available on shared
  imrnes Postgres, so cosine is computed in-app (KB-scale fine); pgvector deferred
- indexer: chunk + embed + upsert per document
- @mcpedia/search: semanticSearch (cosine) + hybridSearch (FTS+cosine RRF)
- apps/api: Hono + tRPC v11 (6 procedures), serve on :4020
- MCP: semantic_search + hybrid_search tools (6 total)
- web: keyword/hybrid toggle; .env.example + README + PHASES updated
This commit is contained in:
asepharyana committed 2026-08-19 19:00:29 +07:00
1 parent ec3a2867d4
commit 9397303f01
32 files changed
+1120 -42

No files matched your search

+34 -7
View File
@@ -51,12 +51,18 @@ bun run mcp # MCP server on stdio (pipe to an MCP client)
### Database
Phase 1 uses Postgres FTS only. Schema is defined in `packages/db/src/schema.ts`
(a `documents` table with a `search_vector` generated `tsvector` column + GIN
index). Apply it with:
Schema is defined in `packages/db/src/schema.ts` (`documents` with a weighted
`search_vector` tsvector + GIN index, and `document_chunks` with an `embedding real[]`).
The `pgvector` extension is **not available** on the shared imrnes Postgres, so
semantic search stores vectors as `real[]` and ranks by in-app cosine similarity.
Migrations live in `packages/db/drizzle/`. They were applied manually via `psql`
(`drizzle-kit push` is unreliable under PgBouncer transaction pooling); to
re-apply on a fresh DB:
```bash
bunx --cwd packages/db drizzle-kit push
psql $DATABASE_URL -f packages/db/drizzle/0000_grey_toro.sql
psql $DATABASE_URL -f packages/db/drizzle/0001_document_chunks.sql
```
> Note: on imrnes (PgBouncer `:6432`) a leaked `DATABASE_URL` shell var can
@@ -84,11 +90,13 @@ updated_at: 2026-08-19
`body` shown in the UI is always read from the on-disk file (source of truth);
the DB stores metadata + the search vector.
## MCP tools (Phase 1)
## MCP tools
| Tool | Purpose |
| --------------------- | ------------------------------------------------ |
| `search_documents` | Postgres FTS over the corpus (ranked + snippet) |
| `semantic_search` | Embedding/cosine search over chunked content |
| `hybrid_search` | FTS + semantic fused via RRF |
| `get_document` | Full markdown body by slug |
| `list_documents` | List, optionally filtered by section |
| `get_related_documents` | Docs sharing tags with a given slug |
@@ -99,10 +107,29 @@ Smoke test (in-memory transport, real JSON-RPC):
bun --cwd apps/mcp run smoke
```
## API (Phase 2)
A tRPC v11 API is also exposed via Hono on **:4020** (all procedures mirror the
MCP tools):
```bash
bun run api # http://localhost:4020 (GET /health, POST/GET /trpc/*)
```
`bun run index` now also chunks + embeds (Phase 2 indexer). Requires `EMBED_*`
vars in `.env` (see `.env.example`).
## Status
**Phase 1 — MVP (DONE):** monorepo, Core, Web UI (home/doc/search), MCP server,
Postgres FTS keyword search, content indexing.
See `PHASES.md` for Phase 2–4 (pgvector semantic/hybrid search, tRPC/Hono API,
auth, Redis/BullMQ background workers, revisions, scale-out).
**Phase 2 — Semantic + API (DONE):** embeddings provider (OpenRouter via 9router),
chunked `document_chunks`, `semanticSearch` + `hybridSearch` (RRF), tRPC/Hono API
(`apps/api`, :4020), MCP `semantic_search`/`hybrid_search` tools, web hybrid toggle.
> pgvector is **not installed** on the shared imrnes Postgres, so vector storage is
> a `real[]` column with in-app cosine similarity (instant at KB scale). pgvector is
> the Phase-4 scale-out path. See `PHASES.md`.
See `PHASES.md` for Phase 3–4 (Redis/BullMQ, auth, revisions, scale-out).