mcpedia/.hermes/plans/2026-08-19_phase2-semantic-api.md

MCPedia Phase 2 — Semantic Search + tRPC/Hono API

For Hermes: implement task-by-task. Spec-first (user rule 2026-08-19).

Goal: Add semantic + hybrid search (pgvector) and a typed tRPC/Hono API so MCPedia is queryable by embeddings, not just keyword FTS — and expose the corpus over a programmatic HTTP API.

Architecture: Content (Markdown) → chunk → embed (OpenRouter) → store document_chunks with vector(N) in Postgres → semanticSearch (cosine) and hybridSearch (FTS + cosine, reciprocal-rank fusion) in @mcpedia/search → exposed via Core, the MCP server (new tools), and a new apps/api (Hono + tRPC v11).

Embedding provider: OpenRouter (openrouter/llama-nemotron-embed-vl-1b-v2:free) via 9router_ai_llm_api_key + 9router_ai_llm_base_url (BWS). Dimension is discovered at first live call (see Step 1.3) and pinned in schema/migration.

Tech stack: drizzle-orm vector column + pgvector extension, HNSW index, @trpc/server v11 (fetch adapter), hono + @hono/node-server.

---

Task P2.1 — packages/embeddings (provider + abstraction)

Files: packages/embeddings/package.json, src/index.ts, src/provider.ts, src/openrouter.ts

• EmbeddingProvider interface: embed(texts: string[]): Promise<number[][]>, readonly model, readonly dimensions.
• OpenRouterEmbeddingProvider: POST ${baseUrl}/embeddings with { model, input }, Authorization: Bearer ${key}. Returns data[].embedding. Validate length === dimensions.
• Read EMBED_BASE_URL, EMBED_API_KEY, EMBED_MODEL from @mcpedia/config (with .env fallback). Dimensions discovered live (Step 1.3) → export EMBED_DIM.
• Chunk helper chunkText(text, { size=1000, overlap=150 }) in src/chunk.ts.

Step 1.3 (discover dim): live call embed(["test"]), read embedding.length, pin EMBED_DIM, assert mismatch throws.

Verify: bun run a temp script: embed(["hello world"]) prints a vector of length N (e.g. 1024). Confirm no key is logged.

---

Task P2.2 — Schema: document_chunks + vector extension

Files: packages/db/src/schema.ts (add), packages/db/drizzle.config.ts (unchanged), new migration.

• CREATE EXTENSION IF NOT EXISTS vector; (idempotent; run once via psql).
• document_chunks table:
  • id uuid pk default gen_random_uuid()
  • document_id text → documents.id on delete cascade
  • slug text (denormalized for convenience)
  • chunk_index integer
  • content text
  • embedding vector(EMBED_DIM)
  • created_at timestamp default now()
  • index chunk_embedding_idx using hnsw (embedding op vector_cosine_ops)
• Generate migration with drizzle-kit generate, apply via psql (drizzle-kit push is unreliable here — known).

Verify: \d document_chunks shows embedding vector(N) + HNSW index; select count(*) from document_chunks = 0.

---

Task P2.3 — Indexer: chunk + embed + upsert

Files: scripts/indexer.ts (extend), packages/core/src/document.service.ts (add indexChunks).

• For each published doc: read body (already on disk), chunkText, embed in batches (≤ 16), delete existing chunks for slug, insert new rows.
• Guard: if embedding provider fails, log + skip (don't crash the whole index).
• Add bun run index:embed (or extend bun run index to also embed).

Verify: after running, select count(*) from document_chunks > 0; a sample row has non-null embedding.

---

Task P2.4 — packages/search: semantic + hybrid

Files: packages/search/src/index.ts (add semanticSearch, hybridSearch).

• semanticSearch(vec, limit): order by embedding <=> ${vec} asc, filter published.
• hybridSearch(q, limit): run FTS (ts_rank) + semantic (cosine) in parallel; fuse with reciprocal-rank (RRF: score = 1/(k+rank), k=60); return merged hits.
• Keep keywordSearch unchanged (Phase 1).

Verify: unit-ish script: embed a query, semanticSearch returns relevant chunks; hybridSearch("websocket") returns ≥ keyword results.

---

Task P2.5 — packages/core expose semantic/hybrid

Files: packages/core/src/search.service.ts, index.ts.

• Re-export semanticSearch, hybridSearch from Core.

---

Task P2.6 — apps/api (Hono + tRPC v11)

Files: apps/api/package.json, tsconfig.json, src/index.ts, src/router.ts, src/trpc.ts.

• initTRPC.create() router with procedures: search, semanticSearch, hybridSearch, getDocument, listDocuments (mirrors MCP tools).
• Mount fetchRequestHandler on a Hono app at /trpc/*; serve via @hono/node-server serve({ fetch: app.fetch, port: 4020 }).
• createContext returns { db }.

Verify: bun run dev → curl -X POST localhost:4020/trpc/search with JSON body returns hits.

---

Task P2.7 — MCP server: semantic + hybrid tools

Files: apps/mcp/src/index.ts (add semantic_search, hybrid_search), extend smoke.test.ts.

• semantic_search: embed query → semanticSearch.
• hybrid_search: embed query → hybridSearch.
• Smoke: assert both return ≥1 hit for "websocket".

---

Task P2.8 — Web: semantic toggle on search

Files: apps/web/app/search/page.tsx.

• Add mode=keyword|hybrid query param; server component calls Core hybridSearch when mode=hybrid. Minimal UI toggle (link/buttons).
• Keep keyword as default.

Verify: bun run build; curl '/search?q=websocket&mode=hybrid' returns hits.

---

Task P2.9 — Verify all + commit

• bunx turbo run build (web + api + mcp), bun run apps/mcp smoke, live API curl, live web hybrid search.
• Update README.md + PHASES.md (mark Phase 2 ✅).
• git add -A (exclude .env), commit as asepharyana (no Co-Authored-By).

---

Risks / decisions

• Dimension unknown until live call → P2.1.3 discovers it; pinned EMBED_DIM=2048.
• pgvector NOT available on shared imrnes Postgres (extension not installed; installing needs host-level apt on a managed/shared DB — deferred). PIVOT: store embedding as real[] and compute cosine similarity in the app layer. Brute-force cosine is instant for a KB-sized corpus (dozens of docs / hundreds of chunks). pgvector+HNSW is the Phase-4 scale-out path.
• PgBouncer + real[]: fine; simple queries, no extension needed.
• API port 4020 (host 4000s range is 4000–4015; 4020 is free for dev). Deploy later.
• YAGNI: no auth/revisions this phase (Phase 3).