feat(mcpedia): Phase 3 — async indexing (BullMQ), git-sync webhook, revisions, MCP Resources
- packages/queue: ioredis singleton + BullMQ Queue/Worker (prefix mcpedia:
on shared imrnes Redis :6379); apps/worker runs startWorker()
- @mcpedia/core: indexContentFile/runFullIndex (single indexing entry point
shared by script/worker/hook) + revision.service (list/get/restore)
- document_revisions table (migration 0002) — snapshots only on body change
- apps/api: POST /hooks/reindex + /hooks/index webhooks; tRPC revisions,
getRevision, restoreRevision, jobStatus, queueStatus
- apps/mcp: register MCP Resources mcpedia://docs{/,+slug/chunks/revisions}
({+slug} RFC6570 reserved expansion for slugs containing /)
- apps/mcp zod pinned to ^4 to match MCP SDK 1.30 compiled types
(resolves registerTool TS2589/ShapeOutput skew)
- scripts/enqueue.ts one-shot job enqueue helper; indexer refactored to runFullIndex
- PHASES.md/README/.env.example/docs updated
This commit is contained in:
@@ -27,12 +27,50 @@ Legend: ✅ built · 🟡 partial · ⬜ deferred
|
||||
- [x] MCP server — added `semantic_search` + `hybrid_search` tools (6 total).
|
||||
- [x] Web search — keyword/hybrid toggle (`?mode=hybrid`), hybrid reaches semantically-related docs keyword misses.
|
||||
|
||||
## Phase 3 — Async + Scale
|
||||
## Phase 3 — Async + Scale ✅ DONE
|
||||
|
||||
- [ ] Redis + BullMQ background indexing / embedding workers
|
||||
- [ ] Git synchronization hook (auto-reindex on push)
|
||||
- [ ] Document revision system (`document_revisions`)
|
||||
- [ ] MCP Resources (`mcpedia://docs/...`) in addition to tools
|
||||
- [x] **Redis + BullMQ background indexing / embedding workers** —
|
||||
`packages/queue` (ioredis singleton + BullMQ `Queue`/`Worker`, prefix
|
||||
`mcpedia:` on shared imrnes Redis `:6379`); `apps/worker` runs
|
||||
`startWorker()`. Three job types: `index-doc`, `index-all`, `reindex`.
|
||||
Single indexing entry point `indexContentFile`/`runFullIndex` in
|
||||
`@mcpedia/core` shared by the script, worker, and git hook. Verified
|
||||
end-to-end against live Redis (job enqueue → worker → Postgres write).
|
||||
- [x] **Git synchronization hook (auto-reindex on push)** — API webhook
|
||||
`POST /hooks/reindex` (full) and `POST /hooks/index?slug=` (single) enqueue
|
||||
BullMQ jobs. Wire a Git provider (GitHub/Gitea) post-receive / webhook to
|
||||
`POST /hooks/reindex` to auto-reindex on push. `scripts/enqueue.ts` is a
|
||||
one-shot enqueue helper (`bun run enqueue --all` / `<slug>`).
|
||||
- [x] **Document revision system (`document_revisions`)** — `packages/db`
|
||||
migration `0002_document_revisions.sql`. Indexer snapshots a revision only
|
||||
when the body actually changes vs the latest revision (pure metadata edits
|
||||
don't bloat history). `listRevisions` / `getRevision` / `restoreRevision`
|
||||
in `@mcpedia/core`; exposed as tRPC `revisions` / `getRevision` /
|
||||
`restoreRevision` and the `mcpedia://docs/{+slug}/revisions` MCP Resource.
|
||||
- [x] **MCP Resources (`mcpedia://docs/...`)** — alongside the 6 tools:
|
||||
`mcpedia://docs` (list), `mcpedia://docs/{+slug}` (body from disk),
|
||||
`mcpedia://docs/{+slug}/chunks` (chunk preview),
|
||||
`mcpedia://docs/{+slug}/revisions` (history). `{+slug}` uses RFC 6570
|
||||
reserved expansion so slugs containing `/` match.
|
||||
|
||||
### New/changed commands
|
||||
```
|
||||
bun run index # full reindex (runFullIndex, writes revisions)
|
||||
bun run enqueue --all # enqueue a full reindex job (no worker needed)
|
||||
bun run enqueue <slug> # enqueue a single-doc reindex job
|
||||
bun run worker # start the BullMQ indexing worker (long-running)
|
||||
bun run api # Hono+tRPC API on :4020 (added /hooks/* webhooks)
|
||||
```
|
||||
|
||||
### Verification done (real, against imrnes Redis + Postgres)
|
||||
- `turbo run typecheck` green across all 13 packages.
|
||||
- BullMQ e2e: enqueue `index-doc` → worker completes → `documents` +
|
||||
`document_chunks` + `document_revisions` rows present.
|
||||
- Revision dedup proven: editing a body creates a new revision; metadata-only
|
||||
reindex does not; `restoreRevision` writes history back into the live row.
|
||||
- MCP smoke test passes (tools + all 4 resources).
|
||||
- API webhook `POST /hooks/reindex` enqueues → worker drains queue →
|
||||
`queueStatus` reflects counts.
|
||||
|
||||
## Phase 4 — Scale-out (only if needed)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user