mcpedia/.hermes/plans/phase3-plan.md

MCPedia — Phase 3 "Async + Scale" Implementation Plan

Status: Phase 1 (MVP) + Phase 2 (Semantic+API) DONE. Phase 3 adds async background work, git-driven reindex, document revision history, and MCP Resources. All logic stays in @mcpedia/core; new packages/queue wires BullMQ; apps/worker runs the worker process; the existing API gets a git-sync webhook + job-status procedures; the MCP server gains Resources.

Scope (4 features from PHASES.md)

1. Redis + BullMQ background indexing/embedding workers
2. Git synchronization hook (auto-reindex on push via webhook)
3. Document revision system (document_revisions)
4. MCP Resources (mcpedia://docs/...) alongside existing tools

Architecture decisions (locked)

• Redis: shared imrnes Redis 100.121.180.82:6379, no auth (verified +PONG). REDIS_URL env (default redis://100.121.180.82:6379), optional REDIS_PASSWORD. BullMQ key prefix mcpedia: to avoid collisions on the shared instance.
• Queue lib: bullmq@6.1.2 + ioredis@6.0.0 (BullMQ peer dep). Pass an ioredis instance; BullMQ duplicates it for blocking commands.
• Single source of truth preserved: per-doc indexing logic moves into @mcpedia/core as indexContentFile(relPath, reason?). The script, the worker, and the git hook ALL call this. Revisions are snapshotted inside it.
• Revisions: created only when body actually changes vs the latest revision (avoids bloat on every sync). Stored in document_revisions.

Files touched

packages/config

• src/index.ts: add REDIS_URL, REDIS_PASSWORD, QUEUE_PREFIX.

packages/db

• src/schema.ts: add documentRevisions table (id, documentId→documents.id cascade, slug, revisionNo int, title, body, meta jsonb, reason text, createdAt). Index (document_id, revision_no DESC), (slug).
• drizzle/0002_document_revisions.sql: migration (applied via psql).
• drizzle/meta/0002_snapshot.json + _journal.json entry (keeps drizzle-kit consistent even though we apply manually).

packages/core (new)

• src/index.service.ts:
  • indexContentFile(relPath: string, reason = "index") — parse → upsert documents → indexChunks → snapshot revision (if changed).
  • runFullIndex(reason?) — walk content, index each, return counts.
• src/revision.service.ts:
  • createRevision(...), listRevisions(slug, limit), getRevision(id), latestRevisionBody(slug), restoreRevision(id).
• src/index.ts: export both.

packages/queue (NEW)

• package.json (@mcpedia/queue): deps bullmq, ioredis, @mcpedia/core, @mcpedia/db, @mcpedia/config.
• src/client.ts: ioredis instance factory from config.
• src/queue.ts:
  • INDEX_QUEUE = "mcpedia-index".
  • enqueueIndexDoc(slug, absPath, reason), enqueueFullIndex(reason).
  • getQueue() lazy singleton.
• src/worker.ts: startWorker() — BullMQ Worker with 3 job types: index-doc (single), index-all (full), reindex (full, reason=git-push). Graceful shutdown on SIGINT/SIGTERM. Job progress + error handling.

apps/worker (NEW)

• package.json (@mcpedia/worker): script start: bun src/index.ts.
• src/index.ts: startWorker() + heartbeat log.

apps/api

• src/index.ts: add POST /hooks/reindex (full) and POST /hooks/index?slug= (single) webhook routes → enqueue jobs. Mount AFTER /trpc.
• src/router.ts: add jobStatus (id→state/prev/failedData), queueStatus (waiting/active/completed/failed counts), revisions (slug→list), restoreRevision (id→new slug/doc).
• package.json: add @mcpedia/queue dep, hooks reused.

apps/mcp

• src/index.ts: register Resources:
  • mcpedia://docs (list all metas)
  • mcpedia://docs/{slug} (full body from disk)
  • mcpedia://docs/{slug}/chunks (chunk previews)
  • mcpedia://docs/{slug}/revisions (revision list)
• src/smoke.test.ts: add listResources + read mcpedia://docs assertion.

scripts

• scripts/indexer.ts: refactor main() to call runFullIndex().

Root

• package.json: add "worker": "bun --cwd apps/worker run start", "reindex": "bun run scripts/worker.ts"? No — worker runs the listener; triggering reindex = bun run api webhook or enqueueFullIndex helper. Add "enqueue-index": "bun run scripts/enqueue.ts" (one-shot enqueue).
• .env.example: add REDIS_URL, REDIS_PASSWORD, QUEUE_PREFIX.

Docs

• PHASES.md: mark Phase 3 items DONE with notes.
• README.md: document worker, webhook, revisions, MCP resources.

Verification (real, not claimed)

1. bun install picks up new deps.
2. bunx turbo run build + typecheck green across workspace.
3. Real BullMQ e2e against imrnes Redis: script that enqueues an index-doc job, starts a Worker, asserts the job completes and the doc row
   • chunks + a revision row appear in Postgres. Verifies Redis+ioredis+bullmq
   • db + core all wired correctly.
4. bun --cwd apps/mcp run smoke passes (incl. new resources).
5. bun run index (runFullIndex) green; verify documents, document_chunks, document_revisions row counts via psql.
6. API webhook: curl -XPOST localhost:4020/hooks/reindex enqueues; worker processes; curl localhost:4020/trpc/queueStatus reflects counts.
7. MCP resource read returns real content.