Commit Graph
8 Commits
Author SHA1 Message Date
asepharyana ca2bd19c48 fix(core): strip YAML frontmatter in getDocument before web/MCP render
getDocument preferred the on-disk file via readFileSync (raw), returning
the full frontmatter block as visible text on doc pages and MCP resources.
ReactMarkdown rendered the '---' delimiters as <hr/>, exposing YAML keys
(id/title/type/etc.) as plain paragraphs.

Root cause: getDocument bypassed parseFile's gray-matter stripping.
The indexer correctly stripped frontmatter (DB body was clean), but
getDocument read raw from disk for the 'source of truth' path.

Fix: use parseFile(abs, relPath).body (same as the indexer) so the
on-disk path returns frontmatter-free markdown. DB fallback unchanged.

Affects web doc pages + MCP mcpedia://docs/{+slug} and /chunks resources.
Verified: frontmatter leakage 0/9 doc pages, headings render as <h2>,
browser snapshot clean.
2026-08-20 11:45:37 +07:00
asepharyana 76f778c10d chore(testing): Phase 9 — bun:test suite + CI gating with no-DB fakes
CI / typecheck + build (turbo) (push) Canceled after 0s
Added a real test suite (32 tests, 0 external services) using bun:test with
in-process module mocking for @mcpedia/db, @mcpedia/queue, @mcpedia/core.

Enablers:
- apps/api: extracted createApp(deps?) factory + dashboard.ts module from
  index.ts so the HTTP surface is unit-testable (real queue is lazy-imported).
- packages/core: exported shouldCreateRevision pure predicate; restoreRevision
  gained an opts.reindex seam for the chunk-rebuild contract.
- apps/mcp: renamed smoke.test.ts -> smoke.ts (bun test now owns .test.ts),
  updated stale assertions (10 tools, 4 docs in docs section).
- infra: turbo test task (cache:false), test scripts across packages,
  @types/bun + tsconfig base types, CI 'Test' step after Build.

Packages with tests: embeddings(5), parser(5), search(8), core(4),
mcp(6 auth-gates), api(8 contracts).

All green: typecheck(4/4), test(6/6 pkgs), build(web). Live API verified
/health, /metrics, /dashboard, /hooks/* auth gate on temp port.
2026-08-20 11:12:32 +07:00
asepharyana b454a0fbac fix(embeddings): defer EMBED_* validation to embed() runtime, not constructor
Constructor threw if EMBED_BASE_URL/API_KEY/MODEL unset, which broke next
build SSG collection (imports search pkg before .env exists). Validation
now happens in embed() so the package is build-safe; a missing config still
errors clearly when a semantic/hybrid search actually runs.

With the DATABASE_URL import-time throw also removed, the whole monorepo
now builds in CI without any .env present.
2026-08-19 22:52:05 +07:00
asepharyana f4fb43c1a5 fix(config): don't throw on missing DATABASE_URL at import time
CI / typecheck + build (turbo) (push) Canceled after 0s
The import-time guard broke 'next build' (SSG data collection imports
@mcpedia/config before any .env exists) and any runtime-injected env.
postgres.js connects lazily on first query, so a missing DATABASE_URL now
surfaces as a clear connect error at runtime instead of a cryptic build
failure. Lets CI build without .env present.
2026-08-19 22:49:03 +07:00
asepharyana b92f6f91fa feat(mcpedia): Phase 4 — operability + correctness hardening
Reinterpreted from the plan's YAGNI 'Scale-out' (OpenSearch/object-storage
/multi-tenant deferred at KB scale). Phase 4 = make the Phase 3 async +
revision system correct, secure, observable, deployable.

- T1 (correctness bug): restoreRevision now rebuilds semantic chunks via new
  @mcpedia/core reindexChunks(slug) so semantic/hybrid search stay consistent
  after a restore (previously document_chunks held the NEW body while
  documents.body held the restored OLD body -> stale search).
- T2 (security): /hooks/* git-sync webhooks now require x-webhook-secret header
  matching WEBHOOK_SECRET (401 otherwise); API fails fast at startup if unset.
  Added WEBHOOK_SECRET to @mcpedia/config + .env.example; set real secret in .env.
- T3 (UX): web doc page shows a History panel (revision no/reason/date/length)
  with per-revision Restore; app/api/revisions/restore/route.ts calls
  restoreRevision + revalidatePath (server-component only, no client JS).
- T4: listRevisions gains offset paging; summary never includes body.
- T5 (ops): deploy/mcpedia-api.service + deploy/mcpedia-worker.service systemd
  units (Restart=on-failure, EnvironmentFile=.env). Not auto-enabled on host.

Verified against live imrnes Redis + Postgres: turbo typecheck+build green;
restore-rebuilds-chunks (marker present -> gone after restore); webhook 401/200;
web restore route redirects to doc + reverts body; revisions API returns summary
(no body); systemd-analyze verify passes.
2026-08-19 21:54:54 +07:00
asepharyana 8f2229d447 feat(mcpedia): Phase 3 — async indexing (BullMQ), git-sync webhook, revisions, MCP Resources
- packages/queue: ioredis singleton + BullMQ Queue/Worker (prefix mcpedia:
  on shared imrnes Redis :6379); apps/worker runs startWorker()
- @mcpedia/core: indexContentFile/runFullIndex (single indexing entry point
  shared by script/worker/hook) + revision.service (list/get/restore)
- document_revisions table (migration 0002) — snapshots only on body change
- apps/api: POST /hooks/reindex + /hooks/index webhooks; tRPC revisions,
  getRevision, restoreRevision, jobStatus, queueStatus
- apps/mcp: register MCP Resources mcpedia://docs{/,+slug/chunks/revisions}
  ({+slug} RFC6570 reserved expansion for slugs containing /)
- apps/mcp zod pinned to ^4 to match MCP SDK 1.30 compiled types
  (resolves registerTool TS2589/ShapeOutput skew)
- scripts/enqueue.ts one-shot job enqueue helper; indexer refactored to runFullIndex
- PHASES.md/README/.env.example/docs updated
2026-08-19 20:18:28 +07:00
asepharyana 9397303f01 feat(mcpedia): Phase 2 — semantic + hybrid search, tRPC/Hono API
- @mcpedia/embeddings: OpenRouter provider (9router /v1, encoding_format float),
  chunkText + embedChunks; EMBED_DIM=2048
- document_chunks table (real[] embedding) — pgvector NOT available on shared
  imrnes Postgres, so cosine is computed in-app (KB-scale fine); pgvector deferred
- indexer: chunk + embed + upsert per document
- @mcpedia/search: semanticSearch (cosine) + hybridSearch (FTS+cosine RRF)
- apps/api: Hono + tRPC v11 (6 procedures), serve on :4020
- MCP: semantic_search + hybrid_search tools (6 total)
- web: keyword/hybrid toggle; .env.example + README + PHASES updated
2026-08-19 19:00:29 +07:00
asepharyana ec3a2867d4 feat(mcpedia): Phase 1 MVP — monorepo, Core, Web UI, MCP server, Postgres FTS
- bun workspaces + Turborepo monorepo (apps/web, apps/mcp; packages/*, scripts)
- @mcpedia/core single business-logic layer (Document/Content/Search services)
- @mcpedia/db Drizzle schema: documents + weighted tsvector (GIN) for FTS
- @mcpedia/parser frontmatter, @mcpedia/search Postgres FTS (ts_rank+ts_headline)
- Next.js 16 Web UI (home/doc SSG, search dynamic) + react-markdown render
- MCP server (stdio) with 4 tools + in-memory smoke test
- scripts/indexer walks content/ -> upserts into Postgres
- 4 seed docs; README + PHASES status
2026-08-19 17:40:14 +07:00