refactor(ai): remove semantic embedding cache + Qdrant vector store

Hapus seluruh fitur embedding/Qdrant (tidak dipakai lagi):

- gateway: drop embeddingClient.ts, qdrantClient.ts, archiveEmbedder.ts
  dan tes qdrantEnsure.test.ts; moderationOrchestrator kembali ke
  exact-hash cache -> LLM (tanpa phase-2 semantic lookup); textCacheStore
  kehilangan findSimilarTextModeration / parseQdrantVerdict /
  isSemanticBandAccepted / upsertBareKeyToQdrant; cache-prune hanya
  menyapu Postgres.
- backend: drop embed.ts + qdrant.ts, endpoint messages.semanticSearch
  dan schema/type terkait; kolom embedding dilepas dari schema
  text_analysis_cache.
- frontend: hapus toggle EXACT/SEMANTIC, hook useSemanticSearch,
  API client + tipe SemanticSearchResult.
- config: buang AI_LLM_EMBEDDING_* dan QDRANT_* (env + .env.example).
- docs: ARCHITECTURE.md / AGENTS.md / README.md / diagram arsitektur
  disesuaikan (LLM caller - vision, cache = exact-hash saja).

Verifikasi: tsc 0 (backend, gateway, frontend); bun test 135 pass +
37 pass, 0 fail; biome 0 error.
This commit is contained in:
mytheclipsebotreview
2026-09-25 01:38:51 +07:00
parent fdc7f01c26
commit 2b6ec59286
30 changed files with 41 additions and 1987 deletions
+2 -6
View File
@@ -53,19 +53,16 @@ src/
1. **LLM is the only judge.** Failed LLM → `status:"error"` + recovery retry.
**Never** reintroduce regex/heuristic content classification.
2. **Discord tokens sanitized** before reaching LLM (`discordTokens.ts`).
3. **Semantic cache is batched** — one embed call + one Qdrant batch search.
4. **Streaming is mandatory** against the router base URL.
3. **Streaming is mandatory** against the router base URL.
## AI moderation pipeline
```
aiAnalyzer.ts → batchScheduler.ts → batchProcessor.ts → individualFallbackProcessor.ts
↓ ↓ ↓ ↓
moderationOrchestrator.ts → (hash cache → Qdrant → LLM)
moderationOrchestrator.ts → (hash cache → LLM)
↓ ↓ ↓
textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts
embeddingClient.ts
qdrantClient.ts
```
- Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`, `startPendingAIAnalysisWorker`)
@@ -85,7 +82,6 @@ textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts
- `messageStore.ts` — DB operations
- `messageMetadata.ts` — metadata extraction
- `messagesDb.ts` / `messagesCrud.ts` — DB schema operations
- `archiveEmbedder.ts` — Qdrant embedding (respect age-restricted guard)
- `retentionDb.ts` / `reviewsDb.ts` / `attachmentsDb.ts` — auxiliary tables
## Redis channels (outbound to backend)