refactor(ai): remove semantic embedding cache + Qdrant vector store
Hapus seluruh fitur embedding/Qdrant (tidak dipakai lagi): - gateway: drop embeddingClient.ts, qdrantClient.ts, archiveEmbedder.ts dan tes qdrantEnsure.test.ts; moderationOrchestrator kembali ke exact-hash cache -> LLM (tanpa phase-2 semantic lookup); textCacheStore kehilangan findSimilarTextModeration / parseQdrantVerdict / isSemanticBandAccepted / upsertBareKeyToQdrant; cache-prune hanya menyapu Postgres. - backend: drop embed.ts + qdrant.ts, endpoint messages.semanticSearch dan schema/type terkait; kolom embedding dilepas dari schema text_analysis_cache. - frontend: hapus toggle EXACT/SEMANTIC, hook useSemanticSearch, API client + tipe SemanticSearchResult. - config: buang AI_LLM_EMBEDDING_* dan QDRANT_* (env + .env.example). - docs: ARCHITECTURE.md / AGENTS.md / README.md / diagram arsitektur disesuaikan (LLM caller - vision, cache = exact-hash saja). Verifikasi: tsc 0 (backend, gateway, frontend); bun test 135 pass + 37 pass, 0 fail; biome 0 error.
This commit is contained in:
@@ -53,19 +53,16 @@ src/
|
||||
1. **LLM is the only judge.** Failed LLM → `status:"error"` + recovery retry.
|
||||
**Never** reintroduce regex/heuristic content classification.
|
||||
2. **Discord tokens sanitized** before reaching LLM (`discordTokens.ts`).
|
||||
3. **Semantic cache is batched** — one embed call + one Qdrant batch search.
|
||||
4. **Streaming is mandatory** against the router base URL.
|
||||
3. **Streaming is mandatory** against the router base URL.
|
||||
|
||||
## AI moderation pipeline
|
||||
|
||||
```
|
||||
aiAnalyzer.ts → batchScheduler.ts → batchProcessor.ts → individualFallbackProcessor.ts
|
||||
↓ ↓ ↓ ↓
|
||||
moderationOrchestrator.ts → (hash cache → Qdrant → LLM)
|
||||
moderationOrchestrator.ts → (hash cache → LLM)
|
||||
↓ ↓ ↓
|
||||
textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts
|
||||
embeddingClient.ts
|
||||
qdrantClient.ts
|
||||
```
|
||||
|
||||
- Entry: `aiAnalyzer.ts` (`queueMessageAnalysis`, `startPendingAIAnalysisWorker`)
|
||||
@@ -85,7 +82,6 @@ textBatchProcessor.ts mediaBatchProcessor.ts llmClient.ts
|
||||
- `messageStore.ts` — DB operations
|
||||
- `messageMetadata.ts` — metadata extraction
|
||||
- `messagesDb.ts` / `messagesCrud.ts` — DB schema operations
|
||||
- `archiveEmbedder.ts` — Qdrant embedding (respect age-restricted guard)
|
||||
- `retentionDb.ts` / `reviewsDb.ts` / `attachmentsDb.ts` — auxiliary tables
|
||||
|
||||
## Redis channels (outbound to backend)
|
||||
|
||||
Reference in New Issue
Block a user