perf(ai-moderation): naikkan cache hit dgn guard akurasi
- Fase-1 exact-cache lookup: N query serial -> SATU query ANY($1::text[]) - Global reuse utk bare key legacy, HANYA verdict non-actionable (clean/flagless/action=none, conf>=0.85, umur<=72h) — flagged/warn tetap context-scoped - Semantic cache dua-band: clean band 0.92 default, actionable tetap 0.97; di antara band -> LLM (fail-open ke akurasi) - hit_count kini di-increment (bulk UPDATE per batch) -> hit-rate terukur - Cache hasil wikipediaSearch di Redis (6h, hanya hasil non-kosong) - Memoize fetchUrlSafely utk type=text (LRU 30m + in-flight dedupe) - makeImageCacheKey strip query CDN Discord (?ex/is/hm, format/width) -> attachment sama = satu key vision, skip re-download+re-vision Spec: .hermes/plans/2026-08-24-ai-analysis-cache-optimization.md Tests: +33 (cacheGuards, discordImageKeyNormalize, cacheBatchLookup)
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# AI Analysis Flow — Audit & Optimization (discord-gateway)
|
||||
|
||||
**Goal:** Analisis alur AI analysis end-to-end, temukan bug/inconsistency yang merusak kualitas verdict, lalu perbaiki root cause-nya.
|
||||
|
||||
## Scope
|
||||
- `services/discord-gateway/src/modules/ai-moderation/**`
|
||||
- Tidak menyentuh chatbot backend / frontend.
|
||||
|
||||
## Alur saat ini (hasil tracing)
|
||||
```
|
||||
message capture → aiAnalyzer.queueMessageAnalysis(messageId)
|
||||
→ batchScheduler.scheduleConversationAnalysis(conversationKey) [debounce 250ms, CB gate]
|
||||
→ messageStore.getPendingMessagesByConversation(≤200)
|
||||
→ skipAgeRestrictedMessages
|
||||
→ pickBatchWithinBudget(14000 tokens, 50/msg)
|
||||
→ processBatch [Piscina worker, ≤4 threads]
|
||||
→ ai-analysis-worker.processBatch
|
||||
→ getConversationContextBefore(20 msgs) + attachments
|
||||
→ attachment-upload race guard (pending upload → skip)
|
||||
→ runModerationAnalysis
|
||||
→ Phase 1: exact-hash cache (PG text_analysis_cache, per channel/thread)
|
||||
→ Phase 2: semantic cache (embedTexts → Qdrant batch search; PG fallback)
|
||||
→ split text-only vs media
|
||||
→ runTextOnlyBatch: URL fetch + wiki search + glossary (paralel)
|
||||
→ dedup short messages → sub-batches (60/sub-batch)
|
||||
→ vision evidence utk URL images (hoisted, 15s cap per image)
|
||||
→ callModerationLLM per sub-batch (stream:true, retries 3, JSON parse + correction retry)
|
||||
→ runMediaBatch: download → vision per image (cache LRU→DB→live, lock) → 1 LLM call
|
||||
→ setCachedTextModeration (PG + Qdrant upsert w/ embedding)
|
||||
→ normalizeResult (confidence clamp, fallback analysis)
|
||||
→ updateMessagesAIAnalysisBulk → broadcast + scheduleAutoDelete
|
||||
→ recovery worker tiap 10s: pending keys → re-schedule; incomplete → individual fallback queue
|
||||
→ individual fallback: 1 msg = 1 worker job (context + full LLM)
|
||||
→ cache prune tiap 6 jam (PG expired + Qdrant expired points)
|
||||
```
|
||||
|
||||
## Temuan audit (ranked)
|
||||
|
||||
### F1 — Cache hit menghapus status "warn" (BUG AKURASI)
|
||||
`moderationOrchestrator.ts` Phase-2 semantic hit & PG-fallback memetakan status via
|
||||
`parseQdrantVerdict`: storedStatus bukan "warn"/"flagged" → dipaksa "clean".
|
||||
TAPI exact-hash lookup (`getCachedTextModeration`, textCacheStore.ts:288-295) lebih parah:
|
||||
hanya menerima "clean"|"flagged" — **"warn" jatuh ke branch flags.length===0 ? clean : flagged**
|
||||
→ warn dengan flags=["conflict_instigation"] dibaca sebagai FLAGGED.
|
||||
Efek: auto-delete eligibility (butuh recommendedAction delete/escalate + severity list) salah baca;
|
||||
dashboard menampilkan flagged padahal verdict asli warn. Root cause: type narrowing legacy
|
||||
(`status: "clean" | "flagged"`) tidak diupdate ketika "warn" ditambahkan ke schema.
|
||||
|
||||
### F2 — Exact-cache key mengabaikan edit (BUG EVASION)
|
||||
Key = sha256(content)+context. Pesan yang DIEDIT (`edited_content`) menghasilkan hash berbeda,
|
||||
tapi verdict lama utk konten pre-edit tetap hidup; lebih penting: pesan edited="true" adalah sinyal
|
||||
evasion di prompt, sedangkan cache bisa menyajikan verdict dari konten lama jika content sama.
|
||||
(Minor, tapi konsistensi: `resolveIsEdited` ada di prompt, tidak ada di cache key.)
|
||||
|
||||
### F3 — `pickBatchWithinBudget` skip-bukan-break (LATENSI/KUALITAS)
|
||||
Loop `if (usedTokens + msgTokens <= maxTokens) {push}` — pesan BESAR di tengah list dilewati
|
||||
dan iterasi lanjut mencoba msg berikutnya. Efek: batch berisi "lubang" (msg pending tetap pending,
|
||||
dianalisis di gelombang berikutnya = LLM call tambahan). Ini by-design tolerable, tapi ada bug halus:
|
||||
pesan >budget tunggal tidak pernah masuk (scheduler sudah punya fallback slice(0,1), OK).
|
||||
Keputusan: biarkan (bukan bug nyata), catat saja.
|
||||
|
||||
### F4 — `callModerationLLM` max_tokens 16384 hardcoded (COST)
|
||||
Sub-batch 60 pesan × output ~150 token/pesan ≈ 9k token cukup; 16k aman. Biarkan.
|
||||
|
||||
### F5 — Dead code builder user-profile/reputation
|
||||
`buildUserProfilesBlock`, `buildUserProfileRef`, `UserProfileEntry` di moderationBuilders.ts
|
||||
tidak dipakai lagi sejak context minimization (hanya tests). `<user_history>` juga tak pernah
|
||||
di-inject (rules masih menyebutnya — misleading bagi model). Bersihkan referensi prompt.
|
||||
|
||||
### F6 — rules.ts menyebut `<user_history>` yang tidak pernah ada di payload
|
||||
Model diberi instruksi tentang blok yang tak pernah muncul → pemborosan token + potensi
|
||||
kelakuan aneh ("menunggu" data yang tak ada). Hapus/ubah kalimat.
|
||||
|
||||
### F7 — system.ts "Blok Data" menyebut `<term_glossary> (SearXNG)` — STALE
|
||||
Sumber sudah Wikipedia. Komentar kode & teks prompt menyebut SearXNG. Perbaiki teks (kecil).
|
||||
|
||||
### F8 — output.ts typo "secifik", baris tabel `-|-` rusak
|
||||
Kualitas prompt: typo + markdown table broken (`||-`) di beberapa baris. Rapikan.
|
||||
|
||||
### F9 — llmCaller parse-error correction tail hanya di SYSTEM
|
||||
Correction tail ditambahkan ke system prompt; provider caching fine, tapi preview invalid
|
||||
content (800 char) ikut SYSTEM — ok. Skip.
|
||||
|
||||
### F10 — `getLlmSemaphore` race kecil saat config berubah di tengah flight
|
||||
Non-issue praktis (config statis per proses). Skip.
|
||||
|
||||
## Keputusan perbaikan (yang dieksekusi sekarang)
|
||||
1. **F1 (utama):** normalisasi status di SATU tempat — `normalizeStoredStatus()` di
|
||||
textCacheStore.ts yang menerima clean/warn/flagged; pakai di getCachedTextModeration
|
||||
DAN parseQdrantVerdict; perluas return types ke union penuh. Orchestrator tinggal pakai.
|
||||
2. **F6+F7+F8:** bersihkan stale references di prompts (user_history, SearXNG, typo).
|
||||
3. **F5:** hapus dead builders + test-nya (biome/tsc yang jaga).
|
||||
4. Regression test untuk F1 (vitest): warn tersimpan → warn terbaca (exact + qdrant path).
|
||||
|
||||
## Files touched
|
||||
- services/discord-gateway/src/modules/ai-moderation/textCacheStore.ts (F1)
|
||||
- services/discord-gateway/src/modules/ai-moderation/moderationOrchestrator.ts (type only)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/rules.ts (F6)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/system.ts (F7)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/output.ts (F8)
|
||||
- services/discord-gateway/src/modules/ai-moderation/moderationBuilders.ts (F5)
|
||||
- services/discord-gateway/tests/contextEnrichment.test.ts (F5 test cleanup + F1 regression test baru)
|
||||
|
||||
## Verification
|
||||
```
|
||||
cd services/discord-gateway
|
||||
npx tsc --noEmit
|
||||
npx biome check --diagnostic-level=error .
|
||||
npx vitest run
|
||||
```
|
||||
Semua harus hijau sebelum commit. Deploy via GHA (push main) — user konfirmasi belakangan.
|
||||
@@ -0,0 +1,127 @@
|
||||
# Spec: Optimasi AI Analysis GMW — Naikkan Cache Hit Tanpa Kehilangan Akurasi
|
||||
|
||||
Tanggal: 2026-08-24 · Repo: `~/GMW` (branch `main`) · Service: `services/discord-gateway`
|
||||
|
||||
## Latar & Evidence (audit 2026-08-24)
|
||||
|
||||
State produksi:
|
||||
- Qdrant `gmw_text_moderation`: **1.550 poin, status green** (vectors size 2048, Cosine).
|
||||
- PG `text_analysis_cache`: 1.634 row `user_moderation`, 277 `vision_llm`; **sum(hit_count) = 0** →
|
||||
hit-rate tidak pernah terukur.
|
||||
- Embedding aktif (`AI_LLM_EMBEDDING_MODEL` set, Nemotron-embed, dim 2048), `AI_LLM_EMBEDDING_MIN_SIMILARITY`
|
||||
tidak diset di BWS → default **0.97** (sangat konservatif).
|
||||
- Messages: 9.375 total; 643 status `error` (banyak retry), 49 pending.
|
||||
|
||||
Temuan audit alur (`moderationOrchestrator.ts` → `textCacheStore.ts` → `qdrantClient.ts`,
|
||||
`textBatchProcessor.ts`, `urlFetcher.ts`, `wikipediaClient.ts`, `visionAnalyzer.ts`):
|
||||
|
||||
| # | Temuan | Dampak |
|
||||
|---|--------|--------|
|
||||
| F1 | Exact-hash cache key menyertakan context (channel/thread) → teks sama di channel lain selalu miss | Killer hit-rate #1 |
|
||||
| F2 | Semantic tier TIDAK memfilter context (Qdrant payload tak punya context) — sudah global tapi hanya aman krn sim 0.97 ketat | Inkonsisten dgn exact tier |
|
||||
| F3 | Phase-1 lookup loop `await getCachedTextModeration(key)` per pesan → N round-trip PgBouncer per batch (60 msg = 60 query serial) | Latensi + beban DB |
|
||||
| F4 | Verdict actionable (flagged/warn) dan clean sama-sama boleh di-serve semantic; toleransi akurasi beda | Risiko akurasi |
|
||||
| F5 | `hit_count` tidak pernah di-increment oleh reader manapun | Hit-rate tak terukur |
|
||||
| F6 | `wikipediaSearch()` (blok `<web_searches>`) tanpa cache — re-fetch tiap batch utk query sama | Latensi + spam ke WP |
|
||||
| F7 | `fetchUrlSafely()` tanpa cache — link sama di batch berikutnya di-download lagi penuh | Latensi + bandwidth |
|
||||
| F8 | Vision cache key dari data-URL base64 hasil resize → attachment sama via jalur berbeda (URL vs embed) = key beda → re-download + re-vision | Duplikasi kerja vision |
|
||||
|
||||
Non-goals: mengubah pipeline enforcement (auto-mute/ban trust-store writes), mengubah prompt
|
||||
kebijakan moderasi, mengubah model/embedding provider.
|
||||
|
||||
## Desain
|
||||
|
||||
Semua perubahan degrade gracefully — cache gagal → perilaku lama (LLM). Akurasi dilindungi
|
||||
asimetris: **hemat boleh untuk verdict non-actionable, konservatif untuk yang memicu aksi.**
|
||||
|
||||
### D1 — Cache metrics (F5)
|
||||
- `textCacheStore.getCachedTextModeration()`: saat hit valid, increment `hit_count`
|
||||
(`UPDATE ... SET hit_count = hit_count + 1`) fire-and-forget (`.catch(()=>{})`), jangan blokir return.
|
||||
- Log info periodik ringkas di orchestrator sudah ada ("User moderation cache applied") — cukup.
|
||||
|
||||
### D2 — Batched exact-cache lookup (F3)
|
||||
- Fungsi baru `getCachedTextModerations(keys: string[]): Promise<Map<string, StoredModerationVerdict>>`
|
||||
di `textCacheStore.ts`: **satu** `SELECT ... WHERE text = ANY($1)` (chunk 200 key/query),
|
||||
parse + `normalizeStoredStatus` per row (reuse helper existing).
|
||||
- Orchestrator fase-1: kumpulkan semua key unik → satu call batched → distribusi hasil.
|
||||
- Semantik identik dengan loop lama (row expired/error-artifact tetap miss); hanya jumlah round-trip
|
||||
yang turun N→1.
|
||||
|
||||
### D3 — Global exact reuse untuk verdict non-actionable (F1)
|
||||
- Key scoped-context TETAP ditulis (kompatibel, invalidasi moderator tetap presisi).
|
||||
- Reader tambahan: kalau key `<ctx>:<hash>` miss, coba key legacy global `text_mod:<hash>` (bare).
|
||||
- Guard akurasi (WAJIB semua terpenuhi):
|
||||
- `status === "clean"` DAN `flags.length === 0`;
|
||||
- `confidence >= AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE` (default 0.85);
|
||||
- `recommendedAction === "none"`;
|
||||
- umur entry ≤ `AI_CACHE_GLOBAL_REUSE_MAX_AGE_H` (default 72h) — cek `analyzed_at`.
|
||||
- Flag baru `policyVersion: "cached-global-clean-2026-08"` supaya terlacak di dashboard/log.
|
||||
- Verdict flagged/warn TETAP context-scoped (tidak pernah lintas channel).
|
||||
|
||||
### D4 — Semantic dua-band similarity (F2+F4)
|
||||
- Config baru: `AI_LLM_EMBEDDING_MIN_SIMILARITY_ACTIONABLE` default **0.97** (perilaku lama),
|
||||
`AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN` default **0.92**, keduanya coerce number 0..1.
|
||||
- Satu Qdrant batch search pakai threshold RENDAH (0.92). Per hit, klasifikasi ulang:
|
||||
- verdict non-actionable (clean, no flags, action=none): terima jika `score >= CLEAN_BAND`;
|
||||
- verdict actionable (warn/flagged atau flags ada / action != none): terima hanya jika
|
||||
`score >= ACTIONABLE_BAND` (0.97 — persis gate lama);
|
||||
- di antara dua band → buang hit, pesan lanjut ke LLM (fail-open ke akurasi).
|
||||
- Legacy PG fallback path: filter serupa di `findSimilarTextModeration` via parameter band.
|
||||
|
||||
### D5 — Cache Wikipedia search (F6)
|
||||
- `wikipediaClient.wikipediaSearch(query)`: cek `cacheGet(makeCacheKey("wikisearch", q))` dulu;
|
||||
miss → fetch (timeout existing) → sukses & hasil non-kosong → `cacheSet(..., TTL 6h)`.
|
||||
Hasil kosong TIDAK di-cache (biar retry nanti). Redis down → langsung fetch (no-op cache).
|
||||
|
||||
### D6 — Cache URL text fetch (F7)
|
||||
- `urlFetcher.fetchUrlSafely(url)`: wrapper async memoize in-process LRU (max 500, TTL 30 menit)
|
||||
untuk `type === "text"` saja (image tetap selalu fresh-download karena dipakai sbg bukti vision
|
||||
+ buffer besar; error tidak di-cache).
|
||||
- Import `LRUCache` dari `lru-cache` (sudah dep gateway).
|
||||
|
||||
### D7 — Unified vision cache key (F8)
|
||||
- `makeImageCacheKey(imageUrl)` di `textCacheStore.ts`: sebelum hash, strip query Discord CDN
|
||||
(`?ex=&is=&hm=` signed tokens, `format/width/height/size`) — regex `(\?[^#]*)$` dibuang bila host
|
||||
CDN discord (`cdn.discordapp.com`, `media.discordapp.net`, `images-ext-*.discordapp.net`);
|
||||
URL non-Discord: hash full URL seperti sekarang.
|
||||
- Efek: attachment sama yang lolos lewat jalur embed vs inline vs re-fetch dgn token beda → SATU
|
||||
entry cache → skip download+vision kedua kali. Data-URL base64 tetap di-hash apa adanya.
|
||||
|
||||
## File yang disentuh
|
||||
|
||||
1. `src/shared/config/index.ts` — 3 config baru (D3×2, D4×2 — total 4 nilai, 3 baris zod + deskripsi).
|
||||
2. `src/modules/ai-moderation/textCacheStore.ts` — hit_count inc (D1), batched getter (D2),
|
||||
global-reuse guard helper (D3), image-key normalize (D7).
|
||||
3. `src/modules/ai-moderation/moderationOrchestrator.ts` — pakai batched getter (D2),
|
||||
global bare-key fallback (D3), dua-band semantic accept (D4).
|
||||
4. `src/modules/ai-moderation/qdrantClient.ts` — `searchQdrantBatch` menerima threshold rendah
|
||||
(sudah parametrik — mungkin tanpa perubahan; verifikasi).
|
||||
5. `src/modules/ai-moderation/wikipediaClient.ts` — cache layer (D5).
|
||||
6. `src/modules/ai-moderation/urlFetcher.ts` — LRU text-fetch memoize (D6).
|
||||
|
||||
## Schema/type changes
|
||||
|
||||
- Tidak ada migrasi DB (kolom `hit_count`, `analyzed_at`, `expires_at` sudah ada).
|
||||
- Tidak ada perubahan kontrak WS/oRPC/frontend.
|
||||
- Type baru: none public; internal `StoredModerationVerdict` dipakai ulang.
|
||||
|
||||
## Verification
|
||||
|
||||
1. Unit tests baru (`tests/`):
|
||||
- `cacheBatchLookup.test.ts`: batched getter — hit/miss/expired/error-artifact mapping,
|
||||
chunking >200 keys (mock executeAll), hit_count increment called.
|
||||
- `globalReuseGuard.test.ts`: guard menerima clean+conf≥0.85+action none+umur ≤72h;
|
||||
menolak flagged/warn/conf rendah/action≠none/stale.
|
||||
- `semanticBands.test.ts`: clean @0.93 diterima, flagged @0.93 ditolak, flagged @0.98 diterima.
|
||||
- `imageKeyNormalize.test.ts`: URL Discord dgn/ex token → key sama; non-Discord beda query → beda.
|
||||
2. Gate service: `pnpm typecheck && pnpm exec biome check --diagnostic-level=error . && pnpm exec vitest run`.
|
||||
3. Deploy via GHA (`git push origin main`) → watch `Build & Deploy (Nix)` → verifikasi
|
||||
`systemctl show gmw-discord-gateway -p ActiveEnterTimestamp` baru.
|
||||
4. Runtime probe pasca-deploy: journalctl level 30 normal; beberapa jam kemudian
|
||||
`SELECT sum(hit_count) FROM text_analysis_cache WHERE source='user_moderation'` > 0 membuktikan
|
||||
metrics jalan; log "User moderation cache applied" menunjukkan hits>0 pada traffic ramai.
|
||||
|
||||
## Rollback
|
||||
|
||||
Semua fitur behind config defaults yang mempertahankan perilaku lama pada nilai konservatif;
|
||||
rollback = redeploy commit sebelumnya (tanpa migrasi DB, tanpa state eksternal).
|
||||
@@ -0,0 +1,205 @@
|
||||
# GMW FE Rombak — "Constellation Ops" Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
> Setiap task = edit → lint → build/test → smoke → commit (author `asepharyana`, tanpa Co-Authored-By).
|
||||
|
||||
**Goal:** Rombak total FE GMW dari template dashboard klasik menjadi **Constellation Node Graph** — graf node interaktif (three.js force layout) di mana channel/event/moderasi adalah node bintang dan navigasi adalah terbang antar node — untuk semua 9 route sekaligus.
|
||||
|
||||
**Architecture:** Hapus total shell template (TopBar + NavRail + MobileNav + main scroll container). Ganti dengan satu `ConstellationStage`: canvas full-bleed fixed di belakang seluruh app yang me-render force-directed graph per-route (scene), plus overlay HTML mengambang (tanpa card box standar) untuk konten detail. Router tetap Next App Router; perpindahan route = kamera "fly-to" ke scene baru. Data tetap lewat pola SSR seed (`page.tsx` server fetch → `view.tsx` client + SWR fallbackData) — tidak ada endpoint baru yang diarankan.
|
||||
|
||||
**Tech Stack:** Next.js 16 App Router · React 19 · TypeScript strict · Tailwind v4 (token oklch existing) · three r180 (+ `@types/three`) · d3-force (+ `@types/d3-force`) untuk simulasi layout deterministik · motion v12 (transisi antar-scene) · biome (0 error **dan** 0 warning) · bun:test untuk modul murni.
|
||||
|
||||
---
|
||||
|
||||
## Keputusan Terkunci (2026-08-24, user MythEclipse)
|
||||
|
||||
| Keputusan | Pilihan user |
|
||||
|---|---|
|
||||
| Cakupan | **Semua 9 route sekali jalan** (dashboard, voice, media, recordings, moderation, messages, analysis, channels, glossary) |
|
||||
| Arah struktural | **Constellation Node Graph** — navigasi = terbang antar node |
|
||||
| Tema | **Pertahankan toggle light/dark** (jangan rebuild sistem tema — harden saja) |
|
||||
|
||||
## Kondisi Saat Ini (fakta repo, 2026-08-24)
|
||||
|
||||
- Repo: `/home/code/GMW/services/frontend` (monorepo GMW, deploy Nix-first via GHA "Build & Deploy (Nix)" ke `imphnen.asepharyana.my.id`; nginx gmw-proxy :4009 → Next standalone :4017).
|
||||
- Shell aktif: `src/components/shell/ambient-app.tsx` → `AppFrame` = NavRail + TopBar + `<main>` scroll + MobileNav + MiniPlayer, dipakai SEMUA route via `(dashboard)/layout.tsx`. AmbientCanvas (233 l) masih ada sebagai background di belakang chrome template.
|
||||
- 9 route di `src/app/(dashboard)/`: dashboard (368 l), voice (311), media (293), moderation (424), messages (611), analysis (196), recordings (201), channels (23), glossary (15).
|
||||
- `next.config.ts`: `output: "standalone"` + `trailingSlash: true` → **nav via `router.push`/Link bisa no-op; pakai `<a href>` polos** (pitfall terverifikasi).
|
||||
- Token desain ada di `globals.css` `@theme`: `--color-canvas/-2`, `--color-ink/-soft/-faint`, `--color-signal(+glow)`, amber, vermilion, hairline; font Bricolage Grotesque/JetBrains Mono; `.light` class override + next-themes sudah jalan — JANGAN dibangun ulang.
|
||||
- Sudah ada & dipertahankan: CommandPalette (⌘K), Chatbot, MiniPlayer, primitives (GlassPanel/GlassCard/Badge/…), shared states+skeletons, `lib/format.ts`, `lib/ai-status.ts`, WS typed events (`message_created/updated/deleted/analyzed/snapshot*`, `voice_state`, dll).
|
||||
- Tooling lokal: `pnpm lint|build|format`, `pnpm exec biome check --write --unsafe .` buat import order; bun 1.3.14 tersedia untuk unit test modul murni.
|
||||
- Smoke test SELALU di port non-prod (mis. 4024), cek `ss -ltnp | grep <port>` dulu; JANGAN pernah bind 4017.
|
||||
|
||||
## Struktur Target
|
||||
|
||||
```
|
||||
src/components/shell/
|
||||
constellation-frame.tsx # pengganti AppFrame: tanpa topbar/rail/bottom-bar
|
||||
constellation-stage.tsx # canvas three.js full-bleed + hit-test + kamera
|
||||
route-scenes.ts # map pathname → scene config (nodes/edges/focus)
|
||||
floating-chrome.tsx # brand mark + status dot + route switcher minim + palette hint
|
||||
(hapus setelah migrasi): topbar.tsx, nav-rail.tsx, mobile-nav.tsx, ambient-app.tsx*
|
||||
src/lib/constellation/
|
||||
graph.ts # tipe GraphNode/GraphEdge + builder dari data API (murni)
|
||||
layout.ts # wrapper d3-force deterministik (seeded) + fallback radial (murni)
|
||||
camera.ts # interpolasi kamera fly-to (murni, testable)
|
||||
*.test.ts # bun:test utk modul murni di atas
|
||||
```
|
||||
|
||||
*ambient-canvas/context dievaluasi: bisa di-refactor jadi bagian dari stage atau dihapus.
|
||||
|
||||
### Scene per route (bahasa visual satu kesatuan)
|
||||
|
||||
| Route | Scene |
|
||||
|---|---|
|
||||
| `/dashboard/` | Guild = bintang pusat; channel = node orbit; event WS = pulse edge; metric cluster floating (tanpa box) |
|
||||
| `/channels/` | Semua channel sebagai konstelasi; klik channel = fly-to + panel detail mengambang |
|
||||
| `/moderation/` | Pesan flagged = node merah (vermilion) mengelilingi hub AI-verdict; LiveModerationFeed jadi ribbon ticker |
|
||||
| `/messages/` | Stream pesan = sabuk orbital; node baru masuk lewat animasi dari tepi |
|
||||
| `/voice/` | Stage node di tengah; speaker aktif = node mengorbit dengan glow intensitas |
|
||||
| `/media/` | Galeri spiral node (thumbnail via div bg-image); klik = MiniPlayer handoff |
|
||||
| `/recordings/` | Variasi spiral media, timeline density sebagai cincin |
|
||||
| `/analysis/` | Hub analitik; chart existing dirender sebagai panel translusen mengambang ter-anchor ke node |
|
||||
| `/glossary/` | Term KB = cincin satelit; search memfilter node real-time |
|
||||
|
||||
### Checklist anti-default (wajib lolos sebelum commit akhir)
|
||||
|
||||
- [ ] Tidak ada top bar / nav rail / side panel / bottom prompt bar
|
||||
- [ ] Tidak ada card grid standar sebagai struktur utama
|
||||
- [ ] Layout beda SPASIAL dari versi lama (bukan ganti warna)
|
||||
- [ ] Verifikasi visual sungguhan: browser_navigate + browser_vision → "bukan template dashboard"
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Phase 0 — Fondasi
|
||||
|
||||
#### Task 1: Dependencies + scaffold direktori
|
||||
**Files:** `package.json`, create `src/lib/constellation/`
|
||||
1. `cd /home/code/GMW/services/frontend && pnpm add three @types/three d3-force @types/d3-force && pnpm add -d @types/bun`
|
||||
2. Buat 4 file kosong bertipe sesuai struktur target (ekspor placeholder bertipe agar tsc lolos).
|
||||
3. Run: `pnpm exec tsc --noEmit` → expected PASS.
|
||||
4. Commit: `chore(frontend): scaffold constellation lib + deps`.
|
||||
|
||||
> Catatan: jika `@types/bun` bentrok dengan tsconfig Next (duplikat globals), fallback: exclude `"**/*.test.ts"` dari `tsconfig.json` dan jalankan test hanya via `bun test` (tanpa typecheck). Putuskan saat eksekusi, dokumentasikan di commit body.
|
||||
|
||||
#### Task 2 (TDD): Modul murni graph + layout + camera
|
||||
**Files:** `src/lib/constellation/{graph,layout,camera}.ts` + `*.test.ts`
|
||||
1. **Tulis test dulu** (bun:test):
|
||||
- `graph.test.ts`: builder `dashboardToGraph()` menghasilkan node guild pusat + N channel + edges benar dari fixture `/api/dashboard/stats|channels` shape (lihat AGENTS.md; nama channel di `message.metadata.channelName`).
|
||||
- `layout.test.ts`: `computeLayout(nodes, edges, {seed:42})` DETERMINISTIK (dua pemanggilan = posisi identik), semua posisi finite, radius dalam bound; `radialLayout()` fallback untuk reduced-motion.
|
||||
- `camera.test.ts`: `flyTo(from,to,t)` easing monotonic, t=0→from, t=1→to.
|
||||
2. Run: `bun test src/lib/constellation` → expected FAIL (belum diimplementasi).
|
||||
3. Implementasi minimal sampai PASS.
|
||||
4. Run: `bun test src/lib/constellation && pnpm exec tsc --noEmit` → PASS.
|
||||
5. Commit: `feat(frontend): constellation graph/layout/camera pure modules (tested)`.
|
||||
|
||||
### Phase 1 — Shell Baru
|
||||
|
||||
#### Task 3: `ConstellationStage` (canvas renderer)
|
||||
**Files:** `src/components/shell/constellation-stage.tsx`
|
||||
- `<canvas>` fixed inset-0 `-z-10`; three.js orthographic 2D-ish render node+edge (sprite/glow shader sederhana — jangan over-engineer; titik + garis + glow cukup).
|
||||
- Interaksi: drag = pan, wheel/pinch = zoom, hover = highlight node + tooltip title, click node ber-`href` = navigasi via `<a>` semantics (window.location assign — hindari router.push no-op).
|
||||
- DPR-aware resize; pause render loop saat `document.hidden`; `prefers-reduced-motion` → render statis tanpa animasi.
|
||||
- Sampling warna dari CSS var (`getComputedStyle`) + `MutationObserver` pada `documentElement.classList` agar ikut toggle dark/light.
|
||||
- Verify: `pnpm lint && pnpm build` PASS → commit `feat(frontend): constellation stage canvas renderer`.
|
||||
|
||||
#### Task 4: `route-scenes.ts` + `ConstellationFrame`
|
||||
**Files:** `src/components/shell/{route-scenes,constellation-frame,floating-chrome}.tsx`
|
||||
- `route-scenes.ts`: `usePathname()` → config scene (builder mana, focus node, overlay slots).
|
||||
- `ConstellationFrame`: render `ConstellationStage` + `children` (overlay HTML absolute, BUKAN flex column bertingkat) + `floating-chrome` (brand mark kiri-atas dgn status dot WS, switcher route minimal kanan-bawah, hint ⌘K). MiniPlayer + Chatbot + CommandPalette tetap termounting.
|
||||
- Switcher & semua nav internal pakai `<a href="/<route>/">` polos (trailingSlash!). Verifikasi klik di browser nanti, bukan cuma baca kode.
|
||||
- Wire `(dashboard)/layout.tsx`: ganti `AppFrame` → `ConstellationFrame` (provider Ambient/WS tetap).
|
||||
- Smoke: `ss -ltnp | grep 4024` kosong → `PORT=4024 pnpm start` → curl semua 9 route = 200 → kill server.
|
||||
- Commit: `feat(frontend): replace classic shell with constellation frame`.
|
||||
|
||||
#### Task 5: Hapus chrome lama + bersih-bersih
|
||||
**Files:** hapus `shell/topbar.tsx`, `shell/nav-rail.tsx`, `shell/mobile-nav.tsx`, update `shell/index.ts`; eval `ambient-*` (refactor jadi layer stage ATAU hapus).
|
||||
- `grep -rn "TopBar\|NavRail\|MobileNav\|AppFrame" src/` → 0 referensi tersisa.
|
||||
- `pnpm exec biome check --write --unsafe .` → import order rapi.
|
||||
- Commit: `refactor(frontend): remove classic dashboard chrome`.
|
||||
|
||||
### Phase 2 — Migrasi 9 Route (batch, tiap batch ship hijau)
|
||||
|
||||
Urutan sengaja dari yang paling graph-native ke paling padat konten:
|
||||
|
||||
#### Task 6: `/dashboard/` + `/channels/`
|
||||
- Dashboard view: guild-star + channel orbit + WS pulse; metric cluster floating (reuse MetricTile tapi tanpa card-box — ubah jadi teks besar + hairline).
|
||||
- Channels: konstelasi channel, klik = fly-to + panel detail (data dari hook `use-channels`/server fetch existing — jangan karang endpoint).
|
||||
- Lint+build+smoke 2 route → commit `feat(frontend): dashboard & channels constellation scenes`.
|
||||
|
||||
#### Task 7: `/glossary/` + `/analysis/`
|
||||
- Glossary: cincin satelit term + search filter node (client-side filter pada node label).
|
||||
- Analysis: hub node + panel chart translusen mengambang (reuse komponen charts/* existing, anchor posisi ke node).
|
||||
- Commit: `feat(frontend): glossary & analysis scenes`.
|
||||
|
||||
#### Task 8: `/messages/` + `/moderation/`
|
||||
- Messages: sabuk orbital stream (reuse sortMessages/logika WS `use-messages.ts` — jangan duplikasi); pesan baru animate-in dari tepi.
|
||||
- Moderation: flagged nodes vermilion + hub verdict; LiveModerationFeed jadi ticker ribbon bawah (bukan panel).
|
||||
- Commit: `feat(frontend): messages & moderation scenes`.
|
||||
|
||||
#### Task 9: `/voice/` + `/media/` + `/recordings/`
|
||||
- Voice: speaker aktif orbit dgn glow (seed dari snapshot `activeSpeakers` server-authoritative — pertahankan perilaku useSpeakers).
|
||||
- Media/Recordings: spiral galeri; thumbnail pakai div background-image (hindari `noImgElement`; kalau terpaksa `<img>` → `// biome-ignore lint/performance/noImgElement: <alasan>`); MiniPlayer handoff tidak boleh regress.
|
||||
- Commit: `feat(frontend): voice, media & recordings scenes`.
|
||||
|
||||
### Phase 3 — Polish, A11y, Tema
|
||||
|
||||
#### Task 10: Motion & micro-interaction
|
||||
- Fly-to antar scene pakai `motion` v12 (interpolasi kamera dari `lib/constellation/camera.ts`).
|
||||
- Overlay: stagger masuk (pattern `animate-stagger` + `staggerDelay(i)` existing); tambah semua `animate-*` baru ke kill-list `prefers-reduced-motion` di `globals.css`.
|
||||
- Commit: `feat(frontend): scene transitions + motion polish`.
|
||||
|
||||
#### Task 11: A11y & fallback semantik
|
||||
- Setiap scene punya fallback semantik: `<nav aria-label="Routes">` visually-hidden berisi link semua route + `<ul>` node (label + href) yang ter-sync dgn graph state.
|
||||
- Keyboard: Tab ke daftar route; panah memindahkan fokus antar node; Enter aktivasi; `focus-visible` ring pakai `--color-ring`.
|
||||
- Kontras ink-on-canvas dicek (WCAG AA utk teks overlay).
|
||||
- Hooks-after-early-return pitfall: compute flags dulu, guard di dalam useEffect.
|
||||
- Commit: `feat(frontend): a11y semantic fallback for constellation`.
|
||||
|
||||
#### Task 12: Hardening tema light/dark
|
||||
- Pastikan canvas sampling var `.light` benar saat toggle (MutationObserver dari Task 3); harden `.light`: `color-scheme: light`, scrollbar/selection, glow opacity turun.
|
||||
- JANGAN menyentuh sistem next-themes.
|
||||
- Commit: `fix(frontend): light theme parity for constellation stage`.
|
||||
|
||||
### Phase 4 — Gerbang Ship (non-negotiable)
|
||||
|
||||
#### Task 13: Verifikasi penuh + deploy
|
||||
1. `pnpm lint` → **Found 0 errors. Found 0 warnings** (warning pun harus bersih).
|
||||
2. `bun test src/lib/constellation` → all PASS; `pnpm exec tsc --noEmit` PASS; `pnpm build` sukses.
|
||||
3. Smoke: port 4024 bersih → start → curl 9 route (dgn trailing slash) semua **200** → kill server.
|
||||
4. `git status` audit: hanya file scope yang ter-commit (`git checkout --` file liar hasil auto-format).
|
||||
5. Push `origin/main` → `gh run watch` "Build & Deploy (Nix)" → hijau.
|
||||
6. Live: curl `https://imphnen.asepharyana.my.id/<route>/` × 9 → 200.
|
||||
7. **Verifikasi visual sungguhan**: browser_navigate ke live URL → browser_vision: "Apakah ini masih template dashboard?" — jawaban HARUS bukan; screenshot dikirim ke user (`MEDIA:<path>`).
|
||||
8. Checklist anti-default dicentang satu per satu di PR/commit message akhir.
|
||||
|
||||
---
|
||||
|
||||
## Files Likely to Change
|
||||
|
||||
```
|
||||
package.json, pnpm-lock.yaml
|
||||
next.config.ts # hanya jika perlu (tidak diharapkan)
|
||||
src/app/(dashboard)/layout.tsx # wire frame baru
|
||||
src/app/(dashboard)/{9 route}/page.tsx|view.tsx # scene migration
|
||||
src/components/shell/** # frame baru + hapus lama
|
||||
src/lib/constellation/** # baru (murni + test)
|
||||
src/app/globals.css # tokens tambahan + reduced-motion kill list
|
||||
biome.json # hanya jika rule intentional baru perlu "off"
|
||||
```
|
||||
|
||||
## Risks / Tradeoffs
|
||||
|
||||
| Risiko | Mitigasi |
|
||||
|---|---|
|
||||
| Perf GPU di device lemah (full-page WebGL) | Node count kecil (<200/route), pause saat tab hidden, fallback radial statis utk reduced-motion; jika tetap berat → downgrade ke Canvas2D renderer dgn API sama |
|
||||
| router.push no-op (standalone+trailingSlash) | Semua nav pakai `<a href>`; verifikasi klik nyata di browser |
|
||||
| Konten padat (messages 611 l) tak muat di metafora graf | Prinsip: graf = struktur/navigasi; DETAIL tetap lewat overlay panel mengambang — konten tidak dikorbankan, hanya wadahnya yang berubah |
|
||||
| Bun test + Next tsconfig konflik types | Exclude `*.test.ts` dari tsconfig (keputusan Task 1) |
|
||||
| Regress fitur existing (theme toggle, palette, chatbot, mini-player, WS sync) | Dipertahankan apa adanya; smoke + klik verifikasi per phase; sortMessages & snapshot voice TIDAK disentuh logikanya |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Tidak ada blocker. Satu preferensi opsional: apakah brand mark perlu jam/uptime seperti TopBar lama — default: tidak (minimal). Keputusan bisa diambil saat review Task 4.
|
||||
@@ -20,11 +20,16 @@ import { isQdrantConfigured, searchQdrantBatch } from "./qdrantClient.js";
|
||||
import { logCacheEvent } from "./responseLogger.js";
|
||||
import { runTextOnlyBatch } from "./textBatchProcessor.js";
|
||||
import {
|
||||
bumpTextModerationHitCounts,
|
||||
ERROR_ARTIFACT_FLAGS,
|
||||
findSimilarTextModeration,
|
||||
getCachedTextModeration,
|
||||
getCachedTextModerations,
|
||||
isGloballyReusableCleanVerdict,
|
||||
isSemanticBandAccepted,
|
||||
makeModerationContextKey,
|
||||
makeTextModerationCacheKey,
|
||||
parseQdrantVerdict,
|
||||
type StoredModerationVerdict,
|
||||
setCachedTextModeration,
|
||||
} from "./textCacheStore.js";
|
||||
|
||||
@@ -73,46 +78,74 @@ export async function runModerationAnalysis(
|
||||
initCacheStore(config.REDIS_URL);
|
||||
if (!targets.length) throw new Error("No targets provided for analysis");
|
||||
|
||||
// ── Phase 1: exact-hash cache (per conversation context) ────────────────
|
||||
// ── Phase 1: exact-hash cache — ONE batched DB query ────────────────────
|
||||
// Key is content + conversation context (channel/thread). On a scoped miss
|
||||
// we also probe the legacy bare key: verdicts that CANNOT trigger an action
|
||||
// (clean, flagless, action=none) may be reused across channels under strict
|
||||
// freshness + confidence guards — flagged/warn verdicts never leave their
|
||||
// conversation. This replaced the old N-sequential-query loop (60-message
|
||||
// burst = 60 PgBouncer round-trips before).
|
||||
const cacheHits: AnalysisResult[] = [];
|
||||
const uncachedTargets: MessageRecord[] = [];
|
||||
// cacheKey → result for identical-content dedupe within one batch
|
||||
// cacheKey → representative result for identical-content dedupe
|
||||
const hitByKey = new Map<string, AnalysisResult>();
|
||||
// Embedding per exact cache key — computed once during lookup, reused
|
||||
// when the fresh LLM verdict is written back to the semantic cache.
|
||||
const embeddingsByKey = new Map<string, number[]>();
|
||||
|
||||
interface ExactCandidate {
|
||||
target: MessageRecord;
|
||||
scopedKey: string;
|
||||
bareKey: string;
|
||||
}
|
||||
const candidates: ExactCandidate[] = [];
|
||||
for (const target of targets) {
|
||||
const hasMedia = hasMediaContent(target, attachments);
|
||||
if (hasMedia) {
|
||||
if (hasMediaContent(target, attachments)) {
|
||||
uncachedTargets.push(target);
|
||||
continue;
|
||||
}
|
||||
|
||||
const rawContent = target.edited_content ?? target.content;
|
||||
if (!rawContent.trim()) {
|
||||
uncachedTargets.push(target);
|
||||
continue;
|
||||
}
|
||||
|
||||
const cacheKey = makeTextModerationCacheKey(
|
||||
candidates.push({
|
||||
target,
|
||||
scopedKey: makeTextModerationCacheKey(
|
||||
rawContent,
|
||||
makeModerationContextKey(target),
|
||||
);
|
||||
const seen = hitByKey.get(cacheKey);
|
||||
if (seen) {
|
||||
// Same content already resolved this batch — reuse the verdict.
|
||||
cacheHits.push({ ...seen, messageId: target.id });
|
||||
continue;
|
||||
),
|
||||
bareKey: makeTextModerationCacheKey(rawContent),
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
const cached = await getCachedTextModeration(cacheKey);
|
||||
if (cached) {
|
||||
// Identical content within one batch resolves once (representative).
|
||||
const firstByScopedKey = new Map<string, ExactCandidate>();
|
||||
for (const c of candidates) {
|
||||
if (!firstByScopedKey.has(c.scopedKey))
|
||||
firstByScopedKey.set(c.scopedKey, c);
|
||||
}
|
||||
|
||||
// Single round-trip for every key we might serve from (scoped + bare).
|
||||
const storedEntries = await getCachedTextModerations([
|
||||
...firstByScopedKey.keys(),
|
||||
...Array.from(firstByScopedKey.values(), (c) => c.bareKey),
|
||||
]);
|
||||
// Keys actually served — bumped in one UPDATE at the end for metrics.
|
||||
const servedCacheKeys = new Set<string>();
|
||||
|
||||
/** Validate + admit one stored verdict for a candidate. */
|
||||
const acceptExactVerdict = (
|
||||
candidate: ExactCandidate,
|
||||
cacheKey: string,
|
||||
entry: { verdict: StoredModerationVerdict },
|
||||
policyVersion: string,
|
||||
): boolean => {
|
||||
const { verdict } = entry;
|
||||
const hasMediaInMeta =
|
||||
target.metadata &&
|
||||
candidate.target.metadata &&
|
||||
(() => {
|
||||
const ev = extractMessageMediaEvidence(target.metadata);
|
||||
const ev = extractMessageMediaEvidence(candidate.target.metadata);
|
||||
return (
|
||||
ev.attachments.length > 0 ||
|
||||
ev.stickers.length > 0 ||
|
||||
@@ -122,48 +155,87 @@ export async function runModerationAnalysis(
|
||||
|
||||
if (hasMediaInMeta) {
|
||||
log.debug(
|
||||
{ messageId: target.id, cacheKey },
|
||||
{ messageId: candidate.target.id, cacheKey },
|
||||
"Cache entry but message has media — treating as miss",
|
||||
);
|
||||
} else if (
|
||||
cached.flags.some((f) =>
|
||||
[
|
||||
"analysis_api_failed",
|
||||
"analysis_parse_failed",
|
||||
"analysis_incomplete",
|
||||
].includes(f),
|
||||
return false;
|
||||
}
|
||||
if (
|
||||
verdict.flags.some((f) =>
|
||||
(ERROR_ARTIFACT_FLAGS as readonly string[]).includes(f),
|
||||
)
|
||||
) {
|
||||
log.warn(
|
||||
{ messageId: target.id, cacheKey },
|
||||
{ messageId: candidate.target.id, cacheKey },
|
||||
"Cache entry contains error artifact — treating as miss",
|
||||
);
|
||||
} else {
|
||||
const hit: AnalysisResult = {
|
||||
messageId: target.id,
|
||||
status: cached.status,
|
||||
flags: cached.flags,
|
||||
score: cached.score,
|
||||
analysis: cached.analysis,
|
||||
categories: cached.categories,
|
||||
severity: cached.severity as AnalysisResult["severity"],
|
||||
confidence: cached.confidence,
|
||||
recommendedAction:
|
||||
cached.recommendedAction as AnalysisResult["recommendedAction"],
|
||||
policyVersion: "cached-user-moderation-2026-06",
|
||||
evidence: [],
|
||||
};
|
||||
cacheHits.push(hit);
|
||||
hitByKey.set(cacheKey, hit);
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* proceed */
|
||||
return false;
|
||||
}
|
||||
|
||||
uncachedTargets.push(target);
|
||||
hitByKey.set(candidate.scopedKey, {
|
||||
messageId: candidate.target.id,
|
||||
status: verdict.status,
|
||||
flags: verdict.flags,
|
||||
score: verdict.score,
|
||||
analysis: verdict.analysis,
|
||||
categories: verdict.categories,
|
||||
severity: verdict.severity as AnalysisResult["severity"],
|
||||
confidence: verdict.confidence,
|
||||
recommendedAction:
|
||||
verdict.recommendedAction as AnalysisResult["recommendedAction"],
|
||||
policyVersion,
|
||||
evidence: [],
|
||||
});
|
||||
servedCacheKeys.add(cacheKey);
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
return true;
|
||||
};
|
||||
|
||||
for (const candidate of firstByScopedKey.values()) {
|
||||
const scopedEntry = storedEntries.get(candidate.scopedKey);
|
||||
if (
|
||||
scopedEntry &&
|
||||
acceptExactVerdict(
|
||||
candidate,
|
||||
candidate.scopedKey,
|
||||
scopedEntry,
|
||||
"cached-user-moderation-2026-06",
|
||||
)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Context-free fallback: ONLY non-actionable clean verdicts qualify
|
||||
// (guard enforces status/flags/action/confidence/freshness). The bare
|
||||
// key equals the scoped key for context-less messages, so the guard
|
||||
// also prevents double-serving the same row.
|
||||
const bareEntry = storedEntries.get(candidate.bareKey);
|
||||
if (
|
||||
bareEntry &&
|
||||
candidate.bareKey !== candidate.scopedKey &&
|
||||
isGloballyReusableCleanVerdict(
|
||||
bareEntry.verdict,
|
||||
bareEntry.analyzedAt ?? undefined,
|
||||
)
|
||||
) {
|
||||
acceptExactVerdict(
|
||||
candidate,
|
||||
candidate.bareKey,
|
||||
bareEntry,
|
||||
"cached-global-clean-2026-08",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Fan-out: every candidate (representative + in-batch duplicates) gets its
|
||||
// own copy of the representative verdict; unresolved ones stay queued.
|
||||
for (const candidate of candidates) {
|
||||
const representative = hitByKey.get(candidate.scopedKey);
|
||||
if (representative) {
|
||||
cacheHits.push({ ...representative, messageId: candidate.target.id });
|
||||
} else {
|
||||
uncachedTargets.push(candidate.target);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase 2: semantic cache — batched (one embed call + one Qdrant
|
||||
@@ -199,10 +271,12 @@ export async function runModerationAnalysis(
|
||||
}
|
||||
|
||||
if (isQdrantConfigured()) {
|
||||
// ONE batch search at the LOOSER threshold; per-hit re-classification
|
||||
// enforces the strict band for actionable verdicts.
|
||||
const batchHits = await searchQdrantBatch(
|
||||
embeddings,
|
||||
config.AI_LLM_EMBEDDING_MAX_CANDIDATES,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN,
|
||||
);
|
||||
for (let i = 0; i < semanticCandidates.length; i++) {
|
||||
const { target, cacheKey } = semanticCandidates[i];
|
||||
@@ -210,6 +284,7 @@ export async function runModerationAnalysis(
|
||||
if (hits.length === 0) continue;
|
||||
const verdict = parseQdrantVerdict(hits[0].payload, hits[0].score);
|
||||
if (!verdict) continue;
|
||||
if (!isSemanticBandAccepted(verdict, verdict.similarity)) continue;
|
||||
log.debug(
|
||||
{
|
||||
messageId: target.id,
|
||||
@@ -242,10 +317,12 @@ export async function runModerationAnalysis(
|
||||
const { target, cacheKey } = semanticCandidates[i];
|
||||
const semantic = await findSimilarTextModeration(
|
||||
embeddings[i],
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN,
|
||||
config.AI_LLM_EMBEDDING_MAX_CANDIDATES,
|
||||
);
|
||||
if (!semantic) continue;
|
||||
if (!isSemanticBandAccepted(semantic, semantic.similarity))
|
||||
continue;
|
||||
log.debug(
|
||||
{
|
||||
messageId: target.id,
|
||||
@@ -290,6 +367,8 @@ export async function runModerationAnalysis(
|
||||
}
|
||||
|
||||
if (cacheHits.length > 0) {
|
||||
// Metrics: one bulk UPDATE for every exact-cache key actually served.
|
||||
bumpTextModerationHitCounts(Array.from(servedCacheKeys));
|
||||
log.info(
|
||||
{
|
||||
cacheHits: cacheHits.length,
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { config } from "../../shared/config/config.js";
|
||||
import { executeAll, executeGet } from "../../shared/database/drizzle.js";
|
||||
import { findBestEmbeddingMatch } from "./embeddingClient.js";
|
||||
import {
|
||||
@@ -84,10 +85,43 @@ export function makeImageCacheKey(imageUrl: string): string {
|
||||
// size is 8191"), which fails acquireMediaAnalysisLock and silently skips
|
||||
// every media analysis. A 32-char sha256 keeps the key well under the limit
|
||||
// and is still deterministic (same attachment → same key).
|
||||
const hash = createHash("sha256").update(imageUrl).digest("hex").slice(0, 32);
|
||||
const hash = createHash("sha256")
|
||||
.update(normalizeDiscordImageUrl(imageUrl))
|
||||
.digest("hex")
|
||||
.slice(0, 32);
|
||||
return `image:${hash}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip volatile query params from Discord CDN URLs so the SAME attachment
|
||||
* always maps to ONE vision-cache key regardless of how it reached us
|
||||
* (signed `?ex=&is=&hm=` tokens rotate per fetch; render variants differ by
|
||||
* `format/width/height/size`). Previously each token variant hashed to its
|
||||
* own key → the same image was re-downloaded and re-analyzed by the vision
|
||||
* model once per variant. Non-Discord URLs and data: URLs are returned
|
||||
* untouched (their query can be semantically meaningful).
|
||||
*/
|
||||
export function normalizeDiscordImageUrl(imageUrl: string): string {
|
||||
try {
|
||||
const parsed = new URL(imageUrl);
|
||||
if (parsed.protocol !== "https:" && parsed.protocol !== "http:") {
|
||||
return imageUrl;
|
||||
}
|
||||
const host = parsed.hostname;
|
||||
const isAttachmentCdn = host === "cdn.discordapp.com";
|
||||
const isRenderOrPreview =
|
||||
host === "media.discordapp.net" ||
|
||||
/^images-ext-\d+\.discordapp\.net$/.test(host);
|
||||
if (!isAttachmentCdn && !isRenderOrPreview) return imageUrl;
|
||||
if (!parsed.search) return imageUrl;
|
||||
// Path IS the stable identity of the attachment; everything after "?" is
|
||||
// signing or a render variant.
|
||||
return `${parsed.origin}${parsed.pathname}`;
|
||||
} catch {
|
||||
return imageUrl;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Lookup a cached media analysis result.
|
||||
* Returns the full cached text (the analysis summary string) or null if not found or expired.
|
||||
@@ -294,10 +328,71 @@ export interface StoredModerationVerdict {
|
||||
recommendedAction: string;
|
||||
}
|
||||
|
||||
/** Raw DB row shape needed to rebuild a StoredModerationVerdict. */
|
||||
interface VerdictRow {
|
||||
flags: string;
|
||||
analyzed_at?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse one `text_analysis_cache` row into a StoredModerationVerdict.
|
||||
* Shared by the single-key and batched getters so their semantics can never
|
||||
* drift apart (status normalization lives in exactly one place).
|
||||
*/
|
||||
export function parseStoredVerdictRow(
|
||||
row: VerdictRow,
|
||||
): StoredModerationVerdict | null {
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(row.flags) as Record<string, unknown>;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object") return null;
|
||||
|
||||
const flags = Array.isArray(parsed.flags) ? (parsed.flags as string[]) : [];
|
||||
const status = normalizeStoredStatus(
|
||||
parsed.status as string | undefined,
|
||||
flags,
|
||||
);
|
||||
return {
|
||||
status,
|
||||
flags,
|
||||
score: (parsed.score as number) ?? 0,
|
||||
analysis: (parsed.analysis as string) ?? "",
|
||||
categories: (parsed.categories as string[]) ?? [],
|
||||
severity: (parsed.severity as string) ?? "none",
|
||||
confidence: (parsed.confidence as number) ?? 0,
|
||||
recommendedAction: (parsed.recommendedAction as string) ?? "none",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Increment the hit counter for a cache key (fire-and-forget).
|
||||
*
|
||||
* Bug history: `hit_count` was written as 0 on insert and never updated by
|
||||
* any reader, so cache effectiveness was unmeasurable. This is best-effort
|
||||
* observability — a failed bump must never affect the read path.
|
||||
*/
|
||||
function bumpHitCount(cacheKey: string): void {
|
||||
executeAll(
|
||||
`UPDATE text_analysis_cache SET hit_count = hit_count + 1 WHERE text = $1`,
|
||||
[cacheKey],
|
||||
).catch(() => {});
|
||||
}
|
||||
|
||||
/** Error-artifact flags that make a cached verdict unusable. */
|
||||
export const ERROR_ARTIFACT_FLAGS = [
|
||||
"analysis_api_failed",
|
||||
"analysis_parse_failed",
|
||||
"analysis_incomplete",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Lookup a cached moderation result for a text content.
|
||||
* Returns the stored result fields or null.
|
||||
*/
|
||||
|
||||
export async function getCachedTextModeration(
|
||||
cacheKey: string,
|
||||
): Promise<StoredModerationVerdict | null> {
|
||||
@@ -311,25 +406,11 @@ export async function getCachedTextModeration(
|
||||
|
||||
if (!row) return null;
|
||||
|
||||
const parsed = JSON.parse(row.flags) as Record<string, unknown>;
|
||||
const flags = (parsed.flags as string[]) ?? [];
|
||||
// Use stored status if available (new entries), otherwise derive from
|
||||
// flags (legacy compatibility). "warn" must survive the round-trip.
|
||||
const status = normalizeStoredStatus(
|
||||
parsed.status as string | undefined,
|
||||
flags,
|
||||
);
|
||||
const verdict = parseStoredVerdictRow(row);
|
||||
if (!verdict) return null;
|
||||
|
||||
return {
|
||||
status,
|
||||
flags,
|
||||
score: (parsed.score as number) ?? 0,
|
||||
analysis: (parsed.analysis as string) ?? "",
|
||||
categories: (parsed.categories as string[]) ?? [],
|
||||
severity: (parsed.severity as string) ?? "none",
|
||||
confidence: (parsed.confidence as number) ?? 0,
|
||||
recommendedAction: (parsed.recommendedAction as string) ?? "none",
|
||||
};
|
||||
bumpHitCount(cacheKey);
|
||||
return verdict;
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
{ error: error instanceof Error ? error.message : String(error) },
|
||||
@@ -339,6 +420,128 @@ export async function getCachedTextModeration(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Batched exact-hash lookup: ONE query for N keys.
|
||||
*
|
||||
* Semantics are identical to calling `getCachedTextModeration` per key
|
||||
* (unexpired rows only, shared row parser). Per-key hit-count bumps are NOT
|
||||
* issued here — the orchestrator logs an aggregate "cache applied" line
|
||||
* instead, keeping a 60-message burst at exactly one round-trip.
|
||||
* `analyzedAt` is surfaced so callers can apply freshness guards.
|
||||
*/
|
||||
export interface BatchedVerdictEntry {
|
||||
verdict: StoredModerationVerdict;
|
||||
analyzedAt: number | null;
|
||||
}
|
||||
|
||||
export async function getCachedTextModerations(
|
||||
cacheKeys: string[],
|
||||
): Promise<Map<string, BatchedVerdictEntry>> {
|
||||
const results = new Map<string, BatchedVerdictEntry>();
|
||||
const uniqueKeys = Array.from(new Set(cacheKeys)).filter(Boolean);
|
||||
if (uniqueKeys.length === 0) return results;
|
||||
|
||||
const CHUNK_SIZE = 200;
|
||||
try {
|
||||
for (let i = 0; i < uniqueKeys.length; i += CHUNK_SIZE) {
|
||||
const chunk = uniqueKeys.slice(i, i + CHUNK_SIZE);
|
||||
// Postgres has a 32k bind-parameter ceiling; ANY($1) keeps it at one
|
||||
// array param per chunk regardless of chunk length.
|
||||
const rows = await executeAll(
|
||||
`SELECT text, flags, analyzed_at
|
||||
FROM text_analysis_cache
|
||||
WHERE text = ANY($1::text[]) AND expires_at > $2`,
|
||||
[chunk, Date.now()],
|
||||
);
|
||||
for (const row of rows ?? []) {
|
||||
if (results.has(row.text)) continue;
|
||||
const verdict = parseStoredVerdictRow(row);
|
||||
if (!verdict) continue;
|
||||
results.set(row.text, {
|
||||
verdict,
|
||||
analyzedAt:
|
||||
typeof row.analyzed_at === "number" ? row.analyzed_at : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
{ error: error instanceof Error ? error.message : String(error) },
|
||||
"Failed batched text moderation lookup",
|
||||
);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fire-and-forget bulk hit-count bump for keys actually served as hits.
|
||||
* Companion to the batched getter (which skips per-row bumps): one UPDATE
|
||||
* per analysis batch keeps hit-rate metrics working at zero extra latency
|
||||
* cost per message.
|
||||
*/
|
||||
export function bumpTextModerationHitCounts(cacheKeys: string[]): void {
|
||||
const uniqueKeys = Array.from(new Set(cacheKeys)).filter(Boolean);
|
||||
if (uniqueKeys.length === 0) return;
|
||||
executeAll(
|
||||
`UPDATE text_analysis_cache SET hit_count = hit_count + 1 WHERE text = ANY($1::text[])`,
|
||||
[uniqueKeys],
|
||||
).catch(() => {});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Semantic two-band acceptance
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* True when a semantic-cache hit may be reused given its verdict class.
|
||||
* Two bands (2026-08-24): non-actionable verdicts (clean / flagless /
|
||||
* action=none) are accepted from the LOOSER clean band; actionable verdicts
|
||||
* (warn/flagged or any flags/action) keep the strict historical gate.
|
||||
* Between the bands → reject → the message falls through to the LLM
|
||||
* (fail-open toward accuracy).
|
||||
*/
|
||||
export function isSemanticBandAccepted(
|
||||
verdict: StoredModerationVerdict,
|
||||
similarity: number,
|
||||
): boolean {
|
||||
const isNonActionable =
|
||||
verdict.status === "clean" &&
|
||||
verdict.flags.length === 0 &&
|
||||
(verdict.recommendedAction ?? "none") === "none";
|
||||
return isNonActionable
|
||||
? similarity >= config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN
|
||||
: similarity >= config.AI_LLM_EMBEDDING_MIN_SIMILARITY;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Global exact-cache reuse guard (context-free fallback)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* True when a stored verdict is safe to reuse OUTSIDE its original channel:
|
||||
* only verdicts that cannot trigger an action and carry no flags qualify,
|
||||
* and they must be confident + fresh. Flagged/warn verdicts are NEVER
|
||||
* globally reused — enforcement is context-sensitive by design.
|
||||
*/
|
||||
export function isGloballyReusableCleanVerdict(
|
||||
verdict: StoredModerationVerdict,
|
||||
analyzedAtMs: number | undefined,
|
||||
): boolean {
|
||||
if (verdict.status !== "clean") return false;
|
||||
if (verdict.flags.length > 0) return false;
|
||||
if ((verdict.recommendedAction ?? "none") !== "none") return false;
|
||||
if (!(verdict.confidence >= config.AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE))
|
||||
return false;
|
||||
if (
|
||||
typeof analyzedAtMs === "number" &&
|
||||
Date.now() - analyzedAtMs >
|
||||
config.AI_CACHE_GLOBAL_REUSE_MAX_AGE_H * 60 * 60 * 1000
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a Qdrant verdict payload into the result shape shared by the
|
||||
* semantic cache lookups. Returns null on malformed payloads (callers then
|
||||
@@ -353,31 +556,13 @@ export function parseQdrantVerdict(
|
||||
similarity: number;
|
||||
})
|
||||
| null {
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(payload.flags) as Record<string, unknown>;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object") return null;
|
||||
|
||||
const flags = (parsed.flags as string[]) ?? [];
|
||||
const status = normalizeStoredStatus(
|
||||
parsed.status as string | undefined,
|
||||
flags,
|
||||
);
|
||||
const parsed = parseStoredVerdictRow({ flags: payload.flags });
|
||||
if (!parsed) return null;
|
||||
|
||||
return {
|
||||
...parsed,
|
||||
text: payload.text,
|
||||
similarity,
|
||||
status,
|
||||
flags,
|
||||
score: (parsed.score as number) ?? 0,
|
||||
analysis: (parsed.analysis as string) ?? "",
|
||||
categories: (parsed.categories as string[]) ?? [],
|
||||
severity: (parsed.severity as string) ?? "none",
|
||||
confidence: (parsed.confidence as number) ?? 0,
|
||||
recommendedAction: (parsed.recommendedAction as string) ?? "none",
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { resolve } from "node:dns/promises";
|
||||
import { isIP } from "node:net";
|
||||
import { LRUCache } from "lru-cache";
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { createAbortControllerWithTimeout } from "@/shared/utils/index";
|
||||
|
||||
@@ -150,6 +151,47 @@ function truncateAndCleanHtml(html: string, maxLen = 1000): string {
|
||||
export async function fetchUrlSafely(
|
||||
url: string,
|
||||
depth = 0,
|
||||
): Promise<FetchedUrlContext> {
|
||||
// Text results are memoized (in-process, short TTL): the same link recurs
|
||||
// across batches and re-downloading + re-parsing the page each time was
|
||||
// pure latency. Images are NEVER cached here — they are vision evidence
|
||||
// and multi-MB buffers don't belong in an LRU. Errors are not cached so a
|
||||
// transient network blip retries on the next batch.
|
||||
if (depth === 0) {
|
||||
const memo = textFetchMemo.get(url);
|
||||
if (memo) return memo;
|
||||
// In-flight dedupe: concurrent callers share one live request.
|
||||
const existing = textInFlight.get(url);
|
||||
if (existing) return existing;
|
||||
const promise = fetchUrlSafelyUncached(url, depth)
|
||||
.then((fetched) => {
|
||||
if (fetched.type === "text") textFetchMemo.set(url, fetched);
|
||||
return fetched;
|
||||
})
|
||||
.finally(() => {
|
||||
textInFlight.delete(url);
|
||||
});
|
||||
textInFlight.set(url, promise);
|
||||
return promise;
|
||||
}
|
||||
return fetchUrlSafelyUncached(url, depth);
|
||||
}
|
||||
|
||||
/** In-process memo of successful TEXT fetches (30 min TTL, bounded size). */
|
||||
const textFetchMemo = new LRUCache<string, FetchedUrlContext>({
|
||||
max: 500,
|
||||
ttl: 30 * 60 * 1000,
|
||||
});
|
||||
|
||||
/** Concurrent same-URL text fetches collapse into one live request. */
|
||||
const textInFlight = new LRUCache<string, Promise<FetchedUrlContext>>({
|
||||
max: 100,
|
||||
ttl: 60_000,
|
||||
});
|
||||
|
||||
async function fetchUrlSafelyUncached(
|
||||
url: string,
|
||||
depth = 0,
|
||||
): Promise<FetchedUrlContext> {
|
||||
if (depth > 1) {
|
||||
return { url, type: "error", error: "Max redirect/meta depth reached" };
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { createAbortControllerWithTimeout } from "@/shared/utils/index";
|
||||
import { config } from "../../shared/config/config.js";
|
||||
import { cacheGet, cacheSet, makeCacheKey } from "./cacheStore.js";
|
||||
|
||||
const log = createChildLogger("wikipedia-client");
|
||||
|
||||
@@ -57,10 +58,19 @@ function stripHtml(snippet: string): string {
|
||||
.trim();
|
||||
}
|
||||
|
||||
/** Redis TTL for cached search results (6h — articles change slowly). */
|
||||
const SEARCH_CACHE_TTL_SECONDS = 6 * 60 * 60;
|
||||
|
||||
/**
|
||||
* Search Wikipedia for a query and return up to MAX_RESULTS structured hits.
|
||||
* Uses the Action API `list=search` (srsearch) which is stable and returns
|
||||
* title + HTML snippet. Graceful: returns [] on any failure.
|
||||
*
|
||||
* Cached in the shared Redis store: the same query recurs across batches
|
||||
* (repeat slang, recurring topics), and an uncached re-search per batch was
|
||||
* pure latency + Wikipedia rate-limit pressure. Only NON-EMPTY results are
|
||||
* cached — an empty result may be a transient limiter/network blip, so it is
|
||||
* retried on a later batch instead of being pinned for 6 hours.
|
||||
*/
|
||||
export async function wikipediaSearch(
|
||||
query: string,
|
||||
@@ -69,6 +79,32 @@ export async function wikipediaSearch(
|
||||
const q = query.trim();
|
||||
if (!q) return [];
|
||||
|
||||
const cacheKey = makeCacheKey("wikisearch", q);
|
||||
const cached = await cacheGet(cacheKey);
|
||||
if (cached) {
|
||||
try {
|
||||
const parsed = JSON.parse(cached) as SearchResult[];
|
||||
if (Array.isArray(parsed) && parsed.length > 0) {
|
||||
log.debug({ query: q }, "Wikipedia search cache HIT");
|
||||
return parsed;
|
||||
}
|
||||
} catch {
|
||||
// Malformed entry — fall through to live fetch.
|
||||
}
|
||||
}
|
||||
|
||||
const mapped = await wikipediaSearchLive(q, timeoutMs);
|
||||
if (mapped.length > 0) {
|
||||
cacheSet(cacheKey, JSON.stringify(mapped), SEARCH_CACHE_TTL_SECONDS);
|
||||
}
|
||||
return mapped;
|
||||
}
|
||||
|
||||
/** Live (uncached) Action API search. Returns [] on any failure. */
|
||||
async function wikipediaSearchLive(
|
||||
q: string,
|
||||
timeoutMs: number,
|
||||
): Promise<SearchResult[]> {
|
||||
const params = new URLSearchParams({
|
||||
action: "query",
|
||||
list: "search",
|
||||
|
||||
@@ -168,6 +168,16 @@ export const configSchema = z
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.97),
|
||||
// Two-band semantic acceptance (2026-08-24): non-actionable verdicts
|
||||
// (clean, no flags, action=none) may be reused from a LOOSER similarity
|
||||
// band than actionable ones (warn/flagged). Actionable verdicts keep the
|
||||
// strict gate above; anything between the two bands falls through to the
|
||||
// LLM (fail-open toward accuracy).
|
||||
AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN: z.coerce
|
||||
.number()
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.92),
|
||||
AI_LLM_EMBEDDING_MAX_CANDIDATES: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
@@ -247,6 +257,20 @@ export const configSchema = z
|
||||
|
||||
// ── AI Analysis Batch ───────────────────────────────────────────────
|
||||
AI_ANALYSIS_MAX_BATCH_SIZE: z.coerce.number().int().positive().default(200),
|
||||
// Global exact-cache reuse guard (2026-08-24): a context-scoped miss may
|
||||
// fall back to the legacy bare (context-free) key, but ONLY for verdicts
|
||||
// that cannot trigger an action and are fresh + confident. These knobs
|
||||
// bound that reuse.
|
||||
AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE: z.coerce
|
||||
.number()
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.85),
|
||||
AI_CACHE_GLOBAL_REUSE_MAX_AGE_H: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
.positive()
|
||||
.default(72),
|
||||
AI_ANALYSIS_MAX_CONTEXT_TOKENS: z.coerce.number().positive().default(8000),
|
||||
AI_ANALYSIS_MAX_TARGET_TOKENS: z.coerce.number().positive().default(14000),
|
||||
AI_ANALYSIS_CONTEXT_MESSAGE_LIMIT: z.coerce
|
||||
|
||||
@@ -0,0 +1,127 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Batched exact-cache lookup (getCachedTextModerations)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): phase-1 cache lookups collapse N sequential per-key
|
||||
// queries into ONE `text = ANY($1::text[])` query. Semantics must match the
|
||||
// single-key getter: unexpired rows only, malformed rows skipped, verdicts
|
||||
// normalized through the shared parser. The DB layer is mocked — no live
|
||||
// Postgres in unit tests.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const executeAll = vi.fn();
|
||||
const executeGet = vi.fn();
|
||||
|
||||
vi.mock("../src/shared/database/drizzle.js", () => ({
|
||||
executeAll: (...args: unknown[]) => executeAll(...args),
|
||||
executeGet: (...args: unknown[]) => executeGet(...args),
|
||||
}));
|
||||
|
||||
import {
|
||||
getCachedTextModerations,
|
||||
parseStoredVerdictRow,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
function rowFor(
|
||||
text: string,
|
||||
status: string,
|
||||
analyzedAt = Date.now() - 1000,
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
text,
|
||||
flags: JSON.stringify({
|
||||
status,
|
||||
flags: [],
|
||||
score: 0,
|
||||
analysis: "ok",
|
||||
categories: [],
|
||||
severity: "none",
|
||||
confidence: 0.9,
|
||||
recommendedAction: "none",
|
||||
}),
|
||||
source: "user_moderation",
|
||||
analyzed_at: analyzedAt,
|
||||
expires_at: Date.now() + 3_600_000,
|
||||
hit_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
executeAll.mockReset();
|
||||
executeGet.mockReset();
|
||||
});
|
||||
|
||||
describe("parseStoredVerdictRow", () => {
|
||||
it("parses a valid stored verdict", () => {
|
||||
const v = parseStoredVerdictRow({
|
||||
flags: JSON.stringify({ status: "warn", flags: ["x"] }),
|
||||
});
|
||||
expect(v).not.toBeNull();
|
||||
expect(v?.status).toBe("warn");
|
||||
expect(v?.flags).toEqual(["x"]);
|
||||
});
|
||||
|
||||
it("returns null on malformed JSON", () => {
|
||||
expect(parseStoredVerdictRow({ flags: "{not-json" })).toBeNull();
|
||||
});
|
||||
|
||||
it("derives legacy status from flags when status is absent", () => {
|
||||
const v = parseStoredVerdictRow({
|
||||
flags: JSON.stringify({ flags: ["a"] }),
|
||||
});
|
||||
expect(v?.status).toBe("flagged");
|
||||
const v2 = parseStoredVerdictRow({ flags: JSON.stringify({}) });
|
||||
expect(v2?.status).toBe("clean");
|
||||
});
|
||||
});
|
||||
|
||||
describe("getCachedTextModerations", () => {
|
||||
it("returns an empty map for empty input and issues no query", async () => {
|
||||
const result = await getCachedTextModerations([]);
|
||||
expect(result.size).toBe(0);
|
||||
expect(executeAll).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("dedupes keys and returns parsed verdicts keyed by cache key", async () => {
|
||||
executeAll.mockResolvedValueOnce([
|
||||
rowFor("text_mod:c1:aaa", "clean"),
|
||||
rowFor("text_mod:c1:bbb", "warn"),
|
||||
]);
|
||||
const result = await getCachedTextModerations([
|
||||
"text_mod:c1:aaa",
|
||||
"text_mod:c1:aaa",
|
||||
"text_mod:c1:bbb",
|
||||
]);
|
||||
expect(result.size).toBe(2);
|
||||
expect(result.get("text_mod:c1:aaa")?.verdict.status).toBe("clean");
|
||||
expect(result.get("text_mod:c1:bbb")?.verdict.status).toBe("warn");
|
||||
// Exactly ONE batched round-trip.
|
||||
expect(executeAll).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("skips rows with malformed payloads instead of failing the batch", async () => {
|
||||
executeAll.mockResolvedValueOnce([
|
||||
{
|
||||
text: "k_good",
|
||||
flags: JSON.stringify({ status: "clean", flags: [] }),
|
||||
analyzed_at: 1,
|
||||
},
|
||||
{ text: "k_bad", flags: "{broken" },
|
||||
]);
|
||||
const result = await getCachedTextModerations(["k_good", "k_bad"]);
|
||||
expect(result.has("k_good")).toBe(true);
|
||||
expect(result.has("k_bad")).toBe(false);
|
||||
});
|
||||
|
||||
it("survives a DB failure and returns an empty map (fail-open)", async () => {
|
||||
executeAll.mockRejectedValueOnce(new Error("connection refused"));
|
||||
const result = await getCachedTextModerations(["k1", "k2"]);
|
||||
expect(result.size).toBe(0);
|
||||
});
|
||||
|
||||
it("chunks queries beyond 200 keys", async () => {
|
||||
executeAll.mockResolvedValue([]);
|
||||
const keys = Array.from({ length: 450 }, (_, i) => `k${i}`);
|
||||
await getCachedTextModerations(keys);
|
||||
expect(executeAll).toHaveBeenCalledTimes(3); // 200 + 200 + 50
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,160 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Semantic two-band acceptance + global exact-cache reuse guard
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): cache hits may be served MORE aggressively for
|
||||
// verdicts that cannot trigger enforcement actions, and NEVER more
|
||||
// aggressively for actionable ones. Two layers enforce this:
|
||||
// - isSemanticBandAccepted: similarity thresholds differ by verdict class
|
||||
// (clean band 0.92 default vs strict actionable band 0.97 default).
|
||||
// - isGloballyReusableCleanVerdict: context-free (cross-channel) reuse of
|
||||
// the legacy bare key only for clean / flagless / action=none verdicts
|
||||
// with high confidence and bounded age.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
isGloballyReusableCleanVerdict,
|
||||
type StoredModerationVerdict,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
function makeVerdict(
|
||||
overrides: Partial<StoredModerationVerdict> = {},
|
||||
): StoredModerationVerdict {
|
||||
return {
|
||||
status: "clean",
|
||||
flags: [],
|
||||
score: 0,
|
||||
analysis: "",
|
||||
categories: [],
|
||||
severity: "none",
|
||||
confidence: 0.95,
|
||||
recommendedAction: "none",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("isSemanticBandAccepted", () => {
|
||||
it("accepts a non-actionable clean verdict at the loose clean band", () => {
|
||||
// Default AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN = 0.92.
|
||||
expect(isBandAccept(makeVerdict(), 0.93)).toBe(true);
|
||||
});
|
||||
|
||||
it("accepts a clean verdict exactly at the clean band boundary", () => {
|
||||
expect(isBandAccept(makeVerdict({ confidence: 0.99 }), 0.92)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a clean verdict below the clean band", () => {
|
||||
expect(isBandAccept(makeVerdict(), 0.91)).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects an actionable flagged verdict between the bands", () => {
|
||||
// 0.93 >= clean band BUT < strict band → must NOT be served.
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "flagged", flags: ["hate_speech"] }),
|
||||
0.93,
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts a flagged verdict at the strict band", () => {
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "flagged", flags: ["hate_speech"] }),
|
||||
0.98,
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a warn verdict below the strict band", () => {
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "warn", recommendedAction: "warn" }),
|
||||
0.96,
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("treats a clean verdict WITH flags as actionable (strict band)", () => {
|
||||
expect(isBandAccept(makeVerdict({ flags: ["borderline"] }), 0.93)).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("treats a clean verdict with a non-none action as actionable", () => {
|
||||
expect(
|
||||
isBandAccept(makeVerdict({ recommendedAction: "review" }), 0.93),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// Import indirection so the describe block reads cleanly.
|
||||
import { isSemanticBandAccepted as isBandAccept } from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
describe("isGloballyReusableCleanVerdict", () => {
|
||||
it("accepts a fresh, confident, flagless clean verdict", () => {
|
||||
const v = makeVerdict({ confidence: 0.9 });
|
||||
expect(isGloballyReusableCleanVerdict(v, Date.now() - 60_000)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects flagged / warn verdicts outright", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ status: "flagged", flags: ["harassment"] }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ status: "warn", flags: ["mild"] }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects clean verdicts carrying flags", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(makeVerdict({ flags: ["x"] }), Date.now()),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects verdicts whose recommended action is not none", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ recommendedAction: "delete" }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects low-confidence verdicts below the guard threshold", () => {
|
||||
// Default AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE = 0.85.
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ confidence: 0.6 }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts confidence exactly at the guard threshold", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ confidence: 0.85 }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects entries older than the freshness window", () => {
|
||||
// Default AI_CACHE_GLOBAL_REUSE_MAX_AGE_H = 72h.
|
||||
const tooOld = Date.now() - 73 * 60 * 60 * 1000;
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), tooOld)).toBe(false);
|
||||
const freshEnough = Date.now() - 71 * 60 * 60 * 1000;
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), freshEnough)).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("skips the age check when analyzedAt is unknown", () => {
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), undefined)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// normalizeDiscordImageUrl — unified vision cache keys for Discord CDN URLs
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): the same attachment reached through different signed
|
||||
// URLs (?ex=&is=&hm= tokens rotate per fetch) or render variants
|
||||
// (?format=&width=) must map to ONE vision-cache key, otherwise the vision
|
||||
// model re-downloads and re-analyzes the identical image once per variant.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
makeImageCacheKey,
|
||||
normalizeDiscordImageUrl,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
describe("normalizeDiscordImageUrl", () => {
|
||||
it("strips rotating signed tokens from cdn.discordapp.com URLs", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://cdn.discordapp.com/attachments/1/2/img.png?ex=67a&is=67b&hm=tokA",
|
||||
);
|
||||
const b = normalizeDiscordImageUrl(
|
||||
"https://cdn.discordapp.com/attachments/1/2/img.png?ex=78c&is=78d&hm=tokB",
|
||||
);
|
||||
expect(a).toBe("https://cdn.discordapp.com/attachments/1/2/img.png");
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it("strips render variants from media.discordapp.net URLs", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://media.discordapp.net/attachments/1/2/img.png?format=webp&width=400&height=300",
|
||||
);
|
||||
const b = normalizeDiscordImageUrl(
|
||||
"https://media.discordapp.net/attachments/1/2/img.png?format=png&width=1024&height=768",
|
||||
);
|
||||
expect(a).toBe(b);
|
||||
expect(a).toBe("https://media.discordapp.net/attachments/1/2/img.png");
|
||||
});
|
||||
|
||||
it("strips query params from images-ext preview hosts", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://images-ext-1.discordapp.net/external/X/https/example.com/cat.jpg?format=webp",
|
||||
);
|
||||
expect(a).toBe(
|
||||
"https://images-ext-1.discordapp.net/external/X/https/example.com/cat.jpg",
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves URLs without query untouched", () => {
|
||||
const u = "https://cdn.discordapp.com/attachments/1/2/img.png";
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
|
||||
it("leaves non-Discord URLs untouched (query may be meaningful)", () => {
|
||||
const u = "https://example.com/image?token=abc&id=1";
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
|
||||
it("leaves data: URLs untouched", () => {
|
||||
const u = "data:image/png;base64,iVBORw0KGgoAAAANS?weird=query";
|
||||
// new URL() parses data: with protocol "data:" → not http(s) → untouched.
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
});
|
||||
|
||||
describe("makeImageCacheKey — Discord URL unification", () => {
|
||||
it("produces the SAME key for the same attachment across token variants", () => {
|
||||
const keyA = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/pic.png?ex=111&is=222&hm=AAA",
|
||||
);
|
||||
const keyB = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/pic.png?ex=333&is=444&hm=BBB",
|
||||
);
|
||||
expect(keyA).toBe(keyB);
|
||||
});
|
||||
|
||||
it("still produces DIFFERENT keys for different attachments", () => {
|
||||
const keyA = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/a.png?ex=1",
|
||||
);
|
||||
const keyB = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/b.png?ex=1",
|
||||
);
|
||||
expect(keyA).not.toBe(keyB);
|
||||
});
|
||||
|
||||
it("preserves the historical full-hash behavior for non-Discord URLs", () => {
|
||||
// Regression guard: external URLs keep pre-change keys.
|
||||
const key = makeImageCacheKey("https://example.com/x.png");
|
||||
expect(key).toBe(makeImageCacheKey("https://example.com/x.png"));
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user