Compare commits
28
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
588e750ede | ||
|
|
31e303c187 | ||
|
|
796c6390ac | ||
|
|
ecbf2617e4 | ||
|
|
9ef7d005fb | ||
|
|
842610b1af | ||
|
|
fca96396b9 | ||
|
|
ccf3fa260e | ||
|
|
1accfd9390 | ||
|
|
440ec41da8 | ||
|
|
1c8c0ca081 | ||
|
|
5f42c17caa | ||
|
|
eda5c752b7 | ||
|
|
25d5097edb | ||
|
|
d3e3b4764a | ||
|
|
33a557c761 | ||
|
|
32de2819df | ||
|
|
a21d252e9b | ||
|
|
84a766db0c | ||
|
|
0581dc3485 | ||
|
|
3d6c07bd91 | ||
|
|
60ae1fb5c3 | ||
|
|
1fafebb16d | ||
|
|
a9e09c38e9 | ||
|
|
3d4236e8df | ||
|
|
16becd5340 | ||
|
|
1397380fe9 | ||
|
|
4ffc99b3fe |
No files matched your search
+3
-3
@@ -84,9 +84,9 @@ BACKLOG_SYNC_BATCH_SIZE=100 # Messages per backlog batch, max 100 (d
|
||||
|
||||
# === AI Analysis ===
|
||||
AI_ANALYSIS_ENABLED=false # Enable AI content moderation (default: false)
|
||||
# AI_LLM_API_KEY= # REQUIRED if AI_ANALYSIS_ENABLED=true. LLM API key
|
||||
AI_LLM_BASE_URL=http://100.121.180.82:20128/api/v1 # LLM API base URL (omniroute on imrnes; /api/v1 exposes OpenAI-compatible chat+embeddings)
|
||||
AI_LLM_MODEL=text # LLM text model name (default: text)
|
||||
AI_LLM_API_KEY= # REQUIRED if AI_ANALYSIS_ENABLED=true. LLM API key
|
||||
AI_LLM_BASE_URL=https://9router.asepharyana.my.id/v1 # LLM API base URL (9router — OpenAI-compatible router, replaces omniroute)
|
||||
AI_LLM_MODEL=claude-opus-5 # LLM text model name (default: claude-opus-5)
|
||||
# AI_LLM_VISION_MODEL= # Vision model for image analysis (falls back to AI_LLM_MODEL)
|
||||
# AI_LLM_EMBEDDING_MODEL= # Embedding model for semantic moderation cache (optional; enables near-duplicate text reuse to save LLM calls)
|
||||
# AI_LLM_EMBEDDING_MIN_SIMILARITY=0.97 # Min cosine similarity to reuse a cached verdict (default: 0.97)
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# AI Analysis Flow — Audit & Optimization (discord-gateway)
|
||||
|
||||
**Goal:** Analisis alur AI analysis end-to-end, temukan bug/inconsistency yang merusak kualitas verdict, lalu perbaiki root cause-nya.
|
||||
|
||||
## Scope
|
||||
- `services/discord-gateway/src/modules/ai-moderation/**`
|
||||
- Tidak menyentuh chatbot backend / frontend.
|
||||
|
||||
## Alur saat ini (hasil tracing)
|
||||
```
|
||||
message capture → aiAnalyzer.queueMessageAnalysis(messageId)
|
||||
→ batchScheduler.scheduleConversationAnalysis(conversationKey) [debounce 250ms, CB gate]
|
||||
→ messageStore.getPendingMessagesByConversation(≤200)
|
||||
→ skipAgeRestrictedMessages
|
||||
→ pickBatchWithinBudget(14000 tokens, 50/msg)
|
||||
→ processBatch [Piscina worker, ≤4 threads]
|
||||
→ ai-analysis-worker.processBatch
|
||||
→ getConversationContextBefore(20 msgs) + attachments
|
||||
→ attachment-upload race guard (pending upload → skip)
|
||||
→ runModerationAnalysis
|
||||
→ Phase 1: exact-hash cache (PG text_analysis_cache, per channel/thread)
|
||||
→ Phase 2: semantic cache (embedTexts → Qdrant batch search; PG fallback)
|
||||
→ split text-only vs media
|
||||
→ runTextOnlyBatch: URL fetch + wiki search + glossary (paralel)
|
||||
→ dedup short messages → sub-batches (60/sub-batch)
|
||||
→ vision evidence utk URL images (hoisted, 15s cap per image)
|
||||
→ callModerationLLM per sub-batch (stream:true, retries 3, JSON parse + correction retry)
|
||||
→ runMediaBatch: download → vision per image (cache LRU→DB→live, lock) → 1 LLM call
|
||||
→ setCachedTextModeration (PG + Qdrant upsert w/ embedding)
|
||||
→ normalizeResult (confidence clamp, fallback analysis)
|
||||
→ updateMessagesAIAnalysisBulk → broadcast + scheduleAutoDelete
|
||||
→ recovery worker tiap 10s: pending keys → re-schedule; incomplete → individual fallback queue
|
||||
→ individual fallback: 1 msg = 1 worker job (context + full LLM)
|
||||
→ cache prune tiap 6 jam (PG expired + Qdrant expired points)
|
||||
```
|
||||
|
||||
## Temuan audit (ranked)
|
||||
|
||||
### F1 — Cache hit menghapus status "warn" (BUG AKURASI)
|
||||
`moderationOrchestrator.ts` Phase-2 semantic hit & PG-fallback memetakan status via
|
||||
`parseQdrantVerdict`: storedStatus bukan "warn"/"flagged" → dipaksa "clean".
|
||||
TAPI exact-hash lookup (`getCachedTextModeration`, textCacheStore.ts:288-295) lebih parah:
|
||||
hanya menerima "clean"|"flagged" — **"warn" jatuh ke branch flags.length===0 ? clean : flagged**
|
||||
→ warn dengan flags=["conflict_instigation"] dibaca sebagai FLAGGED.
|
||||
Efek: auto-delete eligibility (butuh recommendedAction delete/escalate + severity list) salah baca;
|
||||
dashboard menampilkan flagged padahal verdict asli warn. Root cause: type narrowing legacy
|
||||
(`status: "clean" | "flagged"`) tidak diupdate ketika "warn" ditambahkan ke schema.
|
||||
|
||||
### F2 — Exact-cache key mengabaikan edit (BUG EVASION)
|
||||
Key = sha256(content)+context. Pesan yang DIEDIT (`edited_content`) menghasilkan hash berbeda,
|
||||
tapi verdict lama utk konten pre-edit tetap hidup; lebih penting: pesan edited="true" adalah sinyal
|
||||
evasion di prompt, sedangkan cache bisa menyajikan verdict dari konten lama jika content sama.
|
||||
(Minor, tapi konsistensi: `resolveIsEdited` ada di prompt, tidak ada di cache key.)
|
||||
|
||||
### F3 — `pickBatchWithinBudget` skip-bukan-break (LATENSI/KUALITAS)
|
||||
Loop `if (usedTokens + msgTokens <= maxTokens) {push}` — pesan BESAR di tengah list dilewati
|
||||
dan iterasi lanjut mencoba msg berikutnya. Efek: batch berisi "lubang" (msg pending tetap pending,
|
||||
dianalisis di gelombang berikutnya = LLM call tambahan). Ini by-design tolerable, tapi ada bug halus:
|
||||
pesan >budget tunggal tidak pernah masuk (scheduler sudah punya fallback slice(0,1), OK).
|
||||
Keputusan: biarkan (bukan bug nyata), catat saja.
|
||||
|
||||
### F4 — `callModerationLLM` max_tokens 16384 hardcoded (COST)
|
||||
Sub-batch 60 pesan × output ~150 token/pesan ≈ 9k token cukup; 16k aman. Biarkan.
|
||||
|
||||
### F5 — Dead code builder user-profile/reputation
|
||||
`buildUserProfilesBlock`, `buildUserProfileRef`, `UserProfileEntry` di moderationBuilders.ts
|
||||
tidak dipakai lagi sejak context minimization (hanya tests). `<user_history>` juga tak pernah
|
||||
di-inject (rules masih menyebutnya — misleading bagi model). Bersihkan referensi prompt.
|
||||
|
||||
### F6 — rules.ts menyebut `<user_history>` yang tidak pernah ada di payload
|
||||
Model diberi instruksi tentang blok yang tak pernah muncul → pemborosan token + potensi
|
||||
kelakuan aneh ("menunggu" data yang tak ada). Hapus/ubah kalimat.
|
||||
|
||||
### F7 — system.ts "Blok Data" menyebut `<term_glossary> (SearXNG)` — STALE
|
||||
Sumber sudah Wikipedia. Komentar kode & teks prompt menyebut SearXNG. Perbaiki teks (kecil).
|
||||
|
||||
### F8 — output.ts typo "secifik", baris tabel `-|-` rusak
|
||||
Kualitas prompt: typo + markdown table broken (`||-`) di beberapa baris. Rapikan.
|
||||
|
||||
### F9 — llmCaller parse-error correction tail hanya di SYSTEM
|
||||
Correction tail ditambahkan ke system prompt; provider caching fine, tapi preview invalid
|
||||
content (800 char) ikut SYSTEM — ok. Skip.
|
||||
|
||||
### F10 — `getLlmSemaphore` race kecil saat config berubah di tengah flight
|
||||
Non-issue praktis (config statis per proses). Skip.
|
||||
|
||||
## Keputusan perbaikan (yang dieksekusi sekarang)
|
||||
1. **F1 (utama):** normalisasi status di SATU tempat — `normalizeStoredStatus()` di
|
||||
textCacheStore.ts yang menerima clean/warn/flagged; pakai di getCachedTextModeration
|
||||
DAN parseQdrantVerdict; perluas return types ke union penuh. Orchestrator tinggal pakai.
|
||||
2. **F6+F7+F8:** bersihkan stale references di prompts (user_history, SearXNG, typo).
|
||||
3. **F5:** hapus dead builders + test-nya (biome/tsc yang jaga).
|
||||
4. Regression test untuk F1 (vitest): warn tersimpan → warn terbaca (exact + qdrant path).
|
||||
|
||||
## Files touched
|
||||
- services/discord-gateway/src/modules/ai-moderation/textCacheStore.ts (F1)
|
||||
- services/discord-gateway/src/modules/ai-moderation/moderationOrchestrator.ts (type only)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/rules.ts (F6)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/system.ts (F7)
|
||||
- services/discord-gateway/src/modules/ai-moderation/prompts/output.ts (F8)
|
||||
- services/discord-gateway/src/modules/ai-moderation/moderationBuilders.ts (F5)
|
||||
- services/discord-gateway/tests/contextEnrichment.test.ts (F5 test cleanup + F1 regression test baru)
|
||||
|
||||
## Verification
|
||||
```
|
||||
cd services/discord-gateway
|
||||
npx tsc --noEmit
|
||||
npx biome check --diagnostic-level=error .
|
||||
npx vitest run
|
||||
```
|
||||
Semua harus hijau sebelum commit. Deploy via GHA (push main) — user konfirmasi belakangan.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Optimisasi "non-issue" AI analysis pipeline
|
||||
|
||||
## Scope
|
||||
Dua item yang sebelumnya dinyatakan non-issue, kini dioptimalkan + 1 bug ordering
|
||||
yang ditemukan saat menelusuri:
|
||||
|
||||
1. **pickBatchWithinBudget: skip → break.** Pesan diurutkan `created_at ASC`
|
||||
oleh DB. Setelah budget habis, pesan berikutnya pasti lebih besar/lebih kecil
|
||||
arbitrer — skip-then-take menghasilkan batch non-kontigu (ada gap analisis
|
||||
di tengah timeline). Ubah jadi stop at first overflow (break) supaya prefix
|
||||
kronologis utuh; sisanya otomatis diambil gelombang berikutnya
|
||||
(`shouldScheduleNext` sudah selalu true setelah sukses).
|
||||
2. **max_tokens dinamis.** Hard-coded 16384 di llmCaller.ts → parameter
|
||||
opsional `maxTokens?`; default tetap 16384. Caller text/media batch pass
|
||||
nilai berbasis ukuran prompt (tiktoken) dengan floor/ceiling.
|
||||
3. **Bug ordering UPDATE..RETURNING (bonus).** messagesAnalysis.ts
|
||||
`getPendingMessagesByConversation`: SELECT ids di-order `created_at ASC`
|
||||
tapi UPDATE...RETURNING tanpa ORDER BY → urutan rows balik tidak
|
||||
terjamin. Konsumen pakai messages[0] sebagai anchor konteks
|
||||
(beforeCreatedAt) dan pickBatchWithinBudget asumsi urutan. Fix: re-sort in
|
||||
JS by created_at (stable) sebelum return.
|
||||
|
||||
## Files touched
|
||||
- src/modules/ai-moderation/batchProcessor.ts — break bukan skip; test baru.
|
||||
- src/modules/ai-moderation/llmCaller.ts — param maxTokens.
|
||||
- src/modules/ai-moderation/textBatchProcessor.ts / mediaBatchProcessor.ts —
|
||||
hitung token prompt & pass maxTokens.
|
||||
- src/modules/message-capture/messagesAnalysis.ts — sort hasil RETURNING.
|
||||
- tests/batchBudget.test.ts — baru.
|
||||
|
||||
## Verification
|
||||
cd services/discord-gateway && bun run typecheck && bun run lint && bun run test
|
||||
lalu commit+push, watch GHA, restart service via deploy pipeline.
|
||||
@@ -0,0 +1,127 @@
|
||||
# Spec: Optimasi AI Analysis GMW — Naikkan Cache Hit Tanpa Kehilangan Akurasi
|
||||
|
||||
Tanggal: 2026-08-24 · Repo: `~/GMW` (branch `main`) · Service: `services/discord-gateway`
|
||||
|
||||
## Latar & Evidence (audit 2026-08-24)
|
||||
|
||||
State produksi:
|
||||
- Qdrant `gmw_text_moderation`: **1.550 poin, status green** (vectors size 2048, Cosine).
|
||||
- PG `text_analysis_cache`: 1.634 row `user_moderation`, 277 `vision_llm`; **sum(hit_count) = 0** →
|
||||
hit-rate tidak pernah terukur.
|
||||
- Embedding aktif (`AI_LLM_EMBEDDING_MODEL` set, Nemotron-embed, dim 2048), `AI_LLM_EMBEDDING_MIN_SIMILARITY`
|
||||
tidak diset di BWS → default **0.97** (sangat konservatif).
|
||||
- Messages: 9.375 total; 643 status `error` (banyak retry), 49 pending.
|
||||
|
||||
Temuan audit alur (`moderationOrchestrator.ts` → `textCacheStore.ts` → `qdrantClient.ts`,
|
||||
`textBatchProcessor.ts`, `urlFetcher.ts`, `wikipediaClient.ts`, `visionAnalyzer.ts`):
|
||||
|
||||
| # | Temuan | Dampak |
|
||||
|---|--------|--------|
|
||||
| F1 | Exact-hash cache key menyertakan context (channel/thread) → teks sama di channel lain selalu miss | Killer hit-rate #1 |
|
||||
| F2 | Semantic tier TIDAK memfilter context (Qdrant payload tak punya context) — sudah global tapi hanya aman krn sim 0.97 ketat | Inkonsisten dgn exact tier |
|
||||
| F3 | Phase-1 lookup loop `await getCachedTextModeration(key)` per pesan → N round-trip PgBouncer per batch (60 msg = 60 query serial) | Latensi + beban DB |
|
||||
| F4 | Verdict actionable (flagged/warn) dan clean sama-sama boleh di-serve semantic; toleransi akurasi beda | Risiko akurasi |
|
||||
| F5 | `hit_count` tidak pernah di-increment oleh reader manapun | Hit-rate tak terukur |
|
||||
| F6 | `wikipediaSearch()` (blok `<web_searches>`) tanpa cache — re-fetch tiap batch utk query sama | Latensi + spam ke WP |
|
||||
| F7 | `fetchUrlSafely()` tanpa cache — link sama di batch berikutnya di-download lagi penuh | Latensi + bandwidth |
|
||||
| F8 | Vision cache key dari data-URL base64 hasil resize → attachment sama via jalur berbeda (URL vs embed) = key beda → re-download + re-vision | Duplikasi kerja vision |
|
||||
|
||||
Non-goals: mengubah pipeline enforcement (auto-mute/ban trust-store writes), mengubah prompt
|
||||
kebijakan moderasi, mengubah model/embedding provider.
|
||||
|
||||
## Desain
|
||||
|
||||
Semua perubahan degrade gracefully — cache gagal → perilaku lama (LLM). Akurasi dilindungi
|
||||
asimetris: **hemat boleh untuk verdict non-actionable, konservatif untuk yang memicu aksi.**
|
||||
|
||||
### D1 — Cache metrics (F5)
|
||||
- `textCacheStore.getCachedTextModeration()`: saat hit valid, increment `hit_count`
|
||||
(`UPDATE ... SET hit_count = hit_count + 1`) fire-and-forget (`.catch(()=>{})`), jangan blokir return.
|
||||
- Log info periodik ringkas di orchestrator sudah ada ("User moderation cache applied") — cukup.
|
||||
|
||||
### D2 — Batched exact-cache lookup (F3)
|
||||
- Fungsi baru `getCachedTextModerations(keys: string[]): Promise<Map<string, StoredModerationVerdict>>`
|
||||
di `textCacheStore.ts`: **satu** `SELECT ... WHERE text = ANY($1)` (chunk 200 key/query),
|
||||
parse + `normalizeStoredStatus` per row (reuse helper existing).
|
||||
- Orchestrator fase-1: kumpulkan semua key unik → satu call batched → distribusi hasil.
|
||||
- Semantik identik dengan loop lama (row expired/error-artifact tetap miss); hanya jumlah round-trip
|
||||
yang turun N→1.
|
||||
|
||||
### D3 — Global exact reuse untuk verdict non-actionable (F1)
|
||||
- Key scoped-context TETAP ditulis (kompatibel, invalidasi moderator tetap presisi).
|
||||
- Reader tambahan: kalau key `<ctx>:<hash>` miss, coba key legacy global `text_mod:<hash>` (bare).
|
||||
- Guard akurasi (WAJIB semua terpenuhi):
|
||||
- `status === "clean"` DAN `flags.length === 0`;
|
||||
- `confidence >= AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE` (default 0.85);
|
||||
- `recommendedAction === "none"`;
|
||||
- umur entry ≤ `AI_CACHE_GLOBAL_REUSE_MAX_AGE_H` (default 72h) — cek `analyzed_at`.
|
||||
- Flag baru `policyVersion: "cached-global-clean-2026-08"` supaya terlacak di dashboard/log.
|
||||
- Verdict flagged/warn TETAP context-scoped (tidak pernah lintas channel).
|
||||
|
||||
### D4 — Semantic dua-band similarity (F2+F4)
|
||||
- Config baru: `AI_LLM_EMBEDDING_MIN_SIMILARITY_ACTIONABLE` default **0.97** (perilaku lama),
|
||||
`AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN` default **0.92**, keduanya coerce number 0..1.
|
||||
- Satu Qdrant batch search pakai threshold RENDAH (0.92). Per hit, klasifikasi ulang:
|
||||
- verdict non-actionable (clean, no flags, action=none): terima jika `score >= CLEAN_BAND`;
|
||||
- verdict actionable (warn/flagged atau flags ada / action != none): terima hanya jika
|
||||
`score >= ACTIONABLE_BAND` (0.97 — persis gate lama);
|
||||
- di antara dua band → buang hit, pesan lanjut ke LLM (fail-open ke akurasi).
|
||||
- Legacy PG fallback path: filter serupa di `findSimilarTextModeration` via parameter band.
|
||||
|
||||
### D5 — Cache Wikipedia search (F6)
|
||||
- `wikipediaClient.wikipediaSearch(query)`: cek `cacheGet(makeCacheKey("wikisearch", q))` dulu;
|
||||
miss → fetch (timeout existing) → sukses & hasil non-kosong → `cacheSet(..., TTL 6h)`.
|
||||
Hasil kosong TIDAK di-cache (biar retry nanti). Redis down → langsung fetch (no-op cache).
|
||||
|
||||
### D6 — Cache URL text fetch (F7)
|
||||
- `urlFetcher.fetchUrlSafely(url)`: wrapper async memoize in-process LRU (max 500, TTL 30 menit)
|
||||
untuk `type === "text"` saja (image tetap selalu fresh-download karena dipakai sbg bukti vision
|
||||
+ buffer besar; error tidak di-cache).
|
||||
- Import `LRUCache` dari `lru-cache` (sudah dep gateway).
|
||||
|
||||
### D7 — Unified vision cache key (F8)
|
||||
- `makeImageCacheKey(imageUrl)` di `textCacheStore.ts`: sebelum hash, strip query Discord CDN
|
||||
(`?ex=&is=&hm=` signed tokens, `format/width/height/size`) — regex `(\?[^#]*)$` dibuang bila host
|
||||
CDN discord (`cdn.discordapp.com`, `media.discordapp.net`, `images-ext-*.discordapp.net`);
|
||||
URL non-Discord: hash full URL seperti sekarang.
|
||||
- Efek: attachment sama yang lolos lewat jalur embed vs inline vs re-fetch dgn token beda → SATU
|
||||
entry cache → skip download+vision kedua kali. Data-URL base64 tetap di-hash apa adanya.
|
||||
|
||||
## File yang disentuh
|
||||
|
||||
1. `src/shared/config/index.ts` — 3 config baru (D3×2, D4×2 — total 4 nilai, 3 baris zod + deskripsi).
|
||||
2. `src/modules/ai-moderation/textCacheStore.ts` — hit_count inc (D1), batched getter (D2),
|
||||
global-reuse guard helper (D3), image-key normalize (D7).
|
||||
3. `src/modules/ai-moderation/moderationOrchestrator.ts` — pakai batched getter (D2),
|
||||
global bare-key fallback (D3), dua-band semantic accept (D4).
|
||||
4. `src/modules/ai-moderation/qdrantClient.ts` — `searchQdrantBatch` menerima threshold rendah
|
||||
(sudah parametrik — mungkin tanpa perubahan; verifikasi).
|
||||
5. `src/modules/ai-moderation/wikipediaClient.ts` — cache layer (D5).
|
||||
6. `src/modules/ai-moderation/urlFetcher.ts` — LRU text-fetch memoize (D6).
|
||||
|
||||
## Schema/type changes
|
||||
|
||||
- Tidak ada migrasi DB (kolom `hit_count`, `analyzed_at`, `expires_at` sudah ada).
|
||||
- Tidak ada perubahan kontrak WS/oRPC/frontend.
|
||||
- Type baru: none public; internal `StoredModerationVerdict` dipakai ulang.
|
||||
|
||||
## Verification
|
||||
|
||||
1. Unit tests baru (`tests/`):
|
||||
- `cacheBatchLookup.test.ts`: batched getter — hit/miss/expired/error-artifact mapping,
|
||||
chunking >200 keys (mock executeAll), hit_count increment called.
|
||||
- `globalReuseGuard.test.ts`: guard menerima clean+conf≥0.85+action none+umur ≤72h;
|
||||
menolak flagged/warn/conf rendah/action≠none/stale.
|
||||
- `semanticBands.test.ts`: clean @0.93 diterima, flagged @0.93 ditolak, flagged @0.98 diterima.
|
||||
- `imageKeyNormalize.test.ts`: URL Discord dgn/ex token → key sama; non-Discord beda query → beda.
|
||||
2. Gate service: `pnpm typecheck && pnpm exec biome check --diagnostic-level=error . && pnpm exec vitest run`.
|
||||
3. Deploy via GHA (`git push origin main`) → watch `Build & Deploy (Nix)` → verifikasi
|
||||
`systemctl show gmw-discord-gateway -p ActiveEnterTimestamp` baru.
|
||||
4. Runtime probe pasca-deploy: journalctl level 30 normal; beberapa jam kemudian
|
||||
`SELECT sum(hit_count) FROM text_analysis_cache WHERE source='user_moderation'` > 0 membuktikan
|
||||
metrics jalan; log "User moderation cache applied" menunjukkan hits>0 pada traffic ramai.
|
||||
|
||||
## Rollback
|
||||
|
||||
Semua fitur behind config defaults yang mempertahankan perilaku lama pada nilai konservatif;
|
||||
rollback = redeploy commit sebelumnya (tanpa migrasi DB, tanpa state eksternal).
|
||||
@@ -0,0 +1,56 @@
|
||||
# Spec: Perbaiki Delay Attachment 162s→<20s (GMW AI Analysis)
|
||||
|
||||
Tanggal: 2026-08-24 · Repo `~/GMW` · Service discord-gateway
|
||||
|
||||
## Evidence (audit produksi)
|
||||
|
||||
Klaster pesan attachment delay ~330–400 detik. Trace pesan `1541417073245290638` (.gif):
|
||||
19:01:08 dibuat → 19:01:09 batch incomplete → fan-out individual → **guard upload-pending
|
||||
mengembalikan `results:[]`** → diperalakukan sukses (`complete ... (undefined)`) → row
|
||||
tertahan `ai_status='processing'` **tanpa penanggung jawab** → 19:06:12 cleanup mengembalikan
|
||||
ke `pending` (tepat 300s) → baru dianalisis. Plus vision gagal 3× utk GIF besar
|
||||
("Stream ended before producing a non-ping SSE event") → degradasi teks.
|
||||
|
||||
## Root causes
|
||||
|
||||
- **A (fatal)**: `individualFallbackProcessor.processIndividualFallback` memperlakukan
|
||||
`ok:true + results:[]` sebagai sukses. Race-guard upload di `ai-analysis-worker.processIndividual`
|
||||
sengaja balik `results:[]` (desain lama) → pesan yatim `processing` sampai cleanup 300s.
|
||||
- **B**: `llmVision` hanya mencoba `stream:true`; kegagalan SSE truncation pada gambar besar
|
||||
= 3 retry sia-sia (semua jalur sama) → bukti media hilang.
|
||||
- **C**: safety-net cleanup 300s terlalu lambat sbg satu-satunya pemulih `processing`.
|
||||
|
||||
## Fix
|
||||
|
||||
1. **F1 — sinyal eksplisit upload-pending**: `IndividualOkResponse` + field opsional
|
||||
`uploadPending?: boolean`. Worker set `uploadPending:true` saat race guard kena.
|
||||
2. **F2 — processor menangani 3 kondisi** via helper murni baru
|
||||
`classifyIndividualWorkerResult(result): "success" | "upload_pending" | "incomplete" | "error"`
|
||||
(modul baru `fallbackResultClassifier.ts`, zero-dep agar mudah dites):
|
||||
- `upload_pending` → tulis ulang row ke `pending` (pola sama dgn revert apiFailed di
|
||||
batchProcessor) + broadcast + **re-schedule analisis percakapan segera**
|
||||
(dynamic import batchScheduler, pola anti-siklus yg sudah ada) → retry dalam ~250ms
|
||||
begitu upload beres. Bukan error, tidak naikkan CB counter.
|
||||
- `incomplete` (flags analysis_incomplete) → perilaku lama (exhausted path).
|
||||
- `error` / `results kosong tanpa penjelasan` → throw transien (retry oleh recovery),
|
||||
BUKAN sukses palsu. Log "(undefined)" hilang.
|
||||
3. **F3 — vision non-stream fallback**: di `llmVision`, jika error match
|
||||
`/Stream ended before producing a non-ping SSE|stream ended/i` → coba SEKALI lagi dengan
|
||||
`stream:false` (router agregasi penuh; timeout tetap 60s). Konversi hard-fail jadi sukses.
|
||||
4. **F4 — turunkan safety net**: default `revertStuckProcessingMessages` 300000 → 120000 ms.
|
||||
|
||||
## File disentuh
|
||||
|
||||
- `src/modules/ai-moderation/fallbackResultClassifier.ts` (BARU, pure)
|
||||
- `src/modules/ai-moderation/ai-analysis-worker.ts` (tipe + set flag uploadPending)
|
||||
- `src/modules/ai-moderation/individualFallbackProcessor.ts` (konsumsi classifier + reschedule)
|
||||
- `src/modules/ai-moderation/llmClient.ts` (fallback non-stream di llmVision)
|
||||
- `src/modules/message-capture/messagesCleanup.ts` (default 120s)
|
||||
|
||||
## Verifikasi
|
||||
|
||||
- Test baru `tests/fallbackResultClassifier.test.ts` (4 klasifikasi + edge kosong).
|
||||
- Gate: tsc --noEmit, biome error-level, vitest run semua hijau.
|
||||
- Deploy GHA sukses; pasca-deploy: pesan attachment baru p50 < 20s
|
||||
(`SELECT percentile_cont(0.5) ... WHERE metadata attachments>0 AND created_at > deploy`),
|
||||
tidak ada lagi "complete ... (undefined)".
|
||||
@@ -0,0 +1,205 @@
|
||||
# GMW FE Rombak — "Constellation Ops" Implementation Plan
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
> Setiap task = edit → lint → build/test → smoke → commit (author `asepharyana`, tanpa Co-Authored-By).
|
||||
|
||||
**Goal:** Rombak total FE GMW dari template dashboard klasik menjadi **Constellation Node Graph** — graf node interaktif (three.js force layout) di mana channel/event/moderasi adalah node bintang dan navigasi adalah terbang antar node — untuk semua 9 route sekaligus.
|
||||
|
||||
**Architecture:** Hapus total shell template (TopBar + NavRail + MobileNav + main scroll container). Ganti dengan satu `ConstellationStage`: canvas full-bleed fixed di belakang seluruh app yang me-render force-directed graph per-route (scene), plus overlay HTML mengambang (tanpa card box standar) untuk konten detail. Router tetap Next App Router; perpindahan route = kamera "fly-to" ke scene baru. Data tetap lewat pola SSR seed (`page.tsx` server fetch → `view.tsx` client + SWR fallbackData) Line truncated
|
||||
|
||||
**Tech Stack:** Next.js 16 App Router · React 19 · TypeScript strict · Tailwind v4 (token oklch existing) · three r180 (+ `@types/three`) · d3-force (+ `@types/d3-force`) untuk simulasi layout deterministik · motion v12 (transisi antar-scene) · biome (0 error **dan** 0 warning) · bun:test untuk modul murni.
|
||||
|
||||
---
|
||||
|
||||
## Keputusan Terkunci (2026-08-24, user MythEclipse)
|
||||
|
||||
| Keputusan | Pilihan user |
|
||||
|---|---|
|
||||
| Cakupan | **Semua 9 route sekali jalan** (dashboard, voice, media, recordings, moderation, messages, analysis, channels, glossary) |
|
||||
| Arah struktural | **Constellation Node Graph** — navigasi = terbang antar node |
|
||||
| Tema | **Pertahankan toggle light/dark** (jangan rebuild sistem tema — harden saja) |
|
||||
|
||||
## Kondisi Saat Ini (fakta repo, 2026-08-24)
|
||||
|
||||
- Repo: `/home/code/GMW/services/frontend` (monorepo GMW, deploy Nix-first via GHA "Build & Deploy (Nix)" ke `imphnen.asepharyana.my.id`; nginx gmw-proxy :4009 → Next standalone :4017).
|
||||
- Shell aktif: `src/components/shell/ambient-app.tsx` → `AppFrame` = NavRail + TopBar + `<main>` scroll + MobileNav + MiniPlayer, dipakai SEMUA route via `(dashboard)/layout.tsx`. AmbientCanvas (233 l) masih ada sebagai background di belakang chrome template.
|
||||
- 9 route di `src/app/(dashboard)/`: dashboard (368 l), voice (311), media (293), moderation (424), messages (611), analysis (196), recordings (201), channels (23), glossary (15).
|
||||
- `next.config.ts`: `output: "standalone"` + `trailingSlash: true` → **nav via `router.push`/Link bisa no-op; pakai `<a href>` polos** (pitfall terverifikasi).
|
||||
- Token desain ada di `globals.css` `@theme`: `--color-canvas/-2`, `--color-ink/-soft/-faint`, `--color-signal(+glow)`, amber, vermilion, hairline; font Bricolage Grotesque/JetBrains Mono; `.light` class override + next-themes sudah jalan — JANGAN dibangun ulang.
|
||||
- Sudah ada & dipertahankan: CommandPalette (⌘K), Chatbot, MiniPlayer, primitives (GlassPanel/GlassCard/Badge/…), shared states+skeletons, `lib/format.ts`, `lib/ai-status.ts`, WS typed events (`message_created/updated/deleted/analyzed/snapshot*`, `voice_state`, dll).
|
||||
- Tooling lokal: `pnpm lint|build|format`, `pnpm exec biome check --write --unsafe .` buat import order; bun 1.3.14 tersedia untuk unit test modul murni.
|
||||
- Smoke test SELALU di port non-prod (mis. 4024), cek `ss -ltnp | grep <port>` dulu; JANGAN pernah bind 4017.
|
||||
|
||||
## Struktur Target
|
||||
|
||||
```
|
||||
src/components/shell/
|
||||
constellation-frame.tsx # pengganti AppFrame: tanpa topbar/rail/bottom-bar
|
||||
constellation-stage.tsx # canvas three.js full-bleed + hit-test + kamera
|
||||
route-scenes.ts # map pathname → scene config (nodes/edges/focus)
|
||||
floating-chrome.tsx # brand mark + status dot + route switcher minim + palette hint
|
||||
(hapus setelah migrasi): topbar.tsx, nav-rail.tsx, mobile-nav.tsx, ambient-app.tsx*
|
||||
src/lib/constellation/
|
||||
graph.ts # tipe GraphNode/GraphEdge + builder dari data API (murni)
|
||||
layout.ts # wrapper d3-force deterministik (seeded) + fallback radial (murni)
|
||||
camera.ts # interpolasi kamera fly-to (murni, testable)
|
||||
*.test.ts # bun:test utk modul murni di atas
|
||||
```
|
||||
|
||||
*ambient-canvas/context dievaluasi: bisa di-refactor jadi bagian dari stage atau dihapus.
|
||||
|
||||
### Scene per route (bahasa visual satu kesatuan)
|
||||
|
||||
| Route | Scene |
|
||||
|---|---|
|
||||
| `/dashboard/` | Guild = bintang pusat; channel = node orbit; event WS = pulse edge; metric cluster floating (tanpa box) |
|
||||
| `/channels/` | Semua channel sebagai konstelasi; klik channel = fly-to + panel detail mengambang |
|
||||
| `/moderation/` | Pesan flagged = node merah (vermilion) mengelilingi hub AI-verdict; LiveModerationFeed jadi ribbon ticker |
|
||||
| `/messages/` | Stream pesan = sabuk orbital; node baru masuk lewat animasi dari tepi |
|
||||
| `/voice/` | Stage node di tengah; speaker aktif = node mengorbit dengan glow intensitas |
|
||||
| `/media/` | Galeri spiral node (thumbnail via div bg-image); klik = MiniPlayer handoff |
|
||||
| `/recordings/` | Variasi spiral media, timeline density sebagai cincin |
|
||||
| `/analysis/` | Hub analitik; chart existing dirender sebagai panel translusen mengambang ter-anchor ke node |
|
||||
| `/glossary/` | Term KB = cincin satelit; search memfilter node real-time |
|
||||
|
||||
### Checklist anti-default (wajib lolos sebelum commit akhir)
|
||||
|
||||
- [ ] Tidak ada top bar / nav rail / side panel / bottom prompt bar
|
||||
- [ ] Tidak ada card grid standar sebagai struktur utama
|
||||
- [ ] Layout beda SPASIAL dari versi lama (bukan ganti warna)
|
||||
- [ ] Verifikasi visual sungguhan: browser_navigate + browser_vision → "bukan template dashboard"
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Phase 0 — Fondasi
|
||||
|
||||
#### Task 1: Dependencies + scaffold direktori
|
||||
**Files:** `package.json`, create `src/lib/constellation/`
|
||||
1. `cd /home/code/GMW/services/frontend && pnpm add three @types/three d3-force @types/d3-force && pnpm add -d @types/bun`
|
||||
2. Buat 4 file kosong bertipe sesuai struktur target (ekspor placeholder bertipe agar tsc lolos).
|
||||
3. Run: `pnpm exec tsc --noEmit` → expected PASS.
|
||||
4. Commit: `chore(frontend): scaffold constellation lib + deps`.
|
||||
|
||||
> Catatan: jika `@types/bun` bentrok dengan tsconfig Next (duplikat globals), fallback: exclude `"**/*.test.ts"` dari `tsconfig.json` dan jalankan test hanya via `bun test` (tanpa typecheck). Putuskan saat eksekusi, dokumentasikan di commit body.
|
||||
|
||||
#### Task 2 (TDD): Modul murni graph + layout + camera
|
||||
**Files:** `src/lib/constellation/{graph,layout,camera}.ts` + `*.test.ts`
|
||||
1. **Tulis test dulu** (bun:test):
|
||||
- `graph.test.ts`: builder `dashboardToGraph()` menghasilkan node guild pusat + N channel + edges benar dari fixture `/api/dashboard/stats|channels` shape (lihat AGENTS.md; nama channel di `message.metadata.channelName`).
|
||||
- `layout.test.ts`: `computeLayout(nodes, edges, {seed:42})` DETERMINISTIK (dua pemanggilan = posisi identik), semua posisi finite, radius dalam bound; `radialLayout()` fallback untuk reduced-motion.
|
||||
- `camera.test.ts`: `flyTo(from,to,t)` easing monotonic, t=0→from, t=1→to.
|
||||
2. Run: `bun test src/lib/constellation` → expected FAIL (belum diimplementasi).
|
||||
3. Implementasi minimal sampai PASS.
|
||||
4. Run: `bun test src/lib/constellation && pnpm exec tsc --noEmit` → PASS.
|
||||
5. Commit: `feat(frontend): constellation graph/layout/camera pure modules (tested)`.
|
||||
|
||||
### Phase 1 — Shell Baru
|
||||
|
||||
#### Task 3: `ConstellationStage` (canvas renderer)
|
||||
**Files:** `src/components/shell/constellation-stage.tsx`
|
||||
- `<canvas>` fixed inset-0 `-z-10`; three.js orthographic 2D-ish render node+edge (sprite/glow shader sederhana — jangan over-engineer; titik + garis + glow cukup).
|
||||
- Interaksi: drag = pan, wheel/pinch = zoom, hover = highlight node + tooltip title, click node ber-`href` = navigasi via `<a>` semantics (window.location assign — hindari router.push no-op).
|
||||
- DPR-aware resize; pause render loop saat `document.hidden`; `prefers-reduced-motion` → render statis tanpa animasi.
|
||||
- Sampling warna dari CSS var (`getComputedStyle`) + `MutationObserver` pada `documentElement.classList` agar ikut toggle dark/light.
|
||||
- Verify: `pnpm lint && pnpm build` PASS → commit `feat(frontend): constellation stage canvas renderer`.
|
||||
|
||||
#### Task 4: `route-scenes.ts` + `ConstellationFrame`
|
||||
**Files:** `src/components/shell/{route-scenes,constellation-frame,floating-chrome}.tsx`
|
||||
- `route-scenes.ts`: `usePathname()` → config scene (builder mana, focus node, overlay slots).
|
||||
- `ConstellationFrame`: render `ConstellationStage` + `children` (overlay HTML absolute, BUKAN flex column bertingkat) + `floating-chrome` (brand mark kiri-atas dgn status dot WS, switcher route minimal kanan-bawah, hint ⌘K). MiniPlayer + Chatbot + CommandPalette tetap termounting.
|
||||
- Switcher & semua nav internal pakai `<a href="/<route>/">` polos (trailingSlash!). Verifikasi klik di browser nanti, bukan cuma baca kode.
|
||||
- Wire `(dashboard)/layout.tsx`: ganti `AppFrame` → `ConstellationFrame` (provider Ambient/WS tetap).
|
||||
- Smoke: `ss -ltnp | grep 4024` kosong → `PORT=4024 pnpm start` → curl semua 9 route = 200 → kill server.
|
||||
- Commit: `feat(frontend): replace classic shell with constellation frame`.
|
||||
|
||||
#### Task 5: Hapus chrome lama + bersih-bersih
|
||||
**Files:** hapus `shell/topbar.tsx`, `shell/nav-rail.tsx`, `shell/mobile-nav.tsx`, update `shell/index.ts`; eval `ambient-*` (refactor jadi layer stage ATAU hapus).
|
||||
- `grep -rn "TopBar\|NavRail\|MobileNav\|AppFrame" src/` → 0 referensi tersisa.
|
||||
- `pnpm exec biome check --write --unsafe .` → import order rapi.
|
||||
- Commit: `refactor(frontend): remove classic dashboard chrome`.
|
||||
|
||||
### Phase 2 — Migrasi 9 Route (batch, tiap batch ship hijau)
|
||||
|
||||
Urutan sengaja dari yang paling graph-native ke paling padat konten:
|
||||
|
||||
#### Task 6: `/dashboard/` + `/channels/`
|
||||
- Dashboard view: guild-star + channel orbit + WS pulse; metric cluster floating (reuse MetricTile tapi tanpa card-box — ubah jadi teks besar + hairline).
|
||||
- Channels: konstelasi channel, klik = fly-to + panel detail (data dari hook `use-channels`/server fetch existing — jangan karang endpoint).
|
||||
- Lint+build+smoke 2 route → commit `feat(frontend): dashboard & channels constellation scenes`.
|
||||
|
||||
#### Task 7: `/glossary/` + `/analysis/`
|
||||
- Glossary: cincin satelit term + search filter node (client-side filter pada node label).
|
||||
- Analysis: hub node + panel chart translusen mengambang (reuse komponen charts/* existing, anchor posisi ke node).
|
||||
- Commit: `feat(frontend): glossary & analysis scenes`.
|
||||
|
||||
#### Task 8: `/messages/` + `/moderation/`
|
||||
- Messages: sabuk orbital stream (reuse sortMessages/logika WS `use-messages.ts` — jangan duplikasi); pesan baru animate-in dari tepi.
|
||||
- Moderation: flagged nodes vermilion + hub verdict; LiveModerationFeed jadi ticker ribbon bawah (bukan panel).
|
||||
- Commit: `feat(frontend): messages & moderation scenes`.
|
||||
|
||||
#### Task 9: `/voice/` + `/media/` + `/recordings/`
|
||||
- Voice: speaker aktif orbit dgn glow (seed dari snapshot `activeSpeakers` server-authoritative — pertahankan perilaku useSpeakers).
|
||||
- Media/Recordings: spiral galeri; thumbnail pakai div background-image (hindari `noImgElement`; kalau terpaksa `<img>` → `// biome-ignore lint/performance/noImgElement: <alasan>`); MiniPlayer handoff tidak boleh regress.
|
||||
- Commit: `feat(frontend): voice, media & recordings scenes`.
|
||||
|
||||
### Phase 3 — Polish, A11y, Tema
|
||||
|
||||
#### Task 10: Motion & micro-interaction
|
||||
- Fly-to antar scene pakai `motion` v12 (interpolasi kamera dari `lib/constellation/camera.ts`).
|
||||
- Overlay: stagger masuk (pattern `animate-stagger` + `staggerDelay(i)` existing); tambah semua `animate-*` baru ke kill-list `prefers-reduced-motion` di `globals.css`.
|
||||
- Commit: `feat(frontend): scene transitions + motion polish`.
|
||||
|
||||
#### Task 11: A11y & fallback semantik
|
||||
- Setiap scene punya fallback semantik: `<nav aria-label="Routes">` visually-hidden berisi link semua route + `<ul>` node (label + href) yang ter-sync dgn graph state.
|
||||
- Keyboard: Tab ke daftar route; panah memindahkan fokus antar node; Enter aktivasi; `focus-visible` ring pakai `--color-ring`.
|
||||
- Kontras ink-on-canvas dicek (WCAG AA utk teks overlay).
|
||||
- Hooks-after-early-return pitfall: compute flags dulu, guard di dalam useEffect.
|
||||
- Commit: `feat(frontend): a11y semantic fallback for constellation`.
|
||||
|
||||
#### Task 12: Hardening tema light/dark
|
||||
- Pastikan canvas sampling var `.light` benar saat toggle (MutationObserver dari Task 3); harden `.light`: `color-scheme: light`, scrollbar/selection, glow opacity turun.
|
||||
- JANGAN menyentuh sistem next-themes.
|
||||
- Commit: `fix(frontend): light theme parity for constellation stage`.
|
||||
|
||||
### Phase 4 — Gerbang Ship (non-negotiable)
|
||||
|
||||
#### Task 13: Verifikasi penuh + deploy
|
||||
1. `pnpm lint` → **Found 0 errors. Found 0 warnings** (warning pun harus bersih).
|
||||
2. `bun test src/lib/constellation` → all PASS; `pnpm exec tsc --noEmit` PASS; `pnpm build` sukses.
|
||||
3. Smoke: port 4024 bersih → start → curl 9 route (dgn trailing slash) semua **200** → kill server.
|
||||
4. `git status` audit: hanya file scope yang ter-commit (`git checkout --` file liar hasil auto-format).
|
||||
5. Push `origin/main` → `gh run watch` "Build & Deploy (Nix)" → hijau.
|
||||
6. Live: curl `https://imphnen.asepharyana.my.id/<route>/` × 9 → 200.
|
||||
7. **Verifikasi visual sungguhan**: browser_navigate ke live URL → browser_vision: "Apakah ini masih template dashboard?" — jawaban HARUS bukan; screenshot dikirim ke user (`MEDIA:<path>`).
|
||||
8. Checklist anti-default dicentang satu per satu di PR/commit message akhir.
|
||||
|
||||
---
|
||||
|
||||
## Files Likely to Change
|
||||
|
||||
```
|
||||
package.json, pnpm-lock.yaml
|
||||
next.config.ts # hanya jika perlu (tidak diharapkan)
|
||||
src/app/(dashboard)/layout.tsx # wire frame baru
|
||||
src/app/(dashboard)/{9 route}/page.tsx|view.tsx # scene migration
|
||||
src/components/shell/** # frame baru + hapus lama
|
||||
src/lib/constellation/** # baru (murni + test)
|
||||
src/app/globals.css # tokens tambahan + reduced-motion kill list
|
||||
biome.json # hanya jika rule intentional baru perlu "off"
|
||||
```
|
||||
|
||||
## Risks / Tradeoffs
|
||||
|
||||
| Risiko | Mitigasi |
|
||||
|---|---|
|
||||
| Perf GPU di device lemah (full-page WebGL) | Node count kecil (<200/route), pause saat tab hidden, fallback radial statis utk reduced-motion; jika tetap berat → downgrade ke Canvas2D renderer dgn API sama |
|
||||
| router.push no-op (standalone+trailingSlash) | Semua nav pakai `<a href>`; verifikasi klik nyata di browser |
|
||||
| Konten padat (messages 611 l) tak muat di metafora graf | Prinsip: graf = struktur/navigasi; DETAIL tetap lewat overlay panel mengambang — konten tidak dikorbankan, hanya wadahnya yang berubah |
|
||||
| Bun test + Next tsconfig konflik types | Exclude `*.test.ts` dari tsconfig (keputusan Task 1) |
|
||||
| Regress fitur existing (theme toggle, palette, chatbot, mini-player, WS sync) | Dipertahankan apa adanya; smoke + klik verifikasi per phase; sortMessages & snapshot voice TIDAK disentuh logikanya |
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Tidak ada blocker. Satu preferensi opsional: apakah brand mark perlu jam/uptime seperti TopBar lama — default: tidak (minimal). Keputusan bisa diambil saat review Task 4.
|
||||
@@ -0,0 +1,39 @@
|
||||
# GMW FE — Monokrom Hitam-Putih + Sidebar Ala Menu Game + Ringan di Mobile
|
||||
|
||||
Tanggal: 2026-08-24 · Basis: `eda5c75` (shell usable hasil revert)
|
||||
|
||||
## Tujuan
|
||||
1. Tema **monokrom murni** (hitam-putih, tanpa warna) di dark & light.
|
||||
2. Sidebar (desktop NavRail + mobile dock) beranimasi **ala menu game** — corner
|
||||
brackets, sweep, stagger masuk, marker segitiga.
|
||||
3. **Ringan di mobile**: matikan WebGL ambient di layar kecil, kurangi biaya
|
||||
blur/backdrop, animasi transform/opacity saja.
|
||||
|
||||
## Non-goals
|
||||
- Tidak menyentuh backend, endpoint, hooks/data-flow, struktur route.
|
||||
- Tidak menambah dependensi baru (CSS murni untuk semua animasi).
|
||||
|
||||
## File yang disentuh
|
||||
| File | Perubahan |
|
||||
|---|---|
|
||||
| `src/app/globals.css` | Token mono (dark+light): signal/amber/vermilion → skala putih-abu; `.glass` blur adaptif; kelas baru `.game-nav-item` (bracket ::before/::after, sweep, stagger via `--i`), `.game-frame` (panel sudut terpotong + garis tergambar), keyframes `sweep-x`, `draw-line`, `nav-in`; media query `<md`: blur 18→8px, hambat animasi berat |
|
||||
| `src/components/shell/nav-rail.tsx` | Item pakai `.game-nav-item` + `style={{'--i': n}}`; marker aktif jadi segitiga ▸ putih; hapus box-shadow glow besar (ganti sweep) |
|
||||
| `src/components/shell/mobile-nav.tsx` | Dock mono: tab aktif = bar atas putih + sweep sekali; target sentuh ≥44px; hapus glow blob |
|
||||
| `src/components/shell/topbar.tsx` | Aksen mono + `.game-frame` pada container (cek markup dulu) |
|
||||
| `src/components/ambient/ambient-canvas.tsx` | Early-return WebGL bila `(pointer: coarse)` / lebar <768 / `saveData` / core ≤4; fallback statik CSS tetap |
|
||||
| `src/components/ambient/status/signal tone` (`SIGNAL_RGB`) | Semua tone jadi grayscale (putih; intensitas beda per tone) |
|
||||
| `src/app/(dashboard)/dashboard/view.tsx` | Hero + kartu metrik pakai `.game-frame`/cut-corner sebagai showcase |
|
||||
|
||||
## Keputusan desain
|
||||
- **Full monokrom termasuk danger**: flag/moderation tidak lagi merah —
|
||||
ditandai badge putih-di-atlas-hitam inversi + pulse. Kalau user kangen merah,
|
||||
tinggal isi ulang `--color-vermilion`.
|
||||
- Semua animasi hanya `transform`/`opacity` (compositor-friendly), hormati
|
||||
`prefers-reduced-motion` (sudah ada kill-switch global).
|
||||
|
||||
## Verifikasi (gerbang)
|
||||
1. `tsc --noEmit` bersih; biome 0 error 0 warning.
|
||||
2. `pnpm build` sukses; smoke lokal 4024 → 9 route 200.
|
||||
3. Push → GHA "Build & Deploy (Nix)" hijau → live 9×200.
|
||||
4. Visual check live: desktop (rail game-menu terlihat) + cek rule mobile
|
||||
(media query & gate kode) — screenshot disimpan.
|
||||
@@ -93,6 +93,12 @@ type BatchOkResponse = {
|
||||
ok: true;
|
||||
conversationKey: string;
|
||||
rows: MessageRecord[];
|
||||
/**
|
||||
* Race-guard signal (2026-08-25): target ids whose attachment upload is
|
||||
* still in-flight — NO analysis ran for them. The processor must defer
|
||||
* these (requeue + poll), never fan them out as failures.
|
||||
*/
|
||||
uploadPendingIds?: string[];
|
||||
};
|
||||
type BatchErrorResponse = {
|
||||
ok: false;
|
||||
@@ -100,7 +106,16 @@ type BatchErrorResponse = {
|
||||
rows: MessageRecord[];
|
||||
error: string;
|
||||
};
|
||||
type IndividualOkResponse = { ok: true; results: AnalysisResult[] };
|
||||
type IndividualOkResponse = {
|
||||
ok: true;
|
||||
results: AnalysisResult[];
|
||||
/**
|
||||
* Race-guard signal (2026-08-24): the message's attachment upload is still
|
||||
* in-flight — NO analysis ran. The processor must re-queue the message as
|
||||
* `pending` and re-schedule, never treat this as a completed moderation.
|
||||
*/
|
||||
uploadPending?: boolean;
|
||||
};
|
||||
type IndividualErrorResponse = {
|
||||
ok: false;
|
||||
results: AnalysisResult[];
|
||||
@@ -310,7 +325,16 @@ async function processBatch(job: {
|
||||
? messages
|
||||
: messages.filter((m) => !pendingUploadTargetIds.has(m.id));
|
||||
if (readyMessages.length === 0) {
|
||||
return { ok: true, conversationKey, rows: [] };
|
||||
// Explicit signal (2026-08-25): every target is still upload-pending.
|
||||
// Returning bare {ok:true, rows:[]} made the processor classify all of
|
||||
// them "incomplete" and fan out to the individual queue — a hot ~300ms
|
||||
// requeue loop for the whole upload duration.
|
||||
return {
|
||||
ok: true,
|
||||
conversationKey,
|
||||
rows: [],
|
||||
uploadPendingIds: messages.map((m) => m.id),
|
||||
};
|
||||
}
|
||||
|
||||
// The orchestrator handles text/media split + caching + parallel paths
|
||||
@@ -415,7 +439,7 @@ async function processIndividual(job: {
|
||||
(a) => a.message_id === message.id && a.upload_status === "pending",
|
||||
);
|
||||
if (uploadStillPending) {
|
||||
return { ok: true, results: [] };
|
||||
return { ok: true, results: [], uploadPending: true };
|
||||
}
|
||||
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* batchBudget.ts
|
||||
*
|
||||
* Pure batch-sizing helper extracted from batchProcessor.ts so it can be
|
||||
* unit-tested without pulling in the Piscina worker pool, message store,
|
||||
* or any other side-effectful import chain.
|
||||
*/
|
||||
import type { MessageRecord } from "../message-capture/types.js";
|
||||
|
||||
/** Token estimator contract (satisfied by conversationContext.estimateTokens). */
|
||||
export type TokenEstimator = (text: string) => number;
|
||||
|
||||
/**
|
||||
* Picks a batch of messages within a token budget.
|
||||
* `tokensPerMessage` accounts for JSON structure overhead around each entry.
|
||||
* The estimator is injected so this stays a pure function — callers in the
|
||||
* batch pipeline pass the tiktoken-based estimateTokens.
|
||||
*/
|
||||
export function pickBatchWithinBudget(
|
||||
messages: MessageRecord[],
|
||||
maxTokens: number,
|
||||
tokensPerMessage: number,
|
||||
estimateTokens: TokenEstimator,
|
||||
): MessageRecord[] {
|
||||
const batch: MessageRecord[] = [];
|
||||
let usedTokens = 0;
|
||||
|
||||
for (const msg of messages) {
|
||||
const content = msg.edited_content ?? msg.content;
|
||||
const msgTokens = estimateTokens(content) + tokensPerMessage;
|
||||
|
||||
// Stop at the first overflow instead of skipping: input is ordered
|
||||
// created_at ASC, so a contiguous chronological prefix keeps the batch
|
||||
// gap-free. Skipped-over messages would leave unanalyzed holes mid-
|
||||
// timeline; anything past the budget is picked up by the next wave
|
||||
// (processBatch always re-schedules after success).
|
||||
if (usedTokens + msgTokens > maxTokens) {
|
||||
break;
|
||||
}
|
||||
batch.push(msg);
|
||||
usedTokens += msgTokens;
|
||||
}
|
||||
|
||||
return batch;
|
||||
}
|
||||
@@ -0,0 +1,108 @@
|
||||
/**
|
||||
* batchOutcomeClassifier.ts
|
||||
*
|
||||
* Pure partitioner of the batch worker response (2026-08-25).
|
||||
*
|
||||
* Bug history: the batch race guard returned `{ok:true, rows:[]}` when every
|
||||
* target's attachment upload was still in-flight. The processor classified all
|
||||
* of them as "incomplete" and fanned out to the individual queue, where the
|
||||
* guard there requeued + rescheduled at the 250ms debounce — a hot ~300ms loop
|
||||
* for the entire upload duration (~10 cycles in 3s in prod logs). Root fix:
|
||||
* the worker now reports `uploadPendingIds` explicitly and this pure function
|
||||
* partitions the outcome so upload-pending targets NEVER enter the fanout.
|
||||
*/
|
||||
|
||||
export interface BatchRowLike {
|
||||
id?: string;
|
||||
ai_status?: string | null;
|
||||
ai_moderation_flags?: string | null;
|
||||
}
|
||||
|
||||
export interface BatchWorkerResponseLike {
|
||||
ok?: boolean;
|
||||
rows?: BatchRowLike[];
|
||||
/** Explicit race-guard signal from the worker (2026-08-25). */
|
||||
uploadPendingIds?: string[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/** One target's per-message disposition after a batch attempt. */
|
||||
export type BatchTargetKind =
|
||||
| "completed"
|
||||
| "upload_pending"
|
||||
| "incomplete"
|
||||
| "parse_failed"
|
||||
| "api_failed";
|
||||
|
||||
function flagsOf(row: { ai_moderation_flags?: string | null }): string[] {
|
||||
if (!row.ai_moderation_flags) return [];
|
||||
try {
|
||||
const parsed = JSON.parse(row.ai_moderation_flags) as unknown;
|
||||
return Array.isArray(parsed) ? (parsed as string[]) : [];
|
||||
} catch {
|
||||
return [] as string[];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Partition the input message ids into per-message dispositions for one batch
|
||||
* worker response. Pure: no DB/Piscina/logger — unit-testable directly.
|
||||
*
|
||||
* Priority per id: explicit uploadPendingIds → completed row → flag-based
|
||||
* failure kinds → unexplained missing (treated like incomplete).
|
||||
*/
|
||||
export function partitionBatchOutcome(
|
||||
messages: ReadonlyArray<{ id: string }>,
|
||||
response: BatchWorkerResponseLike,
|
||||
): Map<string, BatchTargetKind> {
|
||||
const pendingSet = new Set(response.uploadPendingIds ?? []);
|
||||
const rowsById = new Map(
|
||||
(response.rows ?? [])
|
||||
.filter((r): r is BatchRowLike & { id: string } => Boolean(r?.id))
|
||||
.map((r) => [r.id, r]),
|
||||
);
|
||||
|
||||
const out = new Map<string, BatchTargetKind>();
|
||||
for (const msg of messages) {
|
||||
if (pendingSet.has(msg.id)) {
|
||||
out.set(msg.id, "upload_pending");
|
||||
continue;
|
||||
}
|
||||
const row = rowsById.get(msg.id);
|
||||
if (!row) {
|
||||
// Unexplained drop: LLM silently omitted it. Same retryable bucket as
|
||||
// analysis_incomplete — never a silent success.
|
||||
out.set(msg.id, "incomplete");
|
||||
continue;
|
||||
}
|
||||
if (row.ai_status !== "error") {
|
||||
out.set(msg.id, "completed");
|
||||
continue;
|
||||
}
|
||||
const flags = flagsOf(row);
|
||||
if (flags.includes("analysis_incomplete")) {
|
||||
out.set(msg.id, "incomplete");
|
||||
} else if (flags.includes("analysis_parse_failed")) {
|
||||
out.set(msg.id, "parse_failed");
|
||||
} else if (flags.includes("analysis_api_failed")) {
|
||||
out.set(msg.id, "api_failed");
|
||||
} else {
|
||||
out.set(msg.id, "incomplete");
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Linear backoff ramp for consecutive upload-pending polls:
|
||||
* poll N (1-based) waits min(base × N, cap). Keeps latency low for fast
|
||||
* uploads while bounding total polling cost for long uploads.
|
||||
*/
|
||||
export function computeUploadPollDelayMs(
|
||||
consecutivePolls: number,
|
||||
baseMs: number,
|
||||
capMs: number,
|
||||
): number {
|
||||
const n = Math.max(1, Math.floor(consecutivePolls));
|
||||
return Math.min(Math.round(baseMs * n), Math.round(capMs));
|
||||
}
|
||||
@@ -3,6 +3,11 @@ import { config } from "../../shared/config/config.js";
|
||||
import { isAgeRestrictedMetadata } from "../message-capture/messageMetadata.js";
|
||||
import { messageStore } from "../message-capture/messageStore.js";
|
||||
import type { MessageRecord } from "../message-capture/types.js";
|
||||
import { pickBatchWithinBudget as pickBatchWithinBudgetPure } from "./batchBudget.js";
|
||||
import {
|
||||
computeUploadPollDelayMs,
|
||||
partitionBatchOutcome,
|
||||
} from "./batchOutcomeClassifier.js";
|
||||
import { workerPool } from "./circuitBreaker.js";
|
||||
import { estimateTokens } from "./conversationContext.js";
|
||||
import {
|
||||
@@ -20,11 +25,20 @@ import {
|
||||
|
||||
const logger = createChildLogger("batch-processor");
|
||||
|
||||
/**
|
||||
* Consecutive upload-pending poll counter per conversation (2026-08-25).
|
||||
* Drives the linear backoff ramp while attachments are still uploading;
|
||||
* cleared as soon as a batch comes back with no upload-pending targets.
|
||||
*/
|
||||
const conversationUploadPolls = new Map<string, number>();
|
||||
|
||||
export interface AnalysisWorkerResponse {
|
||||
ok: boolean;
|
||||
conversationKey: string;
|
||||
rows: MessageRecord[];
|
||||
error?: string;
|
||||
/** Explicit upload-in-flight signal from the batch race guard (2026-08-25). */
|
||||
uploadPendingIds?: string[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -39,30 +53,21 @@ export let activeRequests = 0;
|
||||
|
||||
/**
|
||||
* Picks a batch of messages within a token budget.
|
||||
* `tokensPerMessage` accounts for JSON structure overhead around each entry.
|
||||
* Uses a rough character-based token estimate (avoids async formatMessageForPrompt
|
||||
* since this function runs in a synchronous promise chain).
|
||||
* Thin wrapper over the pure helper in batchBudget.ts (kept here so the
|
||||
* existing import surface stays stable); passes the tiktoken-based
|
||||
* estimateTokens. See batchBudget.ts for the overflow-stopping semantics.
|
||||
*/
|
||||
export function pickBatchWithinBudget(
|
||||
messages: MessageRecord[],
|
||||
maxTokens: number,
|
||||
tokensPerMessage: number,
|
||||
): MessageRecord[] {
|
||||
const batch: MessageRecord[] = [];
|
||||
let usedTokens = 0;
|
||||
|
||||
for (const msg of messages) {
|
||||
const content = msg.edited_content ?? msg.content;
|
||||
// Accurate token count via tiktoken (+ overhead for JSON structure)
|
||||
const msgTokens = estimateTokens(content) + tokensPerMessage;
|
||||
|
||||
if (usedTokens + msgTokens <= maxTokens) {
|
||||
batch.push(msg);
|
||||
usedTokens += msgTokens;
|
||||
}
|
||||
}
|
||||
|
||||
return batch;
|
||||
return pickBatchWithinBudgetPure(
|
||||
messages,
|
||||
maxTokens,
|
||||
tokensPerMessage,
|
||||
estimateTokens,
|
||||
);
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -149,6 +154,8 @@ export async function processBatch(
|
||||
|
||||
activeRequests++;
|
||||
let shouldScheduleNext = false;
|
||||
/** Set when upload-pending targets defer the next cycle by this many ms. */
|
||||
let deferredUploadRescheduleMs: number | null = null;
|
||||
try {
|
||||
const result = (await workerPool.run({
|
||||
type: "batch",
|
||||
@@ -207,37 +214,85 @@ export async function processBatch(
|
||||
return;
|
||||
}
|
||||
|
||||
// Batch succeeded -- check for messages the LLM silently dropped or failed
|
||||
const incompleteMessages: MessageRecord[] = [];
|
||||
const parseFailedMessages: MessageRecord[] = [];
|
||||
// Batch succeeded -- partition per-message outcome explicitly (2026-08-25).
|
||||
// upload_pending targets are DEFERRED (never fanned out): the old code
|
||||
// treated them as incomplete -> individual queue -> requeue+250ms
|
||||
// reschedule -> hot ~300ms loop for the whole upload duration.
|
||||
const outcomeById = partitionBatchOutcome(messages, result);
|
||||
const messagesForIndividualQueue: MessageRecord[] = [];
|
||||
const apiFailedMessages: MessageRecord[] = [];
|
||||
const uploadPendingMessages: MessageRecord[] = [];
|
||||
|
||||
for (const msg of messages) {
|
||||
const row = result.rows.find((r) => r.id === msg.id);
|
||||
if (!row) {
|
||||
incompleteMessages.push(msg);
|
||||
continue;
|
||||
}
|
||||
if (row.ai_status === "error") {
|
||||
let flags: string[] = [];
|
||||
try {
|
||||
flags = JSON.parse(row.ai_moderation_flags ?? "[]") as string[];
|
||||
} catch {}
|
||||
|
||||
if (flags.includes("analysis_incomplete")) {
|
||||
incompleteMessages.push(msg);
|
||||
} else if (flags.includes("analysis_parse_failed")) {
|
||||
parseFailedMessages.push(msg);
|
||||
} else if (flags.includes("analysis_api_failed")) {
|
||||
switch (outcomeById.get(msg.id)) {
|
||||
case "upload_pending":
|
||||
uploadPendingMessages.push(msg);
|
||||
break;
|
||||
case "api_failed":
|
||||
// Preserve the dedicated api-failure semantics below: revert +
|
||||
// conversation cooldown instead of an immediate individual retry.
|
||||
apiFailedMessages.push(msg);
|
||||
}
|
||||
break;
|
||||
default:
|
||||
// incomplete / parse_failed / unexplained drops stay retryable via
|
||||
// the individual fallback queue (same semantics as before).
|
||||
messagesForIndividualQueue.push(msg);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
const messagesForIndividualQueue = [
|
||||
...incompleteMessages,
|
||||
...parseFailedMessages,
|
||||
];
|
||||
if (uploadPendingMessages.length > 0) {
|
||||
const polls = (conversationUploadPolls.get(conversationKey) ?? 0) + 1;
|
||||
conversationUploadPolls.set(conversationKey, polls);
|
||||
const delayMs = computeUploadPollDelayMs(
|
||||
polls,
|
||||
config.AI_ANALYSIS_UPLOAD_POLL_MS,
|
||||
config.AI_ANALYSIS_MAX_UPLOAD_POLL_MS,
|
||||
);
|
||||
logger.debug(
|
||||
{
|
||||
conversationKey,
|
||||
count: uploadPendingMessages.length,
|
||||
ids: uploadPendingMessages.map((m) => m.id),
|
||||
pollAttempt: polls,
|
||||
delayMs,
|
||||
},
|
||||
"Attachment upload in-flight for batch targets — deferring with poll backoff",
|
||||
);
|
||||
|
||||
// Put the rows back to `pending` so the scheduler owns them again.
|
||||
await messageStore
|
||||
.updateMessagesAIAnalysisBulk(
|
||||
uploadPendingMessages.map((msg) => ({
|
||||
messageId: msg.id,
|
||||
result: {
|
||||
status: "pending",
|
||||
flags: null,
|
||||
score: null,
|
||||
analysis: null,
|
||||
categories: null,
|
||||
severity: null,
|
||||
confidence: null,
|
||||
recommendedAction: null,
|
||||
analyzedAt: null,
|
||||
error: null,
|
||||
},
|
||||
})),
|
||||
)
|
||||
.catch((err: unknown) => {
|
||||
logger.error(
|
||||
{ error: String(err), ids: uploadPendingMessages.map((m) => m.id) },
|
||||
"Failed to revert upload-pending batch targets to pending",
|
||||
);
|
||||
return [] as MessageRecord[];
|
||||
});
|
||||
|
||||
// Poll backoff instead of the 250ms debounce: the finally-block
|
||||
// schedules the next cycle after this delay instead of immediately.
|
||||
deferredUploadRescheduleMs = delayMs;
|
||||
} else {
|
||||
conversationUploadPolls.delete(conversationKey);
|
||||
}
|
||||
|
||||
if (messagesForIndividualQueue.length > 0) {
|
||||
logger.warn(
|
||||
@@ -315,7 +370,11 @@ export async function processBatch(
|
||||
resetConversationBatchFailures(conversationKey);
|
||||
conversationErrorCooldown.delete(conversationKey);
|
||||
}
|
||||
// Upload-pending defer owns the next-cycle timing; don't let the default
|
||||
// immediate schedule override it.
|
||||
if (deferredUploadRescheduleMs === null) {
|
||||
shouldScheduleNext = true;
|
||||
}
|
||||
} catch (error) {
|
||||
recordConversationBatchFailure(conversationKey);
|
||||
|
||||
@@ -352,7 +411,17 @@ export async function processBatch(
|
||||
if (conversationProcessing.get(conversationKey) === processingStartedAt) {
|
||||
conversationProcessing.delete(conversationKey);
|
||||
}
|
||||
if (shouldScheduleNext) {
|
||||
if (deferredUploadRescheduleMs !== null) {
|
||||
// Upload still in-flight: re-schedule after the backoff delay instead of
|
||||
// immediately (the old path hot-looped at ~250-300ms per cycle).
|
||||
const delayMs = deferredUploadRescheduleMs;
|
||||
setTimeout(() => {
|
||||
// Dynamic import to avoid circular dependency at module scope
|
||||
import("./batchScheduler.js").then((m) =>
|
||||
m.scheduleConversationAnalysis(conversationKey),
|
||||
);
|
||||
}, delayMs).unref();
|
||||
} else if (shouldScheduleNext) {
|
||||
setImmediate(() => {
|
||||
// Dynamic import to avoid circular dependency at module scope
|
||||
import("./batchScheduler.js").then((m) =>
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* fallbackResultClassifier.ts
|
||||
*
|
||||
* Pure classifier for the individual-fallback worker response.
|
||||
*
|
||||
* Bug history (2026-08-24): the worker's upload-pending race guard returned
|
||||
* `{ ok: true, results: [] }` (a legacy "no results yet" signal), but the
|
||||
* processor treated ANY `ok:true` as a successful moderation. Empty results
|
||||
* meant nothing was written to the DB — the message stayed stuck in
|
||||
* `ai_status='processing'` with nobody watching it until the 300s cleanup
|
||||
* reverted it. That single gap produced the ~330-400s attachment delay
|
||||
* cluster. Classification now happens in ONE pure function so every outcome
|
||||
* has an explicit, testable owner.
|
||||
*/
|
||||
|
||||
export type WorkerResultKind =
|
||||
| "success"
|
||||
| "upload_pending"
|
||||
| "incomplete"
|
||||
| "error";
|
||||
|
||||
export interface ClassifiableWorkerResult {
|
||||
ok?: boolean;
|
||||
/** Upload-pending marker set by ai-analysis-worker's race guard. */
|
||||
uploadPending?: boolean;
|
||||
results?: Array<{ status?: string; flags?: string[] | string } | undefined>;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function flagsOf(r: { flags?: string[] | string }): string[] {
|
||||
if (!r.flags) return [];
|
||||
if (Array.isArray(r.flags)) return r.flags;
|
||||
try {
|
||||
const parsed = JSON.parse(r.flags) as unknown;
|
||||
return Array.isArray(parsed) ? (parsed as string[]) : [];
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify an individual-fallback worker response:
|
||||
* - "upload_pending": explicit race-guard signal — retry shortly, NOT an error.
|
||||
* - "success": at least one result and none is analysis_incomplete.
|
||||
* - "incomplete": LLM ran but dropped/failed this message after retries
|
||||
* (analysis_incomplete flag) — terminal exhausted path.
|
||||
* - "error": anything else (ok:false, or ok:true with NO explainable
|
||||
* results). The old code silently succeeded here — never again.
|
||||
*/
|
||||
export function classifyIndividualWorkerResult(
|
||||
result: ClassifiableWorkerResult,
|
||||
): WorkerResultKind {
|
||||
if (result.uploadPending === true) return "upload_pending";
|
||||
const results = (result.results ?? []).filter(
|
||||
(r): r is NonNullable<typeof r> => Boolean(r),
|
||||
);
|
||||
if (results.length === 0) return "error";
|
||||
if (result.ok !== true) return "error";
|
||||
for (const r of results) {
|
||||
const flags = flagsOf(r);
|
||||
if (flags.includes("analysis_incomplete")) return "incomplete";
|
||||
if ((r.status ?? "") === "") return "error";
|
||||
}
|
||||
return "success";
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import type {
|
||||
} from "../message-capture/types.js";
|
||||
import { getConversationKey, workerPool } from "./circuitBreaker.js";
|
||||
import { fireAlert } from "./conversationState.js";
|
||||
import { classifyIndividualWorkerResult } from "./fallbackResultClassifier.js";
|
||||
import {
|
||||
broadcastAnalysisCompleted,
|
||||
LAST_ERROR,
|
||||
@@ -78,26 +79,82 @@ async function processIndividualFallback(
|
||||
message,
|
||||
skipNormalAnalysis: false,
|
||||
} as unknown)) as
|
||||
| { ok: true; results: AnalysisResult[] }
|
||||
| { ok: true; results: AnalysisResult[]; uploadPending?: boolean }
|
||||
| { ok: false; results: AnalysisResult[]; error: string };
|
||||
|
||||
// Explicit outcome classification (2026-08-24): the old code treated any
|
||||
// ok:true as a completed moderation, so the upload-pending race guard's
|
||||
// empty results left messages stuck in `processing` until the 300s
|
||||
// cleanup reverted them — the root cause of the ~330s attachment delays.
|
||||
const kind = classifyIndividualWorkerResult(workerResult);
|
||||
|
||||
if (kind === "upload_pending") {
|
||||
// Attachment still uploading — put the row back to `pending` and
|
||||
// re-schedule this conversation immediately. The next scheduler cycle
|
||||
// (~debounce 250ms) re-fetches; once upload_status flips to done the
|
||||
// race guard passes and analysis proceeds. NOT an error: never touches
|
||||
// the circuit breaker counters.
|
||||
const revertedRows = await messageStore
|
||||
.updateMessagesAIAnalysisBulk([
|
||||
{
|
||||
messageId,
|
||||
result: {
|
||||
status: "pending",
|
||||
flags: null,
|
||||
score: null,
|
||||
analysis: null,
|
||||
categories: null,
|
||||
severity: null,
|
||||
confidence: null,
|
||||
recommendedAction: null,
|
||||
analyzedAt: null,
|
||||
error: null,
|
||||
},
|
||||
},
|
||||
])
|
||||
.catch((dbErr: unknown) => {
|
||||
logger.error(
|
||||
{ messageId, error: String(dbErr) },
|
||||
"Failed to revert upload-pending message to pending",
|
||||
);
|
||||
return [] as MessageRecord[];
|
||||
});
|
||||
for (const row of revertedRows) {
|
||||
broadcastAnalysisCompleted(row);
|
||||
}
|
||||
logger.debug(
|
||||
{ messageId, conversationKey },
|
||||
"Individual fallback: attachment upload in-flight — requeued as pending + rescheduled",
|
||||
);
|
||||
setImmediate(() => {
|
||||
import("./batchScheduler.js")
|
||||
.then((m) => m.scheduleConversationAnalysis(conversationKey))
|
||||
.catch(() => {});
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
let analysisResult: { results: AnalysisResult[] } | null = null;
|
||||
|
||||
if (workerResult.ok) {
|
||||
const stillIncomplete = workerResult.results.some((r) =>
|
||||
r.flags.includes("analysis_incomplete"),
|
||||
);
|
||||
if (stillIncomplete) {
|
||||
if (kind === "success") {
|
||||
analysisResult = workerResult;
|
||||
} else if (kind === "incomplete") {
|
||||
exhaustedOnIncomplete = true;
|
||||
analysisResult = null;
|
||||
} else {
|
||||
analysisResult = workerResult;
|
||||
}
|
||||
// "error" — includes ok:true with unexplainable empty results (the old
|
||||
// silent-success bug). Throw so it is treated as a transient failure.
|
||||
throw new Error(
|
||||
(workerResult as { error?: string }).error ??
|
||||
"Individual worker returned no explainable results",
|
||||
);
|
||||
}
|
||||
|
||||
// No heuristic fallback: an incomplete/errored LLM result stays a
|
||||
// retryable error — the recovery worker picks it up later. Producing a
|
||||
// regex/wordlist verdict here would reintroduce false positives.
|
||||
// (incomplete keeps its exhausted flag so the catch writes the terminal
|
||||
// individual_analysis_exhausted status.)
|
||||
if (!analysisResult) {
|
||||
throw new Error(`LLM analysis failed for message ${messageId}`);
|
||||
}
|
||||
|
||||
@@ -49,6 +49,10 @@ export async function callModerationLLM(
|
||||
targetIds: string[],
|
||||
label: string,
|
||||
signal?: AbortSignal,
|
||||
// Output-side token cap. Defaults to the previous hard-coded value; batch
|
||||
// callers pass a prompt-derived ceiling so small batches don't reserve a
|
||||
// 16k completion budget (some routers pre-allocate KV cache per max_tokens).
|
||||
maxTokens?: number,
|
||||
): Promise<{
|
||||
results: AnalysisResult[];
|
||||
raw: ChatCompletion | null;
|
||||
@@ -75,11 +79,11 @@ export async function callModerationLLM(
|
||||
];
|
||||
const completion = await llmChat({
|
||||
messages,
|
||||
max_tokens: 16384,
|
||||
max_tokens: maxTokens ?? 16384,
|
||||
jsonResponse: { type: "json_object" },
|
||||
retries: 0,
|
||||
signal,
|
||||
// Router (9router/omniroute) always streams SSE even when the
|
||||
// Router (9router / formerly omniroute) always streams SSE even when the
|
||||
// request omits `stream`. In non-stream mode the OpenAI SDK waits
|
||||
// for the FULL body before parsing, so slow/long upstream streams
|
||||
// hit the 30s/60s timeout and abort mid-generation. Streaming mode
|
||||
|
||||
@@ -356,10 +356,10 @@ export async function llmVision(
|
||||
promptText: string,
|
||||
imageUrl: { url: string },
|
||||
): Promise<string | null> {
|
||||
const completion = await llmChat({
|
||||
const params = {
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
role: "user" as const,
|
||||
content: [
|
||||
{ type: "text" as const, text: promptText },
|
||||
{ type: "image_url" as const, image_url: imageUrl },
|
||||
@@ -371,9 +371,26 @@ export async function llmVision(
|
||||
temperature: 0.1,
|
||||
top_p: 0.9,
|
||||
retries: 0,
|
||||
stream: true, // router always streams SSE; non-stream waits for full body and times out
|
||||
timeout: config.AI_LLM_VISION_ANALYSIS_TIMEOUT_MS ?? 60_000,
|
||||
});
|
||||
};
|
||||
|
||||
// Streaming first (the router always streams SSE; a non-stream request
|
||||
// waits for the full body and times out on slow models). Fallback (2026-08-24):
|
||||
// large GIFs/images sometimes get their SSE stream truncated mid-flight by
|
||||
// the upstream ("Stream ended before producing a non-ping SSE event") — all
|
||||
// streaming retries fail identically, so retry ONCE with stream:false where
|
||||
// the router assembles the complete response server-side.
|
||||
let completion: Awaited<ReturnType<typeof llmChat>>;
|
||||
try {
|
||||
completion = await llmChat({ ...params, stream: true });
|
||||
} catch (err) {
|
||||
const msg = err instanceof Error ? err.message : String(err);
|
||||
if (/stream ended before producing a non-ping sse/i.test(msg)) {
|
||||
completion = await llmChat({ ...params, stream: false });
|
||||
} else {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
if (!completion) return null;
|
||||
return completion.choices[0]?.message?.content?.trim() ?? null;
|
||||
|
||||
@@ -13,6 +13,7 @@ import type {
|
||||
MessageRecord,
|
||||
} from "../message-capture/types.js";
|
||||
import { getChannelCulture } from "./channelCultureStore.js";
|
||||
import { estimateTokens } from "./conversationContext.js";
|
||||
import type { RetryState } from "./llmCaller.js";
|
||||
import { callModerationLLM } from "./llmCaller.js";
|
||||
import { prepareMediaMessage } from "./mediaAnalysisClient.js";
|
||||
@@ -87,11 +88,22 @@ export async function runMediaBatch(
|
||||
timeoutId.unref();
|
||||
|
||||
try {
|
||||
// Output budget scales with the prompt (see textBatchProcessor): small
|
||||
// media batches don't need the full 16k completion window.
|
||||
const promptEstimate =
|
||||
2000 +
|
||||
estimateTokens(userContent) +
|
||||
targets.reduce((sum, m) => sum + estimateTokens(m.content ?? "") + 50, 0);
|
||||
const dynamicMaxTokens = Math.min(
|
||||
16384,
|
||||
Math.max(2048, Math.ceil(promptEstimate * 1.5)),
|
||||
);
|
||||
const result = await callModerationLLM(
|
||||
async (_state: RetryState) => ({ system: systemText, user: userContent }),
|
||||
targetIds,
|
||||
`media-batch:${targetIds.length}msgs`,
|
||||
abortController.signal,
|
||||
dynamicMaxTokens,
|
||||
);
|
||||
log.info(
|
||||
{ mediaCount: targets.length, resultCount: result.results.length },
|
||||
|
||||
@@ -75,8 +75,6 @@ export function buildConversationContextBlock(input: {
|
||||
// huge paste (stack traces, log dumps, copypasta). Truncation is explicit so
|
||||
// the model never mistakes the cut for a real message boundary.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Max characters of a message's content sent to the LLM `<content>` payload. */
|
||||
export const AI_CONTENT_MAX_CHARS = 4000;
|
||||
|
||||
/** Marker appended when a message is longer than AI_CONTENT_MAX_CHARS. */
|
||||
@@ -89,79 +87,12 @@ export function truncateForAi(content: string): string {
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// User profile deduplication — a batch can contain many messages from the
|
||||
// same user. Instead of repeating the (up to 3000-char) profile summary on
|
||||
// every message, emit a single <user_profiles> map per batch and reference
|
||||
// entries per message with <user_profile_ref user_id="..."/>.
|
||||
// User profile deduplication — REMOVED (2026-08-22).
|
||||
// Per-user profile/history context was stripped from the moderation prompt
|
||||
// (context minimization): buildUserProfilesBlock / buildUserProfileRef /
|
||||
// UserProfileEntry / buildUserHistoryXml had no remaining production callers.
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface UserProfileEntry {
|
||||
/** Profile summary text (from user_profiles.profile_summary). */
|
||||
text: string;
|
||||
/** Epoch ms when the profile was last generated — staleness signal for
|
||||
* the LLM (a profile from months ago may not reflect current behavior). */
|
||||
asOf?: number | null;
|
||||
}
|
||||
|
||||
/** Build a deduplicated `<user_profiles>` map block, keyed by Discord user id. */
|
||||
export function buildUserProfilesBlock(
|
||||
profiles: ReadonlyMap<string, UserProfileEntry>,
|
||||
): string {
|
||||
const entries = Array.from(profiles.entries()).filter(
|
||||
([, entry]) => entry.text.trim().length > 0,
|
||||
);
|
||||
if (entries.length === 0) return "";
|
||||
const lines = entries.map(([userId, entry]) => {
|
||||
const asOfAttr =
|
||||
typeof entry.asOf === "number" && entry.asOf > 0
|
||||
? ` as_of="${new Date(entry.asOf).toISOString()}"`
|
||||
: "";
|
||||
return ` <user_profile user_id="${escapeXml(userId)}"${asOfAttr}>${sanitizeAiContent(entry.text)}</user_profile>`;
|
||||
});
|
||||
return `<user_profiles>\n${lines.join("\n")}\n</user_profiles>`;
|
||||
}
|
||||
|
||||
/** Per-message reference tag pointing at an entry in the `<user_profiles>` map. */
|
||||
export function buildUserProfileRef(userId: string): string {
|
||||
return `<user_profile_ref user_id="${escapeXml(userId)}"/>`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Per-user history context (last flagged messages only — no trust model).
|
||||
// context to AI moderation.
|
||||
// ---------------------------------------------------------------------------
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
/**
|
||||
* Builds an optional `<user_history>` block (last flagged messages) from
|
||||
* getUserRecentInfractions rows. Only emitted when there is real history —
|
||||
* lets the LLM see the PATTERN (e.g. the same scam link posted repeatedly)
|
||||
* without treating old flags as proof for the current message.
|
||||
*/
|
||||
export function buildUserHistoryXml(
|
||||
history: Array<{
|
||||
content: string;
|
||||
severity: string | null;
|
||||
created_at: number;
|
||||
}>,
|
||||
now: number = Date.now(),
|
||||
): string {
|
||||
const filtered = history.filter((h) => h.content?.trim());
|
||||
if (filtered.length === 0) return "";
|
||||
const lines = filtered.map((h) => {
|
||||
const daysAgo = Math.max(0, Math.floor((now - h.created_at) / DAY_MS));
|
||||
const severityAttr = h.severity
|
||||
? ` severity="${escapeXml(h.severity)}"`
|
||||
: "";
|
||||
const snippet =
|
||||
h.content.length > 100
|
||||
? `${h.content.slice(0, 100).trimEnd()}…`
|
||||
: h.content;
|
||||
return ` <infraction${severityAttr} time_ago_days="${daysAgo}">${escapeXml(snippet)}</infraction>`;
|
||||
});
|
||||
return `<user_history>\n${lines.join("\n")}\n</user_history>`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the message author was a bot (captured in metadata.author.bot).
|
||||
* Bot posts (logging bots, webhook-style automation) deserve different
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
* Orchestrates LLM-based moderation analysis — manages batch splitting,
|
||||
* parallel text+media analysis, LLM calls with retry, and cache handling.
|
||||
*/
|
||||
|
||||
import { LRUCache } from "lru-cache";
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { config } from "../../shared/config/config.js";
|
||||
import { extractMessageMediaEvidence } from "../message-capture/messageMetadata.js";
|
||||
@@ -20,16 +22,29 @@ import { isQdrantConfigured, searchQdrantBatch } from "./qdrantClient.js";
|
||||
import { logCacheEvent } from "./responseLogger.js";
|
||||
import { runTextOnlyBatch } from "./textBatchProcessor.js";
|
||||
import {
|
||||
bumpTextModerationHitCounts,
|
||||
ERROR_ARTIFACT_FLAGS,
|
||||
findSimilarTextModeration,
|
||||
getCachedTextModeration,
|
||||
getCachedTextModerations,
|
||||
isGloballyReusableCleanVerdict,
|
||||
isSemanticBandAccepted,
|
||||
makeModerationContextKey,
|
||||
makeTextModerationCacheKey,
|
||||
parseQdrantVerdict,
|
||||
type StoredModerationVerdict,
|
||||
setCachedTextModeration,
|
||||
upsertBareKeyToQdrant,
|
||||
} from "./textCacheStore.js";
|
||||
|
||||
const log = createChildLogger("moderationOrchestrator");
|
||||
|
||||
/**
|
||||
* Bare keys already written this process (dual-key write-back dedupe).
|
||||
* LRU-bounded so a long-lived gateway can't grow it without limit; the DB
|
||||
* upsert underneath is idempotent anyway — this just avoids redundant writes.
|
||||
*/
|
||||
const globalBareKeysWritten = new LRUCache<string, true>({ max: 5000 });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Types
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -73,46 +88,74 @@ export async function runModerationAnalysis(
|
||||
initCacheStore(config.REDIS_URL);
|
||||
if (!targets.length) throw new Error("No targets provided for analysis");
|
||||
|
||||
// ── Phase 1: exact-hash cache (per conversation context) ────────────────
|
||||
// ── Phase 1: exact-hash cache — ONE batched DB query ────────────────────
|
||||
// Key is content + conversation context (channel/thread). On a scoped miss
|
||||
// we also probe the legacy bare key: verdicts that CANNOT trigger an action
|
||||
// (clean, flagless, action=none) may be reused across channels under strict
|
||||
// freshness + confidence guards — flagged/warn verdicts never leave their
|
||||
// conversation. This replaced the old N-sequential-query loop (60-message
|
||||
// burst = 60 PgBouncer round-trips before).
|
||||
const cacheHits: AnalysisResult[] = [];
|
||||
const uncachedTargets: MessageRecord[] = [];
|
||||
// cacheKey → result for identical-content dedupe within one batch
|
||||
// cacheKey → representative result for identical-content dedupe
|
||||
const hitByKey = new Map<string, AnalysisResult>();
|
||||
// Embedding per exact cache key — computed once during lookup, reused
|
||||
// when the fresh LLM verdict is written back to the semantic cache.
|
||||
const embeddingsByKey = new Map<string, number[]>();
|
||||
|
||||
interface ExactCandidate {
|
||||
target: MessageRecord;
|
||||
scopedKey: string;
|
||||
bareKey: string;
|
||||
}
|
||||
const candidates: ExactCandidate[] = [];
|
||||
for (const target of targets) {
|
||||
const hasMedia = hasMediaContent(target, attachments);
|
||||
if (hasMedia) {
|
||||
if (hasMediaContent(target, attachments)) {
|
||||
uncachedTargets.push(target);
|
||||
continue;
|
||||
}
|
||||
|
||||
const rawContent = target.edited_content ?? target.content;
|
||||
if (!rawContent.trim()) {
|
||||
uncachedTargets.push(target);
|
||||
continue;
|
||||
}
|
||||
|
||||
const cacheKey = makeTextModerationCacheKey(
|
||||
candidates.push({
|
||||
target,
|
||||
scopedKey: makeTextModerationCacheKey(
|
||||
rawContent,
|
||||
makeModerationContextKey(target),
|
||||
);
|
||||
const seen = hitByKey.get(cacheKey);
|
||||
if (seen) {
|
||||
// Same content already resolved this batch — reuse the verdict.
|
||||
cacheHits.push({ ...seen, messageId: target.id });
|
||||
continue;
|
||||
),
|
||||
bareKey: makeTextModerationCacheKey(rawContent),
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
const cached = await getCachedTextModeration(cacheKey);
|
||||
if (cached) {
|
||||
// Identical content within one batch resolves once (representative).
|
||||
const firstByScopedKey = new Map<string, ExactCandidate>();
|
||||
for (const c of candidates) {
|
||||
if (!firstByScopedKey.has(c.scopedKey))
|
||||
firstByScopedKey.set(c.scopedKey, c);
|
||||
}
|
||||
|
||||
// Single round-trip for every key we might serve from (scoped + bare).
|
||||
const storedEntries = await getCachedTextModerations([
|
||||
...firstByScopedKey.keys(),
|
||||
...Array.from(firstByScopedKey.values(), (c) => c.bareKey),
|
||||
]);
|
||||
// Keys actually served — bumped in one UPDATE at the end for metrics.
|
||||
const servedCacheKeys = new Set<string>();
|
||||
|
||||
/** Validate + admit one stored verdict for a candidate. */
|
||||
const acceptExactVerdict = (
|
||||
candidate: ExactCandidate,
|
||||
cacheKey: string,
|
||||
entry: { verdict: StoredModerationVerdict },
|
||||
policyVersion: string,
|
||||
): boolean => {
|
||||
const { verdict } = entry;
|
||||
const hasMediaInMeta =
|
||||
target.metadata &&
|
||||
candidate.target.metadata &&
|
||||
(() => {
|
||||
const ev = extractMessageMediaEvidence(target.metadata);
|
||||
const ev = extractMessageMediaEvidence(candidate.target.metadata);
|
||||
return (
|
||||
ev.attachments.length > 0 ||
|
||||
ev.stickers.length > 0 ||
|
||||
@@ -122,48 +165,87 @@ export async function runModerationAnalysis(
|
||||
|
||||
if (hasMediaInMeta) {
|
||||
log.debug(
|
||||
{ messageId: target.id, cacheKey },
|
||||
{ messageId: candidate.target.id, cacheKey },
|
||||
"Cache entry but message has media — treating as miss",
|
||||
);
|
||||
} else if (
|
||||
cached.flags.some((f) =>
|
||||
[
|
||||
"analysis_api_failed",
|
||||
"analysis_parse_failed",
|
||||
"analysis_incomplete",
|
||||
].includes(f),
|
||||
return false;
|
||||
}
|
||||
if (
|
||||
verdict.flags.some((f) =>
|
||||
(ERROR_ARTIFACT_FLAGS as readonly string[]).includes(f),
|
||||
)
|
||||
) {
|
||||
log.warn(
|
||||
{ messageId: target.id, cacheKey },
|
||||
{ messageId: candidate.target.id, cacheKey },
|
||||
"Cache entry contains error artifact — treating as miss",
|
||||
);
|
||||
} else {
|
||||
const hit: AnalysisResult = {
|
||||
messageId: target.id,
|
||||
status: cached.status,
|
||||
flags: cached.flags,
|
||||
score: cached.score,
|
||||
analysis: cached.analysis,
|
||||
categories: cached.categories,
|
||||
severity: cached.severity as AnalysisResult["severity"],
|
||||
confidence: cached.confidence,
|
||||
recommendedAction:
|
||||
cached.recommendedAction as AnalysisResult["recommendedAction"],
|
||||
policyVersion: "cached-user-moderation-2026-06",
|
||||
evidence: [],
|
||||
};
|
||||
cacheHits.push(hit);
|
||||
hitByKey.set(cacheKey, hit);
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
/* proceed */
|
||||
return false;
|
||||
}
|
||||
|
||||
uncachedTargets.push(target);
|
||||
hitByKey.set(candidate.scopedKey, {
|
||||
messageId: candidate.target.id,
|
||||
status: verdict.status,
|
||||
flags: verdict.flags,
|
||||
score: verdict.score,
|
||||
analysis: verdict.analysis,
|
||||
categories: verdict.categories,
|
||||
severity: verdict.severity as AnalysisResult["severity"],
|
||||
confidence: verdict.confidence,
|
||||
recommendedAction:
|
||||
verdict.recommendedAction as AnalysisResult["recommendedAction"],
|
||||
policyVersion,
|
||||
evidence: [],
|
||||
});
|
||||
servedCacheKeys.add(cacheKey);
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
return true;
|
||||
};
|
||||
|
||||
for (const candidate of firstByScopedKey.values()) {
|
||||
const scopedEntry = storedEntries.get(candidate.scopedKey);
|
||||
if (
|
||||
scopedEntry &&
|
||||
acceptExactVerdict(
|
||||
candidate,
|
||||
candidate.scopedKey,
|
||||
scopedEntry,
|
||||
"cached-user-moderation-2026-06",
|
||||
)
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Context-free fallback: ONLY non-actionable clean verdicts qualify
|
||||
// (guard enforces status/flags/action/confidence/freshness). The bare
|
||||
// key equals the scoped key for context-less messages, so the guard
|
||||
// also prevents double-serving the same row.
|
||||
const bareEntry = storedEntries.get(candidate.bareKey);
|
||||
if (
|
||||
bareEntry &&
|
||||
candidate.bareKey !== candidate.scopedKey &&
|
||||
isGloballyReusableCleanVerdict(
|
||||
bareEntry.verdict,
|
||||
bareEntry.analyzedAt ?? undefined,
|
||||
)
|
||||
) {
|
||||
acceptExactVerdict(
|
||||
candidate,
|
||||
candidate.bareKey,
|
||||
bareEntry,
|
||||
"cached-global-clean-2026-08",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// Fan-out: every candidate (representative + in-batch duplicates) gets its
|
||||
// own copy of the representative verdict; unresolved ones stay queued.
|
||||
for (const candidate of candidates) {
|
||||
const representative = hitByKey.get(candidate.scopedKey);
|
||||
if (representative) {
|
||||
cacheHits.push({ ...representative, messageId: candidate.target.id });
|
||||
} else {
|
||||
uncachedTargets.push(candidate.target);
|
||||
}
|
||||
}
|
||||
|
||||
// ── Phase 2: semantic cache — batched (one embed call + one Qdrant
|
||||
@@ -199,10 +281,12 @@ export async function runModerationAnalysis(
|
||||
}
|
||||
|
||||
if (isQdrantConfigured()) {
|
||||
// ONE batch search at the LOOSER threshold; per-hit re-classification
|
||||
// enforces the strict band for actionable verdicts.
|
||||
const batchHits = await searchQdrantBatch(
|
||||
embeddings,
|
||||
config.AI_LLM_EMBEDDING_MAX_CANDIDATES,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN,
|
||||
);
|
||||
for (let i = 0; i < semanticCandidates.length; i++) {
|
||||
const { target, cacheKey } = semanticCandidates[i];
|
||||
@@ -210,6 +294,7 @@ export async function runModerationAnalysis(
|
||||
if (hits.length === 0) continue;
|
||||
const verdict = parseQdrantVerdict(hits[0].payload, hits[0].score);
|
||||
if (!verdict) continue;
|
||||
if (!isSemanticBandAccepted(verdict, verdict.similarity)) continue;
|
||||
log.debug(
|
||||
{
|
||||
messageId: target.id,
|
||||
@@ -234,6 +319,7 @@ export async function runModerationAnalysis(
|
||||
};
|
||||
cacheHits.push(hit);
|
||||
hitByKey.set(cacheKey, hit);
|
||||
servedCacheKeys.add(cacheKey); // bump hit_count for metrics
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
}
|
||||
} else {
|
||||
@@ -242,10 +328,12 @@ export async function runModerationAnalysis(
|
||||
const { target, cacheKey } = semanticCandidates[i];
|
||||
const semantic = await findSimilarTextModeration(
|
||||
embeddings[i],
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY,
|
||||
config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN,
|
||||
config.AI_LLM_EMBEDDING_MAX_CANDIDATES,
|
||||
);
|
||||
if (!semantic) continue;
|
||||
if (!isSemanticBandAccepted(semantic, semantic.similarity))
|
||||
continue;
|
||||
log.debug(
|
||||
{
|
||||
messageId: target.id,
|
||||
@@ -270,6 +358,7 @@ export async function runModerationAnalysis(
|
||||
};
|
||||
cacheHits.push(hit);
|
||||
hitByKey.set(cacheKey, hit);
|
||||
servedCacheKeys.add(cacheKey); // bump hit_count for metrics
|
||||
logCacheEvent("hit", cacheKey, "text");
|
||||
}
|
||||
}
|
||||
@@ -290,11 +379,14 @@ export async function runModerationAnalysis(
|
||||
}
|
||||
|
||||
if (cacheHits.length > 0) {
|
||||
// Metrics: one bulk UPDATE for every exact-cache key actually served.
|
||||
bumpTextModerationHitCounts(Array.from(servedCacheKeys));
|
||||
log.info(
|
||||
{
|
||||
cacheHits: cacheHits.length,
|
||||
uncached: uncachedTargets.length,
|
||||
total: targets.length,
|
||||
servedKeys: servedCacheKeys.size,
|
||||
},
|
||||
"User moderation cache applied",
|
||||
);
|
||||
@@ -355,9 +447,7 @@ export async function runModerationAnalysis(
|
||||
rawContent,
|
||||
makeModerationContextKey(target),
|
||||
);
|
||||
setCachedTextModeration(
|
||||
cacheKey,
|
||||
{
|
||||
const stored = {
|
||||
flags: result.flags ?? [],
|
||||
score: result.score ?? 0,
|
||||
analysis: result.analysis ?? "",
|
||||
@@ -366,9 +456,50 @@ export async function runModerationAnalysis(
|
||||
confidence: result.confidence ?? result.score ?? 0,
|
||||
recommendedAction: result.recommendedAction ?? "none",
|
||||
status: result.status,
|
||||
},
|
||||
};
|
||||
setCachedTextModeration(
|
||||
cacheKey,
|
||||
stored,
|
||||
embeddingsByKey.get(cacheKey),
|
||||
).catch(() => {});
|
||||
|
||||
// Dual-key write-back (2026-08-24): the FIRST analysis of a message runs
|
||||
// WITH conversation context (accurate), but its verdict is also stored
|
||||
// under the context-free bare key so repeats in OTHER channels hit the
|
||||
// exact cache instead of paying a new LLM call. Same guard as the read
|
||||
// path — only non-actionable clean verdicts may cross channels.
|
||||
//
|
||||
// 2026-08-25 cache-hit fix: the bare key is ALSO upserted to Qdrant
|
||||
// (via upsertBareKeyToQdrant) with the SAME embedding already computed
|
||||
// at lookup time. Previously the bare key was only PG-written with
|
||||
// embedding=null — bare clean verdicts were DB-only and invisible to
|
||||
// searchQdrantBatch, capping the semantic hit-rate below the exact-cache
|
||||
// hit-rate for cross-channel repeats.
|
||||
const bareKey = makeTextModerationCacheKey(rawContent);
|
||||
if (
|
||||
bareKey !== cacheKey &&
|
||||
!globalBareKeysWritten.has(bareKey) &&
|
||||
isGloballyReusableCleanVerdict(
|
||||
{
|
||||
status: stored.status,
|
||||
flags: stored.flags,
|
||||
score: stored.score,
|
||||
analysis: stored.analysis,
|
||||
categories: stored.categories,
|
||||
severity: stored.severity,
|
||||
confidence: stored.confidence,
|
||||
recommendedAction: stored.recommendedAction,
|
||||
},
|
||||
undefined,
|
||||
)
|
||||
) {
|
||||
globalBareKeysWritten.set(bareKey, true);
|
||||
setCachedTextModeration(bareKey, stored, null).catch(() => {});
|
||||
const bareEmbedding = embeddingsByKey.get(cacheKey);
|
||||
if (bareEmbedding && bareEmbedding.length > 0) {
|
||||
upsertBareKeyToQdrant(bareKey, stored, bareEmbedding).catch(() => {});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const allResults = [
|
||||
|
||||
@@ -45,18 +45,18 @@ Wajib sebutkan ISI/KONTEN spesifik apa yang dibicarakan pengirim — bukan templ
|
||||
- **Teks + gambar**: "Pengirim mengirim screenshot chat sambil membahas makanan favorit. Gambar dan teks sama-sama tentang percakapan sehari-hari. Tidak ada pelanggaran." ✓ / "Pesan berisi teks dan gambar tanpa pelanggaran." ✗
|
||||
|
||||
### Per kasus:
|
||||
|- **Melanggar:** "Pengirim <pelanggaran X>. <bukti teks/gambar>. <dampak/konteks>."
|
||||
|- **conflict_instigation:** "Pengirim <ajakan memicu konflik>. <konteks>. Diberi peringatan karena berpotensi memicu drama."
|
||||
|- **Username ofensif (pesan bersih):** "Pengirim memiliki username yang <alasan ofensif>. Isi pesan hanya <isi>. Diberi warning ringan." — (pesan memperkuat): "<username SARA> + isi pesan memperkuat tone kebencian. Pelanggaran berat."
|
||||
|- **Evasi (zalgo/leetspeak):** "Pengirim menggunakan teknik obfuscation untuk menyembunyikan <makna asli>. <dampak>. <kesimpulan>."
|
||||
|- **Spam (repetitions > 1):** "Pengirim mengirim teks yang sama sebanyak N kali dalam waktu singkat. <isi pesan>. Diberi peringatan karena spam berulang." — nilai tetap dari isi; pengulangan saja (mis. "ok" x5 dalam obrolan aktif) bukan pelanggaran.
|
||||
|- **sexual_deviation:** "Pengirim <konten penyimpangan>. <konteks>. Melanggar kebijikan server."
|
||||
|- **SARA/penistaan agama:** "Pengirim <jenis penistaan spesifik: parodi ayat, mengaku Tuhan, mockery ritual, istilah agama sebagai joke, provokasi antar-agama>. <bukti>. Melanggar kebijikan SARA." — JANGAN gunakan kata "bercanda" untuk SARA.
|
||||
- **Melanggar:** "Pengirim <pelanggaran X>. <bukti teks/gambar>. <dampak/konteks>."
|
||||
- **conflict_instigation:** "Pengirim <ajakan memicu konflik>. <konteks>. Diberi peringatan karena berpotensi memicu drama."
|
||||
- **Username ofensif (pesan bersih):** "Pengirim memiliki username yang <alasan ofensif>. Isi pesan hanya <isi>. Diberi warning ringan." — (pesan memperkuat): "<username SARA> + isi pesan memperkuat tone kebencian. Pelanggaran berat."
|
||||
- **Evasi (zalgo/leetspeak):** "Pengirim menggunakan teknik obfuscation untuk menyembunyikan <makna asli>. <dampak>. <kesimpulan>."
|
||||
- **Spam (repetitions > 1):** "Pengirim mengirim teks yang sama sebanyak N kali dalam waktu singkat. <isi pesan>. Diberi peringatan karena spam berulang." — nilai tetap dari isi; pengulangan saja (mis. "ok" x5 dalam obrolan aktif) bukan pelanggaran.
|
||||
- **sexual_deviation:** "Pengirim <konten penyimpangan>. <konteks>. Melanggar kebijikan server."
|
||||
- **SARA/penistaan agama:** "Pengirim <jenis penistaan spesifik: parodi ayat, mengaku Tuhan, mockery ritual, istilah agama sebagai joke, provokasi antar-agama>. <bukti>. Melanggar kebijikan SARA." — JANGAN gunakan kata "bercanda" untuk SARA.
|
||||
|
||||
**CRITICAL — dilarang menulis analysis generik:** JANGAN PERNAH menulis "Pesan hanya berisi...", "Tidak ada indikasi pelanggaran", atau template seperti "Pengirim mengirimkan sebuah file GIF tanpa pelanggaran." Selalu sebutkan ISI/KONTEN secifik, apa yang dibicarakan, apa yang terlihat.
|
||||
**CRITICAL — dilarang menulis analysis generik:** JANGAN PERNAH menulis "Pesan hanya berisi...", "Tidak ada indikasi pelanggaran", atau template seperti "Pengirim mengirimkan sebuah file GIF tanpa pelanggaran." Selalu sebutkan ISI/KONTEN spesifik, apa yang dibicarakan, apa yang terlihat.
|
||||
|
||||
|- **BALASAN (reply):** jelaskan konteks balasannya (apa dibicarakan, siapa dibalas tanpa nama, bagaimana tanggapan pengirim).
|
||||
|- Gunakan Media analysis untuk mendeskripsikan gambar. Analisis harus MEMBERI KONTEKS, bukan hanya status.`;
|
||||
- **BALASAN (reply):** jelaskan konteks balasannya (apa dibicarakan, siapa dibalas tanpa nama, bagaimana tanggapan pengirim).
|
||||
- Gunakan Media analysis untuk mendeskripsikan gambar. Analisis harus MEMBERI KONTEKS, bukan hanya status.`;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Sanitize AI-generated content (channel culture / user profile) to prevent
|
||||
|
||||
@@ -48,7 +48,7 @@ Kata alat kelamin/anatomi seksual (kontol, memek, titten, tit, dick) atau istila
|
||||
- Ageisme ("dasar bocil", "tau aja lo tua") → hate_speech / harassment.
|
||||
- Diskriminasi fisik ("gendut", "iteman", "cungkring") → harassment jika terarah.
|
||||
- Serangan personal, penghinaan, merendahkan = tidak ditoleransi. Perbedaan pendapat wajar.
|
||||
- **PESAN DINILAI SECARA STANDALONE:** Setiap pesan baru dinilai BERDASARKAN ISINYA SENDIRI. <user_history> (jika ada) HANYA untuk mendeteksi POLA PENGULANGAN dengan JAMAK (spam link yang SAMA, provokasi berulang yang MENGANDALKAN KONTEN YANG SAMA). JANGAN gunakan history untuk "menginterpretasi ulang" pesan bersih yang TERPISAH DARI riwayat pelanggaran sebelumnya. Jika pesan tidak mengandung unsur yang BERPANDUAN PADA riwayat → tetap CLEAN.
|
||||
- **PESAN DINILAI SECARA STANDALONE:** Setiap pesan baru dinilai BERDASARKAN ISINYA SENDIRI. Tidak ada data profil/reputasi per-user di payload — nilai murni dari isi pesan + konteks percakapan. Pengulangan teks sama dalam satu batch ditandai atribut repetitions="N" pada <message> (sinyal spam), bukan dari riwayat lampau.
|
||||
|
||||
## LARANGAN BERAT (ZERO TOLERANCE)
|
||||
- **LGBT:** Segala promosi, diskusi, pengakuan orientasi, coming out, atau curhat personal tentang LGBT WAJIB di-flag "sexual_deviation". Tidak ada pengecualian. (Lihat juga pohon keputusan #3.)
|
||||
|
||||
@@ -125,14 +125,14 @@ function buildSystemPromptCore(
|
||||
`- <location_context .../>: metadata channel/thread (channel_name, thread_name, topic, nsfw, age_restricted). topic = tujuan resmi channel; gunakan menilai kesesuaian pesan.\n` +
|
||||
`- <conversation_context>: obrolan SEBELUM target. Baris pertama "[conversation_flow] status=... context_msgs=... dropped=..." = metadata sistem (ongoing/sparse/cold_start), BUKAN pesan dinilai. Baris "[context] id=... time=... user=...: isi" = konteks, BUKAN target.\n` +
|
||||
`- Tidak ada data profil/reputasi per-user di context — nilai tiap pesan murni dari isinya + <conversation_context> + <web_searches> + <location_context>.\n` +
|
||||
`- <web_searches>/<web_content>: bukti web (prioritas tertinggi). <term_glossary>: definisi kata/slang/jargon (SearXNG) — pakai pahami kata asing, JANGAN tebak arti.\n` +
|
||||
`- <web_searches>/<web_content>: bukti web (prioritas tertinggi). <term_glossary>: definisi kata/slang/jargon (Wikipedia) — pakai pahami kata asing, JANGAN tebak arti.\n` +
|
||||
`- <messages_to_analyze>: pesan TARGET yang WAJIB dinilai. Atribut <message>: id, user, time (ISO), repetitions (N = teks sama muncul N× di batch → sinyal spam), bot (true = bot), edited (true = hasil edit setelah posting → evasi potensial).`,
|
||||
);
|
||||
|
||||
parts.push(
|
||||
`## Framing & Aturan Konteks\n` +
|
||||
`- Hasilkan SATU hasil per message_id — jangan gabung, lewati, atau karang id.\n` +
|
||||
`- Setiap target dinilai BERDASARKAN ISINYA SENDIRI. Konteks memengaruhi interpretasi, tapi TIDAK menggantikan isi pesan. Profil/riwayat = REFERENSI personalisasi, BUKAN bukti pelanggaran (lihat "PERSONALITY & MEMORI").\n` +
|
||||
`- Setiap target dinilai BERDASARKAN ISINYA SENDIRI. Konteks memengaruhi interpretasi, tapi TIDAK menggantikan isi pesan.\n` +
|
||||
`- Marker "[pesan dipotong: terlalu panjang]" = TARGET dipotong; "[konteks dipotong: ...]" = konteks dipotong. Nilai dari bagian terlihat; pemotongan BUKAN pelanggaran/evasi.\n` +
|
||||
`- time= = kapan dikirim (rekonsiliasi spam beruntun / bump pesan lama). bot=true = otomatisasi, bukan pelanggaran personal.`,
|
||||
);
|
||||
|
||||
@@ -14,6 +14,7 @@ import type {
|
||||
MessageRecord,
|
||||
} from "../message-capture/types.js";
|
||||
import { getChannelCulture } from "./channelCultureStore.js";
|
||||
import { estimateTokens } from "./conversationContext.js";
|
||||
import type { ModerationPromptContent, RetryState } from "./llmCaller.js";
|
||||
import { callModerationLLM } from "./llmCaller.js";
|
||||
import { analyzeSingleMediaImage } from "./mediaAnalysisClient.js";
|
||||
@@ -341,11 +342,29 @@ export async function runTextOnlyBatch(
|
||||
|
||||
let batchResult: { results: AnalysisResult[]; raw: unknown };
|
||||
try {
|
||||
// Output budget scales with the prompt: the JSON verdict block is
|
||||
// roughly proportional to message count, so a small sub-batch doesn't
|
||||
// need to reserve a full 16k completion window. Estimated here from
|
||||
// raw materials (system/rules baseline ~2k + context + message
|
||||
// bodies) instead of inside buildContent, because max_tokens must be
|
||||
// known at call time.
|
||||
const subBatchPromptEstimate =
|
||||
2000 +
|
||||
estimateTokens(contextBlock ?? "") +
|
||||
batch.reduce(
|
||||
(sum, m) => sum + estimateTokens(m.edited_content ?? m.content) + 50,
|
||||
0,
|
||||
);
|
||||
const dynamicMaxTokens = Math.min(
|
||||
16384,
|
||||
Math.max(2048, Math.ceil(subBatchPromptEstimate * 1.5)),
|
||||
);
|
||||
batchResult = await callModerationLLM(
|
||||
buildContent,
|
||||
targetIds,
|
||||
`text-batch-${i + 1}`,
|
||||
abortController.signal,
|
||||
dynamicMaxTokens,
|
||||
);
|
||||
} catch (err: any) {
|
||||
if (err.name === "AbortError" || abortController.signal.aborted) {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { config } from "../../shared/config/config.js";
|
||||
import { executeAll, executeGet } from "../../shared/database/drizzle.js";
|
||||
import { findBestEmbeddingMatch } from "./embeddingClient.js";
|
||||
import {
|
||||
@@ -84,10 +85,43 @@ export function makeImageCacheKey(imageUrl: string): string {
|
||||
// size is 8191"), which fails acquireMediaAnalysisLock and silently skips
|
||||
// every media analysis. A 32-char sha256 keeps the key well under the limit
|
||||
// and is still deterministic (same attachment → same key).
|
||||
const hash = createHash("sha256").update(imageUrl).digest("hex").slice(0, 32);
|
||||
const hash = createHash("sha256")
|
||||
.update(normalizeDiscordImageUrl(imageUrl))
|
||||
.digest("hex")
|
||||
.slice(0, 32);
|
||||
return `image:${hash}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Strip volatile query params from Discord CDN URLs so the SAME attachment
|
||||
* always maps to ONE vision-cache key regardless of how it reached us
|
||||
* (signed `?ex=&is=&hm=` tokens rotate per fetch; render variants differ by
|
||||
* `format/width/height/size`). Previously each token variant hashed to its
|
||||
* own key → the same image was re-downloaded and re-analyzed by the vision
|
||||
* model once per variant. Non-Discord URLs and data: URLs are returned
|
||||
* untouched (their query can be semantically meaningful).
|
||||
*/
|
||||
export function normalizeDiscordImageUrl(imageUrl: string): string {
|
||||
try {
|
||||
const parsed = new URL(imageUrl);
|
||||
if (parsed.protocol !== "https:" && parsed.protocol !== "http:") {
|
||||
return imageUrl;
|
||||
}
|
||||
const host = parsed.hostname;
|
||||
const isAttachmentCdn = host === "cdn.discordapp.com";
|
||||
const isRenderOrPreview =
|
||||
host === "media.discordapp.net" ||
|
||||
/^images-ext-\d+\.discordapp\.net$/.test(host);
|
||||
if (!isAttachmentCdn && !isRenderOrPreview) return imageUrl;
|
||||
if (!parsed.search) return imageUrl;
|
||||
// Path IS the stable identity of the attachment; everything after "?" is
|
||||
// signing or a render variant.
|
||||
return `${parsed.origin}${parsed.pathname}`;
|
||||
} catch {
|
||||
return imageUrl;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Lookup a cached media analysis result.
|
||||
* Returns the full cached text (the analysis summary string) or null if not found or expired.
|
||||
@@ -260,11 +294,31 @@ export async function invalidateTextModerationCache(
|
||||
}
|
||||
|
||||
/**
|
||||
* Lookup a cached moderation result for a text content.
|
||||
* Returns the stored result fields or null.
|
||||
* Normalize a stored verdict status to the full three-state union.
|
||||
*
|
||||
* Bug history (2026-08-22): both cache readers narrowed their types to
|
||||
* "clean" | "flagged", so a stored "warn" verdict fell into the legacy
|
||||
* `flags.length === 0 ? clean : flagged` branch and was served back as
|
||||
* FLAGGED (breaking auto-delete gating + dashboard labels). New entries
|
||||
* store the exact status; legacy rows without one derive from flags.
|
||||
*/
|
||||
export async function getCachedTextModeration(cacheKey: string): Promise<{
|
||||
status: "clean" | "flagged";
|
||||
export function normalizeStoredStatus(
|
||||
storedStatus: string | undefined,
|
||||
flags: string[],
|
||||
): "clean" | "warn" | "flagged" {
|
||||
if (
|
||||
storedStatus === "clean" ||
|
||||
storedStatus === "warn" ||
|
||||
storedStatus === "flagged"
|
||||
) {
|
||||
return storedStatus;
|
||||
}
|
||||
return flags.length === 0 ? "clean" : "flagged";
|
||||
}
|
||||
|
||||
/** Shape shared by every moderation-cache read path. */
|
||||
export interface StoredModerationVerdict {
|
||||
status: "clean" | "warn" | "flagged";
|
||||
flags: string[];
|
||||
score: number;
|
||||
analysis: string;
|
||||
@@ -272,28 +326,35 @@ export async function getCachedTextModeration(cacheKey: string): Promise<{
|
||||
severity: string;
|
||||
confidence: number;
|
||||
recommendedAction: string;
|
||||
} | null> {
|
||||
}
|
||||
|
||||
/** Raw DB row shape needed to rebuild a StoredModerationVerdict. */
|
||||
interface VerdictRow {
|
||||
flags: string;
|
||||
analyzed_at?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse one `text_analysis_cache` row into a StoredModerationVerdict.
|
||||
* Shared by the single-key and batched getters so their semantics can never
|
||||
* drift apart (status normalization lives in exactly one place).
|
||||
*/
|
||||
export function parseStoredVerdictRow(
|
||||
row: VerdictRow,
|
||||
): StoredModerationVerdict | null {
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
const row = await executeGet(
|
||||
`SELECT flags, source, analyzed_at, expires_at, hit_count
|
||||
FROM text_analysis_cache
|
||||
WHERE text = $1 AND expires_at > $2`,
|
||||
[cacheKey, Date.now()],
|
||||
parsed = JSON.parse(row.flags) as Record<string, unknown>;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object") return null;
|
||||
|
||||
const flags = Array.isArray(parsed.flags) ? (parsed.flags as string[]) : [];
|
||||
const status = normalizeStoredStatus(
|
||||
parsed.status as string | undefined,
|
||||
flags,
|
||||
);
|
||||
|
||||
if (!row) return null;
|
||||
|
||||
const parsed = JSON.parse(row.flags) as Record<string, unknown>;
|
||||
const flags = (parsed.flags as string[]) ?? [];
|
||||
// Use stored status if available (new entries), otherwise derive from flags (legacy compatibility)
|
||||
const storedStatus = parsed.status as string | undefined;
|
||||
const status: "clean" | "flagged" =
|
||||
storedStatus === "clean" || storedStatus === "flagged"
|
||||
? storedStatus
|
||||
: flags.length === 0
|
||||
? "clean"
|
||||
: "flagged";
|
||||
|
||||
return {
|
||||
status,
|
||||
flags,
|
||||
@@ -304,6 +365,52 @@ export async function getCachedTextModeration(cacheKey: string): Promise<{
|
||||
confidence: (parsed.confidence as number) ?? 0,
|
||||
recommendedAction: (parsed.recommendedAction as string) ?? "none",
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Increment the hit counter for a cache key (fire-and-forget).
|
||||
*
|
||||
* Bug history: `hit_count` was written as 0 on insert and never updated by
|
||||
* any reader, so cache effectiveness was unmeasurable. This is best-effort
|
||||
* observability — a failed bump must never affect the read path.
|
||||
*/
|
||||
function bumpHitCount(cacheKey: string): void {
|
||||
executeAll(
|
||||
`UPDATE text_analysis_cache SET hit_count = hit_count + 1 WHERE text = $1`,
|
||||
[cacheKey],
|
||||
).catch(() => {});
|
||||
}
|
||||
|
||||
/** Error-artifact flags that make a cached verdict unusable. */
|
||||
export const ERROR_ARTIFACT_FLAGS = [
|
||||
"analysis_api_failed",
|
||||
"analysis_parse_failed",
|
||||
"analysis_incomplete",
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Lookup a cached moderation result for a text content.
|
||||
* Returns the stored result fields or null.
|
||||
*/
|
||||
|
||||
export async function getCachedTextModeration(
|
||||
cacheKey: string,
|
||||
): Promise<StoredModerationVerdict | null> {
|
||||
try {
|
||||
const row = await executeGet(
|
||||
`SELECT flags, source, analyzed_at, expires_at, hit_count
|
||||
FROM text_analysis_cache
|
||||
WHERE text = $1 AND expires_at > $2`,
|
||||
[cacheKey, Date.now()],
|
||||
);
|
||||
|
||||
if (!row) return null;
|
||||
|
||||
const verdict = parseStoredVerdictRow(row);
|
||||
if (!verdict) return null;
|
||||
|
||||
bumpHitCount(cacheKey);
|
||||
return verdict;
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
{ error: error instanceof Error ? error.message : String(error) },
|
||||
@@ -313,6 +420,128 @@ export async function getCachedTextModeration(cacheKey: string): Promise<{
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Batched exact-hash lookup: ONE query for N keys.
|
||||
*
|
||||
* Semantics are identical to calling `getCachedTextModeration` per key
|
||||
* (unexpired rows only, shared row parser). Per-key hit-count bumps are NOT
|
||||
* issued here — the orchestrator logs an aggregate "cache applied" line
|
||||
* instead, keeping a 60-message burst at exactly one round-trip.
|
||||
* `analyzedAt` is surfaced so callers can apply freshness guards.
|
||||
*/
|
||||
export interface BatchedVerdictEntry {
|
||||
verdict: StoredModerationVerdict;
|
||||
analyzedAt: number | null;
|
||||
}
|
||||
|
||||
export async function getCachedTextModerations(
|
||||
cacheKeys: string[],
|
||||
): Promise<Map<string, BatchedVerdictEntry>> {
|
||||
const results = new Map<string, BatchedVerdictEntry>();
|
||||
const uniqueKeys = Array.from(new Set(cacheKeys)).filter(Boolean);
|
||||
if (uniqueKeys.length === 0) return results;
|
||||
|
||||
const CHUNK_SIZE = 200;
|
||||
try {
|
||||
for (let i = 0; i < uniqueKeys.length; i += CHUNK_SIZE) {
|
||||
const chunk = uniqueKeys.slice(i, i + CHUNK_SIZE);
|
||||
// Postgres has a 32k bind-parameter ceiling; ANY($1) keeps it at one
|
||||
// array param per chunk regardless of chunk length.
|
||||
const rows = await executeAll(
|
||||
`SELECT text, flags, analyzed_at
|
||||
FROM text_analysis_cache
|
||||
WHERE text = ANY($1::text[]) AND expires_at > $2`,
|
||||
[chunk, Date.now()],
|
||||
);
|
||||
for (const row of rows ?? []) {
|
||||
if (results.has(row.text)) continue;
|
||||
const verdict = parseStoredVerdictRow(row);
|
||||
if (!verdict) continue;
|
||||
results.set(row.text, {
|
||||
verdict,
|
||||
analyzedAt:
|
||||
typeof row.analyzed_at === "number" ? row.analyzed_at : null,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
logger.error(
|
||||
{ error: error instanceof Error ? error.message : String(error) },
|
||||
"Failed batched text moderation lookup",
|
||||
);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Fire-and-forget bulk hit-count bump for keys actually served as hits.
|
||||
* Companion to the batched getter (which skips per-row bumps): one UPDATE
|
||||
* per analysis batch keeps hit-rate metrics working at zero extra latency
|
||||
* cost per message.
|
||||
*/
|
||||
export function bumpTextModerationHitCounts(cacheKeys: string[]): void {
|
||||
const uniqueKeys = Array.from(new Set(cacheKeys)).filter(Boolean);
|
||||
if (uniqueKeys.length === 0) return;
|
||||
executeAll(
|
||||
`UPDATE text_analysis_cache SET hit_count = hit_count + 1 WHERE text = ANY($1::text[])`,
|
||||
[uniqueKeys],
|
||||
).catch(() => {});
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Semantic two-band acceptance
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* True when a semantic-cache hit may be reused given its verdict class.
|
||||
* Two bands (2026-08-24): non-actionable verdicts (clean / flagless /
|
||||
* action=none) are accepted from the LOOSER clean band; actionable verdicts
|
||||
* (warn/flagged or any flags/action) keep the strict historical gate.
|
||||
* Between the bands → reject → the message falls through to the LLM
|
||||
* (fail-open toward accuracy).
|
||||
*/
|
||||
export function isSemanticBandAccepted(
|
||||
verdict: StoredModerationVerdict,
|
||||
similarity: number,
|
||||
): boolean {
|
||||
const isNonActionable =
|
||||
verdict.status === "clean" &&
|
||||
verdict.flags.length === 0 &&
|
||||
(verdict.recommendedAction ?? "none") === "none";
|
||||
return isNonActionable
|
||||
? similarity >= config.AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN
|
||||
: similarity >= config.AI_LLM_EMBEDDING_MIN_SIMILARITY;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Global exact-cache reuse guard (context-free fallback)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* True when a stored verdict is safe to reuse OUTSIDE its original channel:
|
||||
* only verdicts that cannot trigger an action and carry no flags qualify,
|
||||
* and they must be confident + fresh. Flagged/warn verdicts are NEVER
|
||||
* globally reused — enforcement is context-sensitive by design.
|
||||
*/
|
||||
export function isGloballyReusableCleanVerdict(
|
||||
verdict: Omit<StoredModerationVerdict, "status"> & { status: string },
|
||||
analyzedAtMs: number | undefined,
|
||||
): boolean {
|
||||
if (verdict.status !== "clean") return false;
|
||||
if (verdict.flags.length > 0) return false;
|
||||
if ((verdict.recommendedAction ?? "none") !== "none") return false;
|
||||
if (!(verdict.confidence >= config.AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE))
|
||||
return false;
|
||||
if (
|
||||
typeof analyzedAtMs === "number" &&
|
||||
Date.now() - analyzedAtMs >
|
||||
config.AI_CACHE_GLOBAL_REUSE_MAX_AGE_H * 60 * 60 * 1000
|
||||
) {
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a Qdrant verdict payload into the result shape shared by the
|
||||
* semantic cache lookups. Returns null on malformed payloads (callers then
|
||||
@@ -321,43 +550,19 @@ export async function getCachedTextModeration(cacheKey: string): Promise<{
|
||||
export function parseQdrantVerdict(
|
||||
payload: QdrantVerdictPayload,
|
||||
similarity: number,
|
||||
): {
|
||||
):
|
||||
| (StoredModerationVerdict & {
|
||||
text: string;
|
||||
similarity: number;
|
||||
status: "clean" | "warn" | "flagged";
|
||||
flags: string[];
|
||||
score: number;
|
||||
analysis: string;
|
||||
categories: string[];
|
||||
severity: string;
|
||||
confidence: number;
|
||||
recommendedAction: string;
|
||||
} | null {
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(payload.flags) as Record<string, unknown>;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (!parsed || typeof parsed !== "object") return null;
|
||||
|
||||
const storedStatus = (parsed.status as string) ?? "clean";
|
||||
const status: "clean" | "warn" | "flagged" =
|
||||
storedStatus === "warn" || storedStatus === "flagged"
|
||||
? storedStatus
|
||||
: "clean";
|
||||
})
|
||||
| null {
|
||||
const parsed = parseStoredVerdictRow({ flags: payload.flags });
|
||||
if (!parsed) return null;
|
||||
|
||||
return {
|
||||
...parsed,
|
||||
text: payload.text,
|
||||
similarity,
|
||||
status,
|
||||
flags: (parsed.flags as string[]) ?? [],
|
||||
score: (parsed.score as number) ?? 0,
|
||||
analysis: (parsed.analysis as string) ?? "",
|
||||
categories: (parsed.categories as string[]) ?? [],
|
||||
severity: (parsed.severity as string) ?? "none",
|
||||
confidence: (parsed.confidence as number) ?? 0,
|
||||
recommendedAction: (parsed.recommendedAction as string) ?? "none",
|
||||
};
|
||||
}
|
||||
|
||||
@@ -373,18 +578,9 @@ export async function findSimilarTextModeration(
|
||||
embedding: number[],
|
||||
minSimilarity: number,
|
||||
limit: number,
|
||||
): Promise<{
|
||||
text: string;
|
||||
similarity: number;
|
||||
status: "clean" | "warn" | "flagged";
|
||||
flags: string[];
|
||||
score: number;
|
||||
analysis: string;
|
||||
categories: string[];
|
||||
severity: string;
|
||||
confidence: number;
|
||||
recommendedAction: string;
|
||||
} | null> {
|
||||
): Promise<
|
||||
(StoredModerationVerdict & { text: string; similarity: number }) | null
|
||||
> {
|
||||
// Qdrant path (primary)
|
||||
if (isQdrantConfigured()) {
|
||||
const hits = await searchQdrant(embedding, limit, minSimilarity);
|
||||
@@ -443,11 +639,10 @@ export async function findSimilarTextModeration(
|
||||
const hit = candidates[match.index];
|
||||
const parsed = hit.parsed;
|
||||
const flags = (parsed.flags as string[]) ?? [];
|
||||
const storedStatus = (parsed.status as string) ?? "clean";
|
||||
const status: "clean" | "warn" | "flagged" =
|
||||
storedStatus === "warn" || storedStatus === "flagged"
|
||||
? storedStatus
|
||||
: "clean";
|
||||
const status = normalizeStoredStatus(
|
||||
parsed.status as string | undefined,
|
||||
flags,
|
||||
);
|
||||
return {
|
||||
text: hit.text,
|
||||
similarity: match.similarity,
|
||||
@@ -469,6 +664,55 @@ export async function findSimilarTextModeration(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Upsert a bare (context-free) clean verdict to the Qdrant vector store,
|
||||
* making global-reuse clean verdicts discoverable by semantic search.
|
||||
*
|
||||
* Why: the main `setCachedTextModeration` writes bare-key rows to Postgres
|
||||
* with embedding=null (deliberate — no duplicate PG embedding column), but
|
||||
* a bare clean verdict that never reaches Qdrant is invisible to
|
||||
* searchQdrantBatch. So two messages with identical clean content in
|
||||
* DIFFERENT channels never match semantically — the semantic hit-rate is
|
||||
* capped below the exact-cache hit-rate. This helper shares the embedding
|
||||
* already computed at lookup time so the bare point is semantically
|
||||
* findable.
|
||||
*
|
||||
* Guard: only non-actionable clean verdicts qualify (same guard as the
|
||||
* read path and as the orchestrator's bare-key write-back). No-op when
|
||||
* Qdrant is disabled or no embedding is available.
|
||||
*/
|
||||
export async function upsertBareKeyToQdrant(
|
||||
bareKey: string,
|
||||
result: {
|
||||
status: string;
|
||||
flags: string[];
|
||||
score: number;
|
||||
analysis: string;
|
||||
categories: string[];
|
||||
severity: string;
|
||||
confidence: number;
|
||||
recommendedAction: string;
|
||||
},
|
||||
embedding: number[] | null | undefined,
|
||||
): Promise<void> {
|
||||
if (!isQdrantConfigured() || !embedding || embedding.length === 0) return;
|
||||
if (!isGloballyReusableCleanVerdict(result, undefined)) return;
|
||||
const now = Date.now();
|
||||
const USER_MOD_CACHE_TTL_MS = 24 * 60 * 60 * 1000;
|
||||
await upsertQdrantPoint(bareKey, embedding, {
|
||||
text: bareKey,
|
||||
flags: JSON.stringify(result),
|
||||
analyzed_at: now,
|
||||
expires_at: now + USER_MOD_CACHE_TTL_MS,
|
||||
content_hash: bareKey.split(":").pop() ?? "",
|
||||
}).catch((err: unknown) => {
|
||||
logger.error(
|
||||
{ error: err instanceof Error ? err.message : String(err), bareKey },
|
||||
"Failed to upsert bare-key clean verdict to Qdrant",
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Store a moderation result for a (user, content) pair.
|
||||
* The `flags` field stores the full result object as JSON.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { resolve } from "node:dns/promises";
|
||||
import { isIP } from "node:net";
|
||||
import { LRUCache } from "lru-cache";
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { createAbortControllerWithTimeout } from "@/shared/utils/index";
|
||||
|
||||
@@ -150,6 +151,47 @@ function truncateAndCleanHtml(html: string, maxLen = 1000): string {
|
||||
export async function fetchUrlSafely(
|
||||
url: string,
|
||||
depth = 0,
|
||||
): Promise<FetchedUrlContext> {
|
||||
// Text results are memoized (in-process, short TTL): the same link recurs
|
||||
// across batches and re-downloading + re-parsing the page each time was
|
||||
// pure latency. Images are NEVER cached here — they are vision evidence
|
||||
// and multi-MB buffers don't belong in an LRU. Errors are not cached so a
|
||||
// transient network blip retries on the next batch.
|
||||
if (depth === 0) {
|
||||
const memo = textFetchMemo.get(url);
|
||||
if (memo) return memo;
|
||||
// In-flight dedupe: concurrent callers share one live request.
|
||||
const existing = textInFlight.get(url);
|
||||
if (existing) return existing;
|
||||
const promise = fetchUrlSafelyUncached(url, depth)
|
||||
.then((fetched) => {
|
||||
if (fetched.type === "text") textFetchMemo.set(url, fetched);
|
||||
return fetched;
|
||||
})
|
||||
.finally(() => {
|
||||
textInFlight.delete(url);
|
||||
});
|
||||
textInFlight.set(url, promise);
|
||||
return promise;
|
||||
}
|
||||
return fetchUrlSafelyUncached(url, depth);
|
||||
}
|
||||
|
||||
/** In-process memo of successful TEXT fetches (30 min TTL, bounded size). */
|
||||
const textFetchMemo = new LRUCache<string, FetchedUrlContext>({
|
||||
max: 500,
|
||||
ttl: 30 * 60 * 1000,
|
||||
});
|
||||
|
||||
/** Concurrent same-URL text fetches collapse into one live request. */
|
||||
const textInFlight = new LRUCache<string, Promise<FetchedUrlContext>>({
|
||||
max: 100,
|
||||
ttl: 60_000,
|
||||
});
|
||||
|
||||
async function fetchUrlSafelyUncached(
|
||||
url: string,
|
||||
depth = 0,
|
||||
): Promise<FetchedUrlContext> {
|
||||
if (depth > 1) {
|
||||
return { url, type: "error", error: "Max redirect/meta depth reached" };
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
import { createChildLogger } from "@/shared/logger/index";
|
||||
import { createAbortControllerWithTimeout } from "@/shared/utils/index";
|
||||
import { config } from "../../shared/config/config.js";
|
||||
import { cacheGet, cacheSet, makeCacheKey } from "./cacheStore.js";
|
||||
|
||||
const log = createChildLogger("wikipedia-client");
|
||||
|
||||
@@ -57,10 +58,19 @@ function stripHtml(snippet: string): string {
|
||||
.trim();
|
||||
}
|
||||
|
||||
/** Redis TTL for cached search results (6h — articles change slowly). */
|
||||
const SEARCH_CACHE_TTL_SECONDS = 6 * 60 * 60;
|
||||
|
||||
/**
|
||||
* Search Wikipedia for a query and return up to MAX_RESULTS structured hits.
|
||||
* Uses the Action API `list=search` (srsearch) which is stable and returns
|
||||
* title + HTML snippet. Graceful: returns [] on any failure.
|
||||
*
|
||||
* Cached in the shared Redis store: the same query recurs across batches
|
||||
* (repeat slang, recurring topics), and an uncached re-search per batch was
|
||||
* pure latency + Wikipedia rate-limit pressure. Only NON-EMPTY results are
|
||||
* cached — an empty result may be a transient limiter/network blip, so it is
|
||||
* retried on a later batch instead of being pinned for 6 hours.
|
||||
*/
|
||||
export async function wikipediaSearch(
|
||||
query: string,
|
||||
@@ -69,6 +79,32 @@ export async function wikipediaSearch(
|
||||
const q = query.trim();
|
||||
if (!q) return [];
|
||||
|
||||
const cacheKey = makeCacheKey("wikisearch", q);
|
||||
const cached = await cacheGet(cacheKey);
|
||||
if (cached) {
|
||||
try {
|
||||
const parsed = JSON.parse(cached) as SearchResult[];
|
||||
if (Array.isArray(parsed) && parsed.length > 0) {
|
||||
log.debug({ query: q }, "Wikipedia search cache HIT");
|
||||
return parsed;
|
||||
}
|
||||
} catch {
|
||||
// Malformed entry — fall through to live fetch.
|
||||
}
|
||||
}
|
||||
|
||||
const mapped = await wikipediaSearchLive(q, timeoutMs);
|
||||
if (mapped.length > 0) {
|
||||
cacheSet(cacheKey, JSON.stringify(mapped), SEARCH_CACHE_TTL_SECONDS);
|
||||
}
|
||||
return mapped;
|
||||
}
|
||||
|
||||
/** Live (uncached) Action API search. Returns [] on any failure. */
|
||||
async function wikipediaSearchLive(
|
||||
q: string,
|
||||
timeoutMs: number,
|
||||
): Promise<SearchResult[]> {
|
||||
const params = new URLSearchParams({
|
||||
action: "query",
|
||||
list: "search",
|
||||
|
||||
@@ -246,6 +246,14 @@ export class MessagesAnalysis {
|
||||
.returning();
|
||||
});
|
||||
|
||||
// UPDATE..RETURNING has no guaranteed row order (Postgres returns rows
|
||||
// in physical update order). Consumers rely on chronological order:
|
||||
// batchScheduler/pickBatchWithinBudget treat the array as a created_at
|
||||
// ASC prefix, and the context anchor uses messages[0].created_at.
|
||||
rows.sort(
|
||||
(a, b) =>
|
||||
(a as MessageRecord).created_at - (b as MessageRecord).created_at,
|
||||
);
|
||||
return rows as MessageRecord[];
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
@@ -386,6 +394,13 @@ export class MessagesAnalysis {
|
||||
.returning();
|
||||
});
|
||||
|
||||
// Same UPDATE..RETURNING ordering guarantee as above: re-sort to
|
||||
// created_at ASC so the individual fallback path also sees a
|
||||
// chronological array.
|
||||
rows.sort(
|
||||
(a, b) =>
|
||||
(a as MessageRecord).created_at - (b as MessageRecord).created_at,
|
||||
);
|
||||
return rows as MessageRecord[];
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
|
||||
@@ -48,7 +48,13 @@ export class MessagesCleanup {
|
||||
}
|
||||
|
||||
async revertStuckProcessingMessages(
|
||||
timeoutMs: number = 300000,
|
||||
// 2026-08-24: lowered from 300000 — this cleanup is the last-resort
|
||||
// recoverer for rows stuck in `processing`. With the upload-pending
|
||||
// race-guard now requeueing properly (fallbackResultClassifier), any row
|
||||
// that still sits here for >2min is a genuine leak; reverting sooner
|
||||
// bounds the worst-case delay without racing legitimate in-flight work
|
||||
// (media batches can legitimately take ~60s+).
|
||||
timeoutMs: number = 120000,
|
||||
): Promise<number> {
|
||||
this.logger.debug({ timeoutMs }, "revertStuckProcessingMessages entry");
|
||||
try {
|
||||
|
||||
@@ -31,7 +31,7 @@ export interface ResolveOptions {
|
||||
timeout?: number;
|
||||
/**
|
||||
* yt-dlp format string override (e.g. "bestaudio[ext=m4a]").
|
||||
* Defaults to "bestaudio".
|
||||
* Defaults to "bestaudio[ext=m4a]/bestaudio/best".
|
||||
*/
|
||||
quality?: string;
|
||||
}
|
||||
@@ -319,7 +319,11 @@ export function resolveMediaUrl(
|
||||
options?: ResolveOptions,
|
||||
): Promise<MediaSourceResolution> {
|
||||
return new Promise<MediaSourceResolution>((resolve, reject) => {
|
||||
const format = options?.quality ?? "bestaudio";
|
||||
// Fallback chain: YouTube gets progressive m4a audio; direct-file hosts
|
||||
// (upload mirror / Telegram files) expose a single generic format with an
|
||||
// arbitrary ID — `bestaudio` alone fails there ("Requested format is not
|
||||
// available"), so fall back to `bestaudio` then plain `best`.
|
||||
const format = options?.quality ?? "bestaudio[ext=m4a]/bestaudio/best";
|
||||
const cookieArgs = buildCookieArgs();
|
||||
const args = [
|
||||
"-f",
|
||||
|
||||
@@ -149,8 +149,8 @@ export const configSchema = z
|
||||
AI_LLM_BASE_URL: z
|
||||
.string()
|
||||
.url()
|
||||
.default("https://omniroute.imrnes.team/v1"),
|
||||
AI_LLM_MODEL: z.string().default("text"),
|
||||
.default("https://9router.asepharyana.my.id/v1"),
|
||||
AI_LLM_MODEL: z.string().default("claude-opus-5"),
|
||||
// Vision uses the SAME router/base URL as text moderation
|
||||
// (AI_LLM_BASE_URL) but a different model alias. The dedicated NVIDIA
|
||||
// multimodal endpoint was removed.
|
||||
@@ -168,11 +168,21 @@ export const configSchema = z
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.97),
|
||||
// Two-band semantic acceptance (2026-08-24): non-actionable verdicts
|
||||
// (clean, no flags, action=none) may be reused from a LOOSER similarity
|
||||
// band than actionable ones (warn/flagged). Actionable verdicts keep the
|
||||
// strict gate above; anything between the two bands falls through to the
|
||||
// LLM (fail-open toward accuracy).
|
||||
AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN: z.coerce
|
||||
.number()
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.92),
|
||||
AI_LLM_EMBEDDING_MAX_CANDIDATES: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
.positive()
|
||||
.default(30),
|
||||
.default(50),
|
||||
// Qdrant vector store for the semantic moderation cache. When
|
||||
// QDRANT_URL is set, embeddings are stored/searched there (Postgres
|
||||
// embedding column remains as a legacy fallback).
|
||||
@@ -244,9 +254,29 @@ export const configSchema = z
|
||||
.positive()
|
||||
.default(10000),
|
||||
AI_ANALYSIS_ERROR_COOLDOWN_MS: z.coerce.number().positive().default(30000),
|
||||
// Upload-pending batch poll (2026-08-25): when a batch is deferred because
|
||||
// attachments are still uploading, the processor re-schedules with this
|
||||
// base delay (linear ramp per consecutive poll, capped) instead of the
|
||||
// 250ms debounce — the old path hot-looped ~300ms for the whole upload.
|
||||
AI_ANALYSIS_UPLOAD_POLL_MS: z.coerce.number().positive().default(1500),
|
||||
AI_ANALYSIS_MAX_UPLOAD_POLL_MS: z.coerce.number().positive().default(8000),
|
||||
|
||||
// ── AI Analysis Batch ───────────────────────────────────────────────
|
||||
AI_ANALYSIS_MAX_BATCH_SIZE: z.coerce.number().int().positive().default(200),
|
||||
// Global exact-cache reuse guard (2026-08-24): a context-scoped miss may
|
||||
// fall back to the legacy bare (context-free) key, but ONLY for verdicts
|
||||
// that cannot trigger an action and are fresh + confident. These knobs
|
||||
// bound that reuse.
|
||||
AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE: z.coerce
|
||||
.number()
|
||||
.min(0)
|
||||
.max(1)
|
||||
.default(0.85),
|
||||
AI_CACHE_GLOBAL_REUSE_MAX_AGE_H: z.coerce
|
||||
.number()
|
||||
.int()
|
||||
.positive()
|
||||
.default(120),
|
||||
AI_ANALYSIS_MAX_CONTEXT_TOKENS: z.coerce.number().positive().default(8000),
|
||||
AI_ANALYSIS_MAX_TARGET_TOKENS: z.coerce.number().positive().default(14000),
|
||||
AI_ANALYSIS_CONTEXT_MESSAGE_LIMIT: z.coerce
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
pickBatchWithinBudget,
|
||||
type TokenEstimator,
|
||||
} from "../src/modules/ai-moderation/batchBudget.js";
|
||||
import type { MessageRecord } from "../src/modules/message-capture/types.js";
|
||||
|
||||
// Deterministic estimator: 1 token per character. Keeps the budget math
|
||||
// exact regardless of tiktoken behavior (the real estimator is injected at
|
||||
// the call site — see batchProcessor.ts).
|
||||
const estimate: TokenEstimator = (text: string) => text.length;
|
||||
|
||||
function msg(id: string, content: string, createdAt: number): MessageRecord {
|
||||
return {
|
||||
id,
|
||||
guild_id: "g",
|
||||
channel_id: "c",
|
||||
thread_id: null,
|
||||
user_id: "u",
|
||||
username: "user",
|
||||
avatar_url: null,
|
||||
content,
|
||||
edited_content: null,
|
||||
created_at: createdAt,
|
||||
edited_at: null,
|
||||
deleted_at: null,
|
||||
type: "text",
|
||||
is_reply: false,
|
||||
is_forward: false,
|
||||
is_crosspost: false,
|
||||
reference_message_id: null,
|
||||
reference_channel_id: null,
|
||||
reference_guild_id: null,
|
||||
metadata: null,
|
||||
};
|
||||
}
|
||||
|
||||
describe("pickBatchWithinBudget", () => {
|
||||
const TOKENS_PER_MESSAGE = 50;
|
||||
|
||||
it("returns a contiguous chronological prefix — no gaps mid-timeline", () => {
|
||||
// sizes: 100, 100, 400 (overflow), 10
|
||||
const messages = [
|
||||
msg("m1", "a".repeat(100), 1),
|
||||
msg("m2", "b".repeat(100), 2),
|
||||
msg("m3", "c".repeat(400), 3),
|
||||
msg("m4", "d".repeat(10), 4),
|
||||
];
|
||||
const batch = pickBatchWithinBudget(
|
||||
messages,
|
||||
500,
|
||||
TOKENS_PER_MESSAGE,
|
||||
estimate,
|
||||
);
|
||||
|
||||
// m1(150)+m2(150)=300 fits; m3 would be 550 > 500 → stop.
|
||||
// m4 must NOT be picked even though it alone fits (no timeline gap).
|
||||
expect(batch.map((m) => m.id)).toEqual(["m1", "m2"]);
|
||||
});
|
||||
|
||||
it("includes a message that exactly hits the budget", () => {
|
||||
const messages = [msg("m1", "a".repeat(450), 1)];
|
||||
const batch = pickBatchWithinBudget(
|
||||
messages,
|
||||
500,
|
||||
TOKENS_PER_MESSAGE,
|
||||
estimate,
|
||||
);
|
||||
expect(batch.map((m) => m.id)).toEqual(["m1"]);
|
||||
});
|
||||
|
||||
it("returns empty when the first message alone exceeds the budget", () => {
|
||||
const messages = [
|
||||
msg("big", "x".repeat(1000), 1),
|
||||
msg("m2", "y".repeat(10), 2),
|
||||
];
|
||||
const batch = pickBatchWithinBudget(
|
||||
messages,
|
||||
500,
|
||||
TOKENS_PER_MESSAGE,
|
||||
estimate,
|
||||
);
|
||||
expect(batch).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("handles empty input", () => {
|
||||
expect(
|
||||
pickBatchWithinBudget([], 500, TOKENS_PER_MESSAGE, estimate),
|
||||
).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,110 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// partitionBatchOutcome — upload-pending defer vs fanout (2026-08-25)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Bug history: the batch worker's race guard returned {ok:true, rows:[]} while
|
||||
// attachments were still uploading; every target was classified "incomplete",
|
||||
// fanned out to the individual queue, requeued there, rescheduled at 250ms —
|
||||
// a hot ~300ms loop for the whole upload duration (~10 cycles in 3s in prod).
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
computeUploadPollDelayMs,
|
||||
partitionBatchOutcome,
|
||||
} from "../src/modules/ai-moderation/batchOutcomeClassifier.js";
|
||||
|
||||
const msgs = (...ids: string[]) => ids.map((id) => ({ id }));
|
||||
|
||||
describe("partitionBatchOutcome", () => {
|
||||
it("marks ALL targets upload_pending when the full-batch guard fires", () => {
|
||||
const out = partitionBatchOutcome(msgs("a", "b"), {
|
||||
ok: true,
|
||||
rows: [],
|
||||
uploadPendingIds: ["a", "b"],
|
||||
});
|
||||
expect(out.get("a")).toBe("upload_pending");
|
||||
expect(out.get("b")).toBe("upload_pending");
|
||||
});
|
||||
|
||||
it("never classifies an explicit upload_pending id as incomplete", () => {
|
||||
// The regression this file exists for: uploadPendingIds must win over the
|
||||
// missing-row heuristic.
|
||||
const out = partitionBatchOutcome(msgs("a"), {
|
||||
ok: true,
|
||||
rows: [],
|
||||
uploadPendingIds: ["a"],
|
||||
});
|
||||
expect(out.get("a")).not.toBe("incomplete");
|
||||
expect(out.get("a")).toBe("upload_pending");
|
||||
});
|
||||
|
||||
it("partitions a mixed batch: completed + upload-pending + missing", () => {
|
||||
const out = partitionBatchOutcome(msgs("ok1", "up1", "gone1"), {
|
||||
ok: true,
|
||||
rows: [
|
||||
{ id: "ok1", ai_status: "clean" },
|
||||
// up1 has NO row but IS in uploadPendingIds -> deferred, not failed
|
||||
],
|
||||
uploadPendingIds: ["up1"],
|
||||
});
|
||||
expect(out.get("ok1")).toBe("completed");
|
||||
expect(out.get("up1")).toBe("upload_pending");
|
||||
expect(out.get("gone1")).toBe("incomplete"); // unexplained drop stays retryable
|
||||
});
|
||||
|
||||
it("routes flag-based failures to their buckets", () => {
|
||||
const out = partitionBatchOutcome(msgs("i", "p", "f"), {
|
||||
ok: true,
|
||||
rows: [
|
||||
{
|
||||
id: "i",
|
||||
ai_status: "error",
|
||||
ai_moderation_flags: JSON.stringify(["analysis_incomplete"]),
|
||||
},
|
||||
{
|
||||
id: "p",
|
||||
ai_status: "error",
|
||||
ai_moderation_flags: JSON.stringify(["analysis_parse_failed"]),
|
||||
},
|
||||
{
|
||||
id: "f",
|
||||
ai_status: "error",
|
||||
ai_moderation_flags: JSON.stringify(["analysis_api_failed"]),
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(out.get("i")).toBe("incomplete");
|
||||
expect(out.get("p")).toBe("parse_failed");
|
||||
expect(out.get("f")).toBe("api_failed");
|
||||
});
|
||||
|
||||
it("treats error rows with no known flag as retryable incomplete", () => {
|
||||
const out = partitionBatchOutcome(msgs("x"), {
|
||||
ok: true,
|
||||
rows: [
|
||||
{ id: "x", ai_status: "error", ai_moderation_flags: '["weird_flag"]' },
|
||||
],
|
||||
});
|
||||
expect(out.get("x")).toBe("incomplete");
|
||||
});
|
||||
|
||||
it("accepts DB-round-trip MessageRecord shape (null ai_status)", () => {
|
||||
const out = partitionBatchOutcome([{ id: "r", ai_status: null }] as never, {
|
||||
ok: true,
|
||||
rows: [{ id: "r", ai_status: null }],
|
||||
});
|
||||
expect(out.get("r")).toBe("completed");
|
||||
});
|
||||
});
|
||||
|
||||
describe("computeUploadPollDelayMs", () => {
|
||||
it("ramps linearly and respects the cap", () => {
|
||||
expect(computeUploadPollDelayMs(1, 1500, 8000)).toBe(1500);
|
||||
expect(computeUploadPollDelayMs(2, 1500, 8000)).toBe(3000);
|
||||
expect(computeUploadPollDelayMs(3, 1500, 8000)).toBe(4500);
|
||||
expect(computeUploadPollDelayMs(9, 1500, 8000)).toBe(8000); // capped
|
||||
});
|
||||
|
||||
it("is safe on degenerate input", () => {
|
||||
expect(computeUploadPollDelayMs(0, 1500, 8000)).toBe(1500);
|
||||
expect(computeUploadPollDelayMs(-5, 1000, 4000)).toBe(1000);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,127 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Batched exact-cache lookup (getCachedTextModerations)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): phase-1 cache lookups collapse N sequential per-key
|
||||
// queries into ONE `text = ANY($1::text[])` query. Semantics must match the
|
||||
// single-key getter: unexpired rows only, malformed rows skipped, verdicts
|
||||
// normalized through the shared parser. The DB layer is mocked — no live
|
||||
// Postgres in unit tests.
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const executeAll = vi.fn();
|
||||
const executeGet = vi.fn();
|
||||
|
||||
vi.mock("../src/shared/database/drizzle.js", () => ({
|
||||
executeAll: (...args: unknown[]) => executeAll(...args),
|
||||
executeGet: (...args: unknown[]) => executeGet(...args),
|
||||
}));
|
||||
|
||||
import {
|
||||
getCachedTextModerations,
|
||||
parseStoredVerdictRow,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
function rowFor(
|
||||
text: string,
|
||||
status: string,
|
||||
analyzedAt = Date.now() - 1000,
|
||||
): Record<string, unknown> {
|
||||
return {
|
||||
text,
|
||||
flags: JSON.stringify({
|
||||
status,
|
||||
flags: [],
|
||||
score: 0,
|
||||
analysis: "ok",
|
||||
categories: [],
|
||||
severity: "none",
|
||||
confidence: 0.9,
|
||||
recommendedAction: "none",
|
||||
}),
|
||||
source: "user_moderation",
|
||||
analyzed_at: analyzedAt,
|
||||
expires_at: Date.now() + 3_600_000,
|
||||
hit_count: 0,
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
executeAll.mockReset();
|
||||
executeGet.mockReset();
|
||||
});
|
||||
|
||||
describe("parseStoredVerdictRow", () => {
|
||||
it("parses a valid stored verdict", () => {
|
||||
const v = parseStoredVerdictRow({
|
||||
flags: JSON.stringify({ status: "warn", flags: ["x"] }),
|
||||
});
|
||||
expect(v).not.toBeNull();
|
||||
expect(v?.status).toBe("warn");
|
||||
expect(v?.flags).toEqual(["x"]);
|
||||
});
|
||||
|
||||
it("returns null on malformed JSON", () => {
|
||||
expect(parseStoredVerdictRow({ flags: "{not-json" })).toBeNull();
|
||||
});
|
||||
|
||||
it("derives legacy status from flags when status is absent", () => {
|
||||
const v = parseStoredVerdictRow({
|
||||
flags: JSON.stringify({ flags: ["a"] }),
|
||||
});
|
||||
expect(v?.status).toBe("flagged");
|
||||
const v2 = parseStoredVerdictRow({ flags: JSON.stringify({}) });
|
||||
expect(v2?.status).toBe("clean");
|
||||
});
|
||||
});
|
||||
|
||||
describe("getCachedTextModerations", () => {
|
||||
it("returns an empty map for empty input and issues no query", async () => {
|
||||
const result = await getCachedTextModerations([]);
|
||||
expect(result.size).toBe(0);
|
||||
expect(executeAll).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("dedupes keys and returns parsed verdicts keyed by cache key", async () => {
|
||||
executeAll.mockResolvedValueOnce([
|
||||
rowFor("text_mod:c1:aaa", "clean"),
|
||||
rowFor("text_mod:c1:bbb", "warn"),
|
||||
]);
|
||||
const result = await getCachedTextModerations([
|
||||
"text_mod:c1:aaa",
|
||||
"text_mod:c1:aaa",
|
||||
"text_mod:c1:bbb",
|
||||
]);
|
||||
expect(result.size).toBe(2);
|
||||
expect(result.get("text_mod:c1:aaa")?.verdict.status).toBe("clean");
|
||||
expect(result.get("text_mod:c1:bbb")?.verdict.status).toBe("warn");
|
||||
// Exactly ONE batched round-trip.
|
||||
expect(executeAll).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("skips rows with malformed payloads instead of failing the batch", async () => {
|
||||
executeAll.mockResolvedValueOnce([
|
||||
{
|
||||
text: "k_good",
|
||||
flags: JSON.stringify({ status: "clean", flags: [] }),
|
||||
analyzed_at: 1,
|
||||
},
|
||||
{ text: "k_bad", flags: "{broken" },
|
||||
]);
|
||||
const result = await getCachedTextModerations(["k_good", "k_bad"]);
|
||||
expect(result.has("k_good")).toBe(true);
|
||||
expect(result.has("k_bad")).toBe(false);
|
||||
});
|
||||
|
||||
it("survives a DB failure and returns an empty map (fail-open)", async () => {
|
||||
executeAll.mockRejectedValueOnce(new Error("connection refused"));
|
||||
const result = await getCachedTextModerations(["k1", "k2"]);
|
||||
expect(result.size).toBe(0);
|
||||
});
|
||||
|
||||
it("chunks queries beyond 200 keys", async () => {
|
||||
executeAll.mockResolvedValue([]);
|
||||
const keys = Array.from({ length: 450 }, (_, i) => `k${i}`);
|
||||
await getCachedTextModerations(keys);
|
||||
expect(executeAll).toHaveBeenCalledTimes(3); // 200 + 200 + 50
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,169 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Semantic two-band acceptance + global exact-cache reuse guard
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): cache hits may be served MORE aggressively for
|
||||
// verdicts that cannot trigger enforcement actions, and NEVER more
|
||||
// aggressively for actionable ones. Two layers enforce this:
|
||||
// - isSemanticBandAccepted: similarity thresholds differ by verdict class
|
||||
// (clean band 0.92 default vs strict actionable band 0.97 default).
|
||||
// - isGloballyReusableCleanVerdict: context-free (cross-channel) reuse of
|
||||
// the legacy bare key only for clean / flagless / action=none verdicts
|
||||
// with high confidence and bounded age.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
isGloballyReusableCleanVerdict,
|
||||
type StoredModerationVerdict,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
function makeVerdict(
|
||||
overrides: Partial<StoredModerationVerdict> = {},
|
||||
): StoredModerationVerdict {
|
||||
return {
|
||||
status: "clean",
|
||||
flags: [],
|
||||
score: 0,
|
||||
analysis: "",
|
||||
categories: [],
|
||||
severity: "none",
|
||||
confidence: 0.95,
|
||||
recommendedAction: "none",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe("isSemanticBandAccepted", () => {
|
||||
it("accepts a non-actionable clean verdict at the loose clean band", () => {
|
||||
// Default AI_LLM_EMBEDDING_MIN_SIMILARITY_CLEAN = 0.92.
|
||||
expect(isBandAccept(makeVerdict(), 0.93)).toBe(true);
|
||||
});
|
||||
|
||||
it("accepts a clean verdict exactly at the clean band boundary", () => {
|
||||
expect(isBandAccept(makeVerdict({ confidence: 0.99 }), 0.92)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a clean verdict below the clean band", () => {
|
||||
expect(isBandAccept(makeVerdict(), 0.91)).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects an actionable flagged verdict between the bands", () => {
|
||||
// 0.93 >= clean band BUT < strict band → must NOT be served.
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "flagged", flags: ["hate_speech"] }),
|
||||
0.93,
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts a flagged verdict at the strict band", () => {
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "flagged", flags: ["hate_speech"] }),
|
||||
0.98,
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects a warn verdict below the strict band", () => {
|
||||
expect(
|
||||
isBandAccept(
|
||||
makeVerdict({ status: "warn", recommendedAction: "warn" }),
|
||||
0.96,
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("treats a clean verdict WITH flags as actionable (strict band)", () => {
|
||||
expect(isBandAccept(makeVerdict({ flags: ["borderline"] }), 0.93)).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("treats a clean verdict with a non-none action as actionable", () => {
|
||||
expect(
|
||||
isBandAccept(makeVerdict({ recommendedAction: "review" }), 0.93),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// Import indirection so the describe block reads cleanly.
|
||||
import { isSemanticBandAccepted as isBandAccept } from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
describe("isGloballyReusableCleanVerdict", () => {
|
||||
it("accepts a fresh, confident, flagless clean verdict", () => {
|
||||
const v = makeVerdict({ confidence: 0.9 });
|
||||
expect(isGloballyReusableCleanVerdict(v, Date.now() - 60_000)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects flagged / warn verdicts outright", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ status: "flagged", flags: ["harassment"] }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ status: "warn", flags: ["mild"] }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects clean verdicts carrying flags", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(makeVerdict({ flags: ["x"] }), Date.now()),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects verdicts whose recommended action is not none", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ recommendedAction: "delete" }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects low-confidence verdicts below the guard threshold", () => {
|
||||
// Default AI_CACHE_GLOBAL_REUSE_MIN_CONFIDENCE = 0.85.
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ confidence: 0.6 }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts confidence exactly at the guard threshold", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
makeVerdict({ confidence: 0.85 }),
|
||||
Date.now(),
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects entries older than the freshness window", () => {
|
||||
// Default AI_CACHE_GLOBAL_REUSE_MAX_AGE_H = 120h.
|
||||
const tooOld = Date.now() - 121 * 60 * 60 * 1000;
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), tooOld)).toBe(false);
|
||||
const freshEnough = Date.now() - 119 * 60 * 60 * 1000;
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), freshEnough)).toBe(
|
||||
true,
|
||||
);
|
||||
});
|
||||
|
||||
it("skips the age check when analyzedAt is unknown", () => {
|
||||
expect(isGloballyReusableCleanVerdict(makeVerdict(), undefined)).toBe(true);
|
||||
});
|
||||
|
||||
it("rejects non-standard statuses such as processing write-backs", () => {
|
||||
expect(
|
||||
isGloballyReusableCleanVerdict(
|
||||
{ ...makeVerdict(), status: "processing" },
|
||||
undefined,
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1,11 +1,10 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Context enrichment builders — <user_history>, <user_profiles> as_of,
|
||||
// bot/edited detection (pure, no DB)
|
||||
// Context enrichment builders — bot/edited detection (pure, no DB)
|
||||
// (buildUserHistoryXml / buildUserProfilesBlock were removed with the
|
||||
// per-user context minimization; their tests went with them.)
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
buildUserHistoryXml,
|
||||
buildUserProfilesBlock,
|
||||
resolveIsBot,
|
||||
resolveIsEdited,
|
||||
} from "../src/modules/ai-moderation/moderationBuilders.js";
|
||||
@@ -39,85 +38,6 @@ function msg(overrides: Partial<MessageRecord> = {}): MessageRecord {
|
||||
};
|
||||
}
|
||||
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
describe("buildUserHistoryXml — last flagged messages for repeat offenders", () => {
|
||||
it("returns empty when there is no real history", () => {
|
||||
expect(buildUserHistoryXml([])).toBe("");
|
||||
expect(
|
||||
buildUserHistoryXml([{ content: " ", severity: "low", created_at: 1 }]),
|
||||
).toBe("");
|
||||
});
|
||||
|
||||
it("renders <infraction> rows with severity and recency", () => {
|
||||
const xml = buildUserHistoryXml(
|
||||
[
|
||||
{
|
||||
content: "beli barang murah disini https://scam.example",
|
||||
severity: "high",
|
||||
created_at: NOW - 3 * DAY_MS,
|
||||
},
|
||||
],
|
||||
NOW,
|
||||
);
|
||||
expect(xml).toContain("<user_history>");
|
||||
expect(xml).toContain('severity="high"');
|
||||
expect(xml).toContain('time_ago_days="3"');
|
||||
expect(xml).toContain("beli barang murah disini");
|
||||
});
|
||||
|
||||
it("caps long snippets and XML-escapes content", () => {
|
||||
const xml = buildUserHistoryXml(
|
||||
[
|
||||
{
|
||||
content: "x".repeat(300),
|
||||
severity: "low",
|
||||
created_at: NOW - DAY_MS,
|
||||
},
|
||||
],
|
||||
NOW,
|
||||
);
|
||||
expect(xml.length).toBeLessThan(250);
|
||||
});
|
||||
});
|
||||
|
||||
describe("buildUserProfilesBlock — deduplicated map with staleness", () => {
|
||||
it("emits as_of when the profile has a last-generated timestamp", () => {
|
||||
const block = buildUserProfilesBlock(
|
||||
new Map([
|
||||
[
|
||||
"u1",
|
||||
{
|
||||
text: "Developer teknis, bahasa Indonesia",
|
||||
asOf: NOW - 3 * DAY_MS,
|
||||
},
|
||||
],
|
||||
]),
|
||||
);
|
||||
expect(block).toContain('<user_profile user_id="u1"');
|
||||
expect(block).toContain(
|
||||
`as_of="${new Date(NOW - 3 * DAY_MS).toISOString()}"`,
|
||||
);
|
||||
expect(block).toContain("Developer teknis");
|
||||
});
|
||||
|
||||
it("omits as_of when absent, and drops empty profiles", () => {
|
||||
const block = buildUserProfilesBlock(
|
||||
new Map([
|
||||
["u1", { text: "profil aktif", asOf: null }],
|
||||
["u2", { text: " " }],
|
||||
]),
|
||||
);
|
||||
expect(block).toContain('user_id="u1"');
|
||||
expect(block).not.toContain("as_of");
|
||||
expect(block).not.toContain("u2");
|
||||
});
|
||||
|
||||
it("returns empty for no profiles", () => {
|
||||
expect(buildUserProfilesBlock(new Map())).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveIsBot / resolveIsEdited — message flags", () => {
|
||||
it("reads author.bot from captured metadata", () => {
|
||||
const bot = msg({
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// normalizeDiscordImageUrl — unified vision cache keys for Discord CDN URLs
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Design (2026-08-24): the same attachment reached through different signed
|
||||
// URLs (?ex=&is=&hm= tokens rotate per fetch) or render variants
|
||||
// (?format=&width=) must map to ONE vision-cache key, otherwise the vision
|
||||
// model re-downloads and re-analyzes the identical image once per variant.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
makeImageCacheKey,
|
||||
normalizeDiscordImageUrl,
|
||||
} from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
describe("normalizeDiscordImageUrl", () => {
|
||||
it("strips rotating signed tokens from cdn.discordapp.com URLs", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://cdn.discordapp.com/attachments/1/2/img.png?ex=67a&is=67b&hm=tokA",
|
||||
);
|
||||
const b = normalizeDiscordImageUrl(
|
||||
"https://cdn.discordapp.com/attachments/1/2/img.png?ex=78c&is=78d&hm=tokB",
|
||||
);
|
||||
expect(a).toBe("https://cdn.discordapp.com/attachments/1/2/img.png");
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
it("strips render variants from media.discordapp.net URLs", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://media.discordapp.net/attachments/1/2/img.png?format=webp&width=400&height=300",
|
||||
);
|
||||
const b = normalizeDiscordImageUrl(
|
||||
"https://media.discordapp.net/attachments/1/2/img.png?format=png&width=1024&height=768",
|
||||
);
|
||||
expect(a).toBe(b);
|
||||
expect(a).toBe("https://media.discordapp.net/attachments/1/2/img.png");
|
||||
});
|
||||
|
||||
it("strips query params from images-ext preview hosts", () => {
|
||||
const a = normalizeDiscordImageUrl(
|
||||
"https://images-ext-1.discordapp.net/external/X/https/example.com/cat.jpg?format=webp",
|
||||
);
|
||||
expect(a).toBe(
|
||||
"https://images-ext-1.discordapp.net/external/X/https/example.com/cat.jpg",
|
||||
);
|
||||
});
|
||||
|
||||
it("leaves URLs without query untouched", () => {
|
||||
const u = "https://cdn.discordapp.com/attachments/1/2/img.png";
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
|
||||
it("leaves non-Discord URLs untouched (query may be meaningful)", () => {
|
||||
const u = "https://example.com/image?token=abc&id=1";
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
|
||||
it("leaves data: URLs untouched", () => {
|
||||
const u = "data:image/png;base64,iVBORw0KGgoAAAANS?weird=query";
|
||||
// new URL() parses data: with protocol "data:" → not http(s) → untouched.
|
||||
expect(normalizeDiscordImageUrl(u)).toBe(u);
|
||||
});
|
||||
});
|
||||
|
||||
describe("makeImageCacheKey — Discord URL unification", () => {
|
||||
it("produces the SAME key for the same attachment across token variants", () => {
|
||||
const keyA = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/pic.png?ex=111&is=222&hm=AAA",
|
||||
);
|
||||
const keyB = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/pic.png?ex=333&is=444&hm=BBB",
|
||||
);
|
||||
expect(keyA).toBe(keyB);
|
||||
});
|
||||
|
||||
it("still produces DIFFERENT keys for different attachments", () => {
|
||||
const keyA = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/a.png?ex=1",
|
||||
);
|
||||
const keyB = makeImageCacheKey(
|
||||
"https://cdn.discordapp.com/attachments/9/8/b.png?ex=1",
|
||||
);
|
||||
expect(keyA).not.toBe(keyB);
|
||||
});
|
||||
|
||||
it("preserves the historical full-hash behavior for non-Discord URLs", () => {
|
||||
// Regression guard: external URLs keep pre-change keys.
|
||||
const key = makeImageCacheKey("https://example.com/x.png");
|
||||
expect(key).toBe(makeImageCacheKey("https://example.com/x.png"));
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,79 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// classifyIndividualWorkerResult — upload-pending vs success vs incomplete vs error
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Bug (2026-08-24): the worker's upload-pending race guard returned
|
||||
// {ok:true, results:[]}; the processor treated it as a completed moderation,
|
||||
// leaving messages stuck in `processing` until the 300s cleanup reverted them.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { classifyIndividualWorkerResult } from "../src/modules/ai-moderation/fallbackResultClassifier.js";
|
||||
|
||||
describe("classifyIndividualWorkerResult", () => {
|
||||
it("classifies the upload-pending race guard signal FIRST", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: true,
|
||||
results: [],
|
||||
uploadPending: true,
|
||||
}),
|
||||
).toBe("upload_pending");
|
||||
});
|
||||
|
||||
it("classifies a normal verdict as success", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: true,
|
||||
results: [{ status: "clean", flags: [] }],
|
||||
}),
|
||||
).toBe("success");
|
||||
});
|
||||
|
||||
it("classifies analysis_incomplete as incomplete", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: true,
|
||||
results: [{ status: "error", flags: ["analysis_incomplete"] }],
|
||||
}),
|
||||
).toBe("incomplete");
|
||||
});
|
||||
|
||||
it("accepts flags as JSON string (DB round-trip shape)", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: true,
|
||||
results: [
|
||||
{ status: "error", flags: JSON.stringify(["analysis_incomplete"]) },
|
||||
],
|
||||
}),
|
||||
).toBe("incomplete");
|
||||
});
|
||||
|
||||
it("classifies ok:false as error", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: false,
|
||||
results: [],
|
||||
error: "boom",
|
||||
}),
|
||||
).toBe("error");
|
||||
});
|
||||
|
||||
it("THE BUG: empty results with ok:true is an ERROR, not a success", () => {
|
||||
// Old code silently succeeded here → stuck `processing` rows.
|
||||
expect(classifyIndividualWorkerResult({ ok: true, results: [] })).toBe(
|
||||
"error",
|
||||
);
|
||||
expect(
|
||||
classifyIndividualWorkerResult({ ok: true, results: [undefined] }),
|
||||
).toBe("error");
|
||||
expect(classifyIndividualWorkerResult({})).toBe("error");
|
||||
});
|
||||
|
||||
it("treats a result with no status as unexplainable (error)", () => {
|
||||
expect(
|
||||
classifyIndividualWorkerResult({
|
||||
ok: true,
|
||||
results: [{ flags: [] }],
|
||||
}),
|
||||
).toBe("error");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,37 @@
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Stored-status normalization — "warn" verdicts must survive the cache
|
||||
// ═══════════════════════════════════════════════════════════════════════════
|
||||
// Bug (2026-08-22): getCachedTextModeration() narrowed its return type to
|
||||
// "clean" | "flagged". A stored "warn" verdict with flags (e.g.
|
||||
// ["conflict_instigation"]) fell into the legacy `flags.length === 0 ?
|
||||
// clean : flagged` branch and was read back as FLAGGED. Downstream this
|
||||
// broke auto-delete eligibility gating and mislabelled warnings on the
|
||||
// dashboard. parseQdrantVerdict had the same narrowing (warn → clean).
|
||||
//
|
||||
// Fix: normalizeStoredStatus() accepts the full clean/warn/flagged union in
|
||||
// BOTH readers; unknown/legacy values still derive from flags.
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { normalizeStoredStatus } from "../src/modules/ai-moderation/textCacheStore.js";
|
||||
|
||||
describe("normalizeStoredStatus — warn survives cache round-trip", () => {
|
||||
it("keeps a stored 'warn' status as 'warn'", () => {
|
||||
expect(normalizeStoredStatus("warn", ["conflict_instigation"])).toBe(
|
||||
"warn",
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps stored 'clean' and 'flagged' unchanged", () => {
|
||||
expect(normalizeStoredStatus("clean", [])).toBe("clean");
|
||||
expect(normalizeStoredStatus("flagged", ["sara"])).toBe("flagged");
|
||||
});
|
||||
|
||||
it("derives from flags for legacy entries without a stored status", () => {
|
||||
expect(normalizeStoredStatus(undefined, [])).toBe("clean");
|
||||
expect(normalizeStoredStatus(undefined, ["spam"])).toBe("flagged");
|
||||
});
|
||||
|
||||
it("treats an unknown stored status like a legacy entry", () => {
|
||||
expect(normalizeStoredStatus("processing", [])).toBe("clean");
|
||||
expect(normalizeStoredStatus("processing", ["spam"])).toBe("flagged");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,91 @@
|
||||
# FE Optimasi: Error Handling, State Refactor, Polish
|
||||
|
||||
**Scope:** `services/frontend/src` — Next.js 16, React 19, SWR, oRPC-over-WS.
|
||||
**Branch:** `feat/fe-error-handling-state` (buat baru dari `main`).
|
||||
**User:** MythEclipse — bahasa Indonesia/Inggris campur, "gas" = eksekusi langsung.
|
||||
|
||||
## 1. Masalah yang teridentifikasi
|
||||
|
||||
### 1.1 Error handling tidak seragam
|
||||
- `ErrorState` (states.tsx:126) sudah ada dengan `onRetry?`, tapi **80% pemanggilan tidak pass `onRetry`** → user stuck, harus refresh manual.
|
||||
- Pola `if (error && !data) return <ErrorState />` berulang di 5+ view, setiap satu hardcode logic-nya.
|
||||
- `useAction` (use-action.ts) set `error` di state tapi komponen pakai `try/catch` lokal juga — **dua error states** (hook `action.error` + lokal `catch (e)`).
|
||||
|
||||
### 1.2 State duplication / double source of truth
|
||||
- `useReview` (use-messages.ts:129) pakai `refreshInterval: 15_000` (polling) padahal WS sudah broadcast `moderation_action` real-time. **Polling + WS = double-fetch + race condition.**
|
||||
- `page.tsx` dan `view.tsx` duplikat state `guildId`/`channelId`/`query` — logic selection ada di 2 tempat.
|
||||
- Chatbot pakai local `failed` state + `setFailed(text)` untuk retry, tapi `useAction` sudah ada `error` field yang tidak dipakai.
|
||||
- `useSpeakers` (use-voice.ts:50) pakai local `useState` untuk speakers, bukan SWR → tidak konsisten dengan pola SWR yang dipakai semua hook lain.
|
||||
|
||||
### 1.3 WebSocket error/user feedback lemah
|
||||
- `WsConnection.onerror` (connection.ts:78) cuma `setStatus("error")` — tidak ada info ke user "reconnecting… (attempt N)".
|
||||
- Tidak ada broadcast error ke komponen (tidak ada `ws_error` handler).
|
||||
|
||||
## 2. Rencana perubahan
|
||||
|
||||
### 2.1 Standardisasi ErrorBoundary + Retry (FE-only)
|
||||
**File:** `src/components/shared/states.tsx`
|
||||
- Tambahkan komponen `ErrorBoundary` (React error boundary untuk crash React, bukan fetch error).
|
||||
- Refactor semua `<ErrorState error={error} />` → `<ErrorState error={error} onRetry={retryFn} />`.
|
||||
- Tambahkan helper `withSWR` atau pattern: tiap view pakai `error` + `mutate`/`refetch` dari SWR dan pass ke ErrorState.
|
||||
|
||||
**File:** `src/app/(dashboard)/*/view.tsx` (6 files)
|
||||
- Setiap `ErrorState` dapat `onRetry` yang memanggil `mutate`/`refetch`.
|
||||
- `DashboardView`: `onRetry={() => void mutate(["dashboard-stats"])}` — tapi SWR keys tersebar. Solusi: export `mutate` via custom hook atau pakai `useSWR` config `onErrorRetry` global.
|
||||
- **Keputusan:** gunakan pola **global SWR config** (`src/lib/swr-config.ts`) dengan `onErrorRetry` backoff, dan tiap view pass `onRetry` explicit ke ErrorState.
|
||||
|
||||
### 2.2 Hapus polling `useReview`, ganti WS-driven
|
||||
**File:** `src/hooks/use-messages.ts`
|
||||
- Hapus `refreshInterval: 15_000` dari `useReview`.
|
||||
- Tambahkan `useReviewWsSync(ws)` — subscribe ke `moderation_action` WS event, mutate key `["messages-review", channelId]`.
|
||||
- Backend sudah broadcast `moderation_action` via WS (redis-channels.ts:130). Review messages yang di-flag akan dapat `moderation_action` event. **Bisa pakai ini.**
|
||||
|
||||
**File:** `src/app/(dashboard)/messages/view.tsx`
|
||||
- Tambahkan `useReviewWsSync(ws)` call.
|
||||
|
||||
### 2.3 Konsistensi state: `useSpeakers` → SWR
|
||||
**File:** `src/hooks/use-voice.ts`
|
||||
- Refactor `useSpeakers` agar pakai SWR key `["voice-speakers"]` + initialData dari `useVoiceStatus`. Ini memungkinkan revalidate + cache sharing.
|
||||
- Tapi `voice_state` dan `voice_active_user` adalah WS events — perlu persist ke SWR cache via `mutate`. Refactor: `useSpeakers` subscribe WS + mutate SWR key.
|
||||
|
||||
### 2.4 Chatbot: gunakan `useAction` error state, hapus duplikat `failed`
|
||||
**File:** `src/components/chatbot/chatbot.tsx`
|
||||
- Hapus `const [failed, setFailed]` — sebalihoikan ke `useAction` return `error`.
|
||||
- Tapi `useAction` `mutateAsync` throw — perlu catch. Refactor: pakai `mutate` (fire-and-forget) + `isPending` + `error`.
|
||||
- **Note:** chatbot pakai `send` yang butuh pemuatan history — tetap pakai local state untuk msgs tapi gunakan `action.error` untuk display.
|
||||
|
||||
### 2.5 WS error feedback
|
||||
**File:** `src/lib/ws/connection.ts`
|
||||
- `onerror` emit kode/status ke status listeners.
|
||||
- Tambahkan method `getReconnectAttempt()` atau expose via status change.
|
||||
|
||||
**File:** `src/lib/ws/context.tsx`
|
||||
- Subscribe `onStatusChange` di `WsProvider`, toast "Reconnecting… (attempt N)" ketika status `error`/`connecting`.
|
||||
|
||||
## 3. Verification
|
||||
- `npx tsc --noEmit` — compile OK
|
||||
- `npx biome check src/` — lint OK
|
||||
- `npm run build` — build OK (Next 16 SSG/SSR)
|
||||
- Manual: refresh halaman, pastikan ErrorState muncul dengan tombol Retry yang bisa diklik.
|
||||
|
||||
## 4. Files yang disentuh
|
||||
```
|
||||
src/components/shared/states.tsx # ErrorBoundary + helper
|
||||
src/lib/swr-config.ts # global SWR config (baru)
|
||||
src/hooks/use-messages.ts # useReviewWsSync, hapus polling
|
||||
src/hooks/use-voice.ts # useSpeakers → SWR
|
||||
src/hooks/use-action.ts # expose resetError
|
||||
src/lib/ws/connection.ts # reconnect info
|
||||
src/lib/ws/context.tsx # WS error toast
|
||||
src/app/(dashboard)/messages/view.tsx # useReviewWsSync
|
||||
src/app/(dashboard)/messages/page.tsx # SSR error passthrough
|
||||
src/app/(dashboard)/moderation/view.tsx # onRetry
|
||||
src/app/(dashboard)/dashboard/view.tsx # onRetry
|
||||
src/app/(dashboard)/voice/view.tsx # onRetry
|
||||
src/app/(dashboard)/media/view.tsx # onRetry
|
||||
src/app/(dashboard)/recordings/view.tsx # onRetry
|
||||
src/components/chatbot/chatbot.tsx # pakai useAction error
|
||||
```
|
||||
|
||||
## 5. Out of scope (BE)
|
||||
- Regex heuristic di moderation.repository.ts (scam domain extraction) — ini BE task, catat tapi jangan sentuh kecuali diminta.
|
||||
@@ -50,7 +50,12 @@ export function DashboardView({
|
||||
initialStats?: DashboardStats;
|
||||
initialActivity?: Awaited<ReturnType<typeof useActivity>>["data"];
|
||||
}) {
|
||||
const { data: stats, isLoading, error } = useStats(initialStats);
|
||||
const {
|
||||
data: stats,
|
||||
isLoading,
|
||||
error,
|
||||
mutate: mutateStats,
|
||||
} = useStats(initialStats);
|
||||
const { data: activity } = useActivity(14, initialActivity as never);
|
||||
const { data: reactors } = useTopReactors();
|
||||
const { data: reactions } = useTopReactions();
|
||||
@@ -65,7 +70,8 @@ export function DashboardView({
|
||||
);
|
||||
}, [stats, ambient]);
|
||||
|
||||
if (error && !stats) return <ErrorState error={error} />;
|
||||
if (error && !stats)
|
||||
return <ErrorState error={error} onRetry={() => void mutateStats()} />;
|
||||
if (!stats && isLoading)
|
||||
return (
|
||||
<div className="space-y-5">
|
||||
@@ -78,7 +84,13 @@ export function DashboardView({
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
if (!stats) return <ErrorState error={error ?? new Error("No data")} />;
|
||||
if (!stats)
|
||||
return (
|
||||
<ErrorState
|
||||
error={error ?? new Error("No data")}
|
||||
onRetry={() => void mutateStats()}
|
||||
/>
|
||||
);
|
||||
|
||||
const s = stats;
|
||||
const total = s.total_flagged + s.total_clean || 1;
|
||||
@@ -87,7 +99,7 @@ export function DashboardView({
|
||||
return (
|
||||
<div className="space-y-5">
|
||||
{/* Hero */}
|
||||
<GlassPanel glow className="relative overflow-hidden">
|
||||
<GlassPanel glow className="game-frame relative overflow-hidden">
|
||||
<div className="scan-line absolute inset-x-0 top-0" />
|
||||
<div className="flex flex-wrap items-end justify-between gap-4">
|
||||
<div>
|
||||
|
||||
@@ -33,7 +33,12 @@ import { useWebSocket } from "@/lib/ws/context";
|
||||
|
||||
export function MediaView({ initialStatus }: { initialStatus?: MediaState }) {
|
||||
const ws = useWebSocket();
|
||||
const { data: media, isLoading, error } = useMediaState(initialStatus);
|
||||
const {
|
||||
data: media,
|
||||
isLoading,
|
||||
error,
|
||||
mutate,
|
||||
} = useMediaState(initialStatus);
|
||||
const queue = useMediaQueue();
|
||||
const skip = useMediaSkip();
|
||||
const stop = useMediaStop();
|
||||
@@ -79,7 +84,8 @@ export function MediaView({ initialStatus }: { initialStatus?: MediaState }) {
|
||||
}
|
||||
};
|
||||
|
||||
if (error && !media) return <ErrorState error={error} />;
|
||||
if (error && !media)
|
||||
return <ErrorState error={error} onRetry={() => void mutate()} />;
|
||||
if (!media && isLoading)
|
||||
return (
|
||||
<div className="space-y-5">
|
||||
|
||||
@@ -39,6 +39,7 @@ import {
|
||||
useMessagesStream,
|
||||
useMessagesWsSync,
|
||||
useRecentEdits,
|
||||
useReviewWsSync,
|
||||
useSemanticSearch,
|
||||
} from "@/hooks";
|
||||
import { aiTone } from "@/lib/ai-status";
|
||||
@@ -94,6 +95,7 @@ export function MessagesView({
|
||||
data: messages,
|
||||
isLoading,
|
||||
error,
|
||||
refetch,
|
||||
} = useMessages(
|
||||
guildId ?? "",
|
||||
channelId ?? undefined,
|
||||
@@ -112,6 +114,7 @@ export function MessagesView({
|
||||
const hasMore = pageInfo?.hasMore ?? false;
|
||||
const loadMore = useLoadMore();
|
||||
useMessagesWsSync(ws, guildId ?? "");
|
||||
useReviewWsSync(ws);
|
||||
const search = useMessageSearch(
|
||||
query,
|
||||
query.trim().length >= 2 && !semanticMode,
|
||||
@@ -326,7 +329,7 @@ export function MessagesView({
|
||||
}
|
||||
/>
|
||||
{error && !messages ? (
|
||||
<ErrorState error={error} />
|
||||
<ErrorState error={error} onRetry={() => refetch()} />
|
||||
) : isLoading && !messages ? (
|
||||
<SkeletonRows rows={8} />
|
||||
) : list.length === 0 ? (
|
||||
|
||||
@@ -81,7 +81,12 @@ export function ModerationView({
|
||||
initialStats?: ModerationStats;
|
||||
initialActions?: ModerationAction[];
|
||||
}) {
|
||||
const { data: stats, isLoading, error } = useModerationStats(initialStats);
|
||||
const {
|
||||
data: stats,
|
||||
isLoading,
|
||||
error,
|
||||
mutate: mutateStats,
|
||||
} = useModerationStats(initialStats);
|
||||
const [statusFilter, setStatusFilter] = useState<string>("");
|
||||
const [typeFilter, setTypeFilter] = useState<string>("");
|
||||
const { data: actions } = useModerationActions(
|
||||
@@ -122,7 +127,8 @@ export function ModerationView({
|
||||
);
|
||||
}, [failedRate, ambient]);
|
||||
|
||||
if (error && !stats) return <ErrorState error={error} />;
|
||||
if (error && !stats)
|
||||
return <ErrorState error={error} onRetry={() => void mutateStats()} />;
|
||||
if (!stats && isLoading)
|
||||
return (
|
||||
<div className="space-y-5">
|
||||
@@ -131,7 +137,13 @@ export function ModerationView({
|
||||
<SkeletonRows rows={6} />
|
||||
</div>
|
||||
);
|
||||
if (!stats) return <ErrorState error={error ?? new Error("No data")} />;
|
||||
if (!stats)
|
||||
return (
|
||||
<ErrorState
|
||||
error={error ?? new Error("No data")}
|
||||
onRetry={() => void mutateStats()}
|
||||
/>
|
||||
);
|
||||
|
||||
const statusOpts: SelectOption[] = [
|
||||
{ value: "", label: "All statuses" },
|
||||
|
||||
@@ -33,7 +33,7 @@ export function RecordingsView({
|
||||
initialItems?: VoiceRecording[];
|
||||
}) {
|
||||
const ws = useWebSocket();
|
||||
const { data: items, isLoading, error } = useRecordings(initialItems);
|
||||
const { data: items, isLoading, error, mutate } = useRecordings(initialItems);
|
||||
const del = useDeleteRecording();
|
||||
useRecordingsWsSync(ws);
|
||||
const ambient = useAmbient();
|
||||
@@ -56,7 +56,8 @@ export function RecordingsView({
|
||||
}
|
||||
};
|
||||
|
||||
if (error && !items) return <ErrorState error={error} />;
|
||||
if (error && !items)
|
||||
return <ErrorState error={error} onRetry={() => void mutate()} />;
|
||||
if (!items && isLoading)
|
||||
return (
|
||||
<GlassPanel>
|
||||
|
||||
@@ -40,7 +40,12 @@ export function VoiceView({
|
||||
initialGuilds?: Guild[];
|
||||
}) {
|
||||
const ws = useWebSocket();
|
||||
const { data: status, isLoading, error } = useVoiceStatus(initialStatus);
|
||||
const {
|
||||
data: status,
|
||||
isLoading,
|
||||
error,
|
||||
mutate,
|
||||
} = useVoiceStatus(initialStatus);
|
||||
const connect = useVoiceConnect();
|
||||
const disconnect = useVoiceDisconnect();
|
||||
const mic = useMicTransmit(ws);
|
||||
@@ -68,7 +73,8 @@ export function VoiceView({
|
||||
else ambient.set("vermilion", 0.35, "voice idle");
|
||||
}, [status?.connected, ambient]);
|
||||
|
||||
if (error && !status) return <ErrorState error={error} />;
|
||||
if (error && !status)
|
||||
return <ErrorState error={error} onRetry={() => void mutate()} />;
|
||||
if (!status && isLoading)
|
||||
return (
|
||||
<div className="space-y-5">
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
/* ── Surfaces (dark-first; light overrides below) ── */
|
||||
--color-canvas: oklch(0.12 0.014 70);
|
||||
--color-canvas-2: oklch(0.16 0.02 70);
|
||||
--color-surface: oklch(0.2 0.022 70 / 0.55);
|
||||
--color-surface: oklch(0.2 0.022 70 / 0.45);
|
||||
--color-surface-2: oklch(0.26 0.024 70 / 0.45);
|
||||
|
||||
/* ── Ink ── */
|
||||
@@ -29,13 +29,13 @@
|
||||
--color-hairline: oklch(1 0 0 / 0.1);
|
||||
--hairline-w: 1px;
|
||||
|
||||
/* ── Semantic signals ── */
|
||||
--color-signal: oklch(0.86 0.19 128);
|
||||
--color-signal-ink: oklch(0.18 0.03 70);
|
||||
--color-signal-glow: oklch(0.86 0.19 128 / 0.4);
|
||||
--color-amber: oklch(0.85 0.15 72);
|
||||
--color-vermilion: oklch(0.68 0.21 25);
|
||||
--color-vermilion-glow: oklch(0.68 0.21 25 / 0.4);
|
||||
/* ── Semantic signals (monochrome: tone = luminance steps) ── */
|
||||
--color-signal: oklch(0.97 0 0);
|
||||
--color-signal-ink: oklch(0.1 0 0);
|
||||
--color-signal-glow: oklch(1 0 0 / 0.35);
|
||||
--color-amber: oklch(0.72 0 0);
|
||||
--color-vermilion: oklch(0.85 0 0);
|
||||
--color-vermilion-glow: oklch(1 0 0 / 0.45);
|
||||
|
||||
--color-ring: var(--color-signal);
|
||||
|
||||
@@ -74,8 +74,8 @@
|
||||
background: oklch(0.22 0.02 70 / 0.34);
|
||||
}
|
||||
.light ::selection {
|
||||
background: var(--color-signal-glow);
|
||||
color: var(--color-ink);
|
||||
background: oklch(0.2 0 0 / 0.9);
|
||||
color: oklch(1 0 0);
|
||||
}
|
||||
|
||||
@layer base {
|
||||
@@ -300,6 +300,144 @@
|
||||
}
|
||||
}
|
||||
|
||||
/* ═══ Game-menu nav (monochrome) ═══
|
||||
Corner brackets + one-shot light sweep + staggered entrance.
|
||||
Pure transform/opacity — compositor only. */
|
||||
|
||||
.game-nav-item {
|
||||
--i: 0;
|
||||
position: relative;
|
||||
isolation: isolate;
|
||||
animation: nav-in 0.45s cubic-bezier(0.22, 1, 0.36, 1) both;
|
||||
animation-delay: calc(var(--i) * 45ms);
|
||||
}
|
||||
/* corner brackets on hover/active */
|
||||
.game-nav-item::before,
|
||||
.game-nav-item::after {
|
||||
content: "";
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
opacity: 0;
|
||||
transition: opacity 0.2s ease;
|
||||
pointer-events: none;
|
||||
}
|
||||
.game-nav-item::before {
|
||||
border-left: 2px solid currentColor;
|
||||
border-top: 2px solid currentColor;
|
||||
border-top-left-radius: 13px;
|
||||
clip-path: polygon(0 0, 62% 0, 62% 30%, 30% 30%, 30% 100%, 0 100%);
|
||||
}
|
||||
.game-nav-item::after {
|
||||
border-right: 2px solid currentColor;
|
||||
border-bottom: 2px solid currentColor;
|
||||
border-bottom-right-radius: 13px;
|
||||
clip-path: polygon(100% 100%, 38% 100%, 38% 70%, 70% 70%, 70% 0, 100% 0);
|
||||
}
|
||||
.game-nav-item:hover::before,
|
||||
.game-nav-item:hover::after,
|
||||
.game-nav-item.is-active::before,
|
||||
.game-nav-item.is-active::after {
|
||||
opacity: 1;
|
||||
}
|
||||
/* one-shot sweep */
|
||||
.game-nav-item .game-sweep {
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
overflow: hidden;
|
||||
border-radius: inherit;
|
||||
pointer-events: none;
|
||||
}
|
||||
.game-nav-item .game-sweep > i {
|
||||
position: absolute;
|
||||
top: -20%;
|
||||
bottom: -20%;
|
||||
width: 34%;
|
||||
left: -60%;
|
||||
background: linear-gradient(
|
||||
105deg,
|
||||
transparent,
|
||||
oklch(1 0 0 / 0.16),
|
||||
transparent
|
||||
);
|
||||
transform: skewX(-14deg);
|
||||
}
|
||||
.game-nav-item:hover .game-sweep > i,
|
||||
.game-nav-item.is-active .game-sweep > i {
|
||||
animation: sweep-x 0.55s ease-out both;
|
||||
}
|
||||
/* active triangle marker (game cursor) */
|
||||
.game-marker {
|
||||
position: absolute;
|
||||
top: 50%;
|
||||
width: 0;
|
||||
height: 0;
|
||||
border-block: 7px solid transparent;
|
||||
border-left: 11px solid currentColor;
|
||||
transform: translateY(-50%);
|
||||
filter: drop-shadow(0 0 6px var(--color-signal-glow));
|
||||
animation: marker-nudge 1.6s ease-in-out infinite;
|
||||
}
|
||||
|
||||
@keyframes marker-nudge {
|
||||
0%, 100% { transform: translateY(-50%) translateX(0); }
|
||||
50% { transform: translateY(-50%) translateX(3px); }
|
||||
}
|
||||
|
||||
@keyframes sweep-x {
|
||||
from { left: -60%; }
|
||||
to { left: 130%; }
|
||||
}
|
||||
@keyframes nav-in {
|
||||
from { opacity: 0; transform: translateX(-10px); }
|
||||
to { opacity: 1; transform: translateX(0); }
|
||||
}
|
||||
|
||||
/* ── Game frame: panel with cut corners + drawn-in edge lines ── */
|
||||
.game-frame {
|
||||
position: relative;
|
||||
}
|
||||
.game-frame::before {
|
||||
content: "";
|
||||
position: absolute;
|
||||
inset: 0;
|
||||
border-radius: inherit;
|
||||
pointer-events: none;
|
||||
background:
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) top left /
|
||||
26px 1.5px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) top left / 1.5px
|
||||
26px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) top right /
|
||||
26px 1.5px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) top right / 1.5px
|
||||
26px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) bottom left /
|
||||
26px 1.5px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) bottom left /
|
||||
1.5px 26px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) bottom right /
|
||||
26px 1.5px,
|
||||
linear-gradient(var(--color-hairline), var(--color-hairline)) bottom right /
|
||||
1.5px 26px;
|
||||
background-repeat: no-repeat;
|
||||
}
|
||||
|
||||
/* ── Mobile lightness (<md): cheaper blur, shorter animations ── */
|
||||
@media (max-width: 767px) {
|
||||
.glass {
|
||||
backdrop-filter: blur(8px);
|
||||
-webkit-backdrop-filter: blur(8px);
|
||||
}
|
||||
.game-nav-item {
|
||||
animation-duration: 0.25s;
|
||||
animation-delay: calc(var(--i) * 18ms);
|
||||
}
|
||||
.scan-line::after,
|
||||
.animate-shimmer {
|
||||
display: none;
|
||||
}
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.scan-line::after,
|
||||
.scan-line,
|
||||
|
||||
@@ -67,6 +67,19 @@ export function AmbientCanvas({
|
||||
const mount = mountRef.current;
|
||||
if (!mount) return;
|
||||
|
||||
// Lightness gate: skip the WebGL layer entirely on small/coarse/data-saver
|
||||
// devices — the static CSS fallback below stays. Desktop keeps the haze.
|
||||
const nav = navigator as Navigator & {
|
||||
connection?: { saveData?: boolean; deviceMemory?: number };
|
||||
deviceMemory?: number;
|
||||
};
|
||||
const lightDevice =
|
||||
window.matchMedia("(max-width: 767px)").matches ||
|
||||
window.matchMedia("(pointer: coarse)").matches ||
|
||||
nav.connection?.saveData === true ||
|
||||
(nav.deviceMemory ?? 8) <= 4;
|
||||
if (lightDevice) return;
|
||||
|
||||
let renderer: THREE.WebGLRenderer;
|
||||
try {
|
||||
renderer = new THREE.WebGLRenderer({
|
||||
|
||||
@@ -12,11 +12,14 @@ import { AmbientCanvas } from "./ambient-canvas";
|
||||
|
||||
export type SignalTone = "signal" | "amber" | "vermilion";
|
||||
|
||||
/** sRGB triplets for the three semantic signals (matches globals.css). */
|
||||
/**
|
||||
* sRGB triplets for the three semantic signals — MONOCHROME build:
|
||||
* tone is expressed as luminance/intensity, not hue (black & white theme).
|
||||
*/
|
||||
export const SIGNAL_RGB: Record<SignalTone, [number, number, number]> = {
|
||||
signal: [0.42, 1.0, 0.52],
|
||||
amber: [1.0, 0.76, 0.28],
|
||||
vermilion: [1.0, 0.34, 0.28],
|
||||
signal: [0.92, 0.94, 0.96], // nominal — near-white
|
||||
amber: [0.62, 0.64, 0.68], // warn — mid grey
|
||||
vermilion: [0.78, 0.8, 0.84], // danger — bright flash grey
|
||||
};
|
||||
|
||||
export interface AmbientState {
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
"use client";
|
||||
|
||||
import { Component, type ReactNode } from "react";
|
||||
import { ErrorState } from "./states";
|
||||
|
||||
/**
|
||||
* React error boundary — catches render/exception crashes in child trees
|
||||
* (e.g. third-party lib throwing on unexpected payload shape) and surfaces
|
||||
* a consistent ErrorState instead of unmounting the whole app shell.
|
||||
*
|
||||
* Usage: wrap leaf views in <ErrorBoundary><SomeView /></ErrorBoundary>.
|
||||
*/
|
||||
interface ErrorBoundaryProps {
|
||||
children: ReactNode;
|
||||
fallback?: (error: Error, reset: () => void) => ReactNode;
|
||||
onReset?: () => void;
|
||||
}
|
||||
|
||||
interface ErrorBoundaryState {
|
||||
error: Error | null;
|
||||
}
|
||||
|
||||
export class ErrorBoundary extends Component<
|
||||
ErrorBoundaryProps,
|
||||
ErrorBoundaryState
|
||||
> {
|
||||
constructor(props: ErrorBoundaryProps) {
|
||||
super(props);
|
||||
this.state = { error: null };
|
||||
}
|
||||
|
||||
static getDerivedStateFromError(error: Error): ErrorBoundaryState {
|
||||
return { error };
|
||||
}
|
||||
|
||||
reset = () => {
|
||||
this.setState({ error: null });
|
||||
this.props.onReset?.();
|
||||
};
|
||||
|
||||
override render() {
|
||||
if (!this.state.error) return this.props.children;
|
||||
|
||||
if (this.props.fallback) {
|
||||
return this.props.fallback(this.state.error, this.reset);
|
||||
}
|
||||
|
||||
return (
|
||||
<ErrorState
|
||||
title="Something went wrong"
|
||||
error={this.state.error}
|
||||
onRetry={this.reset}
|
||||
/>
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ export { GuildChannelPicker } from "./guild-picker";
|
||||
export { MarkdownLite } from "./markdown";
|
||||
export { PageTransition } from "./page-transition";
|
||||
export { MetricTile, SectionHeader } from "./section";
|
||||
export { ErrorBoundary } from "./error-boundary";
|
||||
export {
|
||||
EmptyState,
|
||||
ErrorState,
|
||||
|
||||
@@ -24,16 +24,25 @@ export function MobileNav() {
|
||||
aria-label={item.label}
|
||||
aria-current={active ? "page" : undefined}
|
||||
className={cn(
|
||||
"relative flex flex-1 flex-col items-center justify-center rounded-[12px] rounded-t-[20px] px-2 pb-[calc(0.4rem+env(safe-area-inset-bottom))] pt-2 md:hidden",
|
||||
"game-nav-item relative flex min-h-[44px] flex-1 flex-col items-center justify-center rounded-t-[20px] px-2 pt-2 pb-[calc(0.4rem+env(safe-area-inset-bottom))] md:hidden",
|
||||
active
|
||||
? "bg-signal/15 text-signal"
|
||||
: "text-ink-faint hover:bg-white/5 hover:text-ink-soft",
|
||||
? "is-active text-white"
|
||||
: "text-ink-faint hover:text-ink-soft",
|
||||
)}
|
||||
>
|
||||
{active && (
|
||||
<span className="absolute -top-2 h-8 w-8 rounded-full bg-signal shadow-[0_0_12px_var(--color-signal-glow)]" />
|
||||
<span
|
||||
aria-hidden="true"
|
||||
className="absolute inset-x-4 top-0 h-[2px] bg-white shadow-[0_0_8px_var(--color-signal-glow)]"
|
||||
/>
|
||||
)}
|
||||
<item.icon className="size-[20px]" strokeWidth={active ? 2.4 : 2} />
|
||||
<span className="game-sweep" aria-hidden="true">
|
||||
<i />
|
||||
</span>
|
||||
<item.icon
|
||||
className="relative z-10 size-[20px]"
|
||||
strokeWidth={active ? 2.4 : 2}
|
||||
/>
|
||||
{item.label}
|
||||
</Link>
|
||||
);
|
||||
|
||||
Loaded 50 of 60 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user