Jev (oc/jev-1.13-free via 9router /v1/systemone) added as primary text
analyzer was underperforming. Delete the whole feature:
- jevAnalyzer.ts + its unit & live-smoke tests
- Jev-first branch in textBatchProcessor, restore pure callModerationLLM path
- AI_LLM_JEV_* config vars (zod) and .env.example entries
- @typesafe-ai/sdk dependency (+ lockfile)
Behavior: text moderation is LLM-only again, exactly as before the
Jev feature; AGENTS.md invariant 'LLM is the only judge' holds.
* feat(gateway): Jev (System One) as primary text moderator with LLM fallback
Add TypeSafe Jev via @typesafe-ai/sdk v0.6.0 as the PRIMARY analyzer for
text-only moderation sub-batches; the existing LLM stays as the fallback
for anything Jev cannot decide confidently (per-message gate rejection or
API failure) and for media batches.
- jevAnalyzer.ts: TypeSafeClient wrapper, declarative state builder
(System One models MUST get factual state, not chat-XML — chat framing
made Jev confidently wrong on clean messages at 0.98 confidence),
message-id-keyed question builder (5 typed questions per message),
cross-consistency acceptance gate (noul↔status↔severity↔action↔category),
answer→AnalysisResult mapper, fail-open outcome.
- textBatchProcessor.ts: Jev-first per sub-batch, rejected ids + API
failures fall back to callModerationLLM; no cross-batch pollution.
- config: AI_LLM_JEV_ENABLED/API_KEY/BASE_URL/MODEL/TIMEOUT_MS/MIN_CONFIDENCE.
- .env.example: documented all 6 Jev env vars.
- tests: unit (question/state builders, gate, mapper) + live smoke
(gated behind AI_LLM_JEV_SMOKE=1) verified 4/4 accepted vs real 9router.
* fix: auto-fix code quality [skip ci]
- new tinyFishSearch module: GET api.search.tinyfish.ai with X-API-Key,
maps top-3 to SearchResult shape, never throws (all failure modes -> [])
- wikipediaSearch: on wiki miss, one tinyfish attempt; hits cached 6h
under the same key so fallback latency is paid once
- termGlossary: on summary miss, top tinyfish hit becomes the definition
(persisted permanently like wiki defs); miss keeps 1h sentinel
- config: TINYFISH_API_KEY (empty = fallback disabled), ENABLED,
BASE_URL, TIMEOUT_MS, LOCATION, LANGUAGE knobs
- tests: 6 coverage for disabled/mapping/non-OK/network/bad-json
API key NOT committed — set TINYFISH_API_KEY in BWS gmw secrets.
Verified: typecheck + lint clean, 216/216 tests pass, live probe
'gubernur jawa barat' returned 3 mapped results
Jockie Music (user 411916947773587456) posts now-playing embeds/spotify links
~1347 captured messages — every one consumed a moderation LLM call for zero
signal and contributed to batch timeouts. Config AI_SKIP_ANALYSIS_USER_IDS
(default=Jockie) skips them at ALL three analysis paths:
- queueMessageAnalysis entry (direct skip-result like age-restricted)
- batchScheduler processing (pre-batch filter)
- individual recovery path (no fallback spam for already-skipped authors)
Skip-result mirrors age_restricted: status=clean, flags=[skip_analysis_user],
action=none — stays visible in the dashboard, never analyzed.
The text model behind omniroute/9router consistently takes >45s on long-context
batches. At 45s every such batch fell through to the individual-fallback
queue which re-runs with its own timeout, then exhausted to ai_status=error.
75s keeps the bounded budget while letting the first-pass batch succeed.
- AI_LLM_MEDIA_ANALYSIS_TIMEOUT_MS 60s→120s + vision 60s→120s: vision model
via router regularly exceeded 60s, dropping media batches into the
individual-fallback chain then exhausting into ai_status=error.
- Qdrant upserts: retryWithRetry() wraps PUT /points with exponential
backoff (3 attempts, jitter) for transient 408/abort/ECONNRESET — the
41 six-hour 'Qdrant upsert failed — semantic entry skipped' warnings were
single-hop timeouts on a healthy-but-loaded Qdrant.
- LLM caller: on parse failure, attempt extractJson() structural repair of
the raw content (models with thinking disabled sometimes emit JSON as
plain text) before giving up and re-requesting.
- AI_VOICE_TRANSCRIPTION_MODEL config (default whisper-1) so the model can be a provider-qualified id (openrouter/openai/whisper-1) that actually has credentials through 9router/omniroute — bare whisper-1 maps to the openai provider which has none
- response_format json (not text): 9router proxies only json/verbose_json transcription responses; text returns 400
- parse text from the json response object
- prod env updated: model=openrouter/openai/whisper-1 (still needs OpenRouter STT balance — 402 until funded)
Root-cause fixes for 'banyak miss & terpotong' in the voice->recording flow:
- subscribe BEFORE collecting user metadata. receiver.speaking 'start' fires
on the FIRST opus packet, and onUdpMessage forwards frames to the
subscription only when one exists — every frame during the old
await collectUserMetadata (a Discord REST roundtrip on cache miss) was
dropped, cutting off the start of every burst. Now subscribe synchronously
(guard first, no await in between), then fetch metadata in the background
and discard the burst if the speaker turns out to be a bot.
- one segment per burst: drop the fixed 5s RECORDING_SEGMENT_MS rotation on
the OGG path, which split continuous speech mid-word/sentence. Only the
web-PCM decoder still rotates (bounds memory).
- finalize only once the underlying file has flushed to disk (wait on the
write stream 'finish'), so upload/transcode reads a complete file.
- raise AfterSilence 3000->4000ms so natural pauses (thinking, interruptions)
don't split one utterance into several recordings.
- lower the 'too short to keep' threshold 1000->300ms so brief replies
("ya", "siap") are kept instead of dropped.
All typecheck / biome(src/) / vitest (164) green.
Switch GMW's AI LLM base URL from 9router (https://9router.asepharyana.my.id/v1)
to omniroute on imrnes (http://100.121.180.82:20128/api/v1).
- Update default AI_LLM_BASE_URL in discord-gateway + backend config schemas
- Update .env.example documentation
- Update all 9router references in comments/docs/tests to omniroute
- Production BWS secret gmw_ai_llm_base_url already updated
Omniroute uses /api/v1 prefix (not /v1 like 9router), so the base URL
now correctly points at the right API path for the OpenAI SDK.
- Switch AI_LLM_BASE_URL from omniroute.imrnes.team to 9router.asepharyana.my.id
- Keep AI_LLM_MODEL as 'text' (9router uses alias-based routing, not bare names)
- Update .env.example comments to document 9router
- Per user: multimodal stays 'multimodal' alias
API verified: curl to 9router/v1/chat/completions with model 'text'
returns HTTP 200 (OpenAI-compatible format)
- Change AI_LLM_BASE_URL default from omniroute.imrnes.team to 9router.asepharyana.my.id
- Update AI_LLM_MODEL default from 'text' to 'claude-opus-5' (bare model name
compatible with 9router/OpenAI-compatible router)
- Update .env.example and inline comments to reflect 9router
- discord-gateway config now matches backend (which already uses 9router)
Batch race guard balikin {ok:true, rows:[]} tanpa sinyal saat semua target
masih upload-pending -> processor klasifikasi semua incomplete -> fanout ke
individual queue -> di situ requeue + reschedule 250ms -> balik ke batch:
hot loop ~300ms sepanjang upload (10 siklus/3 dtk di log prod 08:13).
Fix: worker batch kini return uploadPendingIds eksplisit; classifier pure
baru (partitionBatchOutcome) partisi completed/upload_pending/incomplete/
parse_failed/api_failed; target upload-pending DEFERRED dengan poll backoff
linear (AI_ANALYSIS_UPLOAD_POLL_MS 1500 base, cap AI_ANALYSIS_MAX_UPLOAD_POLL_MS
8000), tidak pernah masuk fanout; tail shouldScheduleNext tak menimpa defer.
Test: tests/batchOutcomeClassifier.test.ts (8 kasus, pure tanpa DB/Piscina).
- Fase-1 exact-cache lookup: N query serial -> SATU query ANY($1::text[])
- Global reuse utk bare key legacy, HANYA verdict non-actionable
(clean/flagless/action=none, conf>=0.85, umur<=72h) — flagged/warn
tetap context-scoped
- Semantic cache dua-band: clean band 0.92 default, actionable tetap
0.97; di antara band -> LLM (fail-open ke akurasi)
- hit_count kini di-increment (bulk UPDATE per batch) -> hit-rate terukur
- Cache hasil wikipediaSearch di Redis (6h, hanya hasil non-kosong)
- Memoize fetchUrlSafely utk type=text (LRU 30m + in-flight dedupe)
- makeImageCacheKey strip query CDN Discord (?ex/is/hm, format/width)
-> attachment sama = satu key vision, skip re-download+re-vision
Spec: .hermes/plans/2026-08-24-ai-analysis-cache-optimization.md
Tests: +33 (cacheGuards, discordImageKeyNormalize, cacheBatchLookup)
- Persist structured verdict (flags/severity/confidence/evidence) on
moderation_actions so the public web can show WHY a message was moderated.
- Add a persistent Qdrant archive collection (gmw_message_archive); embed
every captured message at capture time (fire-and-forget, best-effort).
- Public semantic search over the archive (backend oRPC + FE toggle on the
messages view). Both features are read-only/public and fully automatic.
Migration: 0015_add_moderation_explainability.sql
- Memoize buildSystemPrompt by (mode|channelCulture); identical signatures
now reuse the ~5k-token core instead of rebuilding per sub-batch call
(textBatchProcessor rebuilt it inside the loop; a 200-msg batch re-sent
the full system prompt ~4x). Correction tail stays per-attempt (uncached).
- Hoist URL-image -> vision evidence out of the per-sub-batch loop in
textBatchProcessor: it depends only on fetched images + full target set,
so compute once per whole batch, not per sub-batch.
- Compact system instructions: collapse 3x-duplicated 'evaluate by content
alone' statements into one standalone rule; trim output.ts channel-culture
+ context framing already covered by rules.ts/system.ts; drop duplicate
programming-error-log few-shot (id 17, covered by rules AMAN list).
- Fix misleading config default: AI_LLM_BASE_URL default -> omniroute
(gateway already runs omniroute via BWS; 9router was dead/misleading).
typecheck + lint + build green.
- Add wikipediaClient.ts: native fetch to Wikipedia REST/Action APIs
(search + summary), no extra npm dependency.
- Extract shared Redis cache into cacheStore.ts (decoupled from search).
- Term glossary now uses wikipediaSummary for direct article lookup.
- Remove searxngSearch.ts entirely; drop SEARXNG_BASE_URL config,
add WIKIPEDIA_LANG / WIKIPEDIA_TIMEOUT_MS.
- Rename backend searxngCalls metric to webSearchCalls.
User insight: personal profile summaries bloat the prompt (less room per
request) and add a per-user DB/Redis round-trip for little moderation signal.
Only the behavioural <user_reputation> history is kept.
- textBatchProcessor: stop fetching getUserProfile; remove <user_profiles>
block + <user_profile_ref> from message tags. Keep <user_reputation>.
- mediaBatchProcessor + visionAnalyzer: same removal (profile fetch + ref).
- prompts/system.ts + prompts/output.ts: drop stale <user_profiles>/
<user_profile_ref> instructions; point LLM at <user_reputation> instead.
- aiAnalyzer: gate userProfileLearner behind AI_USER_PROFILE_LEARNING_ENABLED
(default false) — generates profiles nobody reads, pure LLM/DB waste.
- Add AI_USER_PROFILE_LEARNING_ENABLED config knob.
Net: smaller prompts (more messages fit per request), fewer DB round-trips
per sub-batch, and no background LLM calls learning unused profiles.
tsc, biome, vitest (129) all clean.
User insight: rather than many small per-batch API requests, pack many
messages into ONE request so a burst is analyzed with far fewer calls.
- AI_LLM_TEXT_BATCH_SIZE 20 -> 60 (one request now carries ~3x more messages).
- AI_ANALYSIS_MAX_TARGET_TOKENS 4000 -> 14000 (the scheduler's token-budget
gate was trimming pending messages to ~20 before they reached the sub-batch
splitter; raising it lets ~60 messages through to a single LLM call).
- AI_LLM_TEXT_ANALYSIS_TIMEOUT_MS 30000 -> 45000 (one larger call needs more
headroom; gemini-flash-lite has a 1M-token context so 14k+8k is trivial).
Net effect when ramai: a 60-message burst = 1-2 API calls instead of 3+,
less semaphore contention, faster throughput.
- Parallelize per-user reputation/profile fetches in textBatchProcessor
(was a serial ~2N DB/Redis round-trip loop per sub-batch; now Promise.all
over unique users). Cuts per-batch latency, biggest win on small/quiet
batches.
- Make the LLM concurrency semaphore dynamic (cached per config value) instead
of frozen at import time, so AI_LLM_MAX_CONCURRENT is tunable without code
change and reflects current config.
- Bump AI_LLM_MAX_CONCURRENT default 5 -> 8 (gemini-flash-lite is cheap; helps
throughput when busy).
- Lower AI_ANALYSIS_DEBOUNCE_MS 500 -> 250 (snappier first-message analysis
when quiet).
- Lower AI_ANALYSIS_RECOVERY_INTERVAL_MS 15000 -> 10000 (stuck/errored
messages re-analyze sooner).
tsc, biome, vitest (129) all clean.
The standalone image analysis path (analyzeSingleMediaImage → llmVision →
llmChat) previously had no request-level timeout of its own — it silently
inherited the shared OpenAI client default (60s), and AI_LLM_MEDIA_ANALYSIS_
TIMEOUT_MS only governed the text+media *batch*, not a single vision call.
- Add AI_LLM_VISION_ANALYSIS_TIMEOUT_MS (default 60000) to config.
- llmChat now accepts an optional per-request `timeout` in LlmCallOpts,
forwarded to the OpenAI request options (falls back to the 60s client
default when omitted).
- llmVision passes config.AI_LLM_VISION_ANALYSIS_TIMEOUT_MS, so a single
image/sticker/emoji analysis gets a guaranteed 1-minute budget and is
independently tunable from the text path.
Verified: tsc + biome green, 129 gateway tests pass.
Co-Authored-By: Claude Opus 5 (Nous Research)
- gateway-metrics: collectors now run per scrape so Prometheus sees real
data (process memory/uptime + live AI-analysis pipeline gauges) instead
of an always-empty stub. bootstrap registers the pipeline collectors.
- systemd: MemoryMax 512M -> 1G (live RSS ~500MiB, peak 508MiB; 512M left
~2% headroom and risked an OOM-kill restart; host has 8GB free).
- config: POSTGRES_POOL_MIN 2 -> 0 so main + 4 Piscina worker threads don't
hold ~10 permanently-open idle pg connections against PgBouncer.
- docs: rewrite stale ARCHITECTURE.md / MODULE_STRUCTURE.md (winston ->
pino, removed mock-crc/indonesianTextNormalizer, renamed
aiAnalysisWorker/llmModerationClient).
Verified: tsc clean, 129 vitest pass, biome clean on changed files.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- config: add AI_LLM_VISION_BASE_URL + AI_LLM_VISION_API_KEY (separate from text router)
- llmClient: llmVision() now calls dedicated vision endpoint when configured
(axios POST to integrate.api.nvidia.com, model nvidia/nemotron-3-nano-omni-30b-a3b-reasoning,
reasoning_budget 16384, non-stream), falls back to router combo otherwise
- keeps text/moderation on omniroute, vision on NVIDIA direct
- Add term_glossary_cache table + migration 0014: resolved definitions are
stored permanently (definitions rarely change); misses stay ephemeral in
Redis/LRU with 1h TTL so transient failures get retried
- Lookup flow: LRU -> Redis -> Postgres (permanent) -> live SearXNG; DB hits
re-warm the fast caches; stale Redis miss sentinels no longer shadow DB
- Rate-limit-aware live lookups: concurrency 2 + stagger, retry once on empty
results, strict definition filter (Wikipedia preferred, rejects
disambiguation/ads/translate-homepages)
- Make SEARXNG_BASE_URL configurable via env (default unchanged)
When the ONLY violation is offensive_username (message content clean):
- Message is NOT deleted (nickname-only violation bypasses auto-delete)
- Member's server nickname is reset to default username via
setNickname(null) (Discord shows the global username again)
- Action 'reset_nickname' logged to moderation_actions; cooldown
10min per guild:user (LRU) so repeated messages by same member
don't hammer the Discord PATCH
- Config: AUTO_NICKNAME_RESET_ENABLED / AUTO_NICKNAME_RESET_COOLDOWN_MS
- Conversation context recency gates (GAP_MS/MAX_AGE_MS): drop stale
messages before silence gaps; cold_start anchor + flow descriptor
tells LLM whether conversation is ongoing or restarted
- [location] block: channel name, thread name, nsfw/age flags from
captured metadata (thread names instead of bare IDs)
- Link media -> multimodal: text-batch URL fetches that resolve to
images now run vision analysis (bounded 15s) and switch prompt to
mixed mode; <web_content> gains og:title for page context
- pnpm-workspace.yaml: approve sharp build script (unblocks install)
- Deleted the text analysis prompt constants and helpers as they are no longer needed.
- Added batch search functionality for Qdrant to optimize vector searches.
- Implemented methods for deleting expired Qdrant points and invalidating cache based on content hash.
- Updated text batch processor to use new timeout configurations and modified content building for moderation prompts.
- Enhanced text cache store to support new Qdrant integration and improved cache invalidation logic.
- Introduced a new user reputation model with a more nuanced trust scoring system, including penalties and rewards for user behavior.
- Added unit tests for the new trust model to ensure correctness of penalty and trust gain calculations.
- Updated configuration schema to reflect new timeout settings and removed deprecated OpenAI moderation keys.
New qdrantClient.ts (zero-dep fetch REST): ensure collection with cosine
distance (auto-recreate on vector-size change), upsert point w/ verdict
payload, search w/ expires_at filter + score threshold.
textCacheStore: when QDRANT_URL set, embeddings are upserted to Qdrant
(primary) and searched there first; Postgres embedding column remains as
legacy fallback for pre-Qdrant rows. Config: QDRANT_URL/COLLECTION/API_KEY.
QDRANT_URL already in repo .env; added to VPS env + GATEWAY_ENV secret.
Build & Deploy / build-and-push (backend) (push) Failing after 35s
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 25s
Build & Deploy / build-and-push (proxy) (push) Failing after 25s
- Remove pnpm workspace, moon repo, and all monorepo tooling
- Delete packages/shared/, embed shared code directly into each service
- Copy packages/shared/src/* -> services/backend/src/shared/ and services/discord-gateway/src/shared/
- Replace all @bete/shared imports with @/shared/ path alias
- Remove @bete/shared workspace dependency from both services
- Update root package.json scripts from --filter to --prefix
- Rewrite Dockerfiles to build each service standalone
- Clean up biome.json, .gitignore, remove root drizzle.config.ts
Build & Deploy / build-and-push (discord-gateway) (push) Failing after 2m22s
Build & Deploy / build-and-push (backend) (push) Failing after 3m22s
Build & Deploy / build-and-push (proxy) (push) Successful in 1m36s
Build & Deploy / deploy (push) Skipped
- Consolidate all DB schema definitions into packages/shared as single source of truth
- Migrate backend from raw SQL to Drizzle ORM across all modules
- Extract frontend inline UI into separate component files
- Refactor discord-gateway circuitBreaker into conversationState + moderationState
- Convert messageStore to Proxy singleton pattern
- Add validateBody/validateQuery middleware + Zod schemas for API endpoints
- Modernize Docker builds with multi-stage + pnpm deploy
- Migrate CI/CD from deployment to image-based pipeline
- Remove 60+ unused/dead files (~15K lines)
- Update color scheme from sky-blue to teal-cyan
- Move DB connection management to @bete/shared/database
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Shared Redis channel constants as single source of truth (redis-channels.ts)
- commandHandler.ts split into VoiceHandler, MediaHandler, GuildHandler,
ModerationHandler with handler-registry.ts dispatch
- messageStore.ts (1322 lines) split into domain-specific DB files:
messages.db.ts, attachments.db.ts, reviews.db.ts,
moderation-actions.db.ts, retention.db.ts
- recorder.ts startSpeaking callback extracted into speakingHandler.ts,
streamSetup.ts, segmentFinalizer.ts
- autoDeleteManager.ts split into autoDeleteEligibility.ts,
autoDeleteNotify.ts, autoDeleteLogger.ts
- Added createChildLogger() logging across 8 service files
- Backend messages.repository.ts migrated from raw SQL to Drizzle ORM
- Fixed biome.json to exclude packages/**/dist/* from lint
- Fixed config.ts GUILD_ID pre-existing type error
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Standardize MessageRecord types — single source of truth from @bete/shared
- Clean up config: remove unused GUILD_ID/TEXT_GUILD_ID/TEXT_CHANNEL_ID, fix WEBSERVER_PORT default (3001), remove default admin password
- Move mascot_chat_messages table to Drizzle schema with proper migration
- Remove runtime DDL (CREATE TABLE IF NOT EXISTS) from mascot-chat repository
- Remove phantom analytics/ module from documentation
- Add better-sqlite3 dependency to root devDependencies
- Replace 'as any' casts with proper type assertions across AI moderation
- Add error logging to silent catch blocks in LLM client
- Apply Biome formatting and import organization
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migrate configuration validation and core moderation types from individual services to the `@bete/shared` package to ensure consistency across the monorepo.
- Move `AppConfig` and moderation-related interfaces to `packages/shared`.
- Replace service-specific Zod schemas with the centralized shared configuration.
- Refactor `services/backend` and `services/discord-gateway` to consume shared config and types.
- Remove redundant type definitions and local configuration logic in services.
- Update `packages/shared` exports to include new `config` and `moderation-types` modules.
- Clean up unused files and deprecated utility functions in `packages/shared`.
asepharyana
asepharyana
asepharyana
Developer
MythEclipse