fix(gateway): separate text and media lanes in AI analysis queue
Image messages previously blocked the whole analysis pipeline:
- conversationProcessing was a single lock per conversation; processBatch
awaited BOTH text and media worker jobs before releasing it, so a fast
text verdict sat unused until the slow vision/media batch finished
- one global LLM semaphore (AI_LLM_MAX_CONCURRENT) was shared by text and
media, so a vision backlog could starve text inference
- recovery worker gated on conversationProcessing.size
Now the queue is split into independent text/media lanes:
- conversationProcessing maps key -> Partial<Record<lane, startedAt>>;
each lane holds its own lock and frees it the moment ITS worker job
resolves (ownership-guarded clear prevents stale timers clearing newer
slots)
- two LLM semaphores: AI_LLM_MAX_CONCURRENT (text, default 8) and
AI_LLM_MEDIA_MAX_CONCURRENT (media, default 4) via
withLlmConcurrency(fn, { lane })
- batchScheduler schedules per conversation+lane (timer keys
'<key>::<lane>'); splitMessagesByLane/laneOfMessage moved to pure
analysisLanes.ts (unit-testable without Piscina)
- ai-analysis-worker batch jobs carry a lane field; per-lane active
request gauges (active_text_requests / active_media_requests)
- added tests/analysisLaneLock.test.ts (7 tests: independent lane locks,
preserving other-lane lock, clear-all, ownership guard, lane split)
Docs: ARCHITECTURE.md + AGENTS.md concurrency model updated.
typecheck/lint/test(138)/build all green.
This commit is contained in:
1 parent
f9fecfc144
commit
6d0d7b34a3
15 files changed
+692
-205
No files matched your search
@@ -46,24 +46,38 @@ services/discord-gateway/
|
||||
## AI moderation pipeline (`ai-moderation/`)
|
||||
|
||||
LLM-only judge — no regex/heuristic classification. One orchestrator call
|
||||
handles a whole batch (text + media split internally, parallel paths).
|
||||
handles a whole batch. **Independent text/media lanes** (2026-09-24): a
|
||||
conversation batch is split into a text lane (messages with no media) and a
|
||||
media lane (attachments/stickers/embeds) that are dispatched to separate
|
||||
pools, hold SEPARATE per-lane processing locks, and run under SEPARATE LLM
|
||||
concurrency semaphores. The text lane frees its lock and saves+broadcasts the
|
||||
moment text analysis finishes — it never waits on a slow vision/media batch
|
||||
of the same conversation, and vice versa.
|
||||
|
||||
- `aiAnalyzer.ts` — public API: `queueMessageAnalysis`, `getAnalysisQueueStatus`,
|
||||
`startPendingAIAnalysisWorker` (recovery worker + cache-prune).
|
||||
- `batchScheduler.ts` — per-conversation debounce → `processBatch`.
|
||||
- `batchProcessor.ts` — batch lock/circuit-breaker, fans failed targets to
|
||||
individual fallback.
|
||||
- `batchScheduler.ts` — per-conversation per-LANE debounce → `processBatch`
|
||||
(lane-aware). `splitMessagesByLane` / `laneOfMessage` live in
|
||||
`analysisLanes.ts` (pure, unit-testable).
|
||||
- `batchProcessor.ts` — per-lane batch lock/circuit-breaker, fans failed
|
||||
targets to individual fallback. `processBatch` releases ITS lane's lock the
|
||||
moment that lane's worker job finishes; the other lane owns its own lock.
|
||||
- `individualFallbackProcessor.ts` — one-message-at-a-time retry path, own CB.
|
||||
- `conversationState.ts` / `circuitBreaker.ts` — per-conversation state,
|
||||
Piscina `workerPool`, `getConversationKey`.
|
||||
- `ai-analysis-worker.ts` — Piscina entry point (`batch` / `individual` jobs).
|
||||
Runs `runModerationAnalysis` off the main thread.
|
||||
- `conversationState.ts` / `circuitBreaker.ts` — per-conversation PER-LANE
|
||||
state (`conversationProcessing` holds a lane → startedAt map per key),
|
||||
Piscina `textWorkerPool`/`mediaWorkerPool`, `getConversationKey`.
|
||||
- `ai-analysis-worker.ts` — Piscina entry point (`batch` (lane) /
|
||||
`individual` jobs). Runs `runModerationAnalysis` off the main thread.
|
||||
- `moderationOrchestrator.ts` — exact-hash cache → batched semantic (Qdrant)
|
||||
cache → LLM. Text and media paths run in parallel.
|
||||
- `textBatchProcessor.ts` / `mediaBatchProcessor.ts` — actual LLM calls
|
||||
(one call per sub-batch, not per message).
|
||||
(one call per sub-batch, not per message). `mediaBatchProcessor` routes its
|
||||
moderation LLM call through the MEDIA semaphore.
|
||||
- `llmClient.ts` — central OpenAI-compatible chat client (streaming, retries,
|
||||
thinking-disable injection). `visionAnalyzer.ts` / `mediaAnalysisClient.ts`
|
||||
thinking-disable injection). TWO concurrency semaphores:
|
||||
`AI_LLM_MAX_CONCURRENT` (text lane, default 8) and
|
||||
`AI_LLM_MEDIA_MAX_CONCURRENT` (media lane, default 4) — a vision backlog
|
||||
can never consume text slots. `visionAnalyzer.ts` / `mediaAnalysisClient.ts`
|
||||
share the same router/base URL (different model alias for vision).
|
||||
- `embeddingClient.ts` + `qdrantClient.ts` — semantic cache (one embed call +
|
||||
one batched Qdrant search for all uncached targets).
|
||||
@@ -72,16 +86,16 @@ handles a whole batch (text + media split internally, parallel paths).
|
||||
|
||||
### Concurrency model
|
||||
|
||||
- Main thread owns the LLM semaphore (`AI_LLM_MAX_CONCURRENT`, default 5) via
|
||||
`llmClient.withLlmConcurrency`.
|
||||
- Main thread owns TWO per-lane LLM semaphores (2026-09-24):
|
||||
`AI_LLM_MAX_CONCURRENT` (text, default 8) and `AI_LLM_MEDIA_MAX_CONCURRENT`
|
||||
(media, default 4) via `llmClient.withLlmConcurrency(fn, { lane })`.
|
||||
- Two Piscina pools run the heavy LLM work off the event loop: a text pool
|
||||
(`PISCINA_MAX_THREADS`, default 4) and a dedicated media pool
|
||||
(`PISCINA_MEDIA_MAX_THREADS`, default 2). A batch is routed to the media
|
||||
pool if ANY of its messages carries an attachment/sticker/embed — this
|
||||
keeps a slow image/vision batch from occupying every thread and blocking
|
||||
unrelated text-only batches behind it. **Each worker thread (in either
|
||||
pool) initializes its own pg Pool** (min 0, grows to `POSTGRES_POOL_MAX`).
|
||||
See "Memory & connections" below.
|
||||
(`PISCINA_MEDIA_MAX_THREADS`, default 2). A batch is routed by lane to the
|
||||
matching pool — this keeps a slow image/vision batch from occupying every
|
||||
thread and blocking unrelated text-only batches behind it. **Each worker
|
||||
thread (in either pool) initializes its own pg Pool** (min 0, grows to
|
||||
`POSTGRES_POOL_MAX`). See "Memory & connections" below.
|
||||
|
||||
## Memory & DB connections
|
||||
|
||||
|
||||
Reference in new issue
Block a user