fix(gateway): enable stream for all LLM calls — router always streams SSE
Build & Deploy (Nix) / build-and-deploy (proxy) (push) Successful in 3m36s
Build & Deploy (Nix) / build-and-deploy (backend) (push) Successful in 4m9s
Build & Deploy (Nix) / build-and-deploy (discord-gateway) (push) Successful in 11m29s

Audit lanjutan: 6x 'LLM API request failed: Request was aborted' per jam.
Root cause: 9router/omniroute SELALU balas SSE (data: chunks) walau request
tanpa stream:true — SDK OpenAI non-stream menunggu FULL body sebelum parse,
jadi batch moderasi besar yang upstream-nya lambat kena timeout 30-60s dan
di-abort. llmClient sudah punya agregasi streaming (chunks → ChatCompletion).

Fix: stream:true di llmCaller (moderasi batch/individual), llmVision,
cultureLearner, userProfileLearner. Verified: SDK stream test 806ms vs
sebelumnya abort. Caller lain (recovery worker dll) lewat llmCaller sama.
This commit is contained in:
asepharyana
2026-08-01 15:00:59 +07:00
parent dfe689bdec
commit 0ef2b715c4
4 changed files with 10 additions and 0 deletions
@@ -79,6 +79,13 @@ export async function callModerationLLM(
jsonResponse: { type: "json_object" },
retries: 0,
signal,
// Router (9router/omniroute) always streams SSE even when the
// request omits `stream`. In non-stream mode the OpenAI SDK waits
// for the FULL body before parsing, so slow/long upstream streams
// hit the 30s/60s timeout and abort mid-generation. Streaming mode
// consumes chunks incrementally — timeout only fires on a real
// stall. llmClient aggregates the stream into a ChatCompletion.
stream: true,
});
if (!completion)