perf(ai): kontiguitas batch budget + max_tokens dinamis + urutan kronologis RETURNING
- pickBatchWithinBudget: stop di overflow pertama (break), bukan skip — batch tetap prefix kronologis tanpa gap analisis di tengah timeline. Diekstrak ke batchBudget.ts (pure, estimator di-inject) + regression test. - callModerationLLM: param opsional maxTokens; text/media caller menghitung ceiling dari estimasi prompt (floor 2048, cap 16384) — batch kecil tak lagi reserve window completion 16k. - getPending/IncompleteMessagesByConversation: sort hasil UPDATE..RETURNING by created_at ASC — Postgres tak menjamin urutan, konsumen (anchor konteks messages[0], prefix batch) bergantung pada urutan kronologis.
This commit is contained in:
@@ -49,6 +49,10 @@ export async function callModerationLLM(
|
||||
targetIds: string[],
|
||||
label: string,
|
||||
signal?: AbortSignal,
|
||||
// Output-side token cap. Defaults to the previous hard-coded value; batch
|
||||
// callers pass a prompt-derived ceiling so small batches don't reserve a
|
||||
// 16k completion budget (some routers pre-allocate KV cache per max_tokens).
|
||||
maxTokens?: number,
|
||||
): Promise<{
|
||||
results: AnalysisResult[];
|
||||
raw: ChatCompletion | null;
|
||||
@@ -75,7 +79,7 @@ export async function callModerationLLM(
|
||||
];
|
||||
const completion = await llmChat({
|
||||
messages,
|
||||
max_tokens: 16384,
|
||||
max_tokens: maxTokens ?? 16384,
|
||||
jsonResponse: { type: "json_object" },
|
||||
retries: 0,
|
||||
signal,
|
||||
|
||||
Reference in New Issue
Block a user