perf: optimasi 7 hot-path bottleneck (buildKey, SSE, cooldowns, encode) (#6)

1. ResponseCache: short-circuit buildKey() via shouldCacheModel() check
   + WeakMap memoization untuk stableStringify (skip recursive sort
   kalau object reference sudah pernah di-stringify).

2. extractTextFromSSEEvent: ganti regex greedy \{.*\} dengan manual
   indexOf untuk JSON bounds (hindari backtracking per SSE chunk).

3. ProxyPool: bound cooldowns Map ke MAX_COOLDOWNS=10_000 dengan
   insertion-order LRU eviction — mencegah memory growth kalau
   banyak unique (proxy, model) pairs kena 429.

4. Single JSON.stringify: hitung responseBody sekali, pakai untuk
   cache.set dan Response constructor (sebelumnya stringified 2x).

5. Shared SHARED_ENCODER singleton: TextEncoder stateless, share
   module-level. Decoder tetap per-stream (stateful).

6. Branch DSML early: skip extractTextFromSSEEvent + JSON.parse
   kalau isDSMLDetectionEnabled() === false (untuk non-DeepSeek model).

7. safeReleaseReader() idempotent guard: gunakan readerReleased flag
   untuk mencegah double-delete di ACTIVE_READERS kalau exception
   terjadi di tengah stream cleanup.

274 tests pass, no regression. Estimated total saving: 5-15ms/req
untuk model non-cached + 30-150ms per streaming response.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Asep Haryana Saputra
2026-06-27 17:46:51 +07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 030c4f884b
commit 173fa68025
5 changed files with 176 additions and 34 deletions
+25 -1
View File
@@ -63,7 +63,7 @@ export class ResponseCache {
/** Build a deterministic cache key from an LLM request. */
static buildKey(model: string, messages: unknown, stream: boolean): string {
const stable = JSON.stringify(messages, stableStringifyReplacer);
const stable = stableStringifyCached(messages);
const raw = `${model}|${stream}|${stable}`;
return simpleHash(raw);
}
@@ -203,6 +203,30 @@ function stableStringifyReplacer(_key: string, value: unknown): unknown {
return value;
}
/**
* WeakMap cache for stableStringify output — avoids re-stringifying the same
* request body across retries or repeated identical requests within the same
* process lifetime. Keyed by the top-level messages reference; entries are
* collected once the request body is GC'd.
*/
const _stableStringifyCache = new WeakMap<object, string>();
/**
* Memoized stable JSON.stringify for buildKey. Re-using the same `messages`
* reference across retries hits the WeakMap and skips the recursive sort.
* Falls back to direct stringify when the value isn't a referenceable object.
*/
function stableStringifyCached(value: unknown): string {
if (value && typeof value === "object") {
const hit = _stableStringifyCache.get(value as object);
if (hit !== undefined) return hit;
const s = JSON.stringify(value, stableStringifyReplacer);
_stableStringifyCache.set(value as object, s);
return s;
}
return JSON.stringify(value, stableStringifyReplacer);
}
/**
* Simple, fast non-cryptographic hash (djb2 variant).
* Collisions are theoretically possible but extremely unlikely for