Compare commits

52 Commits
Author SHA1 Message Date
asepharyana 9f7ce7dbd5 fix(gateway): hash full image data URL for cache key to prevent collision
Root cause: makeImageCacheKey() only hashed the first 128 chars of the
data URL. Since all resized images use the same MIME prefix
('data:image/png;base64,') + identical base64 header bytes, nearly every
image got the same 16-char hash → 'image:<same-hash>' → all images reused
the first cached vision analysis (often a gambling-detection verdict).

Fix: hash the entire data URL instead of just the prefix. Verified
114 stale 'image:' entries + 745 stale 'phash:' entries purged from prod
DB. tsc --noEmit clean, 133 tests pass.
2026-08-12 18:28:19 +07:00
asepharyana bd292fdf3d feat(gateway): Invidious fallback for YouTube 403 in screen share
YouTube blocks anon + cookies terbind ke IP browser (403 download).
Auto-rewrite youtube.com -> yewtu.be/invidious mirror saat cookies gagal.
- mediaSource: export isYoutubeWatchUrl/toInvidiousUrl/INVIDIOUS_INSTANCES
- screenShareController: resolveInputWithRetry tries Invidious instances on 403
2026-08-12 18:19:04 +07:00
asepharyana 7a7f433988 feat(gateway): read YouTube cookies from BWS env (gmw_yt_downloader_cookies) fallback to on-disk file
bws-exec exposes the BWS secret as env GMW_YT_DOWNLOADER_COOKIES.
Materialize to temp Netscape file (yt-dlp --cookies needs a path).
Falls back to /etc/gmw-discord-gateway/ytcookies.txt written by deploy.
2026-08-12 17:50:07 +07:00
asepharyana 84c5c36672 feat(gateway): YouTube cookies support for yt-dlp screen share + music
YouTube now blocks anonymous embeds (403 'Sign in to confirm you're not a
bot'). Resolve with account cookies via --cookies.

- mediaSource: buildCookieArgs() reads GMW_YT_COOKIES_PATH (default
  /etc/gmw-discord-gateway/ytcookies.txt) and injects --cookies into
  resolveMediaUrl + getDirectScreenInput + extractMediaInfo. Falls back
  to anon if file missing (graceful 403, not crash).
- bws-exec now writes cookies file from BWS secret gmw_yt_downloader_cookies
  on service start (systemd ConfigFile).
2026-08-12 17:46:05 +07:00
asepharyana c5898f7cf0 fix(gateway): crash safety on screen-share input timeout + proper error serialization
Root cause of "langsung left": YouTube bot-block/403 on u_c1tRmj7E4 (live
stream, LOGIN_REQUIRED) made yt-dlp timeout in resolveInputWithRetry (12s).
The timeout handler did cleanup() (removing once() listeners) THEN
tee.destroy(new Error(...)) — the PassThrough emitted 'error' with NO
listener left → unhandled stream 'error' event → uncaughtException →
gracefulShutdown → bot left voice.

Fix:
- resolveInputWithRetry: tee.destroy() silently after cleanup (error carried
  in the rejection only); add permanent no-op tee.on('error') safety.
- prepareStream: output.on('error') no-op so ffmpeg spawn failure before
  playStream attaches a demux listener never crashes the gateway.
- bootstrap: serialize uncaughtException/ClientError/DB errors with
  {err, errorMsg, stack} (pino only serializes the 'err' magic key — the old
  {error: err} key printed {} so crashes were invisible).
2026-08-12 16:59:41 +07:00
asepharyana 354e378e74 fix(gateway): output NUT (not raw h264) so Demuxer re-splits video+audio correctly
Revert 392bc35: streaming raw h264 video + opus on separate pipes broke
because prepareStream.output (pipe:1) feeds the Demuxer, but the opus
pipe:3 was never attached to the Demuxer's input — so for audio-capable
streams the Demuxer saw format=h264 (video-only) and emitted -an,
dropping audio RTP.

Correct design (from f1aa08c): prepareStream muxes video+audio into NUT
on a SINGLE pipe:1. The Demuxer then spawns a child ffmpeg that
demuxes NUT → -f h264 pipe:1 (pure AnnexB, start-code scan sees real
IDR type 5) + -f opus pipe:3 (Ogg Opus via createOggOpusDemux). The
start-code parser never touches NUT framing — it runs on the child
ffmpeg's clean h264 stdout.
2026-08-12 16:09:34 +07:00
asepharyana 392bc35a0d fix(gateway): output raw H264+Opus (not NUT) so Demuxer parses NAL keyframes correctly
Root cause: prepareStream muxed video+audio into a NUT container on pipe:1.
The Demuxer scans pipe:1 for AnnexB start codes (00 00 01) to split NAL
units into access units and classify keyframes (nal_type 5). NUT container
framing bytes sat in the stream and were scanned as NALs — NAL type 0
(NUT header) instead of 5 (IDR) → every frame classified key=false →
Discord decoder never got a decodable frame → static/black GoLive tile.

Fix: output raw H264 AnnexB on pipe:1 (demuxer target) and Ogg Opus on
fd3/pipe:3 for audio. NUT is only needed for *input* parsing (single
pipe carries both streams); output is demuxed into separate raw streams.
2026-08-12 15:39:47 +07:00
asepharyana 196cb1d3af fix(gateway): await audio stream line before demux resolve — audio RTP was dropped by metadata race
The demuxer resolved as soon as the VIDEO init line arrived on ffmpeg stderr.
With live NUT input the audio init line ('Stream #0:1: Audio: opus') lands in a
LATER stderr chunk (NUT info-stream packets are read incrementally from the
pipe), so `return { audio: aInfo }` captured undefined → playStream skipped
AudioStream → zero audio RTP on the audio SSRC → Discord showed a static
GoLive tile even though the NUT carried opus audio.

Fix:
- wait for BOTH video and audio init lines (when audio is expected) before
  resolving demux metadata, with a 3s timeout fallback
- default aInfo to opus/48kHz when withAudio instead of undefined, so the
  audio stream is always exposed even if the metadata line races the return
2026-08-12 14:45:06 +07:00
asepharyana 00fc852a32 feat(rtp-capture): add two-peer RTP capture test for H264 frame transmission 2026-08-12 14:26:54 +07:00
asepharyana d9f5592e6e feat(glossary): persist resolved definitions in Postgres + harden live SearXNG lookups
- Add term_glossary_cache table + migration 0014: resolved definitions are
  stored permanently (definitions rarely change); misses stay ephemeral in
  Redis/LRU with 1h TTL so transient failures get retried
- Lookup flow: LRU -> Redis -> Postgres (permanent) -> live SearXNG; DB hits
  re-warm the fast caches; stale Redis miss sentinels no longer shadow DB
- Rate-limit-aware live lookups: concurrency 2 + stagger, retry once on empty
  results, strict definition filter (Wikipedia preferred, rejects
  disambiguation/ads/translate-homepages)
- Make SEARXNG_BASE_URL configurable via env (default unchanged)
2026-08-12 14:22:22 +07:00
asepharyana f1aa08cdf6 fix(gateway): deliver audio + per-IDR SPS/PPS in GoLive screen share
Screen share showed a single frozen frame: the GoLive pipeline sent video
only (-"-an", h264 muxer cannot carry audio) so the audio SSRC never
transmitted and Discord kept the stream in thumbnail state.

- prepareStream: mux NUT when includeAudio (h264 muxer drops audio) and
  return the actual container format
- Demuxer: support NUT input with a second output pipe (fd3) carrying
  Ogg Opus; parse OGG pages into opus frames (20ms, 48kHz) emitted as
  GoLiveFrames; fix metadata parsing that dropped the audio stream line
  when it arrived in a later stderr chunk (early parsedMeta return)
- playStream: pipe audio.stream into AudioStream → RTP on the audio SSRC
- Encoders: -x264-params repeat-headers=1 → SPS/PPS inline before EVERY
  IDR (NUT remux drops container extradata; also enables PLI recovery)
- screenShareController: includeAudio true
- tests: demuxerNut.test.ts — OGG parser unit test + real ffmpeg NUT
  integration (video access units + parsed opus frames)
2026-08-12 14:20:29 +07:00
asepharyana f70a92880e feat(glossary): implement term glossary for LLM moderation with caching and extraction logic 2026-08-12 13:44:52 +07:00
asepharyana 88b13225cd fix(gateway): stream screen-share input from yt-dlp stdout — no more raw-URL 403
Second root cause (2026-08-12): even with yt-dlp http_headers forwarded,
YouTube still returns 403 when a signed DASH URL from --dump-single-json is
fetched raw by ffmpeg/curl on some videos (verified on fONoh7Pc6VU: curl
with the EXACT headers got 403; yt-dlp's own downloader succeeded). The
signature is tied to the extracting client context (po_token/visitor), not
just UA/IP.

Fix: getDirectScreenInput now spawns 'yt-dlp -o -' and returns its stdout
as a Readable — the same mechanism resolveMediaUrl already uses for music.
yt-dlp handles auth, cookies and transient retries internally. Merge
fragments go to /tmp/gmw-ytdlp-tmp (Nix store CWD is read-only → EACCES).
Removed resolveScreenInput + mergeScreenStreams (dead code).

Controller resolveInputWithRetry unchanged: tees the stream, waits for the
first byte (12s), retries with a fresh yt-dlp run up to 3x on error/EOF/
timeout, and destroys stuck inputs (EPIPE) so no process leaks.

Tests: rewritten for streaming (yt-dlp emits bytes; fail mode = exit 8
without stdout → stream must terminate with zero bytes).
2026-08-12 13:23:59 +07:00
asepharyana 67ab289caa fix(gateway): fail-fast + retry on screen share merge failure (black tile zombie)
Root cause (2026-08-12 11:50 test): merge ffmpeg hit a transient YouTube
403 and exited code 8 BEFORE prepareStream attached its input listeners
(voice release+join takes ~10s). The input's end/error events fired into
the void, the encoder stdin never received EOF, demux resolved with
fallback 0x0 metadata, setSpeaking fired anyway → stream 'started' with
zero frames for 8+ minutes (black tile, both ffmpeg processes hung).

Fixes:
- mediaSource: pass yt-dlp http_headers (UA/referer) to the merge ffmpeg
  via -headers to suppress transient 403s; destroy the returned stream
  with an error when the merge exits non-zero before producing bytes.
- screenShareController: resolveInputWithRetry — tee the merge stream and
  wait for the first readable byte (12s timeout) before proceeding; on
  error/EOF/timeout retry the whole resolution with a FRESH yt-dlp run
  (signed DASH URLs expire fast) up to 3 attempts. Stuck merges get
  EPIPE via input.destroy() so no process leaks per attempt.
- prepareStream: race guard — if the input already ended/destroyed before
  listeners attach, EOF the encoder stdin immediately; first-frame
  watchdog in playStream rejects 'started but nothing flowing' after 10s
  instead of resolving with a silent black stream.

Tests: +2 (merge-fail zero-byte terminal state, -headers forwarding).
2026-08-12 12:17:43 +07:00
asepharyana ef9e243609 fix(goLive): encode H264 baseline to match SDP profile-level-id (black tile)
SDP offer advertises profile-level-id=42e01f (constrained baseline) but
x264 encoded the default High profile — Discord's receiver configures its
decoder from the negotiated profile, so the High-profile bitstream failed
to decode → black GoLive tile despite valid access units + correct RTP
timestamps (fixed in 42a503c).

- Add -profile:v baseline to H264 encoder options (matches @dank074's
  proven config; SPS now 6742c01e → profile_idc=66 baseline, aligns with
  the 42e01f fmtp).
- Default x264 tune film → zerolatency (no lookahead — correct for live
  GoLive; @dank074 uses it).
- Update goLive-port test to assert baseline + zerolatency.
2026-08-12 11:37:04 +07:00
asepharyana 42a503c206 fix(goLive): demux access-unit grouping + correct RTP timestamps (black tile)
Demuxer emitted each AnnexB NAL as its own WebRTC frame (SPS/PPS/SEI
separate from slices) with a near-zero timestamp delta (duration=1 in a
1/90000 timebase → RTP +1/frame instead of +3000 @30fps). Discord's H264
receiver never receives a complete decodable access unit → black GoLive
tile despite frames flowing.

- Group NALs into access units: buffer param-set/SEI NALs, flush one
  frame per slice with preceding parameter sets (AnnexB start codes kept
  so the H264RtpPacketizer finds NAL boundaries).
- Timestamp each frame at the video frame rate: duration=1, timeBase
  1/fps → BaseMediaStream frametime=1000/fps ms → RTP +clockRate/fps
  (3000 @ 30fps/90kHz) and correct pacing.
- Thread explicit frameRate from playStream options (raw H264 has no
  timing info; ffmpeg guesses 25fps on stderr).
- Strengthen golive-demux-live-e2e: validates every frame has a slice,
  no bare param-set frames, keyframes carry SPS/PPS, timeBase 1/30.
2026-08-12 11:13:00 +07:00
asepharyana 652974e23a fix(goLive): gateway crash on screenshare stop — unhandledRejection during teardown
Test 00:32 confirmed the video pipeline WORKS (1410 frames @ 1280x720 sent,
ready=true, camera off) but the gateway crashed at stream stop:
unhandledRejection → graceful shutdown → systemd restart (bot offline).

Root cause candidates (both were fire-and-forget promises without .catch):
- BaseMediaConnection.setProtocols().then(...) — rejects when the PC is
  closed while setProtocols is in flight (stream teardown)
- void webRtcConn.createOffer().then(...) — rejects when the PC closes
  while the offer is still gathering

Fixes:
- .catch on both promise chains (log + continue; teardown is expected)
- unhandledRejection handler now treats transient stream errors (EPIPE,
  ERR_STREAM_DESTROYED, ERR_STREAM_WRITE_AFTER_END, ECONNRESET) like
  uncaughtException already does — warn + continue instead of shutting
  down the whole gateway. Non-transient rejections still log + shutdown
  (with String(reason) so the detail actually shows).
2026-08-12 00:36:53 +07:00
asepharyana 407e003399 fix(goLive): black screen root cause — h264 muxer can't carry audio; disable self_video camera
ROOT CAUSE of empty GoLive tile (finally): prepareStream ran with
includeAudio: true + output -f h264. The h264 muxer cannot mux audio
('h264 muxer does not support any stream of type audio') → header write
fails -22 → stdout empty → Demuxer ffmpeg 'Invalid data found when
processing input' → 0 frames → black tile. Reproduced locally end-to-end
(13s backpressure delay + prepareStream + demux).

Fixes:
- screenShareController: includeAudio: false (video-only GoLive; demux
  path never delivers audio anyway)
- Demuxer: pin input format -f h264 for stream inputs (raw AnnexB H264
  has no magic header → auto-detect unreliable on delayed pipes)
- Streamer.signalStream: self_video: false — stop flipping on the bot's
  camera in Discord (user request; screen share ≠ camera)

Verified: local repro now emits 644 frames 1280x720 (was 0); tsc/biome/
vitest all green.
2026-08-12 00:25:01 +07:00
asepharyana 968a43b0f4 debug(goLive): instrument frame pipeline — demux spawn/stderr/frames, playStream resolve, sendVideoFrame drop/send
Tile kosong meski STREAM_CREATE handshake penuh (22:18-22:19 retest):
- Demuxer logs spawn args, ffmpeg stderr errors, frame count every 30
- playStream logs createStream resolved + demux done + setPacketizer
- sendVideoFrame logs DROPPED (ready/track) + sent frame count
2026-08-11 22:31:31 +07:00
asepharyana 91c7a67d2f fix(goLive): stream demux directly instead of spool-to-file (empty screen share)
Root cause of 'tile appears but content empty': demux() spooled the live
NUT/H264 input to a temp file and awaited stream 'finish' — but the merge
ffmpeg output never ends during playback, so demux deadlocked, no probe,
no transcode, 0 frames sent.

- Demuxer: pipe input straight into ffmpeg stdin (-i pipe:0), parse NAL
  frames live from stdout; parse video metadata from ffmpeg stderr with a
  1.5s race (fall back to H264 defaults). No spool, no await-end.
- screenShareController: pass width/height/frameRate (1280x720@30) to
  playStream — matches the prepareStream encode settings, so setVideoAttributes
  gets real dimensions even when ffmpeg can't report metadata on an open pipe.
- Add tests/golive-demux-live-e2e.ts: proves frames flow while input is
  still open (regression test for the deadlock).
2026-08-11 21:43:16 +07:00
asepharyana 8615383829 fix(goLive): STREAM_CREATE handshake — self_video voice state + retry + send instrumentation
- signalStream: flip voice state to self_video:true/self_deaf:false before
  STREAM_CREATE (Discord silently ignores the request while video disabled)
- createStream: attach dispatch listeners before first signal (race), clean
  up listeners on timeout, retry STREAM_CREATE every 3s up to 4 attempts
  (upstream issue #217/#219 — Discord randomly drops the request)
- sendOpcode: direct [goLive:Streamer] log bypassing bootstrap debug filter
  (proves op 18 is actually broadcast)
2026-08-11 20:45:00 +07:00
asepharyana 10d7ecd405 fix(gateway): EPIPE crash on media stop — stream error handlers + no shutdown on transient stream errors 2026-08-11 20:10:05 +07:00
asepharyana ff554fcff2 fix(goLive): instrument voice/stream handshake + createStream timeout (12s) 2026-08-11 20:00:33 +07:00
asepharyana f8b253ba5e merge: libdatachannel-min GoLive stack (build -86pct, node_modules -1.09GB) 2026-08-11 19:07:57 +07:00
asepharyana c8473b0610 build(nix): fix binding link path — LDC_LIB is full .so path
binding.gyp appended '/libdatachannel.so.0.24.0' to LDC_LIB; nixpkgs output
layout is <out>/lib/libdatachannel.so.0.24.1. Make LDC_LIB the complete
library path (env or default) and drop the append.
2026-08-11 18:54:12 +07:00
asepharyana 3deca91ffe build(nix): use nixpkgs libdatachannel (no cmake/fetchFromGitHub)
libdatachannel-src fetchFromGitHub + manual cmake build fails: GitHub tarball
does not include git submodules (deps/plog, libjuice, libsrtp, usrsctp) →
CMake 'source directory does not contain CMakeLists.txt'.

Switch to pkgs.libdatachannel (0.24.1): nixpkgs builds submodules + ships
lib/dev outputs. In the Nix sandbox everything is consistent (store glibc),
so the GLIBC_ABI_GNU2_TLS issue that blocks host-local use of 0.24.1 does
not apply to the Nix build. binding.gyp defaults stay on local 0.24.0 for
dev; Nix sets LDC_INCLUDE/LDC_LIB to the store paths.
2026-08-11 18:45:51 +07:00
asepharyana edec2edf82 build(nix): binding — NAPI_INCLUDE from pnpm store (depth 3), gyp env fallback 2026-08-11 18:35:49 +07:00
asepharyana 17013fe1e5 build(nix): gateway binding — gyp env-var paths, correct cwd, tolerant install
- binding.gyp: resolve libdatachannel include/.so via LDC_INCLUDE/LDC_LIB env
  (node -e expression) instead of hardcoded /tmp/ldc-build paths
- flake buildPhase: run node-gyp from native/libdatachannel-min root (was
  build/ subdir → 'binding.gyp not found'); export LDC_INCLUDE (fetchFromGitHub
  source) + LDC_LIB (cmake build dir)
- flake installPhase: tolerate missing binding (screen share disabled, gateway
  still starts); copy .so real files via -rL
2026-08-11 18:32:09 +07:00
asepharyana 3acb03391a build(nix): gateway flake — build libdatachannel-min binding, drop datachannel/node-av/zeromq
- Replace the per-package rebuild loop (node-datachannel cmake-js, zeromq)
  with: opus build + libdatachannel-min N-API binding build (fetchFromGitHub
  libdatachannel v0.24.0 — pinned because nixpkgs 0.24.1 is glibc-incompatible
  with this host; sha256 1jk53qs…).
- Removes ~760MB of node-datachannel build/cleanup cruft from the build
  phase; node_modules now 423MB (was 1.5GB).
2026-08-11 17:53:47 +07:00
asepharyana 9109d3c898 perf(golive): drop @dank074/discord-video-stream — node_modules 1.5GB → 423MB
Remove the last heavy GoLive dependency now that src/goLive/ replaces it:
- @dank074/discord-video-stream (pulled in @lng2004/node-datachannel
  771MB, node-av 118MB + @seydx/node-av-linux-x64 167MB, zeromq 21MB,
  fluent-ffmpeg 13MB — ~1.09GB total)
- onlyBuiltDependencies: drop node-av/zeromq/@lng2004 (keep opus/esbuild/sharp)
- pnpm.lock regenerated; orphan .pnpm dirs removed locally
- @discordjs/opus prebuild: rebuilt binary copied into
  prebuild/node-v127-napi-v3-linux-x64-glibc-2.39/ (node-pre-gyp find path)

Verified: tsc 0 errors, vitest 8/8, biome clean, opus encode OK.
Fresh CI install now ~423MB instead of ~1.5GB.
2026-08-11 17:49:33 +07:00
asepharyana 9139e225f4 perf(golive): ffmpeg-spawn demuxer (no node-av) + E2E pipeline tests
Phase 2 — replace the 114MB node-av binary with a plain ffmpeg spawn:

Demuxer.ts: spool stream input to temp file → probe via ffmpeg stderr
(ffmpeg-headless ships NO ffprobe — parse 'Stream #0:0: Video: h264...
640x360, 30 fps' from -loglevel info) → ffmpeg -c copy -f h264 pipe:1
→ NAL-split frames. Falls back to h264 defaults when probe fails.

prepareStream.ts: resolve ffmpeg from FFMPEG_PATH env → Nix store
ffmpeg-headless (hash-prefixed entry!) → PATH; split encoder option
strings ('-forced-idr 1' → two argv) — fluent-ffmpeg used to split
automatically, spawn does not.

E2E tests (tsx, need LD_LIBRARY_PATH=/tmp/ldc-build):
- golive-demux-e2e.ts: real H264 file → 33 NAL frames + dims from probe
- golive-pipeline-e2e.ts: prepareStream → demux → 82 frames
- golive-videostream-e2e.ts: local peer pair → demux → VideoStream →
  native setPacketizer/sendFrame/addTimestamp → 33 frames sent connected

Pitfalls captured: setPacketizer before negotiation breaks createOffer
('No DataChannel or Track to negotiate'); track methods are read-only
(no monkeypatching); both peers must declare audio+video tracks or
answer hangs; state() returns 'closed' after close() — snapshot first.
2026-08-11 17:38:06 +07:00
asepharyana 9ae230d047 perf(golive/spike): binding addTrack + TS port of @dank074 media stack
Phase 1 spike: replace @dank074/discord-video-stream + node-datachannel +
node-av (1.3GB) with minimal libdatachannel N-API binding + native RTP
packetizers (H264 FU-A, RTCP SR/NACK, pacer) + pure-TS GoLive stack.

Binding v0.4: addTrack (m=audio/video SDP), TrackWrap w/ setPacketizer +
sendFrame (raw RTP to transport) + addTimestamp — verified by two-peer
handshake emitting SDP with audio(opus 120)+video(H264 101) and 8-frame
RTP roundtrip.

TS layer (src/goLive/, 21 files): CodecPayloadType, VoiceOpCodes,
GatewayOpCodes, utils, BaseMediaConnection (voice WS + DAVE + heartbeat),
VoiceConnection, StreamConnection, Streamer, WebRtcWrapper (SDP mungling,
DAVE encrypt, packetizer chain), BaseMediaStream (pacing/sync), VideoStream,
AudioStream, Demuxer (ffmpeg-spawn NUT/AnnexB, no node-av 114M binary),
Encoders, prepareStream/playStream.

Integration: screenShareController.ts now imports from ../../goLive/index.js —
prepareStream(prepared, ...) + playStream(prepared, streamer, {...}).

Tests: tests/goLive-port.test.ts (8/8 pass). tsc --noEmit clean. biome clean.
2026-08-11 16:16:52 +07:00
asepharyana a1a6d8b418 spike: expose libdatachannel media packetizer chain via Track
Track.setPacketizer(kind, ssrc, pt, clockRate, ...) builds the same
media-handler chain node-datachannel does for @dank074:
  RtpPacketizer (Opus | H264 | H265 | AV1) → RtcpSrReporter →
  RtcpNackResponder → PacingHandler(25Mbps, 1ms) for video
Track.sendFrame(encodedFrame) packetizes into RTP; addTimestamp(delta)
advances the RTP timestamp (node-datachannel contract).

Verified test-packetizer.js: two peers connected over tracks, real opus
frames + AnnexB H264 (SPS/PPS/IDR) flow through the chain without crash.
This removes the need for a JS RTP packetizer entirely — libdatachannel
0.24 has the full media stack built in.
2026-08-11 14:57:04 +07:00
asepharyana 4f06c30c05 spike: add addTrack + TrackWrap to libdatachannel-min binding
Expose rtc::Track with send(binary) for raw RTP — verified:
- SDP from addTrack(audio)+addTrack(video) has m=audio (opus 120)
  and m=video (H264 101 + H265/VP8/VP9/AV1 + RTX)
- libdatachannel Track::send() sends RAW RTP/RTCP when no media
  handler is set (verified in src/track.cpp impl::Track::outgoing) —
  so RTP packetization can live in pure JS, keeping the binding minimal

Also fix: Track class was missing from InitAll exports (crash on
TrackWrap::NewInstance — null FunctionReference).
2026-08-11 14:51:56 +07:00
asepharyana 2203dd5771 spike: minimal N-API libdatachannel binding — WebRTC handshake proven
Phase 0 of GoLive rewrite (drop @dank074/node-datachannel 771MB):
minimal N-API binding exposing PeerConnection/DataChannel/ICE/SDP,
built against libdatachannel 0.24.0 (from node-datachannel _deps source).

Verified: offer/answer/ICE/DataChannel roundtrip between two local
peers (test-handshake.js). Key findings:
- callbacks must be registered in ctor BEFORE createDataChannel
- SDP with candidates comes from localDescription() at gathering Complete
- answer auto-generates on setRemoteDescription(offer); do NOT call
  setLocalDescription() after or role=actpass breaks the peer
2026-08-11 14:23:31 +07:00
asepharyana a53d7b71da fix(voice): screen share GoLive died instantly — neutralize node-av custom ffmpeg filters
prepareStream (from @dank074/discord-video-stream) unconditionally appends
audio filters 'volume@internal_lib' + 'azmq' that exist ONLY in its custom
node-av jellyfin-ffmpeg build. The Nix deployment runs plain ffmpeg-headless
on PATH, so fluent-ffmpeg died instantly with 'Filter not found' (exit 8),
the NUT output stream stayed empty, and playStream's node-av demux failed
with 'Failed to open input from Readable stream: Invalid data found when
processing input' — every screen share failed ~100ms after start.

Fix: pass customFfmpegFlags ['-filter:a','anull'] — ffmpeg applies the LAST
-filter:a for a stream, so the trailing no-op filter overrides the custom
chain (verified: command ends with '-filter:a anull', transcode runs, node-av
demux finds video+audio). Realtime volume control was already removed from
GMW (a690e5b), so dropping the filters is lossless.

Verified end-to-end with the failing URL (youtu.be/fONoh7Pc6VU, AV1+Opus
DASH): getDirectScreenInput → NUT merge → patched prepareStream → node-av
demux finds H264 video + Opus audio streams.
2026-08-11 10:56:49 +07:00
asepharyana c18431bdbf fix(ai-moderation): never cache vision outputs that claim 'no image seen'
Root cause (3rd layer after 50371bd + 4f4c435): a vision model run
(2026-08-10) returned 'Maaf, saya tidak melihat gambar apapun yang terlampir...'
and that text was cached as a VALID vision_llm result (image + phash keys,
24h/7d TTL). Every subsequent analysis of the same image (same hash/phash)
hit the poisoned cache, so image analysis looked broken forever even though
9router responded fine — the moderation LLM wrote 'lampiran yang gagal
terbaca' from a cache hit.

Also: mimo via 9router streams reasoning in delta.reasoning +
delta.reasoning_details[].text (content:"") — extractChunkText only read
delta.reasoning_content, so those runs aggregated empty → 'Vision API null
response' (observed 08:54/09:07/09:38).

Fixes:
- llmClient.extractChunkText: fall back to delta.reasoning and
  reasoning_details[].text (mimo), on top of reasoning_content (gemma).
- visionAnalyzer: isNoImageSeenText() detects 'no image' style outputs;
  such results are NEVER cached, and poisoned entries are purged when hit
  (LRU/DB/phash) so re-analysis actually re-runs vision.
- Tests: reasoning/reasoning_details extraction + isNoImageSeenText
  (Indonesian + English, no false positives on real descriptions).
2026-08-11 09:55:43 +07:00
asepharyana 4f4c43555f fix(ai-moderation): attachment-upload race dropped images before vision
Root cause (2nd layer after 50371bd): the analysis worker could pick up an
image message while its attachment upload was still in flight
(upload_status='pending'). downloadAndExtractFrame then fell back to the
Discord CDN URL (cdn.discordapp.com), which often 404s for old/purged links,
and 'if (!res.ok) return' silently dropped the image — no log, no vision
call, empty image map, and the LLM produced a text-only verdict like
'lampiran yang gagal terbaca oleh sistem'.

Fixes:
- ai-analysis-worker: skip targets whose attachment upload is still pending
  (both batch + individual paths) — they stay ai_status='pending' and the
  next 15s cycle analyzes them after the upload lands.
- mediaDownloader.downloadAndExtractFrame: try uploaded_url first, then
  discord_url as fallback; log non-OK responses (status + host) instead of
  silently returning; log when all candidate URLs fail.
2026-08-11 09:44:34 +07:00
asepharyana 50371bd2d1 fix(ai-moderation): read delta.reasoning_content in stream aggregation — image vision never returned text
Root cause: 9router combo 'multimodal' routes to cloudflare-ai/@cf/google/
gemma-4-26b-a4b-it which streams ALL output in delta.reasoning_content
(content:"") and finishes with 'length' at max_tokens. llmClient only read
delta.content, so llmVision returned empty → every image moderation fell back
to text-only analysis ('Meskipun analisis gambar gagal' in every ai_analysis).

Fix: extractChunkText() prefers delta.content then falls back to
delta.reasoning_content (also handles message/text/response fields), with
unit tests for the exact 9router chunk shape. Verified live against a real
DB image: oc/mimo-v2.5-free (new first model in the multimodal combo) returns
a proper description in delta.content.
2026-08-11 08:06:28 +07:00
asepharyana 0792ff4dc0 perf(nix): prune devDependencies from shipped node_modules
gateway output 1.4G -> 424M (-70%), backend 198M -> 60M (-70%).

- pruneProd: delete every .pnpm dir not in 'pnpm list --prod' graph
  (biome/typescript/esbuild/drizzle-kit/vitest/tsx ~150MB+) then drop
  dangling symlinks (top-level, scoped dirs, hoist, .bin) so stdenv
  noBrokenSymlinks fixup passes.
- NOT using 'pnpm install --prod': it collapses the public-hoist dir
  (.pnpm/node_modules) that peer resolution relies on for
  @lng2004/node-datachannel + @seydx/node-av-linux-x64 (voice breaks).
- node-datachannel: strip build/_deps (cmake FetchContent ~380MB) +
  nested node_modules (nw-gyp/typescript/puppeteer ~380MB) after
  compile; runtime needs only build/Release/node_datachannel.node.
- verified: native binaries (datachannel/opus/zeromq) intact, all 27
  runtime modules resolve, dev tools 0.
2026-08-10 22:17:38 +07:00
asepharyana eb89bb79ed ci(deploy): fix attic push fallbacks - VPS-hop sudo, direct push retry, ssh URL
- VPS-hop attic push now runs via sudo so attic reads root's config
  (~/.config/attic) which has the imrnes-ts server (Tailscale). Without
  it the push ran as the CI user whose config only has pub ->
  'Server imrnes-ts does not exist', silently skipping the cache upload.
- Direct push retried 3x (attic push is idempotent): a transient 502
  (e.g. atticd restart mid-push, Traefik blip) no longer aborts the
  whole closure upload before falling back to VPS-hop.
- Restore $VPS_USER in the 3 ssh:// nix copy fallbacks (was committed
  as masked '***' -> nix copy would ssh as user '***' and fail).
2026-08-10 21:18:25 +07:00
asepharyana 7d6c741bb2 ci(deploy): force narinfo write with --ignore-upstream-cache-filter; Traefik readTimeout=0 on imrnes 2026-08-10 20:18:56 +07:00
asepharyana 4cb4904517 ci(deploy): fix attic fast path - public cache, extra-substituters, self-hosted client bootstrap
Root causes found by reproducing the 2026-08-10 run:
- attic 'gmw' cache was created private -> every narinfo/nix-cache-info
  read returned 401, so the VPS could never actually substitute from
  attic ('Substituted from Attic cache' was a false positive when the
  store path happened to be already present locally).
- VPS nix.conf used extra-trusted-substituters, which Determinate Nix
  never merges for nix-store CLI clients; extra-substituters (all
  users, no trust gate) fixes substitution (verified end-to-end:
  delete path -> nix-store --realise pulls from attic over HTTPS).
- runner bootstrap of the attic client depended on nix copy --from
  ssh:// (fragile, failed on runner); now the prebuilt attic client
  closure lives in the attic cache itself and the runner pulls it over
  HTTPS via extra-substituters configured in the Install Nix step.

Also surfaces bootstrap stderr on fallback for future debugging.
2026-08-10 18:29:00 +07:00
asepharyana 4ee295bd29 ci(deploy): push to attic directly from runner, skip slow SSH closure copy
The old Push-to-Attic step SSH-copied the full closure (~794MB gateway) to
the VPS on every new store path before attic push — at ~500KB/s that took
25+ minutes (observed 40min+ in-flight run). The runner can now push
straight to the public attic endpoint (https://attic.asepharyana.my.id,
token auth validated) after pulling the prebuilt attic client closure
(52MB) from the VPS via nix copy --from. Falls back to the VPS-hop flow
whenever the direct path fails.
2026-08-10 17:28:28 +07:00
asepharyana 65c9c2cd9e feat(ai-moderation): enrich analysis context with recency, repetition, user history and channel topic
- <message> targets now carry time (ISO), repetitions (N identical short texts = spam signal), bot and edited flags; escape id/user XML
- rich <user_reputation>: total_infractions, clean_streak, last_offense_days_ago, repeat_offender (7-day window)
- <user_history> with last flagged messages for repeat offenders (wires dead getUserRecentInfractions)
- <user_profile as_of> staleness signal; <location_context topic> from captured channel topic
- prompt framing + output instructions teach the LLM to use the new signals without treating history as proof
- tests: contextEnrichment.test.ts (13) + topic cases in conversationContext.test.ts
2026-08-10 17:15:33 +07:00
asepharyana 0a5254bf20 feat(ai-moderation): enhance context handling with structured XML blocks and user profiles 2026-08-10 16:46:55 +07:00
asepharyana 4a51f3055c ci(deploy): push builds to attic binary cache (attic.asepharyana.my.id) 2026-08-10 15:27:59 +07:00
asepharyana 185d81f0e0 feat(ai-moderation): reset offensive nickname instead of deleting message
When the ONLY violation is offensive_username (message content clean):
- Message is NOT deleted (nickname-only violation bypasses auto-delete)
- Member's server nickname is reset to default username via
  setNickname(null) (Discord shows the global username again)
- Action 'reset_nickname' logged to moderation_actions; cooldown
  10min per guild:user (LRU) so repeated messages by same member
  don't hammer the Discord PATCH
- Config: AUTO_NICKNAME_RESET_ENABLED / AUTO_NICKNAME_RESET_COOLDOWN_MS
2026-08-10 11:48:27 +07:00
asepharyana ecbb538c9f feat(ai-moderation): use per-server nickname (displayName) in analysis payload
- resolveDisplayName(): member.displayName from captured metadata,
  falls back to global username
- Applied to context lines, target message blocks, and media message
  blocks — LLM sees the name the channel actually sees (nickname can
  carry moderation signal itself)
2026-08-10 11:36:37 +07:00
asepharyana 4049ab4201 feat(ai-moderation): rich context + link media vision analysis
- Conversation context recency gates (GAP_MS/MAX_AGE_MS): drop stale
  messages before silence gaps; cold_start anchor + flow descriptor
  tells LLM whether conversation is ongoing or restarted
- [location] block: channel name, thread name, nsfw/age flags from
  captured metadata (thread names instead of bare IDs)
- Link media -> multimodal: text-batch URL fetches that resolve to
  images now run vision analysis (bounded 15s) and switch prompt to
  mixed mode; <web_content> gains og:title for page context
- pnpm-workspace.yaml: approve sharp build script (unblocks install)
2026-08-10 11:26:26 +07:00
asepharyana 5d094829c4 fix(ui): fit select popup to content and wrap long items
A single long guild name was clipped (whitespace-nowrap + narrow min-width),
so the dropdown rendered as a tiny 144px box with truncated text. Size the
popup to fit-content up to the available width and let option text wrap.
2026-08-07 19:29:36 +07:00
asepharyana abbd78f42b fix(ui): theme popover/input tokens and open select below trigger
Select dropdowns (voice tab, guild selector) rendered with a transparent
background because --color-popover/--color-input were undefined, and the
popup overlapped the trigger due to alignItemWithTrigger. Define the missing
theme tokens (popover, popover-foreground, input, secondary) and default the
select popup to open below the trigger.
2026-08-07 18:18:14 +07:00
81 changed files with 8638 additions and 1411 deletions
+118 -1
View File
@@ -82,6 +82,19 @@ jobs:
extra-conf: |
sandbox = false
accept-flake-config = true
# Attic binary cache as substituter on the runner: lets CI pull the
# prebuilt attic client (and any cached deps/builds) over HTTPS,
# no SSH round-trip needed. extra-substituters (NOT
# extra-trusted-substituters) is required — Determinate Nix never
# merges trusted-* substituters for nix-store CLI clients.
extra-substituters = https://attic.asepharyana.my.id/gmw
extra-trusted-public-keys = gmw:Fq2Anzuhkb+T/hftWnPcveHSi21/RzIgIOeG8pCJa88=
# NOTE: nix-installer-action unconditionally injects
# 'build-provenance-tags' into /etc/nix/nix.conf (a Determinate
# Nix-only setting). With determinate:false the runner's upstream
# nix warns 'unknown setting build-provenance-tags' on every
# invocation — benign, cosmetic. Switching determinate:true would
# silence it but changes the runner's nix flavor.
- name: Cache Nix
uses: DeterminateSystems/magic-nix-cache-action@v14
@@ -107,13 +120,117 @@ jobs:
ssh-keygen -y -f ~/.ssh/id_ed25519 >/dev/null 2>&1 || { echo "SSH key invalid"; exit 1; }
ssh-keyscan -H "$VPS_HOST" >> ~/.ssh/known_hosts 2>/dev/null
# Push build result to Attic binary cache (attic.asepharyana.my.id) so
# the VPS can substitute it instead of a single-stream `nix copy ssh://`.
#
# Fast path: push DIRECTLY from the runner to the public attic endpoint
# (validated 2026-08-10: token auth over public HTTPS works without
# Tailscale). This skips the ~794MB closure SSH copy to the VPS that
# used to take 25+ minutes per new store path.
#
# The attic client is NOT in nixpkgs anymore and has no prebuilt
# releases, so we pull the same prebuilt closure the VPS uses
# (/nix/store/fygyy3yk4rqdknxkiwkqambpnhyax0k4-attic-0.1.0, ~52MB).
# The closure itself lives in the attic cache (pushed once from the
# VPS), so the runner bootstraps it over HTTPS via the configured
# extra-substituters — no SSH round-trip. If that fails we fall back
# to `nix copy --from ssh://`, then the old VPS-hop flow (SSH copy to
# VPS, then attic push from the VPS over Tailscale) so the deploy step
# always has a working closure path.
- name: Push to Attic cache
env:
ATTIC_TOKEN: ${{ secrets.ATTIC_TOKEN }}
run: |
if [ -z "$ATTIC_TOKEN" ]; then
echo "ATTIC_TOKEN not set; skipping attic push"
exit 0
fi
STORE_PATH="${{ steps.build.outputs.store-path }}"
ATTIC_DIR="/nix/store/fygyy3yk4rqdknxkiwkqambpnhyax0k4-attic-0.1.0"
ATTIC_BIN="$ATTIC_DIR/bin/attic"
attic_push_vps_hop() {
echo "Fallback: VPS-hop attic push"
# Copy closure to VPS (fast if attic already has it via substitute)
ssh "$VPS_USER@$VPS_HOST" "sudo /nix/var/nix/profiles/default/bin/nix-store --realise '$STORE_PATH'" 2>/dev/null \
|| nix copy --to "ssh://$VPS_USER@$VPS_HOST" "$STORE_PATH"
# Push from VPS → Attic over Tailscale.
# --ignore-upstream-cache-filter is REQUIRED: without it, attic skips
# writing the narinfo to gmw when chunks exist in the upstream
# cache.nixos.org — leaving the path 404 on gmw so the VPS deploy's
# nix-store --realise can't find it and falls back to ssh copy.
# sudo: attic must read root's config (~/.config/attic), which has
# the imrnes-ts server → Tailscale. Non-root users' configs only
# have the public `pub` server → "Server imrnes-ts does not exist".
ssh "$VPS_USER@$VPS_HOST" "sudo $ATTIC_BIN push imrnes-ts:gmw '$STORE_PATH' --jobs 4 --ignore-upstream-cache-filter" \
|| echo "attic push failed (non-fatal; ssh copy fallback below)"
}
# ── Get an attic client on the runner ────────────────────────────
# Order: PATH → pull the prebuilt closure from the attic cache
# itself (extra-substituters configured in Install Nix step, HTTPS
# only, no SSH) → pull over ssh from the VPS → VPS-hop.
# The attic client closure is stored in the attic cache (pushed
# once from the VPS), so the fast path never depends on SSH.
ATTIC_BIN=""
if command -v attic >/dev/null 2>&1; then
ATTIC_BIN="$(command -v attic)"
elif nix-store --realise "$ATTIC_DIR" 2>/tmp/attic-bootstrap.err; then
echo "✅ Pulled attic client from attic cache (HTTPS substituter)"
ATTIC_BIN="$ATTIC_DIR/bin/attic"
elif nix copy --from "ssh://$VPS_USER@$VPS_HOST" "$ATTIC_DIR" 2>>/tmp/attic-bootstrap.err; then
echo "✅ Pulled attic client from VPS over ssh"
ATTIC_BIN="$ATTIC_DIR/bin/attic"
else
echo "attic client unavailable on runner; using VPS-hop flow"
echo "--- bootstrap errors (stderr) ---"
tail -5 /tmp/attic-bootstrap.err 2>/dev/null || true
attic_push_vps_hop
exit 0
fi
# ── Direct push: runner → attic public endpoint ──────────────────
# --ignore-upstream-cache-filter forces the narinfo write even when
# the path's chunks already exist in upstream cache.nixos.org (which
# attic would otherwise skip, leaving the path 404 on the gmw cache).
mkdir -p "$HOME/.config/attic"
cat > "$HOME/.config/attic/config.toml" <<EOF
default-server = "pub"
[servers.pub]
endpoint = "https://attic.asepharyana.my.id"
token = "$ATTIC_TOKEN"
EOF
# Retry the direct push — a transient 502 (e.g. atticd restart,
# Traefik blip) must not abort the whole closure upload. attic push
# is idempotent, so re-running only uploads what's still missing.
push_ok=""
for attempt in 1 2 3; do
if "$ATTIC_BIN" push pub:gmw "$STORE_PATH" --jobs 4 --ignore-upstream-cache-filter; then
echo "✅ Pushed $STORE_PATH to attic directly from runner"
push_ok=1
break
fi
echo "⚠️ Direct attic push attempt $attempt/3 failed; retrying in 10s..."
sleep 10
done
if [ -z "$push_ok" ]; then
echo "Direct attic push failed after 3 attempts; using VPS-hop flow"
attic_push_vps_hop
fi
# NOTE: env files /etc/gmw/backend.env & /etc/gmw/discord-gateway.env are
# managed MANUALLY on the VPS (source of truth). CI only builds & deploys.
- name: Deploy ${{ matrix.service }} to VPS
run: |
STORE_PATH="${{ steps.build.outputs.store-path }}"
echo "=== Copying ${{ matrix.service }}: $STORE_PATH ==="
nix copy --to "ssh://$VPS_USER@$VPS_HOST" "$STORE_PATH"
if [ -n "${{ secrets.ATTIC_TOKEN }}" ] && ssh "$VPS_USER@$VPS_HOST" "sudo /nix/var/nix/profiles/default/bin/nix-store --realise '$STORE_PATH'" 2>/dev/null; then
echo "Substituted ${{ matrix.service }} from Attic cache"
else
echo "Attic substitute failed; falling back to ssh copy"
nix copy --to "ssh://$VPS_USER@$VPS_HOST" "$STORE_PATH"
fi
echo "=== Updating profile ==="
ssh "$VPS_USER@$VPS_HOST" "sudo /nix/var/nix/profiles/default/bin/nix-env --profile /nix/var/nix/profiles/gmw-${{ matrix.service }} --set '$STORE_PATH'"
+1 -1
View File
@@ -12,7 +12,7 @@ worktrees/
.worktrees/
services/frontend/frontend/dist/
target/
nix/
# Gitea CI runner logs
.gitea/workflows/*.log
+76 -16
View File
@@ -11,6 +11,11 @@
let
pkgs = import nixpkgs { inherit system; };
# libdatachannel for the GoLive N-API binding. nixpkgs 0.24.1 is built
# against this host's glibc and ships both lib + dev headers, so the
# binding links cleanly inside the Nix sandbox (no manual cmake build).
libdatachannel = pkgs.libdatachannel;
# Source filter: `path:` literals do NOT respect .gitignore by default,
# so a dirty local out/ (stale chunks from previous builds) leaks into
# the sandbox. Filter out build artifacts explicitly.
@@ -59,6 +64,36 @@
pnpm rebuild 2>&1 || true
'';
# Shrink the shipped node_modules to production deps only. The full
# install's .pnpm virtual store carries dev-only packages (biome,
# typescript, esbuild, drizzle-kit, vitest, ... ~150MB+) that are never
# needed at runtime, so we delete every .pnpm dir that is not part of
# the resolved production graph (`pnpm list --prod`).
#
# NOTE: do NOT use `pnpm install --prod` here — it collapses the
# public-hoist dir (.pnpm/node_modules) that runtime peer resolution
# relies on (e.g. @lng2004/node-datachannel and @seydx/node-av-linux-x64
# are only reachable through it), silently breaking voice/screenshare.
# Instead we keep the full install's symlink layout and only prune
# orphaned package dirs + broken symlinks.
# Must run AFTER tsc (typescript is a devDep) and after native builds.
pruneProd = ''
echo "=== Pruning devDependencies (production-only node_modules) ==="
pnpm list --prod --depth 999 --parseable 2>/dev/null \
| grep -o '\.pnpm/[^/]*' | sort -u > $TMPDIR/prod-pnms.txt
( cd node_modules/.pnpm \
&& for d in */; do \
d="''${d%/}"; \
[ "$d" = "node_modules" ] && continue; \
grep -qF ".pnpm/$d" $TMPDIR/prod-pnms.txt || rm -rf "$d"; \
done ) || true
# Drop symlinks whose .pnpm target was pruned (top-level, scoped dirs,
# hoist, .bin any depth). Mirrors stdenv's noBrokenSymlinks check,
# which would otherwise fail the fixupPhase.
find node_modules -type l ! -exec test -e {} \; -delete 2>/dev/null || true
du -sh node_modules
'';
# ---- Backend ----
backend = pkgs.stdenv.mkDerivation {
pname = "gmw-backend";
@@ -97,7 +132,7 @@
console.log('Fixed ' + count + ' files');
"
echo "=== Build complete ==="
'';
'' + pruneProd;
installPhase = ''
mkdir -p $out/lib/gmw-backend
@@ -132,6 +167,7 @@ WRAPPER
pkgs.pkg-config
pkgs.openssl
pkgs.openssl.dev
libdatachannel.dev # rtc/rtc.hpp headers for the GoLive binding
pkgs.git # libdatachannel FetchContent clones from GitHub
pkgs.cacert
];
@@ -150,27 +186,34 @@ WRAPPER
# pnpm rebuild aborts on the first failing package and runs scripts
# from the wrong cwd build each native dep explicitly with its own
# install script. Each failure is tolerated (|| true); the packages
# that matter (opus, datachannel, node-av) are verified at runtime.
# that matter (opus) are verified at runtime.
for pkg in \
node_modules/.pnpm/@discordjs+opus@*/node_modules/@discordjs/opus \
node_modules/.pnpm/@lng2004+node-datachannel@*/node_modules/@lng2004/node-datachannel \
node_modules/.pnpm/zeromq@*/node_modules/zeromq
node_modules/.pnpm/@discordjs+opus@*/node_modules/@discordjs/opus
do
if [ -d "$pkg" ]; then
echo "--- native build: $pkg ---"
(cd "$pkg" && npm run install 2>&1 || true)
# node-datachannel's `prebuild -r napi` CLI is broken (TypeError:
# expected first argument to be an array) the install fallback
# populates devDeps incl. cmake-js; build directly via cmake-js.
if [ "$(basename "$pkg")" = "node-datachannel" ]; then
echo "--- datachannel cmake-js compile ---"
# Nix splits OpenSSL headers/libs across outputs merge them
# (opensslDevEnv) so FindOpenSSL finds both include + libcrypto.
(cd "$pkg" && OPENSSL_ROOT_DIR="${opensslDevEnv}" npm run compile 2>&1 || true)
fi
fi
done
echo "=== Compiling TypeScript ==="
echo "=== Building libdatachannel-min N-API binding ==="
# The GoLive screen-share stack uses a minimal N-API binding
# (native/libdatachannel-min) over nixpkgs libdatachannel.
(
cd native/libdatachannel-min
# binding.gyp resolves include/lib from env (LDC_INCLUDE = .dev
# include root, LDC_LIB = lib output dir, NAPI_INCLUDE =
# node-addon-api include root).
NAPI_INCLUDE=$(find ../../node_modules/.pnpm -maxdepth 3 \
-type d -path "*node_modules/node-addon-api" | head -1)
echo "NAPI_INCLUDE=$NAPI_INCLUDE"
LDC_INCLUDE=${libdatachannel.dev} LDC_LIB=${libdatachannel.out}/lib/libdatachannel.so.0.24.1 \
NAPI_INCLUDE=$NAPI_INCLUDE \
npx node-gyp rebuild 2>&1 || true
ls -la build/Release/datachannel_min.node 2>/dev/null \
&& echo "libdatachannel-min binding OK: $(stat -c%s build/Release/datachannel_min.node) bytes" \
|| echo "WARN: libdatachannel-min binding build FAILED (screen share disabled)"
)
echo "=== Compiling TypeScript ===="
npx tsc 2>&1
echo "=== Fixing @/ path aliases to relative paths ==="
node -e "
@@ -198,12 +241,28 @@ WRAPPER
console.log('Fixed ' + count + ' files');
"
echo "=== Build complete ==="
'';
'' + pruneProd;
installPhase = ''
mkdir -p $out/lib/gmw-discord-gateway
cp -r dist node_modules package.json tsconfig.json $out/lib/gmw-discord-gateway/
# GoLive native binding loadNative resolves it relative to
# dist/goLive/native.js, i.e. <root>/native/libdatachannel-min/
# build/Release/datachannel_min.node; libdatachannel .so must sit
# next to it and be on LD_LIBRARY_PATH at runtime.
mkdir -p $out/lib/gmw-discord-gateway/native/libdatachannel-min/build/Release
cp native/libdatachannel-min/build/Release/datachannel_min.node \
$out/lib/gmw-discord-gateway/native/libdatachannel-min/build/Release/ 2>/dev/null || true
mkdir -p $out/lib/gmw-discord-gateway/native/libdatachannel-min/build/ldc
cp -rL native/libdatachannel-min/build/ldc/libdatachannel.so* \
$out/lib/gmw-discord-gateway/native/libdatachannel-min/build/ldc/ 2>/dev/null || true
# If the binding failed to build, screen share is simply disabled
# the gateway itself must still start.
if [ ! -f $out/lib/gmw-discord-gateway/native/libdatachannel-min/build/Release/datachannel_min.node ]; then
echo "WARN: datachannel_min.node missing GoLive screen share disabled in this build"
fi
# Also include drizzle migrations if they exist
cp -r drizzle $out/lib/gmw-discord-gateway/ 2>/dev/null || true
@@ -212,6 +271,7 @@ WRAPPER
#!${pkgs.runtimeShell}
cd $out/lib/gmw-discord-gateway
export PATH=${pkgs.ffmpeg-headless}/bin:${pkgs.yt-dlp}/bin:\$PATH
export LD_LIBRARY_PATH=${libdatachannel.out}/lib:\$LD_LIBRARY_PATH
exec ${nodejs}/bin/node dist/index.js
WRAPPER
chmod +x $out/bin/gmw-discord-gateway
@@ -0,0 +1,9 @@
CREATE TABLE IF NOT EXISTS "term_glossary_cache" (
"term" text PRIMARY KEY NOT NULL,
"definition" text NOT NULL,
"source_url" text DEFAULT '' NOT NULL,
"resolved_at" bigint NOT NULL,
"hit_count" integer DEFAULT 0 NOT NULL
);
--> statement-breakpoint
CREATE INDEX IF NOT EXISTS "idx_term_glossary_cache_resolved_at" ON "term_glossary_cache" USING btree ("resolved_at");
@@ -99,6 +99,13 @@
"when": 1785551832190,
"tag": "0013_rename_mascot_chat_to_chatbot",
"breakpoints": true
},
{
"idx": 14,
"version": "7",
"when": 1785621600000,
"tag": "0014_add_term_glossary_cache",
"breakpoints": true
}
]
}
@@ -0,0 +1,3 @@
node_modules/
build/
package-lock.json
@@ -0,0 +1,526 @@
// libdatachannel-min — minimal N-API binding to libdatachannel.
// Exposes ONLY what GMW GoLive needs:
// PeerConnection (offer/answer, ICE, SDP), DataChannel (signaling),
// Track send (added in media phase).
// Built against libdatachannel 0.24.0 (built from source in /tmp/ldc-build).
#include <napi.h>
#include <rtc/rtc.hpp>
#include <functional>
#include <memory>
#include <string>
#include <variant>
using namespace Napi;
namespace {
std::string stateToString(rtc::PeerConnection::State s) {
switch (s) {
case rtc::PeerConnection::State::New: return "new";
case rtc::PeerConnection::State::Connecting: return "connecting";
case rtc::PeerConnection::State::Connected: return "connected";
case rtc::PeerConnection::State::Disconnected: return "disconnected";
case rtc::PeerConnection::State::Failed: return "failed";
case rtc::PeerConnection::State::Closed: return "closed";
default: return "unknown";
}
}
std::string binaryToString(const rtc::binary& data) {
// rtc::binary is std::vector<std::byte> in libdatachannel >= 0.21
std::string msg(data.size(), '\0');
for (size_t i = 0; i < data.size(); i++) {
msg[i] = static_cast<char>(data[i]);
}
return msg;
}
// Holds a Napi::Promise::Deferred so it can be moved into TSFN lambdas
// without invalid copies (node-addon-api 8.x Deferred is not movable).
struct DeferredHolder {
Promise::Deferred deferred;
explicit DeferredHolder(Promise::Deferred d) : deferred(d) {}
};
class DataChannelWrap : public Napi::ObjectWrap<DataChannelWrap> {
public:
static Function Init(Napi::Env env) {
Function func = DefineClass(env, "DataChannel", {
InstanceMethod("send", &DataChannelWrap::Send),
InstanceMethod("isOpen", &DataChannelWrap::IsOpen),
InstanceMethod("close", &DataChannelWrap::Close),
InstanceMethod("onMessage", &DataChannelWrap::OnMessage),
InstanceMethod("onOpen", &DataChannelWrap::OnOpen),
});
dcConstructor = Napi::Persistent(func);
return func;
}
// Create a JS wrapper (calls the JS constructor, returns instance).
static Object NewInstance(Napi::Env env) {
return dcConstructor.New({});
}
DataChannelWrap(const Napi::CallbackInfo& info)
: Napi::ObjectWrap<DataChannelWrap>(info) {}
void Init(std::shared_ptr<rtc::DataChannel> dc) {
dc_ = dc;
dc_->onMessage([this](rtc::message_variant data) {
std::string msg;
if (std::holds_alternative<rtc::binary>(data)) {
msg = binaryToString(std::get<rtc::binary>(data));
} else {
msg = std::get<std::string>(data);
}
if (msgCb_) {
msgCb_->BlockingCall([msg](Napi::Env env, Function cb) {
cb.Call({String::New(env, msg)});
});
}
});
dc_->onOpen([this]() {
if (openCb_) {
openCb_->BlockingCall([](Napi::Env env, Function cb) {
cb.Call({});
});
}
});
}
private:
static FunctionReference dcConstructor;
std::shared_ptr<rtc::DataChannel> dc_;
std::shared_ptr<ThreadSafeFunction> msgCb_;
std::shared_ptr<ThreadSafeFunction> openCb_;
void Send(const Napi::CallbackInfo& info) {
std::string msg = info[0].As<String>().Utf8Value();
if (dc_) dc_->send(msg);
}
Napi::Value IsOpen(const Napi::CallbackInfo& info) {
bool open = dc_ && dc_->isOpen();
return Boolean::New(info.Env(), open);
}
void Close(const Napi::CallbackInfo& info) {
if (dc_) dc_->close();
}
void OnMessage(const Napi::CallbackInfo& info) {
Function cb = info[0].As<Function>();
msgCb_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(info.Env(), cb, "dc-message", 0, 1));
}
void OnOpen(const Napi::CallbackInfo& info) {
Function cb = info[0].As<Function>();
openCb_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(info.Env(), cb, "dc-open", 0, 1));
}
};
class TrackWrap : public Napi::ObjectWrap<TrackWrap> {
public:
static Function Init(Napi::Env env) {
Function func = DefineClass(env, "Track", {
InstanceMethod("send", &TrackWrap::Send),
InstanceMethod("isOpen", &TrackWrap::IsOpen),
InstanceMethod("close", &TrackWrap::Close),
InstanceMethod("setPacketizer", &TrackWrap::SetPacketizer),
InstanceMethod("sendFrame", &TrackWrap::SendFrame),
InstanceMethod("addTimestamp", &TrackWrap::AddTimestamp),
});
trackConstructor = Napi::Persistent(func);
return func;
}
static Object NewInstance(Napi::Env env) {
return trackConstructor.New({});
}
TrackWrap(const Napi::CallbackInfo& info)
: Napi::ObjectWrap<TrackWrap>(info) {}
void Init(std::shared_ptr<rtc::Track> track, Napi::Env env) {
track_ = track;
(void)env;
}
private:
static FunctionReference trackConstructor;
std::shared_ptr<rtc::Track> track_;
std::shared_ptr<rtc::RtpPacketizationConfig> rtpConfig_;
void Send(const Napi::CallbackInfo& info) {
Buffer<uint8_t> buf = info[0].As<Buffer<uint8_t>>();
if (!track_) return;
rtc::binary data(buf.Length());
for (size_t i = 0; i < buf.Length(); i++) data[i] = (std::byte)buf[i];
try {
track_->send(data);
} catch (const std::exception& e) {
fprintf(stderr, "[binding] track.send THREW: %s\n", e.what());
}
}
// setPacketizer(kind, ssrc, payloadType, clockRate, playoutDelayId,
// playoutDelayMin, playoutDelayMax)
// kind: "audio" | "h264" | "h265" | "av1"
// Builds the media-handler chain (packetizer → RTCP SR → NACK → pacing for
// video) exactly like @dank074's WebRtcWrapper does via node-datachannel.
void SetPacketizer(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
if (!track_) throw Error::New(env, "track closed");
std::string kind = info[0].As<String>().Utf8Value();
uint32_t ssrc = info[1].As<Number>().Uint32Value();
uint8_t pt = (uint8_t)info[2].As<Number>().Uint32Value();
uint32_t clockRate = info[3].As<Number>().Uint32Value();
uint8_t playoutDelayId = (uint8_t)info[4].As<Number>().Uint32Value();
uint16_t playoutDelayMin = (uint16_t)info[5].As<Number>().Uint32Value();
uint16_t playoutDelayMax = (uint16_t)info[6].As<Number>().Uint32Value();
try {
auto cfg = std::make_shared<rtc::RtpPacketizationConfig>(
ssrc, "", pt, clockRate);
cfg->playoutDelayId = playoutDelayId;
cfg->playoutDelayMin = playoutDelayMin;
cfg->playoutDelayMax = playoutDelayMax;
std::shared_ptr<rtc::MediaHandler> handler;
if (kind == "audio") {
handler = std::make_shared<rtc::OpusRtpPacketizer>(cfg);
} else if (kind == "h264") {
handler = std::make_shared<rtc::H264RtpPacketizer>(
rtc::NalUnit::Separator::StartSequence, cfg);
} else if (kind == "h265") {
handler = std::make_shared<rtc::H265RtpPacketizer>(
rtc::NalUnit::Separator::StartSequence, cfg);
} else if (kind == "av1") {
handler = std::make_shared<rtc::AV1RtpPacketizer>(
rtc::AV1RtpPacketizer::Packetization::Obu, cfg);
} else {
throw std::runtime_error("unknown packetizer kind: " + kind);
}
handler->addToChain(std::make_shared<rtc::RtcpSrReporter>(cfg));
handler->addToChain(std::make_shared<rtc::RtcpNackResponder>());
if (kind != "audio") {
handler->addToChain(std::make_shared<rtc::PacingHandler>(
25.0 * 1000 * 1000, std::chrono::milliseconds(1)));
}
track_->setMediaHandler(handler);
rtpConfig_ = cfg;
} catch (const std::exception& e) {
fprintf(stderr, "[binding] setPacketizer THREW: %s\n", e.what());
throw Error::New(env, e.what());
}
}
// sendFrame(buffer) — sends an ENCODED frame (AnnexB H264 / raw opus /
// OBU AV1). The media-handler chain packetizes it into RTP.
void SendFrame(const Napi::CallbackInfo& info) {
Buffer<uint8_t> buf = info[0].As<Buffer<uint8_t>>();
if (!track_) return;
rtc::binary data(buf.Length());
for (size_t i = 0; i < buf.Length(); i++) data[i] = (std::byte)buf[i];
try {
track_->send(data);
} catch (const std::exception& e) {
fprintf(stderr, "[binding] track.sendFrame THREW: %s\n", e.what());
}
}
// addTimestamp(delta) — advances the packetizer RTP timestamp by delta
// (clock-rate units). Called by JS after each frame, matching the
// node-datachannel contract (WebRtcWrapper does the same increment).
void AddTimestamp(const Napi::CallbackInfo& info) {
uint32_t delta = info[0].As<Number>().Uint32Value();
if (rtpConfig_) rtpConfig_->timestamp += delta;
}
Napi::Value IsOpen(const Napi::CallbackInfo& info) {
bool open = track_ && track_->isOpen();
return Boolean::New(info.Env(), open);
}
void Close(const Napi::CallbackInfo& info) {
if (track_) track_->close();
}
void OnStateChange(const Napi::CallbackInfo& info) {
// libdatachannel Track has no state-change callback; kept for API parity.
(void)info;
}
};
class PeerConnectionWrap : public Napi::ObjectWrap<PeerConnectionWrap> {
public:
static Function Init(Napi::Env env) {
Function func = DefineClass(env, "PeerConnection", {
InstanceMethod("state", &PeerConnectionWrap::State),
InstanceMethod("createOffer", &PeerConnectionWrap::CreateOffer),
InstanceMethod("createAnswer", &PeerConnectionWrap::CreateAnswer),
InstanceMethod("setRemoteDescription",
&PeerConnectionWrap::SetRemoteDescription),
InstanceMethod("close", &PeerConnectionWrap::Close),
InstanceMethod("onStateChange", &PeerConnectionWrap::OnStateChange),
InstanceMethod("createDataChannel", &PeerConnectionWrap::CreateDataChannel),
InstanceMethod("onDataChannel", &PeerConnectionWrap::OnDataChannel),
InstanceMethod("addTrack", &PeerConnectionWrap::AddTrack),
});
return func;
}
PeerConnectionWrap(const Napi::CallbackInfo& info)
: Napi::ObjectWrap<PeerConnectionWrap>(info) {
Napi::Env env = info.Env();
if (!info[0].IsObject()) {
throw TypeError::New(env, "config object required");
}
Object config = info[0].As<Object>();
rtc::Configuration rtcConfig;
if (config.Has("iceServers")) {
Array servers = config.Get("iceServers").As<Array>();
for (uint32_t i = 0; i < servers.Length(); i++) {
std::string url = servers.Get(i).As<String>().Utf8Value();
rtcConfig.iceServers.emplace_back(url);
}
}
pc_ = std::make_shared<rtc::PeerConnection>(rtcConfig);
// IMPORTANT: register description/gathering callbacks HERE (constructor),
// BEFORE any createDataChannel call. libdatachannel only fires
// onLocalDescription for negotiations that start AFTER the callback is
// registered — if createDataChannel runs first, the offer callback never
// fires (verified in C++ spike: test3 vs test2).
pc_->onLocalDescription([this](rtc::Description desc) {
latestLocalDesc_ = std::string(desc);
fprintf(stderr, "[binding] trickle desc, %zu bytes\n",
latestLocalDesc_.size());
});
pc_->onGatheringStateChange([this](rtc::PeerConnection::GatheringState gs) {
fprintf(stderr, "[binding] gathering state: %d\n", (int)gs);
if (gs == rtc::PeerConnection::GatheringState::Complete) {
// Use the getter — it returns the FULL SDP including candidates after
// gathering (trickle callbacks only carry the initial fragment).
auto ld = pc_->localDescription();
if (ld) {
latestLocalDesc_ = std::string(*ld);
fprintf(stderr, "[binding] final desc, %zu bytes\n",
latestLocalDesc_.size());
}
resolvePendingLocalDesc_();
}
});
}
private:
std::shared_ptr<rtc::PeerConnection> pc_;
std::shared_ptr<ThreadSafeFunction> stateCb_;
std::shared_ptr<ThreadSafeFunction> dcCb_;
std::string latestLocalDesc_;
std::shared_ptr<DeferredHolder> pendingDescDeferred_;
std::shared_ptr<ThreadSafeFunction> pendingDescTsfn_;
void resolvePendingLocalDesc_() {
if (!pendingDescDeferred_ || !pendingDescTsfn_) return;
auto holder = pendingDescDeferred_;
auto tsfn = pendingDescTsfn_;
pendingDescDeferred_.reset();
pendingDescTsfn_.reset();
std::string sdp = latestLocalDesc_;
tsfn->BlockingCall([sdp, holder](Napi::Env e, Function) {
holder->deferred.Resolve(String::New(e, sdp));
});
}
Napi::Value State(const Napi::CallbackInfo& info) {
return String::New(info.Env(),
pc_ ? stateToString(pc_->state()) : "closed");
}
// createOffer() -> Promise<string> — sets local description, waits for
// ICE gathering to complete (so candidates are in the SDP), resolves SDP.
Napi::Value CreateOffer(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
auto holder = std::make_shared<DeferredHolder>(Promise::Deferred::New(env));
if (!pc_) {
holder->deferred.Reject(Error::New(env, "peer closed").Value());
return holder->deferred.Promise();
}
// createDataChannel already triggers negotiation in libdatachannel 0.24 —
// if gathering already completed, resolve immediately from the cached SDP.
if (!latestLocalDesc_.empty()) {
auto tsfn = std::make_shared<ThreadSafeFunction>(ThreadSafeFunction::New(
env, Function::New(env, [](const CallbackInfo&) {}), "desc", 0, 1));
std::string sdp = latestLocalDesc_;
tsfn->BlockingCall([sdp, holder](Napi::Env e, Function) {
holder->deferred.Resolve(String::New(e, sdp));
});
return holder->deferred.Promise();
}
if (pendingDescDeferred_) {
pendingDescDeferred_->deferred.Reject(
Error::New(env, "previous negotiation still pending").Value());
}
pendingDescDeferred_ = holder;
pendingDescTsfn_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(env, Function::New(env, [](const CallbackInfo&) {}),
"desc", 0, 1));
fprintf(stderr, "[binding] calling setLocalDescription(Offer)\n");
try {
pc_->setLocalDescription(rtc::Description::Type::Offer);
fprintf(stderr, "[binding] setLocalDescription returned OK\n");
} catch (const std::exception& e) {
pendingDescDeferred_.reset();
fprintf(stderr, "[binding] setLocalDescription THREW: %s\n", e.what());
throw Error::New(env, e.what());
}
return holder->deferred.Promise();
}
// createAnswer(offerSdp: string) -> Promise<string>
Napi::Value CreateAnswer(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
std::string offer = info[0].As<String>().Utf8Value();
auto holder = std::make_shared<DeferredHolder>(Promise::Deferred::New(env));
if (!pc_) {
holder->deferred.Reject(Error::New(env, "peer closed").Value());
return holder->deferred.Promise();
}
if (pendingDescDeferred_) {
pendingDescDeferred_->deferred.Reject(
Error::New(env, "previous negotiation still pending").Value());
}
pendingDescDeferred_ = holder;
pendingDescTsfn_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(env, Function::New(env, [](const CallbackInfo&) {}),
"desc", 0, 1));
try {
pc_->setRemoteDescription(
rtc::Description(offer, rtc::Description::Type::Offer));
fprintf(stderr, "[binding] answer: setRemoteDescription OK\n");
// libdatachannel 0.24 AUTO-GENERATES the answer when a remote offer is
// applied (verified in C++ spike test8/9: B desc type=Answer fires
// immediately with a=setup:active). Calling setLocalDescription() again
// would OVERWRITE it with a role=actpass SDP, which A rejects with
// "Illegal role actpass in remote answer description". So we do NOT call
// setLocalDescription here — we just wait for gathering complete and
// resolve with the auto-generated answer. This also matches @dank074's
// Discord voice flow.
} catch (const std::exception& e) {
pendingDescDeferred_.reset();
fprintf(stderr, "[binding] answer THREW: %s\n", e.what());
holder->deferred.Reject(Error::New(env, e.what()).Value());
}
return holder->deferred.Promise();
}
void SetRemoteDescription(const Napi::CallbackInfo& info) {
std::string sdp = info[0].As<String>().Utf8Value();
std::string type = info[1].As<String>().Utf8Value();
rtc::Description::Type t = (type == "answer")
? rtc::Description::Type::Answer
: rtc::Description::Type::Offer;
if (pc_) pc_->setRemoteDescription(rtc::Description(sdp, t));
}
void Close(const Napi::CallbackInfo& info) {
if (pc_) pc_->close();
}
void OnStateChange(const Napi::CallbackInfo& info) {
Function cb = info[0].As<Function>();
stateCb_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(info.Env(), cb, "pc-state", 0, 1));
std::shared_ptr<rtc::PeerConnection> pc = pc_;
pc->onStateChange([this](rtc::PeerConnection::State state) {
if (stateCb_) {
std::string s = stateToString(state);
stateCb_->BlockingCall([s](Napi::Env env, Function cb) {
cb.Call({String::New(env, s)});
});
}
});
}
Napi::Value CreateDataChannel(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
std::string label = info[0].As<String>().Utf8Value();
fprintf(stderr, "[binding] createDataChannel(%s)\n", label.c_str());
auto dc = pc_->createDataChannel(label);
Object obj = DataChannelWrap::NewInstance(env);
DataChannelWrap::Unwrap(obj)->Init(dc);
return obj;
}
Napi::Value AddTrack(const Napi::CallbackInfo& info) {
Napi::Env env = info.Env();
std::string mid = info[0].As<String>().Utf8Value();
std::string kind = info[1].As<String>().Utf8Value();
if (!pc_) throw Error::New(env, "peer closed");
fprintf(stderr, "[binding] addTrack(%s, %s) start\n", mid.c_str(), kind.c_str());
try {
std::shared_ptr<rtc::Track> track;
if (kind == "audio") {
// Opus payload type 120 (matches @dank074 CodecPayloadType.opus)
auto desc = rtc::Description::Audio(mid);
desc.addOpusCodec(120);
track = pc_->addTrack(desc);
} else {
// All video codecs with their payload types, matching WebRtcWrapper:
// H264 101/102, H265 103/104, VP8 105/106, VP9 107/108, AV1 109/110
auto desc = rtc::Description::Video(mid);
desc.addH264Codec(101);
desc.addRtxCodec(102, 101, 90000);
desc.addH265Codec(103);
desc.addRtxCodec(104, 103, 90000);
desc.addVP8Codec(105);
desc.addRtxCodec(106, 105, 90000);
desc.addVP9Codec(107);
desc.addRtxCodec(108, 107, 90000);
desc.addAV1Codec(109);
desc.addRtxCodec(110, 109, 90000);
track = pc_->addTrack(desc);
}
Object obj = TrackWrap::NewInstance(env);
TrackWrap::Unwrap(obj)->Init(track, env);
return obj;
} catch (const std::exception& e) {
fprintf(stderr, "[binding] addTrack THREW: %s\n", e.what());
throw Error::New(env, e.what());
}
}
void OnDataChannel(const Napi::CallbackInfo& info) {
Function cb = info[0].As<Function>();
dcCb_ = std::make_shared<ThreadSafeFunction>(
ThreadSafeFunction::New(info.Env(), cb, "dc", 0, 1));
std::shared_ptr<rtc::PeerConnection> pc = pc_;
pc->onDataChannel([this](std::shared_ptr<rtc::DataChannel> dc) {
if (dcCb_) {
auto dcPtr = dc;
dcCb_->BlockingCall([dcPtr](Napi::Env env, Function cb) {
Object obj = DataChannelWrap::NewInstance(env);
DataChannelWrap::Unwrap(obj)->Init(dcPtr);
cb.Call({obj});
});
}
});
}
};
Object InitAll(Napi::Env env, Object exports) {
exports.Set("PeerConnection", PeerConnectionWrap::Init(env));
exports.Set("DataChannel", DataChannelWrap::Init(env));
exports.Set("Track", TrackWrap::Init(env));
return exports;
}
NODE_API_MODULE(libdatachannel_min, InitAll)
// Definition for the static constructor references.
FunctionReference DataChannelWrap::dcConstructor;
FunctionReference TrackWrap::trackConstructor;
} // namespace
@@ -0,0 +1,21 @@
{
"targets": [
{
"target_name": "libdatachannel_min",
"sources": ["binding.cpp"],
"include_dirs": [
"<!(node -e \"console.log(process.env.NAPI_INCLUDE || (() => { try { return require('node-addon-api').include; } catch { return '/nonexistent'; } })())\")",
"<!(node -e \"const s=process.env.LDC_INCLUDE||'/nix/store/39a85gpfjqy3h3k8jwrwh7m9yc3inqw7-source';console.log(s+'/include')\")"
],
"libraries": [
"<!(node -e \"console.log(process.env.LDC_LIB || '/tmp/ldc-build/libdatachannel.so.0.24.0')\")"
],
"cflags": ["-std=c++17", "-fexceptions"],
"cflags_cc": ["-std=c++17", "-fexceptions"],
"defines": ["NAPI_CPP_EXCEPTIONS"],
"conditions": [
["OS=='linux'", { "cflags": ["-fvisibility=hidden"] }]
]
}
]
}
@@ -0,0 +1,3 @@
// libdatachannel-min — JS entry.
const native = require("./build/Release/datachannel_min.node");
module.exports = native;
@@ -0,0 +1,17 @@
{
"name": "libdatachannel-min",
"version": "0.1.0",
"description": "Minimal N-API binding to libdatachannel — PeerConnection, DataChannel, ICE, SDP (+ media tracks for GoLive)",
"main": "index.js",
"gypfile": true,
"scripts": {
"build": "node-gyp rebuild",
"test": "node test-handshake.js"
},
"dependencies": {
"node-addon-api": "^8.3.0"
},
"devDependencies": {
"node-gyp": "^11.5.0"
}
}
@@ -0,0 +1,82 @@
// Phase 0 spike: prove the minimal binding can do a full WebRTC handshake
// (offer/answer + ICE + DataChannel) between two local PeerConnections.
"use strict";
const { PeerConnection } = require("./build/Release/datachannel_min.node");
function log(...args) {
console.log("[spike]", ...args);
}
async function main() {
const pcA = new PeerConnection({ iceServers: [] });
const pcB = new PeerConnection({ iceServers: [] });
const stateLog = [];
pcA.onStateChange((s) => {
stateLog.push(`A:${s}`);
log("A state:", s);
});
pcB.onStateChange((s) => {
stateLog.push(`B:${s}`);
log("B state:", s);
});
// B waits for incoming DataChannel
const received = new Promise((resolve) => {
pcB.onDataChannel((dc) => {
log("B got incoming DataChannel");
dc.onOpen(() => log("B DataChannel open"));
dc.onMessage((msg) => {
log("B received message:", msg);
dc.send("pong from B");
resolve(msg);
});
});
});
// A creates an outgoing DataChannel
const dcA = pcA.createDataChannel("test");
dcA.onOpen(() => {
log("A DataChannel open — sending hello");
dcA.send("hello from A");
});
dcA.onMessage((msg) => {
log("A received reply:", msg);
});
// Offer/answer dance
log("A createOffer...");
const offer = await pcA.createOffer();
log("Offer SDP bytes:", offer.length);
log("B createAnswer...");
const answer = await pcB.createAnswer(offer);
log("Answer SDP bytes:", answer.length);
const setupMatch = answer.match(/a=setup:(\S+)/);
log("Answer setup role:", setupMatch ? setupMatch[1] : "NONE");
pcA.setRemoteDescription(answer, "answer");
// Wait for message roundtrip
const msg = await Promise.race([
received,
new Promise((_, rej) => setTimeout(() => rej(new Error("TIMEOUT waiting for datachannel message")), 15000)),
]);
log("ROUNDTRIP OK — B got:", msg);
log("States:", stateLog.join(" | "));
const aState = pcA.state();
const bState = pcB.state();
log("Final states — A:", aState, "B:", bState);
pcA.close();
pcB.close();
if (msg !== "hello from A") throw new Error("wrong message");
if (aState !== "connected" && aState !== "disconnected") throw new Error("A not connected: " + aState);
log("SPIKE PASSED ✅");
}
main().catch((e) => {
console.error("SPIKE FAILED:", e.message);
process.exit(1);
});
@@ -0,0 +1,80 @@
// Verify setPacketizer + sendFrame: two peers connect, audio+video tracks
// packetize real encoded frames (opus + AnnexB H264), RTP flows without crash.
"use strict";
const { PeerConnection } = require("./build/Release/datachannel_min.node");
function sleep(ms) { return new Promise((r) => setTimeout(r, ms)); }
async function main() {
const pcA = new PeerConnection({ iceServers: [] });
const pcB = new PeerConnection({ iceServers: [] });
const aAudio = pcA.addTrack("0", "audio");
const aVideo = pcA.addTrack("1", "video");
pcB.addTrack("0", "audio");
pcB.addTrack("1", "video");
let states = { a: "", b: "" };
pcA.onStateChange((s) => (states.a = s));
pcB.onStateChange((s) => (states.b = s));
// A: offer (createDataChannel not needed — tracks trigger negotiation)
const offer = await pcA.createOffer();
pcB.setRemoteDescription(offer, "offer");
const answer = await pcB.createAnswer(offer);
pcA.setRemoteDescription(answer, "answer");
// Wait for connected
for (let i = 0; i < 50; i++) {
if (states.a === "connected" && states.b === "connected") break;
await sleep(100);
}
console.log("[pkt] states:", states.a, states.b);
if (states.a !== "connected" || states.b !== "connected") {
console.log("PKT TEST FAILED: not connected");
process.exit(1);
}
// Setup packetizers on A (sender)
aAudio.setPacketizer("audio", 1234, 120, 48000, 5, 0, 1);
aVideo.setPacketizer("h264", 5678, 101, 90000, 5, 0, 10);
// Fake opus frame (20ms @48kHz stereo — payload can be any bytes)
const opusFrame = Buffer.alloc(160);
for (let i = 0; i < 160; i++) opusFrame[i] = i & 0xff;
// Fake AnnexB H264 frame: SPS + PPS + IDR slice
const sps = Buffer.from([0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0xc0, 0x1e, 0xd9, 0x01, 0x40, 0x7e]);
const pps = Buffer.from([0x00, 0x00, 0x00, 0x01, 0x68, 0xce, 0x3c, 0x80]);
const idr = Buffer.from([0x00, 0x00, 0x00, 0x01, 0x65, 0x88, 0x84, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07]);
const h264Frame = Buffer.concat([sps, pps, idr]);
// Send 10 audio frames (20ms each) + 3 video frames (33ms each)
for (let i = 0; i < 10; i++) {
aAudio.sendFrame(opusFrame);
aAudio.addTimestamp(960); // 20ms @ 48kHz
}
for (let i = 0; i < 3; i++) {
aVideo.sendFrame(h264Frame);
aVideo.addTimestamp(3000); // 33ms @ 90kHz
}
await sleep(500);
console.log("[pkt] after send: states:", states.a, states.b);
console.log("[pkt] audio track open:", aAudio.isOpen(), "| video track open:", aVideo.isOpen());
const ok = states.a === "connected" && aAudio.isOpen() && aVideo.isOpen();
console.log(ok ? "PKT TEST PASSED" : "PKT TEST FAILED");
pcA.close();
pcB.close();
process.exit(ok ? 0 : 1);
}
main().catch((e) => {
console.error("[pkt] FAILED:", e.message);
process.exit(1);
});
setTimeout(() => {
console.error("[pkt] TIMEOUT");
process.exit(1);
}, 25000);
@@ -0,0 +1,138 @@
// Two-peer RTP capture test: A sends REAL H264 frames through the binding's
// packetizer chain to B over localhost. tcpdump (run externally on lo) captures
// the RTP; a Python script reassembles AnnexB and ffmpeg decodes it.
//
// Usage:
// node test-rtp-capture.js <mode> mode = "a" (sender) | "b" (receiver)
// Sender writes the negotiated SDP pieces to /tmp/rtp-a.sdp /tmp/rtp-b.sdp
// Receiver listens and keeps alive.
"use strict";
const { PeerConnection } = require("./build/Release/datachannel_min.node");
const fs = require("fs");
const mode = process.argv[2] || "a";
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function main() {
const pc = new PeerConnection({ iceServers: [] });
const audio = pc.addTrack("0", "audio");
const video = pc.addTrack("1", "video");
if (mode === "a") {
// Sender: generate offer, hand to B via files, get B's answer
const offer = await pc.createOffer();
fs.writeFileSync("/tmp/rtp-offer.sdp", offer);
console.log("[a] offer written", offer.length, "bytes");
// wait for B to write its answer
for (let i = 0; i < 300; i++) {
if (fs.existsSync("/tmp/rtp-answer.sdp")) break;
await sleep(200);
}
const answer = fs.readFileSync("/tmp/rtp-answer.sdp", "utf8");
pc.setRemoteDescription(answer, "answer");
// wait connected
for (let i = 0; i < 50; i++) {
if (pc.state() === "connected") break;
await sleep(100);
}
console.log("[a] state:", pc.state());
if (pc.state() !== "connected") {
console.log("[a] FAILED not connected");
process.exit(1);
}
// Setup packetizer like the gateway does
video.setPacketizer("h264", 5678, 101, 90000, 5, 0, 10);
// Real H264 AnnexB (baseline) — read from file created by ffmpeg
const data = fs.readFileSync("/tmp/rtp-input.h264");
console.log("[a] input h264 bytes:", data.length);
// Split into NAL units by start codes, then group into access units
// the same way Demuxer does (param sets + one slice per frame).
const start3 = Buffer.from([0, 0, 1]);
const start4 = Buffer.from([0, 0, 0, 1]);
const nals = [];
let i = 0;
while (i < data.length) {
let start = -1;
let startLen = 0;
for (let j = i; j < data.length - 3; j++) {
if (data[j] === 0 && data[j + 1] === 0 && data[j + 2] === 1) {
start = j;
startLen = 3;
if (j > 0 && data[j - 1] === 0) {
start = j - 1;
startLen = 4;
}
break;
}
}
if (start === -1) break;
if (start > i) {
nals.push(data.subarray(i, start));
}
i = start + startLen;
}
console.log("[a] NALs:", nals.length);
// Group: buffer param sets, flush on slice (like Demuxer.flushAccessUnit)
let pending = [];
let frameCount = 0;
const flush = () => {
if (pending.length === 0) return;
const parts = pending.map((n) => Buffer.concat([start4, n]));
const au = Buffer.concat(parts);
pending = [];
video.sendFrame(au);
video.addTimestamp(3000); // 30fps @ 90kHz
frameCount++;
};
for (const n of nals) {
const t = n[0] & 0x1f;
if (t === 1 || t === 5) {
flush(); // previous AU closed by this slice
pending.push(n);
} else {
pending.push(n); // param set / SEI
}
}
flush();
console.log("[a] sent frames:", frameCount);
await sleep(3000); // let packets flow
console.log("[a] done");
pc.close();
process.exit(0);
} else {
// Receiver: read offer, answer, keep alive
for (let i = 0; i < 300; i++) {
if (fs.existsSync("/tmp/rtp-offer.sdp")) break;
await sleep(200);
}
const offer = fs.readFileSync("/tmp/rtp-offer.sdp", "utf8");
pc.setRemoteDescription(offer, "offer");
const answer = await pc.createAnswer(offer);
fs.writeFileSync("/tmp/rtp-answer.sdp", answer);
console.log("[b] answer written");
for (let i = 0; i < 50; i++) {
if (pc.state() === "connected") break;
await sleep(100);
}
console.log("[b] state:", pc.state());
await sleep(10000); // hold while sender streams
console.log("[b] done");
pc.close();
process.exit(0);
}
}
main().catch((e) => {
console.error("FAILED:", e.message);
process.exit(1);
});
setTimeout(() => {
console.error("TIMEOUT");
process.exit(1);
}, 30000);
@@ -0,0 +1,33 @@
// Verify addTrack produces SDP with audio+video media sections.
"use strict";
const { PeerConnection } = require("./build/Release/datachannel_min.node");
const pc = new PeerConnection({ iceServers: [] });
const audioTrack = pc.addTrack("0", "audio");
const videoTrack = pc.addTrack("1", "video");
pc.onStateChange((s) => console.log("[test-track] state:", s));
pc.createOffer().then((sdp) => {
const hasAudio = /^m=audio\s/m.test(sdp);
const hasVideo = /^m=video\s/m.test(sdp);
const audioPts = sdp.match(/a=rtpmap:(\d+) opus/g) || [];
const videoPts = sdp.match(/a=rtpmap:(\d+) H264/g) || [];
console.log("[test-track] SDP bytes:", sdp.length);
console.log("[test-track] m=audio:", hasAudio, "| m=video:", hasVideo);
console.log("[test-track] opus pt:", audioPts, "| H264 pt:", videoPts);
console.log("[test-track] audio track send ok:", typeof audioTrack.send === "function");
console.log("[test-track] video track send ok:", typeof videoTrack.send === "function");
const ok = hasAudio && hasVideo && audioPts.length > 0 && videoPts.length > 0;
console.log(ok ? "TRACK TEST PASSED" : "TRACK TEST FAILED");
pc.close();
process.exit(ok ? 0 : 1);
}).catch((e) => {
console.error("[test-track] FAILED:", e.message);
process.exit(1);
});
setTimeout(() => {
console.error("[test-track] TIMEOUT");
process.exit(1);
}, 20000);
+1 -5
View File
@@ -7,11 +7,8 @@
"pnpm": {
"onlyBuiltDependencies": [
"@discordjs/opus",
"@lng2004/node-datachannel",
"esbuild",
"node-av",
"sharp",
"zeromq"
"sharp"
]
},
"scripts": {
@@ -24,7 +21,6 @@
"test": "vitest run"
},
"dependencies": {
"@dank074/discord-video-stream": "6.0.0",
"@discordjs/opus": "^0.10.0",
"@discordjs/voice": "^0.19.2",
"@snazzah/davey": "^0.1.11",
+36 -896
View File
File diff suppressed because it is too large Load Diff
@@ -3,6 +3,7 @@ allowBuilds:
"@lng2004/node-datachannel": true
esbuild: true
node-av: true
sharp: true
zeromq: true
# pnpm 11 requires build-script approvals here (the legacy `pnpm` field in
# package.json is ignored). Native voice deps need their postinstall build.
+56 -4
View File
@@ -222,7 +222,10 @@ export async function initializeDiscordGateway() {
await initializeDatabase();
logger.info("PostgreSQL database initialized");
} catch (err) {
logger.error({ error: err }, "Failed to initialize database");
logger.error(
{ err, errorMsg: err instanceof Error ? err.message : String(err) },
"Failed to initialize database",
);
throw new DatabaseError(
`Database initialization failed: ${err instanceof Error ? err.message : String(err)}`,
);
@@ -267,7 +270,10 @@ export async function initializeDiscordGateway() {
});
client.on("error", (err) => {
logger.error({ error: err }, "Client error");
logger.error(
{ err, errorMsg: err instanceof Error ? err.message : String(err) },
"Client error",
);
});
process.on("SIGINT", () => {
@@ -279,12 +285,58 @@ export async function initializeDiscordGateway() {
});
process.on("uncaughtException", (err) => {
logger.error({ error: err }, "Uncaught exception");
const code =
typeof (err as NodeJS.ErrnoException).code === "string"
? (err as NodeJS.ErrnoException).code
: "";
// Transient stream-teardown errors (voice stop/disconnect races, child
// process stdin closed while we still write) are NOT fatal — crashing the
// gateway on EPIPE takes the whole bot offline mid-music. Log + continue.
if (
code === "EPIPE" ||
code === "ERR_STREAM_DESTROYED" ||
code === "ERR_STREAM_WRITE_AFTER_END" ||
code === "ECONNRESET"
) {
logger.warn(
{ error: err },
"Uncaught transient stream error — continuing",
);
return;
}
logger.error(
{
err,
errorMsg: err instanceof Error ? err.message : String(err),
stack: err?.stack,
},
"Uncaught exception",
);
gracefulShutdown("uncaughtException");
});
process.on("unhandledRejection", (reason, promise) => {
logger.error({ reason, promise }, "Unhandled rejection");
const err =
reason instanceof Error ? reason : new Error(String(reason ?? "unknown"));
const code = (err as NodeJS.ErrnoException).code ?? "";
// Same transient-teardown policy as uncaughtException: a rejection that
// fires while a stream is being torn down (EPIPE after ffmpeg stdin
// closes, write-after-destroy, socket reset) must NOT take the whole
// gateway offline. Log detail + continue. Everything else still shuts
// down so real bugs surface.
if (
code === "EPIPE" ||
code === "ERR_STREAM_DESTROYED" ||
code === "ERR_STREAM_WRITE_AFTER_END" ||
code === "ECONNRESET"
) {
logger.warn(
{ error: err },
"Unhandled rejection transient stream error — continuing",
);
return;
}
logger.error({ error: err, reason: String(reason) }, "Unhandled rejection");
gracefulShutdown("unhandledRejection");
});
@@ -0,0 +1,155 @@
/**
* AnnexB bitstream reader/writer (RBSP + emulation prevention) ported
* from @dank074/discord-video-stream AnnexBBitstreamReaderWriter.js.
*/
export class AnnexBBitstreamReader {
private _buffer: Uint8Array;
private _byteOffset = 0;
private _bitOffset = 0;
constructor(buffer: Uint8Array) {
this._buffer = buffer;
}
readBits(count: number): number {
if (count === 0) return 0;
let result = 0;
while (count > 0) {
if (this._byteOffset >= this._buffer.length) {
throw new Error("Bad byte offset");
}
if (
this._bitOffset === 0 &&
this._byteOffset >= 2 &&
this._buffer[this._byteOffset - 2] === 0 &&
this._buffer[this._byteOffset - 1] === 0 &&
this._buffer[this._byteOffset] === 3
) {
// Skip over emulation prevention
this._byteOffset++;
}
if (this._bitOffset === 0 && count >= 8) {
result = (result << 8) | this._buffer[this._byteOffset++];
count -= 8;
} else {
const numBitsToRead = Math.min(count, 8 - this._bitOffset);
const mask = (1 << numBitsToRead) - 1;
const newBits =
(this._buffer[this._byteOffset] >>
(8 - this._bitOffset - numBitsToRead)) &
mask;
result = (result << numBitsToRead) | newBits;
count -= numBitsToRead;
this._bitOffset += numBitsToRead;
if (this._bitOffset === 8) {
this._bitOffset = 0;
this._byteOffset++;
}
}
}
return result;
}
readUnsigned(bits: number): number {
return this.readBits(bits);
}
readSigned(bits: number): number {
const unsigned = this.readUnsigned(bits);
if (unsigned & (1 << (bits - 1))) return unsigned - (1 << bits);
return unsigned;
}
readUnsignedExpGolomb(): number {
let leading0 = 0;
while (this.readBits(1) === 0) leading0++;
return (1 << leading0) + this.readBits(leading0) - 1;
}
readSignedExpGolomb(): number {
const unsigned = this.readUnsignedExpGolomb();
if (unsigned % 2 === 0) return unsigned / -2;
return (unsigned + 1) / 2;
}
}
export class AnnexBBitstreamWriter {
private _arr: number[] = [];
private _pendingByte = 0;
private _bitOffset = 0;
toBuffer(): Buffer {
return Buffer.from(this._arr);
}
flush(): void {
// Emulation prevention: insert 0x03 before 00 00
if (
this._pendingByte <= 3 &&
this._arr[this._arr.length - 1] === 0 &&
this._arr[this._arr.length - 2] === 0
) {
this._arr.push(3);
}
this._arr.push(this._pendingByte);
this._pendingByte = 0;
this._bitOffset = 0;
}
writeBits(bits: number, count: number): void {
while (count > 0) {
if (this._bitOffset === 0) {
if (count >= 8) {
this._pendingByte = (bits >> (count - 8)) & 0xff;
count -= 8;
this.flush();
} else {
const mask = (1 << count) - 1;
this._pendingByte |= (bits & mask) << (8 - count);
this._bitOffset = count;
count = 0;
}
} else {
const numBitsToWrite = Math.min(8 - this._bitOffset, count);
const bitsToWrite =
(bits >> (count - numBitsToWrite)) & ((1 << numBitsToWrite) - 1);
this._pendingByte |=
bitsToWrite << (8 - this._bitOffset - numBitsToWrite);
count -= numBitsToWrite;
this._bitOffset += numBitsToWrite;
if (this._bitOffset === 8) {
this._bitOffset = 0;
this.flush();
}
}
}
}
writeUnsigned(num: number, count: number): void {
if (num < 0) throw new Error("Expected a non-negative number");
this.writeBits(num, count);
}
writeSigned(num: number, count: number): void {
if (count <= 0) return;
if (count > 32) throw new Error("writeSigned supports up to 32 bits");
const mask =
count === 32 ? 0xffffffff >>> 0 : (((1 << count) >>> 0) - 1) >>> 0;
const unsigned = (num & mask) >>> 0;
this.writeBits(unsigned, count);
}
writeUnsignedExpGolomb(num: number): void {
if (num < 0) throw new Error("Expected a non-negative number");
num++;
const bitCount = 32 - Math.clz32(num >>> 0);
this.writeBits(0, bitCount - 1);
this.writeBits(num, bitCount);
}
writeSignedExpGolomb(num: number): void {
if (num < 0) this.writeUnsignedExpGolomb(-2 * num);
else this.writeUnsignedExpGolomb(2 * num - 1);
}
}
@@ -0,0 +1,112 @@
/**
* H264/H265 NAL helpers ported from @dank074/discord-video-stream
* AnnexBHelper.js. Only the H264 parts are used by GoLive (H264 encoder),
* H265 constants kept for completeness of the port.
*/
export enum H264NalUnitTypes {
Unspecified = 0,
CodedSliceNonIDR = 1,
CodedSlicePartitionA = 2,
CodedSlicePartitionB = 3,
CodedSlicePartitionC = 4,
CodedSliceIdr = 5,
SEI = 6,
SPS = 7,
PPS = 8,
AccessUnitDelimiter = 9,
EndOfSequence = 10,
EndOfStream = 11,
FillerData = 12,
SEIExtenstion = 13,
PrefixNalUnit = 14,
SubsetSPS = 15,
}
export enum H265NalUnitTypes {
TRAIL_N = 0,
TRAIL_R = 1,
TSA_N = 2,
TSA_R = 3,
STSA_N = 4,
STSA_R = 5,
RADL_N = 6,
RADL_R = 7,
RASL_N = 8,
RASL_R = 9,
RSV_VCL_N10 = 10,
RSV_VCL_R11 = 11,
RSV_VCL_N12 = 12,
RSV_VCL_R13 = 13,
RSV_VCL_N14 = 14,
RSV_VCL_R15 = 15,
BLA_W_LP = 16,
BLA_W_RADL = 17,
BLA_N_LP = 18,
IDR_W_RADL = 19,
IDR_N_LP = 20,
CRA_NUT = 21,
RSV_IRAP_VCL22 = 22,
RSV_IRAP_VCL23 = 23,
RSV_VCL24 = 24,
RSV_VCL25 = 25,
RSV_VCL26 = 26,
RSV_VCL27 = 27,
RSV_VCL28 = 28,
RSV_VCL29 = 29,
RSV_VCL30 = 30,
RSV_VCL31 = 31,
VPS_NUT = 32,
SPS_NUT = 33,
PPS_NUT = 34,
AUD_NUT = 35,
EOS_NUT = 36,
EOB_NUT = 37,
FD_NUT = 38,
PREFIX_SEI_NUT = 39,
SUFFIX_SEI_NUT = 40,
}
export const H264Helpers = {
getUnitType(frame: Uint8Array): number {
return frame[0] & 0x1f;
},
splitHeader(frame: Uint8Array): [Uint8Array, Uint8Array] {
return [frame.subarray(0, 1), frame.subarray(1)];
},
isAUD(unitType: number): boolean {
return unitType === H264NalUnitTypes.AccessUnitDelimiter;
},
};
export const H265Helpers = {
getUnitType(frame: Uint8Array): number {
return (frame[0] >> 1) & 0x3f;
},
splitHeader(frame: Uint8Array): [Uint8Array, Uint8Array] {
return [frame.subarray(0, 2), frame.subarray(2)];
},
isAUD(unitType: number): boolean {
return unitType === H265NalUnitTypes.AUD_NUT;
},
};
export const startCode3 = Buffer.from([0, 0, 1]);
/** Split an AnnexB bitstream into NAL units (start codes stripped). */
export function splitNalu(buf: Buffer): Buffer[] {
let temp: Buffer | null = buf;
const nalus: Buffer[] = [];
while (temp?.byteLength) {
let pos: number = temp.indexOf(startCode3);
let length = 3;
if (pos > 0 && temp[pos - 1] === 0) {
pos--;
length++;
}
const nalu = pos === -1 ? temp : temp.subarray(0, pos);
temp = pos === -1 ? null : temp.subarray(pos + length);
if (nalu.byteLength) nalus.push(nalu);
}
return nalus;
}
@@ -0,0 +1,20 @@
/**
* AudioStream feeds encoded opus frames into the WebRTC connection.
* Ported from @dank074/discord-video-stream AudioStream.js.
*/
import { BaseMediaStream } from "./BaseMediaStream.js";
import type { WebRtcConnWrapper } from "./WebRtcWrapper.js";
export class AudioStream extends BaseMediaStream {
_conn: WebRtcConnWrapper;
constructor(conn: WebRtcConnWrapper, noSleep = false) {
super("audio", noSleep);
this._conn = conn;
}
async _sendFrame(frame: Buffer, frametime: number): Promise<void> {
this._conn.sendAudioFrame(frame, frametime);
}
}
@@ -0,0 +1,623 @@
/**
* Base media connection for Discord GoLive ported from
* @dank074/discord-video-stream BaseMediaConnection.js.
*
* Owns the voice WebSocket (identify/select_protocol/heartbeat/resume),
* SDP negotiation against Discord's media server, DAVE E2E voice
* (via @snazzah/davey), and speaking/video attribute signaling.
*/
import { randomUUID } from "node:crypto";
import { EventEmitter } from "node:events";
import Davey from "@snazzah/davey";
import { CodecPayloadType } from "./CodecPayloadType.js";
import type { NativePeerConnection } from "./native.js";
import { isNativeAvailable } from "./native.js";
import { STREAMS_SIMULCAST } from "./utils.js";
import { VoiceOpCodes, VoiceOpCodesBinary } from "./VoiceOpCodes.js";
import { WebRtcConnWrapper } from "./WebRtcWrapper.js";
export interface MediaConnectionStatus {
hasSession: boolean;
hasToken: boolean;
started: boolean;
resuming: boolean;
}
export interface VideoAttribute {
fps: number;
width: number;
height: number;
}
export interface StreamerLike {
opts: Record<string, unknown>;
}
export class BaseMediaConnection extends EventEmitter {
interval: ReturnType<typeof setInterval> | null = null;
guildId: string | null = null;
channelId: string;
botId: string;
ws: WebSocket | null = null;
status: MediaConnectionStatus;
server: string | null = null; // websocket url
token: string | null = null;
session_id: string | null = null;
protected _webRtcWrapper: WebRtcConnWrapper;
_webRtcParams: {
address: string;
port: number;
audioSsrc: number;
videoSsrc: number;
rtxSsrc: number;
supportedEncryptionModes: string[];
} | null = null;
protected _closed = false;
ready: ((conn: WebRtcConnWrapper) => void) | null;
protected _streamer: StreamerLike;
protected _sequenceNumber = -1;
protected _daveSession: Davey.DAVESession | null = null;
protected _connectedUsers = new Set<string>();
protected _daveProtocolVersion = 0;
protected _davePendingTransitions = new Map<number, number>();
protected _daveDowngraded = false;
constructor(
streamer: StreamerLike,
guildId: string | null,
botId: string,
channelId: string,
callback: ((conn: WebRtcConnWrapper) => void) | null,
) {
super();
this._streamer = streamer;
this.status = {
hasSession: false,
hasToken: false,
started: false,
resuming: false,
};
this.guildId = guildId;
this.channelId = channelId;
this.botId = botId;
this.ready = callback;
this._webRtcWrapper = new WebRtcConnWrapper(this);
}
get type(): "guild" | "call" {
return this.guildId ? "guild" : "call";
}
get webRtcConn(): WebRtcConnWrapper {
return this._webRtcWrapper;
}
get webRtcParams(): BaseMediaConnection["_webRtcParams"] {
return this._webRtcParams;
}
get streamer(): StreamerLike {
return this._streamer;
}
/** daveChannelId — overridden in VoiceConnection (channelId) and StreamConnection (serverId - 1n). */
get daveChannelId(): string {
throw new Error("daveChannelId not implemented");
}
stop(): void {
this._closed = true;
this._webRtcWrapper.close();
this.ws?.close();
}
setSession(session_id: string): void {
this.session_id = session_id;
this.status.hasSession = true;
this.start();
}
setTokens(server: string, token: string): void {
this.token = token;
this.server = server;
this.status.hasToken = true;
this.start();
}
start(): void {
if (this.status.hasSession && this.status.hasToken) {
if (this.status.started) return;
this.status.started = true;
this.ws = new WebSocket(`wss://${this.server}/?v=8`);
this.ws.binaryType = "arraybuffer";
this.ws.addEventListener("open", () => {
if (this.status.resuming) {
this.status.resuming = false;
this.resume();
} else {
this.identify();
}
});
this.ws.addEventListener("error", (err) => {
console.error(err);
});
this.ws.addEventListener("close", (e) => {
const wasStarted = this.status.started;
this.interval && clearInterval(this.interval);
this.status.started = false;
const canResume = e.code === 4015 || e.code < 4000;
if (canResume && wasStarted) {
this.status.resuming = true;
this.start();
} else {
this._closed = true;
this._webRtcWrapper?.close();
}
});
this.setupEvents();
}
}
handleReady(d: {
ip: string;
port: number;
ssrc: number;
streams: { ssrc: number; rtx_ssrc: number }[];
modes: string[];
}): void {
// we hardcoded STREAMS_SIMULCAST, which will always be array of 1
const stream = d.streams[0];
console.log(
`[goLive:${this.constructor.name}] READY ssrc=${d.ssrc} ip=${d.ip} port=${d.port} streams=${JSON.stringify(d.streams)}`,
);
this._webRtcParams = {
address: d.ip,
port: d.port,
audioSsrc: d.ssrc,
videoSsrc: stream.ssrc,
rtxSsrc: stream.rtx_ssrc,
supportedEncryptionModes: d.modes,
};
}
async handleProtocolAck(d: {
sdp?: string;
dave_protocol_version?: number;
}): Promise<void> {
if (!("sdp" in d)) throw new Error("Only WebRTC connections are allowed");
// DEBUG: dump Discord's real answer SDP — which payload types did it select?
console.log(
`[goLive:${this.constructor.name}] DISCORD_ANSWER_SDP ${JSON.stringify(d.sdp ?? "").slice(0, 900)}`,
);
this._daveProtocolVersion = d.dave_protocol_version ?? 0;
this.initDave();
// Discord's SDP is garbage — generate our own from its pieces
let ip = "";
let port = "";
let iceUsername = "";
let icePassword = "";
let fingerprint = "";
let candidate = "";
for (const line of (d.sdp ?? "").split("\n")) {
if (line.startsWith("c=")) ip = line;
else if (line.startsWith("a=rtcp")) port = line.split(":")[1];
else if (line.startsWith("a=ice-ufrag")) iceUsername = line;
else if (line.startsWith("a=ice-pwd")) icePassword = line;
else if (line.startsWith("a=fingerprint")) fingerprint = line;
else if (line.startsWith("a=candidate")) candidate = line;
}
const audioPayloadType = CodecPayloadType.opus.payload_type;
const audioSection = `
m=audio ${port} UDP/TLS/RTP/SAVPF ${audioPayloadType}
${ip}
a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=setup:passive
a=mid:0
a=maxptime:60
a=inactive
${iceUsername}
${icePassword}
${fingerprint}
${candidate}
a=rtcp-mux
a=rtpmap:${audioPayloadType} opus/48000/2
a=fmtp:${audioPayloadType} minptime=10;useinbandfec=1;usedtx=1
a=rtcp-fb:${audioPayloadType} transport-cc
a=rtcp-fb:${audioPayloadType} nack
a=ice-lite
`.trim();
const videoPayloads = Object.values(CodecPayloadType).filter(
(el) => el.type === "video",
);
const videoPayloadTypes = videoPayloads.flatMap((el) => [
el.payload_type,
el.rtx_payload_type ?? 0,
]);
const videoSection = `
m=video ${port} UDP/TLS/RTP/SAVPF ${videoPayloadTypes.join(" ")}
${ip}
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:14 urn:ietf:params:rtp-hdrext:toffset
a=extmap:13 urn:3gpp:video-orientation
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=setup:passive
a=mid:1
a=inactive
${iceUsername}
${icePassword}
${fingerprint}
${candidate}
a=rtcp-mux
a=ice-lite
`.trim();
const videoRtpMap = videoPayloads
.flatMap((el) => [
`a=rtpmap:${el.payload_type} ${el.name}/90000`,
`a=rtpmap:${el.rtx_payload_type} rtx/90000`,
`a=fmtp:${el.rtx_payload_type} apt=${el.payload_type}`,
`a=rtcp-fb:${el.payload_type} ccm fir`,
`a=rtcp-fb:${el.payload_type} nack`,
`a=rtcp-fb:${el.payload_type} nack pli`,
`a=rtcp-fb:${el.payload_type} goog-remb`,
`a=rtcp-fb:${el.payload_type} transport-cc`,
])
.join("\n");
const builtAnswer = [audioSection, videoSection, videoRtpMap].join("\n");
this._webRtcWrapper.webRtcConn?.setRemoteDescription(builtAnswer, "answer");
console.log(
`[goLive:${this.constructor.name}] SELECT_PROTOCOL_ACK processed — remote answer set (${builtAnswer.length}B) video_mline=${videoPayloadTypes.join(" ")}`,
);
this.emit("select_protocol_ack");
}
initDave(): void {
if (this._daveProtocolVersion) {
if (this._daveSession) {
this._daveSession.reinit(
this._daveProtocolVersion,
this.botId,
this.daveChannelId,
);
} else {
this._daveSession = new Davey.DAVESession(
this._daveProtocolVersion,
this.botId,
this.daveChannelId,
);
}
this.sendOpcodeBinary(
VoiceOpCodesBinary.MLS_KEY_PACKAGE,
this._daveSession.getSerializedKeyPackage(),
);
} else if (this._daveSession) {
this._daveSession.reset();
this._daveSession.setPassthroughMode(true, 10);
}
}
processInvalidCommit(transitionId: number): void {
this.sendOpcode(VoiceOpCodes.MLS_INVALID_COMMIT_WELCOME, {
transition_id: transitionId,
});
this.initDave();
}
executePendingTransition(transitionId: number): void {
const newVersion = this._davePendingTransitions.get(transitionId);
if (newVersion === undefined) {
console.error("Unrecognized transition ID", { transitionId });
return;
}
const oldVersion = this._daveProtocolVersion;
this._daveProtocolVersion = newVersion;
if (oldVersion !== newVersion && newVersion === 0) {
// Downgraded
this._daveDowngraded = true;
} else if (transitionId > 0 && this._daveDowngraded) {
this._daveDowngraded = false;
this._daveSession?.setPassthroughMode(true, 10);
}
this._davePendingTransitions.delete(transitionId);
}
setupEvents(): void {
this.ws?.addEventListener("message", async (e) => {
if (e.data instanceof ArrayBuffer) {
this.handleBinaryMessages(Buffer.from(e.data));
return;
}
const { op, d, seq } = JSON.parse(e.data as string) as {
op: number;
// biome-ignore lint/suspicious/noExplicitAny: Discord voice WS payload is dynamically typed
d: any;
seq?: number;
};
if (seq) this._sequenceNumber = seq;
if (op === VoiceOpCodes.READY) {
this.handleReady(d);
this.setProtocols()
.then(() => this.ready?.(this._webRtcWrapper))
.catch((err: unknown) => {
// PC can be closed while setProtocols is in flight (stream
// teardown) — don't let that become an unhandledRejection.
console.log(
`[goLive:${this.constructor.name}] setProtocols rejected during teardown: ${err instanceof Error ? err.message : String(err)}`,
);
});
this.setVideoAttributes(false);
} else if (op >= 4000) {
console.error(`${this.constructor.name} connection error`, d);
} else if (op === VoiceOpCodes.HELLO) {
this.setupHeartbeat(d.heartbeat_interval);
} else if (op === VoiceOpCodes.SELECT_PROTOCOL_ACK) {
await this.handleProtocolAck(d);
} else if (op === VoiceOpCodes.SPEAKING) {
// ignore speaking updates
} else if (op === VoiceOpCodes.HEARTBEAT_ACK) {
// ignore heartbeat acknowledgements
} else if (op === VoiceOpCodes.RESUMED) {
this.status.started = true;
} else if (op === VoiceOpCodes.CLIENTS_CONNECT) {
d.user_ids.forEach((id: string) => {
this._connectedUsers.add(id);
});
} else if (op === VoiceOpCodes.CLIENT_DISCONNECT) {
this._connectedUsers.delete(d.user_id);
} else if (op === VoiceOpCodes.DAVE_PREPARE_TRANSITION) {
this._davePendingTransitions.set(d.transition_id, d.protocol_version);
if (d.transition_id === 0) {
this.executePendingTransition(d.transition_id);
} else {
if (d.protocol_version === 0) {
this._daveSession?.setPassthroughMode(true, 120);
}
this.sendOpcode(VoiceOpCodes.DAVE_TRANSITION_READY, {
transition_id: d.transition_id,
});
}
} else if (op === VoiceOpCodes.DAVE_EXECUTE_TRANSITION) {
this.executePendingTransition(d.transition_id);
} else if (op === VoiceOpCodes.DAVE_PREPARE_EPOCH) {
if (d.epoch === 1) {
this._daveProtocolVersion = d.protocol_version;
this.initDave();
}
}
});
}
handleBinaryMessages(msg: Buffer): void {
this._sequenceNumber = msg.readUint16BE(0);
const op = msg.readUint8(2);
switch (op) {
case VoiceOpCodesBinary.MLS_EXTERNAL_SENDER: {
this._daveSession?.setExternalSender(msg.subarray(3));
break;
}
case VoiceOpCodesBinary.MLS_PROPOSALS: {
const optype = msg.readUint8(3);
if (!this._daveSession) break;
const { commit, welcome } = this._daveSession.processProposals(
optype,
msg.subarray(4),
[...this._connectedUsers],
);
if (commit) {
this.sendOpcodeBinary(
VoiceOpCodesBinary.MLS_COMMIT_WELCOME,
welcome ? Buffer.concat([commit, welcome]) : commit,
);
}
break;
}
case VoiceOpCodesBinary.MLS_ANNOUNCE_COMMIT_TRANSITION: {
const transitionId = msg.readUInt16BE(3);
try {
this._daveSession?.processCommit(msg.subarray(5));
if (transitionId) {
this._davePendingTransitions.set(
transitionId,
this._daveProtocolVersion,
);
this.sendOpcode(VoiceOpCodes.DAVE_TRANSITION_READY, {
transition_id: transitionId,
});
}
} catch (e) {
console.debug("MLS commit errored", e);
this.processInvalidCommit(transitionId);
}
break;
}
case VoiceOpCodesBinary.MLS_WELCOME: {
const transitionId = msg.readUInt16BE(3);
try {
this._daveSession?.processWelcome(msg.subarray(5));
if (transitionId) {
this._davePendingTransitions.set(
transitionId,
this._daveProtocolVersion,
);
this.sendOpcode(VoiceOpCodes.DAVE_TRANSITION_READY, {
transition_id: transitionId,
});
}
} catch (e) {
console.debug("MLS welcome errored", e);
this.processInvalidCommit(transitionId);
}
break;
}
}
}
get daveReady(): boolean {
return !!this._daveProtocolVersion && !!this._daveSession?.ready;
}
get daveSession(): Davey.DAVESession | null {
return this._daveSession;
}
setupHeartbeat(interval: number): void {
if (this.interval) {
clearInterval(this.interval);
}
this.interval = setInterval(() => {
try {
this.sendOpcode(VoiceOpCodes.HEARTBEAT, {
t: Date.now(),
seq_ack: this._sequenceNumber,
});
} catch {
/* ignore */
}
}, interval);
}
sendOpcode(code: number, data: unknown): void {
if (this.ws?.readyState !== WebSocket.OPEN) return;
this.ws.send(JSON.stringify({ op: code, d: data }));
}
sendOpcodeBinary(code: number, data: Uint8Array): void {
if (this.ws?.readyState !== WebSocket.OPEN) return;
const buf = Buffer.allocUnsafe(data.length + 1);
buf.writeUInt8(code);
Buffer.from(data).copy(buf, 1);
this.ws.send(buf);
}
/** serverId — overridden in VoiceConnection (guildId ?? channelId) and StreamConnection (rtc_server_id). */
get serverId(): string | null {
throw new Error("serverId not implemented");
}
/** identifies with media server with credentials */
identify(): void {
if (!this.serverId) throw new Error("Server ID is null or empty");
if (!this.session_id) throw new Error("Session ID is null or empty");
if (!this.token) throw new Error("Token is null or empty");
this.sendOpcode(VoiceOpCodes.IDENTIFY, {
server_id: this.serverId,
user_id: this.botId,
session_id: this.session_id,
token: this.token,
video: true,
streams: STREAMS_SIMULCAST,
max_dave_protocol_version: Davey.DAVE_PROTOCOL_VERSION ?? 0,
});
}
resume(): void {
if (!this.serverId) throw new Error("Server ID is null or empty");
if (!this.session_id) throw new Error("Session ID is null or empty");
if (!this.token) throw new Error("Token is null or empty");
this.sendOpcode(VoiceOpCodes.RESUME, {
server_id: this.serverId,
session_id: this.session_id,
token: this.token,
seq_ack: this._sequenceNumber,
});
}
/** Sets protocols and ip data used for video and audio (vp8 video, opus audio). */
async setProtocols(): Promise<void> {
if (!this._webRtcParams) throw new Error("WebRTC parameters not set");
if (!isNativeAvailable()) {
throw new Error(
"libdatachannel-min native binding not built — cannot start GoLive",
);
}
const reconnect = () => {
const webRtcConn = this._webRtcWrapper.initWebRtc();
webRtcConn.onStateChange((state) => {
console.log(`[goLive:${this.constructor.name}] pc state => ${state}`);
if (state === "closed" && !this._closed) reconnect();
});
this._webRtcWrapper.onLocalDescription = (sdp) => {
const rtc_connection_id = randomUUID();
console.log(
`[goLive:${this.constructor.name}] sending SELECT_PROTOCOL (offer ${sdp.length}B, rtc_connection_id=${rtc_connection_id.slice(0, 8)})`,
);
this.sendOpcode(VoiceOpCodes.SELECT_PROTOCOL, {
protocol: "webrtc",
codecs: Object.values(CodecPayloadType),
data: sdp,
sdp,
rtc_connection_id,
});
};
// createOffer (binding resolves full SDP incl. candidates after gathering)
void webRtcConn
.createOffer()
.then((sdp) => {
this._webRtcWrapper.onLocalDescription?.(sdp);
})
.catch((err: unknown) => {
// PC closed while offer is gathering (stream teardown / reconnect) —
// swallow, the reconnect loop will start a fresh offer.
console.log(
`[goLive:${this.constructor.name}] createOffer rejected: ${err instanceof Error ? err.message : String(err)}`,
);
});
};
reconnect();
return new Promise((resolve) => {
this.once("select_protocol_ack", () => resolve());
});
}
setVideoAttributes(enabled: boolean, attr?: VideoAttribute): void {
if (!this._webRtcParams) throw new Error("WebRTC parameters not set");
const { audioSsrc, videoSsrc, rtxSsrc } = this._webRtcParams;
if (!enabled) {
this.sendOpcode(VoiceOpCodes.VIDEO, {
audio_ssrc: audioSsrc,
video_ssrc: 0,
rtx_ssrc: 0,
streams: [],
});
} else {
if (!attr) throw new Error("Need to specify video attributes");
this.sendOpcode(VoiceOpCodes.VIDEO, {
audio_ssrc: audioSsrc,
video_ssrc: videoSsrc,
rtx_ssrc: rtxSsrc,
streams: [
{
type: "video",
rid: "100",
ssrc: videoSsrc,
active: true,
quality: 100,
rtx_ssrc: rtxSsrc,
// hardcode the max bitrate because we don't really know anyway
max_bitrate: 10000 * 1000,
max_framerate: enabled ? attr.fps : 0,
max_resolution: {
type: "fixed",
width: attr.width,
height: attr.height,
},
},
],
});
}
}
/** Set speaking status */
setSpeaking(speaking: boolean): void {
if (!this._webRtcParams) throw new Error("WebRTC connection not ready");
this.sendOpcode(VoiceOpCodes.SPEAKING, {
delay: 0,
speaking: speaking ? 1 : 0,
ssrc: this._webRtcParams.audioSsrc,
});
}
}
export type { NativePeerConnection };
@@ -0,0 +1,175 @@
/**
* BaseMediaStream pacing/sync for GoLive frames. Ported from
* @dank074/discord-video-stream BaseMediaStream.js, minus node-av's
* AVFrame (frames are plain objects here) and debug-level (uses the GMW
* logger instead).
*/
import { Writable } from "node:stream";
import { setTimeout as sleep } from "node:timers/promises";
export interface GoLiveFrame {
data: Buffer | null;
pts: number;
duration: number;
timeBase: { num: number; den: number };
free?: () => void;
}
export class BaseMediaStream extends Writable {
_pts: number | undefined;
_syncTolerance = 20;
_noSleep: boolean;
_startTime: number | undefined;
_startPts: number | undefined;
_sync = true;
_syncStream: BaseMediaStream | undefined;
_type: string;
constructor(type: string, noSleep = false) {
super({ objectMode: true, highWaterMark: 0 });
this._type = type;
this._noSleep = noSleep;
}
get sync(): boolean {
return this._sync;
}
set sync(val: boolean) {
this._sync = val;
}
get syncStream(): BaseMediaStream | undefined {
return this._syncStream;
}
set syncStream(stream: BaseMediaStream | undefined) {
if (stream !== undefined && this === stream.syncStream) {
throw new Error("Cannot sync 2 streams with eachother");
}
this._syncStream = stream;
}
get noSleep(): boolean {
return this._noSleep;
}
set noSleep(val: boolean) {
this._noSleep = val;
if (!val) this.resetTimingCompensation();
}
get pts(): number | undefined {
return this._pts;
}
get syncTolerance(): number {
return this._syncTolerance;
}
set syncTolerance(n: number) {
if (n < 0) return;
this._syncTolerance = n;
}
async _sendFrame(_frame: Buffer, _frametime: number): Promise<void> {
throw new Error("Not implemented");
}
ptsDelta(): number | undefined {
if (this.pts !== undefined && this.syncStream?.pts !== undefined) {
return this.pts - this.syncStream.pts;
}
return undefined;
}
isAhead(): boolean {
const delta = this.ptsDelta();
return (
this.syncStream?.writableEnded === false &&
delta !== undefined &&
delta > this.syncTolerance
);
}
isBehind(): boolean {
const delta = this.ptsDelta();
return (
this.syncStream?.writableEnded === false &&
delta !== undefined &&
delta < -this.syncTolerance
);
}
resetTimingCompensation(): void {
this._startTime = this._startPts = undefined;
}
async _write(
frame: GoLiveFrame,
_encoding: BufferEncoding,
callback: (error?: Error | null) => void,
): Promise<void> {
const { data, pts, duration, timeBase } = frame;
if (!data) {
frame.free?.();
callback();
return;
}
const frametime = (Number(duration) / timeBase.den) * timeBase.num * 1000;
const start_sendFrame = performance.now();
await this._sendFrame(Buffer.from(data), frametime);
const end_sendFrame = performance.now();
this._pts = (Number(pts) / timeBase.den) * timeBase.num * 1000;
this.emit("pts", this._pts);
const sendTime = end_sendFrame - start_sendFrame;
const ratio = sendTime / frametime;
if (ratio > 1) {
// Frame takes longer to send than its frametime — warn once per 100
if (
this._lastWarnedRatio === undefined ||
ratio > this._lastWarnedRatio
) {
this._lastWarnedRatio = ratio;
}
}
this._startTime ??= start_sendFrame;
this._startPts ??= this._pts;
const sleepMs = Math.max(
0,
this._pts -
this._startPts +
frametime -
(end_sendFrame - this._startTime),
);
if (this._noSleep || sleepMs === 0) {
callback(null);
} else if (this.sync && this.isBehind()) {
// Stream is behind — don't sleep for this frame
this.resetTimingCompensation();
callback(null);
} else if (this.sync && this.isAhead()) {
// Stream is ahead — wait until the sync stream catches up
do {
await sleep(frametime);
} while (this.sync && this.isAhead());
this.resetTimingCompensation();
callback(null);
} else {
await sleep(sleepMs);
callback(null);
}
frame.free?.();
}
_lastWarnedRatio: number | undefined;
_destroy(
error: Error | null,
callback: (error?: Error | null) => void,
): void {
super._destroy(error, callback);
this.syncStream = undefined;
}
}
@@ -0,0 +1,71 @@
/** Payload types for Discord GoLive media — ported from @dank074/discord-video-stream. */
export interface CodecPayloadTypeEntry {
name: string;
type: "audio" | "video";
clockRate: number;
priority: number;
payload_type: number;
rtx_payload_type?: number;
encode?: boolean;
decode?: boolean;
}
export const CodecPayloadType: Record<string, CodecPayloadTypeEntry> = {
opus: {
name: "opus",
type: "audio",
clockRate: 48000,
priority: 1000,
payload_type: 120,
},
H264: {
name: "H264",
type: "video",
clockRate: 90000,
priority: 1000,
payload_type: 101,
rtx_payload_type: 102,
encode: true,
decode: true,
},
H265: {
name: "H265",
type: "video",
clockRate: 90000,
priority: 1000,
payload_type: 103,
rtx_payload_type: 104,
encode: true,
decode: true,
},
VP8: {
name: "VP8",
type: "video",
clockRate: 90000,
priority: 1000,
payload_type: 105,
rtx_payload_type: 106,
encode: true,
decode: true,
},
VP9: {
name: "VP9",
type: "video",
clockRate: 90000,
priority: 1000,
payload_type: 107,
rtx_payload_type: 108,
encode: true,
decode: true,
},
AV1: {
name: "AV1",
type: "video",
clockRate: 90000,
priority: 1000,
payload_type: 109,
rtx_payload_type: 110,
encode: true,
decode: true,
},
};
@@ -0,0 +1,600 @@
/**
* Lightweight demuxer replaces node-av's LibavDemuxer for GoLive.
*
* Spawns ffmpeg to remux input into H264 AnnexB on stdout (video only
* screen share doesn't need to mux audio into the demuxer; audio goes
* separately). This replaces the 114MB node-av binary with a plain ffmpeg
* spawn.
*
* Each video "frame" emitted is a complete NAL sequence terminated by a
* keyframe boundary (IDR). Audio is not extracted here for GoLive with
* audio, the NUT mux + full demuxer would be needed; screen share audio is
* handled via a separate ffmpeg instance (see getDirectScreenInput).
*/
import { spawn } from "node:child_process";
import { existsSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { PassThrough } from "node:stream";
import type { GoLiveFrame } from "./BaseMediaStream.js";
/** 4-byte AnnexB start code (00 00 00 01) used when building access units. */
const startCode4 = Buffer.from([0, 0, 0, 1]);
/**
* Resolve ffmpeg/ffprobe binary. Prefers explicit env override, then PATH,
* then a Nix-store ffmpeg-headless (the GMW flake provides it in the service
* profile, but dev shells / tests may not have it on PATH).
*/
function resolveBin(name: "ffmpeg"): string {
const override = process.env.FFMPEG_PATH;
if (override && existsSync(override)) return override;
// Nix store scan: <store>/<hash>-ffmpeg-headless-*/bin/<name>
const store = "/nix/store";
if (existsSync(store)) {
const entries = readdirSync(store);
for (const entry of entries) {
if (!entry.includes("ffmpeg-headless-")) continue;
const candidate = join(store, entry, "bin", name);
if (existsSync(candidate)) return candidate;
}
}
return name; // fall back to PATH
}
const FFMPEG = resolveBin("ffmpeg");
export const AVCodecID = {
AV_CODEC_ID_H264: 27,
AV_CODEC_ID_HEVC: 173,
AV_CODEC_ID_VP8: 139,
AV_CODEC_ID_VP9: 167,
AV_CODEC_ID_AV1: 225,
AV_CODEC_ID_OPUS: 86019,
} as const;
export type AVCodecID = (typeof AVCodecID)[keyof typeof AVCodecID];
export const AV_PKT_FLAG_KEY = 1;
export interface Frame {
data: Buffer | null;
pts: number;
duration: number;
timeBase: { num: number; den: number };
flags: number;
streamIndex: number;
free(): void;
}
export interface DemuxedStream {
codec: number;
codecName: string;
width: number;
height: number;
framerate_num: number;
framerate_den: number;
sample_rate: number;
stream: PassThrough;
}
/**
* Probe a media file for stream info using ffmpeg's stderr (the
* ffmpeg-headless Nix package ships ffmpeg but not ffprobe). Returns
* stream descriptors in the same shape ffprobe -show_streams would.
*/
export async function probeStreams(
url: string,
): Promise<Array<Record<string, unknown>>> {
return new Promise((resolve, reject) => {
const proc = spawn(FFMPEG, [
"-hide_banner",
"-loglevel",
"info",
"-i",
url,
"-f",
"null",
"-",
]);
let stderr = "";
proc.stderr.on("data", (d: Buffer) => (stderr += d.toString()));
proc.on("close", () => {
// Parse "Stream #0:0: Video: h264 (High), yuv420p, 640x360, 30 fps"
const streams: Array<Record<string, unknown>> = [];
const re = /Stream #0:(\d+): (Video|Audio): ([^,]+)/g;
let m: RegExpExecArray | null;
// biome-ignore lint/suspicious/noAssignInExpressions: regex loop idiom
while ((m = re.exec(stderr)) !== null) {
const [full, idx, kind, codecRaw] = m;
void full;
const codecName = codecRaw.split(" ")[0].toLowerCase();
const stream: Record<string, unknown> = {
index: Number(idx),
codec_type: kind.toLowerCase(),
codec_name: codecName,
width: 0,
height: 0,
r_frame_rate: "0/1",
sample_rate: 0,
};
// dimensions: "640x360"
const dim = /(\d{2,5})x(\d{2,5})/.exec(stderr.slice(m.index));
if (dim) {
stream.width = Number(dim[1]);
stream.height = Number(dim[2]);
}
// fps: "30 fps" or "29.97 fps"
const fps = /(\d+(?:\.\d+)?) fps/.exec(stderr.slice(m.index));
if (fps) {
const v = Number(fps[1]);
stream.r_frame_rate = `${Math.round(v * 1000)}/1000`;
}
// sample rate for audio: "48000 Hz"
const sr = /(\d+) Hz/.exec(stderr.slice(m.index));
if (sr) stream.sample_rate = Number(sr[1]);
streams.push(stream);
}
resolve(streams);
});
proc.on("error", (err) => reject(err));
});
}
/**
* Demux input (URL string or readable stream) into video frames on a
* PassThrough. Streams input DIRECTLY into ffmpeg (no spool-to-file the
* live NUT/H264 source never ends, so spooling deadlocks). ffmpeg emits
* AnnexB H264 on stdout; NAL units are split into frames on the fly.
* Video metadata is parsed from ffmpeg stderr during init.
*/
export async function demux(
input: string | PassThrough,
opts: { format: string; frameRate?: number },
): Promise<{
video: DemuxedStream | undefined;
audio: DemuxedStream | undefined;
close: () => void;
}> {
const vPipe = new PassThrough({ objectMode: true, highWaterMark: 128 });
const aPipe = new PassThrough({ objectMode: true, highWaterMark: 128 });
const isStream = typeof input !== "string";
// NUT/matroska input (prepareStream with includeAudio) carries audio; the
// h264 path is video-only raw AnnexB. Video always goes to stdout (pipe:1);
// audio goes to fd3 (pipe:3) so stderr stays free for metadata parsing.
const containerFormat =
isStream && opts.format !== "h264" ? opts.format : null;
const withAudio = isStream && containerFormat !== null;
const args: string[] = [
"-hide_banner",
// info level: stream init lines ("Stream #0:0: Video: h264...") go to
// stderr and are parsed for dimensions/fps.
"-loglevel",
"info",
// Input format hint: raw H264 has NO magic header, so ffmpeg's
// auto-detection fails with "Invalid data found when processing input"
// whenever the first bytes arrive late/buffered. Pin the demuxer input
// format for streams (NUT for the audio-capable path).
...(withAudio
? ["-f", containerFormat as string]
: isStream
? ["-f", "h264"]
: []),
"-i",
isStream ? "pipe:0" : input,
"-map",
"0:v:0",
"-c:v",
"copy",
"-f",
"h264",
"pipe:1",
...(withAudio
? ["-map", "0:a:0?", "-c:a", "copy", "-f", "opus", "pipe:3"]
: ["-an"]),
];
const proc = spawn(FFMPEG, args, {
stdio: isStream
? withAudio
? ["pipe", "pipe", "pipe", "pipe"]
: ["pipe", "pipe", "pipe"]
: ["ignore", "pipe", "pipe"],
});
console.log(
`[goLive:Demuxer] spawn ffmpeg pid=${proc.pid} input=${isStream ? "stream" : input} args=${args.join(" ")}`,
);
// Pipe live input straight into ffmpeg stdin — never await stream end.
if (isStream && proc.stdin) {
input.pipe(proc.stdin);
input.on("error", () => proc.stdin?.destroy());
}
// Audio: ffmpeg writes Ogg Opus on fd3 (pipe:3). Parse OGG pages into
// opus packets and emit them as GoLiveFrames (20ms, 48kHz) on aPipe.
if (withAudio && proc.stdio[3]) {
createOggOpusDemux(
proc.stdio[3] as unknown as NodeJS.ReadableStream,
aPipe,
);
console.log("[goLive:Demuxer] audio pipe wired (fd3 → Ogg Opus → aPipe)");
}
// Set true once stderr metadata has been parsed (see handler below).
let parsedMeta = false;
// Track which stream kinds we've seen. We must NOT stop parsing on the
// first stream found: ffmpeg can print the video line and audio line in
// separate stderr chunks (input arrives slowly), and the old
// early-return (`if (parsedMeta) return`) dropped the audio line forever
// → aInfo undefined → no audio RTP → static GoLive tile.
let seenVideo = false;
let seenAudio = false;
// Parse stream metadata from ffmpeg stderr as it arrives (first chunk has
// the init lines). Fall back to H264 defaults if parsing fails.
let vInfo: DemuxedStream = {
codec: AVCodecID.AV_CODEC_ID_H264,
codecName: "h264",
width: 0,
height: 0,
framerate_num: 0,
framerate_den: 1,
sample_rate: 0,
stream: vPipe,
};
let aInfo: DemuxedStream | undefined;
// With audio expected (NUT input), ALWAYS expose an audio stream even if
// ffmpeg's audio init line hasn't arrived in stderr yet. prepareStream
// encodes libopus into the NUT unconditionally (`-map 0:a:0? -c:a libopus`),
// so fd3 WILL carry Ogg Opus — aInfo must not stay undefined just because
// the metadata line raced the resolve. The stderr handler below upgrades
// this default with real sample_rate metadata when the line lands.
if (withAudio) {
aInfo = {
codec: AVCodecID.AV_CODEC_ID_OPUS,
codecName: "opus",
width: 0,
height: 0,
framerate_num: 0,
framerate_den: 0,
sample_rate: 48000,
stream: aPipe,
};
}
let stderrBuf = "";
if (proc.stderr) {
proc.stderr.on("data", (d: Buffer) => {
const text = d.toString();
stderrBuf = (stderrBuf + text).slice(-16384);
// Surface actionable lines: ffmpeg errors + stream init lines
if (/error|invalid|no such|failed|cannot|not found|unable/i.test(text)) {
console.log(
`[goLive:Demuxer] ffmpeg stderr: ${text.trim().split("\n").slice(0, 4).join(" | ")}`,
);
}
const streamRe = /Stream #0:(\d+): (Video|Audio): ([^,]+)/g;
let m: RegExpExecArray | null;
const found: Array<{ kind: string; codecRaw: string }> = [];
// biome-ignore lint/suspicious/noAssignInExpressions: regex loop idiom
while ((m = streamRe.exec(stderrBuf)) !== null) {
found.push({ kind: m[2], codecRaw: m[3] });
}
if (process.env.GMW_DEMUX_DEBUG) {
console.log(
`[goLive:Demuxer] DEBUG stderrBuf=${JSON.stringify(stderrBuf.slice(0, 300))} found=${JSON.stringify(found)}`,
);
}
const v = found.find((s) => s.kind === "Video");
const a = found.find((s) => s.kind === "Audio");
if (v) {
seenVideo = true;
const codecName = v.codecRaw.split(" ")[0].toLowerCase();
const dim = /(\d{2,5})x(\d{2,5})/.exec(stderrBuf);
const fps = /(\d+(?:\.\d+)?) fps/.exec(stderrBuf);
vInfo = {
codec:
AVCodecID[
(codecName.toUpperCase() as keyof typeof AVCodecID) ??
"AV_CODEC_ID_H264"
] ?? AVCodecID.AV_CODEC_ID_H264,
codecName,
width: dim ? Number(dim[1]) : 0,
height: dim ? Number(dim[2]) : 0,
framerate_num: fps ? Math.round(Number(fps[1]) * 1000) : 0,
framerate_den: fps ? 1000 : 1,
sample_rate: 0,
stream: vPipe,
};
}
if (a) {
seenAudio = true;
const codecName = a.codecRaw.split(" ")[0].toLowerCase();
const sr = /(\d+) Hz/.exec(stderrBuf);
aInfo = {
codec:
AVCodecID[
(codecName.toUpperCase() as keyof typeof AVCodecID) ??
"AV_CODEC_ID_OPUS"
] ?? AVCodecID.AV_CODEC_ID_OPUS,
codecName,
width: 0,
height: 0,
framerate_num: 0,
framerate_den: 0,
sample_rate: sr ? Number(sr[1]) : 0,
stream: aPipe,
};
}
// Mark that at least one stream kind was seen. Note: we must NOT
// resolve the metadata wait on the FIRST stream kind alone. With live
// NUT input, ffmpeg can print the video init line in one stderr chunk
// and the audio init line in the NEXT chunk (NUT info-stream packets
// arrive as ffmpeg reads them from the pipe). The old code returned
// immediately on `parsedMeta=true` — the audio line then landed in the
// handler AFTER `return { audio: aInfo }` had already captured
// `undefined` → no audio RTP → static GoLive tile even though the NUT
// carried audio. Wait for BOTH kinds (when audio is expected).
if (seenVideo || seenAudio) parsedMeta = true;
});
}
// Wait (briefly) for ffmpeg to print its stream init lines on stderr so
// vInfo/aInfo carry real metadata. With audio expected, wait for BOTH the
// video and audio init lines (they may arrive in separate stderr chunks on
// live input); the timeout covers slow starts / genuinely audio-less input.
const allSeen = () =>
withAudio ? seenVideo && seenAudio : seenVideo || seenAudio;
await Promise.race([
new Promise<void>((resolve) => {
const check = setInterval(() => {
if (allSeen()) {
clearInterval(check);
resolve();
}
}, 25);
}),
new Promise<void>((resolve) => setTimeout(resolve, 3000)),
]);
// Scan stdout for AnnexB NAL units and group them into ACCESS UNITS
// (one picture). Discord's H264 decoder requires a complete access unit —
// parameter sets + slice — inside a single RTP frame. Emitting each NAL
// as its own frame (SPS/PPS/SEI separate from the slice) makes the decoder
// unable to produce ANY picture: production showed a black GoLive tile
// despite frames flowing (5892B slices + 4B PPS + 33B SPS as separate
// frames, each with a near-zero RTP timestamp delta). We therefore buffer
// NALs and flush one frame per slice, prepending the parameter sets that
// precede it, and timestamp it as ONE frame at the video frame rate.
let videoBuf = Buffer.alloc(0);
let frameCount = 0;
let pendingNals: Buffer[] = [];
let pendingHasSlice = false;
let pendingIsKey = false;
// Raw H264 streams carry no timing info — ffmpeg's h264 demuxer guesses
// 25fps on stderr. Prefer the caller's explicit frameRate (the encode
// setting); it drives both RTP timestamp advance and pacing.
const videoFps =
opts.frameRate ?? (vInfo.framerate_num / vInfo.framerate_den || 30);
const flushAccessUnit = () => {
if (pendingNals.length === 0) return;
// AnnexB access unit: 00 00 00 01 + NAL for every buffered NAL. The
// packetizer (H264RtpPacketizer, StartSequence separator) needs the
// start codes to find NAL boundaries inside the frame.
const parts: Buffer[] = [];
for (const n of pendingNals) parts.push(startCode4, n);
const au = Buffer.concat(parts);
const isKey = pendingIsKey;
pendingNals = [];
pendingHasSlice = false;
pendingIsKey = false;
vPipe.write({
data: au,
// One frame at videoFps: duration=1 in a 1/fps timebase →
// BaseMediaStream computes frametime=1000/fps ms → the RTP timestamp
// advances clockRate/fps per frame (3000 @ 30fps / 90kHz), which is
// what Discord's receiver expects for real-time video.
pts: frameCount,
duration: 1,
timeBase: { num: 1, den: videoFps },
flags: isKey ? AV_PKT_FLAG_KEY : 0,
streamIndex: 0,
free: () => {},
});
frameCount++;
if (frameCount === 1 || frameCount % 30 === 0) {
console.log(
`[goLive:Demuxer] frames=${frameCount} last=${au.length}B key=${isKey}`,
);
}
};
if (proc.stdout) {
proc.stdout.on("data", (chunk: Buffer) => {
videoBuf = Buffer.concat([videoBuf, chunk]);
// Find start codes (00 00 01 or 00 00 00 01) and split NALs
let start = 0;
// If buffer starts with zeros, that's the first start code — emit from there
while (start < videoBuf.length) {
let scPos = -1;
for (let i = start + 1; i < videoBuf.length - 2; i++) {
if (
videoBuf[i] === 0 &&
videoBuf[i + 1] === 0 &&
videoBuf[i + 2] === 1
) {
scPos = i + 3;
break;
}
}
if (scPos === -1) break;
// Emit the NAL from `start` to `scPos` (but skip the start code bytes at `start`)
if (start < scPos) {
let nalStart = start;
// Skip start code bytes for the NAL itself (00 00 01)
if (
videoBuf[nalStart] === 0 &&
videoBuf[nalStart + 1] === 0 &&
videoBuf[nalStart + 2] === 1
) {
nalStart += 3;
} else if (
nalStart + 3 < scPos &&
videoBuf[nalStart] === 0 &&
videoBuf[nalStart + 1] === 0 &&
videoBuf[nalStart + 2] === 0 &&
videoBuf[nalStart + 3] === 1
) {
nalStart += 4;
}
const nal = videoBuf.subarray(nalStart, scPos);
// Trim trailing zero bytes (from start code overlap)
let end = nal.length;
while (end > 0 && nal[end - 1] === 0) end--;
if (end > 0) {
const nalTrimmed = nal.subarray(0, end);
const nalType = nalTrimmed[0] & 0x1f;
const isSlice = nalType === 1 || nalType === 5;
if (isSlice) {
// A new slice while one is pending closes the previous
// access unit (x264 emits one slice per frame).
if (pendingHasSlice) flushAccessUnit();
pendingNals.push(Buffer.from(nalTrimmed));
pendingHasSlice = true;
if (nalType === 5) pendingIsKey = true;
} else {
// Parameter-set / SEI / AUD / filler NAL. After a slice these
// belong to the NEXT access unit — flush the completed frame.
if (pendingHasSlice) flushAccessUnit();
pendingNals.push(Buffer.from(nalTrimmed));
}
}
}
// Skip the 00 00 01 at scPos-3 to find next
start = scPos;
// But the next start code needs at least 3 bytes
if (start > videoBuf.length - 3) break;
}
// Keep remaining bytes (potential partial NAL or start code)
if (start > 0 && start < videoBuf.length) {
videoBuf = videoBuf.subarray(start);
} else if (videoBuf.length > 4) {
// No full NAL found, but avoid unbounded growth
// Keep a sliding window
videoBuf = videoBuf.subarray(videoBuf.length - 3);
}
});
proc.stdout.on("end", () => {
flushAccessUnit();
vPipe.end();
aPipe.end();
});
}
proc.on("close", () => {
vPipe.end();
aPipe.end();
});
const close = () => {
proc.kill("SIGTERM");
vPipe.end();
aPipe.end();
};
return { video: vInfo, audio: aInfo, close };
}
/**
* Parse an Ogg Opus byte stream (as written by ffmpeg's `-f opus` muxer)
* into individual opus packets and push them onto `out` as GoLiveFrames
* (duration 960 @ 48kHz = 20ms, matching the OpusRtpPacketizer clock).
*
* OGG page structure:
* "OggS" | ver(1) | header_type(1) | granule(8 LE) | serial(4) | seq(4) |
* crc(4) | page_segments(1) | segment_table[n] | payload
* Lacing: a value < 255 ends a packet; 255 continues it (0.5KB chunk).
* The first packet is OpusHead (19B) skipped, as is OpusTags.
*/
function createOggOpusDemux(
input: NodeJS.ReadableStream,
out: PassThrough,
): void {
let buf = Buffer.alloc(0);
// Packets assembled from lacing; packetParts accumulates across pages
// when a packet spans a page boundary (continued flag / 255 lacing).
let packetParts: Buffer[] = [];
let headerDone = false;
let frameIndex = 0;
const emitPacket = (packet: Buffer) => {
if (!headerDone) {
// First packet = OpusHead ("OpusHead"), second = OpusTags. Skip both.
const magic = packet.toString("latin1", 0, 8);
if (magic === "OpusHead" || magic === "OpusTags") return;
headerDone = true;
}
out.write({
data: packet,
pts: frameIndex * 960,
duration: 960,
timeBase: { num: 1, den: 48000 },
free: () => {},
} satisfies GoLiveFrame);
frameIndex++;
};
const processPages = () => {
while (true) {
// Sync to "OggS"
const sync = buf.indexOf("OggS", 0, "latin1");
if (sync === -1) {
// Keep the tail (partial sync pattern) for the next chunk
buf = buf.length > 3 ? buf.subarray(buf.length - 3) : buf;
return;
}
if (sync > 0) buf = buf.subarray(sync);
if (buf.length < 27) return; // need full page header
const numSeg = buf[26];
if (buf.length < 27 + numSeg) return; // need segment table
let payloadLen = 0;
for (let i = 0; i < numSeg; i++) payloadLen += buf[27 + i];
if (buf.length < 27 + numSeg + payloadLen) return; // need payload
const headerType = buf[5];
// Extract packets from the payload using lacing values
let off = 27 + numSeg;
for (let i = 0; i < numSeg; i++) {
const lace = buf[27 + i];
const part = buf.subarray(off, off + lace);
off += lace;
packetParts.push(Buffer.from(part));
if (lace < 255) {
const packet = Buffer.concat(packetParts);
packetParts = [];
if ((headerType & 0x01) === 0) {
// Not a continuation page → packet starts here
emitPacket(packet);
} else if (headerDone) {
// Continued page — packet body, emit directly
emitPacket(packet);
}
// (header packets on continuation pages are dropped)
}
}
buf = buf.subarray(off);
if (buf.length === 0) return;
}
};
input.on("data", (chunk: Buffer) => {
buf = Buffer.concat([buf, chunk]);
processPages();
});
input.on("end", () => {
out.end();
});
input.on("error", () => {
out.end();
});
}
@@ -0,0 +1,69 @@
/**
* Lightweight encoders config ported from @dank074/discord-video-stream
* encoders/software.js. Only software (libx264) is needed for GoLive.
*/
export interface EncoderSettings {
name: string;
options: string[];
outFilters?: string[];
globalOptions?: string[];
}
export interface EncoderSet {
H264: EncoderSettings;
H265: EncoderSettings;
VP8: EncoderSettings;
VP9: EncoderSettings;
AV1: EncoderSettings;
}
/** Software x264 encoder. Matches @dank074's software() defaults. */
export function software(
opts: {
x264?: { preset?: string; tune?: string };
x265?: { preset?: string; tune?: string };
} = {},
): () => EncoderSet {
const { x264, x265 } = opts;
const { preset: x264Preset = "superfast", tune: x264Tune = "zerolatency" } =
x264 ?? {};
const { preset: x265Preset = "superfast", tune: x265Tune } = x265 ?? {};
return () => ({
H264: {
name: "libx264",
// -profile:v baseline is REQUIRED: the SDP advertises
// profile-level-id=42e01f (constrained baseline) and Discord's
// receiver decodes with that profile. x264's default is High — a
// High-profile bitstream against a baseline SDP negotiation fails to
// decode → black GoLive tile (production bug, fixed 2026-08-12).
// zerolatency matches @dank074 (no lookahead — correct for live).
options: [
"-forced-idr 1",
"-profile:v baseline",
// repeat-headers: SPS/PPS inline BEFORE EVERY IDR, not just the
// first. Required for the NUT container path (NUT stores extradata
// in the header and `-c:v copy` remux loses it → decoder sees
// "non-existing PPS 0 referenced" → no picture) and lets Discord's
// decoder recover after any PLI/keyframe request mid-stream.
"-x264-params",
"repeat-headers=1",
`-tune ${x264Tune}`,
`-preset ${x264Preset}`,
],
},
H265: {
name: "libx265",
options: [
"-forced-idr 1",
...(x265Tune ? [`-tune ${x265Tune}`] : []),
`-preset ${x265Preset}`,
],
},
VP8: { name: "libvpx", options: ["-deadline 20000"] },
VP9: { name: "libvpx-vp9", options: ["-deadline 20000"] },
AV1: { name: "libsvtav1", options: [] },
});
}
export const Encoders = { software };
@@ -0,0 +1,41 @@
/** Discord gateway opcodes used by Streamer — ported from @dank074/discord-video-stream. */
export enum GatewayOpCodes {
DISPATCH = 0,
HEARTBEAT = 1,
IDENTIFY = 2,
PRESENCE_UPDATE = 3,
VOICE_STATE_UPDATE = 4,
VOICE_SERVER_PING = 5,
RESUME = 6,
RECONNECT = 7,
REQUEST_GUILD_MEMBERS = 8,
INVALID_SESSION = 9,
HELLO = 10,
HEARTBEAT_ACK = 11,
CALL_CONNECT = 13,
GUILD_SUBSCRIPTIONS = 14,
LOBBY_CONNECT = 15,
LOBBY_DISCONNECT = 16,
LOBBY_VOICE_STATES_UPDATE = 17,
STREAM_CREATE = 18,
STREAM_DELETE = 19,
STREAM_WATCH = 20,
STREAM_PING = 21,
STREAM_SET_PAUSED = 22,
REQUEST_GUILD_APPLICATION_COMMANDS = 24,
EMBEDDED_ACTIVITY_LAUNCH = 25,
EMBEDDED_ACTIVITY_CLOSE = 26,
EMBEDDED_ACTIVITY_UPDATE = 27,
REQUEST_FORUM_UNREADS = 28,
REMOTE_COMMAND = 29,
GET_DELETED_ENTITY_IDS_NOT_MATCHING_HASH = 30,
REQUEST_SOUNDBOARD_SOUNDS = 31,
SPEED_TEST_CREATE = 32,
SPEED_TEST_DELETE = 33,
REQUEST_LAST_MESSAGES = 34,
SEARCH_RECENT_MEMBERS = 35,
REQUEST_CHANNEL_STATUSES = 36,
GUILD_SUBSCRIPTIONS_BULK = 37,
GUILD_CHANNELS_RESYNC = 38,
REQUEST_CHANNEL_MEMBER_COUNT = 39,
}
@@ -0,0 +1,291 @@
/**
* H264 SPS VUI rewriter ported from @dank074/discord-video-stream
* SPSVUIRewriter.js. Rewrites the SPS so Discord's receiver applies
* bitstream restrictions (max_num_reorder_frames=0, max_dec_frame_buffering
* bounded) required for low-latency GoLive decode.
*/
import {
AnnexBBitstreamReader,
AnnexBBitstreamWriter,
} from "./AnnexBBitstreamReaderWriter.js";
export function rewriteSPSVUI(buffer: Uint8Array): Buffer {
const reader = new AnnexBBitstreamReader(buffer.subarray(1));
const writer = new AnnexBBitstreamWriter();
const readBit = (n = 1) => reader.readBits(n);
const writeBit = (v: number, n = 1) => writer.writeBits(v, n);
const readU = (n: number) => reader.readUnsigned(n);
const writeU = (v: number, n: number) => writer.writeUnsigned(v, n);
const readUE = () => reader.readUnsignedExpGolomb();
const writeUE = (v: number) => writer.writeUnsignedExpGolomb(v);
const readSE = () => reader.readSignedExpGolomb();
const writeSE = (v: number) => writer.writeSignedExpGolomb(v);
// Rewrite the NAL header
writeU(buffer[0], 8);
const profile_idc = readU(8);
writeU(profile_idc, 8);
const constraint_flags = readU(8);
writeU(constraint_flags, 8);
const level_idc = readU(8);
writeU(level_idc, 8);
const seq_parameter_set_id = readUE();
writeUE(seq_parameter_set_id);
// If profile in high profiles, additional fields
const highProfiles = new Set([
100, 110, 122, 244, 44, 83, 86, 118, 128, 138, 144,
]);
if (highProfiles.has(profile_idc)) {
const chroma_format_idc = readUE();
writeUE(chroma_format_idc);
if (chroma_format_idc === 3) {
const separate_colour_plane_flag = readBit(1);
writeBit(separate_colour_plane_flag, 1);
}
const bit_depth_luma_minus8 = readUE();
writeUE(bit_depth_luma_minus8);
const bit_depth_chroma_minus8 = readUE();
writeUE(bit_depth_chroma_minus8);
const qpprime_y_zero_transform_bypass_flag = readBit(1);
writeBit(qpprime_y_zero_transform_bypass_flag, 1);
const seq_scaling_matrix_present_flag = readBit(1);
writeBit(seq_scaling_matrix_present_flag, 1);
if (seq_scaling_matrix_present_flag) {
const scalingCount = chroma_format_idc !== 3 ? 8 : 12;
for (let i = 0; i < scalingCount; i++) {
const seq_scaling_list_present_flag = readBit(1);
writeBit(seq_scaling_list_present_flag, 1);
if (seq_scaling_list_present_flag) {
const size = i < 6 ? 16 : 64;
let lastScale = 8;
let nextScale = 8;
for (let j = 0; j < size; j++) {
const delta = readSE();
writeSE(delta);
nextScale = (lastScale + delta + 256) % 256;
if (nextScale !== 0) lastScale = nextScale;
}
}
}
}
}
const log2_max_frame_num_minus4 = readUE();
writeUE(log2_max_frame_num_minus4);
const pic_order_cnt_type = readUE();
writeUE(pic_order_cnt_type);
if (pic_order_cnt_type === 0) {
const log2_max_pic_order_cnt_lsb_minus4 = readUE();
writeUE(log2_max_pic_order_cnt_lsb_minus4);
} else if (pic_order_cnt_type === 1) {
const delta_pic_order_always_zero_flag = readBit(1);
writeBit(delta_pic_order_always_zero_flag, 1);
const offset_for_non_ref_pic = readSE();
writeSE(offset_for_non_ref_pic);
const offset_for_top_to_bottom_field = readSE();
writeSE(offset_for_top_to_bottom_field);
const num_ref_frames_in_pic_order_cnt_cycle = readUE();
writeUE(num_ref_frames_in_pic_order_cnt_cycle);
for (let i = 0; i < num_ref_frames_in_pic_order_cnt_cycle; i++) {
const offset_for_ref_frame = readSE();
writeSE(offset_for_ref_frame);
}
}
const max_num_ref_frames = readUE();
writeUE(max_num_ref_frames);
const gaps_in_frame_num_value_allowed_flag = readBit(1);
writeBit(gaps_in_frame_num_value_allowed_flag, 1);
const pic_width_in_mbs_minus1 = readUE();
writeUE(pic_width_in_mbs_minus1);
const pic_height_in_map_units_minus1 = readUE();
writeUE(pic_height_in_map_units_minus1);
const frame_mbs_only_flag = readBit(1);
writeBit(frame_mbs_only_flag, 1);
if (frame_mbs_only_flag === 0) {
const mb_adaptive_frame_field_flag = readBit(1);
writeBit(mb_adaptive_frame_field_flag, 1);
}
const direct_8x8_inference_flag = readBit(1);
writeBit(direct_8x8_inference_flag, 1);
const frame_cropping_flag = readBit(1);
writeBit(frame_cropping_flag, 1);
if (frame_cropping_flag) {
const frame_crop_left_offset = readUE();
writeUE(frame_crop_left_offset);
const frame_crop_right_offset = readUE();
writeUE(frame_crop_right_offset);
const frame_crop_top_offset = readUE();
writeUE(frame_crop_top_offset);
const frame_crop_bottom_offset = readUE();
writeUE(frame_crop_bottom_offset);
}
// https://webrtc.googlesource.com/src/+/5f2c9278f35e47ff72eb191669d473b7400c9f3e/common_video/h264/sps_vui_rewriter.cc#283
function addBitstreamRestriction() {
// motion_vectors_over_pic_boundaries_flag: u(1) — Default is 1 when not present.
writeBit(1, 1);
// max_bytes_per_pic_denom: ue(v) — Default is 2 when not present.
writeUE(2);
// max_bits_per_mb_denom: ue(v) — Default is 1 when not present.
writeUE(1);
// log2_max_mv_length_horizontal / vertical — both default to 16.
writeUE(16);
writeUE(16);
// IMPORTANT: max_num_reorder_frames must be 0 for low latency.
writeUE(0);
writeUE(max_num_ref_frames);
}
const vui_parameters_present_flag = readBit(1);
writeBit(1, 1);
// If no VUI exists, write one
if (!vui_parameters_present_flag) {
// aspect_ratio_info_present_flag, overscan_info_present_flag. Both u(1).
writeBit(0, 2);
// video_signal_type_present_flag, u(1) — write 0, ignore color space.
writeBit(0, 1);
// chroma_loc_info_present_flag, timing_info_present_flag,
// nal_hrd_parameters_present_flag, vcl_hrd_parameters_present_flag,
// pic_struct_present_flag — all u(1)
writeBit(0, 5);
// bitstream_restriction_flag: u(1)
writeBit(1, 1);
addBitstreamRestriction();
} else {
// VUI parsing and copying
const aspect_ratio_info_present_flag = readBit(1);
writeBit(aspect_ratio_info_present_flag, 1);
if (aspect_ratio_info_present_flag) {
const aspect_ratio_idc = readU(8);
writeU(aspect_ratio_idc, 8);
if (aspect_ratio_idc === 255) {
const sar_width = readU(16);
writeU(sar_width, 16);
const sar_height = readU(16);
writeU(sar_height, 16);
}
}
const overscan_info_present_flag = readBit(1);
writeBit(overscan_info_present_flag, 1);
if (overscan_info_present_flag) {
const overscan_appropriate_flag = readBit(1);
writeBit(overscan_appropriate_flag, 1);
}
// Read the video signal type, but don't copy it
const video_signal_type_present_flag = readBit(1);
writeBit(0, 1);
if (video_signal_type_present_flag) {
readBit(3); // _video_format
readBit(1); // _video_full_range_flag
const colour_description_present_flag = readBit(1);
if (colour_description_present_flag) {
readU(8); // _colour_primaries
readU(8); // _transfer_characteristics
readU(8); // _matrix_coeffs
}
}
const chroma_loc_info_present_flag = readBit(1);
writeBit(chroma_loc_info_present_flag, 1);
if (chroma_loc_info_present_flag) {
const chroma_sample_loc_type_top_field = readUE();
writeUE(chroma_sample_loc_type_top_field);
const chroma_sample_loc_type_bottom_field = readUE();
writeUE(chroma_sample_loc_type_bottom_field);
}
const timing_info_present_flag = readBit(1);
writeBit(timing_info_present_flag, 1);
if (timing_info_present_flag) {
const num_units_in_tick = readU(32);
writeU(num_units_in_tick, 32);
const time_scale = readU(32);
writeU(time_scale, 32);
const fixed_frame_rate_flag = readBit(1);
writeBit(fixed_frame_rate_flag, 1);
}
const nal_hrd_parameters_present_flag = readBit(1);
writeBit(nal_hrd_parameters_present_flag, 1);
if (nal_hrd_parameters_present_flag) {
// hrd_parameters()
const cpb_cnt_minus1 = readUE();
writeUE(cpb_cnt_minus1);
const bit_rate_scale = readBit(4);
writeBit(bit_rate_scale, 4);
const cpb_size_scale = readBit(4);
writeBit(cpb_size_scale, 4);
for (let i = 0; i <= cpb_cnt_minus1; i++) {
const bit_rate_value_minus1 = readUE();
writeUE(bit_rate_value_minus1);
const cpb_size_value_minus1 = readUE();
writeUE(cpb_size_value_minus1);
const cbr_flag = readBit(1);
writeBit(cbr_flag, 1);
}
const initial_cpb_removal_delay_length_minus1 = readBit(5);
writeBit(initial_cpb_removal_delay_length_minus1, 5);
const cpb_removal_delay_length_minus1 = readBit(5);
writeBit(cpb_removal_delay_length_minus1, 5);
const dpb_output_delay_length_minus1 = readBit(5);
writeBit(dpb_output_delay_length_minus1, 5);
const time_offset_length = readBit(5);
writeBit(time_offset_length, 5);
}
const vcl_hrd_parameters_present_flag = readBit(1);
writeBit(vcl_hrd_parameters_present_flag, 1);
if (vcl_hrd_parameters_present_flag) {
// hrd_parameters()
const cpb_cnt_minus1 = readUE();
writeUE(cpb_cnt_minus1);
const bit_rate_scale = readBit(4);
writeBit(bit_rate_scale, 4);
const cpb_size_scale = readBit(4);
writeBit(cpb_size_scale, 4);
for (let i = 0; i <= cpb_cnt_minus1; i++) {
const bit_rate_value_minus1 = readUE();
writeUE(bit_rate_value_minus1);
const cpb_size_value_minus1 = readUE();
writeUE(cpb_size_value_minus1);
const cbr_flag = readBit(1);
writeBit(cbr_flag, 1);
}
const initial_cpb_removal_delay_length_minus1 = readBit(5);
writeBit(initial_cpb_removal_delay_length_minus1, 5);
const cpb_removal_delay_length_minus1 = readBit(5);
writeBit(cpb_removal_delay_length_minus1, 5);
const dpb_output_delay_length_minus1 = readBit(5);
writeBit(dpb_output_delay_length_minus1, 5);
const time_offset_length = readBit(5);
writeBit(time_offset_length, 5);
}
if (nal_hrd_parameters_present_flag || vcl_hrd_parameters_present_flag) {
const low_delay_hrd_flag = readBit(1);
writeBit(low_delay_hrd_flag, 1);
}
const pic_struct_present_flag = readBit(1);
writeBit(pic_struct_present_flag, 1);
const bitstream_restriction_flag = readBit(1);
writeBit(1, 1);
if (!bitstream_restriction_flag) {
addBitstreamRestriction();
} else {
const motion_vectors_over_pic_boundaries_flag = readBit(1);
writeBit(motion_vectors_over_pic_boundaries_flag, 1);
const max_bytes_per_pic_denom = readUE();
writeUE(max_bytes_per_pic_denom);
const max_bits_per_mb_denom = readUE();
writeUE(max_bits_per_mb_denom);
const log2_max_mv_length_horizontal = readUE();
writeUE(log2_max_mv_length_horizontal);
const log2_max_mv_length_vertical = readUE();
writeUE(log2_max_mv_length_vertical);
readUE(); // _num_reorder_frames
writeUE(0);
readUE(); // _max_dec_frame_buffering
writeUE(max_num_ref_frames);
}
}
writeBit(1, 1); // rbsp_stop_one_bit
writer.flush();
return writer.toBuffer();
}
@@ -0,0 +1,45 @@
/**
* StreamConnection GoLive stream connection (screen share).
* Ported from @dank074/discord-video-stream StreamConnection.js.
*/
import { BaseMediaConnection } from "./BaseMediaConnection.js";
import { VoiceOpCodes } from "./VoiceOpCodes.js";
export class StreamConnection extends BaseMediaConnection {
_streamKey: string | null = null;
_serverId: string | null = null;
setSpeaking(speaking: boolean): void {
if (!this.webRtcParams) throw new Error("WebRTC connection not ready");
this.sendOpcode(VoiceOpCodes.SPEAKING, {
delay: 0,
speaking: speaking ? 2 : 0,
ssrc: this.webRtcParams.audioSsrc,
});
}
get daveChannelId(): string {
if (this._serverId === null) {
throw new Error("Server ID not set (this shouldn't happen)");
}
const channelId = BigInt(this._serverId) - 1n;
return channelId.toString();
}
get serverId(): string | null {
return this._serverId;
}
set serverId(id: string | null) {
this._serverId = id;
}
get streamKey(): string | null {
return this._streamKey;
}
set streamKey(value: string | null) {
this._streamKey = value;
}
}
@@ -0,0 +1,351 @@
/**
* Streamer gateway-level GoLive controller. Ported from
* @dank074/discord-video-stream Streamer.js.
*
* Drives the Discord gateway (VOICE_STATE_UPDATE, STREAM_CREATE, ...) and
* hands back a VoiceConnection / StreamConnection once the media server
* session is ready.
*/
import { EventEmitter } from "node:events";
import { GatewayOpCodes } from "./GatewayOpCodes.js";
import type { NativePeerConnection } from "./native.js";
import { StreamConnection } from "./StreamConnection.js";
import { generateStreamKey, parseStreamKey } from "./utils.js";
import { VoiceConnection } from "./VoiceConnection.js";
import type { WebRtcConnWrapper } from "./WebRtcWrapper.js";
/** Minimal surface of a discord.js-selfbot-v13 client used by Streamer. */
export interface StreamerClientLike {
user: { id: string; username?: string } | null;
token: string | null;
on(
event: "raw",
listener: (packet: { t: string; d: unknown }) => void,
): unknown;
ws: {
broadcast(data: { op: number; d: unknown }): void;
};
guilds?: {
// biome-ignore lint/suspicious/noExplicitAny: discord.js-selfbot client shape is dynamic
fetch(id: string): Promise<any>;
};
}
/** Minimal channel shape accepted by joinVoiceChannel. */
export interface VoiceChannelLike {
id: string;
type: string;
guildId?: string | null;
}
export class Streamer {
_voiceConnection: VoiceConnection | null = null;
_client: StreamerClientLike;
_gatewayEmitter = new EventEmitter();
constructor(client: StreamerClientLike) {
this._client = client;
// listen for gateway dispatch events
this.client.on("raw", (packet) => {
const t = packet.t as string;
if (
t === "STREAM_CREATE" ||
t === "STREAM_SERVER_UPDATE" ||
t === "VOICE_STATE_UPDATE" ||
t === "VOICE_SERVER_UPDATE"
) {
console.log(
`[goLive:Streamer] raw dispatch ${t}`,
JSON.stringify(packet.d).slice(0, 220),
);
}
this._gatewayEmitter.emit(t, packet.d);
});
}
get client(): StreamerClientLike {
return this._client;
}
get opts(): Record<string, unknown> {
return {};
}
get voiceConnection(): VoiceConnection | null {
return this._voiceConnection;
}
sendOpcode(code: number, data: unknown): void {
// Direct instrumentation — bypasses the bootstrap debug filter (which
// drops messages without [VOICE / [ffmpeg / error / stream).
console.log(
`[goLive:Streamer] sendOpcode op=${code} d=${JSON.stringify(data)}`,
);
this.client.ws.broadcast({ op: code, d: data });
}
joinVoiceChannel(channel: VoiceChannelLike): Promise<WebRtcConnWrapper> {
let guildId: string | null = null;
if (
channel.type === "GUILD_STAGE_VOICE" ||
channel.type === "GUILD_VOICE"
) {
guildId = channel.guildId ?? null;
}
return this.joinVoice(guildId, channel.id);
}
/**
* Joins a voice channel and resolves with the WebRtcConnWrapper when the
* media session is ready.
*/
joinVoice(
guild_id: string | null,
channel_id: string,
): Promise<WebRtcConnWrapper> {
return new Promise((resolve, reject) => {
if (!this.client.user) {
reject(new Error("Client not logged in"));
return;
}
const user_id = this.client.user.id;
const voiceConn = new VoiceConnection(
this,
guild_id,
user_id,
channel_id,
(conn) => {
resolve(conn);
},
);
this._voiceConnection = voiceConn;
this._gatewayEmitter.on(
"VOICE_STATE_UPDATE",
(d: { user_id: string; session_id: string }) => {
if (user_id !== d.user_id) return;
voiceConn.setSession(d.session_id);
},
);
this._gatewayEmitter.on(
"VOICE_SERVER_UPDATE",
(d: {
guild_id: string | null;
channel_id?: string;
endpoint: string;
token: string;
}) => {
if (guild_id !== d.guild_id) return;
// channel_id is not set for guild voice calls
if (d.channel_id && channel_id !== d.channel_id) return;
voiceConn.setTokens(d.endpoint, d.token);
},
);
this.signalVideo(false);
});
}
/** Create a GoLive stream (screen share) on top of the voice connection. */
createStream(): Promise<WebRtcConnWrapper> {
return new Promise((resolve, reject) => {
if (!this.client.user) {
reject(new Error("Client not logged in"));
return;
}
if (!this.voiceConnection) {
reject(
new Error("cannot start stream without first joining voice channel"),
);
return;
}
const {
guildId: clientGuildId,
channelId: clientChannelId,
session_id,
} = this.voiceConnection;
const clientUserId = this.client.user.id;
if (!session_id) throw new Error("Session doesn't exist yet");
const streamConn = new StreamConnection(
this,
clientGuildId,
clientUserId,
clientChannelId,
(conn) => {
clearTimeout(streamTimeout);
clearInterval(retryInterval);
resolve(conn);
},
);
this.voiceConnection.streamConnection = streamConn;
// Attach listeners BEFORE the first signal so a fast dispatch can't
// be lost between signalStream() and listener registration.
const onStreamCreate = (d: {
stream_key: string;
rtc_server_id: string;
}) => {
const { channelId, guildId, userId } = parseStreamKey(d.stream_key);
if (
clientGuildId !== guildId ||
clientChannelId !== channelId ||
clientUserId !== userId
) {
return;
}
streamConn.serverId = d.rtc_server_id;
streamConn.streamKey = d.stream_key;
streamConn.setSession(session_id);
};
const onStreamServerUpdate = (d: {
stream_key: string;
endpoint: string;
token: string;
}) => {
const { channelId, guildId, userId } = parseStreamKey(d.stream_key);
if (
clientGuildId !== guildId ||
clientChannelId !== channelId ||
clientUserId !== userId
) {
return;
}
streamConn.setTokens(d.endpoint, d.token);
};
this._gatewayEmitter.on("STREAM_CREATE", onStreamCreate);
this._gatewayEmitter.on("STREAM_SERVER_UPDATE", onStreamServerUpdate);
const cleanup = () => {
clearTimeout(streamTimeout);
clearInterval(retryInterval);
this._gatewayEmitter.removeListener("STREAM_CREATE", onStreamCreate);
this._gatewayEmitter.removeListener(
"STREAM_SERVER_UPDATE",
onStreamServerUpdate,
);
};
const streamTimeout = setTimeout(() => {
cleanup();
reject(
new Error(
"Timed out waiting for STREAM_CREATE/STREAM_SERVER_UPDATE from Discord (stream handshake) — voice media session may not be active",
),
);
}, 12_000);
// Discord sometimes drops the STREAM_CREATE request silently (upstream
// issue #217/#219) — resend a few times instead of giving up after one.
let attempt = 0;
const retryInterval = setInterval(() => {
attempt += 1;
if (attempt >= 4) {
clearInterval(retryInterval);
return;
}
console.log(
`[goLive:Streamer] createStream: retrying STREAM_CREATE (attempt ${attempt + 1}/4)`,
);
this.signalStream();
}, 3_000);
console.log(
`[goLive:Streamer] createStream: sending STREAM_CREATE (attempt 1/4)`,
);
this.signalStream();
});
}
async setStreamPreview(image: Buffer): Promise<void> {
if (!this.client.token) throw new Error("Please login :)");
if (!this.voiceConnection?.streamConnection?.guildId) return;
const data = `data:image/jpeg;base64,${image.toString("base64")}`;
const { guildId } = this.voiceConnection.streamConnection;
if (!this.client.guilds) return;
const server = await this.client.guilds.fetch(guildId);
// biome-ignore lint/suspicious/noExplicitAny: discord.js-selfbot dynamic
(server as any).members.me?.voice?.postPreview(data);
}
stopStream(): void {
const stream = this.voiceConnection?.streamConnection;
if (!stream) return;
stream.stop();
this.signalStopStream();
this.voiceConnection.streamConnection = null;
this._gatewayEmitter.removeAllListeners("STREAM_CREATE");
this._gatewayEmitter.removeAllListeners("STREAM_SERVER_UPDATE");
}
leaveVoice(): void {
this.voiceConnection?.stop();
this.signalLeaveVoice();
this._voiceConnection = null;
this._gatewayEmitter.removeAllListeners("VOICE_STATE_UPDATE");
this._gatewayEmitter.removeAllListeners("VOICE_SERVER_UPDATE");
}
signalVideo(video_enabled: boolean): void {
if (!this.voiceConnection) return;
const { guildId: guild_id, channelId: channel_id } = this.voiceConnection;
this.sendOpcode(GatewayOpCodes.VOICE_STATE_UPDATE, {
guild_id: guild_id,
channel_id,
self_mute: false,
self_deaf: true,
self_video: video_enabled,
});
}
signalStream(): void {
if (!this.voiceConnection) return;
const {
type,
guildId: guild_id,
channelId: channel_id,
botId: user_id,
} = this.voiceConnection;
// Un-deafen before requesting the stream (mimic real client). Do NOT
// set self_video: true — that flips on the bot's camera in Discord
// (visible to everyone); screen share should not enable the camera.
this.sendOpcode(GatewayOpCodes.VOICE_STATE_UPDATE, {
guild_id,
channel_id,
self_mute: false,
self_deaf: false,
self_video: false,
});
this.sendOpcode(GatewayOpCodes.STREAM_CREATE, {
type,
guild_id,
channel_id,
preferred_region: null,
});
this.sendOpcode(GatewayOpCodes.STREAM_SET_PAUSED, {
stream_key: generateStreamKey(type, guild_id, channel_id, user_id),
paused: false,
});
}
signalStopStream(): void {
if (!this.voiceConnection) return;
const {
type,
guildId: guild_id,
channelId: channel_id,
botId: user_id,
} = this.voiceConnection;
this.sendOpcode(GatewayOpCodes.STREAM_DELETE, {
stream_key: generateStreamKey(type, guild_id, channel_id, user_id),
});
}
signalLeaveVoice(): void {
this.sendOpcode(GatewayOpCodes.VOICE_STATE_UPDATE, {
guild_id: null,
channel_id: null,
self_mute: true,
self_deaf: false,
self_video: false,
});
}
}
export type { NativePeerConnection };
@@ -0,0 +1,20 @@
/**
* VideoStream feeds encoded H264 frames into the WebRTC connection.
* Ported from @dank074/discord-video-stream VideoStream.js.
*/
import { BaseMediaStream } from "./BaseMediaStream.js";
import type { WebRtcConnWrapper } from "./WebRtcWrapper.js";
export class VideoStream extends BaseMediaStream {
_conn: WebRtcConnWrapper;
constructor(conn: WebRtcConnWrapper, noSleep = false) {
super("video", noSleep);
this._conn = conn;
}
async _sendFrame(frame: Buffer, frametime: number): Promise<void> {
this._conn.sendVideoFrame(frame, frametime);
}
}
@@ -0,0 +1,25 @@
/**
* VoiceConnection guild/DM voice channel GoLive connection.
* Ported from @dank074/discord-video-stream VoiceConnection.js.
*/
import { BaseMediaConnection } from "./BaseMediaConnection.js";
import type { StreamConnection } from "./StreamConnection.js";
export class VoiceConnection extends BaseMediaConnection {
streamConnection: StreamConnection | null = null;
get daveChannelId(): string {
return this.channelId;
}
get serverId(): string | null {
// for guild vc it is the guild id, for dm voice it is the channel id
return this.guildId ?? this.channelId;
}
stop(): void {
super.stop();
this.streamConnection?.stop();
}
}
@@ -0,0 +1,38 @@
/** Discord voice WebSocket opcodes — ported from @dank074/discord-video-stream. */
export enum VoiceOpCodes {
IDENTIFY = 0,
SELECT_PROTOCOL = 1,
READY = 2,
HEARTBEAT = 3,
SELECT_PROTOCOL_ACK = 4,
SPEAKING = 5,
HEARTBEAT_ACK = 6,
RESUME = 7,
HELLO = 8,
RESUMED = 9,
CLIENTS_CONNECT = 11,
VIDEO = 12,
CLIENT_DISCONNECT = 13,
SESSION_UPDATE = 14,
MEDIA_SINK_WANTS = 15,
VOICE_BACKEND_VERSION = 16,
CHANNEL_OPTIONS_UPDATE = 17,
FLAGS = 18,
SPEED_TEST = 19,
PLATFORM = 20,
DAVE_PREPARE_TRANSITION = 21,
DAVE_EXECUTE_TRANSITION = 22,
DAVE_TRANSITION_READY = 23,
DAVE_PREPARE_EPOCH = 24,
MLS_INVALID_COMMIT_WELCOME = 31,
}
/** Binary voice WebSocket opcodes (DAVE / MLS). */
export enum VoiceOpCodesBinary {
MLS_EXTERNAL_SENDER = 25,
MLS_KEY_PACKAGE = 26,
MLS_PROPOSALS = 27,
MLS_COMMIT_WELCOME = 28,
MLS_ANNOUNCE_COMMIT_TRANSITION = 29,
MLS_WELCOME = 30,
}
@@ -0,0 +1,223 @@
/**
* WebRTC connection wrapper for GoLive ported from
* @dank074/discord-video-stream WebRtcWrapper.js, with the media stack
* (packetizers, RTCP SR/NACK, pacing) provided by the libdatachannel-min
* binding instead of node-datachannel's JS-exposed media classes.
*/
import {
H264Helpers,
H264NalUnitTypes,
splitNalu,
startCode3,
} from "./AnnexBHelper.js";
import { CodecPayloadType } from "./CodecPayloadType.js";
import type { NativePeerConnection, NativeTrack } from "./native.js";
import { loadNative } from "./native.js";
import { rewriteSPSVUI } from "./SPSVUIRewriter.js";
import { normalizeVideoCodec } from "./utils.js";
export type WebRtcVideoCodec = "H264" | "H265" | "VP8" | "VP9" | "AV1";
export interface WebRtcParams {
address: string;
port: number;
audioSsrc: number;
videoSsrc: number;
rtxSsrc: number;
supportedEncryptionModes: string[];
}
/** Minimal surface of the media connection that WebRtcWrapper drives. */
export interface VideoAttribute {
fps: number;
width: number;
height: number;
}
export interface MediaConnectionLike {
daveReady: boolean;
daveSession: {
encryptOpus(frame: Buffer): Buffer;
encrypt(mediaType: number, codec: number, frame: Buffer): Buffer;
} | null;
webRtcParams: WebRtcParams | null;
setSpeaking(speaking: boolean): void;
setVideoAttributes(enabled: boolean, attr?: VideoAttribute): void;
}
/** Media types used by DAVE encryption (from @dank074). */
export enum DaveMediaType {
AUDIO = 0,
VIDEO = 1,
}
/** DAVE codec ids (from @dank074). */
export enum DaveCodec {
UNKNOWN = 0,
VP8 = 2,
VP9 = 3,
H264 = 4,
H265 = 5,
AV1 = 6,
}
export class WebRtcConnWrapper {
private _mediaConn: MediaConnectionLike;
private _webRtcConn: NativePeerConnection | null = null;
private _audioTrack: NativeTrack | null = null;
private _videoTrack: NativeTrack | null = null;
private _videoCodec: WebRtcVideoCodec | null = null;
private _videoFrameLog = 0;
/** Assigned by BaseMediaConnection to send the gathered SDP to Discord. */
onLocalDescription: ((sdp: string) => void) | null = null;
constructor(mediaConn: MediaConnectionLike) {
this._mediaConn = mediaConn;
}
initWebRtc(): NativePeerConnection {
const native = loadNative();
this._webRtcConn = new native.PeerConnection({
iceServers: ["stun:stun.l.google.com:19302"],
});
// Track mids must match @dank074: "0" audio, "1" video.
this._audioTrack = this._webRtcConn.addTrack("0", "audio");
this._videoTrack = this._webRtcConn.addTrack("1", "video");
return this._webRtcConn;
}
close(): void {
this._webRtcConn?.close();
this._webRtcConn = null;
}
get webRtcConn(): NativePeerConnection | null {
return this._webRtcConn;
}
get ready(): boolean {
return this._webRtcConn?.state() === "connected";
}
get mediaConnection(): MediaConnectionLike {
return this._mediaConn;
}
sendAudioFrame(frame: Buffer, frametime: number): void {
if (!this.ready || !this._audioTrack) return;
const clockRate = CodecPayloadType.opus.clockRate;
if (this.mediaConnection.daveReady && this.mediaConnection.daveSession) {
frame = this.mediaConnection.daveSession.encryptOpus(frame);
}
this._audioTrack.sendFrame(frame);
this._audioTrack.addTimestamp(Math.round((frametime * clockRate) / 1000));
}
sendVideoFrame(frame: Buffer, frametime: number): void {
if (!this.ready || !this._videoTrack) {
if (this._videoFrameLog === 0) {
console.log(
`[goLive:WebRtc] sendVideoFrame DROPPED ready=${this.ready} track=${this._videoTrack !== null}`,
);
this._videoFrameLog++;
}
return;
}
const clockRate = CodecPayloadType[this._videoCodec ?? "H264"].clockRate;
if (this._videoCodec === "H264") {
let spsRewritten = false;
const nalus = splitNalu(frame).map((el) => {
if (H264Helpers.getUnitType(el) === H264NalUnitTypes.SPS) {
spsRewritten = true;
return rewriteSPSVUI(el);
}
return el;
});
if (spsRewritten)
frame = Buffer.concat(nalus.flatMap((el) => [startCode3, el]));
}
if (this.mediaConnection.daveReady && this.mediaConnection.daveSession) {
let daveCodec = DaveCodec.UNKNOWN;
switch (this._videoCodec) {
case "H264":
daveCodec = DaveCodec.H264;
break;
case "H265":
daveCodec = DaveCodec.H265;
break;
case "VP8":
daveCodec = DaveCodec.VP8;
break;
case "VP9":
daveCodec = DaveCodec.VP9;
break;
case "AV1":
daveCodec = DaveCodec.AV1;
break;
default:
break;
}
frame = this.mediaConnection.daveSession.encrypt(
DaveMediaType.VIDEO,
daveCodec,
frame,
);
}
this._videoTrack.sendFrame(frame);
this._videoTrack.addTimestamp(Math.round((frametime * clockRate) / 1000));
this._videoFrameLog++;
if (this._videoFrameLog === 1 || this._videoFrameLog % 30 === 0) {
console.log(
`[goLive:WebRtc] sendVideoFrame #${this._videoFrameLog} bytes=${frame.length} ready=${this.ready}`,
);
}
}
setPacketizer(videoCodec: string): void {
if (!this.mediaConnection.webRtcParams) {
throw new Error("WebRTC connection not ready");
}
const { audioSsrc, videoSsrc } = this.mediaConnection.webRtcParams;
console.log(
`[goLive:WebRtc] setPacketizer(${videoCodec}) audioSsrc=${audioSsrc} videoSsrc=${videoSsrc} rtxSsrc=${this.mediaConnection.webRtcParams.rtxSsrc}`,
);
this._videoCodec = normalizeVideoCodec(videoCodec);
// Audio packetizer: opus 120 @ 48kHz, playout delay ext id 5 (like @dank074)
this._audioTrack?.setPacketizer(
"audio",
audioSsrc,
CodecPayloadType.opus.payload_type,
CodecPayloadType.opus.clockRate,
5,
0,
1,
);
// Video packetizer: H264/H265/AV1 with their payload types
const codecEntry = CodecPayloadType[this._videoCodec];
if (!codecEntry) {
throw new Error(`Packetizer not implemented for ${this._videoCodec}`);
}
const nativeKind =
this._videoCodec === "H264"
? "h264"
: this._videoCodec === "H265"
? "h265"
: this._videoCodec === "AV1"
? "av1"
: (() => {
throw new Error(
`Packetizer not implemented for ${this._videoCodec}`,
);
})();
this._videoTrack?.setPacketizer(
nativeKind,
videoSsrc,
codecEntry.payload_type,
codecEntry.clockRate,
5,
0,
10,
);
}
}
@@ -0,0 +1,19 @@
/**
* goLive public API re-exports the ported @dank074 modules.
* Drop-in replacement for `@dank074/discord-video-stream` in
* screenShareController.ts.
*/
export { AudioStream } from "./AudioStream.js";
export { BaseMediaConnection } from "./BaseMediaConnection.js";
export { BaseMediaStream } from "./BaseMediaStream.js";
export { CodecPayloadType } from "./CodecPayloadType.js";
export { demux } from "./Demuxer.js";
export { Encoders } from "./Encoders.js";
export { playStream, prepareStream } from "./prepareStream.js";
export { StreamConnection } from "./StreamConnection.js";
export { Streamer } from "./Streamer.js";
export { normalizeVideoCodec } from "./utils.js";
export { VideoStream } from "./VideoStream.js";
export { VoiceConnection } from "./VoiceConnection.js";
export { WebRtcConnWrapper } from "./WebRtcWrapper.js";
@@ -0,0 +1,116 @@
/**
* Loader + typings for the minimal libdatachannel N-API binding
* (native/libdatachannel-min). The binding exposes ONLY what GoLive needs:
* PeerConnection, DataChannel, Track (raw RTP + media packetizer chain).
*
* The .node file is built by node-gyp against libdatachannel 0.24.0 (built
* from source nixpkgs 0.24.1 is glibc-incompatible with this host). It is
* NOT shipped via npm; the Nix flake builds it as part of the gateway.
*/
export interface NativeTrack {
/** Send a RAW RTP/RTCP packet (no media handler installed). */
send(buffer: Uint8Array): void;
/** Send an ENCODED frame; the packetizer chain turns it into RTP. */
sendFrame(buffer: Uint8Array): void;
/** Advance the packetizer RTP timestamp by delta (clock-rate units). */
addTimestamp(delta: number): void;
/** Install the media-handler chain (packetizer → RTCP SR → NACK → pacing). */
setPacketizer(
kind: "audio" | "h264" | "h265" | "av1",
ssrc: number,
payloadType: number,
clockRate: number,
playoutDelayId: number,
playoutDelayMin: number,
playoutDelayMax: number,
): void;
isOpen(): boolean;
close(): void;
}
export interface NativePeerConnection {
/** mid must be "0" (audio) or "1" (video) — matches @dank074's track defs. */
addTrack(mid: string, kind: "audio" | "video"): NativeTrack;
/** Resolves with the full SDP (incl. candidates) after gathering completes. */
createOffer(): Promise<string>;
/** Resolves with the auto-generated answer SDP. */
createAnswer(offerSdp: string): Promise<string>;
setRemoteDescription(sdp: string, type: "offer" | "answer"): void;
state(): string;
close(): void;
onStateChange(cb: (state: string) => void): void;
}
export interface NativeBinding {
PeerConnection: new (config: {
iceServers: string[];
}) => NativePeerConnection;
DataChannel: unknown;
Track: unknown;
}
let cached: NativeBinding | null = null;
/** Load the native binding. Throws only if the .node is truly missing
* callers (screen share) guard with `isNativeAvailable()`. */
export function loadNative(): NativeBinding {
if (cached) return cached;
// Resolve relative to this file: src/goLive/ → native/libdatachannel-min/
const candidates = [
new URL(
"../../native/libdatachannel-min/build/Release/datachannel_min.node",
import.meta.url,
),
new URL(
"../../../native/libdatachannel-min/build/Release/datachannel_min.node",
import.meta.url,
),
];
let lastErr: unknown;
for (const url of candidates) {
try {
// @ts-expect-error — .node modules are not typed; dynamic require via file URL
const mod = process.dlopen ? null : null;
void mod;
const nativePath = url.pathname;
// eslint-disable-next-line @typescript-eslint/no-require-imports
const req = createRequire(import.meta.url);
const binding = req(nativePath) as NativeBinding;
if (typeof binding.PeerConnection === "function") {
cached = binding;
return binding;
}
} catch (e) {
lastErr = e;
}
}
// Fallback: plain relative require (tsx / jest environments)
try {
const req = createRequire(import.meta.url);
const binding = req(
"../../native/libdatachannel-min/build/Release/datachannel_min.node",
) as NativeBinding;
if (typeof binding.PeerConnection === "function") {
cached = binding;
return binding;
}
} catch (e) {
lastErr = e;
}
throw new Error(
`libdatachannel-min native binding not built (${String(lastErr)}). Run: cd native/libdatachannel-min && npx node-gyp rebuild`,
);
}
import { createRequire } from "node:module";
/** True when the native binding is built — screen share stays disabled otherwise. */
export function isNativeAvailable(): boolean {
try {
loadNative();
return true;
} catch {
return false;
}
}
@@ -0,0 +1,446 @@
/**
* prepareStream & playStream ported from @dank074/discord-video-stream
* newApi.js (Encoders/prepareStream/playStream), but uses `child_process.spawn`
* + ffmpeg CLI args directly instead of fluent-ffmpeg + node-av.
*
* Replaces the @dank074 video pipeline entirely:
* input (URL or Readable) ffmpeg spawn H264 AnnexB frames
* Demuxer stream VideoStream/AudioStream WebRtcConnWrapper
*/
import { type ChildProcess, spawn } from "node:child_process";
import { existsSync, readdirSync } from "node:fs";
import { join } from "node:path";
import { PassThrough, type Readable } from "node:stream";
import { AudioStream } from "./AudioStream.js";
import { demux } from "./Demuxer.js";
import { type EncoderSettings, Encoders } from "./Encoders.js";
import { VideoStream } from "./VideoStream.js";
import type { WebRtcConnWrapper } from "./WebRtcWrapper.js";
export interface PrepareStreamResult {
command: ChildProcess;
output: PassThrough;
encoder: () => Record<string, EncoderSettings>;
options: Record<string, unknown>;
videoCodec: string;
width: number;
height: number;
frameRate?: number;
includeAudio: boolean;
/** Container the encoder muxes to: "nut" (audio-capable) or "h264" (raw). */
format: "nut" | "h264";
}
function isFiniteNonZero(n: unknown): n is number {
return typeof n === "number" && !!n && Number.isFinite(n);
}
const DEFAULT_HEADERS = {
"User-Agent":
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36",
Connection: "keep-alive",
};
/** Resolve ffmpeg binary (env override → PATH → Nix store ffmpeg-headless). */
function resolveFfmpeg(): string {
if (process.env.FFMPEG_PATH && existsSync(process.env.FFMPEG_PATH)) {
return process.env.FFMPEG_PATH;
}
const store = "/nix/store";
if (existsSync(store)) {
const entries = readdirSync(store);
for (const entry of entries) {
if (!entry.includes("ffmpeg-headless-")) continue;
const candidate = join(store, entry, "bin", "ffmpeg");
if (existsSync(candidate)) return candidate;
}
}
return "ffmpeg";
}
const FFMPEG_BIN = resolveFfmpeg();
/**
* prepareStream build an ffmpeg command (as spawn args + PassThrough output)
* that transcodes the input into a pipe we can demux. Mirrors @dank074's
* prepareStream but produces a raw spawn instead of a fluent-ffmpeg command.
*/
export function prepareStream(
input: string | Readable,
options: Record<string, unknown> = {},
): PrepareStreamResult {
const mergedOptions = {
noTranscoding: false,
width: isFiniteNonZero(options.width)
? Math.round(options.width as number)
: -2,
height: isFiniteNonZero(options.height)
? Math.round(options.height as number)
: -2,
frameRate:
isFiniteNonZero(options.frameRate) && (options.frameRate as number) > 0
? options.frameRate
: undefined,
videoCodec: (options.videoCodec as string) ?? "H264",
bitrateVideo:
isFiniteNonZero(options.bitrateVideo) &&
(options.bitrateVideo as number) > 0
? Math.round(options.bitrateVideo as number)
: 5000,
bitrateVideoMax:
isFiniteNonZero(options.bitrateVideoMax) &&
(options.bitrateVideoMax as number) > 0
? Math.round(options.bitrateVideoMax as number)
: 7000,
bitrateAudio:
isFiniteNonZero(options.bitrateAudio) &&
(options.bitrateAudio as number) > 0
? Math.round(options.bitrateAudio as number)
: 128,
includeAudio: options.includeAudio ?? true,
encoder:
(options.encoder as () => Record<string, EncoderSettings>) ??
Encoders.software(),
customHeaders: {
...DEFAULT_HEADERS,
...(options.customHeaders as Record<string, string> | undefined),
},
customInputOptions: (options.customInputOptions as string[]) ?? [],
customFfmpegFlags: (options.customFfmpegFlags as string[]) ?? [],
minimizeLatency: options.minimizeLatency ?? false,
};
const output = new PassThrough();
const args: string[] = [
"-hide_banner",
"-loglevel",
"error",
...(typeof input === "string" ? ["-i", input] : ["-i", "pipe:0"]),
...mergedOptions.customInputOptions,
];
if (mergedOptions.minimizeLatency) {
args.push("-fflags", "nobuffer", "-analyzeduration", "0");
}
if (typeof input === "string" && input.startsWith("http")) {
const headerStr = Object.entries(mergedOptions.customHeaders)
.map(([k, v]) => `${k}: ${v}`)
.join("\r\n");
args.push(
"-headers",
headerStr,
"-reconnect",
"1",
"-reconnect_at_eof",
"1",
"-reconnect_streamed",
"1",
"-reconnect_delay_max",
"4294",
);
}
// Video
args.push("-map", "0:v:0");
if (mergedOptions.noTranscoding) {
args.push("-c:v", "copy");
} else {
args.push(`-vf`, `scale=${mergedOptions.width}:${mergedOptions.height}`);
if (mergedOptions.frameRate)
args.push("-r", String(mergedOptions.frameRate));
const enc = mergedOptions.encoder()[mergedOptions.videoCodec];
if (!enc)
throw new Error(
`Encoder settings not specified for ${mergedOptions.videoCodec}`,
);
// Encoder options are declared as single strings like "-forced-idr 1";
// spawn needs each flag and value as separate argv entries.
const encOptions = enc.options.flatMap((opt) =>
opt.split(/\s+/).filter(Boolean),
);
args.push(
"-b:v",
`${mergedOptions.bitrateVideo}k`,
"-maxrate:v",
`${mergedOptions.bitrateVideoMax}k`,
"-bufsize:v",
`${Math.round(mergedOptions.bitrateVideo / 2)}k`,
"-bf",
"0",
"-pix_fmt",
"yuv420p",
"-force_key_frames",
"expr:gte(t,n_forced*1)",
"-c:v",
enc.name,
...encOptions,
...(enc.globalOptions ?? []).flatMap((opt) =>
opt.split(/\s+/).filter(Boolean),
),
);
}
// Audio: transcode to libopus. NUT muxer on stdout carries video (H264
// AnnexB) + audio (Ogg Opus) as ONE stream into the Demuxer, which re-splits
// them via a child ffmpeg -f nut -i pipe:0 -c:v copy -f h264 pipe:1 ... . The
// Demuxer's start-code scan runs on THAT child ffmpeg's stdout (pure H264),
// NOT on NUT — so NAL type 5 (IDR) is parsed correctly. (Outputting raw
// h264+opus on two pipes directly was tried and broke: the audio pipe was
// never attached to the demuxer's input, so audio RTP never flowed.)
if (mergedOptions.includeAudio) {
args.push(
"-map",
"0:a:0?",
"-c:a",
"libopus",
"-b:a",
`${mergedOptions.bitrateAudio}k`,
"-ar",
"48000",
"-ac",
"2",
);
} else {
args.push("-an");
}
// NUT muxer carries video+audio; the raw h264 muxer cannot ("h264 muxer
// does not support any stream of type audio" -> header write fails ->
// empty stdout -> black tile). Audio delivery requires NUT.
const outFormat = mergedOptions.includeAudio ? "nut" : "h264";
args.push("-f", outFormat, "pipe:1");
const isUrl = typeof input === "string";
const proc: ChildProcess = isUrl
? spawn(FFMPEG_BIN, args, { stdio: ["ignore", "pipe", "pipe"] })
: spawn(FFMPEG_BIN, args, { stdio: ["pipe", "pipe", "pipe"] });
if (proc.stdin && !isUrl) {
// Race guard: the merge ffmpeg may have already exited (transient 403
// or stream death) before this function attaches its listeners — the
// input's 'end'/'error' events then fire into the void and the encoder
// stdin NEVER receives EOF, leaving an encoder that waits forever and a
// screen share that shows a black tile with zero frames. Check the
// terminal state eagerly and EOF the encoder immediately.
if (input.readableEnded || input.destroyed) {
proc.stdin.end();
} else {
input.on("data", (chunk: Buffer) => proc.stdin?.write(chunk));
input.on("end", () => proc.stdin?.end());
input.on("error", () => proc.stdin?.destroy());
}
}
// Safety: proc may error before playStream attaches a demux listener on
// `output`. A no-op listener here prevents an unhandled 'error' event
// on the PassThrough from crashing the gateway on ffmpeg spawn failure.
output.on("error", () => {});
proc.stdout?.pipe(output);
proc.stderr?.on("data", () => {
/* swallow ffmpeg stderr */
});
proc.on("error", (err) => {
// spawn failed (e.g. ffmpeg missing). If someone is consuming output
// (demux attaches an 'error' listener) propagate; otherwise just end.
if (output.listenerCount("error") > 0) {
output.destroy(err);
} else {
output.end();
}
});
proc.on("close", () => {
output.end();
});
return {
command: proc,
output,
encoder: mergedOptions.encoder,
options: mergedOptions,
videoCodec: mergedOptions.videoCodec,
width: mergedOptions.width,
height: mergedOptions.height,
frameRate: mergedOptions.frameRate,
includeAudio: !!mergedOptions.includeAudio,
format: outFormat,
};
}
export interface PlayStreamOptions {
type?: "go-live" | "video";
format?: string;
width?: number | ((v: unknown) => number);
height?: number | ((v: unknown) => number);
frameRate?: number | ((v: unknown) => number);
readrateInitialBurst?: number;
streamPreview?: boolean;
}
/**
* playStream demux the prepareStream output and pipe frames into the
* WebRTC connection's video/audio streams. Resolves when the video stream
* ends (natural EOF or the ffmpeg command is killed via cleanup/stop).
*/
export async function playStream(
prepared: PrepareStreamResult,
streamer: { createStream: () => Promise<WebRtcConnWrapper> },
options: PlayStreamOptions = {},
): Promise<void> {
const conn = await streamer.createStream();
console.log("[goLive:playStream] createStream resolved");
const {
video,
audio,
close: demuxClose,
} = await demux(prepared.output, {
format: options.format ?? prepared.format ?? "nut",
frameRate:
typeof options.frameRate === "number" ? options.frameRate : undefined,
});
console.log(
`[goLive:playStream] demux done codec=${video?.codecName ?? "?"} ${video?.width ?? 0}x${video?.height ?? 0} fps=${video ? video.framerate_num / video.framerate_den || 30 : 30} audio=${audio?.codecName ?? "none"}`,
);
if (!video) throw new Error("No video stream in media");
conn.setPacketizer(video.codecName);
conn.mediaConnection.setSpeaking(true);
console.log(
`[goLive:playStream] setPacketizer(${video.codecName}) + setSpeaking done`,
);
const w =
typeof options.width === "function"
? options.width(video)
: (options.width ?? video.width);
const h =
typeof options.height === "function"
? options.height(video)
: (options.height ?? video.height);
const fr =
typeof options.frameRate === "function"
? options.frameRate(video)
: (options.frameRate ??
(video.framerate_num / video.framerate_den || 30));
conn.mediaConnection.setVideoAttributes(true, {
width: Math.round(w),
height: Math.round(h),
fps: Math.round(fr),
});
const vStream = new VideoStream(conn);
video.stream.pipe(vStream);
// Audio: Discord's GoLive pipeline expects RTP on the audio SSRC too —
// a video-only stream (zero audio packets) shows a static tile/thumbnail
// instead of live video. Pipe opus frames from the demuxer (silence is
// injected at the encoder when the source has no audio track).
let aStream: AudioStream | undefined;
if (audio) {
aStream = new AudioStream(conn);
audio.stream.pipe(aStream);
console.log(
`[goLive:playStream] audio stream attached (${audio.codecName})`,
);
}
const cleanup = () => {
try {
prepared.command.kill("SIGTERM");
} catch {
/* already dead */
}
demuxClose();
try {
conn.mediaConnection.setSpeaking(false);
conn.mediaConnection.setVideoAttributes(false);
} catch {
/* connection already torn down */
}
};
// First-frame watchdog: if the encoder never delivers a single frame
// (dead merge input, empty stream, codec mismatch), fail fast instead of
// "playing" a black tile forever. The demuxer resolves with fallback
// metadata even when no frame ever arrives, so this timeout is the only
// place that detects "started but nothing flowing".
let firstFrameTimer: NodeJS.Timeout | null = null;
let gotFirstFrame = false;
const firstFrame = new Promise<void>((resolve, reject) => {
firstFrameTimer = setTimeout(() => {
if (!gotFirstFrame) {
cleanup();
reject(
new Error(
"No video frames within 10s of stream start — input stream failed",
),
);
}
}, 10000);
video.stream.once("data", () => {
gotFirstFrame = true;
if (firstFrameTimer) clearTimeout(firstFrameTimer);
resolve();
});
});
return new Promise<void>((resolve, reject) => {
let settled = false;
const settle = (fn: () => void) => () => {
if (settled) return;
settled = true;
if (firstFrameTimer) clearTimeout(firstFrameTimer);
fn();
};
vStream.once("finish", () => {
settle(() => {
cleanup();
if (!gotFirstFrame) {
reject(new Error("Screen video stream ended without any frame"));
} else {
resolve();
}
})();
});
vStream.once("error", () => {
settle(() => {
cleanup();
if (!gotFirstFrame) {
reject(new Error("Screen video stream errored before first frame"));
} else {
resolve();
}
})();
});
// The stream may end without ever producing a frame (input was
// silently dead) — surface that instead of resolving "successfully".
video.stream.once("end", () => {
settle(() => {
cleanup();
if (!gotFirstFrame) {
reject(new Error("Screen video stream ended before any frame"));
} else {
resolve();
}
})();
});
// Watchdog timeout: no frame arrived within 10s — fail fast instead of
// "playing" a black tile forever. cleanup() kills the encoder so the
// vStream finish/error handlers above still fire, but the settled guard
// ensures this rejection wins.
firstFrame.catch((err) => {
settle(() => {
cleanup();
reject(err);
})();
});
});
}
export { Encoders };
@@ -0,0 +1,82 @@
/** GoLive helpers — ported from @dank074/discord-video-stream/utils.js. */
export function normalizeVideoCodec(
codec: string,
): "H264" | "H265" | "VP8" | "VP9" | "AV1" {
if (/H\.?264|AVC/i.test(codec)) return "H264";
if (/H\.?265|HEVC/i.test(codec)) return "H265";
if (/VP(8|9)/i.test(codec)) return codec.toUpperCase() as "VP8" | "VP9";
if (/AV1/i.test(codec)) return "AV1";
throw new Error(`Unknown codec: ${codec}`);
}
/**
* The available video streams are sent by the client on connection to the
* voice gateway using OpCode Identify (0); the server replies with the ssrc
* and rtxssrc for each available stream using OpCode Ready (2). RID
* distinguishes simulcast streams of the same video source we only send one
* quality stream, so a single entry is hardcoded.
*/
export const STREAMS_SIMULCAST = [{ type: "screen", rid: "100", quality: 100 }];
export const max_int16bit = 2 ** 16;
export const max_int32bit = 2 ** 32;
export function isFiniteNonZero(n: unknown): n is number {
return typeof n === "number" && !!n && Number.isFinite(n);
}
export interface ParsedStreamKey {
type: "guild" | "call";
channelId: string;
guildId: string | null;
userId: string;
}
export function parseStreamKey(streamKey: string): ParsedStreamKey {
const streamKeyArray = streamKey.split(":");
const type = streamKeyArray.shift();
if (type !== "guild" && type !== "call") {
throw new Error(`Invalid stream key type: ${type}`);
}
if (
(type === "guild" && streamKeyArray.length < 3) ||
(type === "call" && streamKey.length < 2)
) {
throw new Error(`Invalid stream key: ${streamKey}`);
}
let guildId: string | null = null;
if (type === "guild") {
guildId = streamKeyArray.shift() ?? null;
}
const channelId = streamKeyArray.shift();
const userId = streamKeyArray.shift();
if (!channelId || !userId) {
throw new Error(`Invalid stream key: ${streamKey}`);
}
return { type, channelId, guildId, userId };
}
export function generateStreamKey(
type: "guild" | "call",
guildId: string | null,
channelId: string,
userId: string,
): string {
return `${type}${type === "guild" ? `:${guildId}` : ""}:${channelId}:${userId}`;
}
export interface VoiceChannelLike {
type: string;
id: string;
guildId?: string | null;
}
export function isVoiceChannel(channel: VoiceChannelLike): boolean {
return (
channel.type === "DM" ||
channel.type === "GROUP_DM" ||
channel.type === "GUILD_STAGE_VOICE" ||
channel.type === "GUILD_VOICE"
);
}
@@ -21,7 +21,11 @@ import { config } from "../../shared/config/config.js";
import { initializeDatabase } from "../../shared/database/drizzle.js";
import { messageStore } from "../message-capture/messageStore.js";
import type { MessageRecord } from "../message-capture/types.js";
import { buildConversationContext } from "./conversationContext.js";
import {
buildConversationContext,
buildLocationContext,
} from "./conversationContext.js";
import { buildConversationContextBlock } from "./moderationBuilders.js";
import { runModerationAnalysis } from "./moderationOrchestrator.js";
const logger = createChildLogger("ai-analysis-worker");
@@ -274,29 +278,54 @@ async function processBatch(job: {
contextBefore,
targets: messages,
maxTokens: config.AI_ANALYSIS_MAX_CONTEXT_TOKENS,
maxAgeMs: config.AI_ANALYSIS_CONTEXT_MAX_AGE_MS,
gapMs: config.AI_ANALYSIS_CONTEXT_GAP_MS,
});
const contextBlock = buildConversationContextBlock({
location: buildLocationContext(messages),
descriptor: contextLines.descriptor,
lines: contextLines.lines,
});
const contextText = contextLines.join("\n");
const targetIds = messages.map((m) => m.id);
const allTargetIds = messages.map((m) => m.id);
const contextIds = contextBefore.map((m) => m.id);
const attachments = await messageStore.getAttachmentsForMessages([
...targetIds,
...allTargetIds,
...contextIds,
]);
// Attachment-upload race guard: a message whose attachment is still being
// uploaded (upload_status='pending') must not be analyzed yet. Its
// uploaded_url is not ready, and falling back to the Discord CDN link often
// 404s (expired/purged) — which used to silently produce a text-only
// verdict ("lampiran yang gagal terbaca"). Leave those targets pending; the
// next worker cycle picks them up after the upload lands.
const pendingUploadTargetIds = new Set(
(attachments ?? [])
.filter((a) => a.upload_status === "pending")
.map((a) => a.message_id),
);
const readyMessages =
pendingUploadTargetIds.size === 0
? messages
: messages.filter((m) => !pendingUploadTargetIds.has(m.id));
if (readyMessages.length === 0) {
return { ok: true, conversationKey, rows: [] };
}
// The orchestrator handles text/media split + caching + parallel paths
// internally, so a 20-message batch = 1 text LLM call (+1 media call
// when media is present), not N per-message calls.
const moderationResult = await runModerationAnalysis({
targets: messages,
contextText,
targets: readyMessages,
contextBlock,
attachments,
});
const results = moderationResult.results.map((r) =>
normalizeResult(
r as unknown as AnalysisResult,
messages.find((m) => m.id === r.messageId),
readyMessages.find((m) => m.id === r.messageId),
),
);
@@ -324,9 +353,10 @@ async function processBatch(job: {
logger.info(
{
total: messages.length,
total: readyMessages.length,
saved: allRows.length,
conversationKey,
skippedPendingUpload: messages.length - readyMessages.length,
},
"LLM batch analysis complete",
);
@@ -359,8 +389,14 @@ async function processIndividual(job: {
contextBefore,
targets: [message],
maxTokens: config.AI_ANALYSIS_MAX_CONTEXT_TOKENS,
maxAgeMs: config.AI_ANALYSIS_CONTEXT_MAX_AGE_MS,
gapMs: config.AI_ANALYSIS_CONTEXT_GAP_MS,
});
const contextBlock = buildConversationContextBlock({
location: buildLocationContext([message]),
descriptor: contextLines.descriptor,
lines: contextLines.lines,
});
const contextText = contextLines.join("\n");
const contextIds = contextBefore.map((m) => m.id);
const attachments = await messageStore.getAttachmentsForMessages([
@@ -368,10 +404,21 @@ async function processIndividual(job: {
...contextIds,
]);
// Same attachment-upload race guard as the batch path: while the upload is
// still in-flight the uploaded_url is not ready and the Discord CDN fallback
// often 404s — analyzing now would silently produce a text-only verdict.
// Return no results so the message stays pending for the next cycle.
const uploadStillPending = (attachments ?? []).some(
(a) => a.message_id === message.id && a.upload_status === "pending",
);
if (uploadStillPending) {
return { ok: true, results: [] };
}
try {
const moderationResult = await runModerationAnalysis({
targets: [message],
contextText,
contextBlock,
attachments,
});
@@ -47,6 +47,40 @@ export function deriveRecommendedAction(msg: MessageRecord): string {
return "none";
}
/** Parse the flag list from a structured result or the stored column. */
export function parseModerationFlags(
message: MessageRecord,
analysisResult?: AnalysisResult,
): string[] {
const flags = analysisResult?.flags ?? null;
if (flags && flags.length > 0) return flags;
const stored = message.ai_moderation_flags;
if (!stored) return [];
try {
const parsed = JSON.parse(stored) as unknown;
return Array.isArray(parsed)
? parsed.filter((f): f is string => typeof f === "string")
: [];
} catch {
return [];
}
}
/**
* True when the ONLY violation is the member's server nickname the message
* content itself is clean. Such messages must NOT be auto-deleted; the
* correct enforcement is resetting the nickname to the default username.
* Any other flag (sara, harassment, vulgar_language, ...) keeps the normal
* delete path.
*/
export function isNicknameOnlyViolation(
message: MessageRecord,
analysisResult?: AnalysisResult,
): boolean {
const flags = parseModerationFlags(message, analysisResult);
return flags.length > 0 && flags.every((f) => f === "offensive_username");
}
/**
* Check whether a message qualifies for auto-deletion.
* Uses the structured `analysisResult` fields when provided, falling back
@@ -1,9 +1,13 @@
import type { Client, PermissionString } from "discord.js-selfbot-v13";
import { LRUCache } from "lru-cache";
import { createChildLogger } from "@/shared/logger/index";
import { config } from "../../shared/config/config.js";
import { messageStore } from "../message-capture/messageStore.js";
import type { MessageRecord } from "../message-capture/types.js";
import { isEligibleForAutoDelete } from "./autoDeleteEligibility.js";
import {
isEligibleForAutoDelete,
isNicknameOnlyViolation,
} from "./autoDeleteEligibility.js";
import { logDeletionToChannel } from "./autoDeleteLogger.js";
import { sendDeletionNotification } from "./autoDeleteNotify.js";
@@ -15,6 +19,83 @@ export interface AutoDeleteResult {
reason: string;
}
// Cooldown per guild:user — a nick violation fires per message, but the
// Discord PATCH is idempotent; hammering it on every message by the same
// member is wasteful and risks rate limits.
const recentNicknameResets = new LRUCache<string, number>({
max: 200,
ttl: config.AUTO_NICKNAME_RESET_COOLDOWN_MS ?? 10 * 60 * 1000,
});
export function isNicknameResetInCooldown(
guildId: string,
userId: string,
): boolean {
return recentNicknameResets.has(`${guildId}:${userId}`);
}
/**
* Resets a member's server nickname to the default (global username)
* Discord's `setNickname(null)` removes the custom nick so the member is
* shown under their default username. Non-blocking; failures are logged
* but never throw into the moderation pipeline.
*/
export async function resetOffensiveNickname(
client: Client | undefined,
guildId: string,
userId: string,
messageId: string,
): Promise<boolean> {
const cooldownKey = `${guildId}:${userId}`;
try {
if (!client?.user?.id) {
logger.warn(
{ messageId, guildId, userId },
"Nick reset skipped: client missing",
);
return false;
}
if (userId === client.user.id) {
logger.debug({ userId }, "Nick reset skipped: operator's own account");
return false;
}
if (recentNicknameResets.has(cooldownKey)) {
logger.debug({ guildId, userId }, "Nick reset skipped: cooldown active");
return false;
}
if (config.AUTO_NICKNAME_RESET_ENABLED === false) return false;
const guild = client.guilds.cache.get(guildId);
if (!guild) {
logger.warn(
{ messageId, guildId },
"Nick reset skipped: guild not found",
);
return false;
}
const member = await guild.members.fetch(userId);
// setNickname(null) = remove nickname → Discord shows global username
await member.setNickname(null, "[auto] nickname melanggar aturan server");
recentNicknameResets.set(cooldownKey, Date.now());
logger.info(
{ messageId, guildId, userId },
"Offensive nickname reset to default username",
);
return true;
} catch (error) {
logger.warn(
{
messageId,
guildId,
userId,
error: error instanceof Error ? error.message : String(error),
},
"Nick reset failed",
);
return false;
}
}
// ─── Error Handling Utilities ────────────────────────────────────────
function getErrorCode(error: unknown): number | string | undefined {
@@ -107,6 +188,57 @@ export async function attemptAutoDeleteFlaggedMessage(
return { deleted: false, skipped: true, reason: "disabled" };
}
// ── Nickname-only violation: reset nick, DO NOT delete ─────────────
// When the only flag is offensive_username (message content is clean),
// the problem is the server nickname, not the message. Enforcement is
// removing the nickname back to the default username — the message stays.
if (isNicknameOnlyViolation(message)) {
if (
!config.AUTO_DELETE_FLAGGED_DRY_RUN &&
config.AUTO_NICKNAME_RESET_ENABLED !== false
) {
const inCooldown = isNicknameResetInCooldown(
message.guild_id,
message.user_id,
);
if (!inCooldown) {
const resetOk = await resetOffensiveNickname(
client,
message.guild_id,
message.user_id,
message.id,
);
try {
await messageStore.createModerationAction({
message_id: message.id,
user_id: message.user_id,
guild_id: message.guild_id,
action_type: "reset_nickname",
reason:
"nickname melanggar aturan server (offensive_username); pesan dibiarkan",
executed_by: "auto-delete-manager",
status: resetOk ? "executed" : "failed",
error: resetOk ? null : "nickname_reset_failed",
executed_at: resetOk ? Date.now() : null,
});
} catch (error) {
logger.warn(
{
messageId: message.id,
error: error instanceof Error ? error.message : String(error),
},
"Failed to persist nickname reset action log",
);
}
}
}
logger.info(
{ messageId: message.id, userId: message.user_id },
"Nickname-only violation: message kept, nickname reset attempted",
);
return { deleted: false, skipped: true, reason: "nickname_only_violation" };
}
// ── Status gate ──────────────────────────────────────────────────
if (message.ai_status !== "flagged" && message.ai_status !== "warn") {
@@ -6,6 +6,7 @@ import {
} from "../message-capture/messageMetadata.js";
import type { MessageRecord } from "../message-capture/types.js";
import { sanitizeDiscordTokens } from "./discordTokens.js";
import { escapeXml, resolveDisplayName } from "./moderationBuilders.js";
const logger = createChildLogger("conversationContext");
@@ -13,6 +14,26 @@ export interface ConversationContextInput {
contextBefore: MessageRecord[];
targets: MessageRecord[];
maxTokens: number;
/**
* Hard age cap for context messages (ms). Messages older than this
* relative to the target are stale conversation noise and dropped.
*/
maxAgeMs?: number;
/**
* Silence threshold (ms). A gap between consecutive context messages
* larger than this means the conversation restarted older messages
* belong to a previous conversation and are dropped.
*/
gapMs?: number;
}
export interface ConversationContextResult {
/** Formatted context lines (oldest → newest, recency-gated). */
lines: string[];
/** One-line flow descriptor: status, span, dropped counts. */
descriptor: string;
/** Number of context messages dropped by the recency gates. */
dropped: number;
}
let _encoder: ReturnType<typeof encodingForModel> | null = null;
@@ -103,26 +124,143 @@ export function formatMessageForPrompt(
msg: MessageRecord,
label: "context" | "target",
): string {
const content = sanitizeDiscordTokens(
renderDiscordMentions(msg.edited_content ?? msg.content, msg.metadata),
const content = truncateContextLine(
sanitizeDiscordTokens(
renderDiscordMentions(msg.edited_content ?? msg.content, msg.metadata),
),
);
const timestamp = formatTimestamp(msg.created_at);
const mediaEvidence = formatMediaEvidenceForPrompt(msg.metadata);
const mediaSuffix = mediaEvidence ? ` ${mediaEvidence}` : "";
const refInfo = formatReferenceInfo(msg);
return `[${label}] id=${msg.id} time=${timestamp} user=${msg.username}: ${content}${mediaSuffix}${refInfo}`;
return `[${label}] id=${msg.id} time=${timestamp} user=${resolveDisplayName(msg)}: ${content}${mediaSuffix}${refInfo}`;
}
/** Max content chars per context line a single huge paste (log dump,
* copypasta) must not eat the whole conversation budget. */
const CONTEXT_LINE_CONTENT_MAX_CHARS = 1500;
/** Marker appended when a context line's content was cut. Distinct from the
* target-content marker so the model knows which side was truncated. */
export const CONTEXT_TRUNC_MARKER = "…[konteks dipotong: terlalu panjang]";
/** Cap one context message's content to CONTEXT_LINE_CONTENT_MAX_CHARS. */
export function truncateContextLine(content: string): string {
if (content.length <= CONTEXT_LINE_CONTENT_MAX_CHARS) return content;
return `${content.slice(0, CONTEXT_LINE_CONTENT_MAX_CHARS).trimEnd()}${CONTEXT_TRUNC_MARKER}`;
}
/**
* Builds a structured `<location_context .../>` element for the batch
* channel/thread name and age-restriction flags from captured message
* metadata. The LLM uses it to judge messages in the right channel context
* (e.g. a thread about a specific topic, or an age-restricted channel).
* Returns "" when no channel metadata was captured.
*/
export function buildLocationContext(targets: MessageRecord[]): string {
const target = targets[0];
if (!target?.metadata) return "";
try {
const meta = JSON.parse(target.metadata) as {
channel?: {
channelName?: string | null;
threadName?: string | null;
topic?: string | null;
nsfw?: boolean;
ageRestricted?: boolean;
nsfwLevel?: string | null;
} | null;
};
const ch = meta?.channel;
if (!ch) return "";
const attrs: string[] = [`channel_id="${escapeXml(target.channel_id)}"`];
if (ch.channelName)
attrs.push(`channel_name="${escapeXml(ch.channelName)}"`);
if (target.thread_id || ch.threadName) {
if (target.thread_id)
attrs.push(`thread_id="${escapeXml(target.thread_id)}"`);
if (ch.threadName)
attrs.push(`thread_name="${escapeXml(ch.threadName)}"`);
}
if (typeof ch.topic === "string" && ch.topic.trim().length > 0) {
const topic =
ch.topic.length > 200
? `${ch.topic.slice(0, 200).trimEnd()}`
: ch.topic;
attrs.push(`topic="${escapeXml(topic)}"`);
}
if (typeof ch.nsfw === "boolean") attrs.push(`nsfw="${ch.nsfw}"`);
if (typeof ch.ageRestricted === "boolean") {
attrs.push(`age_restricted="${ch.ageRestricted}"`);
}
return `<location_context ${attrs.join(" ")}/>`;
} catch {
return "";
}
}
/**
* Builds conversation historical context without including targets.
* Calculates how much token budget targets use, and fills the rest with context.
*
* Two recency gates decide whether a conversation is STILL the same one
* ("obrolan berlanjut") or already restarted:
* - `gapMs`: a silence longer than this between two context messages cuts
* the block there earlier messages belong to a previous conversation.
* - `maxAgeMs`: anything older than this relative to the target is noise.
*
* On a cold start (no recent context), the nearest messages are kept as a
* sparse anchor and the descriptor says `cold_start` instead of `ongoing`,
* so the LLM does not mistake scattered old messages for an active chat.
*/
export function buildConversationContext(
input: ConversationContextInput,
): string[] {
): ConversationContextResult {
const { contextBefore, targets, maxTokens } = input;
const maxAgeMs = input.maxAgeMs ?? 45 * 60 * 1000;
const gapMs = input.gapMs ?? 12 * 60 * 1000;
// Calculate tokens used by targets (parallel)
const targetTime = targets.reduce(
(min, t) => Math.min(min, t.created_at),
targets[0]?.created_at ?? Date.now(),
);
// ── Recency gating (walk newest → oldest) ───────────────────────────────
const gated: MessageRecord[] = [];
let latestSelected: MessageRecord | null = null;
let gapBeforeMs: number | null = null;
let dropped = 0;
for (let i = contextBefore.length - 1; i >= 0; i--) {
const msg = contextBefore[i];
// Age gate
if (targetTime - msg.created_at > maxAgeMs) {
dropped += i + 1; // everything older also exceeds the age cap
break;
}
// Gap gate — silence between this message and the newer one already selected
if (latestSelected && latestSelected.created_at - msg.created_at > gapMs) {
gapBeforeMs = latestSelected.created_at - msg.created_at;
dropped += i + 1;
break;
}
gated.push(msg);
latestSelected = msg;
}
const gatedNewestFirst = gated.reverse();
let status: "ongoing" | "cold_start" | "sparse";
if (gatedNewestFirst.length === 0) {
// Cold start — keep a small anchor of the nearest messages so the LLM
// still senses the channel, but mark it clearly.
status = "cold_start";
gatedNewestFirst.push(...contextBefore.slice(-2)); // ± 2 nearest to target
} else if (gapBeforeMs === null) {
status = "ongoing";
} else {
status = "sparse";
}
// ── Format + token budget (most recent first, like before) ─────────────
const targetLines = targets.map((msg) =>
formatMessageForPrompt(msg, "target"),
);
@@ -131,7 +269,7 @@ export function buildConversationContext(
0,
);
const contextLines = contextBefore.map((msg) =>
const contextLines = gatedNewestFirst.map((msg) =>
formatMessageForPrompt(msg, "context"),
);
const selectedContextLines: string[] = [];
@@ -148,14 +286,26 @@ export function buildConversationContext(
}
}
const descriptorParts = [
`[conversation_flow] status=${status}`,
`context_msgs=${selectedContextLines.length}`,
`dropped=${dropped}`,
];
if (gapBeforeMs !== null) {
descriptorParts.push(`gap_before_min=${Math.round(gapBeforeMs / 60000)}`);
}
const descriptor = descriptorParts.join(" ");
logger.debug(
{
targetCount: targets.length,
contextCount: selectedContextLines.length,
status,
dropped,
usedTokens,
maxTokens,
},
"Conversation context built",
);
return selectedContextLines;
return { lines: selectedContextLines, descriptor, dropped };
}
@@ -56,7 +56,16 @@ export async function withLlmConcurrency<T>(fn: () => Promise<T>): Promise<T> {
*/
type LLMResponseChunk = {
choices?: Array<{
delta?: { content?: string | null };
delta?: {
content?: string | null;
reasoning_content?: string | null;
reasoning?: string | null;
reasoning_details?: Array<{
type?: string;
text?: string;
index?: number;
}> | null;
};
message?: { content?: string | null };
finish_reason?: string | null;
text?: string;
@@ -67,6 +76,39 @@ type LLMResponseChunk = {
finish_reason?: string;
};
/**
* Extract the textual payload from a single streaming chunk. Prefers
* `delta.content`; falls back to reasoning fields so reasoning-only models
* still produce usable aggregated text. Providers differ in the field name:
* - DeepSeek-style / Cloudflare gemma `delta.reasoning_content`
* - mimo (via 9router) streams reasoning in `delta.reasoning` +
* `delta.reasoning_details[].text` (content:"") without these fallbacks
* vision aggregation came back empty ("Vision API null response").
* Exported for unit tests.
*/
export function extractChunkText(
chunk: LLMResponseChunk | null | undefined,
): string {
if (!chunk) return "";
const choice = chunk.choices?.[0];
const reasoningDetails = choice?.delta?.reasoning_details
?.map((d) => d.text ?? "")
.filter(Boolean)
.join("");
return (
choice?.delta?.content ||
choice?.delta?.reasoning_content ||
choice?.delta?.reasoning ||
reasoningDetails ||
choice?.message?.content ||
choice?.text ||
chunk?.message?.content ||
chunk?.response ||
chunk?.content ||
""
);
}
// ---------------------------------------------------------------------------
// Lazy singleton — created on first use so that config is always resolved.
// ---------------------------------------------------------------------------
@@ -167,15 +209,7 @@ export async function llmChat(
let finishReason = "stop";
for await (const chunk of response as unknown as AsyncIterable<LLMResponseChunk>) {
const choice = chunk?.choices?.[0];
const textChunk =
choice?.delta?.content ||
choice?.message?.content ||
choice?.text ||
chunk?.message?.content ||
chunk?.response ||
chunk?.content ||
"";
content += textChunk;
content += extractChunkText(chunk);
const fr = choice?.finish_reason || chunk?.finish_reason;
if (fr) finishReason = fr;
}
@@ -16,8 +16,10 @@ import { getChannelCulture } from "./channelCultureStore.js";
import type { RetryState } from "./llmCaller.js";
import { callModerationLLM } from "./llmCaller.js";
import { prepareMediaMessage } from "./mediaAnalysisClient.js";
import { buildUserProfilesBlock } from "./moderationBuilders.js";
import { buildSystemPrompt as buildSystemPromptModular } from "./moderationPrompt.js";
import { buildCorrectedFewShotExamples } from "./textBatchProcessor.js";
import { getUserProfile } from "./userProfileStore.js";
const log = createChildLogger("mediaBatchProcessor");
@@ -26,7 +28,7 @@ const log = createChildLogger("mediaBatchProcessor");
// ---------------------------------------------------------------------------
export async function runMediaBatch(
targets: MessageRecord[],
contextText: string,
contextBlock: string,
attachments: AttachmentRecord[] | undefined,
): Promise<{ results: AnalysisResult[]; raw: unknown }> {
if (!targets.length) return { results: [], raw: null };
@@ -58,14 +60,41 @@ export async function runMediaBatch(
const channelCulture = channelCultureObj?.culture_summary;
const correctedExamples = await buildCorrectedFewShotExamples();
const systemText = buildSystemPromptModular({
contextText,
mode: "mixed",
correctedExamples,
channelCulture,
});
// Gather user profiles ONCE for the whole batch and emit a deduplicated
// <user_profiles> map (with last-generated timestamp); per-message blocks
// (from prepareMediaMessage) reference it via <user_profile_ref>.
const profileByUser = new Map<
string,
{
text: string;
asOf?: number | null;
}
>();
for (const t of targets) {
if (profileByUser.has(t.user_id)) continue;
const profile = await getUserProfile(t.user_id);
profileByUser.set(t.user_id, {
text: profile?.profile_summary ?? "",
asOf: profile?.last_analyzed_at ?? null,
});
}
const userProfilesBlock = buildUserProfilesBlock(profileByUser);
const messagesBlock = prepared.map((p) => p.messageBlock).join("\n");
const userContent = `<messages_to_analyze>\n${messagesBlock}\n</messages_to_analyze>`;
// Data/instruction separation: the system prompt is stable per mode — all
// per-batch context (profiles, conversation) lives in the USER payload,
// ordered oldest-first so targets come last.
const userBlocks = [
userProfilesBlock?.trimEnd() ?? "",
contextBlock?.trimEnd() ?? "",
`<messages_to_analyze>\n${messagesBlock}\n</messages_to_analyze>`,
].filter((b) => b.trim().length > 0);
const userContent = userBlocks.join("\n\n");
const perMsgTimeout = config.AI_LLM_MEDIA_ANALYSIS_TIMEOUT_MS ?? 60000;
const batchTimeout = Math.min(
@@ -296,101 +296,137 @@ export async function downloadAndExtractFrame(
imageMap: Map<string, MessageImagePart[]>,
): Promise<void> {
const log = createChildLogger("mediaAnalysis");
const urlToUse = att.uploaded_url ?? att.discord_url ?? null;
if (!urlToUse) return;
// Prefer the upload proxy (uploaded_url); the Discord CDN link can expire
// or be purged (404), and a non-OK response used to silently drop the image
// from vision analysis (no log, empty image map → text-only verdict). Try
// each candidate URL in order and surface failures.
const urlCandidates = [
att.uploaded_url,
att.discord_url && att.discord_url !== att.uploaded_url
? att.discord_url
: null,
].filter((u): u is string => Boolean(u));
if (urlCandidates.length === 0) return;
const { controller, clear } = createAbortControllerWithTimeout(15000);
try {
const res = await fetch(urlToUse, { signal: controller.signal });
if (!res.ok || !res.body) return;
let totalBytes = 0;
const chunks: Uint8Array[] = [];
const reader = res.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) break;
if (value) {
totalBytes += value.length;
if (totalBytes > 10 * 1024 * 1024) {
reader.cancel();
return;
}
chunks.push(value);
}
}
const imageBytes = Buffer.concat(chunks);
const sniffedMime = sniffImageMimeType(imageBytes);
if (!sniffedMime && att.type.startsWith("video/")) {
await extractVideoFrames(
att,
imageBytes,
targetId,
maxDimension,
imageMap,
);
return;
}
// Fallback: try attachment type metadata, then filename extension
let resolvedMime = sniffedMime;
if (!resolvedMime) {
if (att.type.startsWith("image/")) {
resolvedMime = att.type;
let imageBytes: Buffer | null = null;
let lastStatus = 0;
let lastError: string | null = null;
for (const urlToUse of urlCandidates) {
const { controller, clear } = createAbortControllerWithTimeout(15000);
try {
const res = await fetch(urlToUse, { signal: controller.signal });
if (!res.ok || !res.body) {
lastStatus = res.status;
log.warn(
{ attachmentId: att.id, filename: att.filename, type: att.type },
"Image MIME sniff failed — using attachment metadata type as fallback",
{
attachmentId: att.id,
urlHost: new URL(urlToUse).host,
status: res.status,
},
"Attachment fetch non-OK — trying next URL",
);
} else {
// Last resort: check file extension
const ext = att.filename?.toLowerCase().split(".").pop();
if (ext && ["jpg", "jpeg", "png", "gif", "webp", "bmp"].includes(ext)) {
const mimeMap: Record<string, string> = {
jpg: "image/jpeg",
jpeg: "image/jpeg",
png: "image/png",
gif: "image/gif",
webp: "image/webp",
bmp: "image/bmp",
};
resolvedMime = mimeMap[ext];
log.warn(
{ attachmentId: att.id, filename: att.filename, ext },
"Image MIME sniff failed — using file extension fallback",
);
continue;
}
let totalBytes = 0;
const chunks: Uint8Array[] = [];
const reader = res.body.getReader();
while (true) {
const { done, value } = await reader.read();
if (done) break;
if (value) {
totalBytes += value.length;
if (totalBytes > 10 * 1024 * 1024) {
reader.cancel();
return;
}
chunks.push(value);
}
}
}
// If all fallbacks fail, still try with generic image/jpeg
if (!resolvedMime) {
resolvedMime = "image/jpeg";
imageBytes = Buffer.concat(chunks);
break;
} catch (err) {
lastError = err instanceof Error ? err.message : String(err);
log.warn(
{ attachmentId: att.id, filename: att.filename },
"All MIME detection failed — forcing image/jpeg as last resort",
{
attachmentId: att.id,
urlHost: new URL(urlToUse).host,
error: lastError,
},
"Attachment download failed — trying next URL",
);
} finally {
clear();
}
}
const { data: resizedBuffer, mimeType: resizedMime } =
await resizeImageForVision(imageBytes, maxDimension);
const dataUrl = `data:${resizedMime};base64,${resizedBuffer.toString("base64")}`;
addImageToMap(imageMap, targetId, {
type: "image_url",
image_url: { url: dataUrl },
sourceLabel: `[gambar di atas adalah attachment ${att.filename} dari pesan id=${att.message_id}]`,
});
} catch (err) {
if (!imageBytes) {
log.warn(
{
attachmentId: att.id,
error: err instanceof Error ? err.message : String(err),
filename: att.filename,
lastStatus,
lastError,
},
"Download failed",
"All attachment URLs failed — skipping media analysis",
);
} finally {
clear();
return;
}
const sniffedMime = sniffImageMimeType(imageBytes);
if (!sniffedMime && att.type.startsWith("video/")) {
await extractVideoFrames(att, imageBytes, targetId, maxDimension, imageMap);
return;
}
// Fallback: try attachment type metadata, then filename extension
let resolvedMime = sniffedMime;
if (!resolvedMime) {
if (att.type.startsWith("image/")) {
resolvedMime = att.type;
log.warn(
{ attachmentId: att.id, filename: att.filename, type: att.type },
"Image MIME sniff failed — using attachment metadata type as fallback",
);
} else {
// Last resort: check file extension
const ext = att.filename?.toLowerCase().split(".").pop();
if (ext && ["jpg", "jpeg", "png", "gif", "webp", "bmp"].includes(ext)) {
const mimeMap: Record<string, string> = {
jpg: "image/jpeg",
jpeg: "image/jpeg",
png: "image/png",
gif: "image/gif",
webp: "image/webp",
bmp: "image/bmp",
};
resolvedMime = mimeMap[ext];
log.warn(
{ attachmentId: att.id, filename: att.filename, ext },
"Image MIME sniff failed — using file extension fallback",
);
}
}
}
// If all fallbacks fail, still try with generic image/jpeg
if (!resolvedMime) {
resolvedMime = "image/jpeg";
log.warn(
{ attachmentId: att.id, filename: att.filename },
"All MIME detection failed — forcing image/jpeg as last resort",
);
}
const { data: resizedBuffer, mimeType: resizedMime } =
await resizeImageForVision(imageBytes, maxDimension);
const dataUrl = `data:${resizedMime};base64,${resizedBuffer.toString("base64")}`;
addImageToMap(imageMap, targetId, {
type: "image_url",
image_url: { url: dataUrl },
sourceLabel: `[gambar di atas adalah attachment ${att.filename} dari pesan id=${att.message_id}]`,
});
}
// ---------------------------------------------------------------------------
@@ -494,8 +530,9 @@ export async function fetchUrlInline(
sourceLabel: `[gambar dari URL ${url} (inline), pesan id=${targetId}]`,
});
} else if (result.type === "text" && result.textContent) {
const titleAttr = result.title ? ` title="${escapeXml(result.title)}"` : "";
webTexts.push(
`<web_content url="${escapeXml(url)}">${escapeXml(result.textContent.slice(0, 2000))}</web_content>`,
`<web_content url="${escapeXml(url)}"${titleAttr}>${escapeXml(result.textContent.slice(0, 2000))}</web_content>`,
);
}
}
@@ -9,6 +9,7 @@ import { renderDiscordMentions } from "../message-capture/messageMetadata.js";
import { messageStore } from "../message-capture/messageStore.js";
import type { MessageRecord } from "../message-capture/types.js";
import { sanitizeDiscordTokens } from "./discordTokens.js";
import { sanitizeAiContent } from "./prompts/output.js";
/** Simple XML-escaping for content text. */
export function escapeXml(s: string): string {
@@ -19,6 +20,216 @@ export function escapeXml(s: string): string {
.replace(/"/g, "&quot;");
}
// ---------------------------------------------------------------------------
// Conversation context block — structured data for the USER message.
//
// All per-batch context lives in the USER message (not the SYSTEM prompt) so
// the system prompt is stable per mode (cacheable on routers/providers) and
// the role boundary is clean: instructions in SYSTEM, data in USER.
// ---------------------------------------------------------------------------
/** Outer char cap for the assembled `<conversation_context>` inner text. */
export const CONVERSATION_CONTEXT_MAX_CHARS = 40_000;
/**
* Wraps per-batch context data into structured XML blocks for the USER
* message:
*
* <location_context channel_id="..." channel_name="..." nsfw="..."/>
* <conversation_context>
* [conversation_flow] status=ongoing context_msgs=12 dropped=0
* [context] id=... time=... user=...: isi pesan
* ...
* </conversation_context>
*
* Empty blocks are omitted entirely (never emit a hollow `<conversation_context>`
* with no content). The inner text is AI/user-derived and passed through
* `sanitizeAiContent` (CDATA + XML-escape) to block prompt injection.
*/
export function buildConversationContextBlock(input: {
/** Pre-built `<location_context .../>` string (or ""). */
location?: string;
/** `[conversation_flow]` descriptor line from buildConversationContext. */
descriptor?: string;
/** `[context]` lines, oldest → newest. */
lines: string[];
}): string {
const blocks: string[] = [];
const location = input.location?.trim();
if (location) blocks.push(location);
const inner = [input.descriptor ?? "", ...input.lines]
.map((line) => line.trim())
.filter((line) => line.length > 0)
.join("\n");
if (inner) {
blocks.push(
`<conversation_context>\n${sanitizeAiContent(inner, CONVERSATION_CONTEXT_MAX_CHARS)}\n</conversation_context>`,
);
}
return blocks.join("\n");
}
// ---------------------------------------------------------------------------
// Per-message content bounds — protects the LLM token budget from a single
// huge paste (stack traces, log dumps, copypasta). Truncation is explicit so
// the model never mistakes the cut for a real message boundary.
// ---------------------------------------------------------------------------
/** Max characters of a message's content sent to the LLM `<content>` payload. */
export const AI_CONTENT_MAX_CHARS = 4000;
/** Marker appended when a message is longer than AI_CONTENT_MAX_CHARS. */
export const AI_CONTENT_TRUNC_MARKER = "\n…[pesan dipotong: terlalu panjang]";
/** Truncate a message's content for the LLM `<content>` payload. */
export function truncateForAi(content: string): string {
if (content.length <= AI_CONTENT_MAX_CHARS) return content;
return `${content.slice(0, AI_CONTENT_MAX_CHARS)}${AI_CONTENT_TRUNC_MARKER}`;
}
// ---------------------------------------------------------------------------
// User profile deduplication — a batch can contain many messages from the
// same user. Instead of repeating the (up to 3000-char) profile summary on
// every message, emit a single <user_profiles> map per batch and reference
// entries per message with <user_profile_ref user_id="..."/>.
// ---------------------------------------------------------------------------
export interface UserProfileEntry {
/** Profile summary text (from user_profiles.profile_summary). */
text: string;
/** Epoch ms when the profile was last generated staleness signal for
* the LLM (a profile from months ago may not reflect current behavior). */
asOf?: number | null;
}
/** Build a deduplicated `<user_profiles>` map block, keyed by Discord user id. */
export function buildUserProfilesBlock(
profiles: ReadonlyMap<string, UserProfileEntry>,
): string {
const entries = Array.from(profiles.entries()).filter(
([, entry]) => entry.text.trim().length > 0,
);
if (entries.length === 0) return "";
const lines = entries.map(([userId, entry]) => {
const asOfAttr =
typeof entry.asOf === "number" && entry.asOf > 0
? ` as_of="${new Date(entry.asOf).toISOString()}"`
: "";
return ` <user_profile user_id="${escapeXml(userId)}"${asOfAttr}>${sanitizeAiContent(entry.text)}</user_profile>`;
});
return `<user_profiles>\n${lines.join("\n")}\n</user_profiles>`;
}
/** Per-message reference tag pointing at an entry in the `<user_profiles>` map. */
export function buildUserProfileRef(userId: string): string {
return `<user_profile_ref user_id="${escapeXml(userId)}"/>`;
}
// ---------------------------------------------------------------------------
// User reputation — richer than a bare trust score.
//
// The trust model tracks total_infractions, a clean-message streak and the
// last infraction timestamp. Feeding all of it to the LLM lets it tell a
// first-timer (same score, 1 infraction) from a repeat offender (score 50,
// 3 infractions, last one yesterday) — the same score means very different
// things in those two contexts.
// ---------------------------------------------------------------------------
export interface ReputationAttrsSource {
trust_score: number;
total_infractions: number;
clean_message_streak: number;
last_infraction_at: number | null;
}
const DAY_MS = 24 * 60 * 60 * 1000;
const REPEAT_OFFENSE_WINDOW_MS = 7 * DAY_MS;
/**
* Formats reputation fields into XML attributes for `<user_reputation .../>`.
* Derived signals: last_offense_days_ago (0 = today) and repeat_offender
* (infraction within the last 7 days) are computed here so both the text and
* media paths emit the exact same shape.
*/
export function formatReputationAttrs(
rep: ReputationAttrsSource,
now: number = Date.now(),
): string {
const attrs = [
`trust_score="${rep.trust_score}"`,
`total_infractions="${rep.total_infractions}"`,
`clean_streak="${rep.clean_message_streak}"`,
];
if (
typeof rep.last_infraction_at === "number" &&
rep.last_infraction_at > 0
) {
const daysAgo = Math.max(
0,
Math.floor((now - rep.last_infraction_at) / DAY_MS),
);
attrs.push(`last_offense_days_ago="${daysAgo}"`);
const isRepeat =
rep.total_infractions > 0 &&
now - rep.last_infraction_at <= REPEAT_OFFENSE_WINDOW_MS;
if (isRepeat) attrs.push(`repeat_offender="true"`);
}
return attrs.join(" ");
}
/**
* Builds an optional `<user_history>` block (last flagged messages) from
* getUserRecentInfractions rows. Only emitted when there is real history
* lets the LLM see the PATTERN (e.g. the same scam link posted repeatedly)
* without treating old flags as proof for the current message.
*/
export function buildUserHistoryXml(
history: Array<{
content: string;
severity: string | null;
created_at: number;
}>,
now: number = Date.now(),
): string {
const filtered = history.filter((h) => h.content?.trim());
if (filtered.length === 0) return "";
const lines = filtered.map((h) => {
const daysAgo = Math.max(0, Math.floor((now - h.created_at) / DAY_MS));
const severityAttr = h.severity
? ` severity="${escapeXml(h.severity)}"`
: "";
const snippet =
h.content.length > 100
? `${h.content.slice(0, 100).trimEnd()}`
: h.content;
return ` <infraction${severityAttr} time_ago_days="${daysAgo}">${escapeXml(snippet)}</infraction>`;
});
return `<user_history>\n${lines.join("\n")}\n</user_history>`;
}
/**
* Whether the message author was a bot (captured in metadata.author.bot).
* Bot posts (logging bots, webhook-style automation) deserve different
* scrutiny than user posts expose the flag instead of hiding it.
*/
export function resolveIsBot(msg: MessageRecord): boolean {
if (!msg.metadata) return false;
try {
const meta = JSON.parse(msg.metadata) as {
author?: { bot?: boolean } | null;
};
return Boolean(meta?.author?.bot);
} catch {
return false;
}
}
/** Whether the shown content is an EDIT of the original post (evasion signal). */
export function resolveIsEdited(msg: MessageRecord): boolean {
return Boolean(msg.edited_content);
}
/**
* Returns the real text content for AI analysis, stripping fallback text
* that getDisplayContent() synthesized ("[Attachment: ...]", "[Sticker: ...]",
@@ -36,6 +247,27 @@ export function getAnalysisContent(message: MessageRecord): string {
).trim();
}
/**
* Server nickname (member.displayName) when captured, else the author
* username. Discord shows the server nickname to other members, so the LLM
* should see the same name the channel sees and a nickname can carry
* moderation signal itself (offensive nick + clean message low warn).
*/
export function resolveDisplayName(msg: MessageRecord): string {
if (msg.metadata) {
try {
const meta = JSON.parse(msg.metadata) as {
member?: { displayName?: string | null } | null;
};
const dn = meta?.member?.displayName;
if (dn && dn.trim().length > 0) return dn;
} catch {
// malformed metadata — fall back to username
}
}
return msg.username;
}
/**
* Builds a <reference> XML element for reply/forward/crosspost context.
*/
@@ -35,7 +35,13 @@ const log = createChildLogger("moderationOrchestrator");
// ---------------------------------------------------------------------------
export interface ModerationInput {
targets: MessageRecord[];
contextText: string;
/**
* Pre-built XML context block for the USER message (from
* `buildConversationContextBlock`): `<location_context .../>` +
* `<conversation_context>...</conversation_context>`. Kept out of the
* system prompt so it stays stable/cacheable per mode.
*/
contextBlock: string;
attachments?: AttachmentRecord[];
}
@@ -62,7 +68,7 @@ export interface ModerationOutput {
export async function runModerationAnalysis(
input: ModerationInput,
): Promise<ModerationOutput> {
const { targets, contextText, attachments } = input;
const { targets, contextBlock, attachments } = input;
initSearxngCache(config.REDIS_URL);
if (!targets.length) throw new Error("No targets provided for analysis");
@@ -320,10 +326,10 @@ export async function runModerationAnalysis(
// Run both paths in parallel
const [textBatchResult, mediaBatchResult] = await Promise.all([
textOnlyTargets.length > 0
? runTextOnlyBatch(textOnlyTargets, contextText)
? runTextOnlyBatch(textOnlyTargets, contextBlock)
: Promise.resolve({ results: [] as AnalysisResult[], raw: null }),
mediaTargets.length > 0
? runMediaBatch(mediaTargets, contextText, attachments)
? runMediaBatch(mediaTargets, contextBlock, attachments)
: Promise.resolve({ results: [] as AnalysisResult[], raw: null }),
]);
@@ -31,11 +31,16 @@ Struktur wajib:
]
}
Instruksi per field:
- "message_id": WAJIB sama persis dengan id di input. Setiap <message> di <messages_to_analyze> menghasilkan SATU hasil. Jangan gabungkan beberapa pesan, jangan lewati, jangan karang id.
- "evidence": kutipan PERSIS frasa yang melanggar (maks 1 baris). Pelanggaran di gambar/sticker kutip deskripsi Media analysis. Pelanggaran lewat balasan/referensi sebut konteks pesan yang dibalas. Boleh tambah label sumber, mis. [media analysis] / [web_search] / [reply]. Kosong jika clean.
## PERSONALITY & MEMORI Profil Pengguna dan Kultur Channel
Data konteks tersedia: <user_profile> (ringkasan kepribadian pengguna) dan <channel_culture> (topik/vibe channel).
Data konteks tersedia: <user_profiles> (peta ringkasan kepribadian, di pesan USER), <user_reputation> (skor trust), dan <channel_culture> (topik/vibe channel). Setiap <message> dapat memuat <user_profile_ref user_id="..."/> yang menunjuk ke entri di peta <user_profiles>.
Gunakan untuk personalisasi analysis, tapi:
- Profil adalah KONTEKS, bukan bukti. Profil mencurigakan flag; profil bersih loloskan pelanggaran.
- Perubahan perilaku mencolok (biasanya teknis tiba-tiba provokatif) layak dicatat di analysis.
- <user_history> (kutipan pesan yang pernah di-flag) = pola pelanggaran lama. Gunakan untuk mendeteksi PENGULANGAN (mis. spam link yang sama, provokasi berulang), tapi JANGAN memflag pesan bersih hanya karena riwayat.
- JANGAN paksa referensi profil jika tidak relevan analysis natural lebih baik.
- Channel culture coding/teknis pesan teknis lebih wajar; channel santai slang lebih wajar. Jangan dipakai mengabaikan pelanggaran nyata.
@@ -54,6 +59,7 @@ Contoh buruk: "Pesan berisi teks dan gambar tanpa pelanggaran." (mengabaikan buk
- **conflict_instigation:** "Pengirim <ajakan memicu konflik>. <konteks>. Diberi peringatan karena berpotensi memicu drama."
- **Username ofensif (pesan bersih):** "Pengirim memiliki username yang <alasan ofensif>. Isi pesan hanya <isi>. Diberi warning ringan." (pesan memperkuat): "<username SARA> + isi pesan memperkuat tone kebencian. Pelanggaran berat."
- **Evasi (zalgo/leetspeak):** "Pengirim menggunakan teknik obfuscation untuk menyembunyikan <makna asli>. <dampak>. <kesimpulan>."
- **Spam (repetitions > 1):** "Pengirim mengirim teks yang sama sebanyak N kali dalam waktu singkat. <isi pesan>. Diberi peringatan karena spam berulang." nilai tetap dari isi; pengulangan saja (mis. "ok" x5 dalam obrolan aktif) bukan pelanggaran.
- **sexual_deviation:** "Pengirim <konten penyimpangan>. <konteks>. Melanggar kebijakan server."
- **SARA/penistaan agama:** "Pengirim <jenis penistaan spesifik: parodi ayat, mengaku Tuhan, mockery ritual, istilah agama sebagai joke, provokasi antar-agama>. <bukti>. Melanggar kebijakan SARA." JANGAN gunakan kata "bercanda" untuk SARA.
@@ -66,7 +72,7 @@ CRITICAL:
- Jika pesan adalah BALASAN (reply) ke pesan lain, jelaskan konteks balasannya: apa yang sedang dibicarakan, siapa yang dibalas (tanpa nama, cukup peran/isi pesan yang dibalas), dan bagaimana tanggapan pengirim terhadapnya.
- Gunakan informasi dari Media analysis untuk mendeskripsikan gambar.
- Analisis harus MEMBERI KONTEKS, bukan hanya menyatakan status.
- GUNAKAN <user_profile> untuk personalisasi analysis jadikan analysis terasa seperti sistem "mengenal" pengguna.
- GUNAKAN <user_profile_ref>/<user_profiles> untuk personalisasi analysis jadikan analysis terasa seperti sistem "mengenal" pengguna.
- Jika perilaku pesan menyimpang dari profil yang diketahui, CATAT dalam analysis sebagai informasi kontekstual yang relevan.
- JANGAN paksa referensi profil jika tidak relevan analysis natural lebih baik dari yang dipaksakan.`;
@@ -12,6 +12,7 @@ export const SYSTEM_RULES = `Kamu adalah asisten moderasi konten untuk server Di
## Normalisasi & Pertahanan Lintas Bahasa (WAJIB)
1. Campuran bahasa (Inggris/Indonesia/daerah) WAJIB dinormalisasi mental ke Bahasa Indonesia sebelum menilai intent. Jangan longgar hanya karena sintaksis campur (Polyglot Obfuscation).
2. Lakukan Named Entity Recognition agresif nama orang/karakter (mis. "ren" setelah kata archaic "diagem") tetap dikenali sebagai nama.
3. <term_glossary> (bila ada) = definisi kata/slang/jargon yang tidak umum. Baca dulu arti kata yang tidak kamu kenal dari sana jangan menebak dari bunyi/kemiripan. Kata yang tampak mencurigakan namun ternyata bermakna netral di glossary = AMAN; kata asing yang ternyata vulgar/terlarang di glossary = FLAG.
## Aturan Umum (AMAN jangan flag)
- Slang: anjay, wkwk, gws, gaskeun, santuy, njir, baka, woy/woi, hadeh, astaga = AMAN.
@@ -73,6 +74,7 @@ RENDAH: harassment, vulgar_language terarah, offensive_username (Scunthorpe: "Sa
## Web Sebagai Bukti Utama
- <web_searches> ADALAH BUKTI UTAMA. Jika ada, WAJIB pakai hasilnya (hentai/scam/narkoba flag; aman clean). JANGAN abaikan. Jika tidak ada gunakan pengetahuan internal.
- <term_glossary> = REFERENSI ARTI KATA, bukan bukti pelanggaran. Dipakai untuk memahami istilah yang tidak dikenal sebelum memutuskan.
- Prioritas bukti: <web_searches> > <web_content> > <media_analysis> > pengetahuan internal. <web_content> (URL fetch): gunakan isi, jangan flag hanya dari domain name.
## Pohon Keputusan
@@ -39,7 +39,6 @@ Gambar/sticker/embed/preview link sudah DIDESKRIPSIKAN vision model sebelum batc
// ---------------------------------------------------------------------------
export interface BuildSystemPromptOptions {
contextText: string;
/** Prompt mode — determines which sections are included. */
mode: PromptMode;
/** @deprecated Use `mode` instead. */
@@ -59,7 +58,6 @@ export interface BuildSystemPromptOptions {
export function buildSystemPrompt(options: BuildSystemPromptOptions): string {
const {
contextText,
mode,
includeMediaInstructions,
correction,
@@ -105,15 +103,40 @@ export function buildSystemPrompt(options: BuildSystemPromptOptions): string {
}
parts.push(
`## Konteks Pengguna\nSetiap pesan mungkin memiliki tag <user_reputation>. Tag ini hanya indikator **referensi**, bukan bukti pelanggaran. Nilai trust_score yang rendah bukan alasan untuk memflag pesan yang bersih. Nilai trust_score yang tinggi bukan alasan untuk mengabaikan pelanggaran nyata. **Setiap pesan harus dinilai berdasarkan isinya sendiri.**`,
`## Blok Data di Pesan USER\n` +
`Semua data dinamis per-batch dikirim di pesan USER — system prompt ini TIDAK memuat data batch:\n` +
`- <location_context .../> = metadata channel/thread (channel_id, channel_name, thread_name, topic, nsfw, age_restricted). topic = deskripsi resmi channel — pakai untuk menilai kesesuaian pesan dengan tujuan channel.\n` +
`- <conversation_context> = obrolan SEBELUM pesan target. Baris "[context]" di dalamnya BUKAN yang dinilai.\n` +
`- <user_profiles> = peta ringkasan kepribadian per user_id (attr as_of = kapan profil terakhir dibuat — profil lama mungkin tidak mencerminkan perilaku terkini); setiap <message> merujuk lewat <user_profile_ref user_id="..."/>.\n` +
`- <web_searches> / <web_content> = bukti web (lihat "Web Sebagai Bukti Utama").\n` +
`- <term_glossary> = kamus istilah: definisi kata/slang/jargon yang jarang dikenal (hasil pencarian Wikipedia via SearXNG). Gunakan untuk memahami arti kata yang tidak kamu kenal — JANGAN menebak atau mengarang arti.\n` +
`- <messages_to_analyze> = pesan-pesan TARGET yang WAJIB dinilai. Atribut <message>: id, user (nama server), time (ISO — kapan pesan dikirim), repetitions (N = teks pendek sama muncul N kali di batch — sinyal spam), bot (true jika dari bot), edited (true jika konten adalah hasil edit setelah posting).`,
);
parts.push(
`## Konteks Pengguna (Referensi, Bukan Bukti)\n` +
`Konteks per pengguna hanya indikator **referensi** untuk personalisasi analisis, BUKAN bukti pelanggaran:\n` +
`- <user_reputation trust_score="..." total_infractions="..." clean_streak="..." last_offense_days_ago="..." repeat_offender="..."> = histori moderasi pengguna. Skor rendah BUKAN alasan memflag pesan bersih; skor tinggi BUKAN alasan mengabaikan pelanggaran nyata. repeat_offender="true" = ada pelanggaran dalam 7 hari terakhir.\n` +
`- <user_history> (di dalam <user_reputation>) = kutipan pesan-pesan pengguna yang PERNAH di-flag. Gunakan untuk mengenali POLA berulang (spam link sama, provokasi), tapi JANGAN memflag pesan bersih hanya karena riwayat.\n` +
`- <user_profiles> (di pesan USER) = peta ringkasan kepribadian per user_id. <user_profile_ref user_id="..."/> dalam sebuah pesan menunjuk ke peta itu. Tanpa ref = tidak ada profil untuk pengguna tersebut.\n` +
`- Profil berguna untuk mengenali penyimpangan perilaku mencolok (mis. pengguna teknis tiba-tiba provokatif), tapi JANGAN memflag atau meloloskan hanya karena profil.\n` +
`**Setiap pesan dinilai berdasarkan isinya sendiri.**`,
);
parts.push(
`## Framing: Konteks vs Target\n` +
`- Baris dalam <conversation_context> berformat "[context] id=... time=<ISO> user=<nama>: isi", diurutkan paling lama → paling baru. Baris pertama biasanya "[conversation_flow] status=... context_msgs=... dropped=..." — metadata sistem tentang status percakapan (ongoing/sparse/cold_start), BUKAN pesan yang dinilai.\n` +
`- <messages_to_analyze> berisi pesan-pesan TARGET yang WAJIB dinilai. Hasilkan SATU hasil per message_id — jangan menggabungkan beberapa pesan, jangan melewati, jangan mengarang id.\n` +
`- Setiap target dinilai berdasarkan isinya sendiri; konteks percakapan memengaruhi interpretasi, bukan menggantikan isi pesan.\n` +
`- Marker "…[pesan dipotong: terlalu panjang]" = konten TARGET sengaja dipotong; marker "…[konteks dipotong: terlalu panjang]" = konten pesan KONTEKS dipotong. Nilai dari bagian yang terlihat; pemotongan BUKAN pelanggaran dan BUKAN teknik evasi.\n` +
`- Atribut time= pada <message> target = kapan pesan dikirim (ISO). Pakai untuk menilai kerelevanan waktu (mis. pesan lama di-bump, spam beruntun dalam menit yang sama).\n` +
`- repetitions="N" pada <message> = teks pendek yang sama muncul N kali dalam batch — pertimbangkan sebagai sinyal spam, tapi nilai tetap dari isi pesan.\n` +
`- bot="true" = pengirim adalah bot (otomatisasi), bukan pengguna manusia — jangan perlakukan sebagai pelanggaran personal, tapi kontennya tetap dinilai.\n` +
`- edited="true" = konten yang ditampilkan adalah hasil edit setelah posting (sinyal potensi evasi), nilai konten saat ini apa adanya.`,
);
parts.push(OUTPUT_INSTRUCTIONS);
// XML-delimited context — prevents prompt injection
const delimitedContext = `<conversation_context>\n${sanitizeAiContent(contextText, 8000)}\n</conversation_context>`;
parts.push(delimitedContext);
let base = parts.join("\n\n");
if (correction) {
@@ -1,10 +1,11 @@
import Redis from "ioredis";
import { createChildLogger } from "@/shared/logger/index";
import { createAbortControllerWithTimeout } from "@/shared/utils/index";
import { config } from "../../shared/config/config.js";
const log = createChildLogger("searxng-search");
const SEARXNG_BASE_URL = "https://searxng.imrnes.team";
const SEARXNG_BASE_URL = config.SEARXNG_BASE_URL;
const MAX_RESULTS = 3;
const TIMEOUT_MS = 8000;
const CACHE_TTL = 86400; // 24 hours
@@ -12,6 +13,42 @@ const CACHE_PREFIX = "searxng:";
let redis: Redis | null = null;
/**
* Exposes the shared SearXNG Redis connection so other modules (e.g. the
* term glossary) reuse the same connection and cache prefix instead of
* opening their own. Returns null when Redis is unavailable.
*/
export function getSearxngRedis(): Redis | null {
return redis;
}
/** Builds a namespaced SearXNG cache key (shared across modules). */
export function makeSearxngCacheKey(namespace: string, key: string): string {
return `${CACHE_PREFIX}${namespace}:${key.toLowerCase().trim()}`;
}
/** Reads a value from the SearXNG Redis cache; null on miss/unavailable. */
export async function searxngCacheGet(key: string): Promise<string | null> {
if (!redis) return null;
try {
return await redis.get(key);
} catch {
return null;
}
}
/** Writes a value to the SearXNG Redis cache, fire-and-forget. */
export function searxngCacheSet(
key: string,
value: string,
ttlSeconds: number,
): void {
if (!redis) return;
redis.setex(key, ttlSeconds, value).catch(() => {
// Cache write failed silently
});
}
/**
* Initialize Redis connection for SearXNG cache.
* Safe to call multiple times only creates one connection.
@@ -51,19 +88,26 @@ export interface SearxngResult {
/**
* Search SearXNG for a query and return structured results.
* Uses Redis cache when available same query within 24h returns cached results.
*
* @param engines Optional comma-separated SearXNG engine list to constrain
* the search (e.g. "wikipedia"). When set, results are cached under a
* separate cache namespace so engine-specific results never collide.
*/
export async function searchSearxng(
query: string,
category: "general" | "news" | "science" = "general",
engines?: string,
timeoutMs: number = TIMEOUT_MS,
): Promise<SearxngResult[]> {
const cacheKey = `${CACHE_PREFIX}${category}:${query.toLowerCase().trim()}`;
const engineNs = engines ? `eng:${engines}` : "auto";
const cacheKey = makeSearxngCacheKey(`${category}:${engineNs}`, query);
// Try cache first
if (redis) {
try {
const cached = await redis.get(cacheKey);
if (cached) {
log.debug({ query, category }, "SearXNG cache HIT");
log.debug({ query, category, engines }, "SearXNG cache HIT");
return JSON.parse(cached) as SearxngResult[];
}
} catch {
@@ -73,8 +117,11 @@ export async function searchSearxng(
// Cache miss — hit SearXNG API
try {
const url = `${SEARXNG_BASE_URL}/search?q=${encodeURIComponent(query)}&format=json&language=id&categories=${category}`;
const { controller, clear } = createAbortControllerWithTimeout(TIMEOUT_MS);
const engineParam = engines
? `&engines=${encodeURIComponent(engines)}`
: "";
const url = `${SEARXNG_BASE_URL}/search?q=${encodeURIComponent(query)}&format=json&language=id&categories=${category}${engineParam}`;
const { controller, clear } = createAbortControllerWithTimeout(timeoutMs);
try {
const response = await fetch(url, {
@@ -0,0 +1,551 @@
/**
* termGlossary.ts
*
* Per-word "kamus" enrichment for LLM moderation.
*
* Problem: the moderation LLM often meets words it does not know regional
* slang (Jawa/Sunda), foreign terms, niche anime/game jargon, or obscure
* technical vocabulary. When it guesses, it either invents a wrong meaning
* (false positive on a safe word) or misses a violation hidden in unfamiliar
* wording (false negative on an unknown vulgar/slang term).
*
* Solution: extract candidate "unknown-looking" words from message content,
* look each one up on Wikipedia via SearXNG, and inject the definitions into
* the LLM prompt as a `<term_glossary>` block so verdicts are based on facts
* instead of guesses.
*
* Cost control & persistence:
* - successfully resolved definitions are PERSISTED PERMANENTLY in Postgres
* (`term_glossary_cache`) definitions rarely change, so a resolved term
* is never searched again; only misses stay ephemeral (Redis/LRU, 1h);
* - in-memory LRU + Redis (shared with the SearXNG cache) sit in front of
* the DB as fast read caches, so repeat lookups are effectively free;
* - lookups per batch are bounded (AI_GLOSSARY_MAX_TERMS);
* - live SearXNG calls are rate-limit aware: concurrency 2 + stagger, retry
* once on empty results, and misses cached for only 1h so a limiter/
* network blip is not treated as a permanent miss;
* - only results that read like actual definitions are accepted (Wikipedia
* preferred; disambiguation/ads/translate-homepages rejected);
* - everything degrades gracefully: no Redis, no SearXNG, no match
* the block is simply omitted and moderation proceeds as before.
*/
import { LRUCache } from "lru-cache";
import pLimit from "p-limit";
import { createChildLogger } from "@/shared/logger/index";
import { delay } from "@/shared/utils/index";
import { config } from "../../shared/config/config.js";
import { escapeXml } from "./moderationBuilders.js";
import {
makeSearxngCacheKey,
searchSearxng,
searxngCacheGet,
searxngCacheSet,
} from "./searxngSearch.js";
import {
getTermDefinitionFromDb,
setTermDefinitionInDb,
} from "./termGlossaryStore.js";
const log = createChildLogger("term-glossary");
// ---------------------------------------------------------------------------
// Constants
// ---------------------------------------------------------------------------
/** Redis TTL for a successfully resolved definition (definitions are stable). */
const DEF_TTL_SECONDS = 7 * 24 * 60 * 60;
/**
* Redis TTL for a lookup that found nothing. Kept SHORT (1h): SearXNG
* instances silently return empty result sets when rate-limited, so an empty
* response is often a transient failure, not a real miss. A short TTL lets
* the term be retried on a later batch instead of poisoning it for a day.
*/
const MISS_TTL_SECONDS = 60 * 60;
const MISS_TTL_MS = MISS_TTL_SECONDS * 1000;
/** Sentinel stored in caches for "term has no resolvable definition". */
const EMPTY_SENTINEL = "__not_found__";
/** Per-search timeout — keep glossary lookups snappy even on a slow SearXNG. */
const GLOSSARY_SEARCH_TIMEOUT_MS = 5000;
/** Delay before retrying a search that returned zero results. */
const RETRY_DELAY_MS = 350;
/** Max definition snippet length kept in the prompt. */
const MAX_DEFINITION_CHARS = 300;
/**
* SearXNG rate-limits aggressive parallel bursts (returns 200 with empty
* results). Never fire all terms at once cap live searches at 2 concurrent
* and stagger the start times slightly.
*/
const LIVE_SEARCH_CONCURRENCY = 2;
const LIVE_SEARCH_STAGGER_MS = 250;
/** In-memory cache: term (lowercase) → definition | NOT_FOUND sentinel. */
const NOT_FOUND: TermDefinition = {
term: "__not_found__",
definition: "",
sourceUrl: "",
};
const termLru = new LRUCache<string, TermDefinition>({
max: 2000,
ttl: 24 * 60 * 60 * 1000,
});
/** Serializes live SearXNG lookups (rate-limit aware) with a small stagger. */
const liveSearchLimit = pLimit(LIVE_SEARCH_CONCURRENCY);
let lastLiveSearchAt = 0;
async function acquireLiveSlot(): Promise<void> {
const now = Date.now();
const wait = lastLiveSearchAt + LIVE_SEARCH_STAGGER_MS - now;
if (wait > 0) await delay(wait);
lastLiveSearchAt = Date.now();
}
// ---------------------------------------------------------------------------
// Term extraction
// ---------------------------------------------------------------------------
/** Word tokenizer letters/digits plus internal -_'· (handles "well-known",
* "node_modules", diacritics). */
const WORD_RE = /[\p{L}\p{N}]+(?:[-_'·][\p{L}\p{N}]+)*/gu;
/** Removes URLs, Discord mentions/custom emoji, code fences, markdown noise. */
function cleanContent(raw: string): string {
return raw
.replace(/https?:\/\/\S+/gi, " ")
.replace(/<@!?\d+>/g, " ")
.replace(/<#\d+>/g, " ")
.replace(/<a?:\w+:\d+>/g, " ")
.replace(/[`*_~|>[\]]/g, " ")
.replace(/[\p{Emoji}\p{Extended_Pictographic}]/gu, " ")
.replace(/\s+/g, " ")
.trim();
}
/** Filters out tokens that are useless as glossary candidates (numbers,
* repeated-char noise, mega-tokens). */
function isNoiseWord(word: string): boolean {
if (word.length > 28) return true;
if (/^\d+$/.test(word)) return true;
const lower = word.toLowerCase();
// "aaaa…", "wwwwww" — single repeated character
if (/^(.)\1{2,}$/.test(lower)) return true;
// "wkwk", "hehe", "69" alternations — repeated 23 char base. "meme" is
// the one legit 4-letter word this matches; it is whitelisted below.
if (/^([a-z]{2,3})\1{1,}$/.test(lower)) return true;
return false;
}
/** Deterministic bonus for words that look like proper nouns or foreign. */
function scoreWord(word: string): number {
let score = 1;
// Capitalized first letter (proper noun / title) but not ALL-CAPS acronyms
if (/^[A-Z]/.test(word) && !/^[A-Z]{2,}$/.test(word)) score += 3;
// Contains a letter outside basic latin → regional/foreign spelling
if (/[\p{L}]/u.test(word.replace(/[A-Za-z]/g, ""))) score += 2;
// Contains an internal apostrophe or hyphen → likely a named entity
if (/[-_'’·]/.test(word)) score += 2;
return score;
}
const STOPWORDS = new Set(
// ── Bahasa Indonesia ────────────────────────────────────────────────
(
" yang dan di ke dari ini itu dengan untuk pada dalam adalah akan telah sudah bisa dapat harus tidak juga saya kamu kita kami mereka dia aku kau gua lu lo gw gue elu anda kalian nya kah lah pun ya yah kan sih dong deh kok loh toh aja saja gitu gini begitu begini tapi tetapi namun atau karena sebab jika kalau bila maka supaya agar meski meskipun walau walaupun ketika saat setelah sebelum selama antara terhadap tentang mengenai bagi oleh secara sebagai seperti daripada tanpa hingga sampai sejak menuju bahwa padahal sebenarnya sepertinya mungkin memang jadi lalu terus akhirnya misalnya contohnya banyak sedikit semua seluruh setiap tiap beberapa ada bukan jangan boleh mau ingin pengen nggak ngak gak ga kagak ngga ndak nanti kemarin besok hari ini sekarang waktu itu masih sedang belum pernah sering selalu kadang jarang cepat lambat awal akhir baru lama besar kecil tinggi rendah panjang pendek baik buruk benar salah sama beda penting biasanya selamat terima kasih makasih sangat sekali paling cuma cuman hanya lebih kurang sekitar hampir ternyata rupanya begitu gimana bagaimana kenapa mengapa siapa apa mana kapan darimana kemana bilang ngomong omong kata tadi dulu terus lagi tetap pasti seharusnya sebaiknya seakan seolah kayaknya keliatan kelihatan ketahuan disini disitu disana kesini kesana bener pake pakai kayak emang lagian mulu istilah istilahnya banget" +
// ── English ───────────────────────────────────────────────────────
" the a an and or but if then else for to in on at by with without from of is are was were be been being have has had do does did will would can could should may might must shall this that these those it its i you he she we they them their there here when where why how what which who whom whose only very just about above after before below under over into onto within upon against between among during through across along around behind beyond near off out up down now then so as not no yes ok okay" +
// ── Common net slang / acronyms the LLM already knows ──────────────
" lol omg wtf idk btw tbh imo aka fyi nsfw smh nvm asap afk brb gg wp ty np mb sry thx kk oke okk ygy frfr"
).split(/\s+/),
);
/**
* Words that are either already defined by the moderation rules, or are so
* common (brands, tech vocabulary, project names) that a Wikipedia lookup is
* a guaranteed miss/waste. Keeps the glossary focused on genuinely unknown
* terms.
*/
const KNOWN_SAFE_TERMS = new Set(
(
"discord youtube google facebook instagram twitter tiktok whatsapp telegram netflix spotify steam github gitlab bitbucket chatgpt openai anthropic claude deepseek gemini llama copilot cursor vscode vscodium jetbrains intellij pycharm webstorm sublime codeblocks" +
" docker kubernetes k8s linux ubuntu debian arch fedora manjaro kali windows macos android ios chrome firefox safari edge opera brave" +
" react nextjs next vue svelte angular node nodejs deno bun pnpm yarn npm javascript typescript python golang go rust java kotlin swift cplusplus cpp css html json xml yaml toml regex backend frontend database mysql postgres postgresql mongodb redis qdrant sqlite nosql graphql rest websocket webhook" +
" bug crash error debug fix issue pr merge commit push pull branch main master dev staging production server client app website web browser" +
" stream streaming video audio voice call camera screen share screenshare gameplay gaming game play steam epic xbox playstation nintendo switch console" +
" bot discordbot moderation moderator admin member user profile avatar channel server guild message chat dm reply forward embed sticker emoji role permission" +
" meme code coding ngoding programmer program developer engineer software hardware cpu gpu ram rom storage disk network internet wifi lan ip dns vpn proxy cloud aws azure gcp vercel netlify heroku railway render vps hosting domain ssl login logout register account password email username" +
" anime manga waifu husbando tsundere moe otaku wibu weeb otome isekai shonen seinen josei manga manhwa manhua doujin" +
" anjay wkwk wkwkwk gws gaskeun santuy njir baka woy woi hadeh astaga asu anjing bangsat ngehe asal alay lebay caper mabar" +
" asus bete imphnen impnhen ngab" +
" syahadat sholat shalat solat puasa zakat haji umrah doa tuhan nabi allah yesus muhammad hashem" +
" loli shota incest exhibition furry fursuit cosplay costume" +
" gaza palestine israel yahudi yahud israel palestina israeli" +
" hokkian mandarin arabic jawa sunda betawi minang bugis batak melayu inggris indonesia"
).split(/\s+/),
);
function isKnownTerm(word: string): boolean {
return STOPWORDS.has(word) || KNOWN_SAFE_TERMS.has(word);
}
/** True when a quoted phrase is mostly filler words (skip it). */
function isMostlyStopwords(phrase: string): boolean {
const words = phrase
.toLowerCase()
.split(/[^a-zà-öø-ÿ]+/i)
.filter(Boolean);
if (words.length === 0) return true;
const stopCount = words.filter((w) => STOPWORDS.has(w)).length;
return stopCount / words.length >= 0.6;
}
export interface ExtractGlossaryOptions {
maxTerms?: number;
minWordLength?: number;
}
/**
* Extracts candidate terms that the LLM might not know from message content.
* Returns at most `maxTerms` terms (default from config), scored by how
* "unknown-looking" they are (proper nouns, foreign spelling, quoted phrases).
*/
export function extractGlossaryTerms(
contents: string[],
options: ExtractGlossaryOptions = {},
): string[] {
const maxTerms = options.maxTerms ?? config.AI_GLOSSARY_MAX_TERMS;
const minWordLength =
options.minWordLength ?? config.AI_GLOSSARY_MIN_WORD_LENGTH;
const candidates = new Map<string, { word: string; score: number }>();
const push = (rawWord: string, score: number): void => {
const clean = rawWord
.trim()
.replace(/^[^\p{L}\p{N}]+|[^\p{L}\p{N}]+$/gu, "");
if (clean.length < minWordLength) return;
const key = clean.toLowerCase();
if (isKnownTerm(key) || isNoiseWord(clean)) return;
const existing = candidates.get(key);
if (existing) {
existing.score += score + 1;
} else {
candidates.set(key, { word: clean, score });
}
};
for (const content of contents) {
if (!content) continue;
const cleaned = cleanContent(content);
if (!cleaned) continue;
// Quoted phrases — explicit terms the user called out
for (const m of cleaned.matchAll(/"([^"]{2,80})"/g)) {
const phrase = m[1].trim();
const wordCount = phrase.split(/\s+/).length;
if (wordCount >= 2 && wordCount <= 6 && !isMostlyStopwords(phrase)) {
push(phrase, 10);
}
}
// Individual words
for (const m of cleaned.matchAll(WORD_RE)) {
const w = m[0];
if (w.length < minWordLength) continue;
if (isNoiseWord(w)) continue;
const key = w.toLowerCase();
if (isKnownTerm(key)) continue;
push(w, scoreWord(w));
}
}
return Array.from(candidates.values())
.sort((a, b) => b.score - a.score)
.slice(0, maxTerms)
.map((c) => c.word);
}
// ---------------------------------------------------------------------------
// Definition lookup (cached: LRU → Redis → SearXNG/Wikipedia)
// ---------------------------------------------------------------------------
export interface TermDefinition {
term: string;
definition: string;
sourceUrl: string;
}
/** Definition-like markers for accepting a non-Wikipedia search result. */
const DEF_MARKERS =
/adalah|merupakan|istilah (?:untuk|yang|yg)|artinya|sebutan|berarti|refers? to|known as|also called|short for|a term (?:for|used)|istilah dalam|kata (?:asing|serapan)? ?untuk/i;
/** True when the term appears in the result text (or a 4+ char word in the
* result is part of the term). Lenient "kafircel" matches a "Kafir"
* article via substring, while a Google-Translate homepage snippet does not. */
function hasTermOverlap(term: string, title: string, snippet: string): boolean {
const termLower = term.toLowerCase();
const text = `${title} ${snippet}`.toLowerCase();
if (text.includes(termLower)) return true;
const words = text.match(/[a-z0-9]{4,}/gi) ?? [];
return words.some((w) => termLower.includes(w));
}
/** Quality gate: is this result good enough to quote as a definition? */
function isUsableDefinition(
r: { title: string; url: string; snippet: string },
term: string,
isWiki: boolean,
): boolean {
const text = `${r.title} ${r.snippet}`;
// Wikipedia disambiguation pages are not definitions
if (/disambiguasi|disambiguation/i.test(text)) return false;
if ((r.snippet ?? "").trim().length < 25) return false;
if (!hasTermOverlap(term, r.title, r.snippet)) return false;
// Wikipedia articles are accepted with just the overlap+length gate;
// everything else must read like an actual definition, not an ad,
// a translate homepage, or a navigation blurb.
if (isWiki) return true;
return DEF_MARKERS.test(r.snippet);
}
/** Picks the best definition from search results, preferring a genuine
* Wikipedia article; otherwise the first result that reads like a
* definition. Returns null when nothing qualifies. */
function pickDefinition(
results: Array<{ title: string; url: string; snippet: string }>,
term: string,
): TermDefinition | null {
const wiki = results.find((r) => /wikipedia\.org/i.test(r.url));
const best = wiki && isUsableDefinition(wiki, term, true) ? wiki : null;
if (!best) {
for (const r of results) {
if (isUsableDefinition(r, term, false)) {
return buildDefinition(r, term);
}
}
return null;
}
return buildDefinition(best, term);
}
function buildDefinition(
best: { title: string; url: string; snippet: string },
term: string,
): TermDefinition {
const snippet = (best.snippet || best.title || "").trim();
const definition =
snippet.length > MAX_DEFINITION_CHARS
? `${snippet.slice(0, MAX_DEFINITION_CHARS - 1).trimEnd()}`
: snippet;
return { term, definition, sourceUrl: best.url };
}
/** Live (network) lookup — runs under the shared SearXNG rate-limit gate. */
async function fetchDefinitionLive(
term: string,
key: string,
cacheKey: string,
): Promise<TermDefinition | null> {
return liveSearchLimit(async () => {
await acquireLiveSlot();
try {
let results = await searchSearxng(
key,
"general",
undefined,
GLOSSARY_SEARCH_TIMEOUT_MS,
);
let def = pickDefinition(results, term);
// Zero results is usually the limiter kicking in, not a real miss —
// retry once. Results-but-unusable = genuine miss, no retry.
if (!def && results.length === 0) {
await delay(RETRY_DELAY_MS);
results = await searchSearxng(
key,
"general",
undefined,
GLOSSARY_SEARCH_TIMEOUT_MS,
);
def = pickDefinition(results, term);
}
if (def) {
// Persist permanently (definitions rarely change) — best-effort,
// then warm the fast caches.
void setTermDefinitionInDb(key, def.definition, def.sourceUrl);
searxngCacheSet(
cacheKey,
JSON.stringify({
definition: def.definition,
sourceUrl: def.sourceUrl,
}),
DEF_TTL_SECONDS,
);
termLru.set(key, def);
log.debug({ term: key }, "Term glossary resolved definition");
return def;
}
} catch (err) {
log.debug(
{ term: key, error: err instanceof Error ? err.message : String(err) },
"Term glossary lookup failed — skipping term",
);
}
// No definition — cache the miss with a SHORT TTL so a transient
// limiter/network failure is retried on a later batch.
searxngCacheSet(cacheKey, EMPTY_SENTINEL, MISS_TTL_SECONDS);
termLru.set(key, NOT_FOUND, { ttl: MISS_TTL_MS });
return null;
});
}
/** Resolve one term: LRU Redis Postgres (permanent) live SearXNG
* (rate-limited). The fast caches sit in front of the DB; the DB is the
* source of truth for successfully resolved definitions. */
async function resolveTerm(term: string): Promise<TermDefinition | null> {
const key = term.toLowerCase().trim();
// 1. In-memory LRU — same process, instant
const lruHit = termLru.get(key);
if (lruHit) return lruHit === NOT_FOUND ? null : lruHit;
// 2. Redis — shared across processes/workers. A miss sentinel here is NOT
// a definitive answer: it may predate a permanent DB entry written by
// another process, so we keep going and let the DB decide.
const cacheKey = makeSearxngCacheKey("def", key);
const cached = await searxngCacheGet(cacheKey);
let redisMiss = false;
if (cached !== null) {
if (cached === EMPTY_SENTINEL) {
redisMiss = true;
} else {
try {
const parsed = JSON.parse(cached) as {
definition?: string;
sourceUrl?: string;
};
if (parsed.definition) {
const def: TermDefinition = {
term,
definition: parsed.definition,
sourceUrl: parsed.sourceUrl ?? "",
};
termLru.set(key, def);
return def;
}
} catch {
// malformed cache entry — fall through to DB/live
}
}
}
// 3. Postgres — permanent store for resolved definitions. A hit re-warms
// the fast caches so the DB is not hit on every batch.
const dbDef = await getTermDefinitionFromDb(key);
if (dbDef) {
const def: TermDefinition = {
term,
definition: dbDef.definition,
sourceUrl: dbDef.sourceUrl,
};
termLru.set(key, def);
searxngCacheSet(
cacheKey,
JSON.stringify({ definition: def.definition, sourceUrl: def.sourceUrl }),
DEF_TTL_SECONDS,
);
log.debug({ term: key }, "Term glossary DB hit");
return def;
}
// 4. Redis already said "miss" recently and the DB has nothing — respect
// that instead of hammering SearXNG again within the miss window.
if (redisMiss) {
termLru.set(key, NOT_FOUND, { ttl: MISS_TTL_MS });
return null;
}
// 5. Live search (rate-limited + staggered)
return fetchDefinitionLive(term, key, cacheKey);
}
/**
* Looks up definitions for a batch of terms, in parallel. Returns a map of
* term definition for the terms that resolved. Errors/misses are skipped.
* Live SearXNG calls are throttled internally (concurrency 2 + stagger).
*/
export async function lookupTermDefinitions(
terms: string[],
): Promise<Map<string, TermDefinition>> {
const map = new Map<string, TermDefinition>();
if (terms.length === 0) return map;
const results = await Promise.allSettled(terms.map(resolveTerm));
for (let i = 0; i < terms.length; i++) {
const r = results[i];
if (r.status === "fulfilled" && r.value) {
map.set(r.value.term, r.value);
}
}
return map;
}
// ---------------------------------------------------------------------------
// Prompt formatting
// ---------------------------------------------------------------------------
/**
* Formats definitions as a `<term_glossary>` XML block for the LLM prompt:
*
* <term_glossary>
* <term word="ngab" source="https://…">definisi</term>
* </term_glossary>
*
* Returns "" when there are no definitions (the block is then omitted).
*/
export function formatTermGlossary(
defs: ReadonlyMap<string, TermDefinition>,
): string {
if (!defs || defs.size === 0) return "";
const lines = Array.from(defs.values()).map(
(d) =>
` <term word="${escapeXml(d.term)}" source="${escapeXml(d.sourceUrl)}">${escapeXml(d.definition)}</term>`,
);
return `<term_glossary>\n${lines.join("\n")}\n</term_glossary>`;
}
// ---------------------------------------------------------------------------
// Convenience: full pipeline
// ---------------------------------------------------------------------------
export interface GlossaryBlockOptions extends ExtractGlossaryOptions {
enabled?: boolean;
}
/**
* One-shot helper: extract terms from message contents, look up definitions,
* and return the formatted `<term_glossary>` block ("" when disabled or no
* definitions found). Safe to call on every batch cached lookups make it
* cheap.
*/
export async function buildTermGlossaryBlock(
contents: string[],
options: GlossaryBlockOptions = {},
): Promise<string> {
const enabled = options.enabled ?? config.AI_GLOSSARY_ENABLED;
if (!enabled) return "";
if (contents.length === 0) return "";
const terms = extractGlossaryTerms(contents, options);
if (terms.length === 0) return "";
const defs = await lookupTermDefinitions(terms);
if (defs.size === 0) return "";
const block = formatTermGlossary(defs);
log.debug(
{ terms: terms.length, definitions: defs.size },
"Term glossary block built",
);
return block;
}
@@ -0,0 +1,86 @@
/**
* termGlossaryStore.ts
*
* Permanent Postgres layer for the term glossary. Resolved definitions
* (which carry content) are persisted here because they rarely change
* Redis/LRU only act as fast read caches in front of this table. Terms with
* no definition (misses) are deliberately NOT persisted; they stay ephemeral
* in Redis with a short TTL so transient lookup failures get retried.
*
* All calls are best-effort: any DB error degrades to a cache miss (the
* glossary then falls through to Redis/live search as if the DB layer
* didn't exist).
*/
import { createChildLogger } from "@/shared/logger/index";
import { executeAll, executeGet } from "../../shared/database/drizzle.js";
const log = createChildLogger("term-glossary-store");
export interface StoredTermDefinition {
definition: string;
sourceUrl: string;
}
/**
* Read a permanently stored definition for a term (lowercase key).
* Returns null when missing or on any DB error (callers fall through).
* A successful read bumps hit_count for observability (fire-and-forget).
*/
export async function getTermDefinitionFromDb(
term: string,
): Promise<StoredTermDefinition | null> {
try {
const row = await executeGet(
`SELECT definition, source_url FROM term_glossary_cache WHERE term = $1`,
[term.toLowerCase().trim()],
);
if (!row) return null;
try {
await executeAll(
`UPDATE term_glossary_cache SET hit_count = hit_count + 1 WHERE term = $1`,
[term.toLowerCase().trim()],
);
} catch {
// hit_count is observability only — never fail a read for it
}
return {
definition: row.definition as string,
sourceUrl: (row.source_url as string | null) ?? "",
};
} catch (error) {
log.debug(
{ error: error instanceof Error ? error.message : String(error) },
"getTermDefinitionFromDb failed — falling back to live search",
);
return null;
}
}
/**
* Persist a resolved definition permanently (UPSERT by term).
* Only called for successful resolutions never for misses.
* Best-effort: a DB write failure does not affect the returned definition.
*/
export async function setTermDefinitionInDb(
term: string,
definition: string,
sourceUrl: string,
): Promise<void> {
try {
await executeAll(
`INSERT INTO term_glossary_cache (term, definition, source_url, resolved_at, hit_count)
VALUES ($1, $2, $3, $4, 0)
ON CONFLICT (term) DO UPDATE SET
definition = EXCLUDED.definition,
source_url = EXCLUDED.source_url,
resolved_at = EXCLUDED.resolved_at`,
[term.toLowerCase().trim(), definition, sourceUrl, Date.now()],
);
} catch (error) {
log.warn(
{ error: error instanceof Error ? error.message : String(error) },
"setTermDefinitionInDb failed — definition stays memory/Redis only",
);
}
}
@@ -6,7 +6,9 @@
* the LLM for analysis. Extracted from moderationOrchestrator.ts.
*/
import { createChildLogger } from "@/shared/logger/index";
import { delay } from "@/shared/utils/index";
import { config } from "../../shared/config/config.js";
import { resizeImageForVision } from "../attachment-upload/imageResizer.js";
import type {
AnalysisResult,
MessageRecord,
@@ -14,25 +16,36 @@ import type {
import { getChannelCulture } from "./channelCultureStore.js";
import type { ModerationPromptContent, RetryState } from "./llmCaller.js";
import { callModerationLLM } from "./llmCaller.js";
import { analyzeSingleMediaImage } from "./mediaAnalysisClient.js";
import {
buildReferenceXml,
buildUserHistoryXml,
buildUserProfileRef,
buildUserProfilesBlock,
escapeXml,
formatReputationAttrs,
getAnalysisContent,
resolveDisplayName,
resolveIsBot,
resolveIsEdited,
truncateForAi,
} from "./moderationBuilders.js";
import {
buildSystemPrompt as buildSystemPromptModular,
sanitizeAiContent,
} from "./moderationPrompt.js";
import { buildSystemPrompt as buildSystemPromptModular } from "./moderationPrompt.js";
import { logModerationAnalysis } from "./responseLogger.js";
import {
extractSearchQueries,
formatSearchResults,
searchSearxng,
} from "./searxngSearch.js";
import { buildTermGlossaryBlock } from "./termGlossary.js";
import { getRecentCorrectedModerations } from "./textCacheStore.js";
import { extractUrlsFromText, fetchUrlSafely } from "./urlFetcher.js";
import { getUserProfile } from "./userProfileStore.js";
import { initializeUserReputation } from "./userReputationStore.js";
import {
getUserRecentInfractions,
initializeUserReputation,
} from "./userReputationStore.js";
import type { MessageImagePart } from "./visionAnalyzer.js";
const log = createChildLogger("textBatchProcessor");
@@ -69,7 +82,7 @@ export async function buildCorrectedFewShotExamples(): Promise<string> {
// ---------------------------------------------------------------------------
export async function runTextOnlyBatch(
targets: MessageRecord[],
contextText: string,
contextBlock: string,
): Promise<{ results: AnalysisResult[]; raw: unknown }> {
if (!targets.length) return { results: [], raw: null };
@@ -84,22 +97,33 @@ export async function runTextOnlyBatch(
allUrls.add(url);
}
const urlArr = Array.from(allUrls).slice(0, 10);
if (urlArr.length === 0) return new Map<string, string>();
if (urlArr.length === 0) {
return {
text: new Map<string, string>(),
image: new Map<string, { data: Buffer; mimeType: string }>(),
title: new Map<string, string>(),
};
}
const results = await Promise.allSettled(
urlArr.map((url) => fetchUrlSafely(url)),
);
const map = new Map<string, string>();
const textMap = new Map<string, string>();
const imageMap = new Map<string, { data: Buffer; mimeType: string }>();
const titleMap = new Map<string, string>();
for (let i = 0; i < urlArr.length; i++) {
const r = results[i];
if (
r.status === "fulfilled" &&
r.value.type === "text" &&
r.value.textContent
) {
map.set(urlArr[i], r.value.textContent);
if (r.status !== "fulfilled") continue;
const v = r.value;
if (v.type === "text" && v.textContent) {
textMap.set(urlArr[i], v.textContent);
if (v.title) titleMap.set(urlArr[i], v.title);
} else if (v.type === "image" && v.data && v.mimeType) {
// Direct image link (or og:image followed from an HTML page) —
// kept for vision analysis below.
imageMap.set(urlArr[i], { data: v.data, mimeType: v.mimeType });
}
}
return map;
return { text: textMap, image: imageMap, title: titleMap };
})();
const searxngPromise = (async () => {
@@ -122,10 +146,19 @@ export async function runTextOnlyBatch(
return map;
})();
const [urlFetchMap, searxngResults] = await Promise.all([
// Term glossary — per-word Wikipedia lookups for words the LLM may not
// know (slang, jargon, regional language). Cached in Redis + in-memory, so
// repeat terms resolve instantly and only genuinely new words hit SearXNG.
const glossaryPromise = buildTermGlossaryBlock(
targets.map((msg) => getAnalysisContent(msg)),
).catch(() => "");
const [urlFetchMaps, searxngResults, glossaryBlock] = await Promise.all([
urlFetchPromise,
searxngPromise,
glossaryPromise,
]);
const urlFetchMap = urlFetchMaps.text;
// Deduplicate identical short messages
const shortContentGroups = new Map<string, MessageRecord[]>();
@@ -171,25 +204,109 @@ export async function runTextOnlyBatch(
const batch = subBatches[i];
const targetIds = batch.map((t) => t.id);
// User reputation + profiles
// User reputation + profiles (raw summary text — deduplicated into a
// single <user_profiles> map per batch; messages only reference it).
const userContexts = new Map<string, string>();
const userProfiles = new Map<string, string>();
const userProfiles = new Map<
string,
{
text: string;
asOf?: number | null;
}
>();
for (const msg of batch) {
if (!userContexts.has(msg.user_id)) {
const rep = await initializeUserReputation(msg.user_id, msg.guild_id);
userContexts.set(
msg.user_id,
`<user_reputation trust_score="${rep.trust_score}" />`,
);
const repAttrs = formatReputationAttrs(rep);
let repXml = `<user_reputation ${repAttrs}/>`;
// Repeat offenders get their last flagged messages as <user_history>
// so the LLM can recognize PATTERNS (same scam link, repeated
// provocation) — history is reference, never proof. Best-effort.
if (rep.total_infractions > 0) {
try {
const history = await getUserRecentInfractions(msg.user_id, 2);
const historyXml = buildUserHistoryXml(
history.map((h) => ({
content: h.content ?? "",
severity: h.severity,
created_at: h.created_at,
})),
);
if (historyXml) {
repXml = `<user_reputation ${repAttrs}>\n${historyXml}\n</user_reputation>`;
}
} catch {
// history is a bonus — fall back to attrs-only reputation
}
}
userContexts.set(msg.user_id, repXml);
}
if (!userProfiles.has(msg.user_id)) {
const profile = await getUserProfile(msg.user_id);
userProfiles.set(
msg.user_id,
profile
? `<user_profile>${sanitizeAiContent(profile.profile_summary)}</user_profile>`
: "",
);
userProfiles.set(msg.user_id, {
text: profile?.profile_summary ?? "",
asOf: profile?.last_analyzed_at ?? null,
});
}
}
const userProfilesBlock = buildUserProfilesBlock(userProfiles);
// ── URL images → multimodal vision evidence ─────────────────────────
// The text batch fetches inline URLs; whenever one resolved to an image
// (direct image link, or og:image followed from an HTML page), run the
// vision model and append its description as media evidence. If any
// message in the sub-batch produced image evidence, the prompt switches
// to "mixed" mode so media-analysis instructions/examples are injected
// — a link to media is analyzed as media, not as bare text.
const batchImageEvidence = new Map<string, string[]>();
let batchHasImageEvidence = false;
const urlImages = urlFetchMaps.image;
const urlTitles = urlFetchMaps.title;
if (urlImages.size > 0) {
const maxDim = config.AI_LLM_IMAGE_MAX_DIMENSION ?? 1024;
const evidenceSets = await Promise.all(
batch.map(async (msg) => {
const content = getAnalysisContent(msg);
const pics = extractUrlsFromText(content)
.slice(0, 3)
.filter((url) => urlImages.has(url));
if (pics.length === 0) return { id: msg.id, lines: [] as string[] };
const lines = await Promise.all(
pics.map(async (url) => {
const img = urlImages.get(url)!;
try {
const { data: resizedBuffer, mimeType: resizedMime } =
await resizeImageForVision(img.data, maxDim);
const part: MessageImagePart = {
type: "image_url",
image_url: {
url: `data:${resizedMime};base64,${resizedBuffer.toString("base64")}`,
},
sourceLabel: `[gambar dari URL ${url} (inline), pesan id=${msg.id}]`,
};
// Bound vision time so a dead vision model can't stall the
// whole text batch — a timeout just skips the evidence.
const timedOut = delay(15000).then(() => null as string | null);
return await Promise.race([
analyzeSingleMediaImage(msg.id, part),
timedOut,
]);
} catch {
return null;
}
}),
);
return {
id: msg.id,
lines: lines.filter((l): l is string => Boolean(l)),
};
}),
);
for (const set of evidenceSets) {
if (set.lines.length > 0) {
batchImageEvidence.set(set.id, set.lines);
batchHasImageEvidence = true;
}
}
}
@@ -204,8 +321,7 @@ export async function runTextOnlyBatch(
: undefined;
const correctedExamples = await buildCorrectedFewShotExamples();
const systemText = buildSystemPromptModular({
contextText,
mode: "text",
mode: batchHasImageEvidence ? "mixed" : "text",
correction,
correctedExamples,
channelCulture,
@@ -214,38 +330,59 @@ export async function runTextOnlyBatch(
const messagesBlock = (
await Promise.all(
batch.map(async (msg) => {
const content = getAnalysisContent(msg);
const content = truncateForAi(getAnalysisContent(msg));
const msgUrls = extractUrlsFromText(content);
const urlContexts = msgUrls
.map((url) => {
const ft = urlFetchMap.get(url);
return ft
? `<web_content url="${escapeXml(url)}">${escapeXml(ft)}</web_content>`
: null;
if (!ft) return null;
const title = urlTitles.get(url);
const titleAttr = title ? ` title="${escapeXml(title)}"` : "";
return `<web_content url="${escapeXml(url)}"${titleAttr}>${escapeXml(ft)}</web_content>`;
})
.filter(Boolean)
.join("\n");
const webContext = urlContexts ? `\n${urlContexts}` : "";
const mediaEvidenceCtx = (batchImageEvidence.get(msg.id) ?? [])
.map((line) => `\n${line}`)
.join("");
const userCtx = userContexts.get(msg.user_id) ?? "";
const userProfileCtx = userProfiles.get(msg.user_id) ?? "";
const userProfileRef = (
userProfiles.get(msg.user_id)?.text ?? ""
).trim()
? buildUserProfileRef(msg.user_id)
: "";
const refXml = await buildReferenceXml(msg);
return `<message id="${msg.id}" user="${msg.username}">\n ${userCtx}${userProfileCtx ? `\n ${userProfileCtx}` : ""}${refXml ? `\n ${refXml}` : ""}\n <content>${escapeXml(content)}</content>${webContext}\n</message>`;
const repetitionCount = groupMapping.get(msg.id)?.length ?? 1;
const isBot = resolveIsBot(msg);
const isEdited = resolveIsEdited(msg);
return `<message id="${escapeXml(msg.id)}" user="${escapeXml(resolveDisplayName(msg))}" time="${new Date(msg.created_at).toISOString()}"${repetitionCount > 1 ? ` repetitions="${repetitionCount}"` : ""}${isBot ? ` bot="true"` : ""}${isEdited ? ` edited="true"` : ""}>\n ${userCtx}${userProfileRef ? `\n ${userProfileRef}` : ""}${refXml ? `\n ${refXml}` : ""}\n <content>${escapeXml(content)}</content>${webContext}${mediaEvidenceCtx}\n</message>`;
}),
)
).join("\n");
const searxngBlock =
searxngResults.size > 0
? `\n\n<web_searches>\n${Array.from(searxngResults.entries())
? `<web_searches>\n${Array.from(searxngResults.entries())
.map(
([q, xml]) =>
` <search_query query="${escapeXml(q)}">\n${xml} </search_query>`,
)
.join("\n")}\n</web_searches>`
: "";
// Data/instruction separation: the system prompt is stable per mode —
// all per-batch context (profiles, conversation, web evidence) lives in
// the USER payload, ordered oldest-first so targets come last.
const userBlocks = [
userProfilesBlock?.trimEnd() ?? "",
contextBlock?.trimEnd() ?? "",
searxngBlock,
glossaryBlock,
`<messages_to_analyze>\n${messagesBlock}\n</messages_to_analyze>`,
].filter((b) => b.trim().length > 0);
return {
system: systemText,
user: `${searxngBlock}\n\n<messages_to_analyze>\n${messagesBlock}\n</messages_to_analyze>`,
user: userBlocks.join("\n\n"),
};
};
@@ -63,12 +63,15 @@ export function makeCustomEmojiCacheKey(emojiId: string): string {
/**
* Generate a deterministic cache key for an image data URL.
* Hashes the first 128 chars of the data URL (enough to identify the image
* without storing the full base64 string as the key).
* Hashes the FULL data URL only hashing a prefix (e.g. first 128 chars)
* causes hash collisions for images that share the same MIME prefix +
* identical base64 header bytes (common when images are resized to the same
* dimensions), which makes every image incorrectly reuse the same cached
* vision analysis. Hashing the entire data URL guarantees uniqueness per
* actual pixel content.
*/
export function makeImageCacheKey(dataUrl: string): string {
const prefix = dataUrl.slice(0, 128);
const hash = createHash("sha256").update(prefix).digest("hex").slice(0, 16);
const hash = createHash("sha256").update(dataUrl).digest("hex").slice(0, 16);
return `image:${hash}`;
}
@@ -11,6 +11,8 @@ export interface FetchedUrlContext {
data?: Buffer;
mimeType?: string;
textContent?: string;
/** Page title from og:title / <title> — strong signal for the LLM. */
title?: string;
error?: string;
}
@@ -86,6 +88,50 @@ function extractOgImage(html: string): string | null {
return null;
}
export interface OgMeta {
title: string | null;
description: string | null;
siteName: string | null;
}
/**
* Extracts OpenGraph / twitter meta + <title> from raw HTML. Both attribute
* orders are accepted (<meta property=... content=...> and reversed).
*/
export function extractOgMeta(html: string): OgMeta {
const metaValue = (name: string): string | null => {
const re = new RegExp(
`<meta[^>]*(?:property|name)=["']${name}["'][^>]*content=["']([^"']+)["']`,
"i",
);
const m = html.match(re);
if (m?.[1]) return m[1].replace(/&amp;/g, "&").replace(/&quot;/g, '"');
const reRev = new RegExp(
`<meta[^>]*content=["']([^"']+)["'][^>]*(?:property|name)=["']${name}["']`,
"i",
);
const mRev = html.match(reRev);
return mRev?.[1]
? mRev[1].replace(/&amp;/g, "&").replace(/&quot;/g, '"')
: null;
};
const title =
metaValue("og:title") ||
metaValue("twitter:title") ||
html.match(/<title[^>]*>([^<]+)<\/title>/i)?.[1]?.trim() ||
null;
const description =
metaValue("og:description") ||
metaValue("twitter:description") ||
metaValue("description") ||
null;
const siteName =
metaValue("og:site_name") || metaValue("application-name") || null;
return { title, description, siteName };
}
function truncateAndCleanHtml(html: string, maxLen = 1000): string {
// Strip <script> and <style> entirely
let text = html.replace(
@@ -176,6 +222,7 @@ export async function fetchUrlSafely(
url,
type: "text",
textContent: cleaned,
title: extractOgMeta(text).title ?? undefined,
};
}
@@ -29,6 +29,35 @@ import {
upsertCachedMediaByPhash,
visionLruCache,
} from "./mediaCache.js";
/**
* Detect vision outputs where the model claims it saw no image at all
* ("Maaf, saya tidak melihat gambar apapun...", "Tidak ada gambar yang
* terlampir...", "I cannot see any image..."). Such text is NOT a valid
* analysis caching it poisons the image cache for 24h (image/phash keys),
* so every re-analysis of the same image returns the "no image" text and the
* moderation LLM writes "lampiran gagal terbaca". These outputs must be
* treated as failures: never cached, and ignored when read back from cache.
*/
export function isNoImageSeenText(text: string | null | undefined): boolean {
if (!text) return false;
const lower = text.toLowerCase();
return (
/tidak (?:melihat|ada|terlihat) (?:gambar|foto|image)/i.test(lower) ||
/tidak (?:ada )?(?:gambar|foto|image) (?:apapun|yang terlampir)/i.test(
lower,
) ||
/gambar apapun/i.test(lower) ||
/tanpa (?:input )?(?:visual|gambar|image)/i.test(lower) ||
/\bno image (?:provided|attached|detected|found|was provided)?/i.test(
lower,
) ||
/(?:cannot|can't) see (?:any |an |the )?image/i.test(lower) ||
/i (?:do not|don't) (?:see|detect) (?:any |an |the )?image/i.test(lower) ||
/there (?:is|are) no image/i.test(lower)
);
}
import {
buildMediaCandidates,
downloadAndExtractFrame,
@@ -37,24 +66,34 @@ import {
} from "./mediaDownloader.js";
import {
buildReferenceXml,
buildUserHistoryXml,
buildUserProfileRef,
escapeXml,
formatReputationAttrs,
getAnalysisContent,
resolveDisplayName,
resolveIsBot,
resolveIsEdited,
truncateForAi,
} from "./moderationBuilders.js";
import {
buildCustomEmojiVisionPrompt,
buildGeneralImageVisionPrompt,
buildStickerTextOnlyWarning,
buildStickerVisionPrompt,
sanitizeAiContent,
} from "./moderationPrompt.js";
import {
extractSearchQueries,
formatSearchResults,
searchSearxng,
} from "./searxngSearch.js";
import { buildTermGlossaryBlock } from "./termGlossary.js";
import { extractUrlsFromText } from "./urlFetcher.js";
import { getUserProfile } from "./userProfileStore.js";
import { initializeUserReputation } from "./userReputationStore.js";
import {
getUserRecentInfractions,
initializeUserReputation,
} from "./userReputationStore.js";
// ---------------------------------------------------------------------------
// Types
@@ -110,18 +149,32 @@ export const analyzeSingleMediaImage = async (
// Layer 0: LRU
const lruCached = visionLruCache.get(cacheKey);
if (lruCached) {
if (lruCached && !isNoImageSeenText(lruCached)) {
log.debug({ cacheKey }, "Vision LRU cache HIT (in-memory)");
return `[Media analysis for message ${messageId}] ${image.sourceLabel}: ${lruCached}`;
}
if (lruCached) {
// Poisoned entry ("I see no image") — drop it and re-analyze.
log.warn({ cacheKey }, "Vision LRU cache HIT was no-image-seen — dropping");
visionLruCache.delete(cacheKey);
}
// Layer 1: DB
const cached = await getCachedMediaAnalysis(cacheKey);
if (cached) {
if (cached && !isNoImageSeenText(cached)) {
visionLruCache.set(cacheKey, cached);
log.debug({ cacheKey }, "Media analysis cache HIT (DB → LRU)");
return `[Media analysis for message ${messageId}] ${image.sourceLabel}: ${cached}`;
}
if (cached) {
// Poisoned DB entry — purge it so later messages re-analyze.
log.warn(
{ cacheKey },
"Media analysis cache HIT was no-image-seen — purging",
);
await deleteCachedMediaAnalysis(cacheKey).catch(() => {});
visionLruCache.delete(cacheKey);
}
// In-flight dedupe
const existing = inFlightVisionCalls.get(cacheKey);
@@ -164,7 +217,7 @@ export const analyzeSingleMediaImage = async (
phash = await computeImagePhash(imgBuffer);
if (phash) {
const phashCached = await getCachedMediaByPhash(phash);
if (phashCached) {
if (phashCached && !isNoImageSeenText(phashCached)) {
visionLruCache.set(cacheKey, phashCached);
await upsertCachedMediaAnalysis(
cacheKey,
@@ -174,6 +227,12 @@ export const analyzeSingleMediaImage = async (
).catch(() => {});
return phashCached;
}
if (phashCached) {
log.warn(
{ phash, cacheKey },
"phash cache HIT was no-image-seen — ignoring",
);
}
}
}
} catch {
@@ -186,7 +245,7 @@ export const analyzeSingleMediaImage = async (
for (let attempt = 0; attempt < 3; attempt++) {
try {
const content = await llmVision(promptText, image.image_url);
if (content) {
if (content && !isNoImageSeenText(content)) {
await upsertCachedMediaAnalysis(
cacheKey,
content,
@@ -204,7 +263,17 @@ export const analyzeSingleMediaImage = async (
}
return content;
}
log.warn({ messageId }, "Vision API null response");
if (content) {
// Model claims it saw no image — same as a null response: NOT a
// valid analysis, and caching it would poison the key for every
// re-analysis of the same image (phash TTL is 7 days).
log.warn(
{ messageId, cacheKey },
"Vision returned no-image-seen text — not caching",
);
} else {
log.warn({ messageId }, "Vision API null response");
}
break;
} catch (err) {
lastError = err instanceof Error ? err : new Error(String(err));
@@ -231,6 +300,7 @@ export const analyzeSingleMediaImage = async (
"Vision failed after 3 attempts",
);
await deleteCachedMediaAnalysis(cacheKey).catch(() => {});
visionLruCache.delete(cacheKey);
return FAILED_ANALYSIS_PREFIX;
})();
@@ -344,6 +414,12 @@ export async function prepareMediaMessage(
searxngXml = `\n<web_searches>\n${parts.join("\n")}\n</web_searches>`;
}
// Term glossary — cached per-word Wikipedia definitions for words the LLM
// may not know. Bounded and cached (in-memory + Redis), so this adds no
// meaningful latency to the media path either.
const glossaryXml = await buildTermGlossaryBlock([content]).catch(() => "");
const glossaryCtx = glossaryXml ? `\n${glossaryXml}` : "";
// Build XML block
const webTexts = webTextMap.get(targetId) ?? [];
const mediaAnalyses = mediaAnalysisMap.get(targetId) ?? [];
@@ -366,7 +442,37 @@ export async function prepareMediaMessage(
const rep = await initializeUserReputation(target.user_id, target.guild_id);
const profile = await getUserProfile(target.user_id);
const refXml = await buildReferenceXml(target);
// Profile is emitted ONCE per batch in a <user_profiles> map (see
// mediaBatchProcessor); here we only reference it to avoid repeating the
// full summary on every message of the same user.
const profileRef = profile?.profile_summary?.trim()
? buildUserProfileRef(target.user_id)
: "";
const messageBlock = `<message id="${escapeXml(target.id)}" user="${escapeXml(target.username)}">\n <user_reputation trust_score="${rep.trust_score}" />${profile ? `\n <user_profile>${sanitizeAiContent(profile.profile_summary)}</user_profile>` : ""}${refXml ? `\n ${refXml}` : ""}\n <content>${escapeXml(content)}</content>${mediaContext ? ` ${escapeXml(mediaContext)}` : ""}${webContext}${mediaAnalysisContext}${searxngXml}\n</message>`;
// Rich reputation — same shape as the text path: attrs + optional
// <user_history> with the last flagged messages for repeat offenders.
const repAttrs = formatReputationAttrs(rep);
let repXml = `<user_reputation ${repAttrs}/>`;
if (rep.total_infractions > 0) {
try {
const history = await getUserRecentInfractions(target.user_id, 2);
const historyXml = buildUserHistoryXml(
history.map((h) => ({
content: h.content ?? "",
severity: h.severity,
created_at: h.created_at,
})),
);
if (historyXml) {
repXml = `<user_reputation ${repAttrs}>\n${historyXml}\n</user_reputation>`;
}
} catch {
// history is a bonus — fall back to attrs-only reputation
}
}
const isBot = resolveIsBot(target);
const isEdited = resolveIsEdited(target);
const messageBlock = `<message id="${escapeXml(target.id)}" user="${escapeXml(resolveDisplayName(target))}" time="${new Date(target.created_at).toISOString()}"${isBot ? ` bot="true"` : ""}${isEdited ? ` edited="true"` : ""}>\n ${repXml}${profileRef ? `\n ${profileRef}` : ""}${refXml ? `\n ${refXml}` : ""}\n <content>${escapeXml(truncateForAi(content))}</content>${mediaContext ? ` ${escapeXml(mediaContext)}` : ""}${webContext}${mediaAnalysisContext}${searxngXml}${glossaryCtx}\n</message>`;
return { targetId, messageBlock };
}
@@ -9,6 +9,10 @@ export interface MessageLocation {
threadId: string | null;
threadName: string | null;
channelName: string | null;
/** Channel topic (resmi/deskripsi channel) strong context for judging
* whether a message fits the channel's purpose. Guarded: some channel
* types (threads on older API builds) expose no topic. */
topic?: string | null;
nsfw?: boolean;
nsfwLevel?: string | null;
ageRestricted?: boolean;
@@ -107,12 +111,17 @@ export function getMessageLocation(message: Message): MessageLocation {
nsfw?: boolean;
nsfwLevel?: string | null;
};
const topic =
"topic" in channel && typeof channel.topic === "string"
? channel.topic
: null;
if (!channel.isThread?.()) {
return {
channelId: message.channelId,
threadId: null,
threadName: null,
channelName: "name" in channel ? channel.name : null,
topic,
nsfw:
typeof safetyChannel.nsfw === "boolean"
? safetyChannel.nsfw
@@ -133,6 +142,7 @@ export function getMessageLocation(message: Message): MessageLocation {
threadId: channel.id,
threadName: channel.name,
channelName: channel.parent?.name ?? null,
topic,
nsfw:
typeof safetyChannel.nsfw === "boolean" ? safetyChannel.nsfw : undefined,
nsfwLevel:
@@ -1,4 +1,13 @@
import { type ChildProcess, spawn } from "node:child_process";
import {
chmodSync,
existsSync,
mkdtempSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { PassThrough, type Readable } from "node:stream";
import { StreamType } from "@discordjs/voice";
import { createChildLogger } from "@/shared/logger/index";
@@ -79,6 +88,22 @@ export function transcodeToHighQualityOgg(
);
input.pipe(proc.stdin);
// ffmpeg teardown closes stdin while the upstream source may still write —
// swallow EPIPE / destroyed-stream errors so they don't crash the gateway.
proc.stdin.on("error", (err: NodeJS.ErrnoException) => {
if (
err.code === "EPIPE" ||
err.code === "ERR_STREAM_DESTROYED" ||
err.code === "ERR_STREAM_WRITE_AFTER_END"
) {
logger.debug(
{ code: err.code },
"Transcode stdin closed during teardown",
);
} else {
logger.error({ error: err.message }, "Transcode stdin error");
}
});
activeProcesses.add(proc);
const cleanup = () => {
@@ -198,6 +223,98 @@ function buildNotInstalledError(): Error {
);
}
/**
* Build the yt-dlp --cookies args. YouTube blocks anonymous embeds with a
* "Sign in to confirm you're not a bot" 403 unless yt-dlp is given a logged-
* in account's cookies. The path is configurable via GMW_YT_COOKIES_PATH
* (default: the BWS-provided file the deploy writes to /etc/.../ytcookies.txt).
* If the file doesn't exist we pass nothing and fall back to anon (YouTube
* may 403 screen share will fail gracefully, not crash).
*/
var _cachedCookiePath: string | null = null;
function buildCookieArgs(): string[] {
// Single source of truth: BWS injects the account cookies via env
// (gmw_yt_downloader_cookies → GMW_YT_DOWNLOADER_COOKIES by bws-exec).
// We materialize them to a temp Netscape file because yt-dlp --cookies
// only accepts a file path, not stdin, and multiline env values are not
// reliable to pass directly on the spawn argv. Falls back to the on-disk
// file at GMW_YT_COOKIES_PATH (or /etc/gmw-discord-gateway/ytcookies.txt)
// which the Nix deploy writes from BWS once at start.
if (_cachedCookiePath) return ["--cookies", _cachedCookiePath];
const envCookies = process.env.GMW_YT_DOWNLOADER_COOKIES?.trim();
if (envCookies && envCookies.includes("LOGIN_INFO")) {
const fdPath = join(tmpdir(), `gmw-ytcookies.${process.pid}.txt`);
writeFileSync(fdPath, envCookies);
try {
chmodSync(fdPath, 0o600);
} catch {
/* best-effort */
}
_cachedCookiePath = fdPath;
logger.info(
{ cookiePath: fdPath, source: "GMW_YT_DOWNLOADER_COOKIES env" },
"Using YouTube cookies (from BWS env)",
);
return ["--cookies", fdPath];
}
const cookiePath =
process.env.GMW_YT_COOKIES_PATH ?? "/etc/gmw-discord-gateway/ytcookies.txt";
try {
if (cookiePath && existsSync(cookiePath)) {
_cachedCookiePath = cookiePath;
logger.info(
{ cookiePath, source: "on-disk file" },
"Using YouTube cookies for yt-dlp",
);
return ["--cookies", cookiePath];
}
} catch {
/* ignore — fallback to anon */
}
logger.warn(
"No YouTube cookies available; yt-dlp will use anonymous (YouTube may 403)",
);
return [];
}
/** Invidious instances for anon YouTube fetch (fallback when cookies 403). */
export const INVIDIOUS_INSTANCES = [
"yewtu.be",
"yewtu.nanomorph.dev",
"invidious.snopyta.org",
"invidious.kavin.rocks",
];
/** True if url is a YouTube watch URL (youtu.be / youtube.com/watch). */
export function isYoutubeWatchUrl(url: string): boolean {
try {
const u = new URL(url);
return (
u.hostname === "youtu.be" ||
(u.hostname === "www.youtube.com" && u.pathname === "/watch") ||
(u.hostname === "youtube.com" && u.pathname === "/watch")
);
} catch {
return false;
}
}
/** Rewrite a YouTube watch URL to an invidious instance (anon, no bot-check). */
export function toInvidiousUrl(url: string, instance: string): string {
try {
const u = new URL(url);
if (u.hostname === "youtu.be") {
const id = u.pathname.slice(1);
return `https://${instance}/watch?v=${id}`;
}
const id = u.searchParams.get("v");
if (id) return `https://${instance}/watch?v=${id}`;
return url;
} catch {
return url;
}
}
// ---------------------------------------------------------------------------
// Public API
// ---------------------------------------------------------------------------
@@ -222,6 +339,7 @@ export function resolveMediaUrl(
): Promise<MediaSourceResolution> {
return new Promise<MediaSourceResolution>((resolve, reject) => {
const format = options?.quality ?? "bestaudio";
const cookieArgs = buildCookieArgs();
const args = [
"-f",
format,
@@ -229,6 +347,7 @@ export function resolveMediaUrl(
"-",
"--no-progress",
"--no-warnings",
...cookieArgs,
"--print",
"before_dl:title",
"--print",
@@ -248,6 +367,20 @@ export function resolveMediaUrl(
// `--print` headers to stderr — pipe stdout immediately so the child
// never blocks on a full pipe while we wait for the headers on stderr.
const mediaStream = new PassThrough();
// Teardown (player stop / ffmpeg exit) destroys this stream while
// yt-dlp may still push bytes — without a listener an EPIPE /
// ERR_STREAM_DESTROYED surfaces as an uncaughtException.
mediaStream.on("error", (err: NodeJS.ErrnoException) => {
if (
err.code === "EPIPE" ||
err.code === "ERR_STREAM_DESTROYED" ||
err.code === "ERR_STREAM_WRITE_AFTER_END"
) {
logger.debug({ code: err.code }, "Media stream closed during teardown");
} else {
logger.error({ error: err.message }, "Media stream error");
}
});
proc.stdout.pipe(mediaStream);
let stderrBuf = "";
@@ -348,241 +481,99 @@ export function resolveMediaUrl(
* Resolve a media URL to a single playable input stream for screen share /
* GoLive streaming.
*
* yt-dlp `--get-url` with `bestvideo+bestaudio` prints the video-only and
* audio-only URLs on SEPARATE lines. The old code took only the first line
* (video-only) ffmpeg had no audio track GoLive stream had no sound.
* Streams the merged video+audio media directly from yt-dlp stdout (`-o -`).
*
* This returns a single input that `prepareStream` (which accepts only ONE
* ffmpeg input) can consume while STILL including audio:
* - If yt-dlp offers a merged progressive URL (one URL, video+audio) it is
* returned directly.
* - Otherwise the video-only + audio-only DASH URLs are fetched in the SAME
* yt-dlp run (signature URLs expire quickly) and merged locally by an
* ffmpeg process into a single NUT stream, which is streamed to the
* consumer over a Readable. NUT over stdin auto-probes cleanly (verified:
* av1+opus merge H264+opus transcode).
* This is deliberately NOT the old --dump-single-json + manual URL-fetch
* approach: YouTube signs DASH URLs for the extracting client and rejects
* them with 403 when fetched raw by ffmpeg/curl (verified 2026-08-12: even
* curl with the EXACT http_headers from the yt-dlp dump got 403 on some
* videos, while yt-dlp's own downloader succeeded). Streaming from yt-dlp
* lets it handle auth, cookies and transient retries internally the same
* mechanism resolveMediaUrl already uses for music playback.
*
* @returns a direct video URL (string) or a Readable of the merged NUT stream.
* @returns a Readable of the merged media stream.
*/
export function getDirectScreenInput(url: string): Promise<string | Readable> {
return new Promise<string | Readable>((resolve, reject) => {
export function getDirectScreenInput(url: string): Promise<Readable> {
return new Promise<Readable>((resolve) => {
// Merge fragments must NOT be written to the process CWD — the Nix
// store dir is read-only for the deployed gateway (EACCES). Use a
// per-run temp dir (world-writable like /tmp) so parallel/retry runs
// never collide on merge fragments and any user can write to it.
const tmpDir = mkdtempSync(join(tmpdir(), "gmw-ytdlp-"));
chmodSync(tmpDir, 0o1777);
const cookieArgs = buildCookieArgs();
const args = [
url,
"--dump-single-json",
"--format",
"-f",
"bestvideo[protocol^=http]+bestaudio[protocol^=http]/best[protocol^=http]/best",
"-o",
"-",
"--no-playlist",
"--no-warnings",
"--quiet",
// NOTE: deliberately NOT --no-simulate. Simulate mode still resolves the
// requested format URLs into the JSON (requested_formats[].url), and it
// avoids yt-dlp writing .part files into the process CWD — which is the
// read-only Nix store dir for the deployed gateway (EACCES).
"--no-progress",
...cookieArgs,
"-P",
tmpDir,
url,
];
logger.info({ url }, "Spawning yt-dlp for screen share input resolution");
logger.info({ url }, "Spawning yt-dlp for screen share input streaming");
const proc = spawn("yt-dlp", args, {
stdio: ["pipe", "pipe", "pipe"],
stdio: ["ignore", "pipe", "pipe"],
});
activeProcesses.add(proc);
let stdoutBuf = "";
const stream = new PassThrough();
proc.stdout.pipe(stream);
let stderrBuf = "";
const MAX_STDERR = 4096;
const MAX_STDOUT = 8 * 1024 * 1024; // JSON metadata + requested format URLs
proc.stderr?.on("data", (chunk: Buffer) => {
if (stderrBuf.length < MAX_STDERR) {
stderrBuf += chunk.toString("utf8");
}
});
if (proc.stdout) {
proc.stdout.on("data", (chunk: Buffer) => {
if (stdoutBuf.length < MAX_STDOUT) {
stdoutBuf += chunk
.toString("utf8")
.slice(0, MAX_STDOUT - stdoutBuf.length);
}
});
}
if (proc.stderr) {
proc.stderr.on("data", (chunk: Buffer) => {
if (stderrBuf.length < MAX_STDERR) {
stderrBuf += chunk
.toString("utf8")
.slice(0, MAX_STDERR - stderrBuf.length);
}
});
}
let producedData = false;
stream.once("data", () => {
producedData = true;
});
proc.on("error", (err: NodeJS.ErrnoException) => {
activeProcesses.delete(proc);
rmSync(tmpDir, { recursive: true, force: true });
if (err.code === "ENOENT") {
reject(buildNotInstalledError());
stream.destroy(buildNotInstalledError());
} else {
reject(new Error(`yt-dlp failed to start: ${err.message}`));
stream.destroy(new Error(`yt-dlp failed to start: ${err.message}`));
}
});
proc.on("close", (code) => {
activeProcesses.delete(proc);
if (code !== 0) {
rmSync(tmpDir, { recursive: true, force: true });
// Fail fast: a download that dies before producing ANY bytes (e.g.
// transient YouTube 403) cannot feed the encoder — destroy the stream
// so the caller retries with a fresh yt-dlp run instead of streaming
// a silent black tile.
if (code !== 0 && !producedData && !stream.destroyed) {
const detail = stderrBuf.trim() ? `: ${stderrBuf.trim()}` : "";
reject(
stream.destroy(
new Error(
`yt-dlp screen input resolution exited with code ${code}${detail}`,
`yt-dlp screen input stream failed (exit ${code})${detail}`,
),
);
return;
}
let parsed: Record<string, unknown>;
try {
parsed = JSON.parse(stdoutBuf.trim()) as Record<string, unknown>;
} catch (parseErr) {
reject(
new Error(
`Failed to parse yt-dlp JSON for screen input: ${(parseErr as Error).message}`,
),
);
return;
}
resolveScreenInput(parsed).then(resolve, (err: unknown) => {
const message = err instanceof Error ? err.message : String(err);
reject(
new Error(`Failed to build screen input for "${url}": ${message}`),
);
});
});
// Resolve immediately — data flows as yt-dlp downloads. The caller's
// resolveInputWithRetry validates the first byte and retries on failure.
resolve(stream);
});
}
/**
* From a parsed yt-dlp JSON info dict, decide how to feed a single ffmpeg
* input with both video and audio.
*/
async function resolveScreenInput(
info: Record<string, unknown>,
): Promise<string | Readable> {
const requested = info.requested_formats as
| Array<Record<string, unknown>>
| undefined;
// Merged/progressive single URL (video+audio in one). Common when yt-dlp
// selects a single format (e.g. format 18 progressive mp4) or when a direct
// muxed URL is available.
const singleUrl = info.url as string | undefined;
const singleHasAudio =
info.acodec !== "none" &&
typeof info.acodec === "string" &&
info.acodec.length > 0;
if (typeof singleUrl === "string" && singleUrl && singleHasAudio) {
logger.debug("Screen share uses merged progressive single URL");
return singleUrl;
}
// Separate video-only + audio-only DASH formats → merge locally via ffmpeg.
if (Array.isArray(requested) && requested.length >= 2) {
const video = requested.find(
(rf) => rf.vcodec && String(rf.vcodec) !== "none",
);
const audio = requested.find(
(rf) => rf.acodec && String(rf.acodec) !== "none",
);
const videoUrl = video?.url as string | undefined;
const audioUrl = audio?.url as string | undefined;
if (
typeof videoUrl === "string" &&
videoUrl.length > 0 &&
typeof audioUrl === "string" &&
audioUrl.length > 0
) {
return mergeScreenStreams(videoUrl, audioUrl);
}
}
throw new Error(
"yt-dlp returned neither a merged progressive URL nor a video+audio format pair",
);
}
/**
* Merge a video-only URL and an audio-only URL into a single NUT stream using
* a child ffmpeg process. Both URLs come from the same yt-dlp run, so they
* share the same signature/expiry and are consumed immediately.
*/
function mergeScreenStreams(videoUrl: string, audioUrl: string): Readable {
logger.info("Merging video+audio DASH streams into a single NUT input");
const ffmpeg = spawn(
"ffmpeg",
[
"-hide_banner",
"-loglevel",
"error",
"-reconnect",
"1",
"-reconnect_streamed",
"1",
"-reconnect_delay_max",
"5",
"-i",
videoUrl,
"-i",
audioUrl,
"-map",
"0:v:0",
"-map",
"1:a:0",
"-c:v",
"copy",
"-c:a",
"copy",
"-f",
"nut",
"pipe:1",
],
{ stdio: ["ignore", "pipe", "pipe"] },
);
// Track so cleanup() can terminate the merge during graceful shutdown.
activeProcesses.add(ffmpeg);
ffmpeg.once("exit", () => {
activeProcesses.delete(ffmpeg);
});
// Prevent the ffmpeg stderr from filling the pipe buffer / leaking.
let stderrBuf = "";
const MAX_STDERR = 4096;
ffmpeg.stderr?.on("data", (chunk: Buffer) => {
if (stderrBuf.length < MAX_STDERR) {
stderrBuf += chunk.toString("utf8");
}
});
ffmpeg.on("error", (err) => {
const msg =
err.message === "spawn ffmpeg ENOENT"
? "FFmpeg not found! Install ffmpeg in the container."
: err.message;
logger.error({ error: msg }, "Screen stream merge ffmpeg error");
});
ffmpeg.on("exit", (code) => {
const stderr = stderrBuf.trim();
logger.warn(
{ code, stderr: stderr.slice(-500) || undefined },
"Screen stream merge ffmpeg exited",
);
});
const stream = ffmpeg.stdout;
stream.setMaxListeners(32);
return stream;
}
/**
* Extract metadata (title, duration, thumbnail) from a media URL
* without downloading the audio stream.
@@ -1,13 +1,19 @@
import { PassThrough, type Readable } from "node:stream";
import type { Client } from "discord.js-selfbot-v13";
import { createChildLogger } from "@/shared/logger/index";
import {
Encoders,
normalizeVideoCodec,
playStream,
prepareStream,
Streamer,
Utils,
} from "@dank074/discord-video-stream";
import type { Client } from "discord.js-selfbot-v13";
import { createChildLogger } from "@/shared/logger/index";
import { getDirectScreenInput } from "./mediaSource.js";
} from "../../goLive/index.js";
import {
getDirectScreenInput,
INVIDIOUS_INSTANCES,
isYoutubeWatchUrl,
toInvidiousUrl,
} from "./mediaSource.js";
import type { ScreenSharePlayback } from "./mediaTypes.js";
import { discordPlayer } from "./player.js";
@@ -54,6 +60,134 @@ export class ScreenShareController {
return this.active !== null;
}
/**
* Resolve the screen-share input with retry + first-byte validation.
*
* Transient YouTube 403s kill the merge ffmpeg BEFORE it produces any
* output; without validation the stream would "start" with a dead input
* and show a black tile forever. So after getDirectScreenInput resolves we
* tee the stream through a PassThrough and wait for the FIRST readable
* byte (or an error / early EOF). On failure the whole resolution is
* retried with a FRESH yt-dlp run (signed DASH URLs expire quickly the
* old URLs cannot simply be re-fetched).
*/
private async resolveInputWithRetry(source: string): Promise<Readable> {
const MAX_ATTEMPTS = 3;
let lastError: Error | null = null;
// YouTube may 403 even with account cookies (IP-bound session / bot check
// on VPS IP). When the source is a YouTube URL and cookies fail, fall back
// to anon Invidious mirror instances — no auth needed.
const isYt = isYoutubeWatchUrl(source);
let invidiousIdx = 0;
for (let attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
// On a 403 against YouTube, try the next Invidious instance for this attempt.
if (
isYt &&
lastError &&
/403|bot|Sign in|not a bot|access denied/i.test(lastError.message) &&
invidiousIdx < INVIDIOUS_INSTANCES.length
) {
const inst = INVIDIOUS_INSTANCES[invidiousIdx];
this.logger.warn(
{ attempt, instance: inst, error: lastError.message },
"YouTube blocked (403); falling back to Invidious mirror",
);
source = toInvidiousUrl(source, inst);
invidiousIdx++;
}
try {
const input = await getDirectScreenInput(source);
const tee = new PassThrough();
input.on("error", (err) => tee.destroy(err));
input.on("end", () => tee.end());
input.pipe(tee);
// If the merge process is stuck (no data, no exit) destroy the raw
// stream too so ffmpeg gets EPIPE on its next write and dies —
// otherwise every failed attempt leaks a merge process.
const destroyInput = () => {
try {
input.destroy();
} catch {
/* already gone */
}
};
await new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
cleanup();
destroyInput();
// Listeners were just removed by cleanup() — destroying tee WITH
// an error would emit "error" on an unlistened PassThrough and
// surface as an unhandled 'error' event (crash). Destroy
// silently; the error lives in the rejection only.
tee.destroy();
reject(
new Error(
"Screen input produced no data within 12s — merge likely failed",
),
);
}, 12000);
const onReadable = () => {
if (tee.readableLength > 0) {
cleanup();
resolve();
}
// readableLength === 0 can mean "EOF reached" — handled by onEnd.
};
const onError = (err: Error) => {
cleanup();
reject(err);
};
const onEnd = () => {
cleanup();
destroyInput();
reject(new Error("Screen input ended before producing any data"));
};
const cleanup = () => {
clearTimeout(timer);
tee.removeListener("readable", onReadable);
tee.removeListener("error", onError);
tee.removeListener("end", onEnd);
};
tee.once("readable", onReadable);
tee.once("error", onError);
tee.once("end", onEnd);
});
// Safety net: cleanup() removes the once() listeners on timeout/error,
// but a late error event from input.pipe(tee) can still fire on an
// unlistened PassThrough and crash the gateway (unhandled 'error').
// A permanent no-op listener guarantees the event is always swallowed.
tee.on("error", () => {});
// Pass the tee onward — the encoder consumes the same buffered
// stream, so no data from the merge is lost.
return tee;
} catch (err) {
lastError = err instanceof Error ? err : new Error(String(err));
this.logger.warn(
{
attempt,
maxAttempts: MAX_ATTEMPTS,
error: lastError.message,
},
"Screen input resolution failed; retrying with fresh yt-dlp",
);
if (attempt < MAX_ATTEMPTS) {
await new Promise((r) => setTimeout(r, 1500 * attempt));
}
}
}
throw (
lastError ??
new Error("Screen input resolution failed after multiple attempts")
);
}
async start(source: string): Promise<ScreenSharePlayback> {
const status = this.getVoiceStatus();
if (!status.connected || !status.activeGuildId || !status.activeChannelId) {
@@ -65,7 +199,7 @@ export class ScreenShareController {
}
try {
const input = await getDirectScreenInput(source);
const input = await this.resolveInputWithRetry(source);
if (!this.streamer) {
this.streamer = new Streamer(this.client);
}
@@ -98,16 +232,23 @@ export class ScreenShareController {
),
]);
const { command, output } = prepareStream(input, {
const prepared = prepareStream(input, {
encoder: Encoders.software({ x264: { preset: "superfast" } }),
width: 1280,
height: 720,
frameRate: 30,
bitrateVideo: 2500,
bitrateVideoMax: 4000,
// GoLive with audio: the encoder muxes to NUT (video h264 + opus
// audio) so the audio SSRC carries RTP too. Discord's GoLive
// pipeline expects audio — a video-only stream shows a static
// tile/thumbnail instead of live video. When the source has no
// audio track, the encoder's `-map 0:a:0?` yields no audio stream
// and the demuxer simply reports none (video still flows).
includeAudio: true,
videoCodec: Utils.normalizeVideoCodec("H264"),
videoCodec: normalizeVideoCodec("H264"),
});
const { command } = prepared;
let stopped = false;
// Restore the @discordjs/voice connection after the stream ends (both
@@ -142,10 +283,13 @@ export class ScreenShareController {
}, 5000);
}
};
const done = playStream(output, this.streamer, {
const done = playStream(prepared, this.streamer, {
type: "go-live",
width: 1280,
height: 720,
frameRate: 30,
})
.catch((err) => {
.catch((err: unknown) => {
// Never let a stream failure become an unhandledRejection — that
// crashed the whole gateway. Log + surface via the done promise.
const message = err instanceof Error ? err.message : String(err);
@@ -56,6 +56,24 @@ export class VoiceTransmitter {
// Create PCM input stream
this.pcmStream = new PassThrough();
this.pcmStream.setMaxListeners(32); // drain listeners accumulate during backpressure
// Voice teardown (stop / disconnect / ffmpeg exit) destroys this stream
// while Redis PCM messages may still be in flight. Without a listener,
// EPIPE / ERR_STREAM_DESTROYED / ERR_STREAM_WRITE_AFTER_END surface as
// an uncaughtException and crash the whole gateway.
this.pcmStream.on("error", (err: NodeJS.ErrnoException) => {
if (
err.code === "EPIPE" ||
err.code === "ERR_STREAM_DESTROYED" ||
err.code === "ERR_STREAM_WRITE_AFTER_END"
) {
logger.debug(
{ code: err.code },
"PCM stream closed during voice teardown — ignoring",
);
} else {
logger.error({ error: err.message }, "PCM stream error");
}
});
// Spawn FFmpeg to encode 24kHz mono PCM → OggOpus
// Input: 24kHz mono s16le (raw PCM)
@@ -146,7 +164,12 @@ export class VoiceTransmitter {
);
this.redisSub.on("message", (channel, message) => {
if (channel !== this.TRANSMIT_CHANNEL || !this.pcmStream) return;
if (
!this.isActive ||
channel !== this.TRANSMIT_CHANNEL ||
!this.pcmStream
)
return;
try {
const data = JSON.parse(message);
@@ -161,11 +184,21 @@ export class VoiceTransmitter {
this.draining = false;
// Re-acquire stream reference (could have been replaced by restart)
const currentStream = this.pcmStream;
if (!currentStream) return;
if (!currentStream || !this.isActive) return;
// Flush queued chunks
while (this.backpressureQueue.length > 0) {
const queued = this.backpressureQueue.shift()!;
if (!currentStream.write(queued)) break;
try {
if (!currentStream.write(queued)) break;
} catch (err) {
logger.debug(
{
error: err instanceof Error ? err.message : String(err),
},
"PCM flush write failed during teardown — ignoring",
);
break;
}
}
});
}
@@ -92,6 +92,10 @@ export const configSchema = z
// ── Redis ────────────────────────────────────────────────────────────
REDIS_URL: z.string().default("redis://localhost:6379"),
// ── SearXNG ───────────────────────────────────────────────────────────
// Instance for web search + term glossary lookups. Override when the
// default instance is down/rate-limited.
SEARXNG_BASE_URL: z.string().url().default("https://searxng.imrnes.team"),
// ── Voice PCM WebSocket (direct gateway→backend, bypasses Redis) ────
VOICE_PCM_WS_ENABLED: z
.string()
@@ -171,6 +175,24 @@ export const configSchema = z
.int()
.positive()
.default(30000),
// Term glossary — per-word Wikipedia lookups (via SearXNG) for words the
// LLM may not know (slang, jargon, regional language, foreign terms).
// Definitions are cached (in-memory + Redis) so repeat lookups are fast.
// Disable to skip glossary lookups entirely and analyze without them.
AI_GLOSSARY_ENABLED: z
.string()
.optional()
.transform((v) => v === "true")
.default(true),
// Max glossary terms looked up per analysis batch (keeps latency bounded).
AI_GLOSSARY_MAX_TERMS: z.coerce.number().int().min(1).max(20).default(6),
// Min word length for a term to be considered glossary-worthy.
AI_GLOSSARY_MIN_WORD_LENGTH: z.coerce
.number()
.int()
.min(2)
.max(20)
.default(5),
// ── AI Analysis Timing ──────────────────────────────────────────────
AI_ANALYSIS_DEBOUNCE_MS: z.coerce.number().positive().default(500),
@@ -189,6 +211,17 @@ export const configSchema = z
.int()
.positive()
.default(20),
// Recency gates for conversation context. A silence longer than GAP_MS
// between context messages = the conversation restarted (older messages
// dropped); MAX_AGE_MS caps how far back context is considered relevant.
AI_ANALYSIS_CONTEXT_GAP_MS: z.coerce
.number()
.positive()
.default(12 * 60 * 1000),
AI_ANALYSIS_CONTEXT_MAX_AGE_MS: z.coerce
.number()
.positive()
.default(45 * 60 * 1000),
AI_ANALYSIS_PROCESSING_TIMEOUT_MS: z.coerce
.number()
.positive()
@@ -242,6 +275,19 @@ export const configSchema = z
.default(false),
AUTO_DELETE_LOG_CHANNEL_ID: z.string().default(""),
// ── Nickname Reset (offensive_username enforcement) ────────────────
// When the only violation is the member's server nickname, reset the
// nickname to the default username instead of deleting the message.
AUTO_NICKNAME_RESET_ENABLED: z
.string()
.optional()
.transform((v) => v === "true")
.default(true),
AUTO_NICKNAME_RESET_COOLDOWN_MS: z.coerce
.number()
.positive()
.default(10 * 60 * 1000),
// ── Retention ───────────────────────────────────────────────────────
RETENTION_MESSAGES_DAYS: z.coerce.number().int().min(0).default(0),
RETENTION_ATTACHMENTS_DAYS: z.coerce.number().int().min(0).default(0),
@@ -435,6 +435,32 @@ export const pgStickerCacheTable = pgTable(
export const stickerCacheTable = pgStickerCacheTable;
/**
* Term Glossary Cache Table (PostgreSQL)
* Permanently stores resolved term definitions (Wikipedia/SearXNG lookups).
* Definitions rarely change, so once a term is successfully resolved it is
* persisted here forever Redis/LRU only act as fast read caches in front.
* Terms with NO definition (misses) are NOT stored here; they stay ephemeral
* in Redis with a short TTL so transient lookup failures get retried.
*/
export const pgTermGlossaryCacheTable = pgTable(
"term_glossary_cache",
{
term: pgText("term").primaryKey(),
definition: pgText("definition").notNull(),
source_url: pgText("source_url").notNull().default(""),
resolved_at: pgBigint("resolved_at", { mode: "number" }).notNull(),
hit_count: pgInteger("hit_count").notNull().default(0),
},
(table) => ({
resolvedAtIdx: pgIndex("idx_term_glossary_cache_resolved_at").on(
table.resolved_at,
),
}),
);
export const termGlossaryCacheTable = pgTermGlossaryCacheTable;
// =============================================================================
// Meta / System
// =============================================================================
@@ -580,6 +606,11 @@ export type TextAnalysisCacheInsert =
export type StickerCacheRecord = typeof stickerCacheTable.$inferSelect;
export type StickerCacheInsert = typeof stickerCacheTable.$inferInsert;
// Term Glossary Cache
export type TermGlossaryCache = typeof termGlossaryCacheTable.$inferSelect;
export type TermGlossaryCacheInsert =
typeof termGlossaryCacheTable.$inferInsert;
// Muxer Jobs
export type MuxerJob = typeof muxerJobsTable.$inferSelect;
export type MuxerJobInsert = typeof muxerJobsTable.$inferInsert;
@@ -615,6 +646,7 @@ export const pgModerationActionsTable = pgTable(
"warn_user",
"kick_user",
"ban_user",
"reset_nickname",
],
}).notNull(),
reason: pgText("reason"),
@@ -198,7 +198,8 @@ export type ModerationActionType =
| "mute_user"
| "warn_user"
| "kick_user"
| "ban_user";
| "ban_user"
| "reset_nickname";
export interface ModerationAction {
id: string;
@@ -0,0 +1,209 @@
// ═══════════════════════════════════════════════════════════════════════════
// Context enrichment builders — rich <user_reputation> attrs, <user_history>,
// <user_profiles> as_of, bot/edited detection (pure, no DB)
// ═══════════════════════════════════════════════════════════════════════════
import { describe, expect, it } from "vitest";
import {
buildUserHistoryXml,
buildUserProfilesBlock,
formatReputationAttrs,
resolveIsBot,
resolveIsEdited,
} from "../src/modules/ai-moderation/moderationBuilders.js";
import type { MessageRecord } from "../src/modules/message-capture/types.js";
const NOW = 1_800_000_000_000;
function msg(overrides: Partial<MessageRecord> = {}): MessageRecord {
return {
id: "m1",
guild_id: "g1",
channel_id: "c1",
thread_id: null,
user_id: "u1",
username: "user1",
avatar_url: null,
content: "hai",
edited_content: null,
created_at: NOW,
edited_at: null,
deleted_at: null,
type: "text",
is_reply: null,
is_forward: null,
is_crosspost: null,
reference_message_id: null,
reference_channel_id: null,
reference_guild_id: null,
metadata: null,
...overrides,
};
}
const DAY_MS = 24 * 60 * 60 * 1000;
describe("formatReputationAttrs — rich reputation signal", () => {
it("emits trust, infraction count and clean streak", () => {
const attrs = formatReputationAttrs({
trust_score: 62,
total_infractions: 3,
clean_message_streak: 45,
last_infraction_at: null,
});
expect(attrs).toContain('trust_score="62"');
expect(attrs).toContain('total_infractions="3"');
expect(attrs).toContain('clean_streak="45"');
});
it("derives last_offense_days_ago and marks repeat offenders (7-day window)", () => {
const attrs = formatReputationAttrs(
{
trust_score: 50,
total_infractions: 2,
clean_message_streak: 0,
last_infraction_at: NOW - 2 * DAY_MS,
},
NOW,
);
expect(attrs).toContain('last_offense_days_ago="2"');
expect(attrs).toContain('repeat_offender="true"');
});
it("does NOT mark repeat offender when the last offense is older than 7 days", () => {
const attrs = formatReputationAttrs(
{
trust_score: 50,
total_infractions: 2,
clean_message_streak: 10,
last_infraction_at: NOW - 30 * DAY_MS,
},
NOW,
);
expect(attrs).toContain('last_offense_days_ago="30"');
expect(attrs).not.toContain("repeat_offender");
});
it("omits offense-derived attrs when the user has no recorded infraction date", () => {
const attrs = formatReputationAttrs({
trust_score: 85,
total_infractions: 0,
clean_message_streak: 120,
last_infraction_at: null,
});
expect(attrs).not.toContain("last_offense_days_ago");
expect(attrs).not.toContain("repeat_offender");
});
it("clamps a future/skewed timestamp to days_ago=0", () => {
const attrs = formatReputationAttrs(
{
trust_score: 50,
total_infractions: 1,
clean_message_streak: 0,
last_infraction_at: NOW + 5 * DAY_MS,
},
NOW,
);
expect(attrs).toContain('last_offense_days_ago="0"');
});
});
describe("buildUserHistoryXml — last flagged messages for repeat offenders", () => {
it("returns empty when there is no real history", () => {
expect(buildUserHistoryXml([])).toBe("");
expect(
buildUserHistoryXml([{ content: " ", severity: "low", created_at: 1 }]),
).toBe("");
});
it("renders <infraction> rows with severity and recency", () => {
const xml = buildUserHistoryXml(
[
{
content: "beli barang murah disini https://scam.example",
severity: "high",
created_at: NOW - 3 * DAY_MS,
},
],
NOW,
);
expect(xml).toContain("<user_history>");
expect(xml).toContain('severity="high"');
expect(xml).toContain('time_ago_days="3"');
expect(xml).toContain("beli barang murah disini");
});
it("caps long snippets and XML-escapes content", () => {
const xml = buildUserHistoryXml(
[
{
content: "x".repeat(300),
severity: "low",
created_at: NOW - DAY_MS,
},
],
NOW,
);
expect(xml.length).toBeLessThan(250);
});
});
describe("buildUserProfilesBlock — deduplicated map with staleness", () => {
it("emits as_of when the profile has a last-generated timestamp", () => {
const block = buildUserProfilesBlock(
new Map([
[
"u1",
{
text: "Developer teknis, bahasa Indonesia",
asOf: NOW - 3 * DAY_MS,
},
],
]),
);
expect(block).toContain('<user_profile user_id="u1"');
expect(block).toContain(
`as_of="${new Date(NOW - 3 * DAY_MS).toISOString()}"`,
);
expect(block).toContain("Developer teknis");
});
it("omits as_of when absent, and drops empty profiles", () => {
const block = buildUserProfilesBlock(
new Map([
["u1", { text: "profil aktif", asOf: null }],
["u2", { text: " " }],
]),
);
expect(block).toContain('user_id="u1"');
expect(block).not.toContain("as_of");
expect(block).not.toContain("u2");
});
it("returns empty for no profiles", () => {
expect(buildUserProfilesBlock(new Map())).toBe("");
});
});
describe("resolveIsBot / resolveIsEdited — message flags", () => {
it("reads author.bot from captured metadata", () => {
const bot = msg({
metadata: JSON.stringify({
author: { id: "x", username: "bot", bot: true },
}),
});
const human = msg({
metadata: JSON.stringify({
author: { id: "y", username: "user", bot: false },
}),
});
expect(resolveIsBot(bot)).toBe(true);
expect(resolveIsBot(human)).toBe(false);
expect(resolveIsBot(msg())).toBe(false);
});
it("flags edited content only when edited_content is present (the edit path)", () => {
expect(resolveIsEdited(msg({ edited_content: "versi baru" }))).toBe(true);
expect(resolveIsEdited(msg())).toBe(false);
});
});
@@ -0,0 +1,299 @@
// ═══════════════════════════════════════════════════════════════════════════
// Conversation context v2 — recency gating + location context (pure, no DB)
// ═══════════════════════════════════════════════════════════════════════════
import { describe, expect, it } from "vitest";
import {
buildConversationContext,
buildLocationContext,
formatMessageForPrompt,
truncateContextLine,
} from "../src/modules/ai-moderation/conversationContext.js";
import { buildConversationContextBlock } from "../src/modules/ai-moderation/moderationBuilders.js";
import { extractOgMeta } from "../src/modules/ai-moderation/urlFetcher.js";
import type { MessageRecord } from "../src/modules/message-capture/types.js";
const NOW = 1_800_000_000_000;
function msg(id: string, createdAt: number, content = "hai"): MessageRecord {
return {
id,
guild_id: "g1",
channel_id: "c1",
thread_id: null,
user_id: `u_${id}`,
username: `user_${id}`,
avatar_url: null,
content,
edited_content: null,
created_at: createdAt,
edited_at: null,
deleted_at: null,
type: "text",
is_reply: null,
is_forward: null,
is_crosspost: null,
reference_message_id: null,
reference_channel_id: null,
reference_guild_id: null,
metadata: null,
};
}
function target(id = "t1", createdAt = NOW): MessageRecord {
return {
...msg(id, createdAt),
content: "pesan yang dianalisis",
};
}
const MIN = 60_000;
describe("buildConversationContext — recency gating", () => {
it("keeps an ONGOING conversation — recent messages, small gaps", () => {
const context = [
msg("a", NOW - 8 * MIN),
msg("b", NOW - 6 * MIN),
msg("c", NOW - 4 * MIN),
msg("d", NOW - 2 * MIN),
];
const { lines, descriptor, dropped } = buildConversationContext({
contextBefore: context,
targets: [target()],
maxTokens: 8000,
gapMs: 12 * MIN,
maxAgeMs: 45 * MIN,
});
expect(lines).toHaveLength(4);
expect(dropped).toBe(0);
expect(descriptor).toContain("status=ongoing");
});
it("drops messages before a silence gap — conversation RESTARTED", () => {
const context = [
msg("old1", NOW - 40 * MIN),
msg("old2", NOW - 38 * MIN),
msg("fresh", NOW - 5 * MIN),
];
const { lines, descriptor, dropped } = buildConversationContext({
contextBefore: context,
targets: [target()],
maxTokens: 8000,
gapMs: 12 * MIN,
maxAgeMs: 45 * MIN,
});
// 40min-old messages are within maxAge but 33min before "fresh" → gap gate
expect(lines.some((l) => l.includes("old1"))).toBe(false);
expect(lines.some((l) => l.includes("fresh"))).toBe(true);
expect(dropped).toBe(2);
expect(descriptor).toContain("status=sparse");
expect(descriptor).toContain("gap_before_min=");
});
it("drops everything older than maxAge — stale noise, cold_start anchor kept", () => {
const context = [
msg("ancient", NOW - 120 * MIN),
msg("stale", NOW - 60 * MIN),
];
const { lines, descriptor, dropped } = buildConversationContext({
contextBefore: context,
targets: [target()],
maxTokens: 8000,
gapMs: 12 * MIN,
maxAgeMs: 45 * MIN,
});
// Age gate drops both from the real context block, but the cold-start
// anchor keeps the nearest 2 so the LLM still senses the channel.
expect(dropped).toBe(2);
expect(descriptor).toContain("status=cold_start");
expect(lines).toHaveLength(2);
});
it("keeps a 2-message anchor on cold start so the LLM senses the channel", () => {
const context = [
msg("far1", NOW - 100 * MIN),
msg("far2", NOW - 99 * MIN),
msg("near1", NOW - 50 * MIN),
];
const { lines, descriptor } = buildConversationContext({
contextBefore: context,
targets: [target()],
maxTokens: 8000,
gapMs: 12 * MIN,
maxAgeMs: 45 * MIN,
});
expect(lines).toHaveLength(2); // nearest 2 kept as anchor
expect(lines.some((l) => l.includes("near1"))).toBe(true);
expect(descriptor).toContain("status=cold_start");
});
it("respects the token budget (older lines dropped first)", () => {
const context = Array.from({ length: 20 }, (_, i) =>
msg(`m${i}`, NOW - (i + 1) * MIN),
);
const { lines } = buildConversationContext({
contextBefore: context,
targets: [target()],
maxTokens: 600,
gapMs: 12 * MIN,
maxAgeMs: 45 * MIN,
});
expect(lines.length).toBeLessThan(20);
expect(lines.length).toBeGreaterThan(0);
});
});
describe("formatMessageForPrompt — server nickname (displayName)", () => {
it("renders member.displayName when captured (per-server nickname)", () => {
const m = msg("n1", NOW - MIN);
m.metadata = JSON.stringify({
member: {
displayName: "Si Goblok Server",
roles: [],
joinedTimestamp: null,
},
});
const line = formatMessageForPrompt(m, "context");
expect(line).toContain("user=Si Goblok Server");
expect(line).not.toContain("user_user_n1");
});
it("falls back to global username when displayName missing", () => {
const line = formatMessageForPrompt(
msg("n2", NOW - MIN, "halo"),
"context",
);
expect(line).toContain("user=user_n2");
});
it("truncates an oversized context message so one paste cannot eat the whole budget", () => {
const huge = "A".repeat(5000);
const line = formatMessageForPrompt(msg("n3", NOW - MIN, huge), "context");
expect(line).toContain("…[konteks dipotong: terlalu panjang]");
expect(line.length).toBeLessThan(2000);
});
it("keeps short context content intact", () => {
expect(truncateContextLine("pendek")).toBe("pendek");
});
});
describe("buildLocationContext — channel/thread/nsfw enrichment", () => {
it("renders a structured <location_context/> element from captured metadata", () => {
const t = target();
t.metadata = JSON.stringify({
channel: {
channelName: "general",
threadName: "tanya coding",
nsfw: false,
ageRestricted: false,
},
});
const line = buildLocationContext([t]);
expect(line).toContain("<location_context");
expect(line).toContain('channel_id="c1"');
expect(line).toContain('channel_name="general"');
expect(line).toContain('thread_name="tanya coding"');
expect(line).toContain('nsfw="false"');
expect(line).toContain('age_restricted="false"');
});
it("includes the channel topic (escaped) when captured", () => {
const t = target();
t.metadata = JSON.stringify({
channel: {
channelName: "rules",
topic: "Diskusi coding & programming — no self-promo",
nsfw: false,
},
});
const line = buildLocationContext([t]);
expect(line).toContain(
'topic="Diskusi coding &amp; programming — no self-promo"',
);
});
it("caps an oversized topic and omits empty/absent topic", () => {
const t = target();
t.metadata = JSON.stringify({
channel: { channelName: "general", topic: "x".repeat(500), nsfw: false },
});
const line = buildLocationContext([t]);
const match = line.match(/topic="([^"]*)"/);
expect(match).not.toBeNull();
expect(match?.[1].length).toBeLessThanOrEqual(201);
const t2 = target();
t2.metadata = JSON.stringify({ channel: { channelName: "general" } });
expect(buildLocationContext([t2])).not.toContain("topic=");
});
it("returns empty when no metadata", () => {
expect(buildLocationContext([target()])).toBe("");
});
});
describe("buildConversationContextBlock — structured USER-message context", () => {
it("wraps location + descriptor + lines into XML blocks", () => {
const block = buildConversationContextBlock({
location: buildLocationContext(
(() => {
const t = target();
t.metadata = JSON.stringify({
channel: { channelName: "general", nsfw: false },
});
return [t];
})(),
),
descriptor: "[conversation_flow] status=ongoing context_msgs=1 dropped=0",
lines: ["[context] id=a time=2027-01-01T00:00:00.000Z user=user_a: hai"],
});
expect(block).toContain("<location_context");
expect(block).toContain("<conversation_context>");
expect(block).toContain("[conversation_flow] status=ongoing");
expect(block).toContain("[context] id=a");
// location block comes before conversation block
expect(block.indexOf("<location_context")).toBeLessThan(
block.indexOf("<conversation_context>"),
);
});
it("omits the conversation block when there are no lines", () => {
const block = buildConversationContextBlock({
location: "",
descriptor: "",
lines: [],
});
expect(block).toBe("");
});
it("keeps only the location block when lines are empty but location exists", () => {
const block = buildConversationContextBlock({
location: '<location_context channel_id="c1"/>',
descriptor: "",
lines: [],
});
expect(block).toBe('<location_context channel_id="c1"/>');
});
});
describe("extractOgMeta — page title/site for <web_content>", () => {
it("extracts og:title, og:description and og:site_name", () => {
const html = `
<html><head>
<title>Fallback title</title>
<meta property="og:title" content="Judul Halaman &amp; Keren" />
<meta property="og:description" content="Deskripsi halaman" />
<meta property="og:site_name" content="Contoh Site" />
<meta property="og:image" content="https://img.example.com/x.png" />
</head></html>`;
const meta = extractOgMeta(html);
expect(meta.title).toBe("Judul Halaman & Keren");
expect(meta.description).toBe("Deskripsi halaman");
expect(meta.siteName).toBe("Contoh Site");
});
it("falls back to <title> when og:title missing", () => {
const html = "<html><head><title>Plain Title</title></head></html>";
expect(extractOgMeta(html).title).toBe("Plain Title");
});
});
@@ -0,0 +1,187 @@
import { describe, expect, it } from "vitest";
import { existsSync, mkdtempSync, readdirSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { PassThrough } from "node:stream";
import { demux } from "../src/goLive/Demuxer.js";
/** Same resolution as Demuxer.resolveBin: env override → Nix store scan. */
function resolveFfmpeg(): string | null {
const override = process.env.FFMPEG_PATH;
if (override && existsSync(override)) return override;
const store = "/nix/store";
if (existsSync(store)) {
for (const entry of readdirSync(store)) {
if (!entry.includes("ffmpeg-headless-")) continue;
const candidate = join(store, entry, "bin", "ffmpeg");
if (existsSync(candidate)) return candidate;
}
}
return existsSync("/usr/bin/ffmpeg") ? "/usr/bin/ffmpeg" : null;
}
/**
* Ogg Opus byte stream built by hand: OpusHead (19B) + a few 20ms opus
* frames, wrapped in valid Ogg pages (CRC 0 the parser ignores CRC).
*/
function buildOggOpusBytes(frames: number): Buffer {
const opusHead = Buffer.from([
0x4f, 0x70, 0x75, 0x73, 0x48, 0x65, 0x61, 0x64, // "OpusHead"
0x01, 0x02, 0x38, 0x01, 0x80, 0xbb, 0x00, 0x00,
0x00, 0x00, 0x00, // version 1, 2ch, 48000
]);
// A minimal valid opus data packet: TOC 0xFC (48kHz stereo 20ms) + payload
const dataPacket = Buffer.alloc(20, 0);
dataPacket[0] = 0xfc;
const makePage = (
seq: number,
headerType: number,
serial: number,
packets: Buffer[],
): Buffer => {
const segmentTable: number[] = [];
const payloadParts: Buffer[] = [];
for (const p of packets) {
let remaining = p.length;
let off = 0;
do {
const chunk = Math.min(255, remaining);
segmentTable.push(chunk);
payloadParts.push(p.subarray(off, off + chunk));
off += chunk;
remaining -= chunk;
} while (remaining > 0);
}
const payload = Buffer.concat(payloadParts);
const header = Buffer.alloc(27 + segmentTable.length);
header.write("OggS", 0, "latin1");
header[4] = 0; // version
header[5] = headerType;
header.writeUInt32LE(0, 6); // granule (unused)
header.writeUInt32LE(0, 10);
header.writeUInt32LE(serial, 14);
header.writeUInt32LE(seq, 18);
header.writeUInt32LE(0, 22); // crc (ignored)
header[26] = segmentTable.length;
for (let i = 0; i < segmentTable.length; i++) header[27 + i] = segmentTable[i];
return Buffer.concat([header, payload]);
};
const pages: Buffer[] = [];
const serial = 0x1234;
let seq = 0;
// Page 0: BOS + OpusHead (19 bytes, single lacing)
pages.push(makePage(seq++, 0x02, serial, [opusHead]));
// Pages 1+: data packets, a few per page
const perPage = 3;
for (let i = 0; i < frames; i += perPage) {
const pkts = [];
for (let j = 0; j < perPage && i + j < frames; j++) pkts.push(dataPacket);
pages.push(makePage(seq++, 0x00, serial, pkts));
}
return Buffer.concat(pages);
}
describe("Demuxer NUT path with audio", () => {
it("emits video access units AND parsed opus audio frames", async () => {
// Real NUT file (video h264 + opus) produced by ffmpeg — generated once
// in this test via ffmpeg, skipped if ffmpeg is unavailable.
const ffmpeg = resolveFfmpeg();
if (!ffmpeg) {
console.warn("ffmpeg not found — skipping NUT integration case");
return;
}
const dir = mkdtempSync(join(tmpdir(), "gmw-nut-test-"));
const inWebm = join(dir, "in.webm");
const inNut = join(dir, "in.nut");
try {
// Build a tiny webm (vpx + opus) then remux to NUT h264+opus — mirrors
// prepareStream(includeAudio) output.
const { spawnSync } = await import("node:child_process");
const gen = spawnSync(
ffmpeg,
[
"-hide_banner", "-loglevel", "error",
"-f", "lavfi", "-i", "testsrc2=size=160x120:rate=10:duration=3",
"-f", "lavfi", "-i", "sine=frequency=440:duration=3",
"-c:v", "libvpx-vp9", "-b:v", "100k", "-pix_fmt", "yuv420p",
"-c:a", "libopus", "-b:a", "48k", "-f", "webm", "-y", inWebm,
],
{ timeout: 20000 },
);
if (gen.status !== 0) {
console.warn("ffmpeg webm gen failed — skipping", gen.stderr?.toString().slice(0, 200));
return;
}
const enc = spawnSync(
ffmpeg,
[
"-hide_banner", "-loglevel", "error", "-i", inWebm,
"-map", "0:v:0", "-c:v", "libx264", "-profile:v", "baseline",
"-x264-params", "repeat-headers=1", "-preset", "superfast",
"-pix_fmt", "yuv420p", "-g", "10", "-forced-idr", "1",
"-map", "0:a:0?", "-c:a", "libopus", "-b:a", "48k", "-ar", "48000", "-ac", "2",
"-f", "nut", "-y", inNut,
],
{ timeout: 20000 },
);
if (enc.status !== 0) {
console.warn("ffmpeg nut gen failed — skipping", enc.stderr?.toString().slice(0, 200));
return;
}
const { readFileSync } = await import("node:fs");
// demux() takes string | PassThrough — wrap the file read. Drip the
// bytes in quickly (well within demux's 1.5s metadata window) so ffmpeg
// prints both stream lines before demux() resolves.
const input = new PassThrough();
const nutBytes = readFileSync(inNut);
const CHUNK = Math.max(1024, Math.floor(nutBytes.length / 20));
let off = 0;
const drip = setInterval(() => {
if (off >= nutBytes.length) {
clearInterval(drip);
input.end();
return;
}
input.write(nutBytes.subarray(off, off + CHUNK));
off += CHUNK;
}, 10);
const { video, audio, close } = await demux(input, {
format: "nut",
frameRate: 10,
});
const vFrames: number[] = [];
const aFrames: number[] = [];
video?.stream.on("data", (f: { data: Buffer | null; flags: number }) => {
if (f.data) vFrames.push(f.data.length);
});
audio?.stream.on("data", (f: { data: Buffer | null; duration: number }) => {
if (f.data) aFrames.push(f.duration);
});
await new Promise<void>((resolve) => {
video?.stream.on("end", resolve);
setTimeout(resolve, 6000);
});
close();
expect(vFrames.length).toBeGreaterThan(0);
// ~10fps × 1s of video → at least 5 access units
expect(vFrames.length).toBeGreaterThanOrEqual(5);
// ~50 opus frames/sec of audio
expect(aFrames.length).toBeGreaterThan(10);
// opus frames are 20ms (duration 960 @ 48kHz)
expect(aFrames[0]).toBe(960);
} finally {
rmSync(dir, { recursive: true, force: true });
}
}, 30000);
it("parses a hand-built Ogg Opus stream into frames", async () => {
// Feed the OGG bytes via the demux's audio path is not directly
// exposed — instead verify the parser contract through the NUT path is
// covered above; here we sanity-check the byte layout our parser reads.
const bytes = buildOggOpusBytes(7);
expect(bytes.subarray(0, 4).toString("latin1")).toBe("OggS");
// 7 frames + header across pages
expect(bytes.includes(Buffer.from("OpusHead"))).toBe(true);
expect(bytes.subarray(28, 36).toString("latin1")).toBe("OpusHead");
});
});
@@ -0,0 +1,97 @@
/**
* goLive port smoke tests verify the TS layer (no native binding needed
* for these; native is covered by the C++/node test-packetizer.js).
*/
import { describe, expect, it } from "vitest";
import { H264Helpers } from "../src/goLive/AnnexBHelper.js";
import { AVCodecID } from "../src/goLive/Demuxer.js";
import {
BaseMediaStream,
CodecPayloadType,
Encoders,
normalizeVideoCodec,
} from "../src/goLive/index.js";
import { rewriteSPSVUI } from "../src/goLive/SPSVUIRewriter.js";
describe("goLive port: codec + encoders", () => {
it("normalizeVideoCodec maps aliases to canonical names", () => {
expect(normalizeVideoCodec("H.264")).toBe("H264");
expect(normalizeVideoCodec("AVC")).toBe("H264");
expect(normalizeVideoCodec("h265")).toBe("H265");
expect(normalizeVideoCodec("vp8")).toBe("VP8");
expect(normalizeVideoCodec("av1")).toBe("AV1");
});
it("software encoder exposes x264 libx264 baseline zerolatency", () => {
const enc = Encoders.software()();
expect(enc.H264.name).toBe("libx264");
expect(enc.H264.options).toContain("-preset superfast");
expect(enc.H264.options).toContain("-tune zerolatency");
// Baseline profile is REQUIRED to match the SDP's profile-level-id=42e01f
// (constrained baseline) — High-profile bitstreams fail to decode → black
expect(enc.H264.options).toContain("-profile:v baseline");
});
it("CodecPayloadType has opus + H264 entries", () => {
expect(CodecPayloadType.opus).toBeDefined();
expect(CodecPayloadType.H264).toBeDefined();
});
});
describe("goLive port: annexb + sps rewriter", () => {
it("H264Helpers detects NAL unit types", () => {
const nal = Buffer.from([0x67, 0x42, 0x00, 0x1e]); // SPS
expect(H264Helpers.getUnitType(nal)).toBe(7); // SPS type
expect(H264Helpers.getUnitType(Buffer.from([0x65, 0x88]))).toBe(5); // IDR
});
it("rewriteSPSVUI returns a buffer for valid SPS", () => {
const sps = Buffer.from([
0x67, 0x42, 0x00, 0x1e, 0x96, 0x54, 0x05, 0x01, 0xec, 0x80,
]);
expect(() => rewriteSPSVUI(sps)).not.toThrow();
});
});
describe("goLive port: streams", () => {
it("BaseMediaStream accepts plain frame objects", () => {
// BaseMediaStream is abstract — use a concrete subclass that no-ops the
// packetizer hook.
class TestStream extends BaseMediaStream {
async _sendFrame(_frame: Buffer, _frametime: number): Promise<void> {
/* no-op */
}
}
const stream = new TestStream("video");
const frame = {
data: Buffer.from([1, 2, 3]),
pts: 0,
duration: 40,
timeBase: { num: 1, den: 48000 },
flags: 0,
streamIndex: 0,
free: () => {},
};
expect(() => stream.write(frame)).not.toThrow();
stream.end();
});
});
describe("goLive port: demuxer codec ids", () => {
it("maps H264/HEVC/opus AVCodecID values", () => {
expect(AVCodecID.AV_CODEC_ID_H264).toBe(27);
expect(AVCodecID.AV_CODEC_ID_HEVC).toBe(173);
expect(AVCodecID.AV_CODEC_ID_OPUS).toBe(86019);
});
});
describe("goLive port: prepareStream option merge", () => {
it("merges default options into the descriptor (no ffmpeg spawn)", () => {
// Import the merge logic directly via the module; prepareStream spawns
// ffmpeg so we verify the descriptors it would build by checking the
// encoder + option functions that prepareStream uses.
const enc = Encoders.software()();
expect(enc.H264.options).toContain("-forced-idr 1");
expect(normalizeVideoCodec("H264")).toBe("H264");
});
});
@@ -0,0 +1,40 @@
// Phase 2 E2E: Demuxer on a real ffmpeg-generated H264 file.
// Run: npx tsx tests/golive-demux-e2e.ts
import { createReadStream } from "node:fs";
import { demux } from "../src/goLive/Demuxer.js";
const input = process.argv[2] ?? "/tmp/sample.h264";
const { video, close } = await demux(createReadStream(input), {
format: "h264",
});
console.log(
"video:",
JSON.stringify({
codecName: video.codecName,
width: video.width,
height: video.height,
duration: video.duration,
fps: Math.round(video.framerate_num / video.framerate_den),
}),
);
let count = 0;
let keyframes = 0;
let bytes = 0;
video.stream.on("data", (frame: { data: Buffer; keyframe: boolean }) => {
count++;
bytes += frame.data.length;
if (frame.keyframe) keyframes++;
});
video.stream.on("end", () => {
console.log(`frames: ${count} (${keyframes} keyframes), ${bytes} bytes`);
close();
process.exit(0);
});
video.stream.on("error", (e: unknown) => {
console.error("stream error:", e);
close();
process.exit(1);
});
@@ -0,0 +1,153 @@
// Regression test: demux must emit frames from a LIVE stream that never
// ends (the NUT/H264 merge output during playback). The old implementation
// spooled the whole stream to a file first → deadlocked forever → 0 frames.
// v2: also validates ACCESS-UNIT grouping — each emitted frame must be a
// complete picture (parameter sets + slice), never a bare SPS/PPS/SEI NAL,
// and must be timestamped at the video frame rate (RTP +clockRate/fps).
// Run: npx tsx tests/golive-demux-live-e2e.ts [ffmpeg-path]
import { spawn } from "node:child_process";
import { PassThrough } from "node:stream";
import { demux } from "../src/goLive/Demuxer.js";
const FFMPEG = process.argv[2] ?? "ffmpeg";
// 1) Generate a 2s H264 test clip to a temp file
const clip = "/tmp/golive-live-test.h264";
await new Promise<void>((resolve, reject) => {
const p = spawn(
FFMPEG,
[
"-hide_banner",
"-loglevel",
"error",
"-f",
"lavfi",
"-i",
"testsrc=size=640x360:rate=30:duration=2",
"-c:v",
"libx264",
"-preset",
"ultrafast",
"-pix_fmt",
"yuv420p",
"-f",
"h264",
clip,
],
{ stdio: ["ignore", "ignore", "pipe"] },
);
let err = "";
p.stderr?.on("data", (d: Buffer) => (err += d.toString()));
p.on("close", (code) => (code === 0 ? resolve() : reject(new Error(err))));
});
// 2) Feed the clip through a PassThrough but DON'T end it (live semantics),
// with a small pause after the first chunk so demux has time to emit.
const input = new PassThrough();
const demuxPromise = demux(input, { format: "h264", frameRate: 30 });
const { video, close } = await demuxPromise;
if (!video) {
console.error("FAIL: demux returned no video stream");
process.exit(1);
}
interface Emitted {
data: Buffer;
duration: number;
timeBase: { num: number; den: number };
flags: number;
}
const frames: Emitted[] = [];
video.stream.on("data", (frame: Emitted) => {
frames.push(frame);
});
const fs = await import("node:fs");
const buf = fs.readFileSync(clip);
const chunkSize = 16384;
for (let i = 0; i < buf.length; i += chunkSize) {
input.write(buf.subarray(i, i + chunkSize));
if (i === 0) await new Promise((r) => setTimeout(r, 1500));
}
// Stream still open — if the old spool logic was here we'd never emit.
await new Promise((r) => setTimeout(r, 500));
// 3) Validate access-unit structure
const nalTypes = (frame: Buffer): number[] => {
const out: number[] = [];
let i = 0;
while (i < frame.length - 3) {
if (frame[i] === 0 && frame[i + 1] === 0 && frame[i + 2] === 1) {
const start = i;
let j = i + 3;
if (frame[j - 4] === 0 && j >= 4) {
// 4-byte start code already consumed by i pointing at the 3-byte tail
}
while (j < frame.length - 3) {
if (frame[j] === 0 && frame[j + 1] === 0 && frame[j + 2] === 1) break;
j++;
}
const nal = frame.subarray(start + 3, j);
if (nal.length > 0) out.push(nal[0] & 0x1f);
i = j;
} else {
i++;
}
}
return out;
};
let bareParamSetFrames = 0;
let framesWithoutSlice = 0;
let keyframesWithParamSets = 0;
let keyframesWithoutParamSets = 0;
for (const f of frames) {
const types = nalTypes(f.data);
const hasSlice = types.some((t) => t === 1 || t === 5);
const hasParams = types.some((t) => t === 7 || t === 8);
const isKey = (f.flags & 1) !== 0;
if (!hasSlice) framesWithoutSlice++;
if (
types.length === 1 &&
(types[0] === 7 || types[0] === 8 || types[0] === 6)
) {
bareParamSetFrames++;
}
if (isKey && hasParams) keyframesWithParamSets++;
if (isKey && !hasParams) keyframesWithoutParamSets++;
}
console.log(
`metadata: ${video.codecName} ${video.width}x${video.height} fps=${video.framerate_num}/${video.framerate_den}`,
);
console.log(`frames while stream OPEN (not ended): ${frames.length}`);
console.log(
`frames w/o slice NAL: ${framesWithoutSlice}, bare param-set frames: ${bareParamSetFrames}`,
);
console.log(
`keyframes with SPS/PPS: ${keyframesWithParamSets}, without: ${keyframesWithoutParamSets}`,
);
if (frames.length === 0) {
console.error("FAIL: no frames emitted while input still open (deadlock)");
close();
process.exit(1);
}
if (bareParamSetFrames > 0 || framesWithoutSlice > 0) {
console.error(
"FAIL: demux emitted bare parameter-set frames (must group into access units)",
);
close();
process.exit(1);
}
if (frames.some((f) => f.duration !== 1 || f.timeBase.den !== 30)) {
console.error(
"FAIL: frame duration/timeBase not 1/30 (RTP timestamp advance wrong)",
);
close();
process.exit(1);
}
input.end();
await new Promise((r) => setTimeout(r, 300));
close();
console.log("PASS: live stream demux works + access units grouped correctly");
process.exit(0);
@@ -0,0 +1,58 @@
// Phase 2 E2E: full pipeline prepareStream → demux → frame stream.
// Run: npx tsx tests/golive-pipeline-e2e.ts
import { demux } from "../src/goLive/Demuxer.js";
import { Encoders } from "../src/goLive/Encoders.js";
import { prepareStream } from "../src/goLive/prepareStream.js";
import { normalizeVideoCodec } from "../src/goLive/utils.js";
// Use a real ffmpeg-generated video file as input (from sample generation).
const input = process.argv[2] ?? "/tmp/sample.h264";
const prepared = prepareStream(input, {
encoder: Encoders.software({ x264: { preset: "superfast" } }),
width: 640,
height: 360,
frameRate: 25,
bitrateVideo: 500,
bitrateVideoMax: 800,
includeAudio: false,
videoCodec: normalizeVideoCodec("H264"),
});
console.log(
"prepareStream ok, videoCodec:",
prepared.videoCodec,
"size:",
prepared.width,
"x",
prepared.height,
);
const { video, close } = await demux(prepared.output, { format: "h264" });
console.log("demux video:", video?.codecName, video?.width, "x", video?.height);
let frames = 0;
let keyframes = 0;
video.stream.on("data", (f: { keyframe?: boolean }) => {
frames++;
if (f.keyframe) keyframes++;
});
video.stream.on("end", () => {
console.log(`pipeline frames: ${frames} (${keyframes} keyframes)`);
close();
prepared.command.kill("SIGTERM");
process.exit(frames > 0 ? 0 : 1);
});
video.stream.on("error", (e: unknown) => {
console.error("pipeline error:", e);
close();
prepared.command.kill("SIGTERM");
process.exit(1);
});
setTimeout(() => {
console.log("timeout after 30s — killing");
close();
prepared.command.kill("SIGTERM");
process.exit(2);
}, 30000);
@@ -0,0 +1,78 @@
// Phase 2 E2E: demux → VideoStream → native packetizer chain (local pair).
// Run: npx tsx tests/golive-videostream-e2e.ts
import { createReadStream } from "node:fs";
import { demux } from "../src/goLive/Demuxer.js";
import { loadNative } from "../src/goLive/native.js";
import { VideoStream } from "../src/goLive/VideoStream.js";
async function main() {
const native = loadNative();
const { PeerConnection } = native;
const pcA = new PeerConnection({ iceServers: [] });
const pcB = new PeerConnection({ iceServers: [] });
pcA.onStateChange(() => {});
pcB.onStateChange(() => {});
// Both peers declare audio+video tracks (exact passing test-packetizer
// pattern — tracks trigger negotiation).
pcA.addTrack("0", "audio");
pcA.addTrack("1", "video");
pcB.addTrack("0", "audio");
const trackB = pcB.addTrack("1", "video");
if (!trackB) throw new Error("no track from addTrack");
const track = trackB;
// NOTE: setPacketizer is called AFTER connected (see below) — calling it
// before negotiation breaks the offer (libdatachannel negotiation state).
const offer = await pcA.createOffer();
console.log("T1 offer");
pcB.setRemoteDescription(offer, "offer");
const answer = await pcB.createAnswer(offer);
console.log("T2 answer");
pcA.setRemoteDescription(answer, "answer");
await new Promise((r) => setTimeout(r, 1500));
console.log("T3 states:", pcA.state(), "/", pcB.state());
// Discord-style SSRC/payload: H264 101 @ 90kHz, playout ext id 5
track.setPacketizer("h264", 0x1234, 101, 90000, 5, 0, 10);
const { video, close } = await demux(createReadStream("/tmp/sample.h264"), {
format: "h264",
});
console.log("video stream:", video.codecName, video.width, "x", video.height);
const conn = {
sendVideoFrame: (frame: Buffer, frametime: number) => {
track.sendFrame(frame);
track.addTimestamp(Math.round((frametime * 90000) / 1000));
},
} as unknown as { sendVideoFrame(frame: Buffer, frametime: number): void };
const vStream = new VideoStream(conn as never);
let sent = 0;
const origSend = conn.sendVideoFrame;
conn.sendVideoFrame = (frame: Buffer, frametime: number) => {
sent++;
origSend(frame, frametime);
};
video.stream.pipe(vStream);
await new Promise((r) => setTimeout(r, 4000));
console.log(`sent ${sent} frames via VideoStream; B state=${pcB.state()}`);
const ok = sent > 0 && pcB.state() === "connected";
close();
pcA.close();
pcB.close();
process.exit(ok ? 0 : 1);
}
main().catch((e) => {
console.error("E2E failed:", e);
process.exit(1);
});
@@ -0,0 +1,105 @@
// ═══════════════════════════════════════════════════════════════════════════
// llmClient chunk extraction — reasoning_content fallback (pure, no network)
// ═══════════════════════════════════════════════════════════════════════════
// Regression: 9router "multimodal" combo routed to cloudflare gemma-4-26b
// which streams ALL output in delta.reasoning_content with content:"" — the
// old extractor returned empty text → llmVision reported "Vision API null
// response" → every image moderation batch fell back to text-only analysis
// (LLM kept writing "Meskipun analisis gambar gagal").
import { describe, expect, it } from "vitest";
import { extractChunkText } from "../src/modules/ai-moderation/llmClient.js";
describe("extractChunkText — streaming chunk text extraction", () => {
it("reads delta.content (standard OpenAI streaming)", () => {
expect(
extractChunkText({
choices: [{ delta: { content: "halo" }, finish_reason: null }],
}),
).toBe("halo");
});
it("falls back to delta.reasoning_content when content is empty — reasoning-only models (cloudflare gemma)", () => {
// Exact shape seen from 9router → cloudflare-ai/@cf/google/gemma-4-26b:
// {"choices":[{"delta":{"content":"","reasoning_content":"Task","role":"assistant"},"finish_reason":null,...}]}
expect(
extractChunkText({
choices: [
{
delta: { content: "", reasoning_content: "Task" },
finish_reason: null,
},
],
}),
).toBe("Task");
});
it('falls back to delta.reasoning — mimo via 9router streams reasoning there with content:""', () => {
// Exact shape seen from 9router → mimo-v2.5-free (2026-08-11):
// {"choices":[{"delta":{"content":"","reasoning":"The user wants a","role":"assistant"},"finish_reason":null,...}]}
expect(
extractChunkText({
choices: [
{
delta: { content: "", reasoning: "The user wants a" },
finish_reason: null,
},
],
}),
).toBe("The user wants a");
});
it("joins delta.reasoning_details[].text when present", () => {
expect(
extractChunkText({
choices: [
{
delta: {
content: "",
reasoning: "",
reasoning_details: [
{ type: "reasoning.text", text: " detailed", index: 0 },
{ type: "reasoning.text", text: " description", index: 1 },
],
},
finish_reason: null,
},
],
}),
).toBe(" detailed description");
});
it("prefers content over reasoning when both present (deepseek-style final answer)", () => {
expect(
extractChunkText({
choices: [
{
delta: { content: "jawaban akhir", reasoning_content: "pikiran" },
finish_reason: null,
},
],
}),
).toBe("jawaban akhir");
});
it("handles Anthropic-style message.content", () => {
expect(extractChunkText({ message: { content: "via message" } })).toBe(
"via message",
);
});
it("handles top-level content / response fields (local LLM proxies)", () => {
expect(extractChunkText({ content: "top-level" })).toBe("top-level");
expect(extractChunkText({ response: "via response" })).toBe("via response");
});
it("returns empty string for null/undefined/empty chunks", () => {
expect(extractChunkText(null)).toBe("");
expect(extractChunkText(undefined)).toBe("");
expect(extractChunkText({})).toBe("");
expect(
extractChunkText({
choices: [{ delta: { content: "", reasoning_content: null } }],
}),
).toBe("");
});
});
@@ -0,0 +1,77 @@
// ═══════════════════════════════════════════════════════════════════════════
// Nickname-only enforcement — offensive username flag handling (pure, no DB)
// ═══════════════════════════════════════════════════════════════════════════
import { describe, expect, it } from "vitest";
import {
isNicknameOnlyViolation,
parseModerationFlags,
} from "../src/modules/ai-moderation/autoDeleteEligibility.js";
import type {
AnalysisResult,
MessageRecord,
} from "../src/modules/message-capture/types.js";
function msg(flagsJson: string | null): MessageRecord {
return {
id: "m1",
guild_id: "g1",
channel_id: "c1",
thread_id: null,
user_id: "u1",
username: "user1",
avatar_url: null,
content: "halo semua",
edited_content: null,
created_at: Date.now(),
edited_at: null,
deleted_at: null,
type: "text",
is_reply: null,
is_forward: null,
is_crosspost: null,
reference_message_id: null,
reference_channel_id: null,
reference_guild_id: null,
metadata: null,
ai_moderation_flags: flagsJson,
};
}
describe("parseModerationFlags", () => {
it("parses JSON array from stored column", () => {
expect(parseModerationFlags(msg('["offensive_username","sara"]'))).toEqual([
"offensive_username",
"sara",
]);
});
it("returns [] for null / malformed values", () => {
expect(parseModerationFlags(msg(null))).toEqual([]);
expect(parseModerationFlags(msg("not-json"))).toEqual([]);
});
it("prefers structured analysisResult flags", () => {
const result = { flags: ["vulgar_language"] } as AnalysisResult;
expect(parseModerationFlags(msg('["old_flag"]'), result)).toEqual([
"vulgar_language",
]);
});
});
describe("isNicknameOnlyViolation", () => {
it("true when the ONLY flag is offensive_username", () => {
expect(isNicknameOnlyViolation(msg('["offensive_username"]'))).toBe(true);
});
it("false when other flags ride along (message itself violated)", () => {
expect(isNicknameOnlyViolation(msg('["offensive_username","sara"]'))).toBe(
false,
);
expect(isNicknameOnlyViolation(msg('["harassment"]'))).toBe(false);
});
it("false when no flags at all", () => {
expect(isNicknameOnlyViolation(msg(null))).toBe(false);
expect(isNicknameOnlyViolation(msg("[]"))).toBe(false);
});
});
@@ -1,6 +1,8 @@
// ═══════════════════════════════════════════════════════════════════════════════
// 1. AppError Hierarchy
// ═══════════════════════════════════════════════════════════════════════════════
import { afterEach, describe, expect, it, vi } from "vitest";
import {
AppError,
ConfigError,
@@ -9,7 +11,6 @@ import {
UnauthorizedError,
ValidationError,
} from "../src/shared/errors/index.js";
import { afterEach, describe, expect, it, vi } from "vitest";
describe("AppError subclasses", () => {
it("AppError carries code, statusCode, and details", () => {
@@ -1,16 +1,21 @@
// ═══════════════════════════════════════════════════════════════════════════════
// Screen share input resolution tests
//
// Verifies the decision logic of getDirectScreenInput:
// - merged progressive URL → returned directly
// - video+audio DASH pair → local ffmpeg merge (Readable)
// - neither → rejection
// getDirectScreenInput now streams the merged video+audio media straight from
// yt-dlp stdout (`-o -`) — same auth-handling mechanism as resolveMediaUrl for
// music. There is no manual URL fetch or local ffmpeg merge anymore.
//
// Both yt-dlp and ffmpeg are faked via PATH shim scripts so the test does not
// hit the network or need real binaries.
// yt-dlp is faked via a PATH shim script so the test does not hit the network
// or need real binaries.
// ═══════════════════════════════════════════════════════════════════════════════
import { chmodSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import {
chmodSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Readable } from "node:stream";
@@ -25,28 +30,21 @@ const realPath = process.env.PATH;
beforeAll(() => {
fakeBinDir = mkdtempSync(join(tmpdir(), "gmw-fake-bins-"));
// Fake yt-dlp: prints the JSON file named in GMW_FAKE_YTDLP_JSON.
// If the file is missing → exits 1 (mimics yt-dlp failure).
// Fake yt-dlp: streams a few bytes to stdout (like `yt-dlp -o -` does).
// Modes (env):
// GMW_FAKE_YTDLP_FAIL=1 → stderr 403 + exit 8 WITHOUT stdout bytes
// (mimics a download rejected by YouTube).
const ytShim = `#!/usr/bin/env bash
if [ -n "$GMW_FAKE_YTDLP_JSON" ] && [ -f "$GMW_FAKE_YTDLP_JSON" ]; then
cat "$GMW_FAKE_YTDLP_JSON"
exit 0
if [ "$GMW_FAKE_YTDLP_FAIL" = "1" ]; then
echo "ERROR: [youtube] ...: 403 Forbidden (access denied)" >&2
exit 8
fi
echo "yt-dlp: fake JSON missing" >&2
exit 1
`;
writeFileSync(join(fakeBinDir, "yt-dlp"), ytShim);
chmodSync(join(fakeBinDir, "yt-dlp"), 0o755);
// Fake ffmpeg: writes a small nut-ish payload to stdout so the returned
// Readable actually emits data (the merge path in mergeScreenStreams).
const ffShim = `#!/usr/bin/env bash
# Fake ffmpeg ignore args, emit a few bytes so consumers see a live stream.
# Fake yt-dlp ignore args, emit a few bytes so consumers see a live stream.
head -c 4096 /dev/urandom
exit 0
`;
writeFileSync(join(fakeBinDir, "ffmpeg"), ffShim);
chmodSync(join(fakeBinDir, "ffmpeg"), 0o755);
writeFileSync(join(fakeBinDir, "yt-dlp"), ytShim);
chmodSync(join(fakeBinDir, "yt-dlp"), 0o755);
process.env.PATH = `${fakeBinDir}:${process.env.PATH}`;
});
@@ -59,84 +57,76 @@ afterAll(() => {
});
// ─── helpers ───────────────────────────────────────────────────────────────────
function writeFakeJson(payload: Record<string, unknown>): string {
const p = join(
tmpdir(),
`gmw-fake-${process.pid}-${Date.now()}-${Math.random().toString(36).slice(2)}.json`,
);
writeFileSync(p, JSON.stringify(payload));
return p;
}
function dashPairInfo(videoUrl: string, audioUrl: string) {
return {
url: null,
acodec: "none", // top-level is not a single merged format
vcodec: "av01",
requested_formats: [
{
format_id: "136",
vcodec: "avc1.4d401f",
acodec: "none",
url: videoUrl,
},
{ format_id: "140", vcodec: "none", acodec: "mp4a.40.2", url: audioUrl },
],
};
function consumeStream(stream: Readable): Promise<string> {
return new Promise<string>((resolve) => {
let got = 0;
stream.on("data", (chunk: Buffer) => {
got += chunk.length;
});
stream.on("error", () => resolve(`error-after-${got}B`));
stream.on("end", () => resolve(`end-after-${got}B`));
stream.resume();
});
}
// ─── tests ─────────────────────────────────────────────────────────────────────
describe("getDirectScreenInput", () => {
it("returns the single merged progressive URL when the info has one", async () => {
process.env.GMW_FAKE_YTDLP_JSON = writeFakeJson({
url: "https://cdn.example/progressive.mp4",
acodec: "mp4a.40.2",
vcodec: "avc1",
});
const result = await getDirectScreenInput("https://youtu.be/abc");
expect(result).toBe("https://cdn.example/progressive.mp4");
});
it("returns a live Readable when a video+audio DASH pair must be merged", async () => {
process.env.GMW_FAKE_YTDLP_JSON = writeFakeJson(
dashPairInfo(
"https://cdn.example/video.mp4",
"https://cdn.example/audio.m4a",
),
);
it("returns a live Readable and streams media bytes from yt-dlp stdout", async () => {
const result = await getDirectScreenInput("https://youtu.be/abc");
expect(Readable.isReadable(result)).toBe(true);
// The fake ffmpeg emits bytes; collect a chunk to prove the stream flows.
const bytes = await new Promise<number>((resolve, reject) => {
const stream = result as Readable;
let got = 0;
stream.on("data", (chunk: Buffer) => {
got += chunk.length;
});
stream.on("error", reject);
stream.on("end", () => resolve(got));
stream.resume();
});
expect(bytes).toBeGreaterThan(0);
const outcome = await consumeStream(result);
// The fake yt-dlp emits 4096 bytes → the stream must deliver them.
expect(outcome).toMatch(/^(error|end)-after-[1-9]\d*B$/);
});
it("rejects when yt-dlp returns neither a merged URL nor a format pair", async () => {
process.env.GMW_FAKE_YTDLP_JSON = writeFakeJson({
url: null,
acodec: "none",
vcodec: "none",
requested_formats: [],
});
await expect(getDirectScreenInput("https://youtu.be/abc")).rejects.toThrow(
/neither a merged progressive URL nor a video\+audio/,
);
it("destroys the stream with an error when yt-dlp fails before producing data (transient 403)", async () => {
// Simulate the production failure: yt-dlp's downloader hits a transient
// YouTube 403 and exits non-zero WITHOUT emitting a single byte. The
// returned Readable must terminate with zero bytes (error OR end) so the
// controller's resolveInputWithRetry retries with a fresh run instead of
// streaming a silent black tile.
process.env.GMW_FAKE_YTDLP_FAIL = "1";
try {
const result = await getDirectScreenInput("https://youtu.be/abc");
expect(Readable.isReadable(result)).toBe(true);
const outcome = await consumeStream(result);
expect(outcome).toMatch(/^(error|end)-after-0B$/);
} finally {
delete process.env.GMW_FAKE_YTDLP_FAIL;
}
});
it("rejects when yt-dlp exits non-zero", async () => {
process.env.GMW_FAKE_YTDLP_JSON = "/nonexistent/gmw-fake.json";
await expect(getDirectScreenInput("https://youtu.be/abc")).rejects.toThrow(
/screen input resolution exited with code 1/,
it("passes -o - (stdout streaming) and a temp dir to yt-dlp", async () => {
const argsDump = join(
tmpdir(),
`gmw-ytargs-${process.pid}-${Date.now()}.txt`,
);
process.env.GMW_FAKE_YTDLP_DUMP_ARGS = argsDump;
// Augment the fake to dump its argv.
const shim = `#!/usr/bin/env bash
printf '%s\\n' "$*" >> "$GMW_FAKE_YTDLP_DUMP_ARGS"
head -c 4096 /dev/urandom
exit 0
`;
const realPath2 = process.env.PATH;
const dir = fakeBinDir as unknown as string;
const existing = join(dir, "yt-dlp");
// Overwrite with the argv-dumping variant.
writeFileSync(existing, shim);
chmodSync(existing, 0o755);
try {
const result = await getDirectScreenInput("https://youtu.be/abc");
await consumeStream(result);
await new Promise((r) => setTimeout(r, 100));
const args = readFileSync(argsDump, "utf8").trim();
expect(args).toContain("-o -");
expect(args).toMatch(/gmw-ytdlp-/);
} finally {
delete process.env.GMW_FAKE_YTDLP_DUMP_ARGS;
rmSync(argsDump, { force: true });
process.env.PATH = realPath2;
}
});
});
@@ -0,0 +1,91 @@
// ═══════════════════════════════════════════════════════════════════════════
// Term glossary — pure extraction/formatting tests (no DB, Redis, or network)
// ═══════════════════════════════════════════════════════════════════════════
import { describe, expect, it } from "vitest";
import {
extractGlossaryTerms,
formatTermGlossary,
} from "../src/modules/ai-moderation/termGlossary.js";
describe("extractGlossaryTerms — filters out words the LLM already knows", () => {
it("returns [] for common conversational Indonesian", () => {
const terms = extractGlossaryTerms(
["anjay mabar yuk gaskeun gua gas", "iya bener banget sih"],
{ maxTerms: 6 },
);
expect(terms).toEqual([]);
});
it("extracts uncommon/foreign-looking words and skips stopwords + brands", () => {
const terms = extractGlossaryTerms(
[
"tadi gua baca soal tempeh di discord",
"kayaknya istilahnya shirkmaxxing deh",
],
{ maxTerms: 6 },
);
// "tempeh" and "shirkmaxxing" are candidates; "discord"/"istilahnya" are not
expect(terms).toContain("tempeh");
expect(terms).toContain("shirkmaxxing");
expect(terms).not.toContain("discord");
expect(terms).not.toContain("istilahnya");
});
it("strips URLs, mentions, and custom emoji before extracting", () => {
const terms = extractGlossaryTerms(
["cek https://example.com/foo <@123456> <:hadeh:987> kafircel"],
{ maxTerms: 6 },
);
expect(terms).toContain("kafircel");
expect(terms.some((t) => /example|hadeh|123/.test(t))).toBe(false);
});
it("extracts quoted phrases as a single term", () => {
const terms = extractGlossaryTerms(['dia bilang "kostum hewan" itu aneh'], {
maxTerms: 6,
});
expect(terms).toContain("kostum hewan");
});
it("skips repeated-char noise like wkwkwk and aaaaa", () => {
const terms = extractGlossaryTerms(["wkwkwkwk aaaaa xixixi"], {
maxTerms: 6,
});
expect(terms).toEqual([]);
});
it("respects maxTerms and prioritizes proper nouns", () => {
const terms = extractGlossaryTerms(
["aku suka Xenogears sama Chrono Cross terus Yakuza"],
{ maxTerms: 2 },
);
expect(terms.length).toBeLessThanOrEqual(2);
expect(terms[0]).toBe("Xenogears");
});
});
describe("formatTermGlossary — XML block shape", () => {
it("returns '' for an empty map", () => {
expect(formatTermGlossary(new Map())).toBe("");
});
it("wraps definitions in <term_glossary> with escaped attributes/content", () => {
const block = formatTermGlossary(
new Map([
[
"kafircel",
{
term: "kafircel",
definition: "sebutan <memes> untuk & orang",
sourceUrl: "https://id.wikipedia.org/wiki/Mem",
},
],
]),
);
expect(block).toContain("<term_glossary>");
expect(block).toContain('<term word="kafircel"');
expect(block).toContain("&lt;memes&gt;");
expect(block).toContain("&amp;");
expect(block).toContain("</term_glossary>");
});
});
@@ -0,0 +1,52 @@
// ═══════════════════════════════════════════════════════════════════════════
// isNoImageSeenText — vision outputs that claim "no image" must not be cached
// ═══════════════════════════════════════════════════════════════════════════
// Regression (2026-08-11): the vision model sometimes answered "Maaf, saya
// tidak melihat gambar apapun yang terlampir..." and that text was cached as
// a VALID vision_llm result. Every later analysis of the same image (same
// hash / phash) then hit the poisoned cache and the moderation LLM wrote
// "lampiran yang gagal terbaca" — image analysis seemed permanently broken
// even though 9router was responding fine.
import { describe, expect, it } from "vitest";
import { isNoImageSeenText } from "../src/modules/ai-moderation/visionAnalyzer.js";
describe("isNoImageSeenText — poisoned vision output detection", () => {
it("detects the exact poisoned strings seen in production", () => {
expect(
isNoImageSeenText(
"Maaf, saya tidak melihat gambar apapun yang terlampir dalam pesan Anda. Mohon kirimkan ulang gambarnya agar saya bisa mendeskripsikannya secara objektif dan spesifik.",
),
).toBe(true);
expect(
isNoImageSeenText(
"Tidak ada gambar yang terlampir. Tidak bisa deskripsi tanpa input visual.",
),
).toBe(true);
});
it("detects English variants", () => {
expect(isNoImageSeenText("I cannot see any image in this message")).toBe(
true,
);
expect(isNoImageSeenText("No image provided")).toBe(true);
expect(isNoImageSeenText("there is no image attached")).toBe(true);
expect(isNoImageSeenText("I don't see an image")).toBe(true);
});
it("does NOT flag legitimate image descriptions", () => {
expect(
isNoImageSeenText(
"Gambar ini menampilkan dua panel komik, seorang gadis berambut biru tersipu saat dipuji.",
),
).toBe(false);
expect(
isNoImageSeenText("Ini adalah screenshot dari sebuah website rekrutmen."),
).toBe(false);
expect(isNoImageSeenText("Emoji menampilkan ekspresi wajah tertawa.")).toBe(
false,
);
expect(isNoImageSeenText(null)).toBe(false);
expect(isNoImageSeenText(undefined)).toBe(false);
expect(isNoImageSeenText("")).toBe(false);
});
});
+14
View File
@@ -56,6 +56,13 @@
/* Ring / focus outline for shadcn outline-ring utility */
--color-ring: var(--color-primary);
/* Base-ui / shadcn primitives */
--color-popover: oklch(0.99 0.005 250 / 0.95);
--color-popover-foreground: var(--color-text-primary);
--color-input: oklch(0.55 0.02 250 / 0.35);
--color-secondary: oklch(0.92 0.01 250 / 0.6);
--color-secondary-foreground: var(--color-text-primary);
/* Sidebar (shadcn) */
--color-sidebar: oklch(1 0 0 / 0.6);
--color-sidebar-foreground: var(--color-text-primary);
@@ -113,6 +120,13 @@
--color-accent-foreground: var(--color-primary-foreground);
--color-ring: var(--color-primary);
/* Base-ui / shadcn primitives */
--color-popover: oklch(0.13 0.02 245 / 0.96);
--color-popover-foreground: var(--color-text-primary);
--color-input: oklch(1 0 0 / 0.18);
--color-secondary: oklch(0.2 0.02 245 / 0.7);
--color-secondary-foreground: var(--color-text-primary);
/* Sidebar (shadcn) */
--color-sidebar: oklch(0.11 0.02 245 / 0.55);
--color-sidebar-foreground: var(--color-text-primary);
@@ -63,7 +63,7 @@ function SelectContent({
sideOffset = 4,
align = "center",
alignOffset = 0,
alignItemWithTrigger = true,
alignItemWithTrigger = false,
...props
}: SelectPrimitive.Popup.Props &
Pick<
@@ -83,7 +83,7 @@ function SelectContent({
<SelectPrimitive.Popup
data-slot="select-content"
data-align-trigger={alignItemWithTrigger}
className={cn("relative isolate z-50 max-h-(--available-height) w-(--anchor-width) min-w-36 origin-(--transform-origin) overflow-x-hidden overflow-y-auto rounded-lg bg-popover text-popover-foreground shadow-md ring-1 ring-foreground/10 duration-100 data-[align-trigger=true]:animate-none data-[side=bottom]:slide-in-from-top-2 data-[side=inline-end]:slide-in-from-left-2 data-[side=inline-start]:slide-in-from-right-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 data-open:animate-in data-open:fade-in-0 data-open:zoom-in-95 data-closed:animate-out data-closed:fade-out-0 data-closed:zoom-out-95", className )}
className={cn("relative isolate z-50 max-h-(--available-height) w-fit min-w-40 max-w-(--available-width) origin-(--transform-origin) overflow-x-hidden overflow-y-auto rounded-lg bg-popover text-popover-foreground shadow-md ring-1 ring-foreground/10 duration-100 data-[align-trigger=true]:animate-none data-[side=bottom]:slide-in-from-top-2 data-[side=inline-end]:slide-in-from-left-2 data-[side=inline-start]:slide-in-from-right-2 data-[side=left]:slide-in-from-right-2 data-[side=right]:slide-in-from-left-2 data-[side=top]:slide-in-from-bottom-2 data-open:animate-in data-open:fade-in-0 data-open:zoom-in-95 data-closed:animate-out data-closed:fade-out-0 data-closed:zoom-out-95", className )}
{...props}
>
<SelectScrollUpButton />
@@ -122,7 +122,7 @@ function SelectItem({
)}
{...props}
>
<SelectPrimitive.ItemText className="flex flex-1 shrink-0 gap-2 whitespace-nowrap">
<SelectPrimitive.ItemText className="flex flex-1 shrink-0 gap-2 whitespace-normal break-words">
{children}
</SelectPrimitive.ItemText>
<SelectPrimitive.ItemIndicator
@@ -71,6 +71,9 @@ export interface ChannelRef {
channelName?: string | null;
threadId?: string | null;
threadName?: string | null;
/** Channel topic (captured in gateway metadata.channel.topic). */
topic?: string | null;
nsfw?: boolean;
}
export interface ReferenceInfo {