Root cause: prepareStream's ffmpeg consumed a YouTube VOD at download/CPU
speed (~10x real-time), so the demuxer buffered a huge frame backlog.
The sender paces at 30fps but always emitted the OLDEST buffered frames, so
the viewer saw frozen/laggy video while audio (tiny, jitter-buffer
recoverable) stayed smooth. That is exactly the 'video stuck, voice normal'
symptom reported live.
Fix: propagate vPipe backpressure UP to the demuxer's ffmpeg stdout — when
the sender can't keep up, pause the source, which stalls the demuxer and
back-pressures the encoder, pinning the whole pipeline to 1x. Also add a
realtime (-re) option for file/URL inputs (no-op for the streaming path,
which is what screen share uses).
Verified: 10s test clip encodes in 1.8s without -re vs 9.5s with it; tsc --noEmit clean.
The demuxer resolved as soon as the VIDEO init line arrived on ffmpeg stderr.
With live NUT input the audio init line ('Stream #0:1: Audio: opus') lands in a
LATER stderr chunk (NUT info-stream packets are read incrementally from the
pipe), so `return { audio: aInfo }` captured undefined → playStream skipped
AudioStream → zero audio RTP on the audio SSRC → Discord showed a static
GoLive tile even though the NUT carried opus audio.
Fix:
- wait for BOTH video and audio init lines (when audio is expected) before
resolving demux metadata, with a 3s timeout fallback
- default aInfo to opus/48kHz when withAudio instead of undefined, so the
audio stream is always exposed even if the metadata line races the return
Screen share showed a single frozen frame: the GoLive pipeline sent video
only (-"-an", h264 muxer cannot carry audio) so the audio SSRC never
transmitted and Discord kept the stream in thumbnail state.
- prepareStream: mux NUT when includeAudio (h264 muxer drops audio) and
return the actual container format
- Demuxer: support NUT input with a second output pipe (fd3) carrying
Ogg Opus; parse OGG pages into opus frames (20ms, 48kHz) emitted as
GoLiveFrames; fix metadata parsing that dropped the audio stream line
when it arrived in a later stderr chunk (early parsedMeta return)
- playStream: pipe audio.stream into AudioStream → RTP on the audio SSRC
- Encoders: -x264-params repeat-headers=1 → SPS/PPS inline before EVERY
IDR (NUT remux drops container extradata; also enables PLI recovery)
- screenShareController: includeAudio true
- tests: demuxerNut.test.ts — OGG parser unit test + real ffmpeg NUT
integration (video access units + parsed opus frames)
Demuxer emitted each AnnexB NAL as its own WebRTC frame (SPS/PPS/SEI
separate from slices) with a near-zero timestamp delta (duration=1 in a
1/90000 timebase → RTP +1/frame instead of +3000 @30fps). Discord's H264
receiver never receives a complete decodable access unit → black GoLive
tile despite frames flowing.
- Group NALs into access units: buffer param-set/SEI NALs, flush one
frame per slice with preceding parameter sets (AnnexB start codes kept
so the H264RtpPacketizer finds NAL boundaries).
- Timestamp each frame at the video frame rate: duration=1, timeBase
1/fps → BaseMediaStream frametime=1000/fps ms → RTP +clockRate/fps
(3000 @ 30fps/90kHz) and correct pacing.
- Thread explicit frameRate from playStream options (raw H264 has no
timing info; ffmpeg guesses 25fps on stderr).
- Strengthen golive-demux-live-e2e: validates every frame has a slice,
no bare param-set frames, keyframes carry SPS/PPS, timeBase 1/30.
ROOT CAUSE of empty GoLive tile (finally): prepareStream ran with
includeAudio: true + output -f h264. The h264 muxer cannot mux audio
('h264 muxer does not support any stream of type audio') → header write
fails -22 → stdout empty → Demuxer ffmpeg 'Invalid data found when
processing input' → 0 frames → black tile. Reproduced locally end-to-end
(13s backpressure delay + prepareStream + demux).
Fixes:
- screenShareController: includeAudio: false (video-only GoLive; demux
path never delivers audio anyway)
- Demuxer: pin input format -f h264 for stream inputs (raw AnnexB H264
has no magic header → auto-detect unreliable on delayed pipes)
- Streamer.signalStream: self_video: false — stop flipping on the bot's
camera in Discord (user request; screen share ≠ camera)
Verified: local repro now emits 644 frames 1280x720 (was 0); tsc/biome/
vitest all green.
Root cause of 'tile appears but content empty': demux() spooled the live
NUT/H264 input to a temp file and awaited stream 'finish' — but the merge
ffmpeg output never ends during playback, so demux deadlocked, no probe,
no transcode, 0 frames sent.
- Demuxer: pipe input straight into ffmpeg stdin (-i pipe:0), parse NAL
frames live from stdout; parse video metadata from ffmpeg stderr with a
1.5s race (fall back to H264 defaults). No spool, no await-end.
- screenShareController: pass width/height/frameRate (1280x720@30) to
playStream — matches the prepareStream encode settings, so setVideoAttributes
gets real dimensions even when ffmpeg can't report metadata on an open pipe.
- Add tests/golive-demux-live-e2e.ts: proves frames flow while input is
still open (regression test for the deadlock).
asepharyana