fix: render fMP4/CMAF HLS by interleaving fragment audio and video#1
Merged
Conversation
HLS with fMP4/CMAF segments loaded a single video frame and played no audio. The fMP4 demuxer emitted a fragment track-major — every video sample of a moof, then every audio sample — and with delivery pacing on (always, for HLS) the paced video burst spans the whole fragment duration in wall time, so the fragment audio behind it arrives a full fragment late. The decoder trims that audio as stale and its audio-gated presentation clock never starts: one frame, silence. Consume a moof mdat as a decode-time-ordered merge across its trun runs instead, interleaving the tracks at frame granularity the way the MPEG-TS demuxer's output naturally is. Decode order within each run is preserved; single-track fragments (split-stream fMP4 legs) are delivered byte-for-byte as before. Verified against ffmpeg-generated CMAF (H.264+AAC, 2s fragments): worst delivery-order PTS gap between tracks drops from ~2.1s (the fragment duration) to ~190ms (B-frame reorder depth), same access-unit counts and timestamps; a video-only fMP4 stream demuxes identically before and after. Rebuilt basis_media_native.dll (win-x64) and libbasis_media_native.so (arm64-v8a).
basis_avcc_to_annexb is bounds-checked, so an output larger than the buffer fails the conversion rather than overrunning — but with a 1-3 byte NAL length prefix, an access unit made of many small NALs could outgrow the fixed 64-byte headroom (each NAL trades its length prefix for a 4-byte start code) and be dropped. Size the buffer from the worst-case growth instead: a NAL is at least nal_len_size + 1 bytes, so growth is bounded by (4 - nal_len_size) * (size / (nal_len_size + 1)). For the common 4-byte prefix this stays exactly size + 64. Rebuilt basis_media_native.dll (win-x64) and libbasis_media_native.so (arm64-v8a).
This was referenced Jul 7, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
HLS with fragmented-MP4 / CMAF segments (
EXT-X-MAP:init.mp4+.m4s) — which includes Twitch's low-latency streams — loaded a single video frame and then stalled with no audio. TS-segment HLS was unaffected.The demux itself was fine; the problem was emission order. The fMP4 demuxer consumed each moof/mdat track-major: every video sample of the fragment, then every audio sample. With delivery pacing on (always, for HLS) the paced video burst spans the whole fragment duration in wall time, so the fragment's audio always arrived a full fragment late. The decoder trims audio that stale, and its audio-gated presentation clock never got a buffer to start from — one frame, silence.
consume_mdatnow emits the moof's trun runs as a decode-time-ordered merge, interleaving the tracks at frame granularity — the same shape the MPEG-TS demuxer's output naturally has. Decode order within each track is preserved, and single-track fragments (split-stream fMP4 legs) are delivered exactly as before.The second commit sizes the Annex B conversion buffer from worst-case NAL growth (a 1–3-byte NAL length prefix with many small NALs could exceed the previous fixed 64-byte headroom and drop the access unit — the conversion helper is bounds-checked, so this was a dropped AU, never an overrun). For the near-universal 4-byte prefix the allocation is unchanged.
Both binaries are rebuilt and committed (win-x64 DLL, arm64-v8a
.so).Verification
basis_mp4_rundirectly and measuring the worst delivery-order PTS gap between tracks: ~2.1 s before (the fragment duration) → ~190 ms after (B-frame reorder depth), with identical access-unit counts and timestamps. A video-only fMP4 stream demuxes identically before and after.Scope note
HLS playback remains single-rendition: masters that put audio in a separate
EXT-X-MEDIArendition group (rather than muxing it into the variant) play the selected variant only. This PR fixes muxed CMAF, which is what Twitch low-latency and ffmpeg-style CMAF origins serve.Re-created from BasisVR/Basis#924 (same commits, paths rebased onto this repo), now that this repository is the media player's home.