feat: VOD seek bar with multiplayer position sync and late-join catch-up#6
Merged
Conversation
Remote MP4 reaches this parser (HLS/CMAF segments, VOD URLs), so treat every stream-supplied field as hostile: - parse_traf: validate tfhd/tfdt/trun field reads against the box body, and reject a trun whose declared per-sample table does not fit in the box (or exceeds a sane per-fragment sample cap). Unchecked counts could previously drive reads past the buffered moof. - Box walkers: rewrite child-size checks in subtraction form (sz > len - off) so a crafted size near INT_MAX cannot overflow the signed addition and bypass the bound. - consume_mdat / consume_progressive: widen run positions to int64 and make the sample fit checks overflow-safe; cap progressive sample sizes at the box cap so a crafted stsz entry cannot drive a multi-gigabyte allocation. - Timestamps: convert ticks to microseconds via quotient/remainder (ticks_to_us) instead of value * 1000000 / timescale, which overflows at stream-controlled timescales. - ctts: honour the FullBox version - version 0 offsets are unsigned, version 1 signed. Previously all offsets were cast signed. - stz2: detect the compact sample-size table and surface a clear unsupported-format error instead of starting playback with the affected track silently missing. - stsd: bound the mp4a fixed-header reads to the sample entry size. - Annex B sizing: move the worst-case AVCC-to-Annex-B output size into a shared helper (basis_avcc_annexb_cap) and use it from both MP4 paths and RTMP. RTMP previously allocated dlen + 64, which undersizes streams using 1-3 byte NAL length prefixes and silently dropped the access unit.
Remove the never-read next_dts track field, update the fMP4-only header comment (the demuxer has handled progressive MP4 since the classic-table work), refresh the top-of-file assumptions to match, and break the nested '/*' in basis_bitstream.h that warned on every translation unit.
Progressive playback previously ignored edts/elst, so every track started at presentation time zero regardless of its edit list. A track delayed by an initial empty edit (common muxer output for A/V alignment) played that much early, permanently desynced from the other track. Parse mvhd for the movie timescale and each track's elst, then map media time onto the movie timeline at emission: initial empty edits become a presentation-time delay, and the first normal edit's media_time becomes the media-time origin (encoder priming / initial trim). Samples ahead of the origin keep a negative presentation time rather than being dropped, since video there can still carry reference frames; multi-segment edits beyond the first normal entry are out of scope for a linear walk. Fragmented MP4 is unaffected - its timeline comes from tfdt.
Delivery pacing gated on the presentation timestamp of whatever the demux thread submitted last. A video AU whose composition offset puts its pts past the pacing lead therefore slept the demux thread out beyond its own decode turn - and any earlier samples of the other track queued behind it on the same thread arrived late enough for the audio queue to trim them as stale. Streams reordering less than the ~400 ms lead masked this. Carry the decode timestamp through the video sink alongside pts and gate on it, while the decoder and caption store keep receiving pts. Demuxers without decode timestamps (TS, RTSP) pass pts for both, which is the old behaviour; RTMP forwards the FLV tag timestamp it already had. The progressive MP4 walk had the file-order variant of the same stall: a muxer that interleaves chunks coarsely (a chunk of video ahead of the matching audio) blocked the forward reader on paced video before it could reach the audio bytes at all. Samples whose file order runs ahead of their delivery turn now park in bounded per-track queues and deliver in decode-time order across tracks; past the 32 MB bound (or once the mdat has nothing more to read) the earliest parked sample delivers regardless, degrading to the previous behaviour instead of buffering without limit. The fragmented path keeps its decode-time merge and hands the merge key to the sink as the decode timestamp.
basis_media_native.dll (win-x64, MSVC 19.51) and libbasis_media_native.so (android arm64-v8a, NDK clang 18) built from this branch.
The load-identity plumbing around metadata had three gaps, all variants of 'which load does this async continuation belong to': - LoadGeneration advanced only in LoadUrl, so a stale resolver continuation could pass its staleness guard after a LoadLocalPath, a direct LoadSource or a CPU Source assignment replaced the source. It now advances on every source replacement, whichever entry point. - The LoadUrl metadata seed was a bool that stayed armed when routing ended without a load (no resolver installed, or a resolver that errored before LoadSource), so the next unrelated load inherited the abandoned URL as its metadata origin. The seed is now keyed to the generation that armed it and is retired on the no-resolver path and by every non-continuation entry point. - LoadSource wrote the resolved metadata origin back into the caller's BasisMediaSource, where it went stale if the caller mutated the descriptor's Uri and loaded it again. The origin is now retained player-side and only trusted while the same instance reloads with an unchanged Uri; the caller's descriptor is no longer touched. Also: Metadata now returns a snapshot (and OnMetadataChanged delivers one), so external code can't mutate the player's live instance past the change event - ApplyMetadata is the mutation path. Assigning a CPU Source clears the previous media's metadata (event fires with null) instead of leaving it reported as current.
Adds basis_media_get_duration_us: 0 while unknown and for live sources, so a non-zero duration doubles as the 'this source has a seekable timeline' signal the managed layer and networking gate on. Demuxers report through a new optional on_duration sink callback: - Progressive MP4: the mvhd duration (now parsed alongside the movie timescale; the all-ones unknown sentinel is ignored), falling back to the longest track's stts total when mvhd carries none. Reported only when the classic tables are walkable - fragmented streams stay at 0 and get their duration from whatever layer knows the timeline. - HLS VOD (EXT-X-ENDLIST): the summed segment EXTINF durations, exposed as basis_hls_duration_ms. A playlist beyond the internal segment cap reports the truncated total, matching what actually plays.
towneh
force-pushed
the
feat/vod-seek-sync
branch
5 times, most recently
from
22ae313 to
5038129
Compare
…LS VOD basis_media_seek_us(engine, target_us): accepted on sources with a seekable timeline (duration > 0), asynchronous, lands on the preceding keyframe / segment boundary. Progressive MP4 over HTTP: the byte source gains a reposition hook - basis_win_http_reseek swaps the response for a ranged GET on the same connection (206 required; a server that ignores Range fails the call instead of silently restarting at byte 0). The engine posts seek requests through a new optional take_seek sink callback, one per demux leg so a split source repositions both; the demuxer maps the target to each track's table cursor (video backs up to the preceding stss sync sample), refetches at the earliest needed byte, and drops parked samples from before the jump. The paced read-ahead ring parks its reader around the refetch and stays alive at EOF so a backward seek after the file has fully buffered still works. HLS VOD with TS segments: the segment list (uri + duration) is retained at open; a seek rebuilds the fetch queue from the segment containing the target and flushes the stitched ring - the TS demuxer resyncs on the 0x47 sync byte, and the decoder's existing >1s hard-resync re-anchors the paced clock. fMP4-segment VOD is excluded (a mid-box ring flush can't be resynchronised) and therefore reports no duration: a non-zero duration is the managed layer's seekability signal, so the progressive path likewise only reports one when its source can actually reposition. Live sources are untouched: no pacing, no reseek hook, seek requests rejected at the ABI.
… native engine Duration consults the native engine's timeline when no CPU seekable source is active (0 stays TimeSpan.Zero, preserving the Duration-gated behaviour everywhere). Seek dispatches to the native asynchronous seek instead of throwing on the native path - it still throws NotSupportedException when the source has no seekable timeline - and resets the audio sync anchor so the scheduled-audio ring re-anchors on the post-seek timeline; OnSeekCompleted reports the requested target (the native landing position is the preceding keyframe). A deferred BasisMediaSource.StartPosition now applies on the native path at ready, best-effort: a live source skips silently, since a late joiner may sync a position into a stream that turns out to have no timeline. Old native libraries without the new exports read as duration 0, which is exactly the pre-feature behaviour.
The networking layer already carried everything needed - FullState positionTicks, the Seek message, drift correction, StartPosition on the late-join apply - all gated on Duration > 0, which was permanently zero for native sources. With the engine now reporting a timeline and honouring absolute seeks, two gaps remained: - Owner position heartbeat: a 9-byte latest-wins ping (Sequenced, like the framework's other position streams) every 5s (inspector-tunable, 0 disables) while playing seekable media, so passive clients re-converge between state events through the existing drift correction. Drift-only: it never starts or pauses playback. - Resolved page URLs skipped the owner's position/pause snapshot entirely (resolution is async, and the old backend couldn't seek anyway). The snapshot is now stashed with the load and applied on the resolved source's OnReady, aged by the local resolve time when the owner was playing - so a late joiner to a YouTube VOD lands near the owner's position instead of at zero, and the heartbeat refines the residual.
…ers panel Playback group gains a timeline scrubber beneath the transport buttons, shown only when the selected player reports a seekable timeline (Duration > 0). The slider API has no drag events, so a seek fires once the handle rests for 0.35s; the per-tick position refresh leaves the knob alone while a drag is pending. Seeks route through the networking component when present (ownership + broadcast), else straight to the player. The Status group now shows elapsed/total time next to the state word (ticking the repaint gate once per second, only while a timeline is showing) and the current metadata - title and uploader - refreshed via OnMetadataChanged. Player-supplied text is <noparse>-wrapped like the existing error strings.
basis_media_get_position_us returned the PTS of the most recently DECODED frame. The demuxer keeps feeding the decoder while presentation is paused or stopped (delivery pacing runs on its own wall anchor), so a stopped player's position kept counting up - visible now that the panel surfaces a time readout and a seek bar. Track the last presented PTS in a field that survives the presenter's resync sentinel resets and report that instead, falling back to the decode-side value before the first frame shows (start-up, audio-only) so early consumers still see the clock move. This also makes the networking position heartbeat broadcast what viewers actually see rather than the decode lead.
basis_media_native.dll (win-x64, MSVC 19.51) and libbasis_media_native.so (android arm64-v8a, NDK clang 18) built from this branch.
towneh
force-pushed
the
feat/vod-seek-sync
branch
from
5038129 to
1fb0de2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VOD seeking, end to end: a native seekable timeline, a seek bar on the Media Players panel, and multiplayer position sync — drift correction, owner seeks reaching every client, and late joiners landing at the owner's position instead of zero.
The networking layer needed almost nothing:
FullStatealready carriedpositionTicks, theSeekmessage and drift correction already existed, and the late-join apply already setStartPosition— all dormant because nativeDurationwas permanently zero andSeekthrew. The substance is in the native engine.Duration (
basis_media_get_duration_us): progressive MP4 reports themvhdduration (falling back to the longest track'ssttstotal); TS-segment HLS VOD reports the summedEXTINFdurations. 0 means unknown/live, and doubles as the "this source can seek" signal — sources that can't actually reposition (fMP4-segment HLS VOD, progressive files on servers withoutAccept-Ranges) deliberately stay at 0 rather than advertising a timeline they can't honour.Absolute seek (
basis_media_seek_us, asynchronous, lands on the preceding keyframe/segment boundary):Rangefails the call instead of silently restarting at byte 0). The engine posts requests through a new optionaltake_seeksink callback, one per demux leg so a split source repositions both; the demuxer maps the target through its sample tables (video backs up to the precedingstsssync sample, edit-list-mapped timelines included), refetches at the earliest needed byte, and drops parked samples from before the jump. The paced read-ahead ring parks its reader around the refetch and stays alive at EOF, so a backward seek after the file fully buffered still works.ResetSyncAnchor. If device testing surfaces an audible blip at the jump, a decoder flush export is the follow-up.Managed:
Duration/Seek/StartPositionroute to the native engine (old native libraries read as duration 0 — exactly the pre-feature behaviour).Seekstill throwsNotSupportedExceptionon timeline-less sources.Networking (all on the existing component, no new sync surfaces beyond one message): an owner position heartbeat — a 9-byte latest-wins ping (
Sequenced, like the framework's other position streams) every 5 s while playing seekable media, feeding the existing drift correction — and the owner's position/pause snapshot now stashes across a remote page-URL resolve and applies on the resolved source's ready, aged by the resolve time, so late joiners to a YouTube VOD land near the owner.Panel: a timeline scrubber under the transport buttons (visible only when the selected player has a timeline; drag-debounced, seeks through the networking component when present), and the Status box now shows elapsed/total time plus the current title and uploader from the player's metadata.
Verification
Closed-loop demux harness against generated and real fixtures:
mvhdduration covers the edited timeline).key=1), audio resumes at 8 s, both tracks interleave immediately; the skipped middle shows in the AU counts.Windows DLL (MSVC 19.51) and Android arm64
.so(NDK clang 18) rebuilt from this branch.Editor-validated (Windows desktop, Unity 6000.5.3f1, single client) against a ~3 min real progressive recording served over HTTP:
2026-07-08_23-08-35_10mb.mp4
Two other fixes came out of the editor pass, folded into their commits: the pacing anchor now re-anchors when a seek is taken (against the old anchor, a forward jump read as far-future and stalled delivery for the jump distance; a backward jump read as late and fast-forwarded back), and seekability is now proven by a
Range: bytes=0-probe on the initial GET rather than trusted fromAccept-Ranges— a server that advertises ranges without implementing them (Python'sSimpleHTTPRequestHandler, for one) gets no seek bar instead of a dead stream on the first seek.Not yet validated: the multiplayer leg (owner-seek propagation, heartbeat drift correction, late-join catch-up) — needs a second client; I'll report back here after that pass.
Notes and limitations