Software path: cold-start clock arming deadlocks when the selected audio stream first appears after the renderer fills (state=playing, currentTime=0 forever; one seek unwedges)

[edde746]

What happened?

A software-path VOD session can come up with state == .playing, frames decoded, the layer isReadyForDisplay, hasFirstFrameReadyForDisplay == true — and currentTime pinned at 0 forever. No error, no timeout, nothing for a host to react to. One seek unwedges it and playback is normal from there.

Trigger: selectAudioTrack(index:) shortly after play(), selecting a track whose first packet sits later in the mux than the point where the video renderer fills. The audio-switch reload rebuilds the host with resumeAt = 0.00, and the rebuilt session deadlocks on its own cold-start clock arming.

Field case: Apple TV 4K (A15, tvOS 26), AV1+Opus MKV via Plex, host applies the viewer's preferred audio language ~20 ms after play() (engine default was track 1/jpn, preference was track 2/eng). Screen shows the first frame, position never moves; the viewer's first seek starts playback. Verbatim from the device log:

[AetherEngine] reload begin: audioStream=2 resumeAt=0.00s embeddedSub=-1 sidecar=nil
[AetherEngine] reload: loadSoftware done (414ms)
[AetherEngine] reload: state=.playing total=420ms
[SWHost] first video frame enqueued: pixfmt=0x34323076 size=1916x1072 pts=0.000s
[Renderer] enqueue #1: status=rendering ready=true error=nil
[SWHost] layer.isReadyForDisplay=true after 5 frames
   ... 19 s of state=playing with the host's progress reports stuck at time=0 ...
[AetherEngine] seek#1 programmatic began target=10.00
[AudioOutput] seekClock to=10.000 rate=1.0     <- first time the clock ever moves
[AetherEngine] seek#1 programmatic landed rendered=10.00 target=10.00

The deadlock

Four facts, each individually deliberate, that close a cycle (all refs 6.15.2, Native/SoftwarePlaybackHost.swift):

1. play() deliberately does not start the clock cold: "demux loop arms the clock on first decoded audio sample" (play(), ~716).
2. Arming happens in the demux loop's selected-stream audio branch, on the first decoded buffer (~1855).
3. The video branch back-pressures on renderer.isReadyForMoreMediaData (~1776). With the synchronizer at rate 0 anchored at 0, enqueued frames never drain, so once the renderer fills, the loop parks — upstream of every packet that could arm the clock.
4. Both escape hatches miss:
   • the video-branch fallback (~1807) needs audioDecoder == nil (it isn't — the decoder opened fine) or audioPacketsSeen >= 50 — but that counter only counts selected-stream packets, and zero of them have arrived;
   • the seek paths arm directly (seekClock + clockArmed = true, ~865-875), which is why the viewer's seek fixes it, and why nobody had seen it from a resume.

So: clock waits for selected audio → selected audio waits for the demux loop → the loop waits for the renderer → the renderer waits for the clock.

Why this mux hits it and most don't

The file is legal Matroska with per-track-grouped interleaving. First packet ordinal per stream (ffprobe, packet order):

stream	codec	first packet #
0 (video, AV1)	av01	18
1 (audio, jpn — the default)	opus	3
2 (audio, eng — the one selected)	opus	435
3 (audio, fre)	opus	11

An instrumented run shows the rebuilt session parking at packet #76 with zero stream-2 packets seen. The initial load never wedges because the default track arms at packet #3. A switch a few seconds in never wedges either, because the reload resumes at startPosition > 0, and videoDecoder.skipUntilPTS / renderer.setSkipThreshold keep the renderer empty until the target — the loop reads straight through to stream 2. Only the start-from-zero rebuild has a fillable renderer between play() and the first selected-audio packet.

Reproduction (headless, macOS, 6.15.2)

EngineLog.handler = { print($0) }
AetherEngine.setForceSoftwarePathForTesting(true)
let engine = try AetherEngine()
_ = try await engine.load(url: url, options: LoadOptions(suppressDisplayCriteria: true))
engine.play()
try? await Task.sleep(for: .milliseconds(20))
engine.selectAudioTrack(index: 2)   // a stream that first appears late in the mux
// currentTime stays 0.00 indefinitely; state stays .playing.

Observed against the field file: 15/15 one-second samples read state=playing current=0.00. Control run with the switch at +5000 ms instead of +20 ms: position resumes at 5.11 and advances normally — same file, same target stream. The wedged session's log shows video enqueues and layer.isReadyForDisplay=true after 6 frames, and never a [AudioOutput] seekClock line.

Any MKV whose selected track's first packet ordinal exceeds the renderer's fill point reproduces it; muxers that lag one track by a second or two produce this shape routinely. Happy to share a trimmed sample privately if useful.

What should the engine do?

The deadlock signature is precise: renderer refuses enqueues while the clock is unarmed. Any of these breaks the cycle; choosing is yours:

• arm off the first video frame when the video branch would otherwise park unarmed (anchor via SWClockAnchorPolicy at the frame's PTS, same shape as the existing video-only fallback), and let the first real audio buffer re-anchor if needed — AU live tv #107's machinery looks built for exactly that;
• or count every demuxed audio-type packet (not only the selected stream's) toward the >= 50 fallback, so a late selected stream degrades to video-anchored arming instead of deadlock;
• or bound the back-pressure park while unarmed.

Also worth a thought regardless of the fix: while wedged, the session publishes .playing with a ready first frame — a host has no signal that transport is dead (#334's ceiling covers the remote-HLS bypass, not this). If arming can still be defeated by some future shape, a "clock never armed" diagnostic after N seconds of .playing at rate 0 would make the next one of these a one-log-line diagnosis.

Area

Software (VideoToolbox) playback path

Would you be willing to open a PR?

Maybe, with guidance

[superuser404notfound]

Fixed in 6.17.1 (b0cbaa6d). The analysis in the report holds verbatim against 6.17.0, all four facts and both missed escape hatches, so there was nothing to re-derive.

What changed. SWClockAnchorPolicy.shouldArmFromParkedVideo decides whether a park on the video gate is terminal, and it is a state test rather than a timeout: unarmed, playing, renderer full, and nothing left that could still arm. In the combined demux loop the last term is unconditional, for the reason the report gives, that the loop is the single reader. When the test holds, the gate anchors on the video the renderer is already holding, through the same SWClockAnchorPolicy.resolve the video-only fallback uses, so a zero-based load keeps its anchor and a mid-stream-joined source re-anchors exactly as before. One release-visible line, which is the diagnostic asked for at the end of the report:

[SWHost] clock unarmed at the video gate: the selected audio stream (index 2) has
produced nothing by the renderer's fill point (audioPacketsSeen=0); anchoring on video

The >= 50 counter was left counting the selected stream only. Widening it would have made this particular shape degrade instead of deadlock, but it would still be a heuristic sitting downstream of a park that is provably terminal at the moment it is entered, and the next shape that reaches the gate by another route would find it again.

The live feeder gate got the same treatment, with a different last term. There a second thread keeps the ring filling and the look-ahead pump can still arm from it, so terminal there is the pump having spent its pre-arm budget without a decoded buffer, which is a declared audio track that never decodes. That loop wedges identically today and had no counterpart to the demux loop's undecodable-audio fallback.

Reproduced first, on a synthetic file. 40 s MKV, 1280x720 H.264, three Opus tracks, first packet ordinal 17 / 1 / 1562 / 2 (the third is the one selected). aetherctl play grew --switch-audio <index>[@ms] so a host applying a language preference just after play() is reproducible from the CLI:

aetherctl play --sw --seconds 15 --switch-audio 2@20 late-audio.mkv

Run	6.17.0	6.17.1
switch to ordinal 1562 at +20 ms	15/15 samples state=playing cur=0.00	clock at 1x from the first tick
switch to ordinal 2 at +20 ms	plays	plays
switch to ordinal 1562 at +5000 ms	wedges	plays
no switch	plays	plays

Same file in every row. The healthy runs never take the new path, which is the part worth checking on a real library.

Note that the +5000 ms control wedges on this fixture where it recovered on the field file, and the difference is informative: here the late track's first packet is late in PTS as well as in ordinal, so a rebuild resuming at 5.1 s still has nothing to arm on, while a grouped-interleaved track whose packets carry early timestamps is read straight through by the skip thresholds. The fixture is the harsher variant of the same defect.

One trade to flag before the retest. The clock now starts from the frame on screen, so samples of the newly selected track that are already behind it when they arrive are dropped by the audio renderer. On the fixture the track genuinely begins at 12 s and its audio starts exactly there with no further gaps, but on a grouped-interleaved mux whose late track carries timestamps from zero, that track can start a beat late instead of at zero. Trading a permanent wedge for that is worth it, and it is not the cheapest possible trade: holding the not-yet-enqueueable video packets instead of parking would remove the gap entirely, at a few MB of compressed packets and real complexity in that loop. Over-enqueueing past the gate to buy the same time is not an option, the post-tvOS-18 renderer answers that with FigVideoQueueRemote -12080.

So if the late start is audible on the AV1+Opus file, that measurement is the useful one to report, and the packet-hold variant is the next step.

Separately, measuring this surfaced that the decoded-PCM tap does not survive the audio-switch reload at all: installAudioTap() binds to the host the switch replaces, and the stream stays open delivering nothing. Tracked as #356, not part of this fix.

Thanks for the report. Tracing the whole cycle including which escapes miss and why a resume never sees it, then listing the possible exits without insisting on one, is what made this a same-day fix rather than an investigation.

Leaving this open for a retest on the field file.