Audio tap does not survive a session-preserving reload: a track switch ends it, nothing re-arms it

[superuser404notfound]

Corrected after measuring (the first version of this body claimed the stream dangles open; it does not). An internal reload runs stopInternal, which tears the tap controller down and finishes the stream. A host's for await therefore ends, so the loss is observable. What remains is smaller than first written, and still worth closing.

What happens

installAudioTap() binds the software delivery path to the SoftwarePlaybackHost live at install time. An audio-track switch is a session-preserving reload: it goes through stopInternal(resetDisplayCriteria: false, keepNativeHost:, keepCustomReader: true), which runs audioTapController?.teardown(); audioTapController = nil, and then builds a new host with no sink.

So a transport-level call (selectAudioTrack) ends an opt-in feature the host installed at load, and nothing re-establishes it. Same for a subtitle-track switch, a title switch and reloadAtCurrentPosition, which take the same path.

The documented contract is that load() and stop() finish the stream, opt-in being per load. A host switching an audio track called neither.

The #311 frame-time observer crosses the identical seam and is deliberately carried across it (AetherEngine+Loading.swift, host.setVideoFrameTimeObserver(softwareVideoFrameTimeObserver), "a load builds a new host and a new renderer, so an observer installed once by the host app has to be carried across the seam"). The tap has no counterpart, and the asymmetry is not intentional as far as the history shows.

Measured

40 s MKV, three audio tracks, software path, tap installed at load, selectAudioTrack at +20 ms:

  AUDIOTAP installed (deliverySource=true)
  HOSTCALL selectAudioTrack(index: 3) at +20 ms (was 1)
  AUDIOTAP stream finished (buffers=1)

Re-installing after the switch resumes delivery immediately (abufs 1 → 177), so nothing about the new session is hostile to the tap; it is only that nobody re-arms it.

aetherctl play --sw --seconds 8 --switch-audio 3@20 --audio-stats <multi-audio file>

Suggested shape

Carry the tap across the reload seam the way the frame-time observer is carried: hold the controller through a session-preserving stopInternal and re-arm the sink where the new host is built. A host that opted in at load then keeps its tap across a track switch, which is what the per-load contract already promises.

Failing that, document that a track switch ends the tap, so hosts re-install on stream end rather than discovering it.

Area

Software (VideoToolbox) playback path

[superuser404notfound]

Working through this, the case for carrying the tap across the seam got weaker the closer I looked, so what landed is the boundary made explicit rather than removed. @dlev02, since yours is the host actually on this API, the question at the end is for you.

Why not carry it across. Three things, in the order they moved me:

The "one line next to setVideoFrameTimeObserver" I wrote above is only true for the software sink. The loopback reader is bound to the session's SegmentCache (captured at install), and the remote-HLS reader to loadedURL plus the playhead mirror. A session-preserving reload rebuilds both sources, so carrying the controller would mean re-running the whole reader selection against the new host, with a backend that may have changed across the reload. That is a re-install with stopInternal surgery in front of it, not a hand-off.

The teardown is also the safer behaviour for the one use case this API was built for. AudioTapMonotonicFilter is per controller, so every install starts a clean timeline. Carrying one across would meet a reload that resumes slightly behind the previous position with the old lastEnd still latched, which is exactly the backward-overlap shape SpeechAnalyzer rejects with SFSpeechErrorDomain Code=17. That filter exists because of that error; re-introducing its input would be a poor trade for the convenience of not re-installing.

And the asymmetry with #311 that I leaned on in the body does not really hold. The frame-time observer is a registration with no resources of its own, the tap is a pipeline with a lifetime tied to the session. One surviving the seam and the other not is a difference in kind, not an inconsistency.

What shipped instead (main): the contract now says a session-preserving reload finishes the stream, on installAudioTap() and in docs/architecture.md, with the reason rather than just the rule. And the teardown has a log line that mirrors the install one, so a device log shows the boundary instead of a tap that was installed once and then went quiet:

[AetherEngine] audio tap installed (backend=software live=true)
[AetherEngine] audio tap torn down with the session (a reload finishes the stream; re-install to follow the new one)

The question. Does your host already treat the stream ending as "re-install", and if so, is the gap across a track switch cheap for you, or does it cost transcript? If a switch is expensive on your side, the right fix is a real re-install against the new session driven by the engine, and that is worth building on a measurement rather than on my guess about how the tap is consumed. Keeping this open for that answer.