Skip to content

fix(mediaplayer): submit the Android Vulkan resolve from a plugin-owned command buffer#923

Merged
dooly123 merged 9 commits into
BasisVR:developerfrom
towneh:fix/mediaplayer-android-vulkan-present
Jul 6, 2026
Merged

fix(mediaplayer): submit the Android Vulkan resolve from a plugin-owned command buffer#923
dooly123 merged 9 commits into
BasisVR:developerfrom
towneh:fix/mediaplayer-android-vulkan-present

Conversation

@towneh

@towneh towneh commented Jul 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes #922 — media player video crashed the client on Quest within about a second of the first decoded frame: a null-pointer dereference inside the Adreno driver's vkCmdExecuteCommands, on Unity's Vulkan submission thread. On-device isolation (detailed in the issue) pinned the trigger to the commands the present path recorded into Unity's command buffer via CommandRecordingState — a pipeline barrier plus the plugin's own render pass and draw. Declaring an outside-render-pass precondition didn't help; skipping the recorded work entirely removed the crash.

The present path no longer touches Unity's command buffers at all:

  • The YCbCr→RGBA resolve records into a plugin-owned command buffer (one per import-ring slot, one shared pool) and is submitted on the graphics queue directly from the render event, which is now configured with kUnityVulkanGraphicsQueueAccess_Allow — Unity holds its own queue users off the queue while the callback runs.
  • A fence per slot replaces Unity's safeFrameNumber for reclaiming imported AHardwareBuffers; release waits in-flight fences (capped) before destroying.
  • The Unity RenderTexture's VkImage is queried with AccessTexture(ObserveOnly) — no barriers recorded on the plugin's behalf. No layout coordination with Unity is needed: the resolve render pass already takes the attachment from UNDEFINED (contents discarded — every draw overwrites the full frame) and returns it in SHADER_READ_ONLY, the layout Unity samples it in. Same-queue submission order plus the pass's external dependencies order the write against Unity's sampling.
  • basis_gfx_vk_begin_record (the Unity command-buffer access) is removed.

Rebuilt libbasis_media_native.so (arm64-v8a, NDK r27c) is included.

Stacked on #921 — this branch is based on the multichannel-audio branch, so the diff shows #921's commits until that merges; the change to review here is the final commit (fix(mediaplayer): submit the Android Vulkan resolve from a plugin-owned command buffer).

Required checks

All boxes below must be ticked before this PR can merge. If a check is genuinely N/A, tick it anyway and explain under Notes.

  • Tested — I built and ran this locally. The change works in the editor and (where relevant) in a built player.
  • Transform access is combined and limited — In hot paths, transform reads/writes go through TransformAccessArray or are otherwise batched. I have not added per-frame transform.position / transform.rotation / transform.localPosition calls inside loops. Whenever I need both position and rotation, I use the combined APIs — SetPositionAndRotation / SetLocalPositionAndRotation for writes, GetPositionAndRotation / GetLocalPositionAndRotation for reads — instead of two separate property accesses; the combined call does one local-to-world matrix traversal instead of two.
  • Addressables used for asset/memory loading — Any new asset loads go through Addressables. No new Resources.Load, no direct asset references that pull large content into memory on scene load.
  • No new GetComponent / AddComponent where avoidable — Where unavoidable, the result is cached on a field, and any GetComponent<T> is replaced with TryGetComponent<T>(out var x) — bare GetComponent will be denied. TryGetComponent is the modern API (Unity 2019.2+) and skips the Editor-only GC allocation GetComponent causes when a component is missing: Unity wraps the null return in a managed "fake null" object so its overloaded == operator can still detect destroyed C++ objects, and constructing that wrapper allocates; TryGetComponent returns a bool plus out parameter and never builds the wrapper. None of these calls run inside Update, LateUpdate, FixedUpdate, jobs, or other per-frame code paths.
  • Per-frame work is scheduled through BasisEventDriver — Any new per-frame work hooks into BasisEventDriver rather than adding standalone Update / LateUpdate / FixedUpdate callbacks on a MonoBehaviour.
  • Anything added to BasisEventDriver is bulletproof, or guarded by try/catchBasisEventDriver runs the single per-frame tick that drives the whole framework (network apply, local player sim, blendshapes, JigglePhysics, nameplates, and more) as one sequential chain. An unhandled exception anywhere in that chain aborts the rest of the tick, so every step after the throwing one is silently skipped for that frame. New work added to the driver must either be guaranteed not to throw, or be wrapped in a try/catch that contains the failure and surfaces it through BasisDebug — logged once / rate-limited, never every frame (see the existing HVRBasisBuiltInAddresses.Simulate() guard for the pattern). Expect this to be scrutinized closely in review.
  • Considered jobification — I asked whether this work can be moved to a Unity Job (Burst-compiled where possible). If it can, it is. If it cannot, the reason is in Notes.
  • No needless { get; set; } properties or access lockdowns — Public fields are fine; Basis is meant to be read and modified freely, so don't wall things off private/internal without a real reason. Don't wrap a field in { get; set; } when the accessors do nothing — property accessors have a real performance cost vs direct field access, and the lead maintainer prefers plain fields (or a method / setter-only property when only the setter needs logic) over a noop-getter pair. For .Instance singletons, callers reassigning Type.Instance is allowed; if that would break your code, log a warning or throw — don't block the assignment. Locking down access is not your call.
  • Camera access goes through BasisLocalCameraDriver — Code that needs the local camera (transform, projection, rig data, etc.) pulls it from BasisLocalCameraDriver rather than looking one up itself. Don't roll a separate camera discovery path.
  • Logging uses BasisDebug — All new logging calls go through BasisDebug.Log / BasisDebug.LogWarning / BasisDebug.LogError (with an appropriate LogTag) instead of UnityEngine.Debug.Log / Debug.LogWarning / Debug.LogError. BasisDebug routes through Basis's tagged, color-coded logger and respects the project-wide LoggingDisabled toggle so logging can be killed at runtime; bare Debug.Log calls bypass that and will be denied.
  • No scene-wide discovery for dependencies — New code is architected so it does not need FindObjectOfType / FindObjectsOfType / GameObject.Find / FindGameObjectsWithTag to locate what it depends on. References are wired in — registered through an existing manager/driver, injected at init, or passed in by the caller — rather than discovered by scanning the scene at runtime. If a scene scan is genuinely unavoidable, justify it under Notes.
  • No allocations in hot paths — Per-frame code (Update / LateUpdate / FixedUpdate, simulation loops, jobs, anything called once per frame or more) does not allocate. No new on reference types, no LINQ, no string concatenation/interpolation, no boxing, no foreach over interface-typed collections. Allocate once at init and reuse the buffer.
  • No debugging in hot paths — No log calls of any kind on per-frame paths, including BasisDebug. Hot-path logging floods the console and incurs cost on every frame regardless of whether the message is filtered out downstream. If a hot-path log is needed while iterating, gate it behind #if UNITY_EDITOR and remove (or leave gated) before merge.
  • Hot-path collection access is optimized — Cache .Count (lists) / .Length (arrays) into a local int before the loop instead of re-reading the property each iteration. Prefer T[] (with a separate length int when the array is over-sized) over List<T> where the data is hot — Unity's mono BCL doesn't expose CollectionsMarshal.AsSpan(List<T>), so a list can't be fed into Span<T> / unsafe paths cleanly. Where the perf justifies it, drop into Span<T> / ref locals / Unsafe.As / unsafe pointer code to skip bounds checks and copies, and call out the invariants you're relying on under Notes so reviewers can sanity-check them.

Testing details

Tick the platforms you actually tested on. Leave the rest unticked — these are informational and do not block merge.

  • Windows
  • Linux
  • Android
  • iOS
  • macOS

Input / control mode coverage:

  • Tested in VR (note headset under Notes)
  • Tested in desktop / non-VR mode
  • Tested with phone controls (mobile touch input)
  • N/A — change does not touch player/XR/input code

Where applicable, confirm these flows still work after your changes:

  • Hot-switching (desktop ↔ VR mode swap at runtime)
  • Avatar swapping
  • Server swapping (joining / leaving / changing servers)
  • N/A — change does not touch any of the above

Notes

Scope / N/A checks: entirely a native C/C++ plugin change (Native~/android, Native~/unity, one internal header) plus the rebuilt arm64 .so — no C# is touched, so the Unity-side checks (transform access, Addressables, GetComponent, BasisEventDriver, properties, camera driver, BasisDebug, scene discovery, hot-path allocations/collections) have nothing to apply to. Jobification: N/A, native present path.

Tested on Quest Pro (Android 14), sideloaded build, H.264 + AAC over RTSP/TCP:

Trade-offs / caveats:

  • The submit happens on the graphics queue once per decoded frame from the render event. GraphicsQueueAccess_Allow means Unity briefly holds its queue users off the queue during the callback; no frame-pacing impact was observed in the soak, but it's the structural cost of not sharing Unity's command buffers.
  • Windows is untouched (the D3D11/D3D12 present path is separate; the Vulkan glue changes compile only on Android).

towneh added 8 commits July 5, 2026 14:23
Multichannel AAC on Android played back wrong: device AAC decoders can
fold 5.1/7.1 down to stereo unless the codec is configured with an
output-channel ceiling, and the Android backend never set one (the
Windows backend already prefers the input channel count for the same
reason). Configure both the generic API 32+ key and the legacy AAC key
on both audio paths (extractor tracks and demuxer-fed streams).

Also harden the PCM path around channel-count changes:

- the PCM ring drops whole interleaved frames on overflow, so a full
  ring can no longer rotate the channel alignment
- a channel-count change re-frames (and clears) the ring instead of
  mixing old- and new-layout floats
- float PCM output (pcm-encoding 4) is written straight to the ring
  instead of being misread as 16-bit
- the audio output format is logged on each format change for device
  diagnosis

Rebuilt libbasis_media_native.so (arm64-v8a, NDK r27c).
Document on the ring what the Windows ring already states: drops are
whole-frame counts (preserving channel phase), reads may return partial
frames, and the managed splitter carries sub-frame remainders — so
neither end anchors to an absolute frame boundary.
The render events record pipeline barriers, an own render pass and
state changes into Unity's command buffer, which is only legal when
Unity's render pass is not in progress. Declare that as an event
precondition via ConfigureEvent so the dispatcher enforces it in every
recording mode, rather than relying on EnsureOutsideRenderPass from
inside the callback. On Quest, playback crashed with a null-deref in
the Adreno driver's vkCmdExecuteCommands on the Vulkan submission
thread as soon as decoded frames reached the present path.
The absent-video guard checked video.pt < 0, but the zero-initialised
struct left pt at 0 - a valid RTP payload type - so an audio-only SDP
walked past the guard into a video SETUP on the base URL and surfaced
as a confusing server 400. Initialise both media slots with a -1
sentinel so the "no usable media" path fires as intended.

Also rebuilds libbasis_media_native.so (arm64-v8a) with this and the
Vulkan render-event precondition change.
- Announce the AAC audio format once per session instead of on every
  RTP packet, using a per-session flag beside the existing
  video_announced (the previous function-level static was dead code,
  and would have been shared across sessions had it been used).
  Downstream decoders already ignore repeat announcements, so this
  removes per-packet churn rather than changing behaviour.
- Always drain an RTSP reply's Content-Length body, including when the
  caller passes no body buffer - an ignored body previously stayed on
  the socket and desynchronised the next reply.
- Zero-initialise the Vulkan plugin event config struct, matching the
  file's existing initialisation style.

Rebuilds libbasis_media_native.so (arm64-v8a).
Android treats a missing pcm-encoding key as signed 16-bit PCM, but the
handler only updated the float flag when the key was present - a later
output format without the key would keep a previously-set float flag
and misread 16-bit buffers. Default the value to 16-bit and assign the
flag unconditionally.

Rebuilds libbasis_media_native.so (arm64-v8a).
…ed command buffer

Recording the YCbCr->RGBA resolve (pipeline barrier + render pass + draw)
into Unity's command buffer corrupts the Adreno driver's command-buffer
state on Quest and crashes the app in vkCmdExecuteCommands on Unity's
submission thread as soon as decoded frames flow (BasisVR#922).
Declaring an outside-render-pass precondition on the event did not help;
a build that skips the recorded work entirely does not crash, isolating
the trigger to the recorded commands themselves.

The present path no longer touches Unity's command buffers at all:

- each import-ring slot gets its own primary command buffer (one shared
  pool) and a fence; the resolve is recorded there and submitted on the
  graphics queue from the render event, which is now configured with
  kUnityVulkanGraphicsQueueAccess_Allow so Unity holds its queue users
  off the queue during the callback
- fence status replaces Unity's safeFrameNumber for reclaiming imported
  AHardwareBuffer slots; release waits in-flight fences (capped) before
  destroying
- the Unity RenderTexture's VkImage is queried with
  AccessTexture(ObserveOnly) - no barriers recorded on our behalf. The
  resolve render pass already takes the attachment from UNDEFINED
  (full-frame overwrite) and returns it SHADER_READ_ONLY, the layout
  Unity samples in, so no layout coordination is needed
- basis_gfx_vk_begin_record (Unity command-buffer access) is removed

Rebuilds libbasis_media_native.so (arm64-v8a).
…ources

Fence-based reclamation left two teardown paths destroying resources a
submitted command buffer may still be using: a mid-stream external-
format change rebuilds the format objects (destroying every ring slot),
and an output-texture change destroys the framebuffer. Both now wait
the in-flight slot fences first, via a helper shared with the release
path (whose inline wait it replaces).

Rebuilds libbasis_media_native.so (arm64-v8a).
@towneh towneh force-pushed the fix/mediaplayer-android-vulkan-present branch from a9a6843 to 26f4880 Compare July 5, 2026 22:07
@towneh towneh requested a review from dooly123 July 5, 2026 22:08
@towneh towneh added the bug Something isn't working label Jul 5, 2026
The teardown wait is deliberately capped (hanging teardown on a wedged
or lost device is worse than proceeding, and destroying objects on a
lost device is the correct cleanup), but a non-success result was
silent. Capture it and log the slot and result code so a
teardown-under-timeout is diagnosable.

Rebuilds libbasis_media_native.so (arm64-v8a).
@dooly123 dooly123 merged commit 8594b56 into BasisVR:developer Jul 6, 2026
12 checks passed
@towneh towneh deleted the fix/mediaplayer-android-vulkan-present branch July 7, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Media player: video crashes on Quest (Adreno null-deref in vkCmdExecuteCommands from the Vulkan present path)

2 participants