feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419
feat(camera): read camera intrinsics + pose off the WebXR view (table stays fallback)#419salmanmkc wants to merge 3 commits into
Conversation
…a access While streaming raw camera frames, read the XRView's projectionMatrix (clip-from-view) and transform.matrix (world-from-view) into the device camera, in the same reference space three renders in. Reset the flag when camera access is lost so consumers fall back cleanly.
getDeviceCameraClipFromView/WorldFromView now use the intrinsics/pose captured off the XRView when available, so the device reports its own camera math instead of a static per-device profile. The DEVICE_CAMERA_PARAMETERS table stays as the fallback for the getUserMedia path and the simulator, and targetDevice remains a manual override.
|
this only kicks in on the webxr raw-camera path and leaves the DEVICE_CAMERA_PARAMETERS table as the │ fallback, so the getUserMedia path and the simulator are unchanged. i haven't been able to validate the numbers on │ real hardware yet though, so it'd be good to confirm the intrinsics + pose line up on a galaxy xr and an aura before │ leaning on it. those two share the android xr user agent, which is the whole reason for reading off the view instead of the device table. |
|
I haven't tested, but I suspect it likely won't work on Galaxy XR or Aura since it's not exposed at the Android or C++ api level afaik. |
|
let me know when this is ready for reviews. |
Technically it's ready for review, if it doesn't work with pass through on aura or galaxy XR it will fallback safely |
3 participants
follow-up to the thread on #268. right now the device camera's intrinsics and pose come from a static per-device table (
DEVICE_CAMERA_PARAMETERS) keyed bytargetDevice, which we pick from the user agent. that's fragile: galaxy xr and aura share the same android xr user agent, so the UA can't always tell them apart.but webxr already hands us the real numbers. the raw camera access spec puts the camera's intrinsics on
XRView.projectionMatrixand its pose onXRView.transform, andupdateXRCamerais already holding that view to read the camera image offview.camera, it just wasn't reading those two fields.so this reads them off the view we already have. while raw camera access is streaming,
updateXRCameracapturesview.projectionMatrix(clip-from-view) andview.transform.matrix(world-from-view) into the device camera, in the same reference space we pass togetViewerPose(and that three renders the scene in). thengetDeviceCameraClipFromView/getDeviceCameraWorldFromViewprefer those when present, so the device reports its own camera math whatever device it is.the
DEVICE_CAMERA_PARAMETERStable stays as the fallback for the getUserMedia path (older Quest with no XRView) and the simulator, andtargetDevicestays as a manual override. the flag resets when camera access drops so we fall back cleanly.added unit tests for the selection (prefer the view params when captured, otherwise the table) and for
isDeviceCameraPoseAvailablepicking up the new source. lint, tests and build are clean.honestly not 100% sure it lines up cleanly on every device, so it only kicks in on the WebXR raw-camera path and leaves the table untouched as the fallback. worth confirming on a real galaxy xr / quest before leaning on it.