feat(addons/objects3d): reusable 3D object detection (2D detect + depth into oriented boxes)#417
feat(addons/objects3d): reusable 3D object detection (2D detect + depth into oriented boxes)#417salmanmkc wants to merge 30 commits into
Conversation
segmenterMaskFromSnapshot resolved with the raw MediaPipe categoryMask, whose backing buffer is only valid for the duration of the segment() callback. The consumer reads it (getAsUint8Array) in a later microtask, after the buffer is freed, giving garbage masks or a crash with the mediapipe backend. Copy the data out inside the callback and resolve a plain snapshot, mirroring the SAM path.
_detectInFlight was set before an unguarded setup block (canvas 2d context, toDataURL, camera clone, depth-mesh snapshot). A throw there left the guard set and wedged the detector for the page lifetime. Wrap the whole detect() body in try/finally so the guard is always cleared.
ruofeidu
left a comment
ruofeidu
left a comment
There was a problem hiding this comment.
Some comments in line.
High-level: after tested on device --- Nels: can you give it a try?
consider runDetection({snapshot, backend}) override in ObjectDetector,
or before this is stable, it can live in addons and enable only when reqeusted.
| export function fuseIntoBoxes( | ||
| records: FusionRecord[], | ||
| newObb: InternalObb, | ||
| cat: string |
There was a problem hiding this comment.
nit: category instead of cat
| ): FusionRecord | null { | ||
| const half = (s: THREE.Vector3) => (s.x + s.y + s.z) / 6; | ||
| for (const rec of records) { | ||
| if (rec.category !== cat) continue; |
There was a problem hiding this comment.
question: will a chair and a sofa near each other (both furniture) fuse into one object?
There was a problem hiding this comment.
Yes, previously they could fuse because matching used only the broad category and center distance. I changed this so same-label observations retain the existing proximity matching, while differently labelled observations fuse only when both OBB centers lie inside each other's box. So tested it, it still handles one object changing from armchair to sofa across views, but keeps merely adjacent furniture separate. I added regression tests for both cases.
|
|
||
| const c = obb.center; | ||
| const farFromRoom = | ||
| Math.abs(c.x) > 6 || Math.abs(c.z) > 6 || c.y < -1 || c.y > 5; |
There was a problem hiding this comment.
nit: declare magic numbers
There was a problem hiding this comment.
Thanks, pulled these into named constants, split by purpose: depth-sampling settings, room bounds, minimum samples per category, and fitted-box size and degeneracy limits.
runDetection({snapshot, backend}) override in ObjectDetector as a follow up? |
3 participants
adds an
objects3daddon that turns 2D detection plus the depth mesh into oriented 3D bounding boxes you can query in world space. it's the pipeline that lived inline in theobjects_3ddemo, pulled out into a reusableScriptso apps and other addons can just ask for detected 3D objects.Object3DDetectorruns the whole thing: snap the camera and depth mesh, run 2D detection, get a per-object segmentation mask, raycast the masked depth samples into world space, fit a yaw-aligned oriented box, fuse across views, and optionally draw debug wireframe boxes. each result is aDetected3DObjectwith a nearest-surface query, so you can ask for the closest point on an object, which is handy for pointing at it or placing something against it.the backends are pluggable:
gemini(open-vocabulary, needs a key) ormediapipe(on-device COCO, no key, fixed class set).@huggingface/transformers) or the mediapipe segmenter.it also exports the pure helpers that are useful on their own:
uvToNdc(the snapshot-vs-camera aspect correction),box2dIoU/unionDetections/snapBoxToFloor(2D fusion and floor snapping), and the labelcategorizehelpers (flat / surface / tiny-flat / light buckets).the heavy deps (
@huggingface/transformers, mediapipe) stay external; the one rollup change just marks transformers external so it isn't bundled.this is split out of the agenthands branch (#416), which originally grounded its pointing through this pipeline before I switched it to a lighter depth raycast. the addon ships standalone here with colocated vitest specs (label categorization, depth sampling, fusion, and the nearest-surface query); the
objects_3ddemo can move onto it as a follow-up. lint, tests and build are clean.