Physics Aligned Simulator Scenes and Detection

[whuang37]

A new more unified simulator scene loading and handling for better more repeatable physical world simulations. One step closer to a more representative simulator mode.

• Simulator environments now reference one JSON file for room mesh, plane data, namesh, transforms, and physical world objects

{
  "scenePath": "./room.glb",
  "scenePlanesPath": "./room-planes.json",
  "navMeshPath": "./room-navmesh.glb",
  "position": [0, 0, 0],
  "quaternion": [0, 0, 0, 1],
  "scale": [1, 1, 1],
  "objects": [
    {
      "id": "table",
      "assetPath": "./table.glb",
      "position": [0, 0, -2],
      "physics": "fixed",
      "detectObject": true,
      "label": "table",
      "data": {"evaluationId": "target-table"}
    },
    {
      "id": "box",
      "assetPath": "https://cdn.example.com/models/box.glb",
      "position": [0, 1.5, -1],
      "physics": "dynamic",
      "detectObject": true,
      "label": "box"
    }
  ]
}

await xb.core.simulator.setEnvironment(1);
console.log(xb.core.simulator.activeEnvironmentManifest);

• physical world objects may be declared that are loaded from glTF and receive an id + physics if desired
• simulator now owns a separate Rapier world for the room and physial world thats separated from the actual XR blocks physics
  • allows simulator objects to collide with each other and the room
  • dynamic physcs on simulator objects
  • no collision with virtual objects
  • mesh detection still remains the same virtual mesh bodies for actually doing collisions
• objectDetection can now take simulatorOverrides for added objects which injects a detected object and circumvents prediction
  • works the same as existing object detection which only detects things that are within camera view. spatial anchors must be made custom
• xb.world.meshes now contains the room mesh + a merged mesh for each object labeled as other in the simulator
• Physical hands are now opt in with options.simulator.handPhysics.enabled = true
  • palm centered kinematic collisions
  • cant move thru simulator geometry anymore
  • shoulder palm tether to prevent hands getting stuck with objects between
• demos/simulator_scene_objects shows the demo

Demo vid

[dli7319]

Can you add a comment to setEnvironment?
Is it supposed to be called after init?

[whuang37]

Can you add a comment to setEnvironment? Is it supposed to be called after init?

its called within init right now as the new way to load environments. thats just the public runtime api that can be called at anytime. Added a comment and made it a bit cleaner by running off manifest + name rather than index