Skip to content

Depth Tracking

Depth tracking uses 3D surface data from depth sensors to match the real object's shape against the 3D model. It provides strong geometric constraints that complement silhouette and edge modalities.

Supported Depth Sources

Source Platform Notes
ARKit LiDAR iPhone Pro, iPad Pro Requires AR Foundation
ARCore Depth API (experimental) Android devices with depth support Requires AR Foundation. Depth tracking on Android is experimental in 2.0. Devices with a ToF sensor give the best depth. See Android
Intel RealSense Windows (desktop) Direct SDK integration

Note

Need depth tracking support for another sensor or platform, such as Azure Kinect? Contact support@formulaxr.com — we can integrate additional depth sources.

When to Use

Works well for:

  • Textureless objects (depth doesn't need visual features)
  • Symmetric objects (depth resolves ambiguities that silhouette can't)
  • Large objects with distinct 3D shape
  • Any scenario where a depth sensor is available

Struggles with:

  • Shiny or reflective surfaces — metallic, glossy, or mirror-like materials scatter or deflect the infrared light used by depth sensors, producing missing or wildly incorrect depth readings
  • Transparent objects — glass, clear plastic, and liquids let infrared pass through rather than reflecting it back to the sensor
  • Very dark or light-absorbing surfaces — matte black materials can absorb too much infrared, returning weak or no depth signal
  • Direct sunlight — outdoor or strong IR light sources overwhelm the depth sensor's infrared emitter, causing noisy or missing depth across the entire frame
  • Very thin objects — insufficient surface area for a reliable depth measurement
  • Objects far from the camera — depth resolution and accuracy decrease with distance (most sensors are effective within 0.2m - 5m)

Configuration

On the TrackedBody component:

  • Enable the Depth Tracking checkbox

Depth tracking is a complementary modality — it is always used in addition to a contour modality (silhouette or edge) and cannot be used alone. If paired with silhouette, the silhouette tracking model must be generated first (depth uses it for detection).

Note

A depth camera feed must be available. For AR Foundation, depth is provided automatically when the device supports it. For desktop, use RealSense or recorded depth sequences.

Quality Metrics

  • Depth Quality — How closely the measured depth aligns with the model surface. Higher = better alignment.
  • Quality ranges from 0.0 (no alignment) to 1.0 (perfect match)
  • With depth tracking on, a body that tracks by the edge method starts, holds and stops on its edge score, because depth only helps the fit. A silhouette body reads the larger of its silhouette score and its depth score

Tips

  • Depth tracking significantly improves robustness when the background is cluttered
  • For best results, ensure the object is within the depth sensor's effective range (typically 0.2m - 5m)
  • Combine with silhouette or edge tracking — depth excels at position accuracy while contour modalities provide rotational precision
  • The Standard Deviations of the depth settings are the measurement noise of the depth correspondences in meters, one value for each correspondence iteration. They default to 0.05, 0.03 and 0.02, and larger values make the tracker trust the depth measurement less