Import upstream snapshot c19f713c415a699a79d71cd96aa13c3104a05047

Upstream: https://github.com/michaelgillett/mjlab
Upstream-Commit: c19f713c415a699a79d71cd96aa13c3104a05047
Upstream-Branch: main

docs/source/sensors/rgbd_camera.rst

@@ -0,0 +1,219 @@
+.. _rgbd_camera:
+
+RGB-D Camera
+============
+
+``CameraSensor`` renders RGB and depth images on the GPU using MuJoCo
+Warp's ray-traced rendering pipeline. It can either wrap an existing
+MuJoCo camera defined in your XML or create a new one programmatically.
+
+
+Quick start
+-----------
+
+.. code-block:: python
+
+    from mjlab.sensor import CameraSensorCfg
+
+    # Wrap an existing MuJoCo camera from the robot's XML.
+    cam = CameraSensorCfg(
+        name="wrist_cam",
+        camera_name="robot/wrist_camera",
+        data_types=("rgb", "depth"),
+        width=160,
+        height=120,
+    )
+
+    scene_cfg = SceneCfg(
+        entities={"robot": robot_cfg},
+        sensors=(cam,),
+    )
+
+    # Access at runtime.
+    data = env.scene["wrist_cam"].data
+    data.rgb      # [B, 120, 160, 3] uint8
+    data.depth    # [B, 120, 160, 1] float32
+
+
+Creating vs wrapping cameras
+-----------------------------
+
+There are two ways to set up a camera sensor.
+
+**Wrap an existing camera.** If your MJCF model already defines a
+camera, pass its name via ``camera_name``. The sensor uses the
+camera's position, orientation, and field of view from the model.
+You can optionally override ``fovy`` or switch to orthographic
+projection.
+
+.. code-block:: python
+
+    # Wrap the camera named "front_cam" in the robot's XML.
+    CameraSensorCfg(
+        name="front",
+        camera_name="robot/front_cam",
+        data_types=("rgb",),
+    )
+
+**Create a new camera.** When ``camera_name`` is ``None`` (the
+default), the sensor adds a new camera to the MjSpec during scene
+construction. Specify ``pos``, ``quat``, and optionally ``fovy`` to
+place it.
+
+.. code-block:: python
+
+    # Fixed overhead camera on the worldbody.
+    CameraSensorCfg(
+        name="overhead",
+        pos=(0.0, 0.0, 2.0),
+        quat=(0.0, 0.707, 0.707, 0.0),
+        fovy=60.0,
+        width=320,
+        height=240,
+        data_types=("rgb", "depth"),
+    )
+
+
+Camera parameterization
+-----------------------
+
+MuJoCo supports two ways to define a camera's projection, and both
+work with ``CameraSensor``. See the `MuJoCo camera documentation
+<https://mujoco.readthedocs.io/en/stable/XMLreference.html#body-camera>`_
+for full details.
+
+**FOV-based.** The simpler approach. A single ``fovy`` (vertical field
+of view in degrees) combined with the image resolution defines the
+projection. This is the default when creating cameras
+programmatically via ``CameraSensorCfg``.
+
+**Intrinsic-based.** For matching real camera hardware, MuJoCo cameras
+can be parameterized with ``sensorsize``, ``focal`` (or
+``focalpixel``), and ``principal`` (or ``principalpixel``). These
+fields are set in the MJCF XML and provide direct control over the
+intrinsic matrix. When intrinsic parameters are present, ``fovy`` is
+ignored by MuJoCo.
+
+When wrapping an existing camera, the sensor inherits whichever
+parameterization the XML defines. When creating a new camera, the
+sensor uses ``fovy``. To use intrinsic parameters for a new camera,
+define it in your XML and wrap it with ``camera_name``.
+
+.. note::
+
+   If you plan to randomize the field of view with domain
+   randomization, use ``dr.cam_fovy`` for FOV-based cameras or
+   ``dr.cam_intrinsic`` for intrinsic-based cameras. Randomizing
+   ``cam_fovy`` has no effect on cameras that use intrinsic parameters.
+
+
+Body-mounted cameras
+--------------------
+
+Set ``parent_body`` to attach a new camera to a specific body rather
+than the worldbody. The ``pos`` and ``quat`` are then relative to the
+parent body frame, so the camera moves with the body.
+
+.. code-block:: python
+
+    # Camera mounted on the robot's end-effector.
+    CameraSensorCfg(
+        name="ee_cam",
+        parent_body="robot/link_6",
+        pos=(0.0, 0.0, 0.05),
+        quat=(1.0, 0.0, 0.0, 0.0),
+        fovy=45.0,
+        width=160,
+        height=120,
+        data_types=("rgb", "depth"),
+    )
+
+
+Data types
+----------
+
+The ``data_types`` tuple selects which image modalities to render.
+Only requested types are allocated; the other field on
+``CameraSensorData`` is ``None``.
+
+.. list-table::
+   :header-rows: 1
+   :widths: 15 20 65
+
+   * - Type
+     - Shape
+     - Description
+   * - ``"rgb"``
+     - ``[B, H, W, 3]`` uint8
+     - Color image. Rendered as packed ABGR uint32 by MuJoCo Warp,
+       then unpacked to RGB channels on the GPU.
+   * - ``"depth"``
+     - ``[B, H, W, 1]`` float32
+     - Depth image. Values are distances from the camera plane.
+   * - ``"segmentation"``
+     - ``[B, H, W, 2]`` int32
+     - Typed segmentation. Channel 0 stores object IDs and channel 1 stores
+       MuJoCo object types. Background pixels are ``(-1, -1)``.
+
+
+Render settings
+---------------
+
+All camera sensors in a scene must share identical values for
+``use_textures``, ``use_shadows``, and ``enabled_geom_groups``. This
+is a constraint of the underlying MuJoCo Warp rendering system, which
+uses a single ``RenderContext`` for all cameras. Mismatched settings
+raise a ``ValueError`` at scene construction.
+
+.. code-block:: python
+
+    # These two cameras must agree on render settings.
+    cam_a = CameraSensorCfg(
+        name="cam_a",
+        camera_name="robot/front_cam",
+        use_textures=True,
+        use_shadows=False,
+        enabled_geom_groups=(0, 1, 2),
+        data_types=("rgb",),
+    )
+    cam_b = CameraSensorCfg(
+        name="cam_b",
+        camera_name="robot/wrist_cam",
+        use_textures=True,       # Must match cam_a
+        use_shadows=False,       # Must match cam_a
+        enabled_geom_groups=(0, 1, 2),  # Must match cam_a
+        data_types=("depth",),
+    )
+
+
+Output
+------
+
+``CameraSensorData`` is a dataclass with one field per data type.
+
+.. code-block:: python
+
+    @dataclass
+    class CameraSensorData:
+        rgb: Tensor | None      # [B, H, W, 3] uint8
+        depth: Tensor | None    # [B, H, W, 1] float32
+        segmentation: Tensor | None  # [B, H, W, 2] int32
+
+By default, the returned tensors are zero-copy views into the render
+buffer. Set ``clone_data=True`` on the config if you modify them in
+place, to avoid corrupting the shared buffer.
+
+
+Visualization in Viser
+----------------------
+
+The Viser viewer automatically discovers all ``CameraSensor`` instances
+in the scene and displays their RGB and depth outputs as live image
+panels in the GUI sidebar. A camera frustum is rendered in the 3D
+viewport showing the camera's position, orientation, and field of view.
+Depth images include an interactive scale slider for adjusting the
+visualization range.
+
+.. image:: ../_static/viser_camera_pane.png
+   :align: center
+   :alt: Viser viewer showing camera image panels and frustum visualization