Choose a Camera Renderer for the Simulator

Required Version

This feature will not be released until MoveIt Pro version 10.2.

The MoveIt Pro Simulator renders every simulated camera and camera-based lidar with MuJoCo's built-in OpenGL renderer. A robot configuration can also render its color images with an optional physically based renderer, for images that are closer to what a real camera sees. This page explains the difference, lists what each renderer supports, and shows how to switch a robot configuration between them.

Rendering Concepts

Rasterization and Path Tracing

A rasterizer projects each triangle onto the image and shades each pixel from the lights that reach it directly. Shadows and reflections are approximations added in extra passes. It is fast and deterministic, and it produces ideal depth as a by-product, which is why it is the source of depth images, point clouds, and camera-based lidar.

A path tracer follows light paths through the scene, bouncing between surfaces before they reach a light. Soft shadows, indirect light, and reflections come from the same calculation, and materials are described with physically based rendering (PBR) properties: base color, roughness, metalness, normal maps, and emission. Each pixel averages many random paths (samples per pixel), so an image costs much more GPU time and has noise that a denoiser removes.

	Rasterization (MuJoCo OpenGL)	Path tracing (physically based)
Strengths	Fast on any GPU, deterministic, ideal depth	Realistic lighting, shadows, and materials, closer to real camera images
Weaknesses	Flat look, legacy material model, no PBR textures	Needs a ray-tracing GPU, lower frame rate, noise to denoise
Use for	Depth, point clouds, lidar, fast iteration, CI	Perception testing, data generation, and demos where appearance matters

Renderers in MoveIt Pro

• MuJoCo OpenGL is the default and the only renderer for depth, point clouds, and lidar. It runs wherever the Simulator runs.
• Genesis Nyx is a GPU path tracer that MoveIt Pro can use for camera color images. MoveIt Pro does not redistribute it. You install it into your own workspace image and are responsible for complying with its license.

Nyx is added beside MuJoCo, not in place of it. MuJoCo keeps rendering depth, point clouds, and lidar in every mode; Nyx renders only the color image and its camera_info.

note

Nyx support is experimental, and its API can change. The physically based renderer that MoveIt Pro supports may change in a future release. The renderer setting described below names the renderer, so a robot configuration selects a new one by value.

Supported Sensor Features

The tables below describe the Simulator integration in MoveIt Pro 10.2.

Sensors

Sensor	MuJoCo OpenGL	Nyx
RGB camera	Yes	Yes
Depth image and point cloud (RGB-D camera)	Yes, ideal depth with no sensor noise model	No. Nyx has no depth output; MuJoCo keeps publishing depth
Active RGB-D camera (projected pattern)	Depth has no noise model, so it stands in for an ideal active sensor; the projector and its noise are not simulated	Color only
Passive stereo RGB pair	Yes, two RGB-only cameras with matching stamps. The Simulator does not publish the stereo camera_info with the baseline; a separate node publishes it, as in hangar_sim	Not supported. Each camera renders in its own Nyx node, so the two images are not guaranteed to share a stamp
Planar (2D) lidar	Yes, with <rangefinder> sensors, or one ring of a camera lidar published as a LaserScan	No. <rangefinder> sensors keep working in every mode
3D lidar	Yes, camera-based, with optional range noise and dropouts	No

The Simulator computes no stereo depth; a passive stereo pair needs your own stereo matching pipeline. <rangefinder> sensors are ray cast by the physics step, not rendered, so the renderer choice never affects them.

The Create MoveIt Pro Simulator Models guide shows how to add each sensor to the MJCF.

Surfaces, materials, and lighting

Feature	MuJoCo OpenGL	Nyx
Base color and color textures	Yes	Yes
PBR material layers (normal, orm, emissive)	No, ignored	Yes
Roughness and metalness	No	Yes. Without them, they are derived from the MuJoCo shininess, specular, and reflectance values
Emission	Yes	Yes
Glass and other transparent materials	Alpha blending	No. Geometry behind a transparent surface is not drawn, and refraction is not rendered
Mirror reflections	Approximated on the first reflective plane or box	Yes, from the material
Shadows	Off by default. With the camera_shadows hardware parameter set to true, shadow maps from each light with castshadow; see Image quality	Path traced, from each light with castshadow
Indirect light	No, a flat ambient term	Yes
Directional and spot lights	Yes	Yes
Point lights	No	Yes
Skybox	Background	Background and environment lighting
Heightfields	Yes	No. Render terrain as a mesh
Mocap bodies	Yes	Rendered at their MJCF pose only
Cube textures on materials	Yes	First face only

Hardware

	MuJoCo OpenGL	Nyx
Platforms	All MoveIt Pro images	amd64 images with an NVIDIA or AMD ROCm GPU with Vulkan ray tracing
Frame rate	Images and depth: render_publish_rate. Point clouds and camera lidar: point_cloud_publish_rate. <rangefinder> scans: lidar_publish_rate	Set per camera with the rate launch argument, 10 Hz by default; a GPU that cannot keep up publishes slower

On AMD GPUs, Nyx uses a reference path tracer that does not render specular transmission, so skylights render black and glass loses its reflections.

Use the Physically Based Renderer

The steps below use lab_sim from the MoveIt Pro example workspace, which has the renderer switch wired in. The other MuJoCo robot configurations in the example workspace use the same switch.

Install Nyx

Nyx and the PyTorch build it needs are installed into your user workspace image, so they persist across moveit_pro run and moveit_pro build. Add this step to the base stage of your workspace's Dockerfile, directly above the # Install additional dependencies comment and its rosdep install step:

RUN --mount=type=cache,target=/root/.cache/pip \
    if [ -f /opt/nyx_shim/lib/liboidn_device_shim.so ]; then \
      /opt/nyx_shim/install_runtime.sh; \
    elif [ -f /etc/vulkan/icd.d/nvidia_icd.json ]; then \
      pip install --break-system-packages --only-binary=:all: --index-url https://download.pytorch.org/whl/cu130 torch==2.10.0+cu130 && pip install --break-system-packages --only-binary=:all: gs_nyx==0.1.3 && pip check; \
    fi

The step does nothing on an image without a supported GPU runtime. It adds several GB of PyTorch to the image, so add it only to a workspace that uses the renderer. On AMD, install_runtime.sh also replaces the image's Mesa graphics stack with Mesa 26, which MuJoCo's cameras then use too.

Then rebuild the workspace image:

moveit_pro build

Export the Scene

Nyx reads a scene exported from the same MJCF that the simulation runs. Export it once per robot configuration, and again after you change the MJCF or its assets. Open a shell in the Runtime container:

moveit_pro shell

Then export the scene to the directory that lab_sim reads it from:

ros2 run picknik_nyx_renderer nyx_export_scene \
    --mjcf $(ros2 pkg prefix --share lab_sim)/description/scene.xml \
    --out-dir $USER_WS/.nyx_scenes/lab_sim

Export on the same type of GPU (NVIDIA or AMD) that renders the scene, because the export records the path tracer and sample count for that GPU type. The exporter lists every feature it could not translate; add --strict to fail instead. On AMD, prefix the command with /opt/nyx_shim/nyx_env.sh, which applies the environment the renderer needs to a command run outside the launch file. Other options:

• --spp sets the samples per pixel. Higher values reduce noise and frame rate.
• --cam-res WxH overrides the camera resolution and keeps the vertical field of view. In nyx mode, <camera>/camera_info then no longer matches the depth image.
• --visual-groups sets the geom groups to export, 0,1,2 by default, which matches what MuJoCo draws.

A scene export can be hundreds of MB. The example workspace's .gitignore excludes .nyx_scenes/; add it to your own workspace's .gitignore.

Instrument a Robot Configuration

A robot configuration needs two changes, both driven by the MOVEIT_PRO_CAMERA_RENDERER environment variable so that they cannot disagree.

1. In the robot description, read the variable as a xacro argument and add a <sensor> block for each RGB camera inside <ros2_control>:

   • For nyx, set color_enabled to false, so MuJoCo publishes only depth and point clouds and Nyx publishes <camera>/color.
   • For both, set rgb_topic and camera_info_topic to <camera>/opengl/color and <camera>/opengl/camera_info.
   • Fail the description on any other value, so a typo does not leave the robot configuration without a camera.

   See the camera_renderer argument and the camera_renderer_sensors macro in lab_sim/description/picknik_ur.xacro.

2. In runtime.launch.xml, start one Nyx camera node per camera when the variable is nyx or both. Pass the same MJCF the simulation runs and the export directory, and set use_sim_time to match the robot configuration's ros_global_params. On a mismatch, the node misjudges the age of each simulation state: it either drops every image or keeps publishing the last pose of a simulation that has stopped. lab_sim runs on wall time, so it keeps the launch default, false. Set topic_namespace to an empty value for nyx, so the image publishes on <camera>/color, and to nyx for both.

   See lab_sim/launch/runtime.launch.xml, which includes nyx_camera.launch.py from picknik_nyx_renderer for scene_camera and wrist_camera.

The Nyx node follows the simulation through the /mujoco/qpos_stamped topic. The node stops publishing when the newest state is older than its qpos_timeout launch argument, 0.2 s by default. Keep qpos_publish_rate in the MuJoCo hardware parameters above 5 Hz, or raise qpos_timeout to match. It defaults to 60 Hz, capped at the model's step rate.

Select the Renderer

Set MOVEIT_PRO_CAMERA_RENDERER before you start MoveIt Pro, in the shell or in the workspace .env file:

MOVEIT_PRO_CAMERA_RENDERER=nyx moveit_pro run -c lab_sim

Value	<camera>/color, <camera>/camera_info	<camera>/depth, <camera>/points	<camera>/opengl/*	<camera>/nyx/*
opengl, or unset for the robot configuration's default (opengl in lab_sim)	MuJoCo	MuJoCo	None	None
nyx	Nyx	MuJoCo	None	None
both	None	MuJoCo	MuJoCo	Nyx

moveit_pro run stops on any other value. Use both only to compare the renderers side by side to help tune your renderer assets: Objectives and detectors that read <camera>/color get no images in that mode. The Desktop App camera picker lists the Nyx image as nyx/color under the camera. A camera with an absolute rgb_topic gets a separate top-level entry for the Nyx image instead.

Limitations

• In nyx mode, depth shares <camera>/camera_info with the Nyx image. If the Nyx node exits, for example because the scene was not exported, the camera has no color image and depth has no camera_info for the session.
• In nyx mode, MuJoCo stamps depth when it renders, and Nyx stamps color and <camera>/camera_info with the simulation state it rendered. An exact-time consumer of depth and camera_info, such as depth_image_proc, gets no pairs. Use <camera>/points or an approximate-time synchronizer.
• In nyx mode, color and depth do not show exactly the same scene. Nyx renders a simulation state that can be one state update plus one render older than the depth image, so the two misalign on a moving camera. Mocap bodies are at their MJCF pose in color and at their true pose in depth. <camera>/points is colored from MuJoCo's own render, not from Nyx.
• A paused simulation stops the Nyx image, because the simulation publishes its state only when it changes.
• Nyx renders on GPU device 0. On NVIDIA, set CUDA_VISIBLE_DEVICES for the Nyx node to use another GPU.
• In both mode, <camera>/depth has no camera_info beside it; <camera>/opengl/camera_info describes it.
• Both renderers compute camera_info from fovy with a centered principal point and ignore sensorsize, focal, and principal. Use fovy cameras for anything that reads camera_info.
• On AMD GPUs, Nyx can draw the skybox black for some pairs of camera clip planes. The scene geometry is not affected. The export sets the near and far planes from the MJCF's <visual><map znear zfar> values, multiplied by the model extent. If a camera shows black where MuJoCo shows the skybox, change znear or zfar slightly and export the scene again.
• Nyx images are often darker than MuJoCo's. In a scene with a skybox, Nyx lights the scene only from its lights and the skybox, while MuJoCo also adds a camera headlight and an ambient term. Across the example workspace cameras, the median Nyx image is about half a stop darker. Add lights to the MJCF where a camera needs more light.