Creating a MuJoCo Config from a URDF

To create a Mujoco world for your MoveIt Pro config to run in simulation, you will need to create a MuJoCo Modeling XML File (MJCF) that defines the robot and the environment. The easiest way to create this file is by starting with your existing URDF file.

This how-to guide will walk through the URDF to MJCF migration process.

Configuration package references:

• Kinova Gen3 MuJoCo config
• UR5e MuJoCo config

MJCF references:

• MuJoCo Menagerie

Steps

Retrieve the rendered URDF of the robot

1. Run the non-MuJoCo MoveIt Pro site config:

   moveit_pro run -v

2. Enter the MoveIt Pro docker container:

   moveit_pro shell

3. Copy the auto-generated URDF file from the /robot_description topic to a file.

   ros2 topic echo --once --field data --full-length /robot_description | head -n -1 > ~/user_ws/robot_description.urdf

4. You may also have to rectify the filename paths for the mesh files. Example:

   • From: <mesh filename="file:///home/USERNAME/user_ws/install/kortex_description/share/kortex_description/arms/gen3/7dof/meshes/base_link.dae"/>
   • To: <mesh filename="package://kortex_description/arms/gen3/7dof/meshes/base_link.dae"/>

Create the MuJoCo config

Create a new config based on the site config called [robot_name]_mujoco_config and create the [robot_name]_mujoco_config/description/mujoco directory.

Inside of the mujoco directory create and populate the following:

• assets directory:
  • Add in all of the .stl collision files for the robot and gripper.
  • If the visual files are currently in .stl or .dae files, use Blender to convert them to .obj files.
    • NOTE 1: Make sure the axes are aligned correctly when you export the obj files from blender. You may need to change the "Up Axis" to Z and the "Forward Axis" to X when you export.
    • NOTE 2: May need to use the obj2mjcf tool to create .obj files that have material information embedded in them. This tool can also perform convex mesh decomposition if needed.
• [robot_name]_mujoco.xml MJCF file:
  • You can convert a URDF to MJCF in XML via either of the following 2 methods:
    1. Using the MuJoCo binaries, which can be installed via the instructions in the MuJoCo Getting started docs: ./compile <robot_name>.urdf <robot_name>.xml
    2. Using the MuJoCo pip package, which can be installed via pip install mujoco: python3 -c "import mujoco; model = mujoco.MjModel.from_xml_path('<robot_name>.urdf'); mujoco.mj_saveLastXML('<robot_name>.xml', model)"
  • ROS 2 package resolution does not work, so you may have to do some find and replace if your URDF uses ROS style references to mesh locations.
  • Confirm all the links from the robot.urdf file are present.
  • Use .stl assets to define collision geometries and obj files to define visual geometries.
  • Add actuators and tune.
  • NOTE: You can preview your robot model and tune the actuators using python3 -m mujoco --mjcf=/path/to/robot.xml
• scene.xml MJCF file.
  • Include robot MJCF file.
  • Add sites if the simulation includes a scene camera.

The file structure beneath the description directory should resemble the following:

description
├── mujoco
│   ├── assets
│   │   ├── base_link.obj
│   │   ├── base_link.STL
│   │   └── ...
│   ├── robot.xml
│   └── scene.xml
└── robot_description.xacro

Use the MujocoSystem plugin in your robot description Xacro file

Change the xacro defining your robot's <ros2_control>/<hardware> tag to use the picknik_mujoco_ros/MujocoSystem plugin. Refer to the example in our lab_sim example here.

We recommend exposing mujoco_model as a xacro argument to enable quickly switching the robot between simulated worlds.

Optional Params

MuJoCo Interactive Viewer

You can launch MuJoCo's interactive viewer (in passive mode) on startup and safely use it in parallel with commands sent by your Objectives. Do this by setting the mujoco_viewer parameter to true in the above MujocoSystem plugin.

Learn more about the interactive viewer's capabilities in this video.

MujoCo Keyframes

You can start the simulation with a specific keyframe (including initial joint values) as specified in the MJCF file:

1. Capture the keyframe the qpos entry for a keyframe by launching with the MuJoCo interactive viewer, expanding the Simulation left sidebar, and clicking Copy pose.

2. Paste the keyframe output to your MJCF file. Optionally add a name if you paste in multiple keyframes.

     <keyframe>
       <key
         name='zeros'
         qpos='0 0 0 0 0 0'
       />
       <key
         name='ones'
         qpos='1 1 1 1 1 1'
       />
     </keyframe>
     ...
   </mujoco>

   Some configurations may also require recording the ctrl values, but this is rare.

3. If keyframes are specified in the MJCF, the simulation will default to the first keyframe.

4. You can launch a specific, named keyframe by referencing it in the xacro defining your robot's <ros2_control>/<hardware> tag:

     <ros2_control>
       <hardware>
         <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
         ...
         <param name="mujoco_keyframe">ones</param>
       </hardware>
     </ros2_control>

   Alternatively, using the interactive viewer, you can load a specific keyframe by expanding the Simulation left sidebar, dragging the Key value to the index of the keyframe you want to load, and clicking Load Key.

tip

For complex scenes with many objects, the keyframe can be quite large. The starting joint values of the robot will be the last X items in the keyframe qpos, where X is the ordered list of joints in the interactive viewer's Joint right sidebar. You can use this knowledge to quickly iterate additional keyframes from an initial keyframe.

Ignore Joint Validation

• By default, the simulation enforces that every joint in the URDF exists in the MJCF model. You may want to skip this in cases where the joint isn't intended to be simulated, e.g. mock joints.

• A list of joints to ignore can be specified using the ignore_joint_validation parameter.

    <ros2_control>
      <hardware>
        <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
        ...
        <param name="ignore_joint_validation">linear_x_joint linear_y_joint rotational_yaw_joint</param>
      </hardware>
    </ros2_control>

Simulated Sensors

note

All <site> tags with a name attribute defined in the MJCF file will be picked up by the MuJoCo ros2_control plugin and published to tf2. The transforms will be published in world frame. Like a physical sensor, our simulated sensors do not know where they are mounted and do not include this transform information along with their sensor data. A controller and/or broadcaster is typically configured with the additional information of what frame the sensor has been mounted at.

Simulated Force Torque Sensor

1. Add a site for the FTS pose in the MJCF file. The simulator will publish the transform of the site at the rate defined by tf_publish_rate. Refer to the tcp_fts_sensor site in our lab_sim example here.
2. Add <force> and <torque> tags referencing the site in the MJCF file. The created hardware interface will have the same name as the site. Refer to the tags referencing tcp_fts_sensor in our lab_sim example here.
3. Optionally, add a FTS broadcaster to your ros2_control configuration yaml to publish the values to a ROS topic. Refer to the force_torque_sensor_broadcaster declaration and definition in our lab_sim example here.
4. Optionally, add a VelocityForceController to your ros2_control configuration yaml to directly leverage the simulated hardware interface. Refer to the velocity_force_controller declaration and definition in our lab_sim example here.

Simulated Depth Camera Sensor

1. Add a <camera> sensor tag and a <site> tag corresponding to the camera frame and camera optical frame, respectively. Refer to the wrist_camera and wrist_camera_optical_frame tags in our lab_sim example here.

   

   • Positive X points to right of camera.
   • Positive Y points up towards the top of the camera.
   • Positive Z points to behind the camera.

   

   The camera optical frame should be set to the <camera> position rotated 180 degrees about its x-axis.

2. Add a render_publish_rate param in the ros2_control hardware section of the robot description file. Refer to the example in our lab_sim example here.

tip

When using MuJoCo's interactive viewer, you can quickly obtain the <camera> and <site> definitions for a scene camera by adjusting your viewpoint and copying the equivalent camera pose using the Copy camera button under the Rendering tab on the left hand side of the viewer. You can paste the camera position in your XML description, which will contain a position (pos) and orientation (xyaxes). Be sure to add the name, fovy, mode, and resolution, as shown in the example above. To determine the camera optical frame, multiply the camera xyaxes orientation by 1 1 1 -1 -1 -1 and use the same position as the camera.

Simulated Lidar Sensor

1. Add a site for each lidar laser angle using the <replicate> tag in the MJCF along with a <rangefinder>. Ex:

     <replicate count="360" sep="-" offset="0 0 0" euler="0.0174533 0 0">
         <site name="lidar" size="0.01" pos="-0.075 0.0 0.0" quat="0.0 0.0 0.0 1.0"/>
     </replicate>
   ...
     <sensor>
       <rangefinder site="lidar" user="0.1 -3.14159 3.14159 0.0174533 0.1 6.0"/>
     </sensor>

2. Add a lidar_publish_rate param in the ros2_control hardware section of the robot description file. Ex:

     <ros2_control>
       <hardware>
         <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
         ...
         <param name="lidar_publish_rate">10</param>
       </hardware>
     </ros2_control>

Other configuration package changes

• In the [robot_name]_mujoco_config/config/config.yaml file, make sure to change all package references from [robot_name]_base_config to [robot_name]_mujoco_config.

XML files in multiple packages

Mujoco does not have the equivalent of URDF’s of package:// package finding. The location of the top level XML defines the location from which all file references are made. The simulate binary does not even respect the folder structures in XML include tags, so when first building your XML it’s easiest to just put all the STL and include XMLs in the same folder.

The picknik_accessories package contains reusable xml scenes and components. To use them with an XML in another package, the symlinks from picknik_accessories can be copied over to the package’s install folder that contains the top level XML. See here.

Troubleshooting

See our Digital Twin Troubleshooting page here.