Skip to main content
Version: 8

3. Physics Simulator Setup

The MoveIt Pro Simulator uses MuJoCo as its underlying simulation engine. This means in order to set up the MoveIt Pro Simulator, you must create a MuJoCo Modeling XML File (MJCF) that defines the robot and the environment. The easiest way to create this file is by starting with your existing URDF file.

This how-to guide will walk through the URDF to MJCF migration process.

Configuration package references:

MJCF references:

Create the MuJoCo config​

Create a new config based on the site config called [robot_name]_mujoco_config and create the [robot_name]_mujoco_config/description/mujoco/assets directory. The assets directory will be populated with all the collision and visual files for the robot. You can populate this directory manually, or use a script to copy the files over automatically (see below).

warning

If the visual files are currently in the .dae format, use Blender or Meshlab to convert them to .obj files. The .stl file format is also accepted by MuJoCo.

  • Note 1: Make sure the axes are aligned correctly when you export the .obj files from Blender. You may need to change the "Up Axis" to Z and the "Forward Axis" to X when you export.
  • Note 2: May need to use the obj2mjcf tool to create .obj files that have material information embedded in them. This tool can also perform convex mesh decomposition if needed.

The file structure beneath the description directory should resemble the following:

description
├── mujoco
│   ├── assets
└── robot_description.xacro

Other configuration package changes​

  • In the [robot_name]_mujoco_config/config/config.yaml file, make sure to change all package references from [robot_name]_base_config to [robot_name]_mujoco_config.

Prepare the robot description xacro​

Use the MujocoSystem plugin​

Change the xacro defining your robot's <ros2_control>/<hardware> tag to use the picknik_mujoco_ros/MujocoSystem plugin. Refer to the example in our lab_sim example here.

We recommend exposing mujoco_model as a xacro argument to enable quickly switching the robot between simulated worlds.

Optional Params​

MuJoCo Interactive Viewer​

You can launch MuJoCo's interactive viewer (in passive mode) on startup and safely use it in parallel with commands sent by your Objectives. Do this by setting the mujoco_viewer parameter to true in the above MujocoSystem plugin.

Learn more about the interactive viewer's capabilities in this video.

Hand of God

Ignore Joint Validation​

  • By default, the simulation enforces that every joint in the URDF exists in the MJCF model. You may want to skip this in cases where the joint isn't intended to be simulated, e.g. mock joints.

  • A list of joints to ignore can be specified using the ignore_joint_validation parameter.

      <ros2_control>
        <hardware>
          <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
          ...
          <param name="ignore_joint_validation">linear_x_joint linear_y_joint rotational_yaw_joint</param>
        </hardware>
      </ros2_control>

Retrieve the rendered URDF of the robot​

tip

This guide assumes you generate a URDF file for your robot from a xacro file. If you already have a URDF file, you can skip to the next section.

You can generate a rendered URDF file from the non-MuJoCo MoveIt Pro site config using the following commands.

  1. Run the non-MuJoCo MoveIt Pro site config:

    moveit_pro run -v

  2. Enter the MoveIt Pro Docker container:

    moveit_pro shell

  3. Copy the auto-generated URDF file from the /robot_description topic to a file.

    ros2 topic echo --once --field data --full-length /robot_description | head -n -1 > ~/user_ws/robot_description.urdf

Prepare the rendered URDF for conversion​

Copy your robot_description.urdf file into the [robot_name]_mujoco_config/description directory.

Manipulators with a mobile base​

warning

If you are migrating just an arm to MuJoCo, skip this section.

For mobile bases/manipulators, additional processing of the URDF is required so that the MJCF is properly generated.

In the URDF, if the root link is base_link of the robot, an additional joint needs to be added so that the MuJoCo URDF to MJCF compiler properly nests bodies in the MJCF.

In the URDF, add the following frame and joint:

<link name="base_link_reference">
<joint name="base_link_reference_to_base_link" type="floating">
  <origin rpy="0 0 0" xyz="0 0 0"/>
  <parent link="base_link_reference"/>
  <child link="base_link"/>
</joint>

Generating MuJoCo xml from the rendered URDF​

Move files into a flat directory​

MuJoCo does not have a built-in mechanism for resolving package:// URIs or relative paths. We provide a utility to locate meshes referenced in a URDF and copy them into the current directory.

Inside your moveit_pro shell terminal, run the following commands.

cd ~/user_ws/src/[robot_name]_mujoco_config/description/assets
ros2 run moveit_studio_utils_py flatten_meshes ../robot_description.urdf

If this script fails to locate the mesh files for your robot, you may need to copy into the assets directory manually. This step is necessary in order to complete the MJCF conversion. After the conversion, you may need to modify the location of the meshes in your MJCF file.

PickNik Accessories​

The picknik_accessories package contains reusable xml scenes and components. To use them with an XML in another package, the symlinks from picknik_accessories can be copied over to the package’s install folder that contains the top level XML. See here.

Compile your MuJoCo xml​

MuJoCo provides a compile utility to convert URDF to MJCF. We wrap this utility with additional post-processing functions that are often helpful. Now we will start running the conversion script and resolve some possible warnings and errors. It may be necessary to run this script several times until all errors are addressed. At the end of this step we will have the beginning of a functional mujoco model.

Inside your moveit_pro shell terminal, run the following commands.

cd ~/user_ws/src/[robot_name]_mujoco_config/description
ros2 run moveit_studio_utils_py urdf_to_mjcf robot_description.urdf

Possible WARNING: Geom with duplicate name​

warning

Geom with duplicate name '' encountered in URDF, creating an unnamed geom.

This is expected, as most URDFs do not name these tags.

<collision name="link1_collision">
  <geometry>
    <box size="1 1 1"/>
  </geometry>
</collision>

<visual name="link1_visual">
  <geometry>
    <box size="1 1 1"/>
  </geometry>
</visual>

We do not need mujoco to name the geoms so this is fine.

Possible ERROR: Error opening file​

danger

Error opening file 'temp/base_link.STL': No such file or directory

This is typically a sign you forgot to copy a file into your flat "temp" directory.

Possible ERROR: Mesh decimation​

To improve collision checking performance, MuJoCo limits the number of faces that can be in a mesh file.

danger

Error: number of faces should be between 1 and 200000 in STL file 'temp/base_link.stl'; perhaps this is an ASCII file? Element name 'body', id 0

We have a built in tool for decimating meshes to reduce the number of faces. You may need to run it multiple times for large files.

decimate_mesh base_link.stl base_link.stl

Possible ERROR: Missing inertia​

Some URDF generating tools may only generate partial dynamics information in the exported URDF. This is important information for physics engines and must be included.

Either:

  • Generate missing values in CAD software and copy over to xacro
    • Regenerate URDF in temp folder
  • If the missing values cannot be easily generated, delete the partially populated inertial tag. MuJoCo will by default provide a guess of the values via inertiafromgeom. You can use these values initially and resolve the inaccuracies later.
    • You can get a sanity check of the inertia values it generates via the Inertia Rendering capability.
    • TODO would be nice to find a way to extract these estimates, so we could have the inertia matrix, but set our own mass estimate (to a low value) to make the actuators more responsive.

Inertia Rendering

Change

<link name="left_arm_wrist_link_x"/>

To

<link name="left_arm_wrist_link_x">
   <inertial>
     <origin rpy="0 0 0" xyz="0 0 0"/>
     <mass value="0.05"/>
     <inertia ixx="1.0E-05" ixy="0" ixz="0" iyy="1.0E-05" iyz="0" izz="1.0E-05"/>
   </inertial>
 </link>

Final config contents​

When the urdf_to_mjcf script completes without errors, your mujoco_config should have the following structure:

my_robot_mujoco_config
├── description
│   ├── robot_description.urdf
│   ├── base_link.obj
│   ├── base_link.STL
│   │   └── ...
│   ├── robot.xml # generated by urdf_to_mjcf
│   └── scene.xml

Modify the MJCF file​

Gravcomp​

To simulate arms with brakes that hold the arm in place when it is not actively commanded, we add gravity compensation to robot bodies. This also makes tuning actuator strength simpler. Gravity compensation can be added to a body by adding the gravcomp="1" attribute to the <body> tag. This is done automatically by the urdf_to_mjcf script. If you don't want gravcomp on a body, you can remove it by changing the body tags in the robot_description.xml file from:

<body name="link1" gravcomp="1">
  ...
</body>

back to:

<body name="link1">
  ...
</body>

Use implicitfast integration​

The default Euler integration typically does not perform well for the instructions in this guide. Use implicitfast integration with multi contact consideration.

<mujoco>
  <option integrator="implicitfast">
    <flag multiccd="enable" />
  </option>
...

MujoCo Keyframes​

You can start the simulation with a specific keyframe (including initial joint values) as specified in the MJCF file:

  1. Capture the keyframe the qpos entry for a keyframe by launching with the MuJoCo interactive viewer, expanding the Simulation left sidebar, and clicking Copy pose.

  2. Paste the keyframe output to your MJCF file. Optionally add a name if you paste in multiple keyframes.

tip

Name all your keyframes so they are accessible by the ResetMujocoKeyframe Behavior. Name your first keframe 'defaut' to work with the Reset MuJoCo Sim Objective.

  <keyframe>
    <key
      name='default'
      qpos='0 0 0 0 0 0'
    />
    <key
      name='ones'
      qpos='1 1 1 1 1 1'
    />
  </keyframe>
  ...
</mujoco>

Some configurations may also require recording the ctrl values, but this is rare.

  1. If keyframes are specified in the MJCF, the simulation will default to the first keyframe.

  2. You can launch a specific, named keyframe by referencing it in the xacro defining your robot's <ros2_control>/<hardware> tag:

      <ros2_control>
        <hardware>
          <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
          ...
          <param name="mujoco_keyframe">ones</param>
        </hardware>
      </ros2_control>

    Alternatively, using the interactive viewer, you can load a specific keyframe by expanding the Simulation left sidebar, dragging the Key value to the index of the keyframe you want to load, and clicking Load Key.

tip

For complex scenes with many objects, the keyframe can be quite large. The starting joint values of the robot will be the last X items in the keyframe qpos, where X is the ordered list of joints in the interactive viewer's Joint right sidebar. You can use this knowledge to quickly iterate additional keyframes from an initial keyframe.

Simulated Sensors​

note

All <site> tags with a name attribute defined in the MJCF file will be picked up by the MuJoCo ros2_control plugin and published to tf2. The transforms will be published in world frame. Like a physical sensor, our simulated sensors do not know where they are mounted and do not include this transform information along with their sensor data. A controller and/or broadcaster is typically configured with the additional information of what frame the sensor has been mounted at.

Simulated Force Torque Sensor​

  1. Add a site for the FTS pose in the MJCF file. The simulator will publish the transform of the site at the rate defined by tf_publish_rate. Refer to the tcp_fts_sensor site in our lab_sim example here.
  2. Add <force> and <torque> tags referencing the site in the MJCF file. The created hardware interface will have the same name as the site. Refer to the tags referencing tcp_fts_sensor in our lab_sim example here.
  3. Optionally, add a FTS broadcaster to your ros2_control configuration yaml to publish the values to a ROS topic. Refer to the force_torque_sensor_broadcaster declaration and definition in our lab_sim example here.
  4. Optionally, add a VelocityForceController to your ros2_control configuration yaml to directly leverage the simulated hardware interface. Refer to the velocity_force_controller declaration and definition in our lab_sim example here.

Simulated Depth Camera Sensor​

  1. Add a <camera> sensor tag and a <site> tag corresponding to the camera frame and camera optical frame, respectively. Refer to the wrist_camera and wrist_camera_optical_frame tags in our lab_sim example here.

    Camera Frame Convention

    • Positive X points to right of camera.
    • Positive Y points up towards the top of the camera.
    • Positive Z points to behind the camera.

    Camera Optical Frame Convention

    The camera optical frame should be set to the <camera> position rotated 180 degrees about its x-axis.

    tip

    When using MuJoCo's interactive viewer, you can quickly obtain the <camera> and <site> definitions for a scene camera by adjusting your viewpoint and copying the equivalent camera pose using the Copy camera button under the Rendering tab on the left hand side of the viewer. You can paste the camera position in your XML description, which will contain a position (pos) and orientation (xyaxes). Be sure to add the name, fovy, mode, and resolution, as shown in the example above. To determine the camera optical frame, multiply the camera xyaxes orientation by 1 1 1 -1 -1 -1 and use the same position as the camera.

  2. Add a render_publish_rate param in the ros2_control hardware section of the robot description file. Refer to the example in our lab_sim example here.

tip

When using MuJoCo's interactive viewer, you can quickly obtain the <camera> and <site> definitions for a scene camera by adjusting your viewpoint and copying the equivalent camera pose using the Copy camera button under the Rendering tab on the left hand side of the viewer. You can paste the camera position in your XML description, which will contain a position (pos) and orientation (xyaxes). Be sure to add the name, fovy, mode, and resolution, as shown in the example above. To determine the camera optical frame, multiply the camera xyaxes orientation by 1 1 1 -1 -1 -1 and use the same position as the camera.

Simulated Lidar Sensor​

  1. Add a site for each lidar laser angle using the <replicate> tag in the MJCF along with a <rangefinder>. Ex:

      <replicate count="360" sep="-" offset="0 0 0" euler="0.0174533 0 0">
          <site name="lidar" size="0.01" pos="-0.075 0.0 0.0" quat="0.0 0.0 0.0 1.0"/>
      </replicate>
    ...
      <sensor>
        <rangefinder site="lidar" user="0.1 -3.14159 3.14159 0.0174533 0.1 6.0"/>
      </sensor>

  2. Add a lidar_publish_rate param in the ros2_control hardware section of the robot description file. Ex:

      <ros2_control>
        <hardware>
          <plugin>picknik_mujoco_ros/MujocoSystem</plugin>
          ...
          <param name="lidar_publish_rate">10</param>
        </hardware>
      </ros2_control>

Additional steps for a mobile manipulator​

If trying to migrating a URDF for a mobile base/manipulator to MuJoCo, additional steps need to be taken so that the base can move in MuJoCo.

Adding mecanum wheels to the robot base​

Currently in MoveIt Pro, we only support holonomic mobile bases. To properly support holonomic movement in MuJoCo, we generate mecanum wheels for the MJCF using this repository. As of the time of writing, we were using the latest commit off of main: 4d0fb46f363fc23fe4eb58653a6a4653e73a3c76 (add images).

Clone the repository and from the root directory, run the following:

python3 wheel_code_gen.py --link_name <wheel-name> --size <radius> <half-height> --type <type-of-wheel>

In the command, the link_name argument should designate what wheel is being generated, i.e. front_right, back_left, etc.

The size arguments should be the radius of the wheel and the half-height of the wheel, not the full height.

The type argument is important because it designates the orientation of the rollers for the mecanum wheels. For the front right and back left wheels, the type should be 0, and for the front left and back right wheels, the type should be 1.

Whenever the wheel is generated, it is saved as a wheel.xml file. Rename the file depending on what wheel it is, i.e. the front right wheel should be named front_right_wheel.xml.

Move the wheel files to the mujoco directory where the MJCF lives.

You can add a wheel to the MJCF by adding the following:

<body name="name-of-body" pos="0 0 0">
  <include file="file-name.xml" />
</body>

Set the name, pose, and file arguments and do this for each generated wheel.

Load the MJCF in the mujoco viewer to see if the mecanum wheels are loaded properly.

Ensure that there are no contacts internally between the mecanum wheels and the body of the robot. This is explained in the Possible ERROR: colliding adjacent meshes section.

The next step is to add an actuator for each mecanum wheel to the MJCF.

This can be done by adding the following to the MJCF for each wheel:

<actuator>
  <velocity name="name-of-velocity-actuator" joint="name-of-joint-to-actuate" ctrlrange="-15 15" kv="50" />
</actuator>

The joint field should match the name of the main joint in the corresponding wheel xml file. i.e.

<actuator>
  <velocity name="front_right_wheel" joint="R1_front_right_wheel_rolling_joint" ctrlrange="-15 15" kv="50" />
</actuator>

Reload the MJCF in the mujoco viewer and in the "Control" section of the right sidebar, you should see each wheel actuator listed. Try actuating each wheel to ensure that the wheels are being controlled as expected.

Simulate your robot​

Next we will use the mujoco simulate binary to start making actuation work.

simulate ~/user_ws/my_robot_mujoco_config/description/robot_description.xml

This should launch the mujoco viewer without any errors in the UI, but the robot will immediately flop and hang under its base link.

Make sure the robot is upright​

If the robot immediately flops and hangs under its base link, it is likely that gravcomp is not enabled on each body. Ensure each <body> tag contains a gravcomp="1" attribute.

In the mujoco viewer, under the Rendering tab to the left, the "Inertia" model element can be turned on to see the inertias of the various bodies. More information on where to find this toggle can be found in the Possible ERROR: Missing inertia section. If the inertias do not make physical sense, it is highly recommended that the <inertial> tag in the offending <body> be deleted, so that mujoco can estimate the inertia from the mesh.

Tune arm actuators​

Again, if your xml file only contains robot elements, and not environment elements, then it is easy to extract every joint and then make a corresponding actuator for it.

MuJoCo supports a number of different types of actuators, but for ease of getting a robotic arm up and running, we are going to use position actuators.

The urdf_to_mjcf script adds position actuators for any named joints in the MJCF file by adding the following tags.

<mujoco>
  <actuator>
    <position joint="<name-of-joint-to-actuate>" name="<name-of-actuator>" ctrlrange="-3.14 3.14" kp="1000" forcelimited="false" dampratio="1"/>
  ...
  </actuator>
</mujoco>

More information on position actuators can be found here.

The right sidebar's "Control" entry should have entries for each joint. Command some values, see how well it tracks and settles.

Control

Recommendation: kp and dampratio tuning (Experimental)​

  • Small robot arms / fingers:
    • kp: 10-100
    • dampratio: 1-1.5
  • Medium-scale limbs / manipulators:
    • kp: 100-500
    • dampratio: 0.7-1
  • Large or high-inertia systems:
    • kp: 500-2000
    • dampratio: 0.6-0.9
  • Start with 1.0 (critical damping)
    • If you want faster response, reduce dampratio (e.g. 0.5).
    • If you see oscillations or jitter, increase it toward 1.2–1.5.
    • Keep consistent units: since dampratio is used alongside kp, make sure kp is realistic for your joint scale and mass.

After making changes, Reload and re-test.

If your robot arm has an internal controller handling the PID loop (e.g. most commercial arms) or the physical modeling of the actuators is not a concern, it is recommended that kp for arm joints be set to a large value to quickly achieve behavior similar to what the onboard controller produces.

Recommendation: Remove body joint force limits​

info

If the robot is still very weak after increasing kp to >10,000, it may be because your MuJoCo joints have limits on the force that can be applied to them (by the actuators) which are too conservative. Check for "actuatorfrcrange" attributes in your joint entries. These are generated if your URDF joint entries have a non-zero effort limit, for example below:

 <joint name="feet_joint" type="revolute">
    <origin rpy="-3.1416 0 0" xyz="0 0 0.0156"/>
    <parent link="base_link"/>
    <child link="feet_link"/>
    <axis xyz="0 0 -1"/>
    <limit effort="100" lower="-3.14" upper="3.14" velocity="0"/>
  </joint>

Consider increasing or removing these “actuatorfrcrange” statements from the Xacro and XML.

Possible ERROR: colliding adjacent meshes​

If certain actuators do not move at all despite the control value sent, it could be colliding with another link. Enable "Contact Point" visualization as well as "Convex Hull" visualization. Look for a cylinder as you command the joint, which indicates a collision. Setting "Label" to "Contact" under "Rendering" will also show the names of the geoms that are under collision.

Contact Point

There are two ways to resolve collisions:
1. via exclude (body to body) tag

<mujoco>
  <contact>
    <exclude body1="base_link" body2="J1_link" />
    <exclude body1="fork" body2="J5_link" />
    <exclude body1="fork" body2="J6_link" />
    <exclude body1="fork" body2="head" />
  </contact>
</mujoco>

Note that the exclude tag is not inherited by the nested bodies, so if you want to exclude contacts for multiple nested bodies, an exclude tag needs to be created for each nested body.

2. via contype and conaffinity tags (for things attached to "world" - geom to geom)
URDF meshes connected to one another via a urdf fixed joint are combined under the same mujoco body by the compile tool. This means your base_link, and any other fixed link attached to it, are all part of the mujoco "world" body, as will be any environment asset. We still want assets in the environment to interact with the base link, so we use contype and conaffinity, which can be set per-geom instead of per-body.

<mujoco>
  <worldbody>
    <geom type="mesh" rgba="0.898039 0.917647 0.929412 1" mesh="base_link" contype="1" conaffinity="2"/>
    <body name="feet_link" pos="0 0 0.0156" quat="-3.67321e-06 -1 0 0" gravcomp="1">
      <inertial pos="3.959e-07 -0.031579 -0.19136" quat="0.694431 -0.133517 0.133511 0.694344" mass="43.789" diaginertia="0.453199 0.385 0.281311"/>
      <joint name="feet_joint" pos="0 0 0" axis="0 0 -1" range="-3.14 3.14"/>
      <geom type="mesh" rgba="0.89804 0.91765 0.92941 1" mesh="feet_link" contype="1" conaffinity="2"/>
    </body>
  </worldbody>
</mujoco>

Alternatively, we could wrap the base_link in its own body and use the exclude tag approach.

Iterate through these recommendations and you should end up with responsive actuators.

Check for meshes that were not convex hulls and are not poorly converted​

Left sidebar ---> header ??? ---> Convex Hulls
Rendering
Make sure convex hulls make sense.
Otherwise decompose - we have a guide here for that.

ROBOT_description/
  description/
   ROBOT_DETAILED.xml
    meshes/
      base_link.stl
      link1.stl # decomposed mesh
temp/
  ROBOT_DETAILED.test.urdf
  base_link.stl
  link1.stl # decomposed mesh
  ROBOT_DETAILED.xml

Copy mujoco xml and decimated/decomposed meshes back to ROBOT_description​

ROBOT_description/  
  description/  
   ROBOT_DETAILED.xml  
    meshes/  
      base_link.stl # decimated mesh  
      link1.stl # decimated mesh  
  urdf/  
    ROBOT_model.urdf.xacro # new inertia values  
    ROBOT_COMPONENT_macro.urdf.xacro # new inertia values  
  launch/  
    view_robot.launch  
    view_robot.rviz  
  CMakeLists.txt  
  package.xml  
  LICENSE  
  README.md

Set up ros2_control and MoveIt config files​

a. Make ros2_control macro xacro for urdf tags​

In a future revision, we will add a script to automatically add ros2_control tags to xacro file(s).

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro # now calls ros2_control xacro macro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py

b. Make ros2_control controllers file​

Manually create and populate with Pro controllers

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/ # or ROBOT_DETAILED_config/ if you expect to have variations of gripper type, etc
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro # now calls ros2_control xacro macro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

c. Configuring "ros2_control" for a mobile base​

To control an omnidiretional mobile base, we use the clearpath_mecanum_drive_controller. As of writing this, we are using the latest commit of the pr-add-world-frame-control-option branch: 50ff0df61042ca5d16cf5d98c386661023bd6185 (fix test).

This controller is used to control the mecanum wheels that were generated in the "Adding mecanum wheels to the robot base" section.

To use the controller, clone the repo into your user workspace and build the package.

Add the parameters under platform_velocity_controller to the controllers.yaml file.

Most of the parameters are self explanatory, except kinematics.sum_of_robot_center_projection_on_X_Y_axis.

That parameter is equal to l + w in the image below (in meters).

Control

d. Generate MoveIt SRDF ACM​

Two options:

  1. Use MoveIt Pro CLI (Recommended) - From inside the dev container:
moveit_pro dev
update_acm src/[robot_name]_mujoco_config/description/robot_description.urdf --output src/[robot_name]_mujoco_config/config/moveit/robot.srdf

Or from the host:

moveit_pro update_acm ~/user_ws/src/[robot_name]_mujoco_config/description/robot_description.urdf --output ~/user_ws/src/[robot_name]_mujoco_config/config/moveit/robot.srdf
  1. Use the MoveIt Setup Assistant - create the .setup_assistant file so it will let you save your changes. Only use the SRDF tab and only save the new SRDF file - no other changes.
🐙
ros2 run moveit_setup_assistant collisions_updater --urdf <PATH_TO_URDF> --srdf <PATH_TO_OLD_SRDF> --output out.srdf --trials 100000 --default --always --min-collision-fraction 0.99 --verbose

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf # Just ACM
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

e. Manually define groups in SRDF​

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf # Now with group definitions
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

f. Manually create ompl file​

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      ompl_planning.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

g. Manually create PoseIK kinematics file​

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      ompl_planning.yaml
      pose_ik_distance.yaml 
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

h. Create servo.yaml file​

I copy lab_sim and start manually replacing things.

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

i. Create joint_limits.yaml file​

Can probably write a script to use the extracted joints to make these.

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

Set up Pro config files​

a. Copy over ROBOT_sim boilerplate package​

~/user_ws contents:

ROBOT_sim/
  config/
    config.yaml
  description/
    meshes/
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
  .gitattributes

b. Create ROBOT_sim TASK-scene.xml​

<mujoco model="gen3_proto1">
  <option integrator="implicitfast">
    <flag multiccd="enable" />
  </option>

  <include file="ROBOT_DETAILED.xml" />

  <compiler angle="radian"/>
  <asset>
    <texture
      type="2d"
      name="groundplane"
      builtin="checker"
      mark="edge"
      rgb1="0.2 0.3 0.4"
      rgb2="0.1 0.2 0.3"
      markrgb="0.8 0.8 0.8"
      width="300"
      height="300"
    />
    <material
      name="groundplane"
      texture="groundplane"
      texuniform="true"
      texrepeat="5 5"
      reflectance="0.2"
    />
  </asset>
  <statistic center="0.3 0 0.4" extent="0.8" />
  <visual>
    <headlight diffuse="0.6 0.6 0.6" ambient="0.1 0.1 0.1" specular="0 0 0" />
    <rgba haze="0.15 0.25 0.35 1" />
    <global azimuth="120" elevation="-20" />
    <map zfar="50" />
  </visual>
  <worldbody>
    <light pos="0 0 2.5" dir="0 0 -1" directional="true" />
    <geom
      name="floor"
      pos="0 0 0"
      size="0 0 0.05"
      type="plane"
      material="groundplane"
    />
  </worldbody>
</mujoco>

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  ROBOT_DETAILED_config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

Optional - add scene geometry​

Tables, widget, doodads, etc

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    meshes/
      interesting_object.stl
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

Optional - configure scene camera​

https://docs.picknik.ai/how_to/configuration_tutorials/migrate_to_mujoco_config/#simulated-depth-camera-sensor

Optional - configure keyframes​

https://docs.picknik.ai/how_to/configuration_tutorials/migrate_to_mujoco_config/#mujoco-keyframes

c. Configure ROBOT_sim CMakeLists.txt to install ROBOT_description mujoco xml & mesh files into its description and meshes folder​

# Specify all packages we will install files from here to populate '_DIR' path
find_package(ROBOT_description REQUIRED)

...

# MuJoCo models in this package use relative paths to assets such as meshes and textures
# To simplify building relative paths to assets in other packages, we will copy those assets into this project's file structure

# Paths we will copy from - the assets we want to easily reference in our MuJoCo model
# set(PICKNIK_ACCESSORIES_PATH "${picknik_accessories_DIR}/../mujoco_assets/")
set(ROBOT_MESHES_PATH "${ROBOT_description_DIR}/../meshes/")
set(ROBOT_MODEL_PATH "${ROBOT_description_DIR}/../description/")

# Path we will copy to - the common folder in this project to copy all MuJoCo related files to
set(MUJOCO_MESHES_DIR "share/${PROJECT_NAME}/meshes/")
set(MUJOCO_MODEL_DIR "share/${PROJECT_NAME}/description/")

message(STATUS"ROBOT_MESHES_PATH ${ROBOT_MESHES_PATH}")
message(STATUS"ROBOT_MODEL_PATH ${ROBOT_MODEL_PATH}")
message(STATUS"MUJOCO_MESHES_DIR ${MUJOCO_MESHES_DIR}")
message(STATUS"MUJOCO_MODEL_DIR ${MUJOCO_MODEL_DIR}")

# Copy all contents into the installed description folder
install(DIRECTORY "${ROBOT_MESHES_PATH}"
        DESTINATION "${MUJOCO_MESHES_DIR}"
        FILES_MATCHING PATTERN "*")
install(DIRECTORY "${ROBOT_MODEL_PATH}"
        DESTINATION "${MUJOCO_MODEL_DIR}"
        FILES_MATCHING PATTERN "*")
🐙
cd ~/user_ws
colcon build --packages-select ROBOT_sim
find . -name ROBOT_DETAILED.xml
find . -name base_link.stl

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

d. Expose mujoco model settings in xacro files​

This is so that the config.yaml can be quickly updated to start a different mujoco world or tun on/off the mujoco viewer.

ROBOT_model.urdf.xacro

 <robot>
  <!-- Use $() for xacro processing that must be done at runtime (roslaunch or CLI arguments, filepaths, etc)  -->
  <!-- Use ${} for xacro processing that is not done at runtime (passing values between xacro files) -->
  <xacro:arg name="left_gripper_type" default="default_vacuum" />
  <xacro:arg name="right_gripper_type" default="default_vacuum" />
  <xacro:arg name="base_type" default="none" />
  <!-- hardware_interface options: sim, physical (TODO), or mock (TODO) -->
  <xacro:arg name="hardware_interface" default="sim" />
  <xacro:arg name="mujoco_model_package" default="rapid_rad_base_config" />
  <xacro:arg
    name="mujoco_model_relpath"
    default="description/simple-scene.xml"
  />
  <xacro:arg name="mujoco_viewer" default="true" />

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

e. Configure "hardware" dictionary entries in config.yaml​

# Baseline hardware configuration parameters for MoveIt Pro.
# [Required]
hardware:
  # Set simulated to false if you are using this as a configuration for real hardware.
  # This allows users to switch between mock and real hardware by changing a single parameter with config inheritance.
  # [Required]
  simulated: true
  # If simulated is false, launch the following file:
  # [Optional, defaults to a blank launch file if not specified]
  robot_driver_persist_launch_file:
    package: "moveit_studio_agent"
    path: "launch/blank.launch.py"

  # If the MoveIt Pro Agent should launch the ros2 controller node.
  # [Optional, default=True]
  launch_control_node: True

  # If the MoveIt Pro Agent should launch the robot state publisher.
  # This should be false if you are launching the robot state publisher as part of drivers.
  # [Optional, default=True]
  launch_robot_state_publisher: True

  # Parameters used to configure the robot description through XACRO.
  # A URDF and SRDF are both required.
  # [Required]
  robot_description:
    urdf:
      package: "factory_sim"
      path: "description/picknik_fanuc.xacro"
    srdf:
      package: "factory_sim"
      path: "config/moveit/picknik_fanuc.srdf"
    # Specify any additional parameters required for the URDF.
    # Many of these are specific to the descriptions packages, and can be customized as needed.
    # [Optional]
    urdf_params:
      - hardware_interface: "sim"
      - mujoco_model_package: "ROBOT_sim"
      - mujoco_model_relpath: "description/TASK-scene.xml"
      - mujoco_viewer: "true"

f. Configure "moveit_params" dictionary entries in config.yaml​

TODO - unclear which are required in 7.x

# Configuration files for MoveIt.
# For more information, refer to https://moveit.picknik.ai/main/doc/how_to_guides/moveit_configuration/moveit_configuration_tutorial.html
# [Required]
moveit_params:
  # Used by the Waypoint Manager to save joint states from this joint group.
  joint_group_name: "whole_body"

  joint_limits:
    package: "ROBOT_description"
    path: "config/moveit/joint_limits.yaml"
  kinematics:
    package: "ROBOT_description"
    path: "config/moveit/pose_ik_speed.yaml"
  ompl_planning:
    package: "ROBOT_description"
    path: "config/moveit/ompl_planning.yaml"
  servo:
    package: "ROBOT_description"
    path: "config/moveit/servo.yaml"
  # servo_kinematics:
  #   package: "ROBOT_description"
  #   path: "config/moveit/pose_ik_speed.yaml"
  # servo_joint_limits:
  #   package: "ROBOT_description"
  #   path: "config/moveit/joint_limits.yaml"

g. Configure "ros2_control" dictionary entries in config.yaml​

# Configuration for launching ros2_control processes.
# [Required, if using ros2_control]
ros2_control:
  config:
    package: "ROBOT_description"
    path: "config/ros2_control/controllers.yaml"
  # MoveIt Pro will load and activate these controllers at start up to ensure they are available.
  # If not specified, it is up to the user to ensure the appropriate controllers are active and available
  # for running the application.
  # [Optional, default=[]]
  controllers_active_at_startup:
    - "joint_trajectory_controller"
    - "joint_state_broadcaster"
    - "tool_attach_controller"
    - "suction_cup_controller"
  # Load but do not start these controllers so they can be activated later if needed.
  # [Optional, default=[]]
  controllers_inactive_at_startup:
    - "servo_controller"
  # Any controllers here will not be spawned by MoveIt Pro.
  # [Optional, default=[]]
  controllers_not_managed: []
  # Optionally configure remapping rules to let multiple controllers receive commands on the same topic.
  # [Optional, default=[]]
  controller_shared_topics: []

Don't forget to add the mobile base controller if required.

h. Configure "objectives" dictionary entries in config.yaml​

make blank waypoints.yaml file

# Configuration for loading behaviors and objectives.
# [Required]
objectives:
  # List of plugins for loading custom behaviors.
  # [Required]
  behavior_loader_plugins:
    # This plugin will load the core MoveIt Pro Behaviors.
    # Add additional plugin loaders as needed.
    core:
      - "moveit_studio::behaviors::CoreBehaviorsLoader"
      - "moveit_studio::behaviors::MTCCoreBehaviorsLoader"
      - "moveit_studio::behaviors::ServoBehaviorsLoader"
      - "moveit_studio::behaviors::VisionBehaviorsLoader"
  # Specify source folder for objectives
  # [Required]
  objective_library_paths:
    core:
      package_name: "ROBOT_sim"
      relative_path: "objectives"
  # Specify the location of the saved waypoints file.
  # [Required]
  waypoints_file:
    package_name: "ROBOT_sim"
    relative_path: "waypoints/waypoints.yaml"

i. Launch! 🤞🤞​

Launch pro and iteratively check terminal output for errors in finding and/or loading files.

moveit_pro build user_workspace
moveit_pro run -v -c ROBOT_sim

Configure Teleop objectives​

If the Pro UI loads and you can see the 3D visualization of your robot, then we are ready to start configuring teleoperation objectives:
https://docs.picknik.ai/how_to/configuration_tutorials/configure_custom_robot/#configuring-robot-description-and-semantics

~/user_ws contents:

Dockerfile
docker-compose.yaml
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch # from xacro
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
    clear_snapshot.xml
    disengage_grasp.xml
    engage_grasp.xml
    interpolate_to_joint_State.xml
    move_to_joint_state.xml
    move_to_pose_with_approval.xml
    request_teleoperation.xml
    reset_planning_scene.xml
    take_snapshot.xml
    teleoperation.xml
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md

Final directory contents​

Dockerfile
docker-compose.yaml
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
mujoco-3.3.1/
  bin/
  ...
ROBOT_description/
  config/
    moveit/
      groups.srdf
      joint_limits.yaml
      ompl_planning.yaml
      pose_ik_distance.yaml 
      servo.yaml
    ros2_control/
      controllers.yaml
  description/
    meshes/
      base_link.stl
      link1.stl
    ROBOT_DETAILED.xml
  urdf/
    ROBOT_model.urdf.xacro
    ROBOT_COMPONENT_macro.urdf.xacro
    ROBOT_macro.ros2_control.xacro
  launch/
    view_robot.launch
    view_robot.rviz
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
helper-scripts/
  add-gravcomp.py
  add-position-actuators.py
ROBOT_sim/
  config/
    config.yaml
  description/
    TASK-scene.xml
  launch/
    agent_bridge.launch
    moveit_pro.launch.xml
  objectives/
    clear_snapshot.xml
    disengage_grasp.xml
    engage_grasp.xml
    interpolate_to_joint_State.xml
    move_to_joint_state.xml
    move_to_pose_with_approval.xml
    request_teleoperation.xml
    reset_planning_scene.xml
    take_snapshot.xml
    teleoperation.xml
  waypoints/
    waypoints.yaml
  CMakeLists.txt
  package.xml
  LICENSE
  README.md
temp/
  ROBOT_DETAILED.test.urdf
  base_link.stl
  link1.stl
  ROBOT_DETAILED.xml

Troubleshooting​

See our Digital Twin Troubleshooting page here.

Visual Issues​

If your robot model appears oversaturated or visually "blown out" in the MoveIt Pro UI, see the Robot model looks oversaturated or blown out troubleshooting entry.