Troubleshooting

Installation and Launch

Problem: I see this error in the logs for the web_ui container:

moveit_studio-web_ui-1        | 2023/09/26 16:23:26 [emerg] 1#1: bind() to 0.0.0.0:80 failed (98: Address already in use)
moveit_studio-web_ui-1        | nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)

Solution: Check for other processes using localhost port 80 and kill them:

lsof -i tcp:80
kill $(lsof -t -i:80)

Problem: Web app does not finish loading. The robot or other objects are not displayed.

Solution: A previous process likely did not shut down properly. Close the terminals that were previously running MoveIt Pro with Ctrl+D. Then open new terminals and launch again.

Problem: When running the MoveIt Pro Agent, the controller manager is not available and then dies:

[ERROR] [1640282574.233456180] [spawner_servo_controller]: Controller manager not available
...
[ERROR] [ros2 run controller_manager spawner --stopped joint_trajectory_controller-7]: process has died

Solution:

Remove the following line from your cyclonedds.xml file:

<AllowMulticast>false</AllowMulticast>

Problem: When building MoveIt Pro containers locally, such as when running docker compose build dev, you may run into the following errors if you are using ZFS as your host filesystem:

=> ERROR [dev ros-core 13/13] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=locked     apt-get update && apt-get install -y --no-install-  18.4s
> [dev ros-core 13/13] RUN --mount=type=cache,target=/var/cache/apt,sharing=locked     --mount=type=cache,target=/var/lib/apt,sharing=locked     apt-get update && apt-get install -y --no-install-recommends    ros-humble-ros-base:
...
16.02 Processing triggers for libc-bin (2.35-0ubuntu3.3) ...
16.04 Processing triggers for sgml-base (1.30) ...
16.07 cannot rename /var/lib/sgml-base/supercatalog to /var/lib/sgml-base/supercatalog.old: Invalid argument at /usr/sbin/update-catalog line 322.
16.07 dpkg: error processing package sgml-base (--configure):
16.07  installed sgml-base package post-installation script subprocess returned error exit status 22
16.07 Setting up libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
...
18.26 Setting up ros-humble-ros-base (0.10.0-1jammy.20230919.234834) ...
18.28 Processing triggers for libc-bin (2.35-0ubuntu3.3) ...
18.31 Errors were encountered while processing:
18.31  sgml-base
18.33 E: Sub-process /usr/bin/dpkg returned an error code (1)
...
failed to solve: process "/bin/bash -o pipefail -c apt-get update && apt-get install -y --no-install-recommends         ros-${ROS_DISTRO}-ros-base" did not complete successfully: exit code: 100

Solution:

Perform the following steps to update the docker daemon to use the ZFS storage driver:

  • Stop docker: sudo system stop docker

  • Backup the docker data: sudo cp -au /var/lib/docker /var/lib/docker.bk

  • Clear out the docker data: sudo rm -rf /var/lib/docker/*

  • Find available zpools: sudo zfs list which gives bpool and rpool on my system. If you have no existing pools jump to the next step. The pool rpool has most of the disk space, so we will add a dataset to this pool.

    Adding to an existing zpool (rpool):

  • sudo zfs create rpool/docker

  • sudo zfs set mountpoint=/var/lib/docker rpool/docker

    Adding a new zpool (Skip this if you already added to an existing zpool):

  • sudo zfs create create -f zpool-docker -m /var/lib/docker /dev/xvdf /dev/xvdg (replace the /dev/ devices appropriately).

    (Optional) Restoring backed up docker data:

  • sudo cp -a /var/lib/docker.bk /var/lib/docker

    Restart docker:

  • sudo system restart docker

Problem: When installing MoveIt Pro and building a Docker image, I see the following error:

# Ign:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
 => => # Ign:3 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease
 => => # Ign:4 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease
 => => # Ign:5 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease
 => => # Err:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
 => => #   Temporary failure resolving 'ports.ubuntu.com'
failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF

Solution:

If you are using a virtual machine (VM), e.g. through VMWare or Parallels, this can sometimes happen when the VM has been suspended for a long time. Try rebooting the VM and verify whether the issue persists.

Problem: When running the MoveIt Pro Agent with a Gazebo configuration, startup hangs when requesting the list of world names:

[ros_gz_sim]: Requesting list of world names.
[ros_gz_sim]: Requesting list of world names.

Solution:

This can be caused by the default Ubuntu firewall package blocking Gazebo’s internal multicast communication. You can work around this by temporarily disabling the firewall:

ufw disable

You can re-enable the firewall later with:

ufw enable

Problem: When starting MoveIt Pro with a Gazebo simulation, the agent fails with an Unable to open display: error.

Solution: Gazebo requires access to your display server to correctly visualize simulated scenes. Two common issues that could prevent the environment from accessing your display are:

Running MoveIt Pro from a CLI environment that does not have access to an X server, such as over an ssh connection.

The simplest solution to starting a Gazebo simulation is to launch it from a graphical desktop environment. However, if ssh is a requirement, ensure that the session has X forwarding enabled. Though there may be other issues with hardware acceleration depending on your machine setup.

Running as a non-UID 1000 user without permission to the X server in the graphical desktop.

By default, the MoveIt Pro application runs as a user with UID 1000. If this does not match your user’s UID, then you will need to explicitly give xhost permissions to the application. At each login to your host machine, run the following to ensure the application has permissions to access display resources:

xhost +local:docker

Build and Compilation

Problem:

the following packages/stacks could not have their rosdep keys resolved to system dependencies:
moveit_servo: Cannot locate rosdep definition for [control_toolbox]

Solution: you forgot to run rosdep update. Alternatively, you may need to run rosdep install:

sudo apt update
rosdep update
rosdep install --from-paths src -y --ignore-src

Problem: I get compilation errors in MoveIt 2 packages when building my workspace. Example shown below:

--- stderr: moveit_planners_ompl
/usr/bin/ld: CMakeFiles/moveit_generate_state_database.dir/scripts/generate_state_database.cpp.o: in function `main':
generate_state_database.cpp:(.text.startup+0x76f): undefined reference to `planning_scene_monitor::PlanningSceneMonitor::PlanningSceneMonitor(std::shared_ptr<rclcpp::Node> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<tf2_ros::Buffer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
collect2: error: ld returned 1 exit status
make[2]: *** [ompl_interface/CMakeFiles/moveit_generate_state_database.dir/build.make:313: ompl_interface/generate_state_database] Error 1
make[1]: *** [CMakeFiles/Makefile2:328: ompl_interface/CMakeFiles/moveit_generate_state_database.dir/all] Error 2
make: *** [Makefile:141: all] Error 2

Solution: Uninstall conflicting MoveIt 2 binary packages and reinstall MoveIt Pro workspace dependencies:

sudo apt remove --purge ros-rolling-moveit*
cd $STUDIO_WS/src
rosdep install --ignore-src --from-paths . -y -r

Problem: MoveIt Pro and other ROS processes are throwing network errors along the lines of ddsi_udp_conn_write_failed

Solution: Try restarting the ROS daemon with ros2 daemon stop, then ros2 daemon start. If that doesn’t work, try rebooting.

Using the MoveIt Studio Developer Tool

Problem: MoveIt Pro Behavior is failing to plan.

Solution: Launch the Developer RViz configuration and use the Motion Planning Tasks plugin to inspect each stage’s solution and why they are failing:

./moveit_pro rviz

Problem: Unsure how the planning scene is modeled.

Solution: Launch the Developer RViz configuration and use the Planning Scene widget view the scene robot and geometry used by MoveIt:

./moveit_pro rviz

Problem: Robot fails planning after a very short time trying.

Solution: Check that the robot isn’t in a position outside of the specified joint limits. Try to use the teach pendant to set the joints to near zero and see if that fixes things.

Problem: The robot arm got into collision during manual control and the robot cannot be moved.

Solution:

This can happen if the rate of moving the arm is greater than the rate of collision checking. To increase the rate of collision checking, the following parameters can be tweaked in the MoveIt Servo YAML configuration file in your robot configuration package:

collision_check_rate: 10.0  # [Hz] Collision-checking can easily bog down a CPU if done too often.

Restart MoveIt Pro as that will reset the robot’s position in simulation.

Problem: I am not seeing logged messages published at the Info level come through in the UI:

I have a line in my custom behavior shared_resources_->logger->publishInfoMessage(…); that I expect to run but I don’t see the logged message coming through in the UI.

Solution: Change the log level output in the UI by pressing the log level bell icon in the top right corner of the UI and checking the info level box.

../_images/log_level_bell.png ../_images/log_level_checkboxes.png

Networking and PC Setup

Problem: In a multi-PC setup, I can ping the other computer but can’t communicate with it via ros2 run demo_nodes_py talker + ros2 run demo_nodes_py listener

Solution: Check that ~/.ros/cyclonedds.xml files on each machine are pointing to the right network interface adapter, say if you changed USB ethernet adapters.

Problem: Multiple PCs are connected over the network, but the agent cannot seem to access action servers or services provided by the robot drivers, or discover controllers, resulting in errors such as:

[ERROR] [1658269925.062171689] [MoveGripperAction]: Error code: -25; Message: Failed to send trajectory execution action goal to server.; Details: Action server not available.
...
[ERROR] [1658269931.800549146] [Teleoperate]: Error code: 99999; Message: Failed to enable required controller servo_controller; Details: Service /ensure_controller_is_active failed with message: Failed to ensure that specified controllers are active.
...
[ERROR] [1658269939.054476603] [moveit_ros.trajectory_execution_manager]: Controller '/joint_trajectory_controller' is not known

Solution:

Check that the network interface specified by the environment variable CYCLONEDDS_NETWORK_INTERFACE matches the network interface specified in ~/.ros/cyclonedds.xml, and that the IP address of the network interface is listed in the environment variable CYCLONEDDS_PEER_ADDRESSES.

Problem: When using multi-machine setups, the machines’ clocks go out of sync. This can been seen by msgs coming from the future. Or TF complaining about old msgs.

Solution: To solve this, one or both of the machines should install chrony:

sudo apt-get install chrony

If any machine in the setup does not have access to NTP, the machine that has chrony installed can serve as a NTP server. Suppose the drivers RTPC does not have an internet connection, but has a direct link to the agent PC. On the main PC (usually the PC running the Agent), install chrony and then add lines to /etc/chrony/chrony.conf to allow the drivers RTPC to connect to it. For example:

# Allow drivers PC to use the agent PC as NTP server
allow 192.168.1.2  # This should be the IP address of drivers RTPC

Then restart the service:

sudo systemctl restart chrony.service

On the drivers RTPC, reconfigure timesyncd to use the agent PC as an NTP server. Modify /etc/systemd/timesyncd.conf to have:

[Time]
# Use the Agent PC as the primary NTP source
NTP=192.168.1.1  # This should be the IP address of the agent PC

And restart the service:

systemctl restart systemd-timesyncd.service

For more information on more complex installations: Setup guide

Robot Hardware and Cameras

Problem: I cannot connect to my Realsense camera.

Solution: Reconnect the Realsense and then check if you can view it in the realsense program: if the program is not on your system, follow the Realsense Drivers instructions. The expected set of realsense packages should look something like this:

$ apt list --installed | grep realsense

librealsense2-dbg/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2-dev/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2-dkms/focal,now 1.3.18-0ubuntu1 all [installed]
librealsense2-gl-dbg/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-gl-dev/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-gl/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-net-dbg/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-net-dev/focal 2.48.0-0~realsense0.4976 amd64
librealsense2-net/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-udev-rules/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
librealsense2-utils/focal,now 2.48.0-0~realsense0.4976 amd64 [installed]
librealsense2/focal,now 2.48.0-0~realsense0.4976 amd64 [installed,automatic]
ros-humble-librealsense2-dbgsym/focal 2.48.0-1focal.20210701.133935 amd64
ros-humble-librealsense2/focal,now 2.48.0-1focal.20210701.133935 amd64 [installed]
ros-humble-realsense-msgs-dbgsym/focal 2.0.8-2focal.20210630.230828 amd64
ros-humble-realsense-msgs/focal 2.0.8-2focal.20210630.230828 amd64
ros-humble-realsense2-camera-dbgsym/focal 3.2.2-1focal.20210705.140737 amd64
ros-humble-realsense2-camera-msgs-dbgsym/focal 3.2.2-1focal.20210705.135802 amd64
ros-humble-realsense2-camera-msgs/focal 3.2.2-1focal.20210705.135802 amd64
ros-humble-realsense2-camera/focal 3.2.2-1focal.20210705.140737 amd64
ros-humble-realsense2-description/focal 3.2.2-1focal.20210705.140354 amd64

Problem: Can’t use 2 Realsense video feeds after required driver update.

Solution: Use realsense_ros - v3.2.2, librealsense2 - v2.49.00. To downgrade these packages run install_librealsense.sh in moveit_studio/bin and rebuild your workspace.

Problem: One of the cameras does not come up properly when MoveIt Pro is launched:

[realsense2_camera_node-15] [RealSenseCameraNode]: Device 123456789012 is connected using a 2.1 port. Reduced performance is expected.

Solution:

  1. Verify that the cable and USB port are both rated USB 3.
  2. Unplug the cable from the camera and the computer and plug it back in.
  3. Open the realsense-viewer and perform a hardware reset by clicking “More” and selecting “Hardware Reset”.
  4. Try connecting a different cable.