4. Runtime SDK & Extensions

🕒 Duration: 1-2 hours
💪 Level: Advanced

Tutorial Overview

In this tutorial you will create custom robot configuration packages, build new Behavior plugins, and learn how to interact with MoveIt Pro through the runtime SDK.

By the end of this tutorial, you should understand:

• How to create and customize robot configuration packages for your own robot (including inheriting from existing configurations).
• How to develop custom Behaviors (plugins)
• How to integrate external ROS 2 interfaces (like topics, services, and actions) into MoveIt Pro Behaviors.
• How to use the MoveIt Pro SDK Python interface and standard ROS 2 CLI tools to programmatically run Objectives (Behavior Trees) without the GUI.

Pre-reqs

You should have already installed MoveIt Pro. We will assume you have already completed Tutorial 3 and know how to build Behavior Trees.

MoveIt Pro Workspace

In MoveIt Pro, all robot configurations, Behaviors, and external code are represented as ROS 2 packages in a ROS workspace. This is where the moveit_pro launcher will look to build and run your robot configuration.

Below is an example of a typical MoveIt Pro workspace:

moveit_pro_example_ws/
├── Dockerfile
├── docker-compose.yaml
├── colcon-defaults.yaml
└── src/
    ├── lab_sim/
    │   ├── config/
    │   ├── launch/
    │   ├── objectives/
    │   ├── src/
    │   ├── package.xml
    │   └── CMakeLists.txt
    ├── additional_configuration_packages/
    └── external_dependencies/

An empty workspace is provided as a starting point, called moveit_pro_empty_ws, which you can fork and add your own robot configuration package to. If there are changes or additions to our recommended configuration, your fork will nudge you to sync the upstream changes.

User Workspaces

In production systems, it is often desirable to have a customized Docker build and bring-up procedure specific to that hardware. The MoveIt Pro Runtime is distributed through Docker images. When the robot starts, Docker services will launch the MoveIt Pro runtime, drivers, developer platform user interface, and/or additional Docker containers using your user image. A user image is built using the base MoveIt Pro image, your ROS 2 workspace, and MoveIt Pro configuration packages according to the steps in your Dockerfile.

The following diagram illustrates how a production system would share and build packages in the MoveIt Pro runtime:

The ROS 2 workspace can be built into the image as part of the Docker build context, mounted at runtime, or some combination of both. See our Docker Containerization page for more detail.

Robot Configuration Packages

A robot configuration package is a ROS 2 package that describes your robot and its capabilities.

It defines:

• Robot and environment (URDF and MJCF) models
• Planner settings (motion planners, inverse kinematics solvers, allowable collisions, etc.)
• Real-time Controller settings
• Camera settings
• Objectives (Applications)
• Saved waypoints
• Launch files: To start drivers/simulators, additional ROS nodes/processes, etc.
• Custom Behaviors (C++ source files)

Creating a New Robot Configuration

Creating a custom robot configuration package from scratch is a huge topic covered in this how to guide, but it is outside the scope of this tutorial. Instead we will "fork" an existing package through inheritance.

Inherited Robot Configuration Packages

Package inheritance is a powerful tool that allows you to create base robot configuration packages that contain a generic configuration of your robot, then use inherited robot config packages to add or override certain aspects.

Some example use cases for inherited packages:

• Minor hardware differences
  • Differences in end effectors (lacks a gripper, or the robot has multiple end-effectors to choose from)
  • IP addresses, serial numbers, specific kinematic calibrations, or other unique details
• Additional Objectives, Behaviors, or sensors
  • Mobile base and navigation Behaviors
  • Perception and ML Behaviors

tip

An example inheritance structure:

Create a new robot config based on an existing config

In the following we will create a robot configuration package (called example_config) that will be built on an existing base robot config package (in this case, lab_sim):

tip

We will assume your user workspace is located at the standard ~/moveit_pro/moveit_pro_example_ws location. You can modify the below paths for any location, however.

1. First we begin with a config.yaml. Lets create it now:

   mkdir -p ~/moveit_pro/moveit_pro_example_ws/src/example_config/config
   touch ~/moveit_pro/moveit_pro_example_ws/src/example_config/config/config.yaml

2. In the editor of your choice, add the following to config.yaml:

   based_on_package: lab_sim

3. Navigate to the top level of your package

   cd ~/moveit_pro/moveit_pro_example_ws/src/example_config

4. Add three files: CMakeLists.txt, package.xml, and launch/agent_bridge.launch.xml with the following contents:

   CMakeLists.txt

   cmake_minimum_required(VERSION 3.8)
   project(example_config)
   find_package(ament_cmake REQUIRED)
   # Add waypoints, Objectives, and other folders here if you want to override them
   install(DIRECTORY config launch DESTINATION share/${PROJECT_NAME})
   ament_package()
   package.xml

   <?xml version="1.0"?>
   <package format="3">
     <name>example_config</name>
     <version>0.1.0</version>
     <description>Example derived MoveIt Pro config package</description>
     <maintainer email="support@picknik.ai">MoveIt Pro Maintainer</maintainer>
     <license>BSD-3-Clause</license>
     <buildtool_depend>ament_cmake</buildtool_depend>
     <exec_depend>lab_sim</exec_depend>
     <export>
       <build_type>ament_cmake</build_type>
     </export>
   </package>
   launch/agent_bridge.launch.xml

   <?xml version="1.0" encoding="UTF-8" ?>
   <launch>
     <include file="$(find-pkg-share moveit_studio_agent)/launch/studio_agent_bridge.launch.xml"/>
   </launch>

After creating your new robot configuration package, you can rebuild and launch using:

moveit_pro build user_workspace
moveit_pro run -c example_config

When MoveIt Pro opens, you shouldn't really see any differences to the normal lab_sim world, yet.

Override the Objectives folder

In order to create new Objectives specifically in this example_config, we first need to override that folder:

1. Create the objectives folder:

mkdir -p ~/moveit_pro/moveit_pro_example_ws/src/example_config/objectives

2. Add to your config.yaml:

config.yaml

objectives:
  objective_library_paths:
    example_objective:
      package_name: "example_config"
      relative_path: "objectives"

3. Update the install statement in your CMakeLists.txt to look like this:

CMakeLists.txt

    install(DIRECTORY config launch objectives DESTINATION share/${PROJECT_NAME})

note

Make sure colcon builds your objectives folder (it may not get symlinked to your install folder if it is empty)!

Developing Custom Behavior Plugins

Next we will build some example Behaviors plugins in our example_config.

tip

In the Creating Custom Behaviors how to guide, more details for creating MoveIt Pro Behaviors are covered.

Using the graphical interface or command line, create a new Behavior and select the AsyncBehavior type.

Open your new Behavior's .cpp file and modify its doWork() method with this implementation:

doWork()

tl::expected<bool,std::string> MyBehavior::doWork() {
  auto ports = moveit_studio::behaviors::getRequiredInputs(getInput<int>("int_1"),
                                                            getInput<int>("int_2"));
  if (!ports.has_value()) {
    return tl::make_unexpected("Missing inputs: "+ports.error());
  }
  const auto& [a,b] = ports.value();
  setOutput<int>("int_out", a + b);
  return { true };
}

In the providePorts() function add input and output ports to the Behavior:

providedPorts()

BT::PortsList MyBehavior::providedPorts(){
  return {
    BT::InputPort<int>("int_1",0,"The first number"),
    BT::InputPort<int>("int_2",0,"The second number"),
    BT::OutputPort<int>("int_out","The result")
  };
}

Now, when we run our Behavior, it will add two blackboard integers for us.

tip

Try out blackboard introspection to see your result

Adding Multiple Behavior Plugins in a Single Package

By default, each new Behavior gets put into its own ROS package, but it is often more convenient to have a collection of associated Behavior plugins in the same ROS package. In this section we walk you through manually adding a new Behavior.

To add additional Behaviors to your package, we must create the relevant .cpp and .hpp files and register the Behavior in register_behaviors.cpp. Lets create a new Behavior to add two blackboard values using an external ROS service:

service_client.hpp

#pragma once

#include <example_interfaces/srv/add_two_ints.hpp>
#include <moveit_studio_behavior_interface/service_client_behavior_base.hpp>

using moveit_studio::behaviors::BehaviorContext;
using moveit_studio::behaviors::ServiceClientBehaviorBase;
using AddTwoInts = example_interfaces::srv::AddTwoInts;

namespace my_behavior
{
class MyAddTwoIntsServiceClient final : public ServiceClientBehaviorBase<AddTwoInts>
{
public:
  MyAddTwoIntsServiceClient(const std::string& name, const BT::NodeConfiguration& config,
                                 const std::shared_ptr<BehaviorContext>& shared_resources);

  /** @brief Implementation of the required providedPorts() function for the hello_world Behavior. */
  static BT::PortsList providedPorts();

  /**
   * @brief Implementation of the metadata() function for displaying metadata, such as Behavior description and
   * subcategory, in the MoveIt Studio Developer Tool.
   * @return A BT::KeyValueVector containing the Behavior metadata.
   */
  static BT::KeyValueVector metadata();

private:
  /** @brief User-provided function to get the name of the service when initializing the service client. */
  tl::expected<std::string, std::string> getServiceName() override;

  /**
   * @brief User-provided function to create the service request.
   * @return Returns a service request message. If not successful, returns an error message. Note that the criteria for
   * success or failure is defined by the user's implementation of this function.
   */
  tl::expected<AddTwoInts::Request, std::string> createRequest() override;

  /** @brief Optional user-provided function to process the service response after the service has finished. */
  tl::expected<bool, std::string> processResponse(const AddTwoInts::Response& response) override;

  /** @brief Classes derived from AsyncBehaviorBase must implement getFuture() so that it returns a shared_future class member */
  std::shared_future<tl::expected<bool, std::string>>& getFuture() override
  {
    return future_;
  }

  /** @brief Classes derived from AsyncBehaviorBase must have this shared_future as a class member */
  std::shared_future<tl::expected<bool, std::string>> future_;
};
}  // namespace my_behavior

service_client.cpp

#include <my_behavior/service_client.hpp>

// Include the template implementation for GetMessageFromTopicBehaviorBase<T>.
#include <moveit_studio_behavior_interface/impl/service_client_behavior_base_impl.hpp>
namespace my_behavior
{
MyAddTwoIntsServiceClient::MyAddTwoIntsServiceClient(
    const std::string& name, const BT::NodeConfiguration& config,
    const std::shared_ptr<moveit_studio::behaviors::BehaviorContext>& shared_resources)
  : ServiceClientBehaviorBase<example_interfaces::srv::AddTwoInts>(name, config, shared_resources)
{
}

BT::PortsList MyAddTwoIntsServiceClient::providedPorts()
{
  // This node has three input ports and one output port
  return BT::PortsList({
      BT::InputPort<std::string>("service_name", "/add_two_ints", "The name of the service to call."),
      BT::InputPort<int>("addend1", "The first int to add to the other."),
      BT::InputPort<int>("addend2", "The second int to add to the other."),
      BT::OutputPort<int>("result", "{result}", "Result of the AddTwoInts service."),
  });
}

BT::KeyValueVector MyAddTwoIntsServiceClient::metadata()
{
  return { { "subcategory", "Example Behaviors" },
           { "description", "Calls a service to add two integers and makes the result available on an output port." } };
}

tl::expected<std::string, std::string> MyAddTwoIntsServiceClient::getServiceName()
{
  const auto service_name = getInput<std::string>("service_name");
  if (const auto error = moveit_studio::behaviors::maybe_error(service_name))
  {
    return tl::make_unexpected("Failed to get required value from input data port: " + error.value());
  }
  return service_name.value();
}

tl::expected<AddTwoInts::Request, std::string> MyAddTwoIntsServiceClient::createRequest()
{
  const auto a = getInput<int>("addend1");
  const auto b = getInput<int>("addend2");
  if (const auto error = moveit_studio::behaviors::maybe_error(a, b))
  {
    return tl::make_unexpected("Failed to get required value from input data port: " + error.value());
  }
  return example_interfaces::build<AddTwoInts::Request>().a(a.value()).b(b.value());
}

tl::expected<bool, std::string> MyAddTwoIntsServiceClient::processResponse(const AddTwoInts::Response& response)
{
  setOutput<int>("result", response.sum);
  return { true };
}
}  // namespace my_behavior

tip

To publish a toast message with your result, add the following to your processResponse() method:

shared_resources_->logger->publishInfoMessage("Sum:", std::to_string(response.sum));

Register the Behavior in register_behaviors.cpp:

moveit_studio::behaviors::registerBehavior<MyAddTwoIntsServiceClient>(factory, "MyServiceClient", shared_resources);

Click to reference the full register_behaviors.cpp

#include <behaviortree_cpp/bt_factory.h>
#include <moveit_studio_behavior_interface/behavior_context.hpp>
#include <moveit_studio_behavior_interface/shared_resources_node_loader.hpp>

#include <my_behavior/my_behavior.hpp>
#include <my_behavior/service_client.hpp>

#include <pluginlib/class_list_macros.hpp>

namespace my_behavior
{
class MyBehaviorBehaviorsLoader : public moveit_studio::behaviors::SharedResourcesNodeLoaderBase
{
public:
  void registerBehaviors(BT::BehaviorTreeFactory& factory,
    [[maybe_unused]] const std::shared_ptr<moveit_studio::behaviors::BehaviorContext>& shared_resources) override
  {
    moveit_studio::behaviors::registerBehavior<MyBehavior>(factory, "MyBehavior", shared_resources);
    moveit_studio::behaviors::registerBehavior<MyAddTwoIntsServiceClient>(factory, "MyServiceClient", shared_resources);
  }
};
}  // namespace my_behavior

PLUGINLIB_EXPORT_CLASS(my_behavior::MyBehaviorBehaviorsLoader,
                       moveit_studio::behaviors::SharedResourcesNodeLoaderBase);

Update CMakeLists.txt to build your new Behavior:

set(THIS_PACKAGE_INCLUDE_DEPENDS moveit_studio_behavior_interface pluginlib example_interfaces)
add_library(
  my_behavior
  SHARED
  src/my_behavior.cpp
  src/service_client.cpp
  src/register_behaviors.cpp)

Click to reference the full CMakeLists.txt

cmake_minimum_required(VERSION 3.22)
project(my_behavior CXX)

find_package(moveit_studio_common REQUIRED)
moveit_studio_package()

set(THIS_PACKAGE_INCLUDE_DEPENDS moveit_studio_behavior_interface pluginlib example_interfaces)
foreach(package IN ITEMS ${THIS_PACKAGE_INCLUDE_DEPENDS})
  find_package(${package} REQUIRED)
endforeach()

add_library(
  my_behavior
  SHARED
  src/my_behavior.cpp
  src/service_client.cpp
  src/register_behaviors.cpp)
target_include_directories(
  my_behavior
  PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
         $<INSTALL_INTERFACE:include>)
ament_target_dependencies(my_behavior
                          ${THIS_PACKAGE_INCLUDE_DEPENDS})

# Install Libraries
install(
  TARGETS my_behavior
  EXPORT my_behaviorTargets
  ARCHIVE DESTINATION lib
  LIBRARY DESTINATION lib
  RUNTIME DESTINATION bin
  INCLUDES
  DESTINATION include)

if(BUILD_TESTING)
  moveit_pro_behavior_test(my_behavior)
endif()

# Export the behavior plugins defined in this package so they are available to
# plugin loaders that load the behavior base class library from the
# moveit_studio_behavior package.
pluginlib_export_plugin_description_file(
  moveit_studio_behavior_interface my_behavior_plugin_description.xml)

ament_export_targets(my_behaviorTargets HAS_LIBRARY_TARGET)
ament_export_dependencies(${THIS_PACKAGE_INCLUDE_DEPENDS})
ament_package()

Update package.xml with our new dependency:

<depend>example_interfaces</depend>

Click to reference the full package.xml

<?xml version="1.0" encoding="utf-8" ?>
<package format="3">
  <name>my_behavior</name>
  <version>0.0.0</version>
  <description>My description</description>

  <maintainer email="support@picknik.ai">
    MoveIt Pro User
  </maintainer>
  <author email="support@picknik.ai">
    MoveIt Pro User
  </author>

  <license>TODO</license>

  <buildtool_depend>ament_cmake</buildtool_depend>

  <build_depend>moveit_studio_common</build_depend>

  <depend>moveit_studio_behavior_interface</depend>
  <depend>example_interfaces</depend>

  <test_depend>ament_lint_auto</test_depend>
  <test_depend>ament_cmake_gtest</test_depend>

  <export>
    <build_type>ament_cmake</build_type>
  </export>
  <buildtool_depend>python3-colcon-common-extensions</buildtool_depend>
</package>

Communicating With External ROS Interfaces

Next we will test out our previously built custom Behavior.

You will need to install an example service server in your Docker container using the following command (while MoveIt Pro is running):

moveit_pro shell
sudo apt update && sudo apt install ros-humble-examples-rclcpp-minimal-service

Then run in that same shell:

ros2 run examples_rclcpp_minimal_service service_main

tip

The above steps only install packages in the running instance of MoveIt Pro. When you end the session (moveit_pro down) the changes will not persist. To make them permanent, install them in your production image by adding this to your Dockerfile:

Click to expand

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
   --mount=type=cache,target=/var/lib/apt,sharing=locked \
   --mount=type=bind,target=${USER_WS}/,source=. \
   . /opt/overlay_ws/install/setup.sh && \
   apt-get update && \
   rosdep install -q -y \
     --from-paths src \
     --ignore-src && \
   apt-get install -y --no-install-recommends \
     ros-humble-examples-rclcpp-minimal-service ros-humble-ros2action

tip

Check out the Behavior Specialization section for more examples of specialized Behaviors to communicate with ROS interfaces like topics, services, and actions.

Programmatic SDK Control

While MoveIt Pro can run as a standalone application with its own UI, or as simple a runtime API. Many users integrate MoveIt Pro into their own framework with their application acting as the overarching coordinator.

While the MoveIt Pro UI is useful for building, editing, and executing Objectives, it is by no means required. MoveIt Pro can be run headless and its Objectives can be launched and communicated from external processes through ROS interfaces.

Using the MoveIt Pro SDK to Start Objectives

In this section, we will use the SDK to remotely start an Objective without the UI. As an example, we will trigger the 3 Waypoints Pick and Place Objective that ships in lab_sim. While you can do this with Python, in this example we will use the command line.

First we need to add the MoveIt Pro SDK repository as a git submodule to the moveit_pro_example_ws that you already have downloaded:

cd ~/moveit_pro/moveit_pro_example_ws/src/external_dependencies
git submodule add https://github.com/PickNikRobotics/moveit_studio_sdk.git

Then we will build our user image and launch MoveIt Pro:

moveit_pro build
moveit_pro run -c lab_sim

And from a separate terminal window (shell) start the Objective using a ROS service:

moveit_pro shell
ros2 service call /execute_objective moveit_studio_sdk_msgs/srv/ExecuteObjective "{objective_name: '3 Waypoints Pick and Place'}"

To cancel a running Objective, for example an Objective that loops forever like 3 Waypoints Pick and Place, run:

ros2 service call /cancel_objective moveit_studio_sdk_msgs/srv/CancelObjective

Using the MoveIt Pro SDK to Execute Behavior Tree XML

MoveIt Pro also supports your external application programmatically generating your own Behavior Tree XML, and then sending it to the MoveIt Pro runtime to execute it. You can send the entire XML contents programatically via an action server.

note

While Python examples are provided here, for now we will use the command line.

First, start MoveIt Pro and in a separate terminal enter a shell (moveit_pro shell):

Install the action client interface in the shell instance:

sudo apt update && sudo apt install ros-humble-ros2action

Send the action request with a trivial Behavior Tree XML example:

ros2 action send_goal /do_objective moveit_studio_sdk_msgs/action/DoObjectiveSequence '
objective_xml_string: |
  <root BTCPP_format="4" main_tree_to_execute="test simple">
    <BehaviorTree ID="test simple">
      <Control ID="Sequence" name="TopLevelSequence">
        <Action ID="AlwaysSuccess"/>
      </Control>
    </BehaviorTree>
  </root>
'

You should get the following result from the action, indicating it was successful:

Goal finished with status: SUCCEEDED

Waiting for an action server to become available...
Sending goal:
     objective_name: ''
objective_xml_string: "<root BTCPP_format=\"4\" main_tree_to_execute=\"test simple\">\n  <BehaviorTree ID=\"test simple\">\n    <Control ID=\"Sequence\" name=\"TopLevelSequence\">\n      <Action ID=\"AlwaysSuccess\"/>\n    </Control>\n  </BehaviorTree>\n</root>\n"
parameter_overrides: []

Goal accepted with ID: d5f48811588240f19a3ab303ccdc78ce

Result:
    error_code:
  val: 1
  message: ''
  source: ''
error_message: ''

Goal finished with status: SUCCEEDED

Nice work!

Summary

This tutorial introduced advanced usage of MoveIt Pro for customizing and deploying robotic applications. You learned how to use the moveit_pro CLI, create and extend robot configuration packages, develop custom Behaviors, and integrate with external ROS interfaces. You also saw how to run Objectives programmatically using the MoveIt Pro SDK—equipping you with the tools to build and deploy production-grade robotic systems.