Version: 8

2. Runtime SDK & Extensions

🕒 Duration: 1-2 hours
💪 Level: Intermediate

Tutorial Overview

By the end of this tutorial, you should understand how to:

Use the SDK Python interface and standard ROS 2 CLI tools to programmatically run Objectives.
Create and customize robot configuration packages for your own robot.
Develop custom Behaviors (plugins).

While we really like the MoveIt Pro UI, in this tutorial we will show you how to use the MoveIt Pro Runtime without using much of the UI. Let's get started!

Pre-reqs

Tutorial 1, though this tutorial is significantly more advanced and you may want to skip to Tutorial 3 and 4 first. However, we've ordered it this way because many users want to jump right into the code and programmatic interfaces.
Some experience with the Linux command line.
Experience with basic code editing.
A basic understanding of the Robot Operating System (ROS). MoveIt Pro currently uses ROS 2 Humble within a Docker container, and does not require you to install it yourself.

Software dependencies:

You should have already installed MoveIt Pro.
Have your favorite code editor installed, we recommend VS Code.
We recommend an enhanced multi-window terminal tool in Ubuntu called Terminator, as well as the basic text editor gedit.
```
sudo apt install -y terminator gedit
```

note

Throughout this tutorial we will assume your user workspace is located at the standard ~/moveit_pro/moveit_pro_example_ws location.

Advanced Run Commands

MoveIt Pro provides a command-line tool called moveit_pro with multiple subcommands (or “verbs”) to manage the development and runtime environment. You can see all the verbs by running moveit_pro --help.

You should already be familiar with the moveit_pro run command from Tutorial 1.

note

There are many additional arguments for the moveit_pro run command:

--help: Print the description for all moveit_pro run arguments.
-w, --user-workspace <PATH>: Specify a user workspace path to use (overrides the configured default).
-c, --config-package <NAME>: Specify the MoveIt Pro configuration package to launch.
-n, --no-drivers: Do not launch any robot driver containers.
-o, --only-drivers: Only launch the robot driver containers.
-h, --headless: Run in headless mode (no UI front-end).
-v, --verbose: Enable verbose output for debugging.
--no-browser: Prevent automatically opening the web UI in your browser.

To get started, let's launch lab_sim without the UI in headless mode. We'll also run in verbose mode to allow you to see the full logging output.

moveit_pro run -c lab_sim -v --headless

No viewer or browser will open, but you should see a lot of output in the terminal. This is the MoveIt Pro Runtime starting up and launching the robot simulation in the background.

Using Terminator for launching multiple terminal windows

As mentioned in the pre-reqs, we recommend using Terminator to run multiple terminal windows at once. In the next few steps we will be running multiple commands in different terminal windows, so this will make it easier to follow along. Here's how to do this:

Launch Terminator from your applications menu or by running terminator in a terminal.
Right-click in the Terminator window and select "Split Horizontally" or "Split Vertically" to create a new terminal pane.
You can also use the keyboard shortcut Ctrl-Shift+O to split horizontally or Ctrl-Shift+E to split vertically.
You can navigate between the panes using Alt+Arrow keys (note: varies by OS).

Terminator Window

View your robot in Rviz

While we have built a powerful set of user interfaces and visualization tools in the MoveIt Pro UI, the runtime is fully compatible with third party tools like Rviz and Foxglove through ROS 2 interfaces. The MoveIt Pro launcher includes a verb for starting RViz with our recommended configuration.

Since we are running without the browser in the previous command, start the Rviz visualizer now:

moveit_pro rviz

Rviz Window

info

Rviz can be helpful for certain data types that MoveIt Pro does not support yet, such as viewing the full TF tree and using the MoveIt "Motion Planning Task" plugin for MTC, as we'll see in Tutorial 3.

Programmatic SDK Control to Start Objectives

As discussed above, MoveIt Pro can run as a standalone application with its own UI, or as a runtime API. Many users integrate MoveIt Pro into their own framework with their application acting as the overarching coordinator. MoveIt Pro can have its Objectives launched and communicated from external processes through programmatic SDK interfaces, as we will explore in this tutorial.

External API for MoveIt Pro

In a new terminal window, open a shell inside the container:

moveit_pro shell

You should see some fun octopuses 🐙 to welcome you into the MoveIt Pro Docker environment:

Shell Prompt

We added this to make it easier to know which environment you are connected to.

tip

Throughout this tutorial, commands that should run in the MoveIt Pro shell have a 🐙 emoji.

What is moveit_pro shell and moveit_pro dev?

Both of these commands have similar functionality, but with subtle differences.

Using moveit_pro shell opens a shell inside the MoveIt Pro runtime Docker container. Within it, all the dependencies and code you've built are sourced and ready to run.

Using moveit_pro dev you can start a runtime container that does not launch a robot configuration. You can use it a development environment, or temporary instance to install packages in. When the container stops, all runtime changes are lost except for files saved inside the container at ~/user_ws

Trigger via ROS Service

Start the example 3 Waypoints Pick and Place Objective using a ROS service: 🐙

ros2 service call /execute_objective moveit_studio_sdk_msgs/srv/ExecuteObjective "{objective_name: '3 Waypoints Pick and Place'}"

What is a ROS Service?

A ROS Service is a synchronous communication mechanism in ROS that allows a node to send a request and wait for a response, similar to calling a function. It’s ideal for short, transactional interactions that require a guaranteed reply from another node. Learn more.

You should now see the UR5 in Rviz moving between pick and place poses.

To cancel a running Objective, for example an Objective that loops forever, Ctrl-C the previous service call and run in the same shell: 🐙

ros2 service call /cancel_objective moveit_studio_sdk_msgs/srv/CancelObjective

note

Calling a new Objective while one is already running will also cancel the running Objective, and start the new one.

Trigger via Python

Next we will build a short Python script to trigger an Objective. Start by installing some dependencies into the same MoveIt Pro shell you should already have open: 🐙

sudo apt update
sudo apt install python3-tk tree -y
pip install roslibpy

Why do we have to develop within the MoveIt Pro shell?

Similar to the way Python manages multiple configurations with virtual environments, MoveIt Pro uses a Docker container with all necessary dependencies installed, including the MoveIt Pro SDK and ROS 2 Python libraries. You actually do not have to run the example Python scripts in this tutorial in the MoveIt Pro shell, but to avoid dependency issues such as Ubuntu 24.04's stricter externally-managed Python environment, we recommend a containerized approach for new users.

Install dependencies via the Dockerfile

To persist dependencies as you open and close the MoveIt Pro shell, and avoid having to install them every time you restart MoveIt Pro, you can also add dependencies directly to your Dockerfile. In your editor of choice, open the user workspace Dockerfile:

gedit ~/moveit_pro/moveit_pro_example_ws/Dockerfile

Modify line ~218 of the file, where it installs nano and other developer dependencies. Add the following line below nano:

        nano \
        python3-tk tree \
    && pip install roslibpy

Save this file and now re-build your entire Docker image, which can take quite a while:

moveit_pro build

Next create a new example Python script from your host computer (not shell):

gedit ~/moveit_pro/moveit_pro_example_ws/src/start_my_application.py

tip

Throughout this tutorial feel free to create, edit, and move files in whatever way you prefer. We will suggest various tools like gedit and mv, but you can use your favorite code editor or file manager instead - developer's choice!

Once the file is open, add this simple Python code:

start_my_application.py

Click here to copy the script directly to your clipboard.

import tkinter as tk
import roslibpy

# ROS client setup
client = roslibpy.Ros(host='localhost', port=3201)

def call_start_objective():
    if not client.is_connected:
        client.run()

    start_service = roslibpy.Service(
        client,
        '/execute_objective',
        'moveit_studio_sdk_msgs/srv/ExecuteObjective'
    )

    request = roslibpy.ServiceRequest({
        "objective_name": "3 Waypoints Pick and Place"
    })

    def success(response):
        print("Objective started:", response)

    def failure(error):
        print("Failed to start objective:", error)

    start_service.call(request, callback=success, errback=failure)

def call_cancel_objective():
    if not client.is_connected:
        client.run()

    cancel_service = roslibpy.Service(
        client,
        '/cancel_objective',
        'moveit_studio_sdk_msgs/srv/CancelObjective'
    )

    request = roslibpy.ServiceRequest({})

    def success(response):
        print("Objective cancelled:", response)

    def failure(error):
        print("Failed to cancel objective:", error)

    cancel_service.call(request, callback=success, errback=failure)

# UI setup
window = tk.Tk()
window.title("Start My Application Example")

frame = tk.Frame(window)
frame.pack(pady=20)

start_button = tk.Button(frame, text="Start", command=call_start_objective, height=2, width=20)
start_button.pack(side="left", padx=10)

stop_button = tk.Button(frame, text="Stop", command=call_cancel_objective, height=2, width=20)
stop_button.pack(side="left", padx=10)

window.mainloop()

# Clean up ROS client on close
client.terminate()

To make this application you're own, we encourage you to change the window title to something fun. Find this line in the Python script and edit it:

window.title("Start My Application Example")

Save the script.

In your shell terminal, run the script to start the example Objective again: 🐙

python3 ~/user_ws/src/start_my_application.py

You should see an example application for remotely starting and stopping a MoveIt Pro Objective.

We encourage you to play around with this Python script, changing out the Objective that it triggers, names of the buttons, or anything else.

Example Application

note

For more advanced SDK scenarios, see the how to guide.

You can now Ctrl-C the Python application to continue the tutorial, though we will come back to this application later.

Safety Checks with the Heartbeat Topic

MoveIt Pro publishes a heartbeat signal to the /objective_server_heartbeat ROS topic with the system clock and the current Objective's name, status, and runtime. This topic publishes the moveit_studio_agent_msgs/msg/ObjectiveServerStatus message.

Try it yourself in a new shell terminal: 🐙

ros2 topic echo /objective_server_heartbeat

You should see something like:

Heartbeat Command Line Output

  process_timestamp:
    sec: 562
    nanosec: 921124227
  ros_timestamp:
    sec: 1753122094
    nanosec: 477107495
  objective_name: 3 Waypoints Pick and Place
  objective_status: Cancelled
  ---

Keep echoing the heartbeat topic while also running the Python script from the prior terminal. You should see the heartbeat objective_status change when clicking Start and Stop in your Python GUI. Ctrl-C this command when you are ready.

Understanding MoveIt Pro Workspaces

In MoveIt Pro, all robot configurations, Behavior plugins, and source code are represented as industry standard ROS 2 packages in a ROS 2 workspace. We also call this the "user workspace", and you can think of it as typically a git repo that contains multiple ROS packages for your robot configuration. The user workspace is where the moveit_pro launcher will look to build and run your robot software.

Out of the box MoveIt Pro ships with various reference applications in a workspace called the moveit_pro_example_ws, which you can access from two locations:

Option 1: From your host computer:

cd ~/moveit_pro/moveit_pro_example_ws/

Option 2: From the moveit_pro_shell, which is inside a Docker container. It mounts your workspace in to the container's home folder:

moveit_pro shell
cd ~

tip

This is an important concept to understand: MoveIt Pro bind mounts your Workspace with Docker container... the same folders are accessible from two locations!

Let's see for ourselves what our workspace looks like. From your host computer terminal, run:

tree ~/moveit_pro/ -L 2

tip

If you are currently in a shell you can exit the shell simply by typing exit or Ctrl-D.

There can be a lot of noise in a MoveIt Pro workspace folder, but some of the key files are below:

Details

Key Workspace Files

Note: the following files and folders are not exhaustive and only give a high level representation.

moveit_pro_example_ws/
├── Dockerfile
├── docker-compose.yaml
├── colcon-defaults.yaml
└── src/
    ├── lab_sim/
    │── factory_sim/
    │── grinding_sim/
    │── hangar_sim/
    ├── additional_configuration_packages/
    └── external_dependencies/

In particular, within the src folder you can see a number of example simulation worlds, which we call robot configuration packages, as explored in the next section.

Here's a very simplified diagram relating these different concepts we've just reviewed:

Robot Configuration Package

A MoveIt Pro robot configuration package, often called the "robot config", is a ROS 2 package that describes your specific robot. Let's explore what is in the lab_sim robot configuration package, which is the default one we have been using so far:

tree ~/moveit_pro/moveit_pro_example_ws/src/lab_sim -L 2

You should see lots of plain text files for all the concepts we've been learning in other tutorials:

Details

Example contents of lab_sim

Note: the following files and folders are not exhaustive and only give a high level representation.

lab_sim
├── CMakeLists.txt
├── config
│   ├── config.yaml
│   ├── control
│   └── moveit
├── CONTRIBUTING.md
├── description
│   ├── LICENSE
│   ├── picknik_ur.xacro
│   ├── scene.xml
│   └── simple_scene.xml
├── launch
│   └── agent_bridge.launch.xml
├── LICENSE
├── models
│   ├── clip.onnx
│   ├── clipseg.onnx
│   ├── decoder.onnx
│   ├── l2g.onnx
│   └── sam2_hiera_large_encoder.onnx
├── objectives
│   ├── 3_waypoint_pick_and_place.xml
│   ├── add_point_cloud_to_vector.xml
│   ├── add_poses_to_mtc_task.xml
│   ├── add_waypoints_to_mtc_task.xml
│   ├── apriltag_pick_object.xml
│   ├── classical_pick_and_place.xml
│   ├── clear_previous_obstacles.xml
│   ├── close_gripper.xml
│   ├── constrained_pick_and_place_subtree.xml
│   ├── constrained_pick_place.xml
│   ├── create_pose_vector.xml
│   ├── cycle_between_waypoints.xml
│   ├── force_relaxation.xml
│   ├── fuse_multiple_views_subtree.xml
│   ├── fuse_multiple_views.xml
│   ├── get_april_tag_pose_from_image.xml
│   ├── get_candidate_grasps_subtree.xml
│   ├── get_collision_free_grasp_subtree.xml
│   ├── get_grasp_from_text_prompt_subtree.xml
│   ├── get_transform_frame_pose.xml
│   ├── grasp_planning.xml
│   ├── grasp_pose_tuning_with_april_tag.xml
│   ├── grasp_pose_using_yaml.xml
│   ├── gripper_token.dae
│   ├── joint_diagnostic.xml
│   ├── joint_trajectory.yaml
│   ├── load_and_execute_joint_trajectory.xml
│   ├── load_mesh_as_green_poincloud.xml
│   ├── load_mesh_as_red_pointcloud.xml
│   ├── look_at_table.xml
│   ├── ml_auto_grasp_object_from_clicked_point.xml
│   ├── ml_auto_grasp_object_from_text_prompt.xml
│   ├── ml_grasp_object_from_text_prompt.xml
│   ├── ml_segment_image_from_text_prompt.xml
│   ├── ml_segment_point_cloud_from_clicked_point.xml
│   ├── ml_segment_point_cloud_from_text_prompt.xml
│   ├── move_along_square.xml
│   ├── move_with_velocity_and_force.xml
│   ├── mpc_pose_tracking_with_point_cloud_avoidance.xml
│   ├── mpc_pose_tracking.xml
│   ├── my_constraints.yaml
│   ├── open_gripper.xml
│   ├── pick_and_place_example.xml
│   ├── pick_april_tag_object_with_approval.xml
│   ├── pick_from_pose_with_approval.xml
│   ├── pick_from_pose.xml
│   ├── pick_object.xml
│   ├── pick_up_cube.xml
│   ├── pill_bottle_place_poses.yaml
│   ├── place_at_pose_with_approval.xml
│   ├── place_at_pose.xml
│   ├── place_object.xml
│   ├── plan_and_save_trajectory.xml
│   ├── plan_move_to_pose.xml
│   ├── playback_square_trajectory.xml
│   ├── push_button_admittance_params.yaml
│   ├── push_button_with_a_trajectory.xml
│   ├── record_and_replay_scanning_motion.xml
│   ├── record_square_trajectory.xml
│   ├── reflect_poses_subtree.xml
│   ├── register_cad_part.xml
│   ├── relative_grasp_pose.yaml
│   ├── _scan_scene.xml
│   ├── square_traj.yaml
│   ├── take_picture_with_wrist_camera.xml
│   ├── take_snap.xml
│   └── wrist_snap.xml
├── package.xml
├── README.md
├── test
│   └── objectives_integration_test.py
└── waypoints
    └── ur_waypoints.yaml

The following is a high level description of the lab_sim folders:

config: Real-time controller settings, planner settings, camera settings, etc
description: Robot and environment (URDF and MJCF) models
launch: To start drivers/simulators, additional ROS nodes/processes, etc.
models: Machine Learning models used by the robot
objectives: XML-based Behavior Trees that make up the applications and subtrees.
test: unit and integration tests
waypoints: named joint states that can be used in Objectives

Creating a New Robot Configuration

Onboarding a new robot model from scratch is a large topic covered in this how to guide, but it is outside the scope of this 2 hour tutorial. Instead, we will base our new configuration on an existing robot configuration package using inheritance in the next section.

Inherited Robot Configuration Package

Package inheritance is a powerful tool that allows you to create and leverage reusable base robot configuration packages that contain a generic configuration of your robot, then use inherited robot config packages to add or override certain aspects. This is especially useful if you have a fleet of robots with similar but not exactly the same configurations.

Example use cases for inherited packages:

Minor hardware differences
- Differences in end effectors (lacks a gripper, multiple end-effectors options, etc)
- IP addresses, serial numbers, specific kinematic calibrations, etc
Additional Objectives, Behaviors, or sensors
- Mobile base and navigation Behaviors
- Perception and ML Behaviors

Create a new robot config using inheritance

In the following we will create our own robot configuration package called my_robot_config, that will be built on lab_sim. The inheritance structure will look like this:

note

In the above diagram, notice the robot config that connects to hardware, called picknik_ur_site_config. The simulation and hardware implementations inherit from the same base robot config, which is a generic UR5 robot configuration. This allows us to share common code and configuration between the simulation and hardware implementations.

Let's create this new robot configuration package using a Python script that will automatically create the necessary files and folders:

From your host computer go to the src directory of your workspace:
```
cd ~/moveit_pro/moveit_pro_example_ws/src
```
Use our provided script for automatically creating a new robot config. First create a new file:
```
gedit create_robot_config.py
```
Then add the following Python code:

create_robot_config.py

#!/usr/bin/env python3

import os
from pathlib import Path

def write_file(path, content):
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_text(content)
    print(f"Created: {path}")

def main():
    print("Create a new MoveIt Pro Robot Config in the current directory")

    package = input("Enter the name for the new robot config package (e.g., my_robot_config): ").strip()

    if not package:
        print("❌ Package name cannot be empty.")
        return

    base_path = Path.cwd() / package
    if base_path.exists():
        print(f"❌ Package already exists at: {base_path}")
        return

    print(f"Creating config at: {base_path}")
    print(f"")

    # Create core files
    write_file(base_path / "CMakeLists.txt", f"""\
cmake_minimum_required(VERSION 3.10)
project({package})

find_package(ament_cmake REQUIRED)

install(DIRECTORY config launch objectives
  DESTINATION share/${{PROJECT_NAME}}
)

ament_package()
""")

    write_file(base_path / "package.xml", f"""\
<?xml version="1.0"?>
<package format="3">
  <name>{package}</name>
  <version>0.0.1</version>
  <description>My blank robot config package</description>

  <maintainer email="user@example.com">Your Name</maintainer>
  <license>Apache-2.0</license>

  <buildtool_depend>ament_cmake</buildtool_depend>

  <export>
    <build_type>ament_cmake</build_type>
  </export>
</package>
""")

    write_file(base_path / "config" / "config.yaml", f"""\
based_on_package:

objectives:
  objective_library_paths:
    example_objective:
      package_name: "{package}"
      relative_path: "objectives"
""")

    write_file(base_path / "launch" / "agent_bridge.launch.xml", f"""\
<?xml version="1.0" encoding="UTF-8" ?>
<launch>
  <include file="$(find-pkg-share moveit_studio_agent)/launch/studio_agent_bridge.launch.xml"/>
</launch>
""")

    # Create empty objectives folder
    (base_path / "objectives").mkdir(parents=True, exist_ok=True)
    print(f"Created empty folder: {base_path / 'objectives'}")

    print("Blank robot config created successfully.")

if __name__ == "__main__":
    main()

Run this Python script in your src folder and input the name my_robot_config and it will automatically create the needed file structure.
```
python3 create_robot_config.py
```
Open the newly created file my_robot_config/config/config.yaml and add the following text to specify which robot config to inherit from:
```
based_on_package: lab_sim
```
Open the file my_robot_config/package.xml and add the following text after the <buildtool_depend> tag:
```
  <exec_depend>lab_sim</exec_depend>
```
After creating your new robot configuration package, you can rebuild just our specific package using:
```
moveit_pro build user_workspace --colcon-args="--packages-select my_robot_config"
```

The Build Command

The moveit_pro build command compiles and builds any customizations you make to the robot workspace, in particular plugin extensions you develop. It uses the ROS 2 standard colcon and ament tools under the hood to manage the build process.

Several command line options are available:

moveit_pro build: Build everything. Use this if you've made underlying changes to MoveIt Pro, or as a default option if you are unsure of what to rebuild.
moveit_pro build user_image: Build only the MoveIt Pro Docker overlay (user) images. Use this when you have modified the Dockerfile or added new system dependencies (e.g. via package.xml dependencies) that require rebuilding the containers.
moveit_pro build user_workspace: Build only your ROS workspace packages (your robot’s configuration package, additional ROS packages and source code dependencies, and any custom behaviors). This is fastest if you've only changed your code.

To speed up build time, you can also pass additional colcon arguments to skip or select certain packages using:

moveit_pro build user_workspace --colcon-args "--packages-select my_package"

Shut down moveit_pro if it's currently running.

How to shutdown MoveIt Pro?

There are two ways to turn off MoveIt Pro. You can Ctrl+C the terminal run command, or you can run moveit_pro down (a shortcut for docker-compose down).

Next, start MoveIt Pro using our new robot config. Let's run in verbose mode so we can see any errors that may occur while building a custom Behavior:

moveit_pro run -c my_robot_config --headless -v

And re-launch Rviz too, since all shell commands automatically shutdown when MoveIt Pro is shutdown:

moveit_pro rviz

Everything should look the same at this point, but now we have our own custom robot configuration package that inherits from the lab_sim package that will let us do further customizations.

Creating an Objective without the UI

In Tutorial 1 we learned how to make an Objective using the MoveIt Pro UI, which is our recommended approach with its debug tools and drag & drop capabilities.

However we can also create Objectives manually as an XML file through your favorite editor. There are good uses cases for doing a mix of development both in the UI and via XML.

Create a new Objective called MyApplication in your existing my_robot_config robot config:

gedit ~/moveit_pro/moveit_pro_example_ws/src/my_robot_config/objectives/my_application.xml

Copy in this very simple application into that file:

my_application.xml

<?xml version="1.0" encoding="UTF-8" ?>
<root BTCPP_format="4" main_tree_to_execute="My Application">
<!--//////////-->
<BehaviorTree
ID="My Application"
_description="Example of cycling through the min and max limits of a single joint, for hardware testing"
_favorite="true"
>
<Decorator ID="KeepRunningUntilFailure">
  <Control ID="Sequence">
    <!--If you use two waypoints that have identical joint values, except for one joint, you can use this Objective as a way to test a single joint repeatedly. This is useful for things like hardware diagnostics and hardware bringup.-->
    <SubTree
      ID="Move to Waypoint"
      waypoint_name="Wrist 2 Max"
      joint_group_name="manipulator"
      controller_names="joint_trajectory_controller"
      link_padding="0.0"
      acceleration_scale_factor="1.0"
      controller_action_server="/joint_trajectory_controller/follow_joint_trajectory"
      keep_orientation="false"
      keep_orientation_tolerance="0.05"
      seed="0"
      velocity_scale_factor="1.0"
      _collapsed="false"
    />
    <SubTree
      ID="Move to Waypoint"
      waypoint_name="Wrist 2 Min"
      joint_group_name="manipulator"
      controller_names="joint_trajectory_controller"
      link_padding="0.0"
      acceleration_scale_factor="1.0"
      controller_action_server="/joint_trajectory_controller/follow_joint_trajectory"
      keep_orientation="false"
      keep_orientation_tolerance="0.05"
      seed="0"
      velocity_scale_factor="1.0"
      _collapsed="false"
    />
  </Control>
</Decorator>
</BehaviorTree>
<TreeNodesModel>
<SubTree ID="My Application">
  <MetadataFields>
    <Metadata subcategory="Application - Basic Examples" />
  </MetadataFields>
</SubTree>
</TreeNodesModel>
</root>

Build your workspace again:

moveit_pro build user_workspace --colcon-args="--packages-select my_robot_config"

Restart MoveIt Pro, this time with the UI so we can use it in the next section:
```
moveit_pro run -c my_robot_config -v
```
From within the UI run the My Application Objective by finding it in the sidebar. Confirm that the robot starts moving.

Build A Custom Behavior Plugin

Next we will build a Behavior plugins for our my_robot_config, although it will actually be in a new package called my_behavior.

Using the MoveIt Pro UI, open your new My Application Objective up for editing.
Click the + button and then New Behavior. Name your new customer Behavior my_behavior.
Select the AsyncBehavior node type
Give the new Behavior a description of Generates a sine wave output for velocity scaling between min and max values..

note

To learn more about the different node types, see the full how-to guide on Making a New Behavior.

New Custom Behavior Screen

Click Create.
Click Done.
Switch to the command line and close MoveIt Pro again.
Then build your entire user workspace to find the new Behavior.
```
moveit_pro build user_workspace
```

warning

We can't just build the my_robot_config package like we did previously, because our new Behavior is created in a new package called my_behavior.

The new Behavior creation assistant automatically creates unit tests that check if the empty Behavior correctly builds and registers. Let's run that now to make sure everything is working correctly.
```
moveit_pro test --colcon-args="--packages-select my_behavior"
```

note

For more information on testing Behaviors, see Build, Test, and Run your Behavior.

Adding Functionality to the Behavior

Now let's add some actual functionality to this new Behavior! In this example we will have the Behavior do a simple task of changing the velocity scale factor of the robot using a sine wave, with a variable input min and max velocity range.

Open your new Behavior's .cpp file in your editor of choice. This file is located in a new package called my_behavior. It should be located here:
```
gedit ~/moveit_pro/moveit_pro_example_ws/src/my_behavior/src/my_behavior.cpp
```
Add an include at the top of the cpp file:
```
#include <cmath>
```
Review the comments on logging in the existing doWork() method. We'll use some of these later. Now replace the file's doWork() method with this implementation:

doWork()

tl::expected<bool,std::string> MyBehavior::doWork() {
  const auto ports = moveit_studio::behaviors::getRequiredInputs(getInput<double>("velocity_min"),
                                                                 getInput<double>("velocity_max"));
  if (!ports.has_value()) {
    return tl::make_unexpected("Missing inputs: "+ports.error());
  }
  const auto& [velocity_min, velocity_max] = ports.value();

  // Use a static counter to create a phase for the sine wave
  static int call_count = 0;
  call_count++;

  // Create a phase that increments with each call
  // Adjust the divisor to change how fast the sine wave oscillates
  const double phase = (call_count % 100) * 2.0 * M_PI / 100.0;  // Complete cycle every 100 calls

  // Generate sine wave
  const double sine_value = std::sin(phase);

  // Map sine wave from [-1, 1] to [velocity_min, velocity_max]
  const double amplitude = (velocity_max - velocity_min) / 2.0;
  const double offset = (velocity_max + velocity_min) / 2.0;
  const double velocity_scale_factor = offset + amplitude * sine_value;

  setOutput<double>("velocity_scale_factor", velocity_scale_factor);
  return { true };
}

Replace the file's providedPorts() function with this implementation to add input and output ports to the Behavior:

providedPorts()

BT::PortsList MyBehavior::providedPorts(){
  return {
    BT::InputPort<double>("velocity_min", 0.5, "The minimum velocity scale factor"),
    BT::InputPort<double>("velocity_max", 1.0, "The maximum velocity scale factor"),
    BT::OutputPort<double>("velocity_scale_factor", "{velocity_scale_factor}", "The sine wave output for velocity scaling")
  };
}

Build the Behavior again:

moveit_pro build user_workspace --colcon-args="--packages-select my_behavior"

Add the new Behavior to your Objective

Restart the my_robot_config workspace so that it detects the new Behavior we just created:
```
moveit_pro run -c my_robot_config -v
```
Next, go into edit mode for your My Application Objective.
Add your new Behavior to the Objective by searching for MyBehavior on the left.
Add the Behavior to the top of sequence, as pictured below:
Edit both of the Move to Waypoint subtrees to include the {velocity_scale_factor} variable as input to the port also called velocity_scale_factor. This will allow the Behavior to raise and lower the velocity of the waypoint execution motion.

Running the new Objective

Run the My Application Objective with the new changes. You should see the robot moving a single joint moving back and forth, with variable speed. This is a subtle behavior because the velocity is only changed once per cycle.
To see the effects of the velocity scaling more clearly, open the Blackboard viewer and expand the My Application row to view the live changes to that variable. It only changes once per motion cycle.

Using Toast Notifications

To show case tighter integration of your plugins with MoveIt Pro, let's publish a toast message with each updated velocity scaling factor value.

Toast Messages

A toast message in a UI is a non-intrusive, temporary notification pop-up that provides feedback to the user about an action they've taken or a system event. It's designed to be brief and self-dismissing, requiring no user interaction to close. We use these extensively in MoveIt Pro.

Open my_behavior.cpp again to edit it.

Add the following to the end the doWork() function, right before the return { true };:

shared_resources_->logger->publishInfoMessage("Velocity Scale Factor:", std::to_string(velocity_scale_factor));

Build the workspace again:

moveit_pro build user_workspace --colcon-args="--packages-select my_behavior"

Restart MoveIt Pro and run the My Application Objective again.
You should see a toast message appear in the UI with the current velocity scale factor value showing at each run. If you do not see this, you need to enable the "Info" toast notification type in the UI settings by clicking on the bell icon at the top right.

Update our UI Application

Finally to bring everything together, let's update our example Python application to start and stop My Application instead of 3 Waypoints Pick and Place.

First let's move the Python script to our new robot configuration package so it is grouped in a more logical place that can be committed to git. We need to create a src folder in the my_robot_config package first:
```
mkdir ~/moveit_pro/moveit_pro_example_ws/src/my_robot_config/src/
```

Then we move the Python script into that folder:

mv ~/moveit_pro/moveit_pro_example_ws/src/start_my_application.py ~/moveit_pro/moveit_pro_example_ws/src/my_robot_config/src/start_my_application.py

Now let's edit that file and replace the string 3 Waypoints Pick and Place with My Application.
```
gedit ~/moveit_pro/moveit_pro_example_ws/src/my_robot_config/src/start_my_application.py
```
Save the file and run it again from the MoveIt Pro shell: 🐙
```
python3 ~/user_ws/src/my_robot_config/src/start_my_application.py
```
warning
You will need to reinstall some dependencies if you have restarted moveit_pro since the last time you ran this script (unless you added them to your Dockerfile):
sudo apt update sudo apt install python3-tk tree -y pip install roslibpy
You should see the same UI application as before, but now it will start the My Application Objective instead of the 3 Waypoints Pick and Place Objective.

Summary

This tutorial introduced advanced usage of MoveIt Pro for customizing and deploying robotic applications!

We've created:

A custom robot configuration package using inheritance.
A custom Objective using XML.
A custom Behavior plugin in C++.
A custom Python application to start and stop Objectives programmatically.

We hope you enjoyed this tutorial and learned how to use the MoveIt Pro Runtime SDK to build your own robotic applications.

Tutorial Overview​

Pre-reqs​

Advanced Run Commands​

View your robot in Rviz​

Programmatic SDK Control to Start Objectives​

Trigger via ROS Service​

Trigger via Python​

Safety Checks with the Heartbeat Topic​

Understanding MoveIt Pro Workspaces​

Robot Configuration Package​

Creating a New Robot Configuration​

Inherited Robot Configuration Package​

Create a new robot config using inheritance​

Creating an Objective without the UI​

Build A Custom Behavior Plugin​

Adding Functionality to the Behavior​

Add the new Behavior to your Objective​

Running the new Objective​

Using Toast Notifications​

Update our UI Application​

Summary​

Tutorial Overview

Pre-reqs

Advanced Run Commands

View your robot in Rviz

Programmatic SDK Control to Start Objectives

Trigger via ROS Service

Trigger via Python

Safety Checks with the Heartbeat Topic

Understanding MoveIt Pro Workspaces

Robot Configuration Package

Creating a New Robot Configuration

Inherited Robot Configuration Package

Create a new robot config using inheritance

Creating an Objective without the UI

Build A Custom Behavior Plugin

Adding Functionality to the Behavior

Add the new Behavior to your Objective

Running the new Objective

Using Toast Notifications

Update our UI Application

Summary