Skip to main content
Version: 10

Tutorial 2 - Build with an AI Agent and MCP

Required Version
This feature requires MoveIt Pro version 10.2 or newer.

๐Ÿ’ช Level: Intermediate

Tutorial Overviewโ€‹

This tutorial teaches the recommended extension and integration workflow using an AI agent and the release-matched MoveIt Pro MCP server. For the manual workflow using the CLI, Runtime SDK, and C++, follow Tutorial 5.

By the end, you will understand how to:

  • Select and inspect one MoveIt Pro workspace, robot configuration package, and Runtime.
  • Use robot configuration package inheritance and identify the files that define it.
  • Create, build, test, and inspect a custom C++ Behavior plugin.
  • Author and validate an Objective that connects Behavior ports through the blackboard.
  • Start, observe, and cancel an Objective through MCP and the Runtime SDK.
  • Distinguish persistent Objective edits from per-run parameter overrides.

The agent performs MoveIt Pro operations through MCP. The same workflow works with simulated and physical robot configuration packages. This tutorial uses lab_sim so you can focus on the development flow.

Prerequisitesโ€‹

  • Complete Tutorial 1.
  • Install MoveIt Pro 10.2 or later.
  • Configure the MoveIt Pro CLI for the standard example workspace at ~/moveit_pro/moveit_pro_example_ws, the installer default.
  • Install one supported AI client.
  • Install the MoveIt Pro Desktop App and keep it available for visual checkpoints.

Use a local or SSH host-user MCP connection. Shared HTTPS connections provide inspection tools and cannot create files, build, launch, or command motion.

Connect Your AI Agentโ€‹

For everyday use, register moveit_pro mcp serve without target options. For example, Claude Code connects with:

claude mcp add --scope user moveit-pro -- moveit_pro mcp serve

Each new session selects the running Runtime, or uses your MoveIt Pro CLI configuration when none is running. If several Runtimes are running, the client's working directory can select a workspace; add --instance <name> if the target is still ambiguous. A session keeps its target until you reconnect. See MCP connection setup for other clients.

For this tutorial, use one of the commands below instead. The two extra options select the new robot configuration package (--config-package mcp_walkthrough_sim) and its dedicated Runtime (--instance mcp-walkthrough). The agent creates that package later, so it cannot be selected from a running Runtime yet. The workspace comes from your MoveIt Pro CLI configuration. If that is not the tutorial workspace, add --workspace "$HOME/moveit_pro/moveit_pro_example_ws".

  1. Run:

    claude mcp add --scope user moveit-pro -- moveit_pro mcp serve --config-package mcp_walkthrough_sim --instance mcp-walkthrough
  2. Enter /mcp in Claude Code and confirm that moveit-pro is connected.

If your AI agent runs on another computer, follow Connect over SSH, keeping this tutorial's --config-package mcp_walkthrough_sim and --instance mcp-walkthrough options on the remote moveit_pro mcp serve command, then return here.

Understand the Development Loopโ€‹

The MCP connection selects the target for every agent operation:

SettingTutorial valuePurpose
Workspace~/moveit_pro/moveit_pro_example_wsROS packages and source code
Robot configuration packagemcp_walkthrough_simRobot, environment, Objectives, and extensions
Instancemcp-walkthroughRuntime processes, ports, logs, and execution state

The loop is:

If a step fails, send this prompt before changing anything:

Diagnose why this MoveIt Pro target is not ready. Inspect its capabilities, job output, launch output, and recent logs. Identify the first causal failure and recommend one next action. Do not start or stop another Runtime.

To stop this tutorial's named Runtime, first confirm that no Objective is executing, then run this on the Runtime computer:

moveit_pro down --instance mcp-walkthrough

Know the Main CLI Commandsโ€‹

MCP handles the routine workflow, but these commands remain useful for direct development and recovery:

CommandUse
moveit_pro run --helpList launch options
moveit_pro run -w <workspace> -c <package> -vLaunch a robot configuration package with verbose output
moveit_pro run -c <package> --instance <name>Launch an isolated named Runtime
moveit_pro run --list-instancesShow named instances and their endpoints
moveit_pro shell --instance <name>Enter a running Runtime container with its workspace sourced
moveit_pro devStart a development container without launching a robot configuration package
moveit_pro rvizOpen the legacy RViz debug view for the default deployment

This tutorial uses a named Runtime and the MoveIt Pro Desktop App. Run moveit_pro --help to discover other commands.

Inspect the Targetโ€‹

Send:

Inspect the selected MoveIt Pro target without changing it. Confirm the workspace, robot configuration package, Runtime instance, identity, permissions, and readiness.

On a first run, check that:

  • The target identifies mcp-walkthrough and mcp_walkthrough_sim.
  • The principal type is local_os_user.
  • The scopes include runtime:config, runtime:build, runtime:test, runtime:launch, and runtime:execute.
  • WORKSPACE, VALIDATOR, LOGS, TOOLCHAIN, and HOST_CONTROL are ready.
  • The active configuration and Runtime are unavailable because the destination package has not been created or launched.

Ignore a suggestion to build the missing destination package at this point. Create it in the next step.

On a fresh workspace or after upgrading MoveIt Pro, build the existing workspace once before continuing:

Build the entire bound workspace and follow the job to completion. Report the first actionable failure.

Later steps select only the package that changed.

Create the Robot Configuration Packageโ€‹

Send:

Create the bound mcp_walkthrough_sim robot configuration package based on lab_sim. Do not overwrite an existing package. Explain what it inherits and stop before building.

Open src/mcp_walkthrough_sim/config/config.yaml and find:

based_on_package: "lab_sim"
What just happened

The agent created a ROS package in the bound workspace. Its config.yaml inherits the UR5 robot, simulated lab, controllers, Objectives, and Waypoints from lab_sim. Files in mcp_walkthrough_sim can override only the parts that differ.

Build and launch it:

Build only mcp_walkthrough_sim, follow the job to completion, then launch the bound Runtime and wait until it is ready. Stop and report the first failure.

The package-scoped build passes --packages-up-to mcp_walkthrough_sim to colcon, including its workspace dependencies. The Runtime now has the inherited configuration but no custom Behavior yet.

Connect the Desktop Appโ€‹

On the Runtime computer, run:

moveit_pro run --list-instances
moveit_pro frontend-key

Open the MoveIt Pro Desktop App, select Add Connection, and enter the address, REST API port listed for mcp-walkthrough, and frontend key. See Connect to a Runtime for details.

Confirm that the UR5, lab scene, Objectives, and Waypoints match lab_sim. Keep the app open for the remaining checkpoints.

The lab_sim scene and UR5 in the MoveIt Pro Desktop App

Create a Custom Behaviorโ€‹

Send:

Create a package named mcp_tutorial_behavior with a synchronous C++ Behavior class named TutorialVelocityScale and the exact registered Behavior ID Tutorial Velocity Scale. Add required double inputs velocity_min and velocity_max and a double output velocity_scale_factor. Alternate between the limits on each tick, reject velocity_min greater than velocity_max, write the selected value to the output port, and publish it as an info notification. Add focused tests for the limits and alternating output. Stop before building.

Record the exact Behavior ID returned by the agent. You will use it in the Objective.

What just happened

The agent used the release-matched Behavior authoring guide and package template. Open these generated files:

  • The header declares the ports in providedPorts().
  • src/register_behaviors.cpp registers the exact Behavior ID.
  • behavior_plugin.yaml describes the plugin to MoveIt Pro.
  • The test source checks the Behavior logic; the template also checks registration.

MoveIt Pro merges behavior_plugin.yaml from packages in the workspace, so mcp_walkthrough_sim/config/config.yaml does not need a plugin entry.

Build and test the new package:

Build only mcp_tutorial_behavior, then test only that package. Follow both jobs to completion and report the first actionable failure.

The build uses --packages-up-to mcp_tutorial_behavior, including workspace dependencies. The test job also uses --packages-select mcp_tutorial_behavior so only that package is tested.

Restart the Runtime and Find the Behaviorโ€‹

A running Runtime does not hot-load a newly compiled C++ plugin. Confirm that no Objective is executing, then run:

moveit_pro down --instance mcp-walkthrough

Send:

Launch the bound Runtime and wait until it is ready. Find the exact registered Behavior ID returned during creation and report its description, node type, and ports. Do not run motion.

Compare the reported ports with providedPorts() in the generated header. In the Desktop App Objective editor, search for Tutorial Velocity Scale and confirm the same description and three ports appear.

Create and Validate the Objectiveโ€‹

Send:

Create an Objective named MCP Joint Exercise in mcp_joint_exercise.xml under My Tutorials. Use the existing joint diagnostic Objective as a reference. First move the manipulator to Look at Table. Then loop between Wrist 2 Min and Wrist 2 Max, calling the exact custom Behavior ID returned earlier before each wrist move and passing velocity_scale_factor to that motion. Expose root blackboard inputs velocity_min and velocity_max with defaults of 0.2 and 0.6. Use collision_check_params to override scene link padding to 0.0 for both wrist moves. Verify the Waypoints and ports, fully validate the Objective against the Runtime, save it, and stop before running it.

The setup move puts the robot in a repeatable starting pose. Zero link padding prevents the diagnostic wrist Waypoints from being rejected by additional self-collision padding.

What just happened

The agent used the Runtime's plugin manifest to validate every Behavior ID, node type, and port before writing the XML. Open src/mcp_walkthrough_sim/objectives/mcp_joint_exercise.xml and find the custom Behavior ID, the two blackboard inputs, the output connection, and collision_check_params="[{override_scene_link_padding: true, link_padding: 0.0}]" on both wrist moves.

Objective XML loads dynamically, so saving later XML changes does not require another build or Runtime restart.

Run, Observe, and Cancelโ€‹

Before the run, open a second terminal:

moveit_pro shell --instance mcp-walkthrough
ros2 topic echo /objective_server_heartbeat --once

Send:

Ask me before starting MCP Joint Exercise. Retain its goal ID, confirm that exact run is executing, report which joints are moving, and find the current velocity-scale notification or first execution error. Then cancel that goal and report its terminal state.

While the Objective is running, execute the heartbeat command again. Its Objective name, per-run execution ID, and status should change. In the Desktop App, watch the UR5 alternate between the wrist Waypoints at different speeds and check the alert history for the selected scale.

Run and cancel the Objective several times. Each run should begin at Look at Table, alternate between both wrist Waypoints without a self-collision error, and end in CANCELED.

MCP sends a DoObjectiveSequence goal to the Runtime's /do_objective ROS 2 action. Retaining the goal ID allows the agent to observe and cancel the exact run.

Change the Persistent Defaultsโ€‹

Send:

Update MCP Joint Exercise so the default velocity limits are 0.1 and 0.4. Preserve everything else, fully validate the complete Objective, use its current revision to avoid overwriting concurrent edits, and confirm the saved result. Do not rebuild or restart the Runtime.

To see info notifications as toast popups, open Alert Settings from the bell drawer, enable Info under Toasts, and close the Alert Sidebar.

Run it again and compare the motion and notification values. This edit changes the saved Objective. The revision check rejects the write if another editor saved a newer version first.

Use the Runtime SDKโ€‹

MCP is convenient for agent-driven work. Application code controls the same Runtime through /do_objective, using moveit_studio_sdk_msgs/action/DoObjectiveSequence.

From a shell in the named Runtime, start the finite Look at Table Objective directly:

moveit_pro shell --instance mcp-walkthrough
ros2 action send_goal /do_objective moveit_studio_sdk_msgs/action/DoObjectiveSequence \
"{objective_name: 'Look at Table'}"

Only one Objective executes at a time. A new valid goal preempts the current run. For exact cancellation, an application retains its goal handle.

The following rclpy client starts MCP Joint Exercise with one-run velocity limits, then cancels the same goal. Save it as start_mcp_joint_exercise.py in the workspace root.

start_mcp_joint_exercise.py
import time

import rclpy
from rclpy.action import ActionClient
from rclpy.signals import SignalHandlerOptions
from action_msgs.msg import GoalStatus

from moveit_studio_sdk_msgs.action import DoObjectiveSequence
from moveit_studio_sdk_msgs.msg import BehaviorParameter


def override(name: str, value: str) -> BehaviorParameter:
parameter = BehaviorParameter()
parameter.description.name = name
parameter.string_value = value
return parameter


# Keep ROS available while a Ctrl-C runs the cancellation cleanup below.
rclpy.init(signal_handler_options=SignalHandlerOptions.NO)
node = rclpy.create_node("mcp_joint_exercise_client")
client = ActionClient(node, DoObjectiveSequence, "/do_objective")

try:
if not client.wait_for_server(timeout_sec=60.0):
raise RuntimeError("Objective server is not ready")

goal = DoObjectiveSequence.Goal()
goal.objective_name = "MCP Joint Exercise"
goal.parameter_overrides = [
override("velocity_min", "0.15"),
override("velocity_max", "0.35"),
]

send_future = client.send_goal_async(goal)
rclpy.spin_until_future_complete(node, send_future)
goal_handle = send_future.result()
if not goal_handle or not goal_handle.accepted:
raise RuntimeError("Objective was rejected")

result_future = goal_handle.get_result_async()
try:
time.sleep(10)
finally:
cancel_future = goal_handle.cancel_goal_async()
rclpy.spin_until_future_complete(node, cancel_future, timeout_sec=10.0)
if not cancel_future.done():
raise RuntimeError("Cancellation timed out; inspect the Runtime before continuing")
response = cancel_future.result()
if response is None or not response.goals_canceling:
raise RuntimeError("Cancellation was not accepted; inspect the exact goal status")

rclpy.spin_until_future_complete(node, result_future, timeout_sec=30.0)
if not result_future.done():
raise RuntimeError("Goal has not stopped; inspect the Runtime before continuing")
result = result_future.result()
if result is None or result.status != GoalStatus.STATUS_CANCELED:
raise RuntimeError("Goal ended without confirming cancellation; inspect its result")
print("The exact goal reached CANCELED")
finally:
node.destroy_node()
rclpy.shutdown()

Run it from the Runtime shell:

cd ~/user_ws
python3 start_mcp_joint_exercise.py

The client waits for the action server, sends a goal, retains its handle, overrides two root blackboard values for that run, and confirms cancellation of the same goal. Once the goal is accepted, Ctrl-C during the wait also attempts cancellation before closing ROS. A timeout or rejected cancellation requires checking the Runtime before continuing. The saved defaults remain 0.1 and 0.4.

See Run an Existing Objective for complete native and Web Bridge clients. See Passing Parameters to Objectives at Runtime for supported types and Subtree targeting.

Stop the Tutorial Runtime and MCP Sessionโ€‹

Ending an MCP session does not stop the Runtime or a running Objective, so stop them in this order:

  1. Ask the agent to call get_execution_status. If the tutorial Objective is still running, have the agent cancel its goal_id with cancel_objective and wait until that goal reaches a terminal state. If a different Objective is running, leave the Runtime running.

  2. On the Runtime host, as the account that owns the workspace, stop the named Runtime:

    moveit_pro down --instance mcp-walkthrough
  3. Disable this tutorial's MCP server in your AI client to end the session. Claude Code starts a new session on its next tool call after a session ends, so disabling the server is what keeps it closed.

To check which MCP sessions are still connected, run this on the computer and account that run the MCP server (the SSH host for an SSH connection):

moveit_pro mcp session list

An orphaned entry belongs to an MCP server that stopped without cleaning up, for example after being force-killed, and the next MCP session you start on that host removes it. See List MCP Sessions.

Summaryโ€‹

You inspected an MCP target, created an inherited robot configuration package, and verified it in the Desktop App. You then created, built, tested, and loaded a C++ Behavior plugin; authored and validated an Objective around its ports; and controlled that Objective through MCP, the heartbeat topic, the ROS 2 CLI, and rclpy.