Connect a VLA Policy to the ExecutePolicy Behavior

Required Version

This feature will not be released until MoveIt Pro version 10.0.

ExecutePolicy runs a learned policy on a robot. That can be a VLA such as SmolVLA or pi0.5, a diffusion policy, or anything else that outputs chunks of actions. The Behavior streams those chunks as one continuous motion, and checks every point against the robot's joint limits and the planning scene before it reaches the controller.

Connecting your own policy takes one ROS 2 service. You write an adapter node that serves GetActionChunk in front of your inference server, and ExecutePolicy handles the rest of the loop: gathering observations, stitching chunks together, checking limits and collision, and executing.

Read this page when you want to connect your own inference server to ExecutePolicy. The vla_sim package is a worked example of exactly that, so running the cube-stacking example first is a good way to see the pieces working before you wire up your own. Its adapter and server are the reference implementations this page points back to throughout.

Executing a Policy Safely

A policy's raw output is a list of numbers. MoveIt Pro never passes those numbers straight to the robot. Four protections sit in between.

• Joint limits. Position, velocity, and acceleration are sampled per point and checked against the commanded joints' limits. A chunk that violates them is rejected before it reaches the controller.
• Collision. The robot is checked against itself, and against the world as the planning scene knows it, with link_padding meters of clearance added around the robot's geometry for the world check. A colliding chunk is rejected before it reaches the controller, and the scene is monitored for the whole run, so an object that appears mid-run stops the robot.
• Force-torque abort. The controller stops the run when a reading crosses the per-axis threshold in absolute_force_torque_threshold, catching contact the policy did not expect.
• Continuity. Chunks are smoothed, densified, and blended together so the motion stays smooth and continuous, and the next chunk is fetched before the current one runs out, so the robot does not pause for inference as long as the committed window covers the latency (see Sizing the Committed Window).

Run in simulation until you trust the policy

Collision checking covers the planning scene only. Obstacles the scene does not contain are not checked, and no check can tell whether the policy is about to do something reasonable.

Plan for one thing up front. Collision rules and object attachments are read once when the Behavior starts, and are never refreshed during the run. If the policy is meant to touch something, such as closing a grasp on a scene object, allow that contact with the SetCollisionRule Behavior before ExecutePolicy starts. Otherwise every chunk from the moment of contact is rejected.

The Pipeline

inference server (LeRobot, torch, GPU)
        ▲  HTTP or gRPC, freely chosen
adapter node (serves GetActionChunk)
        ▲  ROS 2 service: moveit_pro_ml_msgs/srv/GetActionChunk
ExecutePolicy Behavior
        ▼  FollowJointTrajectoryWithAdmittance goals, stitched on one clock
joint_trajectory_admittance_controller
        ▼
      robot

The GetActionChunk service is the boundary between the two halves. Because it is a plain ROS 2 service, any node able to serve it can back ExecutePolicy. That might be an adapter in front of an inference server, a replay of recorded actions, or something you write from scratch. Swapping models never touches the execution side.

Prerequisites

• A policy checkpoint trained on the robot it will command. Collecting demonstrations and training the policy are outside the scope of this guide, which starts from a checkpoint you already have.
• A Python environment that can load and run it. The worked example uses LeRobot with predict_action_chunk.
• A robot configuration package where:
  • The joint_trajectory_admittance_controller is available and active during the run. ExecutePolicy sends it moveit_pro_controllers_msgs/action/FollowJointTrajectoryWithAdmittance goals on /joint_trajectory_admittance_controller/follow_joint_trajectory. A plain FollowJointTrajectory server cannot stand in for it, because mid-run goal stitching is what lets one chunk hand off to the next without stopping.
  • The SRDF has a joint group matching the joints the policy commands, and that group resolves to exactly one end-effector tip.
  • /joint_states is published with reliable QoS.
  • The camera topics the policy needs publish sensor_msgs/Image.
  • A GripperCommand action server exists if the policy also outputs a gripper joint.
• The adapter node itself, which the rest of this guide covers.

The GetActionChunk Contract

Inspect the service with ros2 interface show moveit_pro_ml_msgs/srv/GetActionChunk. The comments in that definition are the authoritative contract, and what follows summarizes them from the adapter author's side. Treat the policy as a pure function of the request. It holds no robot state between calls, so one running model can serve any caller.

Four request fields exist only for real-time chunking (RTC), a technique where the policy, while still iteratively refining its output, is guided to continue the chunk the robot is already executing. If your policy does not use RTC, leave previous_action_chunk, previous_anchor_state, frozen_prefix_steps, and guidance_horizon alone.

Request

Field	Type	What the adapter does with it
prompt	string	Pass to the policy as the task instruction. Empty for policies that take no prompt.
robot_state	sensor_msgs/JointState	The current unnormalized joint state, with position filled in and nothing else. name gives the ordering. The commanded group's joints come first, then the configured gripper joint when the Objective sets one. A policy that conditions on velocity or effort has to obtain those itself. Normalization is the policy's job.
images, image_names	sensor_msgs/Image[], string[]	One raw camera frame per view, index-aligned. Frames arrive as captured, so the adapter owns resize, normalization, and encoding conversion (bgr8 to rgb8 when needed, per the encoding field).
new_episode	bool	True on the run's first request. Reset any per-episode state. Note that ExecutePolicy sends one extra warmup request before the run starts, also with new_episode true, and throws its chunk away. Use the field to clear state, not to count runs, since a single run sets it twice.
previous_action_chunk	std_msgs/Float64MultiArray	RTC carryover: the unexecuted tail of the chunk your previous response returned in policy_action_chunk, still in the policy's own action space. Empty on the first call and whenever RTC is off.
previous_anchor_state	sensor_msgs/JointState	The state the previous chunk was anchored to, sent with every carryover. A delta-action policy (pi0-style) re-references the carried tail from it. An absolute-action policy ignores it.
frozen_prefix_steps	uint32	RTC's inference delay, in steps. While the policy computes a new chunk, the robot keeps following the previous one, so the new chunk's first steps cover time that will already be over by the time it arrives. The policy freezes those steps to the previous chunk instead of re-deciding them.
guidance_horizon	uint32	How many steps past the frozen prefix the guidance keeps the new chunk close to the previous one. Zero, the default, means use whatever the inference server is configured for. Anything else overrides the server for this run, and the adapter forwards it to the policy.

Response

Field	Type	What to fill
status, message	uint8, string	CHUNK_PRODUCED, NO_MORE_ACTIONS, or ERROR. Always set it, because the default UNKNOWN fails the run. See Reporting Status.
chunk	trajectory_msgs/JointTrajectory	The chunk as absolute joint positions, one point per policy step. Set joint_names by echoing back robot_state.name, which is correct in every case. Building the list yourself means following two different rules: with no gripper configured the positions go to the joints in list order, so the names must be the commanded group's joints in that same order, while with a gripper configured each position is matched to its joint by name, so every joint must be named. Any time_from_start you set is ignored, since the caller plays the chunk at its configured dt.
native_control_period	float64	Seconds per step the policy was trained at, so 1 / training fps. This is advisory. ExecutePolicy warns when it disagrees with the Objective's dt, but never retimes the chunk. Zero is reserved for a chunk source with no training rate to report, such as a replay or a planner serving the same contract.
policy_action_chunk	std_msgs/Float64MultiArray	The same chunk in the policy's own normalized action space, one row per step and one column per action dimension. The caller stores it untouched and returns its tail as the next previous_action_chunk. Leave it empty when RTC is off, and mind the layout warning under Real-Time Chunking Conventions.

Reporting Status

• CHUNK_PRODUCED means chunk carries the next actions. This is the only status for which chunk is read at all.
• NO_MORE_ACTIONS means the chunk source chose to stop, as a replay does when it reaches its end. Goals are cancelled, the robot stops, and the Behavior finishes successfully, with message shown as information. A source ending its stream should return its final actions in a CHUNK_PRODUCED response first, then answer NO_MORE_ACTIONS on the following call.
• ERROR means no chunk could be produced. The robot stops and the Behavior fails, with message shown as the failure reason.

Map every operational failure to ERROR

Server unreachable, model still loading, malformed inference output: all of these are ERROR, never NO_MORE_ACTIONS. The Objective branches on the Behavior's status, so an error reported as a graceful end would let downstream steps act on a rollout that never ran.

Real-Time Chunking Conventions

RTC keeps consecutive chunks consistent. While the model is still iteratively refining its output (denoising), the policy guides the new chunk to continue the part of the previous one the robot is still executing. That guidance runs inside the policy's sampling loop, so it belongs in the inference server rather than in MoveIt Pro. Three conventions matter.

• Alignment. The tail in previous_action_chunk starts at the action for the moment the request is made, the same moment the new chunk starts, so tail step i pairs with new-chunk step i. The robot keeps following the first frozen_prefix_steps steps of that tail while the new chunk is being computed. Those are the steps the guidance freezes.
• Action space. The tail is the policy's normalized output, echoed back untouched. MoveIt Pro never interprets it, only slices it by step.
• Applicability. RTC exists for policies with an iterative denoising loop, such as SmolVLA, pi0, pi0.5, and diffusion policies. Autoregressive and single-forward-pass policies have no loop to guide, so they rely on the executor's blending alone and should leave policy_action_chunk empty.

data_offset must be zero

Lay policy_action_chunk out with the step dimension first in layout.dim and layout.data_offset at 0. A non-zero offset makes the caller read the echo as empty, which silently disables the carryover. RTC then appears to be on and does nothing.

Adapting a LeRobot Policy

MoveIt Pro's container has rclpy and the moveit_pro_ml_msgs interfaces but no ML stack; your inference environment has lerobot and torch but usually no rclpy. In practice that means two processes: a thin ROS 2 bridge next to the MoveIt Pro stack, and an inference server wherever the model lives, connected over HTTP or gRPC. When both stacks can live in one environment, a single node that calls predict_action_chunk in-process works the same way and needs no network hop.

The vla_sim package ships both halves as reference implementations.

• get_action_chunk_adapter.py is the bridge node. It encodes the request's camera frames for transport, forwards the RTC carryover, and validates what comes back, checking for a non-empty chunk with one action column per observed joint. It reuses the request's joint names as the chunk's joint names, which satisfies both the ordering rule and the gripper naming rule. Its service callback is wrapped so that any failure, from an unreachable server to a malformed response, answers with ERROR and a reason rather than leaving the caller to time out.
• vla_inference_server.py is the inference server. It loads a LeRobot checkpoint, configures RTC guidance, normalizes the observation, denormalizes the actions, and serves both the absolute chunk and the raw normalized chunk over HTTP.

When MoveIt Pro starts the inference server for you, it passes the deployment's MOVEIT_FRONTEND_KEY into the container. Your server must require that key as an Authorization: Bearer token on its inference endpoint and refuse inference without it. Leave the health endpoint token-free, since the launcher's health poll and any external probe depend on reaching it unauthenticated. See Endpoint Security.

The field mapping onto LeRobot's API:

GetActionChunk	LeRobot
previous_action_chunk	predict_action_chunk(..., prev_chunk_left_over=...)
frozen_prefix_steps	predict_action_chunk(..., inference_delay=...)
guidance_horizon	predict_action_chunk(..., execution_horizon=...), after the conversion below
chunk	the postprocessed (denormalized, absolute) actions
policy_action_chunk	the normalized (T, A) output of predict_action_chunk, before the postprocessor

Convert guidance_horizon before calling LeRobot

The two count from different places. guidance_horizon counts steps from the end of the frozen prefix, while LeRobot's execution_horizon counts from the start of the chunk. A guidance_horizon of 0 also means "use the width the server is configured with", so substitute that width before adding:

width = guidance_horizon if guidance_horizon > 0 else server_default_guidance
execution_horizon = frozen_prefix_steps + width

You do not have to estimate the inference delay yourself. ExecutePolicy measures it and sends it as frozen_prefix_steps on every request.

Both halves of that conversion matter. Adding a zero width instead of substituting the server's default leaves the guided region empty, so RTC reports as on and guides nothing. Passing the width through without adding the prefix shrinks the frozen prefix whenever the width is smaller than the inference delay, and the policy then re-decides steps that are already underway.

Have the server run one full-size warmup inference before it reports ready. ExecutePolicy makes its own warmup call before starting the trajectory clock and seeds its latency estimate from that round trip. That estimate only ever grows, so if the call pays for model compilation or cache setup, that inflated figure makes every later chunk be requested earlier than it needs to be, for the rest of the run.

LeRobot also ships its own gRPC PolicyServer (python -m lerobot.async_inference.policy_server) for its RobotClient. An adapter can target it, but the async client protocol does not expose the RTC carryover arguments. Wrapping predict_action_chunk directly, as the reference server does, keeps control of them.

Serving from Another Machine

Nothing in the contract ties the inference server to the robot's computer. An adapter can forward requests to a GPU workstation or a cloud endpoint just as well as to a local container. The vla_sim adapter does not allow that, though. It accepts only a loopback infer_url and rejects anything else at startup, so a remote setup starts from your own adapter or a modified copy of that one. What changes when you do:

• Secure the transport yourself. The vla_sim deployment speaks plain HTTP, so the bearer key its requests carry travels in cleartext. Publishing on loopback only is what makes that acceptable, and only on one machine. A remote hop needs its own protection, whether TLS, a VPN, or a private network.
• Budget for the round trip. Every request carries one full camera frame per view, so bandwidth is part of the inference latency. Set policy_call_timeout above the worst-case round trip, and size committed_action_steps to the real latency.
• One request, one response. The service call is single-shot, so a dropped response fails that call. Keep the link reliable enough that this stays rare.

How This Maps onto LeRobot's Async Inference Stack

If you know LeRobot's async inference stack (PolicyServer and RobotClient), ExecutePolicy replaces the RobotClient side entirely.

LeRobot async concept	MoveIt Pro counterpart
RobotClient control loop	ExecutePolicy and the joint_trajectory_admittance_controller
actions_per_chunk	the length of the chunk the server returns
chunk_size_threshold (when to send a fresh observation)	committed_action_steps plus prefetch: ExecutePolicy requests the next chunk before the committed motion drains
aggregate_fn_name (overlap aggregation)	blending in the executor, plus RTC guidance in the policy
fps	1 / dt

What MoveIt Pro adds over that client stack is everything in Executing a Policy Safely, along with gripper streaming through the standard GripperCommand action.

Wire ExecutePolicy into an Objective

The vla_sim Objective shows the pattern. It activates the joint_trajectory_admittance_controller with SwitchController, opens the gripper, moves to a start pose, and only then runs the policy. That starting gripper state is not incidental, since it is part of matching the checkpoint's training distribution.

<Action
  ID="ExecutePolicy"
  policy_service_name="/get_action_chunk"
  joint_group_name="manipulator"
  prompt="stack the blue cube on the green cube"
  total_action_steps="300"
  dt="0.1"
  committed_action_steps="20"
  num_interpolation_points="200"
  goal_path_tolerance="0.3"
  policy_tracking_weight="300.0"
  policy_uses_real_time_chunking="true"
  guidance_horizon="18"
  policy_call_timeout="10.0"
  image_topics="/scene_camera/color;/wrist_camera/color;/overview_camera/color"
  image_names="scene;wrist;overview"
  joint_states_topic="/joint_states"
  absolute_force_torque_threshold="250;250;250;80;80;80"
  gripper_command_action_name="/robotiq_gripper_controller/gripper_cmd"
  gripper_joint_name="robotiq_85_left_knuckle_joint"
/>

policy_service_name, joint_group_name, and total_action_steps are the required ports, and everything else has a default. image_names must match the keys the adapter and checkpoint expect, index-aligned with image_topics. The port descriptions in the Behavior's sidebar are the authoritative reference.

Several defaults suit a fast local policy rather than a VLA, so compare them against the example above before assuming they will work.

Port	Default	How to choose it
dt	0.05	The inverse of the rate the policy was trained at, so a 10 Hz policy uses 0.1. Get it wrong and the motion runs at the wrong speed, scaling every joint velocity with it.
committed_action_steps	32	How many points execute from each chunk before the next takes over. See Sizing the Committed Window.
total_action_steps	required	The run length. The motion lasts total_action_steps * dt seconds, after which the Behavior stops successfully whether or not the task is done.
num_interpolation_points	100	Densified points per chunk, and the sampling density of the limit and collision checks. Keep it high relative to committed_action_steps and dt, or a brief over-speed between samples can slip through.
link_padding	0.01	Padding in meters added around the robot's collision geometry when checking against the world. A configuration coming within this distance of a world object is rejected. Self-collision is checked with the unpadded geometry. Raise it for more clearance, lower it when valid close-quarters motion is rejected.
policy_tracking_weight	300.0	How closely the smoothing spline follows the raw chunk points. 300 suits SmolVLA-style chunks. Lower it if the motion oscillates, raise it if it lags the policy.
policy_uses_real_time_chunking	false	true only when the inference server actually applies the carryover. The executor blends every chunk into the previous one either way, so RTC is a refinement rather than a requirement.
policy_call_timeout	3.0	Above the worst-case service round trip. It also bounds the warmup call, usually the slowest of the run, so the default is too tight for most VLAs.
goal_path_tolerance	0.01	Error in radians accepted along the path. It is also the tolerance the controller applies to the position and velocity deviation where one chunk joins the next, so a tight value rejects those joins and stalls the stream. The default is far tighter than policy execution allows. The example uses 0.3.
absolute_force_torque_threshold	45;45;45;10;10;10	Per-axis abort limits, ordered Fx, Fy, Fz, Tx, Ty, Tz.
controller_action_name	/joint_trajectory_admittance_controller/follow_joint_trajectory	The admittance controller's action. Change it only for a namespaced or renamed controller.
gripper_command_action_name, gripper_joint_name	empty	Set these together to observe and command the gripper, since either one alone fails at startup. The gripper's sensed position is appended to the observation, and the chunk's one extra joint, which must carry exactly gripper_joint_name, is streamed to the action. Leave both empty to run the arm only. Single-DOF grippers only.
guidance_horizon	0	Steps past the frozen prefix that RTC keeps the new chunk close to the previous one. Leave it at 0 to use the inference server's configured setting, or set it to sweep guidance per run without touching the server. Only meaningful when policy_uses_real_time_chunking is true.

Sizing the Committed Window

committed_action_steps must be smaller than the policy's chunk size, and (committed_action_steps - 1) * dt should cover at least twice your inference latency. Below that, the committed motion drains before the next chunk lands, and the robot stalls or the run aborts. Measure your own inference latency and size the window from it, rather than tuning up from the default.

When inference takes seconds rather than a fraction of one, a bigger committed window cannot rescue it. The window can never be longer than the chunk the policy returns, so there is a ceiling on how far you can raise it. The only lever left is a larger dt, which spreads the same number of steps over more time. That buys the window, but it also plays the policy slower than the rate it was trained at, scaling down every joint velocity, and ExecutePolicy warns about the mismatch. Treat it as a way to see a slow policy move at all, not as a working configuration.

Bring-Up and Verification

Work through these in order, since each step depends on the one before it.

1. Start the inference server and confirm it reports ready. Give it a health endpoint that reports ready only once the startup warmup inference has finished. The vla_sim server's GET /health is one example.
2. Start the adapter where it can reach both the ROS graph and the server. In vla_sim the adapter launches with the robot configuration package, so it comes up with the stack.
3. Smoke-test the service before running any motion. A throwaway rclpy script catches nearly every wiring mistake in seconds. Fill robot_state the way ExecutePolicy will, with the commanded group's joints and the gripper joint appended last when one is configured. Attach one frame per camera topic, then call the service once. Check that the chunk has the expected step count and joint names, that all values are finite, and that policy_action_chunk is present when RTC is on.
4. Run the Objective from the UI. Reset the scene first, re-zeroing force-torque sensors if the configuration has them. Then watch the adapter's log as the run proceeds. Its served-chunk counter should climb steadily from start to finish.

Troubleshooting

• Run fails saying the policy service is not available. Start the adapter before you run the Objective. ExecutePolicy gives the service 5 seconds to appear, and the same to the controller and gripper action servers.
• Run fails warming up the policy service. The warmup call is bounded by policy_call_timeout, and it is usually the slowest call of the run. Raise the timeout, or make the server finish its startup warmup before it reports ready.
• Run fails saying the seam time exceeds the end of the current trajectory. The chunk arrived too late to join the motion still playing. Revisit Sizing the Committed Window, then check goal_path_tolerance, because the controller rejects a join whose position or velocity deviates by more than that value. A large enough jump where two chunks meet can also trip the force-torque threshold.
• Robot pauses at chunk boundaries, or the run dies with a drained buffer. The committed window is too small for the latency. The latency estimate only ever grows, so one slow inference makes every later chunk be requested earlier, for the rest of the run.
• Every chunk rejected for a velocity or acceleration limit. Check dt first, since playing a 10 Hz policy at dt=0.05 doubles every joint velocity. Then check that the checkpoint actually matches the robot.
• Chunk rejected for a world collision the motion should be allowed to make. Either the contact is intended, in which case allow it with SetCollisionRule before ExecutePolicy starts, or link_padding is inflating the robot's geometry into a scene object. An object the robot already holds belongs attached to the robot, not left in the scene as a world obstacle.
• Chunk rejected saying the joint names do not match the joint group. With no gripper configured, each point's positions are handed to the joint group's joints in list order rather than looked up by name, so joint_names has to list the same joints in that same order. The simplest way to get it right is to copy the name list out of robot_state and send it straight back.
• Model output looks unrelated to the scene. Verify the request-key to camera-slot mapping against the checkpoint's trained input features, and check the bgr8/rgb8 conversion. Both failure modes are silent.
• robot_state positions look stale or frozen. The joint-state publisher must be reliable QoS. With a best-effort publisher the subscription never connects, and the Behavior falls back to the initial snapshot positions.
• Run fails before the first chunk. Each camera topic must produce a frame within 5 seconds, and the joint group must resolve to exactly one end-effector tip.
• Run fails at startup naming the gripper ports. gripper_command_action_name and gripper_joint_name must be set together, and the joint must be a single-DOF joint in the robot model outside the arm group.
• Run fails at the first observation naming the gripper joint. With a gripper configured, that joint must be published on joint_states_topic alongside the arm joints.
• Chunk rejected naming two gripper joints. The chunk's one extra joint must carry exactly the configured gripper_joint_name. The easiest way to guarantee that is to reuse the request's joint names as the chunk's, as the reference adapter does.

Current Limits

• Collision checking covers only what the planning scene contains. Obstacles the scene does not know about are never checked. World geometry is tracked for the whole run, so an object that appears mid-run stops the motion, but collision rules and object attachments are read once when the Behavior starts.
• A run ends by step count or by the chunk source stopping. total_action_steps bounds the run, and a NO_MORE_ACTIONS response ends it early and reports success. Nothing checks whether the task itself succeeded. If you need that, detect it in the Objective after ExecutePolicy returns.
• Single-DOF grippers only, carried as exactly one joint beyond the arm group.