Skip to main content
Version: 10

Connect a VLA Policy to the ExecutePolicy Behavior

Required Version
This feature will not be released until MoveIt Pro version 10.2.

ExecutePolicy runs a learned policy on a robot. That can be a VLA such as SmolVLA or pi0.5, a diffusion policy, or anything else that outputs chunks of actions. The Behavior streams those chunks as one continuous motion, and checks every point against the robot's joint limits and the planning scene before it reaches the controller.

An action chunk is a sequence of future robot actions predicted by a policy in a single inference call, rather than one action at a time. The robot executes part of a chunk while the next one is being computed, which is what the rest of this page is about: asking for chunks, joining them together, and checking them before they move the robot.

Connecting your own policy takes one ROS 2 service. You write an adapter node that serves GetActionChunk in front of your inference server, and ExecutePolicy handles the rest of the loop: gathering observations, stitching chunks together, checking limits and collision, and executing.

Read this page when you want to connect your own inference server to ExecutePolicy. The vla_sim package is a worked example of exactly that, so running the cube-stacking example first is a good way to see the pieces working before you wire up your own. Its adapter and server are the reference implementations this page points back to throughout.

Executing a Policy Safely​

A policy's raw output is a list of numbers. MoveIt Pro never passes those numbers straight to the robot. Four protections sit in between.

  • Joint limits. Position, velocity, and acceleration are sampled per point and checked against the commanded joints' limits. A chunk that violates them is rejected before it reaches the controller.
  • Collision. The robot is checked against itself, and against the world as the planning scene knows it, with link_padding meters of clearance added around the robot's geometry for the world check. A colliding chunk is rejected before it reaches the controller, and the scene is monitored for the whole run, so an object that appears mid-run stops the robot.
  • Force-torque abort. The controller stops the run when a reading crosses the per-axis threshold in absolute_force_torque_threshold, catching contact the policy did not expect.
  • Continuity. Chunks are smoothed, densified with intermediate points, and blended into one another so the motion stays smooth and continuous, and the next chunk is fetched before the current one runs out, so the robot does not pause for inference as long as the committed window covers the latency (see Sizing the Committed Window). When a chunk is late anyway, the robot pauses at rest and resumes when it arrives; lateness never fails the run on its own, though a policy call that fails or times out still does. The robot resumes without warning the moment the chunk lands, so never treat a pause as a safe state to approach. The chunk that ends a pause was computed from an observation taken up to policy_call_timeout earlier, so the images it acted on can be seconds old. Collision revalidation covers the geometry the planning scene knows about, not a policy acting on a stale picture.

The gripper gets none of these. Its targets are only clamped to the joint's position limits, so a policy can drive the gripper anywhere in its mechanical range, including opening it while the robot is holding something.

Run in simulation until you trust the policy

Collision checking covers the planning scene only. Obstacles the scene does not contain are not checked, and no check can tell whether the policy is about to do something reasonable.

Plan for one thing up front. Collision rules and object attachments are read once when the Behavior starts, and are never refreshed during the run. If the policy is meant to touch something, such as closing a grasp on a scene object, allow that contact with the SetCollisionRule Behavior before ExecutePolicy starts. Otherwise every chunk from the moment of contact is rejected.

The Pipeline​

inference server (LeRobot, torch, GPU)
        ▲  HTTP or gRPC, freely chosen
adapter node (serves GetActionChunk)
        ▲  ROS 2 service: moveit_pro_ml_msgs/srv/GetActionChunk
ExecutePolicy Behavior
        ▼  FollowJointTrajectoryWithAdmittance goals, stitched on one clock
joint_trajectory_admittance_controller
        ▼
      robot

The GetActionChunk service is the boundary between the two halves. Because it is a plain ROS 2 service, any node able to serve it can back ExecutePolicy. That might be an adapter in front of an inference server, a replay of recorded actions, or something you write from scratch. Swapping models never touches the execution side.

Prerequisites​

  • A policy checkpoint trained on the robot it will command. Collecting demonstrations and training the policy are outside the scope of this guide, which starts from a checkpoint you already have. To produce one, see Train a VLA Policy with Trainer or Train a VLA Policy on Your Own Runpod.
  • A Python environment that can load and run it. The worked example uses LeRobot with predict_action_chunk.
  • A robot configuration package where:
    • The joint_trajectory_admittance_controller is available and active during the run. ExecutePolicy sends it moveit_pro_controllers_msgs/action/FollowJointTrajectoryWithAdmittance goals on /joint_trajectory_admittance_controller/follow_joint_trajectory. A plain FollowJointTrajectory server cannot stand in for it, because mid-run goal stitching is what lets one chunk hand off to the next without stopping.
    • The SRDF has a joint group matching the joints the policy commands, and that group resolves to exactly one end-effector tip.
    • /joint_states is published with reliable QoS.
    • The camera topics the policy needs publish sensor_msgs/Image.
    • A GripperCommand action server exists if the policy also outputs a gripper joint.
  • The adapter node itself, which the rest of this guide covers.

The GetActionChunk Contract​

Inspect the service with ros2 interface show moveit_pro_ml_msgs/srv/GetActionChunk. The comments in that definition are the authoritative contract, and what follows summarizes them from the adapter author's side. Treat the policy as a pure function of the request. It holds no robot state between calls, so one running model can serve any caller.

Four request fields exist only for real-time chunking (RTC), a technique where the policy, while still iteratively refining its output, is guided to continue the chunk the robot is already executing. If your policy does not use RTC, leave previous_action_chunk, previous_anchor_state, frozen_prefix_steps, and guidance_horizon alone.

Request​

FieldTypeWhat the adapter does with it
promptstringPass to the policy as the task instruction. Empty for policies that take no prompt.
robot_statesensor_msgs/JointStateThe current unnormalized joint state, with position filled in and nothing else. name gives the ordering. The commanded group's joints come first, then the configured gripper joint when the Objective sets one. A policy that conditions on velocity or effort has to obtain those itself. Normalization is the policy's job.
images, image_namessensor_msgs/Image[], string[]One raw camera frame per view, index-aligned. Frames arrive as captured, so the adapter owns resize, normalization, and encoding conversion (bgr8 to rgb8 when needed, per the encoding field).
new_episodeboolTrue on the run's first request. Reset any per-episode state. Note that ExecutePolicy sends one extra warmup request before the run starts, also with new_episode true, and throws its chunk away. Use the field to clear state, not to count runs, since a single run sets it twice.
previous_action_chunkstd_msgs/Float64MultiArrayRTC carryover: the unexecuted tail of the chunk your previous response returned in policy_action_chunk, still in the policy's own action space. Empty on the first call and whenever RTC is off.
previous_anchor_statesensor_msgs/JointStateThe state the previous chunk was anchored to, sent with every carryover. A delta-action policy (pi0-style) re-references the carried tail from it. An absolute-action policy ignores it.
frozen_prefix_stepsuint32RTC's inference delay, in steps. While the policy computes a new chunk, the robot keeps following the previous one, so the new chunk's first steps cover time that will already be over by the time it arrives. The policy freezes those steps to the previous chunk instead of re-deciding them.
guidance_horizonuint32How many steps past the frozen prefix the guidance keeps the new chunk close to the previous one. Zero, the default, means use whatever the inference server is configured for. Anything else overrides the server for this run, and the adapter forwards it to the policy.

Response​

FieldTypeWhat to fill
status, messageuint8, stringCHUNK_PRODUCED, NO_MORE_ACTIONS, or ERROR. Always set it, because the default UNKNOWN fails the run. See Reporting Status.
chunktrajectory_msgs/JointTrajectoryThe chunk as absolute joint positions, one point per policy step. Set joint_names by echoing back robot_state.name, which is correct in every case. Building the list yourself means following two different rules: with no gripper configured the positions go to the joints in list order, so the names must be the commanded group's joints in that same order, while with a gripper configured each position is matched to its joint by name, so every joint must be named. Any time_from_start you set is ignored, since the caller plays the chunk at its configured dt.
native_control_periodfloat64Seconds per step the policy was trained at, so 1 / training fps. This is advisory. ExecutePolicy warns when it disagrees with the Objective's dt, but never retimes the chunk. Zero is reserved for a chunk source with no training rate to report, such as a replay or a planner serving the same contract.
policy_action_chunkstd_msgs/Float64MultiArrayThe same chunk in the policy's own normalized action space, one row per step and one column per action dimension. The caller stores it untouched and returns its tail as the next previous_action_chunk. Leave it empty when RTC is off, and mind the layout warning under Real-Time Chunking Conventions.

Reporting Status​

  • CHUNK_PRODUCED means chunk carries the next actions. This is the only status for which chunk is read at all.
  • NO_MORE_ACTIONS means the chunk source chose to stop, as a replay does when it reaches its end. Goals are cancelled, the robot stops, and the Behavior finishes successfully, with message shown as information. A source ending its stream should return its final actions in a CHUNK_PRODUCED response first, then answer NO_MORE_ACTIONS on the following call.
  • ERROR means no chunk could be produced. The robot stops and the Behavior fails, with message shown as the failure reason.
Map every operational failure to ERROR

Server unreachable, model still loading, malformed inference output: all of these are ERROR, never NO_MORE_ACTIONS. The Objective branches on the Behavior's status, so an error reported as a graceful end would let downstream steps act on a rollout that never ran.

Real-Time Chunking Conventions​

RTC keeps consecutive chunks consistent. While the model is still iteratively refining its output (denoising), the policy guides the new chunk to continue the part of the previous one the robot is still executing. That guidance runs inside the policy's sampling loop, so it belongs in the inference server rather than in MoveIt Pro. Three conventions matter.

  • Alignment. The tail in previous_action_chunk and the new chunk start at the same moment, so tail step i pairs with new-chunk step i. The robot keeps following the first frozen_prefix_steps steps of that tail while the new chunk is being computed. Those are the steps the guidance freezes.
  • Action space. The tail is the policy's normalized output, echoed back untouched. MoveIt Pro never interprets it, only slices it by step.
  • Applicability. RTC exists for policies with an iterative denoising loop, such as SmolVLA, pi0, pi0.5, and diffusion policies. Autoregressive and single-forward-pass policies have no loop to guide, so they rely on the executor's blending alone and should leave policy_action_chunk empty.
data_offset must be zero

Lay policy_action_chunk out with the step dimension first in layout.dim and layout.data_offset at 0. A non-zero offset makes the caller read the echo as empty, which silently disables the carryover. RTC then appears to be on and does nothing.

Adapting a LeRobot Policy​

MoveIt Pro's container has rclpy and the moveit_pro_ml_msgs interfaces but no ML stack; your inference environment has lerobot and torch but usually no rclpy. In practice that means two processes: a thin ROS 2 bridge next to the MoveIt Pro stack, and an inference server wherever the model lives, connected over HTTP or gRPC. When both stacks can live in one environment, a single node that calls predict_action_chunk in-process works the same way and needs no network hop.

The vla_sim package ships both halves as reference implementations.

  • get_action_chunk_adapter.py is the bridge node. It encodes the request's camera frames for transport, forwards the RTC carryover, and validates what comes back, checking for a non-empty chunk with one action column per observed joint. It reuses the request's joint names as the chunk's joint names, which satisfies both the ordering rule and the gripper naming rule. Its service callback is wrapped so that any failure, from an unreachable server to a malformed response, answers with ERROR and a reason rather than leaving the caller to time out.
  • vla_inference_server.py is the inference server. It loads a LeRobot checkpoint, configures RTC guidance, normalizes the observation, denormalizes the actions, and serves both the absolute chunk and the raw normalized chunk over HTTP. To serve a checkpoint of your own with it, set checkpoint in config/vla_serving.yaml to your Hugging Face repo id; Run the VLA Cube-Stacking Example covers that file's settings.

When a completed Trainer run is eligible, choose Configure for ExecutePolicy on its card or in Details instead of editing the serving file by hand. With a local inference server, the action is available only when the active robot configuration package provides a writable config/vla_serving.yaml whose inference server declares support for immutable Trainer checkpoints. Trainer verifies the checkpoint, securely provisions the selected Hugging Face token to the inference server, and records the exact revision and private-model access requirement. A running inference server reloads automatically, and the run card reports Connected to ExecutePolicy only after the authenticated status endpoint confirms the exact repository revision. Copy the reported dt and dataset-native image_names into the Objective. Copy image_topics when Trainer resolves all source cameras; otherwise review the topics in the Objective. The action does not edit the Objective.

When MoveIt Pro starts the inference server for you, it passes the inference key into the container as MOVEIT_INFERENCE_KEY. An adapter you write must send the same variable as its bearer token, as the reference adapter does on every request. Your server must require that key as an Authorization: Bearer token on its inference endpoint, and on its status endpoint if it has one, and refuse inference without it. Leave the health endpoint token-free, since the launcher's health poll and any external probe depend on reaching it unauthenticated. MoveIt Pro never passes the frontend key to a remote inference server. A server it starts locally still receives MOVEIT_FRONTEND_KEY so that a server written against 10.0 or 10.1 keeps working. Passing it is deprecated and will be removed in a future release, so switch such a server to MOVEIT_INFERENCE_KEY. See Endpoint Security and, for a server on another machine, Serving from Another Machine.

The field mapping onto LeRobot's API:

GetActionChunkLeRobot
previous_action_chunkpredict_action_chunk(..., prev_chunk_left_over=...)
frozen_prefix_stepspredict_action_chunk(..., inference_delay=...)
guidance_horizonpredict_action_chunk(..., execution_horizon=...), after the conversion below
chunkthe postprocessed (denormalized, absolute) actions
policy_action_chunkthe normalized (T, A) output of predict_action_chunk, before the postprocessor
Convert guidance_horizon before calling LeRobot

The two count from different places. guidance_horizon counts steps from the end of the frozen prefix, while LeRobot's execution_horizon counts from the start of the chunk. A guidance_horizon of 0 also means "use the width the server is configured with", so substitute that width before adding:

width = guidance_horizon if guidance_horizon > 0 else server_default_guidance
execution_horizon = frozen_prefix_steps + width

You do not have to estimate the inference delay yourself. ExecutePolicy measures it and sends it as frozen_prefix_steps. The first request carries zero, since nothing has executed yet.

Both halves of that conversion matter. Adding a zero width instead of substituting the server's default leaves the guided region empty, so RTC reports as on and guides nothing. Passing the width through without adding the prefix shrinks the frozen prefix whenever the width is smaller than the inference delay, and the policy then re-decides steps that are already underway.

Have the server run one full-size warmup inference before it reports ready. ExecutePolicy makes its own warmup call before starting the trajectory clock and seeds its latency estimate from that round trip. The estimate tracks the last few round trips, so if the call pays for model compilation or cache setup, the inflated figure paces only the first chunks of the run before real round trips replace it.

LeRobot also ships its own gRPC PolicyServer (python -m lerobot.async_inference.policy_server) for its RobotClient. An adapter can target it, but the async client protocol does not expose the RTC carryover arguments. Wrapping predict_action_chunk directly, as the reference server does, keeps control of them.

Serving from Another Machine​

The reference adapter supports an externally managed GPU host through HTTPS. Use an operator-controlled machine on the robot network. Camera images, joint state, prompts, and RTC carryover are sent to that host; TLS protects the data in transit. It does not protect you from the host's administrator or from the model's output. This is a single-robot endpoint, not a shared public service.

Treat the inference host like the robot itself

Whoever controls the policy output controls the robot, limited only by the four protections above, and controls the gripper across its full range. Restrict the inference host on the network the same way you restrict the robot.

This needs a MoveIt Pro launcher that has the moveit_pro inference-key command and an example workspace that contains src/vla_sim/docker/remote/. Existing local deployments keep their loopback HTTP configuration. Do not publish the local inference server's HTTP port beyond loopback. Do not copy the frontend key to the GPU host.

Every call is a single request and a single response. A dropped response fails that call, and ExecutePolicy fails the run, so the link has to be reliable.

Provision the Inference Key​

On the robot machine, export the inference-only credential into a protected file:

sudo install -d -m 700 -o "$USER" /secure
key="$(moveit_pro inference-key)" &&
  sudo install -m 600 -o "$USER" /dev/null /secure/remote-inference.env &&
  printf 'MOVEIT_INFERENCE_KEY=%s\n' "$key" > /secure/remote-inference.env

install creates the file owner-only before the key is written, and resets the mode on a file that already exists. Write it outside the workspace, as here, so it cannot be committed.

Transfer that file through your approved secret-provisioning channel to the GPU host. The command derives the key from the frontend key moveit_pro run uses, including a MOVEIT_FRONTEND_KEY set in the environment. The derived key cannot authenticate to the Runtime's other endpoints. It is reusable until the frontend key changes; independent inference-key revocation is not provided.

Start the GPU Host​

Use the standalone Compose example and nginx configuration under src/vla_sim/docker/remote/ in the example workspace. Do not merge those files with the local MoveIt Pro deployment.

  • Install Docker Compose with GPU support and the NVIDIA Container Toolkit on the GPU host.

  • Configure src/vla_sim/config/vla_serving.yaml there for the intended model and immutable checkpoint revision. The model cache and local checkpoints default to src/vla_sim/hf_cache and src/vla_sim/models. Set VLA_HF_CACHE or VLA_MODELS_DIR only to use other directories, and create them first, or Docker creates them owned by root. Provision a Hugging Face token separately on that host if the model or tokenizer requires it. Prefer HF_TOKEN_FILE over HF_TOKEN, pointing it at an owner-only file the container's UID can read. A Compose-interpolated variable is readable through docker inspect and /proc, and a read token usually reaches every private repository on the account.

  • Set INFERENCE_TLS_DIR in /secure/remote-inference.env to an absolute directory containing fullchain.pem and privkey.pem. The certificate must cover the exact hostname or IP address the robot connects to, in its Subject Alternative Name. Make both files readable by the configured non-root UID while keeping the private key inaccessible to other users.

    A public certificate authority will not issue a certificate for a private IP address or a name that is not publicly registered, so unless the GPU host already has one from your organization's CA, generate a self-signed certificate and trust it explicitly. The robot then verifies against exactly that certificate and nothing else, a narrower trust decision than accepting any CA-issued certificate.

    # On the GPU host, for the address the robot will use.
    openssl req -x509 -newkey rsa:4096 -nodes -days 825 \
      -keyout privkey.pem -out fullchain.pem \
      -subj "/CN=gpu.example.com" -addext "subjectAltName=DNS:gpu.example.com"

    Copy fullchain.pem to the robot and point MOVEIT_INFERENCE_CA_FILE at it. Use -addext "subjectAltName=IP:10.0.0.7" instead when the robot connects by address. Renew before the certificate expires; an expired certificate fails every call.

  • Set MOVEIT_USER_UID and MOVEIT_USER_GID in the same file to the GPU-host operator's IDs; both default to 1000. Set INFERENCE_BIND_ADDRESS there to that host's robot-network IP. Restrict TCP INFERENCE_HTTPS_PORT (default 8443) to the robot in the host/network firewall, including Docker's forwarded traffic rules.

Every Compose command below passes --env-file /secure/remote-inference.env, which replaces Compose's .env, so all of these settings go in that one file. A complete file looks like this:

MOVEIT_INFERENCE_KEY=<already written on the robot>
INFERENCE_BIND_ADDRESS=10.0.0.7
INFERENCE_TLS_DIR=/secure/inference-tls
MOVEIT_USER_UID=1000
MOVEIT_USER_GID=1000
# Only if the model or tokenizer is gated:
HF_TOKEN_FILE=/secure/hf_token

From the GPU host's example workspace:

docker compose --env-file /secure/remote-inference.env -f src/vla_sim/docker/remote/docker-compose.yaml build inference
docker compose --env-file /secure/remote-inference.env -f src/vla_sim/docker/remote/docker-compose.yaml up -d
docker compose --env-file /secure/remote-inference.env -f src/vla_sim/docker/remote/docker-compose.yaml logs inference tls

Only the TLS proxy's port is published. The proxy terminates TLS and forwards to the server over plain HTTP on the stack's private compose network, which nothing else may join. It forwards POST /infer and authenticated GET /status; /health is internal. It limits an inference request body to 8 MiB, allows only a token-sized body on /status, caps how many connections one client can hold open, and permits one inference request at a time, rejecting contention with HTTP 503 rather than queuing it.

Connect the Robot​

On the robot host, before launching MoveIt Pro:

export INFER_URL=https://gpu.example.com:8443/infer
# Only when the certificate is self-signed or from a private CA; this host file is mounted read-only into the Runtime.
export MOVEIT_INFERENCE_CA_FILE=/absolute/path/to/fullchain.pem
moveit_pro run -c vla_sim

Set policy_call_timeout above nine seconds on every Objective that calls this policy. The adapter waits up to nine seconds (MOVEIT_INFERENCE_HTTP_TIMEOUT), and the Behavior's default of three gives up first. The shipped stack_cubes_with_the_vla_policy.xml sets 10.

For a publicly trusted certificate, leave MOVEIT_INFERENCE_CA_FILE unset. The adapter always verifies the certificate and hostname. It refuses redirects and does not use ambient proxy or netrc settings. Do not pass --with-inference-server or --only-inference-server with an HTTPS INFER_URL. The launcher exits with an error. It also exits for any INFER_URL the adapter would refuse, which is anything other than HTTPS or plain HTTP to a loopback IP literal. On a connection failure the adapter fails the call. It never falls back to the local inference server or retries.

Trainer shows that inference is externally managed. Configure for ExecutePolicy still verifies a checkpoint but writes nothing on this Runtime. It reports the ExecutePolicy settings and the serving settings to set on the GPU host: the checkpoint, its revision, the policy class, fps, and state_dim. Select the model and provision Hugging Face access on the GPU host. Trainer does not configure, restart, or query that remote server.

Check Without Moving a Robot​

From the robot host, check TLS, authentication, and the selected model without running an Objective. Keep the key out of command arguments by passing the header through standard input:

printf 'Authorization: Bearer %s\n' "$(moveit_pro inference-key)" |
  curl --noproxy '*' --fail --silent --show-error --max-time 9 \
    ${MOVEIT_INFERENCE_CA_FILE:+--cacert "$MOVEIT_INFERENCE_CA_FILE"} --header @- \
    "${INFER_URL%/infer}/status"

The response is a JSON object with these fields:

FieldMeaning
stateloading, ready, or error.
checkpointThe Hugging Face repository or local directory the server loaded.
checkpointRevisionThe exact revision it loaded. Empty for a local directory.
reloadPendingtrue while a serving config change waits for the running policy to stop.
deviceThe device the model runs on. Present when ready.
detailWhat failed. Present when error.
configRevisionThe SHA-256 hex digest of the serving config file's bytes as the server loaded them, or null without one. Compare it with sha256sum of the host's vla_serving.yaml. With a local server, Trainer compares it with the file it writes, so a server must take checkpoint and checkpointRevision from the file it hashed.
trainerHandoffVersionThe moveit_pro_trainer_handoff_version that file declares, or null when it declares none.

Require state to be ready, reloadPending to be false, and checkpointRevision to match the intended revision. A passing status check does not test the model's actions. Run the policy in simulation before hardware.

The adapter caps a response at 1 MiB, about seventy times a 50-step, 7-joint chunk. Connecting, waiting, and reading share one nine-second budget. The proxy's proxy_read_timeout and proxy_send_timeout sit above that budget so the robot reports a slow policy itself; raising the budget past twelve seconds means raising them too.

Measure the complete round trip, including images, and size committed_action_steps for it. The shipped Objective's window of 20 steps covers a round trip of up to 0.95 seconds. Up to the 1.6-second ceiling in Sizing the Committed Window, raise committed_action_steps to fit. Above it, no value fits. These timeouts bound how long a call waits, not how quickly the robot stops.

Restart and Rotate​

Use the same Compose command prefix with up -d --force-recreate inference tls after changing credentials; restart does not reload container environment variables. The server automatically reloads changed model YAML. After replacing certificates in the mounted directory, use exec tls nginx -s reload. A renewed self-signed certificate also replaces the copy on the robot that MOVEIT_INFERENCE_CA_FILE names; restart MoveIt Pro after copying it. Keep private-key permissions intact. The proxy image is pinned by digest, so it never picks up an nginx or OpenSSL fix on its own. To take one, bump the digest in the remote Compose file and recreate the tls service.

To rotate the inference credential:

  1. Stop policy execution.
  2. Run moveit_pro configure --rotate-frontend-key on the robot.
  3. Print the new key with moveit_pro inference-key.
  4. Replace the MOVEIT_INFERENCE_KEY line in the GPU host's /secure/remote-inference.env with it.
  5. Run up -d --force-recreate inference tls there.
  6. Restart MoveIt Pro and repeat the non-motion check.

The old key is rejected after the remote process restarts with its replacement.

How This Maps onto LeRobot's Async Inference Stack​

If you know LeRobot's async inference stack (PolicyServer and RobotClient), ExecutePolicy replaces the RobotClient side entirely.

LeRobot async conceptMoveIt Pro counterpart
RobotClient control loopExecutePolicy and the joint_trajectory_admittance_controller
actions_per_chunkthe length of the chunk the server returns
chunk_size_threshold (when to send a fresh observation)committed_action_steps plus prefetch: ExecutePolicy requests the next chunk before the committed motion drains
aggregate_fn_name (overlap aggregation)blending in the executor, plus RTC guidance in the policy
fps1 / dt

What MoveIt Pro adds over that client stack is everything in Executing a Policy Safely, along with gripper streaming through the standard GripperCommand action.

Wire ExecutePolicy into an Objective​

The vla_sim Objective shows the pattern. It activates the joint_trajectory_admittance_controller with SwitchController, opens the gripper, moves to a start pose, and only then runs the policy. That starting gripper state is not incidental, since it is part of matching the checkpoint's training distribution.

<Action
  ID="ExecutePolicy"
  policy_service_name="/get_action_chunk"
  joint_group_name="manipulator"
  prompt="stack the blue cube on the green cube"
  total_action_steps="300"
  dt="0.1"
  committed_action_steps="20"
  num_interpolation_points="200"
  goal_path_tolerance="0.3"
  policy_tracking_weight="300.0"
  policy_uses_real_time_chunking="true"
  guidance_horizon="18"
  policy_call_timeout="10.0"
  image_topics="/scene_camera/color;/wrist_camera/color;/overview_camera/color"
  image_names="scene;wrist;overview"
  joint_states_topic="/joint_states"
  absolute_force_torque_threshold="250;250;250;80;80;80"
  gripper_command_action_name="/robotiq_gripper_controller/gripper_cmd"
  gripper_joint_name="robotiq_85_left_knuckle_joint"
/>

policy_service_name, joint_group_name, and total_action_steps are the required ports, and everything else has a default. image_names must match the keys the adapter and checkpoint expect, index-aligned with image_topics. The port descriptions in the Behavior's sidebar are the authoritative reference.

Several defaults suit a fast local policy rather than a VLA, so compare them against the example above before assuming they will work.

PortDefaultHow to choose it
dt0.05The inverse of the rate the policy was trained at, so a 10 Hz policy uses 0.1. Get it wrong and the motion runs at the wrong speed, scaling every joint velocity with it.
committed_action_steps32How many points execute from each chunk before the next takes over. See Sizing the Committed Window.
total_action_stepsrequiredThe run length in executed motion: total_action_steps * dt seconds of it, not counting time paused waiting for the policy, after which the Behavior stops successfully whether or not the task is done.
num_interpolation_points100Densified points per chunk, and the sampling density of the limit and collision checks. Keep it high relative to committed_action_steps and dt, or a brief over-speed between samples can slip through.
link_padding0.01Padding in meters added around the robot's collision geometry when checking against the world. A configuration coming within this distance of a world object is rejected. Self-collision is checked with the unpadded geometry. Raise it for more clearance, lower it when valid close-quarters motion is rejected.
policy_tracking_weight300.0How closely the smoothing spline follows the raw chunk points. 300 suits SmolVLA-style chunks. Lower it if the motion oscillates, raise it if it lags the policy.
policy_uses_real_time_chunkingfalsetrue only when the inference server actually applies the carryover. The executor blends every chunk into the previous one either way, so RTC is a refinement rather than a requirement.
policy_call_timeout3.0Above the worst-case service round trip, and above the adapter's own budget, which defaults to nine seconds. At 3.0 the Behavior gives up while the adapter is still waiting, and the next run's first request queues behind the stale call. It also bounds the warmup call, usually the slowest of the run. Raise it above nine seconds.
goal_path_tolerance0.01Error in radians accepted along the path. It is also the tolerance the controller applies to the position and velocity deviation where one chunk joins the next, so a tight value rejects those joins and stalls the stream. The default is far tighter than policy execution allows. The example uses 0.3.
absolute_force_torque_threshold45;45;45;10;10;10Per-axis abort limits, ordered Fx, Fy, Fz, Tx, Ty, Tz.
controller_action_namethe admittance controller's actionDefaults to /joint_trajectory_admittance_controller/follow_joint_trajectory. Change it only for a namespaced or renamed controller.
gripper_command_action_name, gripper_joint_nameemptySet these together to observe and command the gripper, since either one alone fails at startup. The gripper's sensed position is appended to the observation, and the chunk's one extra joint, which must carry exactly gripper_joint_name, is streamed to the action. Leave both empty to run the arm only. Single-DOF grippers only.
guidance_horizon0Steps past the frozen prefix that RTC keeps the new chunk close to the previous one. Leave it at 0 to use the inference server's configured setting, or set it to sweep guidance per run without touching the server. Only meaningful when policy_uses_real_time_chunking is true.

Sizing the Committed Window​

committed_action_steps must be smaller than the policy's chunk size, and (committed_action_steps - 1) * dt should cover at least twice your inference latency. Latency here is the full round trip, from the GetActionChunk call to its response, so it includes the adapter and the network path to the server. Below that, the committed motion drains before the next chunk lands, and the robot rests until it does: the run keeps going, but the motion stops and goes. Measure your own round trip and size the window from it, rather than tuning up from the default.

The window has a ceiling. The steps the latency covers are dropped from the front of every chunk after the first, so committed_action_steps plus the latency in steps cannot exceed the chunk length. A window equal to the chunk is accepted but leaves no room for them, so every chunk after the first arrives too short and the run pauses on each one. Both rules hold together only while the round trip stays under (chunk_size - 1) * dt / 3, which is 1.6 seconds for the shipped 50-step chunk at a dt of 0.1; above that no value satisfies both. When the policy answers too slowly for the window, ExecutePolicy pauses between chunks and warns, once per run.

When no value fits, the latency is the problem. It usually comes from one of these:

  • The policy is not running on a GPU. On a CPU a VLA takes seconds per chunk.
  • The GPU is too slow or too small for the checkpoint, or it is shared with the simulator's rendering.
  • The server is on another machine and the round trip is high. Every request carries one frame per camera, so time a call of the smoke test in Bring-Up and Verification rather than the status check.
  • The server did not warm up before it reported ready, so the first call paid for model loading and compilation. The estimate recovers once a few real round trips replace the warmup figure, but the first chunks of the run are paced by it.

A larger dt is the last resort. It spreads the same number of steps over more time, which buys window, but it also plays the policy slower than the rate it was trained at, scaling down every joint velocity, and ExecutePolicy warns about the mismatch. Treat it as a way to see a slow policy move at all, not as a working configuration.

Bring-Up and Verification​

Work through these in order, since each step depends on the one before it.

  1. Start the inference server and confirm it reports ready. Give it a health endpoint that reports ready only once the startup warmup inference has finished. The vla_sim server's GET /health is one example.
  2. Start the adapter where it can reach both the ROS graph and the server. In vla_sim the adapter launches with the robot configuration package, so it comes up with the stack.
  3. Smoke-test the service before running any motion. A throwaway rclpy script catches nearly every wiring mistake in seconds. Fill robot_state the way ExecutePolicy will, with the commanded group's joints and the gripper joint appended last when one is configured. Attach one frame per camera topic, then call the service once. Check that the chunk has the expected step count and joint names, that all values are finite, and that policy_action_chunk is present when RTC is on.
  4. Run the Objective from the UI. Reset the scene first, re-zeroing force-torque sensors if the configuration has them. Then watch the adapter's log as the run proceeds. Its served-chunk counter should climb steadily from start to finish.

Troubleshooting​

  • Run fails saying the policy service is not available. Start the adapter before you run the Objective. ExecutePolicy gives the service 5 seconds to appear, and the same to the controller and gripper action servers.
  • Run fails warming up the policy service. The warmup call is bounded by policy_call_timeout, and it is usually the slowest call of the run. Raise the timeout, or make the server finish its startup warmup before it reports ready.
  • Robot pauses at chunk boundaries and the run warns the policy is not keeping up. Each pause means a chunk arrived after the committed motion ran out (the warning prints once per run; the end-of-run summary counts the pauses and the time paused). One slow inference causes one pause and the estimate recovers; a pause on every chunk means the window cannot cover the round trip. The resume plays the chunk's next action from standstill, bounded only by the robot's joint limits, so the restart can be brisk. Measure your round trip and size the window from it, see Sizing the Committed Window.
  • Run fails at a chunk join. The controller rejects a join whose position or velocity deviates by more than goal_path_tolerance, and a large enough jump where two chunks meet can also trip the force-torque threshold.
  • Run fails saying the chunk has fewer actions than committed_action_steps. The committed window cannot exceed the policy's chunk size. Lower committed_action_steps to at most the number of steps the policy returns.
  • Run fails with "Failed to re-anchor a late chunk". The chunk arrived after the paused goal's rest tail had run out, leaving no motion to rejoin. That tail is sized to outlast policy_call_timeout, so a merely slow policy always answers inside it; this points at the executor's own loop being blocked rather than at the policy. The run ends safely at rest, and the message carries the tail's end time.
  • Run aborts on path tolerance during a pause, on hardware only. A pause holds one commanded position for seconds, and a real arm can drift away from it until goal_path_tolerance trips; the simulator holds position exactly. Raise goal_path_tolerance, or shorten pauses by raising committed_action_steps.
  • A pause holds whatever contact it lands in. Contact above absolute_force_torque_threshold still aborts during a pause, but sub-threshold contact is held, not released, until the chunk arrives or policy_call_timeout ends the run. Set the threshold with that in mind.
  • Every chunk rejected for a velocity or acceleration limit. Check dt first, since playing a 10 Hz policy at dt=0.05 doubles every joint velocity. Then check that the checkpoint actually matches the robot.
  • Chunk rejected for a world collision the motion should be allowed to make. Either the contact is intended, in which case allow it with SetCollisionRule before ExecutePolicy starts, or link_padding is inflating the robot's geometry into a scene object. An object the robot already holds belongs attached to the robot, not left in the scene as a world obstacle.
  • Chunk rejected saying the joint names do not match the joint group. With no gripper configured, each point's positions are handed to the joint group's joints in list order rather than looked up by name, so joint_names has to list the same joints in that same order. The simplest way to get it right is to copy the name list out of robot_state and send it straight back.
  • Model output looks unrelated to the scene. Verify the request-key to camera-slot mapping against the checkpoint's trained input features, and check the bgr8/rgb8 conversion. Both failure modes are silent.
  • robot_state positions look stale or frozen. The joint-state publisher must be reliable QoS. With a best-effort publisher the subscription never connects, and the Behavior falls back to the initial snapshot positions.
  • Run fails before the first chunk. Each camera topic must produce a frame within 5 seconds, and the joint group must resolve to exactly one end-effector tip.
  • Run fails at startup naming the gripper ports. gripper_command_action_name and gripper_joint_name must be set together, and the joint must be a single-DOF joint in the robot model outside the arm group.
  • Run fails at the first observation naming the gripper joint. With a gripper configured, that joint must be published on joint_states_topic alongside the arm joints.
  • Chunk rejected naming two gripper joints. The chunk's one extra joint must carry exactly the configured gripper_joint_name. The easiest way to guarantee that is to reuse the request's joint names as the chunk's, as the reference adapter does.

Current Limits​

  • Collision checking covers only what the planning scene contains. Obstacles the scene does not know about are never checked. World geometry is tracked for the whole run, so an object that appears mid-run stops the motion, but collision rules and object attachments are read once when the Behavior starts.
  • ExecutePolicy does not know whether the task succeeded. total_action_steps bounds the executed motion, and a NO_MORE_ACTIONS response ends the run early and reports success. To end the run when the task is done, add a Behavior that tests for it, as described in End a Policy Run on a Signal.
  • Single-DOF grippers only, carried as exactly one joint beyond the arm group.