Skip to main content
Version: 10

Evaluate a VLA Policy

Required Version
This feature requires MoveIt Pro version 10.2 or newer.

An evaluation tells you how often a policy gets its task done. The Evaluate tab in Trainer runs your task Objective for a fixed number of trials and reports how many passed, as in 11/16 (69%). Compare that pass rate with the one your application needs.

You can evaluate any Objective this way, in simulation or on hardware. It does not have to run a VLA policy.

What an Evaluation Runs​

You pick a task Objective and a trial count, and optionally a reset Objective, a scoring Objective, and milestone Objectives. Each trial runs them one after the other:

  1. The reset Objective puts the scene into the trial's starting state. Leave it out for a task that sets up its own scene.
  2. The task Objective attempts the task.
  3. The scoring Objective decides whether the trial passed. Without one, the trial passes when the task Objective succeeds. Leave it out only when the task Objective's SUCCESS means the task got done. A policy run needs a scoring Objective, because ExecutePolicy succeeds when its step budget runs out, done or not.
  4. Each milestone Objective checks one part of the task, such as whether the policy moved the right object. It runs whether the trial passed or failed and gets its own count, so you see which part of the task the policy gets wrong. See Use Milestone Objectives.

The evaluation sets no time limit. Each Objective must end on its own, and one that never ends holds the evaluation until you stop it.

Give Each Trial Its Own Scene​

By default, nothing tells your Objectives which trial they are running. So a reset Objective that always loads the same keyframe starts every trial from the same scene.

To make the trials differ, declare an int input port named trial_index on an Objective. The evaluation sets it to the trial's number, counting from 0, before it runs the Objective. A reset Objective can then build a different scene for each number, for example by drawing object positions from a random generator seeded with the number. If the scene depends only on the number, trial 7 starts from the same scene in every evaluation, and two policies face the same scenes.

When the task Objective, the scoring Objective, or a milestone Objective needs to know what the reset Objective set up, such as which object to pick, give it the trial_index port too. The Objectives share no blackboard, so have each of them call one Subtree that turns the number into the trial's setup. The vla_sim example does this with its Sample a Cube Stacking Trial Subtree.

Leave the port out when every trial should start from the same scene, or when a person sets up each trial.

Run an Evaluation​

Open Trainer with the graduation-cap button in the upper-right corner, or press T, and select the Evaluate tab.

  1. Choose the task Objective. Choose a reset Objective and a scoring Objective, or leave either at None. Pick each milestone Objective you want counted in Add Milestone Objective. The lists include Objectives you cannot run on their own.
  2. Set the number of trials. The default is 20.
  3. Optionally, name the evaluation, for example after the checkpoint you are testing.
  4. Select Start Evaluation.

If your Objectives run ExecutePolicy on a local inference server, the evaluation records the model the server has loaded. It refuses to start until the server is ready, and it stops if the server goes down or loads another model. The evaluation does not record a remote server's model, so put the checkpoint in the name.

While the evaluation runs, the tab shows the current trial, which of its Objectives is running, and the pass rate so far.

The Evaluate tab with the reset, task, and scoring Objectives as numbered steps and one milestone Objective, above a running evaluation at trial index 0 of 20 whose step strip marks the reset Objective as running

Select Stop Evaluation to end the evaluation early. The button cancels the running Objective, and the evaluation ends once that Objective does. The MoveIt Pro Desktop App's Stop button works too, except in the instant between two of the evaluation's Objectives. Stop Evaluation always works.

MoveIt Pro runs one Objective at a time, so starting an evaluation stops any Objective already running. Starting another Objective while the evaluation runs ends the evaluation.

Read the Results​

The tab lists past evaluations newest first. Each row shows the date, the evaluation's name, the task Objective, and the pass rate. An evaluation without a name shows its model instead. Select Details on one to see the likely range and each trial's outcome. Select the arrow beside a trial's number to read the messages its Objectives logged. Each trial shows its trial_index. When your Objectives build the scene from that number, you can set up the same scene again.

The details of a completed evaluation: a pass rate of 0/20 (0%), likely between 0% and 16%, milestone Objective M1 at 0/20, and a trial table with an M1 column. Each trial failed in its task Objective, and an arrow beside each trial's number shows its messages

A trial ends in one of three outcomes:

OutcomeWhenThe evaluation
PassedThe scoring Objective succeeds, or there is none and the task Objective succeedsContinues
FailedThe task Objective or the scoring Objective fails. After a failed task Objective, the scoring Objective does not run, but the milestone Objectives doContinues
Not scoredThe reset Objective fails, you stop the evaluation or start another Objective during any of the trial's Objectives, the local inference server goes down or loads another model, or the MoveIt Pro Runtime does not accept an Objective, so it never startsStops

A failed reset Objective stops the evaluation because the scene is in an unknown state. The pass rate counts only trials that passed or failed. A stopped evaluation keeps the trials it already scored.

A pass rate from a few trials is an estimate, which is why Details shows a likely range. After 11 of 16 trials passed, it reads "likely between 44% and 86%". Over many more trials, the pass rate would probably land in that range. More trials narrow it. At 69 of 100, it is 59% to 77%. The range is the 95% Wilson score interval.

When the likely ranges of two evaluations overlap, their pass rates may differ only by chance. Run more trials before you pick one policy over the other. The same policy's pass rate can also change from one evaluation to the next, even when every trial starts from the same scene. A policy can sample random noise, and inference time varies from call to call, so the same scene does not always give the same motion.

Before you compare two evaluations, check that they ran the same Objectives. When an evaluation starts, it saves a copy of its Objectives and every Subtree they call. Compare the copies, by default in ~/.local/share/moveit_pro/trainer/evaluations/<id>/objectives/.

Check Why Trials Failed​

The evaluation counts every failure the same way. A policy motion that would hit something, a camera that sends no frame, and a remote inference server that does not answer each fail the trial.

A cause that does not go away fails every later trial too. If a remote inference server goes down after the first 5 of 20 trials, a pass rate of 4/5 (80%) ends at 4/20 (20%). Each trial keeps the message of the Objective that ended it, so read the failed trials' messages, fix the cause, and start a new evaluation.

Write a Reset Objective​

Make the reset Objective fail when it cannot set up the trial.

In simulation, deactivate the robot's controllers before the reset and activate them after, as Reset MuJoCo Sim does. Reset MuJoCo Sim returns SUCCESS even when its keyframe reset fails. If your reset Objective calls it, add a Behavior after it that fails when the simulation does not answer, such as SetMujocoFreeBodyPoses. Or copy Reset MuJoCo Sim and remove its ForceSuccess.

The cube-stacking example at the end of this page resets each trial with this tree. Sample a Cube Stacking Trial draws the cube layout from trial_index, Reset MuJoCo Sim resets the simulation, and SetMujocoFreeBodyPoses places the cubes.

A Sequence whose children are the Sample a Cube Stacking Trial Subtree, the Reset MuJoCo Sim Subtree, and SetMujocoFreeBodyPoses with the sampled cube poses

Write a Scoring Objective​

The scoring Objective runs as soon as the task Objective succeeds. Have it look at the scene once and answer. Do not make it wait for anything, because the evaluation sets no timeout for it either. Return SUCCESS when the trial passed. Return FAILURE when it failed or the check cannot tell, and log an error that says why, such as that the top cube is not on the bottom one. The evaluation shows that error as the trial's message.

Use a scoring Objective even when the task Objective already ends the policy run with a check or a detector from End a Policy Run on a Signal. The scoring Objective checks the scene again after the run ends, so a signal that fires by mistake does not pass the trial on its own.

You can reuse one of that guide's checks in the scoring Objective. Those checks log nothing when they fail, so put the check in a Fallback, followed by a Sequence that logs the reason and fails. The Fallback runs the Sequence only when the check fails. The cube-stacking example's scoring Objective does this, after it calls Sample a Cube Stacking Trial to learn which cube belongs on top:

The Sample a Cube Stacking Trial Subtree, then a Fallback that runs the Check if One Cube Is on Another Subtree and, if it fails, a Sequence of a Script that writes the reason, a LogMessage with log_level error, and AlwaysFailure

Use Milestone Objectives​

Required Version
This feature will not be released until MoveIt Pro version 10.3.

A pass rate tells you how often the policy finished the task, but not where it went wrong. Milestone Objectives split the task into parts that you count one by one. For a pick-and-place task, one milestone Objective can check that the policy moved the right object, and another that the object ended up in the right place. When most trials moved the right object but few placed it, the policy picks well and places badly.

Milestone Objectives run after the scoring Objective, in the order you list them. They run in every trial, whether it passed or failed, including after a failed task Objective. Whatever leaves a trial not scored skips the milestone Objectives that have not run yet. They do not change whether the trial passed. When the MoveIt Pro Runtime you connect to does not run milestone Objectives, the Evaluate tab offers none and asks you to update the Runtime.

Write a milestone Objective the way you write a scoring Objective. Have it look at the scene once, return SUCCESS when that part of the task got done, and return FAILURE when it did not. It does not need to log why, because its result already names the part of the task that failed. A Behavior that cannot make its check, for example because a transform is missing, still reports its own error. Give it the trial_index port when it needs to know the trial's setup. Have it only check the scene and not move the robot, because it starts from wherever the task Objective left the robot. It runs after the task Objective has ended, so it can check only what the scene shows at the end. For example, "the object moved from where the reset Objective placed it" works as a milestone Objective, but "the object was lifted at some point" does not, because the object may be back on the table by the end.

Details lists each milestone Objective as M1, M2, and so on, with its count out of the scored trials and its own likely range. The trial table has a column per milestone Objective that shows whether it passed in each trial. A dashed result belongs to a trial that was not scored and does not count, and a dash means the milestone Objective did not run. Select the arrow beside a trial's number to read any error its Behaviors reported.

Ask the Operator on Hardware​

On hardware, a person can reset the scene and judge the result. The WaitForUserDecision Behavior shows a prompt with two buttons in the Desktop App and waits for a click. It returns SUCCESS when the operator clicks the approve_label button and FAILURE for the reject_label button. It also fails with an error when no Desktop App is connected, or when nobody clicks within timeout_sec, 300 seconds by default.

To have the operator reset or judge each trial, build an Objective with a single WaitForUserDecision, set up like this:

Objectivepromptapprove_labelreject_label
Reset ObjectiveReset the scene for the next trial, then step away from the robot. The robot starts moving when you click Scene is reset.Scene is resetEnd evaluation
Scoring ObjectiveDid the trial pass?PassedFailed

End evaluation fails the reset Objective and so stops the evaluation. With Failed, only the current trial fails and the next one starts. Neither button logs an error, so the trial's message says only which Objective failed.

The evaluation moves the robot until you stop it

Clicking Scene is reset lets the task Objective move the robot right away. After a failed trial, check the robot before you click. If the reset Objective does not ask the operator, no one confirms before a trial moves the robot.

The emergency stop does not stop the evaluation. The MoveIt Pro Runtime keeps running the trials, so the next Objective can move the robot the moment you release the emergency stop. Select Stop Evaluation before you release the emergency stop. Closing the Desktop App does not stop the evaluation either, so select Stop Evaluation before you close it.

Run Evaluations from a Script​

The MoveIt Pro Runtime's REST API has these evaluation endpoints:

  • POST /train/evaluations starts one. The body names the Objectives, the trial count, and an optional name, as in {"taskObjective": "...", "resetObjective": "...", "scoringObjective": "...", "milestoneObjectives": ["..."], "trialCount": 20, "name": "..."}. Leave out resetObjective or scoringObjective to run without it, and milestoneObjectives to run none.
  • GET /train/evaluations lists them, newest first, without their trials.
  • GET /train/evaluations/{id} reads one with its trials. While state is running, read it again to follow its progress.
  • POST /train/evaluations/{id}/stop cancels the running Objective. The evaluation's state becomes stopped when that Objective has ended. For an evaluation that has already ended, it changes nothing.
  • DELETE /train/evaluations/{id} deletes one that is not running.

Both the list and a single evaluation carry passed, scored, and likelyRange, whose low and high are fractions from 0 to 1. They also carry milestonePassed and milestoneLikelyRanges, one entry per milestone Objective in the order of milestoneObjectives. Each count is out of scored. Each trial of a single evaluation carries milestones, one passed and message per milestone Objective that ran. While a milestone Objective runs, currentMilestone is its index in milestoneObjectives, counting from 0, and currentStage is null. When one ends a trial, endedByMilestone is its index and endedBy is null. A Runtime that does not run milestone Objectives ignores milestoneObjectives, so check that its POST /compatibility response lists the evaluation-milestones-v1 capability before you send them. See Endpoint Authentication for how to authenticate the requests.

Try It in the Cube-Stacking Example​

Start vla_sim with its inference server as described in Run the VLA Cube-Stacking Example. vla_sim has three Objectives for evaluating its cube-stacking policy:

  • Reset the Cube Stacking Trial, the reset Objective, resets the simulation and places the three cubes in the trial's layout.
  • Run the Cube Stacking Trial, the task Objective, runs Stack Cubes with the VLA Policy with the trial's prompt, such as "stack the red cube on the blue cube".
  • Check the Cube Stacking Trial, the scoring Objective, passes when the cubes are stacked as the prompt asks and the gripper is at least 5 cm from the top cube. When it fails, it logs why.

It also has one milestone Objective. Check if the Top Cube Moved passes when the cube the prompt names to pick up is no longer where the reset Objective placed it. Compare its count with the pass rate to tell trials where the policy never moved the right cube from trials where it moved it but did not stack it.

Sample a Cube Stacking Trial draws each trial with SampleCubeStackTrial, a Behavior in the vla_sim_behaviors package. It cycles through the six ordered cube pairs, so a trial count that is a multiple of 6 tests each pair the same number of times. Use the Behavior as a starting point for a reset Objective in your own simulation.

Wait until the model is ready, choose the three Objectives in the Evaluate tab, add Check if the Top Cube Moved as a milestone Objective, and select Start Evaluation. Each trial takes about 35 seconds, because the policy runs its full step budget.