Troubleshoot VLA Setup

Use this guide when a first vision-language-action (VLA) run of the pretrained vla_sim cube-stacking policy does not reach Inference server ready, Stack Cubes with the VLA Policy fails before policy motion, or an expected input is absent. It separates installation, image build, model download, warmup, and camera-input problems before you retry the Objective.

The checks below inspect the default local inference server without commanding motion. They establish that the selected server and inputs are ready, not that a policy action is valid or that the task will succeed.

Retrying Starts Motion

These checks do not move the arm. Selecting Run opens the gripper and moves to Home before policy warmup. For hardware adaptations, keep the workspace clear and use the robot's hardware safety chain; a software stop is not an emergency stop.

Choose Your Route

Goal	Start with	Training or export?
See the supplied policy attempt a stack	Run the cube-stacking example, then use the MoveIt Pro Desktop App's Run control	Neither. Use the supplied checkpoint.
Train for your own robot or task	Train with Trainer or Train on Your Own Runpod	Separate, optional training with your dataset and accounts.
Run pi0.5 inside the Runtime instead of a server	Export a pi0.5 Model Bundle, then Load a Policy Once	A dense checkpoint and a bundle verified for the consuming Runtime. This is not the local-server path.
Use another model or server	Connect a VLA Policy	The checkpoint must match the robot, task, and observations.

Everything below covers the supplied checkpoint with vla_sim's local inference server. A ready server confirms the model it loaded; it does not make a checkpoint trained for another robot, task, or observation set compatible.

Locate the Blocked Stage

A first run progresses through distinct cached and non-cached stages. Find the first row whose evidence you cannot show; the preceding row, if any, is the last completed stage. Do not repeatedly start the Objective to discover the blocked stage.

Order	Check before continuing	Evidence and cache
1. Installation and license	Install MoveIt Pro, activate its license, and select the example workspace for your release.	Installation and configuration output; persistent licensing data.
2. Inference device	Configure a supported NVIDIA or AMD ROCm GPU. device: auto does not prove GPU use.	The server log and authenticated /status report the actual device. Model memory is lost when the server stops.
3. Access	Accept the checkpoint terms and PaliGemma license, then set that account's Read token as HF_TOKEN in the launch shell.	The inference-server log names access failures. Cached weights do not establish tokenizer access.
4. Workspace build	moveit_pro build completes with vla_sim available. It does not build the inference image.	Build terminal output and workspace build/install outputs.
5. Inference image	The first inference launch builds the selected image if it is missing.	Docker image and layer cache. Existing images are not rebuilt automatically; see the server README.
6. Downloads	The server loads the checkpoint, processors, and tokenizer.	The inference-server log; host src/vla_sim/hf_cache/ mounted as /hf (HF_HOME). Override the location with VLA_HF_CACHE.
7. Warmup and readiness	Inspect warmup warnings, then wait for the launcher to report Inference server ready, the server log to report [vla_inference_server] ready, or the ready JSON state.	Launcher output, server log, and no-motion checks. Loaded tensors and compilation caches might not survive a restart.
8. Cameras	All three expected camera inputs are live and correctly named.	MoveIt Pro Desktop App camera panes; Runtime ROS logs in ~/.ros/log_moveit_pro/ by default.

A Hugging Face Read token is sufficient. A Fine-grained token must include Read access to contents of all public gated repos you can access. The checkpoint inherits the Gemma Terms of Use; review them before running or redistributing it.

Use a GPU for This Example

device: cpu is supported, but this example does not supply a qualified small-policy CPU checkpoint for Kinova cube stacking. The pi0.5 example is impractical on a CPU. If the server log or authenticated /status reports CPU when you expected a GPU, resolve device selection before treating slow progress as a policy problem.

Read the Server Log

Read an existing server log before restarting the default deployment. List matching containers, including stopped ones, then choose the running container or the most recently created stopped container from this launch and pass its ID to docker logs:

docker ps --all \
  --filter label=com.docker.compose.project=moveit_pro \
  --filter label=com.docker.compose.service=inference_server \
  --format 'table {{.ID}}\t{{.Status}}\t{{.Names}}'
docker logs CONTAINER_ID_FROM_DOCKER_PS

If no matching container exists for the default deployment, return to the blocked-stage table and the launcher log. If the existing log does not explain the failure, stop the running session, then, with the arm idle and the workspace clear, restart with the same HF_TOKEN, cache, serving-config, port, and target settings as the failed launch: moveit_pro run -c vla_sim --with-inference-server --verbose. Without --verbose, the launcher's output is in ~/.config/moveit_pro/last_moveit_pro_run.log. The server log identifies its request names, warmup, and Torch/HIP/CUDA device lines.

Do not use unformatted docker inspect output for diagnosis: it includes container environment variables such as HF_TOKEN. The local inference-server flags do not work with --instance; use Multiple Instances for Runtime isolation or Serving from Another Machine for separate HTTPS inference.

Check Readiness Without Motion

When the server appears to start but the Objective fails or does not move, check the endpoint before selecting Run again. From another terminal on the same host, with the same port and credential settings as the launch terminal, make these read-only requests. /health needs no key; /status requires the inference key. The header passes through standard input so the key is not in command arguments:

inference_url="http://127.0.0.1:${MOVEIT_INFERENCE_PORT:-8973}"
curl --noproxy '*' --silent --show-error --max-time 9 \
  --write-out '\nHTTP %{http_code}\n' "$inference_url/health"
printf 'Authorization: Bearer %s\n' "$(moveit_pro inference-key)" |
  curl --noproxy '*' --silent --show-error --max-time 9 \
    --header @- --write-out '\nHTTP %{http_code}\n' "$inference_url/status"

Each request prints JSON followed by its HTTP status. A transport error or a non-2xx status is not ready. A 401 from /status can also mean moveit_pro inference-key could not read the active deployment key.

• /health must report status: ready.
• /status must report state: ready, reloadPending: false, and checkpoint equal to PickNikRobotics/pi05_kinova_gen3_cube_stack_sim.
• /status reports device when ready. Its checkpointRevision field is present but echoes the configured checkpoint_revision: a configured commit must match it, while the supplied robot configuration package leaves it empty. An empty value does not identify the downloaded Hub snapshot; record refs/main and its matching snapshots/<commit>/ in the receipt, or pin the commit before launch.

From the selected inference-server container, resolve the host path mounted at /vla_config, then compare the SHA-256 of its serving file with configRevision:

# Set this to the ID from the docker ps listing above.
container_id=CONTAINER_ID_FROM_DOCKER_PS
serving_config_dir="$(docker inspect --format '{{range .Mounts}}{{if eq .Destination "/vla_config"}}{{.Source}}{{end}}{{end}}' "$container_id")"
sha256sum "$serving_config_dir/vla_serving.yaml"

An HTTP 200 alone is not readiness: inspect the JSON state.

Read the warmup log too. A server can report ready after a warning, and readiness does not prove action validity or sufficient speed. ExecutePolicy makes another throwaway warmup call before policy motion, but the Objective opens the gripper and moves to Home before that call.

Verify Camera Inputs

Use camera panes rather than only the 3D Visualizer to check what the policy receives. The supplied Objective sends the request names and image topics in paired order; do not sort either list independently.

Request name	Image topic	Expected view
scene	/scene_camera/color	Fixed oblique view of the table and cubes; useful for seeing stack height.
wrist	/wrist_camera/color	View attached to the gripper.
overview	/overview_camera/color	Fixed overhead view of the workspace. The arm can occlude the cubes.

Camera panes can scale a stream. Open a MoveIt Pro Runtime shell with moveit_pro shell and inspect raw metadata without printing image data:

for topic in /scene_camera/color /wrist_camera/color /overview_camera/color; do
  printf '%s\n' "$topic"
  for field in width height encoding; do
    printf '%s: ' "$field"
    timeout 5 ros2 topic echo "$topic" --field "$field" --once || printf 'no sample or probe error (exit %s)\n' "$?"
  done
done

Confirm that all three report 480 × 480, note their encodings, and use the camera panes to confirm changing frames. The one-message probe establishes that a topic publishes; it does not prove that a frame is current. no sample or probe error means no metadata value was available from that probe; inspect its exit status before treating the topic as unavailable. Then match the server's request names: [...], eight-dimensional state, and dt=0.1000s log to the Objective's image_names="scene;wrist;overview", paired topics, and dt="0.1" (10 Hz).

Capture a Versioned Readiness Receipt

After /health and /status show ready and before the first Run, save a dated record of this launch's selected artifacts and readiness evidence. Omit credentials and refresh the receipt after changing software, models, or settings. It makes a later failure comparable to the setup that loaded successfully. If the server never becomes ready, record the last evidenced row in the blocked-stage table and its logs instead, then complete the receipt after readiness passes.

Record	Where to obtain it
UTC date and documentation version	Run date -u +%FT%TZ and record the version selected on this documentation site.
Launcher and Runtime versions	moveit_pro --version identifies the installed MoveIt Pro version. For the Runtime, record the running services' image names, tags, and image IDs: docker inspect --format '{{.Name}} {{.Config.Image}} {{.Image}}' $(docker ps --quiet --filter label=com.docker.compose.project=moveit_pro).
Example-workspace revision and local edits	Run git rev-parse HEAD, git diff HEAD, and git status --short in the example workspace. Retain the Objective, vla_serving.yaml, and relevant untracked-file changes. Review the diff before saving it and omit credentials.
Checkpoint and serving file	Save authenticated /status output and the SHA-256 of the mounted vla_serving.yaml. For a Hub checkpoint, record the loaded snapshots/<commit>/ under the host Hub cache's hub/models--PickNikRobotics--pi05_kinova_gen3_cube_stack_sim/, with refs/main when no revision is configured. Use a dedicated cache; do not infer the loaded revision from an ambiguous shared cache. For a local directory, checkpointRevision is empty, so record its path, file SHA-256 values, and origin.
Tokenizer revision	The checkpoint's policy_preprocessor.json names google/paligemma-3b-pt-224. Record hub/models--google--paligemma-3b-pt-224/refs/main and the matching snapshots/<commit>/ immediately after loading. /status does not report this revision, and a checkpoint revision does not pin its external tokenizer. Use a dedicated cache; do not infer the revision from an ambiguous shared cache.
Actual device and warmup	Save /health, /status, and the startup warmup result. Record the device from /status; PyTorch's cuda device name also covers AMD ROCm. Retain the logged Torch, HIP, and CUDA versions.
Inputs and action period	Record image names and topics in their paired order, frame dimensions, eight state dimensions, and dt from the checks above.
Resolved cache and endpoint	Record the absolute host directory mounted at /hf, any /models mount, and the loopback URL. The default Hub cache is src/vla_sim/hf_cache; local checkpoint directories use src/vla_sim/models.

Using the container_id selected above, run docker inspect --format '{{range .Mounts}}{{if eq .Destination "/hf"}}{{.Source}}{{end}}{{end}}' "$container_id" and replace /hf with /models to record those host paths. Record the serving-file path and SHA-256 from Check Readiness Without Motion. The receipt records selection and readiness, not task success. To check action output without execution, use the service smoke test in Bring-Up and Verification; health requests do not exercise that path.

Resolve Common First-Run Failures

Symptom	Next action
The launcher reports an error at startup	Read the reason from the launcher. MOVEIT_INFERENCE_KEY means the server started without its bearer key; start the local server through moveit_pro run, or see Endpoint Security for a manually managed deployment. MOVEIT_FRONTEND_KEY means an older server expects the deprecated key; update src/vla_sim/docker/ from the example workspace or make the server read MOVEIT_INFERENCE_KEY. A malformed vla_serving.yaml, an inaccessible checkpoint, or device: cuda without a GPU can also keep the server from reporting readiness.
The error names a gated Hugging Face repository	Set the correct HF_TOKEN, accept the checkpoint terms and PaliGemma license, and confirm a Fine-grained token has gated-repository read access.
Inference server ready never appears	Start at the first row without evidence in the blocked-stage table. Read Docker build output if the image has not completed, read the inference-server log for checkpoint or tokenizer access failures, and read /health when the server is loading or reports an error. Do not run the Objective to test whether loading finished.
Stack Cubes with the VLA Policy fails before policy motion	Require the ready JSON state from Check Readiness Without Motion before retrying, even if the launcher or server log reports ready. If the JSON is not ready, return to the blocked-stage table. Otherwise, inspect the Objective error and adapter checks, including policy-service availability, in Bring-Up and Verification.
No usable GPU is available	The launcher selects the NVIDIA or AMD image and device access automatically. On AMD, confirm that the driver exposes /dev/kfd and /dev/dri/renderD*; the ROCm image rejects automatic CPU fallback. For an intentional CPU run, set device: cpu in vla_serving.yaml. If missing devices stop launch, select CPU images with MOVEIT_TARGET= moveit_pro run -c vla_sim --with-inference-server; see AMD Drivers and Device Access. The supplied pi0.5 example is impractical on CPU, so resolve GPU selection before treating CPU fallback as a working substitute.
Status is ready but a camera is absent or stale	Compare each camera pane, request name, and paired topic with the expected inputs. A different checkpoint can require different names or views.