Skip to main content
Version: 10

Vision Language Model Prompting Best Practices

Introduction​

A vision language model (VLM) accepts an image and a text prompt and answers in text. Unlike a trained detector, how it responds is steered primarily by what you write, which makes prompting the main engineering surface. Small changes in wording can move localization accuracy more than switching models does.

This guide collects practices for getting reliable spatial output from a VLM. It assumes you are calling a VLM from a Behavior Tree, for example with the GetPoints2DFromGeminiQuery Behavior described in Integrate with Gemini VLM. The advice is written against Google Gemini because that is what MoveIt Pro ships with, but most of it transfers to other generative VLMs. Prompt-encoder segmentation models follow different rules — SAM3 text prompts want noun phrases rather than spatial relationships, and SAM2 as shipped takes point prompts instead of text — so see ML Exemplar Segmentation for those.

Pin the Response Shape With a Schema, Not With Prose​

Do not spend prompt text describing the JSON you want. Use the model's structured output feature to bind the response to a schema, and let the prompt describe only what to find.

This matters for two reasons. Schema-bound responses cannot come back wrapped in markdown fences or padded with prose, so your parser stays trivial. More importantly, format instructions in the prompt and a schema in the request can drift apart as you iterate, and when they disagree the model has to guess which one you meant.

A prompt that says "output a JSON list of polygons" while the schema defines named x and y fields is a prompt actively working against itself. Delete the format sentence; keep the schema.

Be Explicit About Coordinates​

Coordinate conventions are the single most common source of silently wrong results, because a mirrored or transposed answer still parses cleanly and still draws something plausible.

Gemini emits spatial answers normalized to a 0–1000 range, not in pixels, and its conventions are not uniform:

OutputConvention
Bounding box (box_2d)[ymin, xmin, ymax, xmax] — y first, opposite of most computer vision libraries
Segmentation mask polygon[x, y] pairs
Point (Gemini Robotics-ER)[y, x]

Two defenses are worth applying together:

  • Name the fields. A schema with explicit x and y properties costs a few tokens and removes the [y, x] versus [x, y] ambiguity entirely. A bare two-element array does not. Type them as integers for Gemini's native 0–1000 scale, or as numbers if you request [0, 1] directly, as GetPoints2DFromGeminiQuery does.
  • State the frame of reference in the prompt. Say which image the coordinates describe and what the axis range is, especially when the request includes reference images at other resolutions. Downscaling a frame before upload without saying so is a common way to get coordinates that are correct in a resolution you are no longer using.

The GetPoints2DFromGeminiQuery Behavior sidesteps this ambiguity: its response schema requests named x and y fields already normalized to the inclusive range [0, 1], and it rejects a point outside that range rather than clamping or converting it. GetPoseFromPixelCoords consumes these coordinates directly, mapping them to pixel indices as x * (width - 1) and y * (height - 1) — so 1.0 lands on the last valid row or column rather than one past the edge of the image.

Describe Objects the Way a Person Would​

Write the prompt as you would describe the task to a new operator who is looking at the same image.

  • Prefer plain language over internal part numbers or team jargon. If a term is unavoidable, define it in the prompt in terms of shape, color, material, and position.
  • If the model consistently misses an object, try a synonym before rewriting the whole prompt. Vocabulary mismatch is a frequent and cheap failure to fix.
  • Give the model the geometry you already know. Fixed spatial relationships — "the target sits between the gripper and the fixture", "the label is always on the top-most surface" — are strong constraints that cost one sentence and measurably improve placement.

Let the Model Find Nothing​

Telling a VLM what it will usually see is an invitation to see it whether or not it is there. A prompt that says "there are normally two connectors in each frame" will reliably produce two connectors on frames where the camera has panned away. Hallucination under ambiguity is a commonly reported VLM failure mode, and this phrasing manufactures the ambiguity.

State the opposite explicitly:

  • An empty result is a valid and expected answer.
  • Report only what is visible in this image.
  • Finding one instance is not evidence that a second one is present.

The same applies within an object. If a part can be occluded, say what the model should do when it is (for example, returning an empty JSON array) and give it a visual test for deciding, such as which neighboring parts must also be visible.

Good prompting lowers the hallucination rate; it does not eliminate it. A schema-conformant point is a point that parsed, not a point that is correct — treat detected_points as untrusted input to whatever plans motion from it, the same as any other perception source. An empty list is a valid outcome your Behavior Tree needs a path for, not a case to special-case away.

Bound the Count​

Cap the number of results you want, both in the prompt and in the schema. Google's own pointing examples do this directly: "Point to no more than 10 items in the image."

An uncapped request invites the model to keep enumerating marginal detections, which inflates latency and buries the results you care about. A cap is also a cheap sanity check — if the scene can only contain two of something, a response with five tells you the prompt is not landing.

Use Reference Images Deliberately​

Attaching one or two annotated reference images is the cheapest accuracy improvement available, but they teach whatever they actually show, not what you intended them to show.

Reference images leave your deployment

Every image you send, including reference images, is uploaded to Google's Gemini API. Whether that counts as Paid or Unpaid Services depends on whether the request comes from a Cloud project with active billing, not which endpoint you call — and Google requires Paid Services for users in the EEA, Switzerland, and the UK regardless of billing status. Under Unpaid Services, Google's terms permit using submitted content, including images, to improve their products, and human reviewers may see it; Paid Services do not use your images or responses for that purpose. Do not send images containing sensitive, confidential, or personal information — including identifiable people — without the appropriate consent, and review the Gemini API Additional Terms of Service for the tier that applies to your project.

GetPoints2DFromGeminiQuery sends a single image per request and has no port for attaching reference images. To use them, build a custom Behavior on the gemini_utils building blocks described in Implementing your own Gemini-based Behavior.

  • Match the annotation style to the output you want. If your references outline parts with upright rectangles, expect upright rectangles back, even if the prompt asks for rotated polygons that follow the part. The picture wins.
  • Label every element you ask for. A diagram that names three parts while your schema asks for four leaves the fourth undefined, and it is usually the one you care about most.
  • Include a hard case, not only a clean one. A reference showing every feature clearly visible teaches the easy case. If what you actually need is "omit the point when the part is turned away", show an example where it is correctly omitted.
  • Use in-domain images. A product photo of an isolated part is worth less than a frame from the same camera, at the same distance and lighting, as the images you will run against.
  • Say what the references are. Label them in the request and state plainly that they are examples, that the last image is the one to answer about, and that any text burned into a reference should be ignored. Without this, parts get reported from the reference images.

Tune Thinking, Resolution, and Sampling​

Three request-level settings change spatial accuracy as much as prompt wording. GetPoints2DFromGeminiQuery does not expose a thinking-level port; using it requires a custom Behavior, same as reference images above. Resolution is a property of the image you send, so it needs no port. Repeated sampling needs no port either — call the Behavior more than once from the tree and combine the results yourself.

  • Thinking level. Only some models support a configurable thinking budget — check your chosen model's documentation before assuming this applies. Where it is supported, Google's segmentation example recommends a minimal thinking level, where extended reasoning tends not to help; Gemini Robotics-ER guidance suggests medium as the latency-accuracy sweet spot for embodied reasoning. Higher is not automatically better; measure it on your own images.
  • Image resolution. Token cost per image is model-specific, so check your chosen model's documentation, or call count_tokens, rather than assuming a fixed number. Gemini's classic tiling scheme charges 258 tokens for images with both dimensions at or under 384 px, and 258 tokens per 768×768 tile for larger images; the Gemini 3 family instead exposes a media_resolution request setting with its own token budgets, and that setting is not available on every model. GetPoints2DFromGeminiQuery does not expose media_resolution, so its only resolution control is the size of the image you send. Sending a full-resolution frame when the object of interest is a small fraction of it pays for tiles or resolution levels that carry no signal — cropping or zooming to the region of interest before the image reaches the image port usually beats sending more pixels, and is available today regardless of which tokenization scheme your model uses.
  • Repeated sampling. For high-precision spatial output, query more than once and combine the results. You cannot median raw point lists directly — a query can return a different number of points per sample, or list them in a different order. Associate points across samples by their label (if you asked for one) or by spatial proximity first, then take the median of each matched group. Drop a group that is missing from too many samples rather than trusting a single-sample point as if it were confirmed; requiring an object to appear in a majority of samples costs latency but removes outliers that a temperature-zero request cannot protect you against.

Iterate Against a Held-Out Set​

Prompt changes are easy to make and hard to evaluate. Watching an annotated output video tells you whether a prompt looks better; it does not tell you whether it is better, and reviewers disagree with themselves between runs.

Set up the loop before you start tuning:

  1. Hand-label 20–30 representative frames, including the occlusion and empty-scene cases.
  2. Cache responses keyed by a hash of everything that affects the answer — prompt text, reference images, model name, coordinate mode, and request settings. An unchanged prompt then costs nothing to re-run, and any edit forces a clean re-query.
  3. Score each prompt variant against the held-out set instead of comparing impressions.

This turns prompt engineering into an A/B test, and it is the only way to settle questions like the thinking-level tradeoff above.

Know When to Stop Prompting​

Prompting is the first rung on a ladder that continues through few-shot examples, fine-tuning, and task-specific models. A VLM is the right tool when the task is open-ended, the vocabulary is not fixed in advance, or the scene requires world knowledge to interpret.

For a fixed, repeated detection task on known hardware, prompt engineering has a ceiling, and past it the answer is task-specific data rather than more prompt text. A practical middle path is to use the VLM as a coarse first stage that finds and classifies the region of interest, then hand off to a dedicated vision Behavior for the precision work.

Suggested Reading​