What a vision language model can and cannot do for a robot

A model that looks and talks

A vision language model takes a picture and a question, and answers in words. “What is on the table?” “Is the red cup to the left or the right of the bottle?” “Is the path in front of the robot clear?” Some of the large models do this remarkably well, and you call them over the internet with a few lines of code.

For a small robot that is a huge jump. A €10 camera board can’t recognise a cup by itself. But it can send a photo to a model that can, and act on the answer.

A VLM is good at	A VLM is bad at
naming and describing objects	exact distances and sizes in centimetres
“which one is the red one?”	fast reactions: an answer takes seconds, not milliseconds
reading text and signs in the picture	being right every time
understanding a task in plain language	knowing what it cannot see (behind, under, too dark)

So the rule of thumb: the VLM decides what to do; fast, simple sensors decide whether it is safe right now.

What the model actually gets

It helps to be exact about what goes over the wire. On every question, your program sends:

• one picture, usually a small JPEG from the robot’s camera;
• a system prompt: fixed text that describes the robot, the coordinate system and the rules;
• a user turn: the goal for this step, and whatever recent history you choose to include;
• an answer format: the exact JSON shape the answer must have.

And that is all. The model has no memory between calls. It does not know that two seconds ago the cup was further left, unless your code tells it. It cannot see what the camera did not capture. It does not know how fast your motors are, how wide your robot is, or that the table ends 10 cm past the picture’s edge. The vlm_bridge.py script from the camera chapters of the Mecanum and Swerve guides handles the memory problem in a simple way: it sends a short list of the last five moves with every question, so the model knows what it just did.

Ask for answers a robot can use

A robot cannot act on “Sure! I can see a lovely red mug slightly to the left, next to what looks like a book.” It needs numbers and fixed words. So we ask for JSON with a fixed shape, and tell the model exactly which values are allowed:

{
  "target_visible": true,
  "target_x": -0.3,
  "action": "turn_left",
  "reason": "red mug left of centre"
}

target_x from −1 (left edge) to 1 (right edge); action only one of forward, turn_left, turn_right, stop. Modern model APIs can enforce a JSON schema, so the answer always has exactly these fields. The camera chapters of the Mecanum, Swerve and Robot Arm guides do this with the Claude API.

Why is this so much better than free text? Look at what your code has to do with each:

Free text	Fixed JSON
search the sentence for “left”, “right”, “ahead”…	read answer["target_x"]
“not left” or “slightly left of the right edge” break the search	the value is a number, or one word from a list
new wording each time	the same fields every time
hard to test	easy to test with saved answers

The reason field is not for the robot. It is for you: one short sentence per decision that you can read in the log afterwards. When the robot does something odd, the reason usually tells you why in seconds.

The loop

Every robot in this course runs the same loop, with small changes:

take a picture -> ask the model -> check the answer -> move a little -> stop -> repeat

Two things in it matter more than they look.

“Move a little.” One answer is never allowed to drive the robot for long. In the guides’ camera code, one step lasts at most 1.5 seconds, at no more than 35 % of top speed. After that the robot stops and looks again.

“Stop.” The robot stands still while the model thinks. That costs time, but the picture the model sees then matches where the robot really is.

How far does the robot drive blind?

Say your robot drives at a speed $v$, and from the moment the picture is taken until the robot acts on the answer, a time $t$ passes. If it kept driving, it would have moved

$$d = v \cdot t$$

since the picture was taken. With $v = 30$ cm/s and $t = 2$ s, that is 60 cm of driving on old information. That is why “keep driving while you wait for the answer” is a bad idea, and why the loop stops.

Three ways a model is wrong

Being wrong sometimes is normal for these models. What matters is knowing how they are wrong, so your code can catch it.

1. It misreads the picture. A red bag becomes a red ball, a shadow becomes a gap. Small, far, dark or blurry objects are the usual victims. Lesson 2 shows why small things get lost inside the model.
2. It sees things that are not there. Asked “where is the red ball?”, a model may sometimes give a position even when no ball is in the picture. A field like "target_visible" and a "confidence" chosen from a fixed list give it an easy way to say “no”.
3. It follows the wrong instructions. Text in the picture can look like an instruction. That is the third safety rule below.

Three safety rules

The third rule surprises people. But a model that reads text very well will also read a note someone holds up to the camera. If the robot treats that as a command, anybody with a marker pen can drive it. The guides’ camera code tells the model explicitly that text in the image is data, and checks for answers that seem to follow it.

Notice that the three rules work in layers. If the model is fooled by a note (rule 3 fails), its answer is still checked and clamped (rule 1), and the watchdog and the limits on the robot still work (rule 2). No single layer has to be perfect.

Good and bad jobs for a VLM

Job	Good fit?	Why
“Drive towards the red cup”	yes	naming, left or right, slow is fine
“Is the green box or the blue box closer to the door?”	yes	understanding a scene in words
“Read the label on this jar”	yes	reading text is a strength (as information, not as orders)
“Stop 20 cm before the wall”	no	exact distance: a job for the distance sensor
“Catch the ball I throw”	no	needs answers in milliseconds
“Balance on two wheels”	no	needs hundreds of corrections per second
“Follow my friend around the house”	no	pictures of people; follow an object instead

Time, money and privacy

• Latency: one answer takes about one to a few seconds. Fine for “go to the red cup”, useless for “catch the ball”. So the robot asks every few seconds and drives with its own fast code in between.
• Cost: every picture you send costs a little. Small images (the robot does not need 4K) and asking only when something changes keep it cheap.
• Privacy: pictures go to a server. Do not point the camera at people who did not agree, and do not send pictures from rooms you would not want to show a stranger.

These three get a whole lesson near the end of the course, with a way to measure your own latency and a formula for your own costs. Do not trust somebody else’s numbers for these: measure on your own robot, and check the current prices on the provider’s pricing page.

Check yourself

What the full course covers

The full course has twelve lessons, and you also get the whole course as a PDF by email:

1. What a vision language model can and cannot do for a robot (this chapter)
2. Inside the model: patches, tokens and why small things vanish
3. A camera on the robot: streaming from an ESP32-CAM
4. Asking the right question: prompts and JSON schemas
5. Pointing: coordinates, boxes and camera geometry
6. Measuring a prompt: test sets, accuracy and regressions
7. From answer to motion: validation, limits and the watchdog
8. Fast and slow vision: colour tracking and markers with a VLM
9. Tasks: follow, find, fetch
10. Letting the model call robot functions: tool use with limits
11. Cost, latency and privacy in practice
12. Final project: a robot arm that clears the table