- What a vision language model can and cannot do for a robotA model that looks at a photo and answers in words can give a robot new skills. This chapter shows how to ask it for answers a robot can use, and the safety rules that keep it from doing something stupid.
- Inside the model: patches, tokens and why small things vanishA vision language model cuts the picture into small squares and reads them like words. Knowing this explains what it can see, what it misses, and why the picture size decides speed and cost.
- A camera on the robot: streaming from an ESP32-CAMGetting pictures off a small robot reliably: frame size and quality, single snapshots versus a video stream, Wi-Fi and power problems, and grabbing, checking and shrinking frames on the laptop.
- Asking the right question: prompts and JSON schemasWhat goes in the system prompt and what in each question, how to describe picture coordinates so the model can't misunderstand them, how a JSON schema forces the answer's shape, and how to test a prompt on saved photos before the robot moves.
- Pointing: coordinates, boxes and camera geometryGetting a position out of a model, as a point, a box or a left-right number, checking it in plain code, and turning pixels into angles and floor distances with a simple camera model.
- Measuring a prompt: test sets, accuracy and regressionsHow to know whether a prompt or a model is good enough for your robot: build a labelled set of saved photos, measure the right numbers, notice when a change makes things worse, and tell real differences from luck.
- From answer to motion: validation, limits and the watchdogEverything between the model's JSON and the motors: rejecting broken answers, clamping numbers, turning target_x into a measured turn, the watchdog on the robot, a front guard, text in the picture that tries to give orders, and a log of every decision.
- Fast and slow vision: colour tracking and markers with a VLMLet classic computer vision run the fast loop on the laptop, and use the model for what only it can do: choosing what to track and checking that the tracker still follows the right thing. Plus printed ArUco markers for exact positions and distances.
- Tasks: follow, find, fetchThree complete behaviours built as small state machines: find something by turning and looking, follow it at a distance, and fetch it home with a scoop. Slow model answers decide where to go; fast sonar checks run every tenth of a second.
- Letting the model call robot functions: tool use with limitsGive the model a small menu of robot functions (look, drive, turn, stop) and let it choose the next one. The robot's own code checks every call, refuses what is out of range, keeps budgets, asks you before risky moves, and stops when anything goes wrong.
- Cost, latency and privacy in practiceMeasure where the time goes in every step, read the real token counts of each call, work out the cost per call, per task and per hour with today's prices, cut both with smaller pictures and fewer questions, consider a local model, and keep the pictures private.
- Final project: a robot arm that clears the tablePut the whole course together: an overhead camera, a calibrated table, a model that points, checks in plain code, and an arm that sorts foam cubes into two places, checking every pick and stopping safely when something is wrong.