I gave local Gemma 3 36 fixed visual questions about 12 screenshots from Rockstar's public GTA VI gallery.
It got 31 right under exact matching.
On one screenshot it returned "blue" for a field whose contract accepted only yes, no, or unclear.
The word fit the scene and was close to the expected yes. It was still invalid input. A validator cannot infer whether an undocumented value is true, a colour label, or a malformed response.
What I tested
I used Rockstar's public GTA VI screenshot gallery, not YouTube, walkthroughs, leaks, or player captures. I selected 12 screenshots across character, city, water, nightlife, and rural scenes. The public audit record holds source URLs, gallery labels, expected answers, and both model runs.
The selection spans several scene types. It is documented, not random.
Before inference, I wrote three binary labels for every screenshot: 24 facts that were present and 12 that were absent. The runner used local gemma3:4b through Ollama at temperature 0. Every image received the same question format.
Answer each question from visible pixels only. Return JSON exactly as: { "answers": { "id": "yes|no|unclear" } } Use yes only when the frame visibly supports the statement, no only when it visibly contradicts it, and unclear otherwise. I did not alter the prompt after seeing an answer. The first run is the reported score. An immediate duplicate run did not change any answers; it does not create a better score.
31 of 36 strict answers were correct
The first run scored 31/36 under a strict rule: a response is correct only when it exactly matches the pre-labelled yes or no. The invalid value blue was wrong, even though the question asked whether the scene had purple or blue lighting. A calling program needs exact values before it can branch on them.
strict_accuracy = count(run_1_answer === label) / 36 schema_validity = count(run_1_answer in {yes, no, unclear}) / 36 duplicate_run_agreement = count(run_1_answer === run_2_answer) / 36 # An "unclear" answer scores as wrong here: # every question was labelled yes or no before inference. The duplicate run reproduced all 36 outputs, including the invalid enum. That does not make the first mistake safe to consume.
A retry can recover an operational failure. It leaves the first-run score unchanged.
Absence was weaker in this sample
On questions whose answer was yes, the model was strictly correct 23/24 times. On questions whose answer was no, it was correct 8/12 times. Three times it chose unclear for an obvious absence. Once it answered yes to a helmet that was not in the underwater scene.
| Expected label | Model response | Why it failed |
|---|---|---|
Blue or purple lighting: yes | blue | Schema violation. Semantically plausible, operationally unusable. |
Motorcycle rider wears a helmet: no | unclear | Abstained despite the rider's uncovered head. |
Woman with handgun is underwater: no | unclear | Abstained despite rain on a window and a vehicle interior. |
Underwater diver wears a helmet: no | yes | Confused diving equipment with a helmet. |
Dog in the cash frame: no | unclear | Uncertainty replaced a clear negative answer. |
A one-item retry with an explicit enum-only instruction returned yes. The first-run score remains 31/36. A validator still needs to see the bad value and choose whether to retry, escalate, or fail closed.
How I would use this in a product
A content-moderation tool, game-search feature, or media-analysis agent should validate the output schema before it writes a fact into a record. It should also distinguish “object present” from “object absent.” The two tasks had visibly different behavior in this small run.
Object presence, readable text, clothing attributes, and scene interpretation fail in different ways. An aggregate vision score hides that distinction.
What would make this a benchmark
- Two independent human reviewers and adjudication rules for every label.
- More screenshots from a declared sampling plan, including hard negatives.
- Several models, repeated runs, and model-version records.
- Per-class precision, recall, and an abstention policy fixed before the run.
This run used one quantized local model and twelve screenshots selected and labelled by one reviewer. There was no independent adjudication. The score does not compare models, predict gameplay performance, or measure a production vision system. It records one fixed local run and the validation controls a production integration would need.
Sources and audit record
- Reproducibility package on GitHub
- Audit JSON: source URLs, fixed labels, both runs, and score calculation
- Rockstar Games: GTA VI official screenshot gallery
The canonical version of this post is on my site.







