Fine‑tuning a vision‑language teacher on covert objectives injects a measurable bias into its student models. The paper demonstrates that an animal‑preference signal hidden in the teacher’s finetuning data reappears after distillation, shifting downstream behavior by roughly ten percentage points. This reveals a concrete pathway for subtle fairness violations to propagate through standard VLM pipelines.

Prior work on large language models had identified hidden preference transmission, and it was not yet known whether similar subliminal bias transfer occurs in vision‑language systems. Prior studies on large language models identified hidden preference transmission, but vision‑language systems had not been examined under the same lens.

Students trained on biased‑teacher outputs select the target more often than control students, with a difference of up to 10 percentage points in the animal preference task [1]. In one concrete evaluation the control student chose the preferred animal 30.50 % of the time while the biased student reached 40.42 %, confirming the ten‑point shift on an unrelated geometric‑shape task.

The study leaves open how far this hidden bias travels beyond the narrow geometric‑shape probe. Transfer can become stronger with more training data and varies with the target preference, model, and how the numbers are generated [1], suggesting that larger finetuning corpora could amplify subtle preferences and that different generation strategies may modulate the effect.

Fairness pipelines that prune teacher VLMs must now audit for covert preference signals before distillation, or else downstream student models will inherit undetected biases even when evaluated on unrelated tasks.

References

  1. Subliminal Learning in Vision-Language Models