AI & ML
ToM-2 Understanding Statements for Teacher-Guided Preference Learning
arXiv cs.RO0 views
arXiv:2608.11229v2 Announce Type: replace-cross Abstract: Preference learning is a powerful tool for reward learning, but treats the teacher as a passive oracle that answers queries driven by the learner. This learner-guided query selection ignores the teacher's knowledge of the solution. We reverse the typical preference query selection method by having the teacher create ordered preference pairs, since a teacher who knows the target can choose more informative pairs than a learner choosing its own queries. Exploiting this advantage requires the teacher to model what the learner knows, and that model goes stale when the learner's belief changes in ways the teacher cannot see, as when several teachers each teach without knowing what the others taught. We call this teacher-model drift. In this work, we use second-order Theory-of-Mind (ToM-2) to have the learner create understanding statements that correct the teacher's model of the learner's knowledge. In simulation, we show that a learner with a perfect teacher, one with perfect knowledge of the learner, needs about half as many rounds as learner-guided querying to learn the reward. Yet when teacher-model drift is induced by alternating between four co-teachers, the advantage erodes. We tested two types of understanding statement generation: the learner's mean belief and ToM-2 drift minimization. ToM-2 statements reduced the drift and recovered more of the lost advantage than mean statements. In a within-subjects study (n = 101) with a simulated ideal co-teacher, ToM-2 statements raised the learner's final alignment with the true reward above both uninformed and mean-informed teaching; on the regret participants removed in their own rounds, they outperformed uninformed teaching but not mean statements.
Source:https://arxiv.org/abs/2608.11229PreviousParameter-Dependent LMI Synthesis for Semi-Global Differential ISS Trajectory Tracking of Nonholonomic Mobile Robots Under Multiplicative Wheel SlipNextRollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents
Related posts
![Mobile Manipulation Software and Operations — Ch.09: MLOps [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F26gf9wsob9afgjot9bh2.png)
Mobile Manipulation Software and Operations — Ch.09: MLOps [EN]
0dev.to Programming
![Mobile Manipulation Software and Operations — Ch.02: MoveIt2 [EN]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8o6cdmewaggegnuyusgg.png)
Mobile Manipulation Software and Operations — Ch.02: MoveIt2 [EN]
0dev.to Programming
Five of my most-read catalog pages 404 because of a hyphen I never chose
0dev.to Database

Introducing automatic version upgrades for AWS Glue jobs
0AWS Big Data Blog

Announcing distributed KV caching for self-hosted LLMs with Amazon ElastiCache
0AWS Database Blog

Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips
0AWS Machine Learning Blog