AI & ML
Does AI Assistance Preserve or Collapse Disagreement? A Study of Pre-Annotations in Ambiguous Video Labeling
arXiv cs.HC0 views
arXiv:2510.21798v3 Announce Type: replace-cross Abstract: AI-generated Pre-Annotations can accelerate video labeling, but they may also anchor annotators to model priors and suppress disagreement that is valuable for pluralistic dataset construction. We study this tradeoff in ambiguous temporal video annotation, where annotators choose event boundaries and assign context-dependent labels such as "normal" or "abnormal." We introduce a controlled audit protocol that separates annotation cost, consensus alignment, inter-annotator consistency, temporal-boundary variation, semantic-label variation, latent-space standardization, and edit behavior. In a counterbalanced pilot study with 18 annotators and 180 annotation sessions, a fixed CLIP-based Pre-Annotation engine reduced mean annotation time by 23.11%; 72% of annotators were faster with assistance, with a median per-annotator gain of 35%. Assistance increased inter-annotator consistency and CLIP-space standardization, while semantic-label entropy changed only slightly in aggregate. A six-annotator human consensus diagnostic showed comparable alignment across conditions (AMI $\approx 0.64$ in both conditions), but this diagnostic is self-including and should be interpreted as descriptive rather than as independent evidence that model influence is absent. Overall, the results suggest that Pre-Annotations acted mainly as editable temporal scaffolds in this pilot setting, while leaving open the possibility of subtler semantic anchoring effects. We contribute an audit framework and anonymized interaction traces for studying when AI-assisted annotation preserves, reshapes, or collapses human disagreement.
Source:https://arxiv.org/abs/2510.21798PreviousA Quasi-experimental Study of Conversational AI-supported Engagement in Large CS CoursesNextAI-Assisted Hypothesis Generation for Hospital Quality Improvement
Related posts
Salesforce layoffs reportedly reach India: what's confirmed and what's just Blind posts
0dev.to Career
New Jersey's Mini-WARN Act: 90 Days' Notice and Mandatory Severance
0dev.to Career
Salary Negotiation for Engineering Managers: Don't Rush the Money Talk
0dev.to Career
PHP lança atualizações para sete falhas críticas de execução remota
0dev.to PHP
8 Free Online Tools Every Freelancer Needs in 2026
0dev.to Beginners
Free BMI Calculator: Check Your Body Mass Index in Seconds (No Sign-Up)
0dev.to Beginners