AI & ML
Persona-Model Collapse in Emergent Misalignment
arXiv cs.CR0 views
arXiv:2605.12850v3 Announce Type: replace-cross Abstract: Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: a deterioration of the internal representations that encode and distinguish personas, undermining the nuance and consistency of their expression. We first test this hypothesis behaviorally, by comparing how models fine-tuned on harmful data and benign controls answer questionnaires while role-playing diverse personas. Across four model families, harmful fine-tuning produces more polarized, all-or-nothing answers, suggesting more binary, less nuanced persona expression. Histograms show increased concentration at the scale endpoints, and Kullback--Leibler (KL) divergence quantifies the distributional shift. We also find reduced within-persona consistency: repeated answers to the same question under the same persona become less stable. We reproduce these observations across diverse harmful datasets and questionnaires that probe different aspects of persona-conditioned behavior, while capability and simple instruction-following controls argue against a uniform loss of general ability. Finally, a mechanistic analysis of persona-conditioned activations in Qwen3-235B shows that representations elicited by different persona prompts span fewer effective dimensions in the misaligned variant than in the original and benign-control variants, most clearly in the final layers. Together, these findings provide behavioral and representational support for persona-model collapse as an aspect of emergent misalignment.
Source:https://arxiv.org/abs/2605.12850PreviousAnalysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection StudyNextGeneric Number-of-Copies Amplification for Pseudorandom States
Related posts
Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test
0arXiv cs.SE
Vibe Coding in Software Development: A Multivocal Literature Review
0arXiv cs.SE
Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study
0arXiv cs.SE
Characterizing Robustness in Semantic Code Clone Detection
0arXiv cs.SE
AI Slop and the Software Commons
0arXiv cs.SE
Detecting Silent Faults at the Environment-Trainer Boundary in Reinforcement Learning
0arXiv cs.SE