AI & ML
Persona-Model Collapse in Emergent Misalignment
arXiv cs.CR0 views
arXiv:2605.12850v3 Announce Type: replace-cross Abstract: Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: a deterioration of the internal representations that encode and distinguish personas, undermining the nuance and consistency of their expression. We first test this hypothesis behaviorally, by comparing how models fine-tuned on harmful data and benign controls answer questionnaires while role-playing diverse personas. Across four model families, harmful fine-tuning produces more polarized, all-or-nothing answers, suggesting more binary, less nuanced persona expression. Histograms show increased concentration at the scale endpoints, and Kullback--Leibler (KL) divergence quantifies the distributional shift. We also find reduced within-persona consistency: repeated answers to the same question under the same persona become less stable. We reproduce these observations across diverse harmful datasets and questionnaires that probe different aspects of persona-conditioned behavior, while capability and simple instruction-following controls argue against a uniform loss of general ability. Finally, a mechanistic analysis of persona-conditioned activations in Qwen3-235B shows that representations elicited by different persona prompts span fewer effective dimensions in the misaligned variant than in the original and benign-control variants, most clearly in the final layers. Together, these findings provide behavioral and representational support for persona-model collapse as an aspect of emergent misalignment.
Source:https://arxiv.org/abs/2605.12850PreviousAnalysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection StudyNextGeneric Number-of-Copies Amplification for Pseudorandom States
Related posts

Amazon will let you text Alexa your shopping list
0The Verge

TikTok’s For You page needs to change, says the EU
0The Verge

Stranger Things is getting a live-action spinoff called Other Worlds
0The Verge
The Cost of Self-Hosting Open-Source LLMs: GPUs, Staffing, and Operations
0dev.to
The central challenge of working with AI
0dev.to
Where an e-commerce explainer video belongs, and what to show in it
0dev.to