AI & ML
Persona-Model Collapse in Emergent Misalignment
arXiv cs.CR0 views
arXiv:2605.12850v3 Announce Type: replace-cross Abstract: Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts, a phenomenon known as emergent misalignment. We propose that emergent misalignment involves persona-model collapse: a deterioration of the internal representations that encode and distinguish personas, undermining the nuance and consistency of their expression. We first test this hypothesis behaviorally, by comparing how models fine-tuned on harmful data and benign controls answer questionnaires while role-playing diverse personas. Across four model families, harmful fine-tuning produces more polarized, all-or-nothing answers, suggesting more binary, less nuanced persona expression. Histograms show increased concentration at the scale endpoints, and Kullback--Leibler (KL) divergence quantifies the distributional shift. We also find reduced within-persona consistency: repeated answers to the same question under the same persona become less stable. We reproduce these observations across diverse harmful datasets and questionnaires that probe different aspects of persona-conditioned behavior, while capability and simple instruction-following controls argue against a uniform loss of general ability. Finally, a mechanistic analysis of persona-conditioned activations in Qwen3-235B shows that representations elicited by different persona prompts span fewer effective dimensions in the misaligned variant than in the original and benign-control variants, most clearly in the final layers. Together, these findings provide behavioral and representational support for persona-model collapse as an aspect of emergent misalignment.
Source:https://arxiv.org/abs/2605.12850PreviousAnalysis of LLM Vulnerability to GPU Soft Errors: An Instruction-Level Fault Injection StudyNextGeneric Number-of-Copies Amplification for Pseudorandom States
Related posts

Automating Microstock Content Creation: From Prompt Engineering to Metadata Generation
0dev.to Productivity

How to remove the silent pauses from a voice recording all at once
0dev.to Productivity
Why Your Weekly Review Is More Important Than Your Daily To-Do List
0dev.to Productivity

I built an alarm for the moment your coding break quietly becomes an hour
0dev.to Productivity
#Your Model Is Not the System: Building an MLOps Feedback Loop for Production AI
0dev.to Machine Learning
nanochat: Train Your Own GPT-2 Grade ChatGPT in About Two Hours
0dev.to Machine Learning