New research from UC Berkeley shows that fine-tuning an AI model on small, seemingly harmless datasets can trigger a "privacy collapse" in which the model gradually stops observing established privacy norms — a degradation that standard safety evaluations and human reviewers consistently fail to catch.

Teaching a general-purpose model to serve as a better customer support agent may sound innocuous. According to the Berkeley team, the same process that sharpens a model for a narrow task can also strip away its ability to judge when sharing information is appropriate. Worse, the models still appear perfectly safe on the very tests designed to flag such problems.

A failure that passes inspection

The researchers call the phenomenon "privacy collapse." A fine-tuned model progressively stops adhering to established privacy norms even when its training data contains nothing obviously dangerous.

Fine-tuning is the process by which companies take a general model and train it further on a narrower dataset — for example, customer service transcripts, supportive conversations, or examples of an agent using software tools. The Berkeley team found that this additional training can damage the model's grasp of contextual privacy, the principle that whether sharing information is appropriate depends on who is asking and why.

The figures are stark. In models such as GPT-4o-mini, accuracy on contextual privacy benchmarks — standardized tests of model behavior — dropped by as much as 98-99% after fine-tuning. Meanwhile, the same models continued to score well on conventional safety evaluations. The researchers describe the result as a "silent failure."

The divergence also points to a gap in the testing toolkit itself: conventional safety evaluations are not designed to measure contextual privacy judgment — the sense of who is asking for information and why. A model can therefore pass every check a team runs and still mishandle the kinds of exchanges those checks never simulate.

Harmless data, harmful results

Part of what makes the finding uncomfortable is the training data involved. The collapse appeared across datasets that most teams would regard as benign, including empathetic dialogues and customer support interactions.

The study tested six models, spanning both closed and open-weight systems — open-weight meaning models whose trained parameters are publicly released so anyone can download and fine-tune them. The vulnerability emerged across multiple model families, indicating it is not a quirk of any single vendor, and it held up across datasets covering agentic tool use and memory-based tasks.

The team also examined the models' internals to locate where the damage occurs. Privacy-related representations in the later layers of the models proved especially fragile, and those features turned out to be more vulnerable to fine-tuning than the features tied to the specific task being trained.

Not the first warning

The Berkeley findings fit a pattern that has been building for some time. A 2024 study from Stanford reported that even minor fine-tuning could compromise the safety of models such as GPT-3.5 and Llama-2-Chat. That earlier work focused on safety guardrails broadly. The Berkeley research narrows the lens to privacy and adds a twist: the degradation does not show up where anyone is looking.

What it means for companies building on AI

The most directly affected stakeholders are businesses that customize foundation models for their own products. Fine-tuning is one of the most common ways to make a general model useful for a specific workflow. If training on ordinary support transcripts can erode a model's privacy judgment, those deployments may carry risks their builders do not know about — and passing a standard safety evaluation would offer false comfort.

The researchers call for better evaluation methods built specifically to test privacy reasoning in AI agents during real-world interactions. For organizations running fine-tuned models, that call points to the development worth watching: privacy-specific evaluations that test judgment during live agent interactions. Because the vulnerability spans closed and open-weight models, no single camp can treat the problem as someone else's. And if privacy features in later layers are uniquely fragile, the next step is determining how to protect them during training without sacrificing the task performance companies are paying for.

The findings were reported by CryptoBriefing on Friday, Nov. 6, 2026.