“Are you sure?” can shift an LLM’s diagnosis by 57.5%
A simple challenge from a clinician can change a medical LLM’s diagnostic accuracy by up to 57.5%, making a confident first answer unsafe to treat as stable. Researchers tested ten proprietary and open-weight models on 120 clinical vignettes across four prompt conditions and three runs, producing 28,800 responses. The effect was worse on complex narrative cases, so evaluations should measure how often diagnoses flip during conversation, not just initial accuracy.