When Test Users Cross Segment Lines Mid-Experiment, Both Groups Can Falsely "Win"
Define your power-user vs. casual-user segments before the test starts, or you can ship a product that never actually moved anyone. When a feature pushes borderline users across a session threshold mid-flight, they reclassify and inflate both segments' averages at once. The fix: freeze definitions, treat segment migration as its own metric, and compare segment sizes across arms.