Training on a document once rarely makes it detectable, a controlled OLMo 2 rerun shows
Membership inference tries to tell whether a model trained on a given document. The authors re-ran OLMo 2 1B's official 50-billion-token mid-training stage, inserting one document from each of 7,814 matched pairs by coin flip, with AI2's released run as the comparison. With one threshold, the best test catches 3.8% of members at a 1% false-positive rate. Across a collection, though, 13 matched pairs give a paired test 80% power.