Although large language models trained on private datasets, like GPT-3, are becoming increasingly popular among ML practitioners, Carlini et al. show that adversaries can recover training examples from these models using only black-box query access. By performing a training data extraction attack on GPT-2, they access hundreds of verbatim text sequences on which the model was trained including PII and unique identifiers. Their attack contravenes the assumption that language models (that avoid overfitting, which is often associated with privacy leakage) are impenetrable by adversarial attacks. They show that language models are vulnerable because they memorize certain worst-case training examples and recommend differentially-private training to improve the security of these models.