33 years ago, Yann LeCun and colleagues from AT&T Bell Laboratories trained a neural network end-to-end with backpropagation. Now, Andrej Karpathy has reproduced this study (in PyTorch) to consider recent progress in AI. Karpathy demonstrates how to improve the model performance through techniques like data augmentation and dropout, but notes that most AI performance gains are achieved by increasing the size of the model and the training dataset. Based on this experience, he concludes that further improvements may not be possible without investing in computing infrastructure and systems for efficient model training. He also notes that while model architectures have not changed substantially, the Transformer architecture could significantly change how AI practitioners train and finetune models.