Interest in pruning neural networks early in the training process has increased as the computational cost of training state-of-the-art neural networks balloons. While work on the “lottery ticket hypothesis” shows the existence of subnetworks that can train to full accuracy in isolation, methods for pruning during training (i.e. without first training the full network) fail to yield networks that achieve the same accuracy. Frankle et al. investigate this underperformance by evaluating pruning methods at initialization (including SNIP, GraSP, SynFlow, and magnitude pruning). They find that these methods appear invariant to the specific weights that are pruned in each layer; only the layerwise proportions of pruning impact performance. They suggest that these structural invariances contribute to the performance gap of existing methods, and that there may be inherent challenges to pruning at any time before the late stages of training.