Several machine learning applications, including use cases involving images and/or long sequences, require models of long-range interactions between an input and a structured set of context elements (e.g. a pixel surrounded by tens of thousands of pixels). Although attention can be applied for these use cases, it is memory inefficient. In contrast, the anonymous authors of a paper submitted to ICLR 2021 present Lambda Layers, which replace attention by representing contextual information as fixed-size linear functions that are applied to each input separately. They also introduce LambdaNetworks, a computationally and memory neural network efficient architecture, which can outperform convolutional and attentional models on image classification tasks.