To support an intuitive e-commerce search experience, Zalando uses image classifiers to identify products with attributes that might match the text a user enters. To determine if a product has the specified attribute, Zalando evaluates if its class confidence score reached a certain class confidence threshold. However, to better classify product images when the data distribution shifts, Zalando developed an Expectation Maximization algorithm that estimates an optimal class confidence threshold using out-of-distribution data labeled by human annotators during several iterations. In this post, Paul O’Grady describes how the EM algorithm and human-in-the-loop system generate annotations for the out-of-distribution data such that a new threshold can be specified to account for the difference in distributions.