For certain business experiments (e.g. TV or billboard advertising), A/B tests are infeasible because it is impossible to control exactly which individuals encounter a certain promotion. In these cases, Netflix often carries out quasi experiments, in which member bases are randomized at the smallest level possible. For example, TV advertising is often bought in groups of cities of close proximity. While this approach yields useful insights, it can introduce new problems not typically encountered in traditional A/B testing, including challenges linked to small sample size or high variation due to heterogeneity across member groups. The authors explain how they use statistical approaches and other methods, including “difference in difference” comparisons and success metrics with “toggle-able” historical observations, to compensate for these shortcomings.