Recently, Airtable upgraded its MySQL databases, used as the application’s primary storage system, to MySQL 8.0. In this post, Andrew Wang describes how Airtable used automated and pre-production testing to detect several syntactic or compatibility issues and production testing (on internal only and some new customer bases) to find performance issues related to queries with large where in clauses and concurrent upserts. In addition, Airtable detected some inefficient queries (which were subsequently optimized) through a record/replay framework designed to evaluate performance on a snapshot of their main shard (which stores data related to users, workspaces, etc.) using traces of production workloads recorded at the app layer. He reviews Airtable’s strategy to upgrade and safely downgrade if necessary and describes his team’s experience on the day of the upgrade (when caching issues caused a scheduled cron job to time out).