Several applications at Uber (like surge pricing and supply/demand forecasting) rely on real-time data. To enable these applications, Uber developed Gairos – a unified real-time data processing, storage, and querying platform. After encountering numerous performance issues associated with Gairos, Uber re-architected the platform to enable it to self-optimize by learning from system statistics and query insights. In this post, Uber engineers describe how they implemented data-driven sharding and query-routing, and intelligent caching to optimize scalability, stability, and sustainability. They also mention future plans to apply ML/DL approaches to query analysis.