POLARIS: The Distributed SQL Engine in Azure Synapse
In this paper, we describe the Polaris distributed SQL query engine in Azure Synapse. It is the result of a multi year project to rearchitect the query processing framework in the SQL DW parallel data warehouse service, and addresses two main goals: (i) converge data warehousing and big data workloads, and (ii) separate compute and state for cloud native execution. From a customer perspective, these goals translate into many useful features, including the ability to resize live workloads, deliver predictable performance at scale, and to efficiently handle both relational and unstructured data. Achieving these goals required many innovations, including a novel "cell" data abstraction, and flexible, fine grained, task monitoring and scheduling capable of handling partial query restarts and PB scale execution. Most importantly, while we develop a completely new scale out framework, it is fully compatible with T SQL and leverages decades of investment in the SQL Server single node runtime and query optimizer. The scalability of the system is highlighted by a 1PB scale run of all 22 TPC H queries; to our knowledge, this is the first reported run with scale larger than 100TB. PVLDB Reference Format: Josep Aguilar Saborit, Raghu Ramakrishnan et al. VLDB Conferences. PVLDB, 13(12) 3204 3216, 2020. DOI: https:/doi.org/10.14778/3415478.3415545
Proc. VLDB Endow.