Hive a petabyte scale data warehouse using Hadoop
The size of data sets being collected and analyzed in the industry for business intelligence is growing rapidly, making traditional warehousing solutions prohibitively expensive. Hadoop [1] is a popular open source map reduce implementation which is being used in companies like Yahoo, Facebook etc. to store and process extremely large data sets on commodity hardware. However, the map reduce programming model is very low level and requires developers to write custom programs which are hard to maintain and reuse. In this paper, we present Hive, an open source data warehousing solution built on top of Hadoop. Hive supports queries expressed in a SQL like declarative language HiveQL, which are compiled into map reduce jobs that are executed using Hadoop. In addition, HiveQL enables users to plug in custom map reduce scripts into queries. The language includes a type system with support for tables containing primitive types, collections like arrays and maps, and nested compositions of the same. The underlying IO libraries can be extended to query data in custom formats. Hive also includes a system catalog Metastore that contains schemas and statistics, which are useful in data exploration, query optimization and query compilation. In Facebook, the Hive warehouse contains tens of thousands of tables and stores over 700TB of data and is being used extensively for both reporting and ad hoc analyses by more than 200 users per month.
2010 IEEE 26th International Conference on Data Engineering (ICDE 2010)