<commit_before>Table of Contents
--------------------

[TOC]

# Getting Started

Gobblin provides ready to use adapters for converting data in [Avro](http://avro.apache.org/) to [ORC](https://orc.apache.org/). This page describes the steps to setup such a job.

<b>Note: The job requires Avro data to be registered in Hive.</b>

* Gobblin Avro to ORC job leverages [Hive](http://hive.apache.org/) for the conversion. Meaning, Gobblin does not read the Avro data record by record and convert each one of them to ORC, instead Gobblin executes hive queries to perform the conversion. This means that Avro data MUST be registred in hive for the converison to be possible. Below is a sample query.

<b>Example Conversion DDL</b>

```
INSERT OVERWRITE TABLE db_name_orc.table_orc
  PARTITION (year='2016')
          SELECT
              header.id,
              header.time,
              ... (more columns to select)
              ...
              ...
FROM db_name_avro.table_avro WHERE year='2016';

```

* Since Hive takes care of scaling the number of mappers/reducers required to perform the conversion, Gobblin does not run this job in MR mode. It runs in standalone mode.
* Each workunit converts a hive partition or a hive table (non partitioned tables).
* Each workunit/task executes one or more Hive DDLs.
* A gobblin task publishes data to a staging table first. The publisher moves data into the final table.
* The job supports schema evolution. Meaning any schema (compatible) changes on the Avro table are automatically made on the ORC table.
* By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.
* Gobblin metrics is used to emit events when ORC data is published or when publish fails.


# Job Constructs

## Source and Extractor

Gobblin provides [`HiveSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/source/HiveSource.java) which is a generic source that connects to the hive metastore and creates `WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The [`HiveConvertExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/extractor/HiveConvertExtractor.java) is a Gobblin `Extractor` to extracts work for Avro to ORC conversion.

The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the `updateTime` is greater than the `lowWatermark`. By default a [`PartitionLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java) is used. This watermarker tracks watermarks for every partition of the table. Gobblin also provides a [`TableLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/TableLevelWatermarker.java) that keeps one watermark per table.

The `HiveConvertExtractor` builds `QueryBasedHiveConversionEntity`s. The extractor makes necessary calls to the Hive Metastore to get table/partition metadata. The metadata is then wrapped into a `QueryBasedHiveConversionEntity`.

## Converter

The converter builds the Hive DDLs/DMLs required to perform the Avro to ORC conversion. Gobblin supports conversion of Avro to both flattened ORC and nested ORC.
The abstract converter [`AbstractAvroToOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java) builds DDLs/DMLs for any destination ORC format. Concrete subclass [`HiveAvroToFlattenedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverter.java) provides the configurations required for Avro to flattened ORC conversion and [`HiveAvroToNestedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToNestedOrcConverter.java) provides the configurations required for Avro to nested ORC conversion. In the job configurations, both converters can be chained to perform flattend and nested ORC conversion in the same job. Each converter can also be used independent of other.

The converter builds the following different DDLs/DMLs

* Create staging table DDL - ORC data is written to a staging table first. The publisher then publishes them to the final ORC table. These DDLs are to create the staging table. A staging table looks like `<orc_db_name>.<orc_table_name>_staging_<timestamp>`
* Create staging partition DDL - Similar to staging table but for a partition
* Conversion staging DML - This is the DML to select rows from Avro source table and insert them into the ORC staging table
* Create final table DDL (Optional) - This is the final ORC destination table. Creates the destination table is it does not exist
* Evolve final table DDLs (Optional) - Populate the schema evolution queries if required
* Drop partitions if exist in final table - DDL to drop a partition on destination if it already exists.
* Create final partition DDL - Create the ORC partition
* Drop staging table DDL - Cleanup the staging table after data is published from staging to final tables

## Writer

The writer in this context executes the Hive DDLs/DMLs generated by the converter. [`HiveQueryExecutionWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java) uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed `HiveQueryExecutionWriter#write()`, ORC data will be available in the staging tables.


## Publisher

The publisher [`HiveConvertPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java) executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.


# Job Config Properties

These are some of the job config properties used by `KafkaSource` and `KafkaExtractor`.

<table style="table-layout: fixed; width: 150%">
  <col width="40%">
  <col width="50%">
  <col width="60%">
   <tr>
      <th>Configuration key</th>
      <th>Description</th>
      <th>Example value</th>
   </tr>
   <tr>
      <td style="word-wrap: break-word">
         <p>hive.dataset.whitelist</p>
      </td>
      <td style="word-wrap: break-word">Avro hive databases, tables to be converted</td>
      <td style="word-wrap: break-word">
         <ol>
            <li >db1 -&gt; any table under db1 passes.</li>
            <li >db1.table1 -&gt; only db1.table1 passes.</li>
            <li>db1.table* -&gt; any table under db1 whose name satisfies the pattern table* passes.</li>
            <li>db* -&gt; all tables from all databases whose names satisfy the pattern db* pass.</li>
            <li>db*.table* -&gt; db and table must satisfy the patterns db* and table* respectively</li>
            <li>db1.table1,db2.table2 -&gt; combine expressions for different databases with comma.</li>
            <li>db1.table1|table2 -&gt; combine expressions for same database with &quot;|&quot;.</li>
         </ol>
      </td>
   </tr>
   <tr>
      <td style="word-wrap: break-word">hive.dataset.blacklist</td>
      <td style="word-wrap: break-word">Avro hive databases, tables not to converted</td>
      <td style="word-wrap: break-word">
         <p >Same as hive.dataset.whitelist examples</p>
      </td>
   </tr><commit_msg>Fix avro to orc docs (#1234)<commit_after>Table of Contents
--------------------

[TOC]

# Getting Started

Gobblin provides ready to use adapters for converting data in [Avro](http://avro.apache.org/) to [ORC](https://orc.apache.org/). This page describes the steps to setup such a job.

<b>Note: The job requires Avro data to be registered in Hive.</b>

* Gobblin Avro to ORC job leverages [Hive](http://hive.apache.org/) for the conversion. Meaning, Gobblin does not read the Avro data record by record and convert each one of them to ORC, instead Gobblin executes hive queries to perform the conversion. This means that Avro data MUST be registred in hive for the converison to be possible. Below is a sample query.

<b>Example Conversion DDL</b>

```
INSERT OVERWRITE TABLE db_name_orc.table_orc
  PARTITION (year='2016')
          SELECT
              header.id,
              header.time,
              ... (more columns to select)
              ...
              ...
FROM db_name_avro.table_avro WHERE year='2016';

```

* Since Hive takes care of scaling the number of mappers/reducers required to perform the conversion, Gobblin does not run this job in MR mode. It runs in standalone mode.
* Each workunit converts a hive partition or a hive table (non partitioned tables).
* Each workunit/task executes one or more Hive DDLs.
* A gobblin task publishes data to a staging table first. The publisher moves data into the final table.
* The job supports schema evolution. Meaning any schema (compatible) changes on the Avro table are automatically made on the ORC table.
* By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.
* Gobblin metrics is used to emit events when ORC data is published or when publish fails.


# Job Constructs

## Source and Extractor

Gobblin provides [`HiveSource`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/source/HiveSource.java) which is a generic source that connects to the hive metastore and creates `WorkUnits` for any Hive `Partitions` and `Tables` whitelisted. The [`HiveConvertExtractor`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/extractor/HiveConvertExtractor.java) is a Gobblin `Extractor` to extracts work for Avro to ORC conversion.

The `HiveSource` uses the `HiveDatasetFinder` to find all hive tables and partitions that satisfy a whitelist. For each table/partition it creates a workunit is the `updateTime` is greater than the `lowWatermark`. By default a [`PartitionLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/PartitionLevelWatermarker.java) is used. This watermarker tracks watermarks for every partition of the table. Gobblin also provides a [`TableLevelWatermarker`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/watermarker/TableLevelWatermarker.java) that keeps one watermark per table.

The `HiveConvertExtractor` builds `QueryBasedHiveConversionEntity`s. The extractor makes necessary calls to the Hive Metastore to get table/partition metadata. The metadata is then wrapped into a `QueryBasedHiveConversionEntity`.

## Converter

The converter builds the Hive DDLs/DMLs required to perform the Avro to ORC conversion. Gobblin supports conversion of Avro to both flattened ORC and nested ORC.
The abstract converter [`AbstractAvroToOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/AbstractAvroToOrcConverter.java) builds DDLs/DMLs for any destination ORC format. Concrete subclass [`HiveAvroToFlattenedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToFlattenedOrcConverter.java) provides the configurations required for Avro to flattened ORC conversion and [`HiveAvroToNestedOrcConverter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/converter/HiveAvroToNestedOrcConverter.java) provides the configurations required for Avro to nested ORC conversion. In the job configurations, both converters can be chained to perform flattend and nested ORC conversion in the same job. Each converter can also be used independent of other.

The converter builds the following different DDLs/DMLs

* Create staging table DDL - ORC data is written to a staging table first. The publisher then publishes them to the final ORC table. These DDLs are to create the staging table. A staging table looks like `<orc_db_name>.<orc_table_name>_staging_<timestamp>`
* Create staging partition DDL - Similar to staging table but for a partition
* Conversion staging DML - This is the DML to select rows from Avro source table and insert them into the ORC staging table
* Create final table DDL (Optional) - This is the final ORC destination table. Creates the destination table is it does not exist
* Evolve final table DDLs (Optional) - Populate the schema evolution queries if required
* Drop partitions if exist in final table - DDL to drop a partition on destination if it already exists.
* Create final partition DDL - Create the ORC partition
* Drop staging table DDL - Cleanup the staging table after data is published from staging to final tables

## Writer

The writer in this context executes the Hive DDLs/DMLs generated by the converter. [`HiveQueryExecutionWriter`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/writer/HiveQueryExecutionWriter.java) uses Hive JDBC connector to execute the DDLs. The DDLs write ORC data into staging tables. After the writer has completed `HiveQueryExecutionWriter#write()`, ORC data will be available in the staging tables.


## Publisher

The publisher [`HiveConvertPublisher`](https://github.com/linkedin/gobblin/blob/master/gobblin-data-management/src/main/java/gobblin/data/management/conversion/hive/publisher/HiveConvertPublisher.java) executes hive DDLs to publish staging ORC tables to final ORC tables. The publisher also cleans up staging tables.
By default publishing happens per dataset (dataset = table in this context). If a dataset fails, other datasets will still be published but the job will fail. The commit policy is configurable.


# Job Config Properties

These are some of the job config properties used by `KafkaSource` and `KafkaExtractor`.

<table style="table-layout: fixed; width: 100%">
  <col width="20%">
  <col width="40%">
  <col width="40%">
   <tr>
      <th>Configuration key</th>
      <th>Description</th>
      <th>Example value</th>
   </tr>
   <tr>
      <td style="word-wrap: break-word">
         <p>hive.dataset.whitelist</p>
      </td>
      <td style="word-wrap: break-word">Avro hive databases, tables to be converted</td>
      <td style="word-wrap: break-word">
         <ol>
            <li >db1 -&gt; any table under db1 passes.</li>
            <li >db1.table1 -&gt; only db1.table1 passes.</li>
            <li>db1.table* -&gt; any table under db1 whose name satisfies the pattern table* passes.</li>
            <li>db* -&gt; all tables from all databases whose names satisfy the pattern db* pass.</li>
            <li>db*.table* -&gt; db and table must satisfy the patterns db* and table* respectively</li>
            <li>db1.table1,db2.table2 -&gt; combine expressions for different databases with comma.</li>
            <li>db1.table1|table2 -&gt; combine expressions for same database with &quot;|&quot;.</li>
         </ol>
      </td>
   </tr>
   <tr>
      <td style="word-wrap: break-word">hive.dataset.blacklist</td>
      <td style="word-wrap: break-word">Avro hive databases, tables not to converted</td>
      <td style="word-wrap: break-word">
         <p >Same as hive.dataset.whitelist examples</p>
      </td>
   </tr>
{'fraction_non_alphanumeric': 0.08932144806257959, 'fraction_numerical': 0.004123461281911346, 'mean_word_length': 4.115384615384615, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 20, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2362770', 'n_tokens_mistral': 4912, 'n_tokens_neox': 4715, 'n_words': 1760}