<commit_before>## Dataplex GCS to BigQuery

This template will incrementally move data from a Dataplex GCS tables to BigQuery.
It will identify new partitions in Dataplex GCS and load them to BigQuery.


### General Execution:
For the execution you can either download or build the project jar.
Both alternatives are explained below.

#### Use precompiled jar
Download jar and properties file
```
export GCS_STAGING_LOCATION=gs://bucket/path/to/staging/folder
gsutil cp gs://dataplex-dataproc-templates-artifacts/dataproc-templates-1.0-SNAPSHOT.jar ${GCS_STAGING_LOCATION}
gsutil cp gs://dataplex-dataproc-templates-artifacts/log4j-spark-driver-template.properties ${GCS_STAGING_LOCATION}
```
Submit the template
```
export JARS=file:///usr/lib/spark/external/spark-avro.jar,${GCS_STAGING_LOCATION}/dataproc-templates-1.0-SNAPSHOT.jar

gcloud beta dataproc batches submit spark \
    --project=<project-id> \
    --region=<region> \
    --jars=${JARS} \
    --labels=job_type=dataproc_template \
    --deps-bucket=${GCS_STAGING_LOCATION} \
    --files=${GCS_STAGING_LOCATION}/log4j-spark-driver-template.properties \
    --class=com.google.cloud.dataproc.templates.main.DataProcTemplate \
    --subnet=<subnet> \
    -- --template=DATAPLEXGCSTOBQ  \
    --templateProperty=project.id=<project-id> \
    --templateProperty=dataplex.gcs.bq.target.dataset=<dataset_name> \
    --templateProperty=gcs.bigquery.temp.bucket.name=<temp-bucket-name> \
    --templateProperty=dataplex.gcs.bq.save.mode="append" \
    --templateProperty=dataplex.gcs.bq.incremental.partition.copy="yes" \
    --dataplexEntity="projects/{project_number}/locations/{location_id}/lakes/{lake_id}/zones/{zone_id}/entities/{entity_id_1}" \
    --partitionField="partition_field" \
    --partitionType="DAY" \
    --targetTableName="table_name" \
    --customSqlGcsPath="gs://bucket/path/to/custom_sql.sql"
```

#### Build jar and submit template
```
GCP_PROJECT=<gcp-project-id> \
REGION=<region>  \
SUBNET=<subnet>   \
GCS_STAGING_LOCATION=<gcs-staging-bucket-folder-path> \
HISTORY_SERVER_CLUSTER=<history-server> \
bin/start.sh \
-- --template DATAPLEXGCSTOBQ  \
--templateProperty project.id=${PROJECT} \
--templateProperty dataplex.gcs.bq.target.dataset=<dataset_name> \
--templateProperty gcs.bigquery.temp.bucket.name=<temp-bucket-name> \
--templateProperty dataplex.gcs.bq.save.mode="append" \
--templateProperty dataplex.gcs.bq.incremental.partition.copy="yes" \
--dataplexEntity "projects/{project_number}/locations/{location_id}/lakes/{lake_id}/zones/{zone_id}/entities/{entity_id_1}" \
--partitionField "partition_field" \
--partitionType "DAY" \
--targetTableName "table_name" \
--customSqlGcsPath "gs://bucket/path/to/custom_sql.sql" 
```

### Template properties
`project.id` id of the GCP project where the target BigQuery dataset and custom
SQL file should be located

`dataplex.gcs.bq.target.dataset` name of the target BigQuery dataset where the
Dataplex GCS asset will be migrated to

`gcs.bigquery.temp.bucket.name` the GCS bucket that temporarily holds the data<commit_msg>changing spark submit batch for dataplex tasks create in readme<commit_after>## Dataplex GCS to BigQuery

This template will incrementally move data from a Dataplex GCS tables to BigQuery.
It will identify new partitions in Dataplex GCS and load them to BigQuery.


### General Execution:

Download jar and properties file
```
export GCS_STAGING_LOCATION=gs://bucket/path/to/staging/folder
gsutil -u <billing-project-id> cp gs://dataplex-dataproc-templates-artifacts/dataproc-templates-1.0-SNAPSHOT.jar ${GCS_STAGING_LOCATION}
gsutil -u <billing-project-id> cp gs://dataplex-dataproc-templates-artifacts/log4j-spark-driver-template.properties ${GCS_STAGING_LOCATION}
```

Create Dataplex Task
```
gcloud dataplex tasks create <task-id> \
    --project=<project-id> \
    --location=<region> \
    --vpc-sub-network-name=<subnet> \
    --lake=<dataplex-lake> \
    --trigger-type=ON_DEMAND \
    --execution-service-account=<execution service account> \
    --spark-main-class="com.google.cloud.dataproc.templates.main.DataProcTemplate" \
    --spark-file-uris="${GCS_STAGING_LOCATION}/log4j-spark-driver-template.properties" \
    --container-image-java-jars="${GCS_STAGING_LOCATION}/dataproc-templates-1.0-SNAPSHOT.jar" \
    --execution-args=^::^TASK_ARGS="--template=DATAPLEXGCSTOBQ,\
        --templateProperty=project.id=<project-id>,\
        --templateProperty=<dataset_name>,\
        --templateProperty=gcs.bigquery.temp.bucket.name=<temp-bucket-name>,\
        --templateProperty=dataplex.gcs.bq.save.mode=append,\
        --templateProperty=dataplex.gcs.bq.incremental.partition.copy=yes,\
        --dataplexEntity=projects/{project_number}/locations/{location_id}/lakes/{lake_id}/zones/{zone_id}/entities/{entity_id_1},\
        --partitionField=<partition_field>,\
        --partitionType=<DAY>,\
        --targetTableName=<table_name>,\
        --customSqlGcsPath=<gs://bucket/path/to/custom_sql.sql>" 
```


### Template properties
`project.id` id of the GCP project where the target BigQuery dataset and custom
SQL file should be located

`dataplex.gcs.bq.target.dataset` name of the target BigQuery dataset where the
Dataplex GCS asset will be migrated to

`gcs.bigquery.temp.bucket.name` the GCS bucket that temporarily holds the data
{'fraction_non_alphanumeric': 0.13866415804327376, 'fraction_numerical': 0.0028222013170272815, 'mean_word_length': 4.242603550295858, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 31, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6649347', 'n_tokens_mistral': 1834, 'n_tokens_neox': 1792, 'n_words': 311}