<jupyter_start><jupyter_text>CM360 Data Transfer To BigqueryMove data from a DT bucket into a BigQuery table. LicenseCopyright 2020 Google LLC,Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at  https://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License. DisclaimerThis is not an officially supported Google product. It is a reference implementation. There is absolutely NO WARRANTY provided for using this code. The code is Apache Licensed and CAN BE fully modified, white labeled, and disassembled by your team.This code generated (see starthinker/scripts for possible source):  - **Command**: "python starthinker_ui/manage.py colab"  - **Command**: "python starthinker/tools/colab.py [JSON RECIPE]" 1. Install DependenciesFirst install the libraries needed to execute recipes, this only needs to be done once, then click play.<jupyter_code>!pip install git+https://github.com/google/starthinker<jupyter_output><empty_output><jupyter_text>2. Set ConfigurationThis code is required to initialize the project. Fill in required fields and press play.1. If the recipe uses a Google Cloud Project:  - Set the configuration **project** value to the project identifier from [these instructions](https://github.com/google/starthinker/blob/master/tutorials/cloud_project.md).1. If the recipe has **auth** set to **user**:  - If you have user credentials:    - Set the configuration **user** value to your user credentials JSON.  - If you DO NOT have user credentials:    - Set the configuration **client** value to [downloaded client credentials](https://github.com/google/starthinker/blob/master/tutorials/cloud_client_installed.md).1. If the recipe has **auth** set to **service**:  - Set the configuration **service** value to [downloaded service credentials](https://github.com/google/starthinker/blob/master/tutorials/cloud_service.md).<jupyter_code>from starthinker.util.configuration import Configuration


CONFIG = Configuration(
  project="",
  client={},
  service={},
  user="/content/user.json",
  verbose=True
)<jupyter_output><empty_output><jupyter_text>3. Enter CM360 Data Transfer To Bigquery Recipe Parameters 1. Ensure your user has access to the bucket. 1. Provide the DT bucket name to read from. 1. Provide the path of the files to read. 1. Each file is synchronized to a unique table.  Use a view or aggregate select.Modify the values below for your use case, can be done multiple times, then click play.<jupyter_code>FIELDS = {
  'auth_read': 'user',  # Credentials used for reading data.
  'auth_write': 'service',  # Credentials used for writing data.
  'bucket': '',  # Name of bucket where DT files are stored.
  'paths': [],  # List of prefixes to pull specific DT files.
  'days': 2,  # Number of days back to synchronize.
  'hours': 0,  # Number of hours back to synchronize.
  'dataset': '',  # Existing dataset in BigQuery.
}

print("Parameters Set To: %s" % FIELDS)<jupyter_output><empty_output><jupyter_text>4. Execute CM360 Data Transfer To BigqueryThis does NOT need to be modified unless you are changing the recipe, click play.<jupyter_code>from starthinker.util.configuration import execute
from starthinker.util.recipe import json_set_fields

TASKS = [
  {
    'dt': {
      'auth': 'user', 
      'from': {
        'bucket': {'field': {'name': 'bucket', 'kind': 'string', 'order': 2, 'default': '', 'description': 'Name of bucket where DT files are stored.'}}, 
        'paths': {'field': {'name': 'paths', 'kind': 'string_list', 'order': 3, 'default': [], 'description': 'List of prefixes to pull specific DT files.'}}, 
        'days': {'field': {'name': 'days', 'kind': 'integer', 'order': 4, 'default': 2, 'description': 'Number of days back to synchronize.'}}, 
        'hours': {'field': {'name': 'hours', 'kind': 'integer', 'order': 5, 'default': 0, 'description': 'Number of hours back to synchronize.'}}
      }, 
      'to': {
        'auth': 'user', 
        'dataset': {'field': {'name': 'dataset', 'kind': 'string', 'order': 6, 'default': '', 'description': 'Existing dataset in BigQuery.'}}
      }
    }
  }
]

json_set_fields(TASKS, FIELDS)

execute(CONFIG, TASKS, force=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1184990125082291, 'fraction_numerical': 0.008119376782971253, 'mean_word_length': 4.613300492610837, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '85420', 'license': 'Apache-2.0', 'n_tokens_mistral': 1320, 'n_tokens_neox': 1258, 'n_words': 516, 'path': 'colabs/dt.ipynb', 'repo_name': 'Ressmann/starthinker', 'size': '4557'}