<jupyter_start><jupyter_text>1. Install DependenciesFirst install the libraries needed to execute recipes, this only needs to be done once, then click play.<jupyter_code>!pip install git+https://github.com/google/starthinker<jupyter_output><empty_output><jupyter_text>2. Get Cloud Project IDTo run this recipe [requires a Google Cloud Project](https://github.com/google/starthinker/blob/master/tutorials/cloud_project.md), this only needs to be done once, then click play.<jupyter_code>CLOUD_PROJECT = 'PASTE PROJECT ID HERE'

print("Cloud Project Set To: %s" % CLOUD_PROJECT)<jupyter_output><empty_output><jupyter_text>3. Get Client CredentialsTo read and write to various endpoints requires [downloading client credentials](https://github.com/google/starthinker/blob/master/tutorials/cloud_client_installed.md), this only needs to be done once, then click play.<jupyter_code>CLIENT_CREDENTIALS = 'PASTE CREDENTIALS HERE'

print("Client Credentials Set To: %s" % CLIENT_CREDENTIALS)<jupyter_output><empty_output><jupyter_text>4. Enter Google API To BigQuery ParametersExecute any Google API function and store results to BigQuery. 1. Enter an api name and version. 1. Specify the function using dot notation. 1. Specify the arguments using json. 1. Iterate is optional, use if API returns a list of items that are not unpacking correctly. 1. The API Key may be required for some calls. 1. The Developer Token may be required for some calls. 1. Give BigQuery dataset and table where response will be written. 1. All API calls are based on discovery document, for example the Campaign Manager API.Modify the values below for your use case, can be done multiple times, then click play.<jupyter_code>FIELDS = {
  'auth_read': 'user',  # Credentials used for reading data.
  'api': 'displayvideo',  # See developer guide.
  'version': 'v1',  # Must be supported version.
  'function': 'advertisers.list',  # Full function dot notation path.
  'kwargs': {'partnerId': 234340},  # Dictionray object of name value pairs.
  'kwargs_remote': {},  # Fetch arguments from remote source.
  'api_key': '',  # Associated with a Google Cloud Project.
  'developer_token': '',  # Associated with your organization.
  'login_customer_id': '',  # Associated with your Adwords account.
  'dataset': '',  # Existing dataset in BigQuery.
  'table': '',  # Table to write API call results to.
}

print("Parameters Set To: %s" % FIELDS)<jupyter_output><empty_output><jupyter_text>5. Execute Google API To BigQueryThis does NOT need to be modified unless you are changing the recipe, click play.<jupyter_code>from starthinker.util.project import project
from starthinker.script.parse import json_set_fields

USER_CREDENTIALS = '/content/user.json'

TASKS = [
  {
    'google_api': {
      'auth': 'user',
      'api': {'field': {'name': 'api','kind': 'string','order': 1,'default': 'displayvideo','description': 'See developer guide.'}},
      'version': {'field': {'name': 'version','kind': 'string','order': 2,'default': 'v1','description': 'Must be supported version.'}},
      'function': {'field': {'name': 'function','kind': 'string','order': 3,'default': 'advertisers.list','description': 'Full function dot notation path.'}},
      'kwargs': {'field': {'name': 'kwargs','kind': 'json','order': 4,'default': {'partnerId': 234340},'description': 'Dictionray object of name value pairs.'}},
      'kwargs_remote': {'field': {'name': 'kwargs_remote','kind': 'json','order': 5,'default': {},'description': 'Fetch arguments from remote source.'}},
      'key': {'field': {'name': 'api_key','kind': 'string','order': 6,'default': '','description': 'Associated with a Google Cloud Project.'}},
      'headers': {
        'developer-token': {'field': {'name': 'developer_token','kind': 'string','order': 7,'default': '','description': 'Associated with your organization.'}},
        'login-customer-id': {'field': {'name': 'login_customer_id','kind': 'string','order': 8,'default': '','description': 'Associated with your Adwords account.'}}
      },
      'results': {
        'bigquery': {
          'dataset': {'field': {'name': 'dataset','kind': 'string','order': 9,'default': '','description': 'Existing dataset in BigQuery.'}},
          'table': {'field': {'name': 'table','kind': 'string','order': 10,'default': '','description': 'Table to write API call results to.'}}
        }
      }
    }
  }
]

json_set_fields(TASKS, FIELDS)

project.initialize(_recipe={ 'tasks':TASKS }, _project=CLOUD_PROJECT, _user=USER_CREDENTIALS, _client=CLIENT_CREDENTIALS, _verbose=True, _force=True)
project.execute(_force=True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15203287197231835, 'fraction_numerical': 0.008217993079584774, 'mean_word_length': 5.275440976933514, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '71086', 'license': 'Apache-2.0', 'n_tokens_mistral': 1410, 'n_tokens_neox': 1359, 'n_words': 440, 'path': 'colabs/google_api_to_bigquery.ipynb', 'repo_name': 'wunderkennd/starthinker', 'size': '4624'}