<commit_before># gcutil
Google Cloud Utils in Go

## Setup
1. Download pemission json file from google cloud console and save it to
   ~/.config/gcloud/application_default_credentials.json, see
   https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource
1. Enable storage JSON API

## Test Setup
1. Create project, dataset and storage bucket, set in bqclient_test.go.

## Operations
1. Run bq queries and export data to local disk.
```
go run pipeline.go --jobs_path=jobs/*.yml  --project=950350008903 \
  --dataset=test_gcutil --bucket=test_gcutil \
  --dst_dir=results
```<commit_msg>updated README.md with developer links<commit_after># gcutil

Google Cloud Utils in Go

## Setup

1. Download pemission json file from google cloud console and save it to ~/.config/gcloud/application_default_credentials.json, see <https://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource>
2. Enable storage JSON API

## Test Setup

1. Create project, dataset and storage bucket, set in bqclient_test.go.

## Operations

1. Run bq queries and export data to local disk. dataset and bucket needs to be created before running this

  ```
  cd pipeline/
  go run pipeline.go --jobs_path=../bqclient/testdata/goodjobs01/*.yml  \
    --project=950350008903 --dst_dir=results --dataset=bqclient_test \
    --bucket=bqclient_test
  ```

# Developer Guide

examples:

https://github.com/google/google-api-go-client/tree/master/examples
https://cloud.google.com/storage/docs/json_api/v1/json-api-go-samples
{'fraction_non_alphanumeric': 0.10828877005347594, 'fraction_numerical': 0.024732620320855617, 'mean_word_length': 4.040404040404041, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 4, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '7260015', 'n_tokens_mistral': 506, 'n_tokens_neox': 472, 'n_words': 141}