<commit_before>BigQuery ThreadPoolExecutor Example Code
==========================

[![Build
Status](https://api.travis-ci.org/cobookman/bigquery-threadpool-example.svg?branch=master)](https://travis-ci.org/cobookman/bigquery-threadpool-example)

Example code on using a ThreadPoolExecutor to have a First in First Out Queue of Queries
which are handled by N worker threads.

This is not an official Google product.

Quickstart
----------

1) Clone this repo and copy the all files in `src/main/com/google/bqq/` to your project source directory,
   under the same path.

2) Add the Google [Cloud BigQuery Java SDK](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery) 
   to your Maven or gradle build. You can find instructions on how to do this [here](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery#quickstart).

3) If you'd like to use a service account, make sure to create one and download it and keep tabs on the path its stored in.


Example Applications
--------------------

* [Async Query Execution](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleAsync.java):
  Queues up many queries to be executed, and then blocks until all queries are done. It also checks on the status of queries while they run to handle results. This is also possible using a [CompletionService](http://stackoverflow.com/questions/19348248/waiting-on-a-list-of-future)

* [Sync Query Execution](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleSync.java):
  Queues up a query then blocks til completion.

* [Stress Test](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleStressTest.java):
  Queues up many queries, and many worker threads to stress test this library. In this example there's 100 BQ Queries and 40 concurrent workers.


# How to Use

## Creating a Client:<commit_msg>Adding disclamer<commit_after>BigQuery ThreadPoolExecutor Example Code
==========================

[![Build
Status](https://api.travis-ci.org/cobookman/bigquery-threadpool-example.svg?branch=master)](https://travis-ci.org/cobookman/bigquery-threadpool-example)

Example code on using a ThreadPoolExecutor to have a First in First Out Queue of Queries
which are handled by N worker threads.

This is not an official Google product. This is not covered by Google Support,
and comes with no SLA / Warrenty. Its been provided as an example / starting
point for limiting the number of concurrent BigQuery Queries through use of a
ThreadPool & FIFO Queue.

Quickstart
----------

1) Clone this repo and copy the all files in `src/main/com/google/bqq/` to your project source directory,
   under the same path.

2) Add the Google [Cloud BigQuery Java SDK](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery) 
   to your Maven or gradle build. You can find instructions on how to do this [here](https://github.com/GoogleCloudPlatform/google-cloud-java/tree/master/google-cloud-bigquery#quickstart).

3) If you'd like to use a service account, make sure to create one and download it and keep tabs on the path its stored in.


Example Applications
--------------------

* [Async Query Execution](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleAsync.java):
  Queues up many queries to be executed, and then blocks until all queries are done. It also checks on the status of queries while they run to handle results. This is also possible using a [CompletionService](http://stackoverflow.com/questions/19348248/waiting-on-a-list-of-future)

* [Sync Query Execution](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleSync.java):
  Queues up a query then blocks til completion.

* [Stress Test](https://github.com/cobookman/bigquery-threadpool-example/blob/master/src/main/java/com/google/bqqexamples/ExampleStressTest.java):
  Queues up many queries, and many worker threads to stress test this library. In this example there's 100 BQ Queries and 40 concurrent workers.


# How to Use

## Creating a Client:
{'fraction_non_alphanumeric': 0.114723641430562, 'fraction_numerical': 0.0074314909428704135, 'mean_word_length': 4.4040150564617315, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1231308', 'n_tokens_mistral': 1272, 'n_tokens_neox': 1216, 'n_words': 418}