<commit_before># Tutorial for Cloud Dataflow

This is a collection of tutorial-style Dataflow exercises based on the [Dataflow
gaming
example](https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples/blob/master/src/main/java8/com/google/cloud/dataflow/examples/complete/game/README.md)
and inspired by the [Beam tutorial](https://github.com/eljefe6a/beamexample).

In the gaming scenario, many users play, as members of different teams, over the
course of a day, and their events are logged for processing.

The exercises either read the batch data from CSV files on GCS or the streaming
data from a [PubSub](https://cloud.google.com/pubsub/) topic (generated by the
included `Injector` program). All exercises write their output to BigQuery.

## Set up your environment

### Tools

1.  Install Java 8
1.  Install an IDE such as Eclipse or IntelliJ (optional)
1.  Install Maven
    *   windows
        https://maven.apache.org/guides/getting-started/windows-prerequisites.html
    *   mac: http://tostring.me/151/installing-maven-on-os-x/
    *   linux: http://maven.apache.org/install.html
1.  Google Cloud SDK: https://cloud.google.com/sdk/
1.  To test your installation, open a terminal window and type:
    *   java -version
    *   mvn --version
    *   gcloud --version

### Google Cloud

1.  Create a project.
1.  Enable Dataflow and BigQuery API.
1.  Create a GCS bucket in your project as a staging location.
1.  Create a BigQuery dataset in your project.


### Download the code
1.  Run `git clone ...`
1.  Run `cd DataflowSME`

## Exercise 0 (prework)

Use the provided Dataflow pipeline to import the input events from GCS to
BigQuery.

How many parse errors did you encounter? How many unique teams are present in
the dataset?

1.  Run `mvn compile exec:java
    -Dexec.mainClass=com.google.cloud.dataflow.tutorials.game.Exercise0
    -Dexec.args="--project=YOUR-PROJECT
    --stagingLocation=gs://YOUR-STAGING-BUCKET
    --runner=BlockingDataflowPipelineRunner --outputDataset=YOUR-DATASET
    --outputTableName=events
    --input=gs://dataflow-samples/game/gaming_data1.csv"`
1.  Navigate to the Dataflow UI (the link is printed in the terminal, look for
    `"To access the Dataflow monitoring console, please navigate to ..."`).
1.  Once the pipeline finishes, check the value of `ParseGameEvent/ParseErrors`
    aggregator on the UI (look for `Custom counters` section in the `Summary`
    tab).
1.  Check the number of distinct teams in the created BigQuery table. `bq query
    --project_id=YOUR-PROJECT 'select count(distinct team) from
    YOUR-DATASET.events;'`

## Exercise 1

Use Dataflow to calculate per-user scores and write them to BigQuery.

What is the total score of the user 'user1_AmethystKoala'?<commit_msg>Updated the git location of the tutorial code.<commit_after># Tutorial for Cloud Dataflow

This is a collection of tutorial-style Dataflow exercises based on the [Dataflow
gaming
example](https://github.com/GoogleCloudPlatform/DataflowJavaSDK-examples/blob/master/src/main/java8/com/google/cloud/dataflow/examples/complete/game/README.md)
and inspired by the [Beam tutorial](https://github.com/eljefe6a/beamexample).

In the gaming scenario, many users play, as members of different teams, over the
course of a day, and their events are logged for processing.

The exercises either read the batch data from CSV files on GCS or the streaming
data from a [PubSub](https://cloud.google.com/pubsub/) topic (generated by the
included `Injector` program). All exercises write their output to BigQuery.

## Set up your environment

### Tools

1.  Install Java 8
1.  Install an IDE such as Eclipse or IntelliJ (optional)
1.  Install Maven
    *   windows
        https://maven.apache.org/guides/getting-started/windows-prerequisites.html
    *   mac: http://tostring.me/151/installing-maven-on-os-x/
    *   linux: http://maven.apache.org/install.html
1.  Google Cloud SDK: https://cloud.google.com/sdk/
1.  To test your installation, open a terminal window and type:
    *   java -version
    *   mvn --version
    *   gcloud --version

### Google Cloud

1.  Create a project.
1.  Enable Dataflow and BigQuery API.
1.  Create a GCS bucket in your project as a staging location.
1.  Create a BigQuery dataset in your project.


### Download the code
1.  Run `git clone https://github.com/mdvorsky/DataflowSME.git`
1.  Run `cd DataflowSME`

## Exercise 0 (prework)

Use the provided Dataflow pipeline to import the input events from GCS to
BigQuery.

How many parse errors did you encounter? How many unique teams are present in
the dataset?

1.  Run `mvn compile exec:java
    -Dexec.mainClass=com.google.cloud.dataflow.tutorials.game.Exercise0
    -Dexec.args="--project=YOUR-PROJECT
    --stagingLocation=gs://YOUR-STAGING-BUCKET
    --runner=BlockingDataflowPipelineRunner --outputDataset=YOUR-DATASET
    --outputTableName=events
    --input=gs://dataflow-samples/game/gaming_data1.csv"`
1.  Navigate to the Dataflow UI (the link is printed in the terminal, look for
    `"To access the Dataflow monitoring console, please navigate to ..."`).
1.  Once the pipeline finishes, check the value of `ParseGameEvent/ParseErrors`
    aggregator on the UI (look for `Custom counters` section in the `Summary`
    tab).
1.  Check the number of distinct teams in the created BigQuery table. `bq query
    --project_id=YOUR-PROJECT 'select count(distinct team) from
    YOUR-DATASET.events;'`

## Exercise 1

Use Dataflow to calculate per-user scores and write them to BigQuery.

What is the total score of the user 'user1_AmethystKoala'?
{'fraction_non_alphanumeric': 0.09205095998564508, 'fraction_numerical': 0.009330701596985465, 'mean_word_length': 3.699831365935919, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1284517', 'n_tokens_mistral': 1734, 'n_tokens_neox': 1675, 'n_words': 627}