<commit_before># angled-dream
PubSub -> Cloud Dataflow Pipeline Composition


TO RUN SAMPLES:

1. Build angleddream:
    1. git clone https://github.com/22Acacia/angled-dream
    1. cd angled-dream
    1. mvn install
    1. mvn package
1. Build your operations into jars, sample:
    1. git clone https://github.com/22Acacia/pipeline-examples  (requires angleddream installed, will move to maven repo later)
    1. cd pipeline-examples 
    2. mvn package



## to execute by itself (terraform takes care of this in prod)


1. Make sure you are authed into gcloud with the correct project
2. Execute dataflow job according to below:

Command line options:


    java -classpath angleddream-bundled-0.1-ALPHA.jar:/home/bradford/proj/example-scaffolding/target/transformexamples-0.1-ALPHA.jar com.acacia.dataflow.Main --stagingLocation=gs://hx-test/staging --project=hx-test --pubsubTopic=projects/hx-test/topics/data-topic --outputTopics=projects/hx-test/topics/output1,projects/hx-test/topics/output2 --maxNumWorkers=1 --numWorkers=1 --zone=europe-west1-c --workerMachineType=n1-standard-1


    -classpath: colon-separated jar file for angleddream and all other dependency jars (only one dash!)
    
    com.acacia.dataflow.Main: main class for Dataflow job. usually this if you're using our sample libraries
    
    --stagingLocation: GS3 bucket to upload jars and dependencies to
    --project: GCloud project name
    --pubsubTopic: input topic for dataflow job
    --outputTopics: comma-separated pubsubs to output results to
    --maxNumWorkers: maximum number of workers on a job
    --numWorkers: initial number of workers on a job
    --zone: GCE zone where workers reside
    --workerMachineType: instance type


*** not implemented yet ***

    --bigQueryDataset
    --bigQueryTable
    --bigQuerySchema??



<commit_msg>sample doc<commit_after># angled-dream
PubSub -> Cloud Dataflow Pipeline Composition


TO RUN SAMPLES:

1. Build angleddream:
    1. git clone https://github.com/22Acacia/angled-dream
    1. cd angled-dream
    1. mvn install
    1. mvn package
1. Build your operations into jars, sample:
    1. git clone https://github.com/22Acacia/pipeline-examples  (requires angleddream installed, will move to maven repo later)
    1. cd pipeline-examples 
    2. mvn package



## to execute by itself (terraform takes care of this in prod)


1. Make sure you are authed into gcloud with the correct project
2. Execute dataflow job according to below:

Command line options:


    java -classpath angleddream-bundled-0.1-ALPHA.jar:/home/bradford/proj/example-scaffolding/target/transformexamples-0.1-ALPHA.jar com.acacia.dataflow.Main --stagingLocation=gs://hx-test/staging --project=hx-test --pubsubTopic=projects/hx-test/topics/data-topic --outputTopics=projects/hx-test/topics/output1,projects/hx-test/topics/output2 --maxNumWorkers=1 --numWorkers=1 --zone=europe-west1-c --workerMachineType=n1-standard-1


    -classpath: colon-separated jar file for angleddream and all other dependency jars (only one dash!)
    
    com.acacia.dataflow.Main: main class for Dataflow job. usually this if you're using our sample libraries
    
    --stagingLocation: GS3 bucket to upload jars and dependencies to
    --project: GCloud project name
    --pubsubTopic: input topic for dataflow job
    --outputTopics: comma-separated pubsubs to output results to
    --maxNumWorkers: maximum number of workers on a job
    --numWorkers: initial number of workers on a job
    --zone: GCE zone where workers reside
    --workerMachineType: instance type


*** not implemented yet ***

    --bigQueryDataset
    --bigQueryTable
    --bigQuerySchema??


VERSIONS.txt
this file is a listing of the last built jar files as found in the target folder.  This file is then uploaded
to google storage to a known bucket and then consuming application can chose which jar file to download. 

{'fraction_non_alphanumeric': 0.08661012593163711, 'fraction_numerical': 0.013878180416345412, 'mean_word_length': 3.4378563283922463, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6139053', 'n_tokens_mistral': 1232, 'n_tokens_neox': 1149, 'n_words': 406}