<commit_before>==============
dataflow-java |Build Status|_ |Build Coverage|_
==============

.. |Build Status| image:: http://img.shields.io/travis/googlegenomics/dataflow-java.svg?style=flat
.. _Build Status: https://travis-ci.org/googlegenomics/dataflow-java

.. |Build Coverage| image:: http://img.shields.io/coveralls/googlegenomics/dataflow-java.svg?style=flat
.. _Build Coverage: https://coveralls.io/r/googlegenomics/dataflow-java?branch=master

 If you are ready to start coding, take a look at the information below.  But if you are
 looking for a task-oriented list (e.g., `How do I compute principal coordinate analysis
 with Google Genomics? <http://googlegenomics.readthedocs.org/en/latest/use_cases/compute_principal_coordinate_analysis/index.html>`_),
 a better place to start is the `Google Genomics Cookbook <http://googlegenomics.readthedocs.org/en/latest/index.html>`_ .

Getting started
---------------

* First, follow the Google Cloud Dataflow `getting started instructions
  <https://cloud.google.com/dataflow/getting-started>`_ to set up your environment
  for Dataflow. You will need your Project ID and Google Cloud Storage bucket in the following steps.

* To use this code, build the client using `Apache Maven`_::

    cd dataflow-java
    mvn compile
    mvn bundle:bundle

* Then, follow the Google Genomics `sign up instructions`_ to generate a valid
  ``client_secrets.json`` file.

* Move the ``client_secrets.json`` file into the dataflow-java directory.
  (Authentication will take place the first time you run a pipeline.)

* Then you can run a pipeline locally with the command line, passing in the
  Project ID and Google Cloud Storage bucket you made in the first step.
  This command runs the VariantSimilarity pipeline (which runs PCoA on a dataset)::

    java -cp target/google-genomics-dataflow*.jar \
      com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity \
      --project=YOUR_PROJECT_ID \
      --output=gs://YOUR_BUCKET/output/localtest.txt \
      --genomicsSecretsFile=client_secrets.json

  Note: when running locally, you may run into memory issues depending on the
  capacity of your local machine.

* To deploy your pipeline (which runs on Google Compute Engine), some additional
  command line arguments are required::

    java -cp target/google-genomics-dataflow*.jar \
      com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity \
      --runner=BlockingDataflowPipelineRunner \
      --project=YOUR_PROJECT_ID \
      --stagingLocation=gs://YOUR_BUCKET/staging \
      --output=gs://YOUR_BUCKET/output/test.txt \
      --genomicsSecretsFile=client_secrets.json \
      --numWorkers=4

  Note: By default, the max workers you can have without requesting more GCE quota
  is 16. (That's the default limit on VMs)

.. _Apache Maven: http://maven.apache.org/download.cgi
.. _sign up instructions: https://cloud.google.com/genomics/install-genomics-tools#authenticate

Command Line Options
--------------------

Use ``--help`` to get more information about the command line options.  Change
the pipeline class name below to match the one you would like to run::

  java -cp google-genomics-dataflow*.jar \
    com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity --help
<commit_msg>Switch back to lowercase bucket names<commit_after>==============
dataflow-java |Build Status|_ |Build Coverage|_
==============

.. |Build Status| image:: http://img.shields.io/travis/googlegenomics/dataflow-java.svg?style=flat
.. _Build Status: https://travis-ci.org/googlegenomics/dataflow-java

.. |Build Coverage| image:: http://img.shields.io/coveralls/googlegenomics/dataflow-java.svg?style=flat
.. _Build Coverage: https://coveralls.io/r/googlegenomics/dataflow-java?branch=master

 If you are ready to start coding, take a look at the information below.  But if you are
 looking for a task-oriented list (e.g., `How do I compute principal coordinate analysis
 with Google Genomics? <http://googlegenomics.readthedocs.org/en/latest/use_cases/compute_principal_coordinate_analysis/index.html>`_),
 a better place to start is the `Google Genomics Cookbook <http://googlegenomics.readthedocs.org/en/latest/index.html>`_ .

Getting started
---------------

* First, follow the Google Cloud Dataflow `getting started instructions
  <https://cloud.google.com/dataflow/getting-started>`_ to set up your environment
  for Dataflow. You will need your Project ID and Google Cloud Storage bucket in the following steps.

* To use this code, build the client using `Apache Maven`_::

    cd dataflow-java
    mvn compile
    mvn bundle:bundle

* Then, follow the Google Genomics `sign up instructions`_ to generate a valid
  ``client_secrets.json`` file.

* Move the ``client_secrets.json`` file into the dataflow-java directory.
  (Authentication will take place the first time you run a pipeline.)

* Then you can run a pipeline locally with the command line, passing in the
  Project ID and Google Cloud Storage bucket you made in the first step.
  This command runs the VariantSimilarity pipeline (which runs PCoA on a dataset)::

    java -cp target/google-genomics-dataflow*.jar \
      com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity \
      --project=your-project-id \
      --output=gs://your-bucket/output/localtest.txt \
      --genomicsSecretsFile=client_secrets.json

  Note: when running locally, you may run into memory issues depending on the
  capacity of your local machine.

* To deploy your pipeline (which runs on Google Compute Engine), some additional
  command line arguments are required::

    java -cp target/google-genomics-dataflow*.jar \
      com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity \
      --runner=BlockingDataflowPipelineRunner \
      --project=your-project-id \
      --stagingLocation=gs://your-bucket/staging \
      --output=gs://your-bucket/output/test.txt \
      --genomicsSecretsFile=client_secrets.json \
      --numWorkers=4

  Note: By default, the max workers you can have without requesting more GCE quota
  is 16. (That's the default limit on VMs)

.. _Apache Maven: http://maven.apache.org/download.cgi
.. _sign up instructions: https://cloud.google.com/genomics/install-genomics-tools#authenticate

Command Line Options
--------------------

Use ``--help`` to get more information about the command line options.  Change
the pipeline class name below to match the one you would like to run::

  java -cp google-genomics-dataflow*.jar \
    com.google.cloud.genomics.dataflow.pipelines.VariantSimilarity --help

{'fraction_non_alphanumeric': 0.11160170523751523, 'fraction_numerical': 0.0009135200974421437, 'mean_word_length': 3.7293016558675305, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4167762', 'n_tokens_mistral': 2000, 'n_tokens_neox': 1849, 'n_words': 611}