<commit_before># TensorFlow Cloud

## What is this repo?

The TensorFlow Cloud repository provides APIs that will allow to easily go from debugging and training your Keras and TensorFlow code in a local environment to distributed training in the cloud.

## Installation

### Requirements

- Python >= 3.5
- [A Google Cloud project](https://cloud.google.com/ai-platform/docs/getting-started-keras#set_up_your_project)
- An [authenticated GCP account](https://cloud.google.com/ai-platform/docs/getting-started-keras#authenticate_your_gcp_account)
- [Google AI platform](https://cloud.google.com/ai-platform/) APIs enabled for your GCP account. We use the AI platform for deploying docker images on GCP.
- Either a functioning version of [docker](https://docs.docker.com/engine/install/) if you want to use a local docker process for your build, or [create a cloud storage bucket](https://cloud.google.com/ai-platform/docs/getting-started-keras#create_a_bucket) to use with [Google Cloud build](https://cloud.google.com/cloud-build) for docker image build and publishing.
- [Authenticate to your Docker Container Registry](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper)
- (optional) [nbconvert](https://nbconvert.readthedocs.io/en/latest/) if you are using a notebook file as `entry_point` as shown in [usage guide #4](#usage-guide).

For detailed end to end setup instructions, please see [Setup instructions](#setup-instructions).

### Install latest release

```shell
pip install -U tensorflow-cloud
```

### Install from source

```shell
git clone https://github.com/tensorflow/cloud.git
cd cloud
pip install src/python/.
```

## High level overview

TensorFlow Cloud package provides the `run` API for training your models on GCP. To start, let's walk through a simple workflow using this API.

1. Let's begin with a Keras model training code such as the following, saved as `mnist_example.py`.

```python
import tensorflow as tf

(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

x_train = x_train.reshape((60000, 28 * 28))
x_train = x_train.astype('float32') / 255

model = tf.keras.Sequential([
  tf.keras.layers.Dense(512, activation='relu', input_shape=(28 * 28,)),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=10, batch_size=128)
```

2. After you have tested this model on your local environment for a few epochs, probably with a small dataset, you can train the model on Google Cloud by writing the following simple script `scale_mnist.py`.

```python
import tensorflow_cloud as tfc
tfc.run(entry_point='mnist_example.py')
```

Running `scale_mnist.py` will automatically apply TensorFlow [one device strategy](https://www.tensorflow.org/api_docs/python/tf/distribute/OneDeviceStrategy) and train your model at scale on Google Cloud Platform. Please see the [usage guide](#usage-guide) section for detailed instructions and additional API parameters.

3. You will see an output similar to the following on your console. This information can be used to track the training job status. 

```shell
user@desktop$ python scale_mnist.py
Job submitted successfully.
Your job ID is:  tf_cloud_train_519ec89c_a876_49a9_b578_4fe300f8865e
Please access your job logs at the following URL:
https://console.cloud.google.com/mlengine/jobs/tf_cloud_train_519ec89c_a876_49a9_b578_4fe300f8865e?project=prod-123
```

## Setup instructions

End to end instructions to help set up your environment for Tensorflow Cloud.

1. Create a new local directory

```shell
mkdir tensorflow_cloud
cd tensorflow_cloud
```

2. Make sure you have `python >= 3.5`

```shell
python -V
```

3. Set up virtual environment

```shell
virtualenv tfcloud --python=python3
source venv/bin/activate
```

4. [Set up your Google Cloud project](https://cloud.google.com/ai-platform/docs/getting-started-keras#set_up_your_project)

Verify that gcloud sdk is installed.

```shell
which gcloud
```

Set default gcloud project

```shell
export PROJECT_ID=<your-project-id>
gcloud config set project $PROJECT_ID
```

5. [Authenticate your GCP account](https://cloud.google.com/ai-platform/docs/getting-started-keras#authenticate_your_gcp_account)

Create a service account.

```shell
export SA_NAME=<your-sa-name>
gcloud iam service-accounts create $SA_NAME
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
    --role 'roles/editor'
```
<commit_msg>Update virtual environment creation demo<commit_after># TensorFlow Cloud

## What is this repo?

The TensorFlow Cloud repository provides APIs that will allow to easily go from debugging and training your Keras and TensorFlow code in a local environment to distributed training in the cloud.

## Installation

### Requirements

- Python >= 3.5
- [A Google Cloud project](https://cloud.google.com/ai-platform/docs/getting-started-keras#set_up_your_project)
- An [authenticated GCP account](https://cloud.google.com/ai-platform/docs/getting-started-keras#authenticate_your_gcp_account)
- [Google AI platform](https://cloud.google.com/ai-platform/) APIs enabled for your GCP account. We use the AI platform for deploying docker images on GCP.
- Either a functioning version of [docker](https://docs.docker.com/engine/install/) if you want to use a local docker process for your build, or [create a cloud storage bucket](https://cloud.google.com/ai-platform/docs/getting-started-keras#create_a_bucket) to use with [Google Cloud build](https://cloud.google.com/cloud-build) for docker image build and publishing.
- [Authenticate to your Docker Container Registry](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper)
- (optional) [nbconvert](https://nbconvert.readthedocs.io/en/latest/) if you are using a notebook file as `entry_point` as shown in [usage guide #4](#usage-guide).

For detailed end to end setup instructions, please see [Setup instructions](#setup-instructions).

### Install latest release

```shell
pip install -U tensorflow-cloud
```

### Install from source

```shell
git clone https://github.com/tensorflow/cloud.git
cd cloud
pip install src/python/.
```

## High level overview

TensorFlow Cloud package provides the `run` API for training your models on GCP. To start, let's walk through a simple workflow using this API.

1. Let's begin with a Keras model training code such as the following, saved as `mnist_example.py`.

```python
import tensorflow as tf

(x_train, y_train), (_, _) = tf.keras.datasets.mnist.load_data()

x_train = x_train.reshape((60000, 28 * 28))
x_train = x_train.astype('float32') / 255

model = tf.keras.Sequential([
  tf.keras.layers.Dense(512, activation='relu', input_shape=(28 * 28,)),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer=tf.keras.optimizers.Adam(),
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=10, batch_size=128)
```

2. After you have tested this model on your local environment for a few epochs, probably with a small dataset, you can train the model on Google Cloud by writing the following simple script `scale_mnist.py`.

```python
import tensorflow_cloud as tfc
tfc.run(entry_point='mnist_example.py')
```

Running `scale_mnist.py` will automatically apply TensorFlow [one device strategy](https://www.tensorflow.org/api_docs/python/tf/distribute/OneDeviceStrategy) and train your model at scale on Google Cloud Platform. Please see the [usage guide](#usage-guide) section for detailed instructions and additional API parameters.

3. You will see an output similar to the following on your console. This information can be used to track the training job status. 

```shell
user@desktop$ python scale_mnist.py
Job submitted successfully.
Your job ID is:  tf_cloud_train_519ec89c_a876_49a9_b578_4fe300f8865e
Please access your job logs at the following URL:
https://console.cloud.google.com/mlengine/jobs/tf_cloud_train_519ec89c_a876_49a9_b578_4fe300f8865e?project=prod-123
```

## Setup instructions

End to end instructions to help set up your environment for Tensorflow Cloud.

1. Create a new local directory

```shell
mkdir tensorflow_cloud
cd tensorflow_cloud
```

2. Make sure you have `python >= 3.5`

```shell
python -V
```

3. Set up virtual environment

```shell
virtualenv tfcloud --python=python3
source tfcloud/bin/activate
```

4. [Set up your Google Cloud project](https://cloud.google.com/ai-platform/docs/getting-started-keras#set_up_your_project)

Verify that gcloud sdk is installed.

```shell
which gcloud
```

Set default gcloud project

```shell
export PROJECT_ID=<your-project-id>
gcloud config set project $PROJECT_ID
```

5. [Authenticate your GCP account](https://cloud.google.com/ai-platform/docs/getting-started-keras#authenticate_your_gcp_account)

Create a service account.

```shell
export SA_NAME=<your-sa-name>
gcloud iam service-accounts create $SA_NAME
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member serviceAccount:$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com \
    --role 'roles/editor'
```

{'fraction_non_alphanumeric': 0.10650128314798973, 'fraction_numerical': 0.019461077844311378, 'mean_word_length': 4.953532781667728, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 11, 'https://': 26, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1807632', 'n_tokens_mistral': 3023, 'n_tokens_neox': 2794, 'n_words': 910}