<jupyter_start><jupyter_text>Get Started with Jupyter on Google CloudIn the following you find various helpful code examples which show you how to access data or start ML routines on Google Cloud resources like CPUs, GPUs, or TPUs.Our first task is to import all necessary libraries used in the examples below.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Access Data on Google Cloud Storage Cloud Storage is a storage service in the Google Cloud. It can store virtually infinite amounts of data. Typically, Cloud Storage is used to store files with unstructured data, such as images, text files, and semi-structured file formats, such as CSV, Avro, Parquet, and TFRecords.We start by creating a Cloud Storage client in Python. The client allows us to interact with the Cloud Storage service. With the client we can download and upload files.<jupyter_code>from google.cloud import storage
client = storage.Client()
print("Client created using default project: {}".format(client.project))<jupyter_output><empty_output><jupyter_text>To explicitly specify a project when constructing the client, set the `project` parameter:<jupyter_code># client = storage.Client(project='your-project-id')<jupyter_output><empty_output><jupyter_text>First, we work with a bucket which is a root folder in Cloud Storage. Buckets can contain many files and have (practically) no size limit. Here is how we access our bucket for the hackathon:<jupyter_code>bucket_name = "ecb-fsf-hackathon-base-data"
bucket = client.get_bucket(bucket_name)

print("Bucket name: {}".format(bucket.name))
print("Bucket location: {}".format(bucket.location))
print("Bucket storage class: {}".format(bucket.storage_class))<jupyter_output><empty_output><jupyter_text>Let's list all files in the bucket:<jupyter_code>blobs = bucket.list_blobs()

print("Blobs in {}:".format(bucket.name))
for item in blobs:
    print("\t" + item.name)<jupyter_output><empty_output><jupyter_text>We can also use the gsutil command line tool for a list:<jupyter_code>!gsutil ls gs://{bucket_name}<jupyter_output><empty_output><jupyter_text>Now we can get details about one of the files, download it, and load into a dataframe:<jupyter_code>blob_name = "sample.csv"
blob = bucket.get_blob(blob_name)

print("Name: {}".format(blob.id))
print("Size: {} bytes".format(blob.size))
print("Content type: {}".format(blob.content_type))
print("Public URL: {}".format(blob.public_url))

output_file_name = "/tmp/sample.csv"
blob.download_to_filename(output_file_name)

print("Downloaded blob {} to {}.".format(blob.name, output_file_name))<jupyter_output><empty_output><jupyter_text>Again, the same can be achieved using the gsutil command line tool:<jupyter_code>!gsutil cp gs://{bucket_name}/{blob_name} /tmp/{blob_name}<jupyter_output><empty_output><jupyter_text>With the file stored locally, we can load it into a Pandas dataframe:<jupyter_code>df = pd.read_csv(output_file_name, header=None)
df.describe()<jupyter_output><empty_output><jupyter_text>And we should have a look into the dataframe:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Let's use Panda's built-in support for Google Cloud Storage:<jupyter_code>df = pd.read_csv('gs://ecb-fsf-hackathon-base-data/sample.csv', dtype=str)
df.describe()<jupyter_output><empty_output><jupyter_text>And .head() should return the same lines as with our manual download:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>**Learn more about interacting with Cloud Storage in the following tutorials:**- [Cloud Storage client library](../tutorials/storage/Cloud%20Storage%20client%20library.ipynb)- [Storage command-line tool](../tutorials/storage/Storage%20command-line%20tool.ipynb)  Access Tables & Views on Google BigQuery<jupyter_code>from google.cloud import bigquery
client = bigquery.Client(location="EU")
print("Client creating using default project: {}".format(client.project))<jupyter_output><empty_output><jupyter_text>To explicitly specify a project when constructing the client, set the `project` parameter:<jupyter_code># client = bigquery.Client(location="US", project="your-project-id")
query = """
    SELECT `Set`, URL, Label
    FROM `ecb-fsf-hackathon-base.hackathon_dataset.data_table`
    LIMIT 60
"""
query_job = client.query(query, location="EU")
df = query_job.to_dataframe()
df.describe()
df.head()
query = """
    SELECT category, url, product_name
    FROM `ecb-fsf-hackathon-base.hackathon_dataset.food_unique_products_classified`
    LIMIT 60
"""
query_job = client.query(query, location="EU")
df = query_job.to_dataframe()
df.describe()
df.head()<jupyter_output><empty_output><jupyter_text>You can also execute a query using the BigQuery magic expression in a cell:<jupyter_code>%%bigquery --verbose df
SELECT category, Count(*) as Occurence
FROM `ecb-fsf-hackathon-base.hackathon_dataset.food_unique_products_classified`
GROUP BY category
ORDER BY Occurence DESC
LIMIT 10
df.head()<jupyter_output><empty_output><jupyter_text>**Learn more about interacting with BigQuery in the following tutorials:**- [BigQuery basics](../tutorials/bigquery/BigQuery%20basics.ipynb)- [BigQuery command-line tool](../tutorials/bigquery/BigQuery%20command-line%20tool.ipynb)- [BigQuery query magic](../tutorials/bigquery/BigQuery%20query%20magic.ipynb)  Cloud AI APIs and Cloud AutoML Some useful resources to get started with our Cloud APIs for NLP and [AutoML](https://cloud.google.com/automl/) for NLP:* [Cloud NLP Intro](https://cloud.google.com/natural-language/)* [Cloud Natural Language API Docs](https://cloud.google.com/natural-language/docs/)* [Cloud AutoML Get Started Guides](https://cloud.google.com/natural-language/overview/docs/get-started)* [Cloud AutoML NLP in the Console](https://console.cloud.google.com/natural-language)There is also [Cloud AutoML Tables](https://cloud.google.com/automl-tables/) to build ML models on tabular data (e.g. from BigQuery):* [Cloud AutoML Tables Intro](https://cloud.google.com/automl-tables/)* [Cloud AutoML Tables Docs](https://cloud.google.com/automl-tables/docs/)* [Cloud AutoML Tables in the Console](https://console.cloud.google.com/automl-tables)  Data Transformation with Apache Beam (and Cloud Dataflow)<jupyter_code>!pip3 install apache-beam[gcp]
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

pipeline_options = PipelineOptions.from_dictionary({
    'runner': 'DirectRunner',
# Run it massively parallel on Dataflow with
#   'runner': 'DataflowRunner'
    'job_name': 'notebook',
    'streaming': True
})

def collect(i):
    output.append(i)
    return True

output = []

p = beam.Pipeline(options=pipeline_options)

pipeline = (
    p 
    | 'generate' >> beam.Create(range(1000))
    | 'square' >> beam.Map(lambda x: x**2)
    | "print" >> beam.Map(collect)
)

result = p.run()
result.wait_until_finish()

output[:10]<jupyter_output><empty_output><jupyter_text>Train Models with Google Cloud AI Platform Training We want to enable the ML and Container Registry APIs in our project.<jupyter_code>!gcloud services enable ml.googleapis.com
!gcloud services enable containerregistry.googleapis.com<jupyter_output><empty_output><jupyter_text>Then, we need to create a bucket for the staging and training results. Replace with your favorite name (needs to be globally unique!):<jupyter_code>!gsutil mb gs://[YOUR_GCS_BUCKET]<jupyter_output><empty_output><jupyter_text>Ready to start our Training Job! Fill in in your bucket name where you find brackets. You can modify the model_dir parameter to change where the training output is stored.<jupyter_code>gcloud ml-engine jobs submit training $JOB_NAME \
    --staging-bucket [YOUR_GCS_BUCKET] \
    --runtime-version 1.8 \
    --scale-tier BASIC_TPU \
    --module-name resnet.resnet_main \
    --package-path resnet/ \
    --region us-central1 \
    -- \
    --data_dir=gs://cloud-tpu-test-datasets/fake_imagenet \
    --model_dir=gs://[YOUR_GCS_BUCKET]/training_result/ \
    --resnet_depth=50 \
    --train_steps=1024<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11572810716943686, 'fraction_numerical': 0.0053336641032001985, 'mean_word_length': 5.980952380952381, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 83, 'https://': 9, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '530659', 'license': 'Apache-2.0', 'n_tokens_mistral': 2629, 'n_tokens_neox': 2534, 'n_words': 685, 'path': 'get-started.ipynb', 'repo_name': 'mugglmenzel/ecb-fsf-hackathon', 'size': '8062'}