<jupyter_start><jupyter_text>Efficient training for image classification _Transfer learning using Inception Package - Local Run Experience_Traditionally, image classification required a very large corpus of training data - often millions of images which may not be available and a long time to train on those images which is expensive and time consuming. That has changed with transfer learning which can be readily used with Cloud ML Engine and without deep knowledge of image classification algorithms using the ML toolbox in Datalab.This notebook codifies the capabilities discussed in this [blog post](https://cloud.google.com/blog/big-data/2016/12/how-to-train-and-classify-images-using-google-cloud-machine-learning-and-cloud-dataflow). In a nutshell, it uses the pre-trained inception model as a starting point and then uses transfer learning to train it further on additional, customer-specific images. For explanation, simple flower images are used. Compared to training from scratch, the training data requirements, time and costs are drastically reduced.This notebook does all operations in the Datalab container without calling CloudML API. Hence, this is called "local" operations - though Datalab itself is most often running on a GCE VM. See the corresponding cloud notebook for cloud experience which only adds the --cloud parameter and some config to the local experience commands. The purpose of local work is to do some initial prototyping and debugging on small scale data - often by taking a suitable (say 0.1 - 1%) sample of the full data. The same basic steps can then be repeated with much larger datasets in cloud.  SetupAll data is available under gs://cloud-datalab/sampledata/flower. eval100 is a subset of eval300, which is a subset of eval670. Same for train data.<jupyter_code>!mkdir -p /content/flowerdata
!gsutil -m cp gs://cloud-datalab/sampledata/flower/* /content/flowerdata<jupyter_output>Copying gs://cloud-datalab/sampledata/flower/all.csv...
Copying gs://cloud-datalab/sampledata/flower/eval100.csv...
Copying gs://cloud-datalab/sampledata/flower/eval300.csv...
Copying gs://cloud-datalab/sampledata/flower/eval670.csv...
Copying gs://cloud-datalab/sampledata/flower/train1000.csv...
Copying gs://cloud-datalab/sampledata/flower/train200.csv...
Copying gs://cloud-datalab/sampledata/flower/train300.csv...
Copying gs://cloud-datalab/sampledata/flower/train3000.csv...<jupyter_text>Define directories for preprocessing, model, and prediction.<jupyter_code>import mltoolbox.image.classification as model
from google.datalab.ml import *

worker_dir = '/content/datalab/tmp/flower'
preprocessed_dir = worker_dir + '/flowerrunlocal'
model_dir = worker_dir + '/tinyflowermodellocal'
prediction_dir = worker_dir + '/flowermodelevallocal'
images_dir = worker_dir + '/images'
local_train_file = '/content/flowerdata/train200local.csv'
local_eval_file = '/content/flowerdata/eval100local.csv'
!mkdir -p {images_dir}<jupyter_output><empty_output><jupyter_text>In order to get best efficiency, we download the images to local disk, and create our training and evaluation files to reference local path instead of GCS path. Note that the original training files referencing GCS image paths work too, although a bit slower.<jupyter_code>import csv
import datalab.storage as gcs
import os


def download_images(input_csv, output_csv, images_dir):
  with open(input_csv) as csvfile:
    data = list(csv.DictReader(csvfile, fieldnames=['image_url', 'label']))
  for x in data:
    url = x['image_url']
    out_file = os.path.join(images_dir, os.path.basename(url))
    with open(out_file, 'w') as f:
      f.write(gcs.Item.from_url(url).read_from())
    x['image_url'] = out_file

  with open(output_csv, 'w') as w:
    csv.DictWriter(w, fieldnames=['image_url', 'label']).writerows(data)


download_images('/content/flowerdata/train200.csv', local_train_file, images_dir)    
download_images('/content/flowerdata/eval100.csv', local_eval_file, images_dir)<jupyter_output><empty_output><jupyter_text>The above code can best be illustrated by the comparison below.<jupyter_code>!head /content/flowerdata/train200.csv -n 5
!head {local_train_file} -n 5<jupyter_output>/content/datalab/tmp/flower/images/754296579_30a9ae018c_n.jpg,daisy

/content/datalab/tmp/flower/images/18089878729_907ed2c7cd_m.jpg,dandelion

/content/datalab/tmp/flower/images/284497199_93a01f48f6.jpg,dandelion

/content/datalab/tmp/flower/images/3554992110_81d8c9b0bd_m.jpg,dandelion

/content/datalab/tmp/flower/images/4065883015_4bb6010cb7_n.jpg,daisy<jupyter_text>PreprocessPreprocessing uses a Dataflow pipeline to convert the image format, resize images, and run the converted image through a pre-trained model to get the features or embeddings. You can also do this step using alternate technologies like Spark or plain Python code if you like.  The following cell takes ~5 min on a n1-standard-1 VM. Preprocessing the full 3000 images takes about one hour.<jupyter_code># instead of local_train_file, it can take '/content/flowerdata/train200.csv' too, but processing will be slower.
train_set = CsvDataSet(local_train_file, schema='image_url:STRING,label:STRING')
model.preprocess(train_set, preprocessed_dir)<jupyter_output>/usr/local/lib/python2.7/dist-packages/apache_beam/coders/typecoders.py:136: UserWarning: Using fallback coder for typehint: Any.
  warnings.warn('Using fallback coder for typehint: %r.' % typehint)
WARNING:root:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.<jupyter_text>TrainThe next step is to train the inception model with the preprocessed images using transfer learning. Transfer learning retains most of the inception model but replaces the final layer as shown in the image.![inception](https://cloud.google.com/blog/big-data/2016/12/images/148114735559140/image-classification-3.png)<jupyter_code>import logging
logging.getLogger().setLevel(logging.INFO)
model.train(preprocessed_dir, 30, 800, model_dir)
logging.getLogger().setLevel(logging.WARNING)<jupyter_output>INFO:tensorflow:global_step/sec: 0<jupyter_text>Run TensorBoard to visualize the completed training. Review accuracy and loss in particular.<jupyter_code>tb_id = TensorBoard.start(model_dir)<jupyter_output><empty_output><jupyter_text>We can check the TF summary events from training.<jupyter_code>summary = Summary(model_dir)
summary.list_events()
summary.plot('accuracy')
summary.plot('loss')<jupyter_output><empty_output><jupyter_text>PredictLet's start with a quick check by taking a couple of images and using the model to predict the type of flower locally.<jupyter_code>images = [
  'gs://cloud-ml-data/img/flower_photos/daisy/15207766_fc2f1d692c_n.jpg',
  'gs://cloud-ml-data/img/flower_photos/tulips/6876631336_54bf150990.jpg'
]
# set show_image to False to not display pictures.
model.predict(model_dir, images, show_image=True)<jupyter_output>Predicting...<jupyter_text>EvaluateWe did a quick test of the model using a few samples. But we need to understand how the model does by evaluating it against much larger amount of labeled data. In the initial preprocessing step, we did set aside enough images for that purpose. Next, we will use normal batch prediction and compare the results with the previously labeled targets. The following batch prediction and loading of results takes ~3 minutes.<jupyter_code>import google.datalab.bigquery as bq
bq.Dataset('flower').create()
eval_set = CsvDataSet(local_eval_file, schema='image_url:STRING,label:STRING')
model.batch_predict(eval_set, model_dir, output_bq_table='flower.eval_results_local')<jupyter_output>completed<jupyter_text>Now that we have the results and expected results loaded in a BigQuery table, let's start analyzing the errors and plot the confusion matrix.<jupyter_code>%%bq query --name wrong_prediction

SELECT * FROM flower.eval_results_local where target != predicted
wrong_prediction.execute().result()<jupyter_output><empty_output><jupyter_text>Confusion matrix is a common way of comparing the confusion of the model - aggregate data about where the actual result did not match the expected result.<jupyter_code>ConfusionMatrix.from_bigquery('flower.eval_results_local').plot()<jupyter_output><empty_output><jupyter_text>More advanced analysis can be done using the feature slice view. For the feature slice view, let's define SQL queries that compute accuracy and log loss and then use the metrics.<jupyter_code>%%bq query --name accuracy

SELECT
  target,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END) as correct,
  COUNT(*) as total,
  SUM(CASE WHEN target=predicted THEN 1 ELSE 0 END)/COUNT(*) as accuracy
FROM
  flower.eval_results_local
GROUP BY
  target
accuracy.execute().result()
%%bq query --name logloss

SELECT feature, AVG(-logloss) as logloss, count(*) as count FROM
(
SELECT feature, CASE WHEN correct=1 THEN LOG(prob) ELSE LOG(1-prob) END as logloss
FROM
(
SELECT
target as feature, 
CASE WHEN target=predicted THEN 1 ELSE 0 END as correct,
target_prob as prob
FROM flower.eval_results_local))
GROUP BY feature
FeatureSliceView().plot(logloss)<jupyter_output><empty_output><jupyter_text>Clean up<jupyter_code>import shutil
import google.datalab.bigquery as bq

TensorBoard.stop(tb_id)
bq.Table('flower.eval_results_local').delete()
shutil.rmtree(worker_dir)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08642239077021685, 'fraction_numerical': 0.02360858882598013, 'mean_word_length': 5.858608058608058, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 51, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '251444', 'license': 'Apache-2.0', 'n_tokens_mistral': 3019, 'n_tokens_neox': 2784, 'n_words': 892, 'path': 'docs/samples/ML Toolbox/Image Classification/Flower/Local End to End.ipynb', 'repo_name': 'JeffersonH44/datalab-r', 'size': '9361'}