<jupyter_start><jupyter_text>Environment is available as a publically available docker container: `hamelsmu/ml-gpu` Pre-requisite: Familiarize yourself with sequence-to-sequence modelsIf you are not familiar with sequence to sequence models, please refer to [this tutorial](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8). Pre-Requisite: Make Sure you have the right files prepared from Step 1You should have these files in the root of the `./data/processed_data/` directory:1. `{train/valid/test.function}` - these are python function definitions tokenized (by space), 1 line per function.2. `{train/valid/test.docstring}` - these are docstrings that correspond to each of the python function definitions, and have a 1:1 correspondence with the lines in *.function files.3. `{train/valid/test.lineage}` - every line in this file contains a link back to the original location (github repo link) where the code was retrieved.  There is a 1:1 correspondence with the lines in this file and the other two files. This is useful for debugging. Set the value of `use_cache` appropriately.  If `use_cache = True`, data will be downloaded where possible instead of re-computing.  However, it is highly recommended that you set `use_cache = False` for this tutorial as it will be less confusing, and you will learn more by runing these steps yourself. **This notebook takes approximately 4 hours to run on an AWS `p3.8xlarge` instance.**<jupyter_code>use_cache = False
# # Optional: you can set what GPU you want to use in a notebook like this.  
# # Useful if you want to run concurrent experiments at the same time on different GPUs.
# import os
# os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
# os.environ["CUDA_VISIBLE_DEVICES"]="3"
# This will allow the notebook to run faster
from pathlib import Path
from general_utils import get_step2_prerequisite_files, read_training_files
from keras.utils import get_file
OUTPUT_PATH = Path('./data/seq2seq/')
OUTPUT_PATH.mkdir(exist_ok=True)<jupyter_output>/opt/conda/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.<jupyter_text>Read Text From FileWe want to read in raw text from files so we can pre-process the text for modeling as described in [this tutorial](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8)<jupyter_code>if use_cache:
    get_step2_prerequisite_files(output_directory = './data/processed_data')

# you want to supply the directory where the files are from step 1.
train_code, holdout_code, train_comment, holdout_comment = read_training_files('./data/processed_data/')
# code and comment files should be of the same length.

assert len(train_code) == len(train_comment)
assert len(holdout_code) == len(holdout_comment)<jupyter_output><empty_output><jupyter_text>Tokenize TextIn this step, we are going to pre-process the raw text for modeling.  For an explanation of what this section does, see the [Preapre & Clean Data section of this Tutorial](https://towardsdatascience.com/how-to-create-data-products-that-are-magical-using-sequence-to-sequence-models-703f86a231f8)<jupyter_code>from ktext.preprocess import processor

if not use_cache:    
    code_proc = processor(heuristic_pct_padding=.7, keep_n=20000)
    t_code = code_proc.fit_transform(train_code)

    comment_proc = processor(append_indicators=True, heuristic_pct_padding=.7, keep_n=14000, padding ='post')
    t_comment = comment_proc.fit_transform(train_comment)

elif use_cache:
    logging.warning('Not fitting transform function because use_cache=True')<jupyter_output>WARNING:root:....tokenizing data
WARNING:root:Setting maximum document length to 55 based upon hueristic of 0.7 percentile.
 See full histogram by insepecting the `document_length_stats` attribute.
WARNING:root:(1/2) done. 111 sec
WARNING:root:....building corpus
WARNING:root:(2/2) done. 35 sec
WARNING:root:Finished parsing 1,227,989 documents.
WARNING:root:...fit is finished, beginning transform
WARNING:root:...padding data
WARNING:root:done. 33 sec
WARNING:root:....tokenizing data
WARNING:root:Setting maximum document length to 15 based upon hueristic of 0.7 percentile.
 See full histogram by insepecting the `document_length_stats` attribute.
WARNING:root:(1/2) done. 38 sec
WARNING:root:....building corpus
WARNING:root:(2/2) done. 11 sec
WARNING:root:Finished parsing 1,227,989 documents.
WARNING:root:...fit is finished, beginning transform
WARNING:root:...padding data
WARNING:root:done. 13 sec<jupyter_text>**Save tokenized text** (You will reuse this for step 4)<jupyter_code>import dill as dpickle
import numpy as np

if not use_cache:
    # Save the preprocessor
    with open(OUTPUT_PATH/'py_code_proc_v2.dpkl', 'wb') as f:
        dpickle.dump(code_proc, f)

    with open(OUTPUT_PATH/'py_comment_proc_v2.dpkl', 'wb') as f:
        dpickle.dump(comment_proc, f)

    # Save the processed data
    np.save(OUTPUT_PATH/'py_t_code_vecs_v2.npy', t_code)
    np.save(OUTPUT_PATH/'py_t_comment_vecs_v2.npy', t_comment)<jupyter_output><empty_output><jupyter_text>Arrange data for modeling<jupyter_code>%reload_ext autoreload
%autoreload 2
from seq2seq_utils import load_decoder_inputs, load_encoder_inputs, load_text_processor


encoder_input_data, encoder_seq_len = load_encoder_inputs(OUTPUT_PATH/'py_t_code_vecs_v2.npy')
decoder_input_data, decoder_target_data = load_decoder_inputs(OUTPUT_PATH/'py_t_comment_vecs_v2.npy')
num_encoder_tokens, enc_pp = load_text_processor(OUTPUT_PATH/'py_code_proc_v2.dpkl')
num_decoder_tokens, dec_pp = load_text_processor(OUTPUT_PATH/'py_comment_proc_v2.dpkl')<jupyter_output>Shape of encoder input: (1227989, 55)
Shape of decoder input: (1227989, 14)
Shape of decoder target: (1227989, 14)
Size of vocabulary for data/seq2seq/py_code_proc_v2.dpkl: 20,002
Size of vocabulary for data/seq2seq/py_comment_proc_v2.dpkl: 14,002<jupyter_text>If you don't have the above files on disk because you set `use_cache = True` you can download the files for the above function calls here: - https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_t_code_vecs_v2.npy - https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_t_comment_vecs_v2.npy - https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_code_proc_v2.dpkl - https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_comment_proc_v2.dpkl  Build Seq2Seq Model For Summarizing CodeWe will build a model to predict the docstring given a function or a method.  While this is a very cool task in itself, this is not the end goal of this exercise.  The motivation for training this model is to learn a general purpose feature extractor for code that we can use for the task of code search.<jupyter_code>from seq2seq_utils import build_seq2seq_model<jupyter_output><empty_output><jupyter_text>The convenience function `build_seq2seq_model` constructs the architecture for a sequence-to-sequence model.  The architecture built for this tutorial is a minimal example with only one layer for the encoder and decoder, and does not include things like [attention](https://nlp.stanford.edu/pubs/emnlp15_attn.pdf).  We encourage you to try and build different architectures to see what works best for you!<jupyter_code>seq2seq_Model = build_seq2seq_model(word_emb_dim=800,
                                    hidden_state_dim=1000,
                                    encoder_seq_len=encoder_seq_len,
                                    num_encoder_tokens=num_encoder_tokens,
                                    num_decoder_tokens=num_decoder_tokens)
seq2seq_Model.summary()<jupyter_output>__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Decoder-Input (InputLayer)      (None, None)         0                                            
__________________________________________________________________________________________________
Decoder-Word-Embedding (Embeddi (None, None, 800)    11201600    Decoder-Input[0][0]              
__________________________________________________________________________________________________
Encoder-Input (InputLayer)      (None, 55)           0                                            
__________________________________________________________________________________________________
Decoder-Batchnorm-1 (BatchNorma (None, None, 800)    3200        Decoder-Word-Embedding[0][0]     
__________[...]<jupyter_text>Train Seq2Seq Model<jupyter_code>from keras.models import Model, load_model
import pandas as pd
import logging

if not use_cache:

    from keras.callbacks import CSVLogger, ModelCheckpoint
    import numpy as np
    from keras import optimizers

    seq2seq_Model.compile(optimizer=optimizers.Nadam(lr=0.00005), loss='sparse_categorical_crossentropy')

    script_name_base = 'py_func_sum_v9_'
    csv_logger = CSVLogger('{:}.log'.format(script_name_base))

    model_checkpoint = ModelCheckpoint('{:}.epoch{{epoch:02d}}-val{{val_loss:.5f}}.hdf5'.format(script_name_base),
                                       save_best_only=True)

    batch_size = 1100
    epochs = 16
    history = seq2seq_Model.fit([encoder_input_data, decoder_input_data], np.expand_dims(decoder_target_data, -1),
              batch_size=batch_size,
              epochs=epochs,
              validation_split=0.12, callbacks=[csv_logger, model_checkpoint])
if use_cache:
    logging.warning('Not re-training function summarizer seq2seq model because use_cache=True')
    # Load model from url
    loc = get_file(fname='py_func_sum_v9_.epoch16-val2.55276.hdf5',
                   origin='https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_func_sum_v9_.epoch16-val2.55276.hdf5')
    seq2seq_Model = load_model(loc)
    
    # Load encoder (code) pre-processor from url
    loc = get_file(fname='py_code_proc_v2.dpkl',
                   origin='https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_code_proc_v2.dpkl')
    num_encoder_tokens, enc_pp = load_text_processor(loc)
    
    # Load decoder (docstrings/comments) pre-processor from url
    loc = get_file(fname='py_comment_proc_v2.dpkl',
                   origin='https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/py_comment_proc_v2.dpkl')
    num_decoder_tokens, dec_pp = load_text_processor(loc)<jupyter_output><empty_output><jupyter_text>Note that the above procedure will automatically download a pre-trained model and associated artifacts from https://storage.googleapis.com/kubeflow-examples/code_search/data/seq2seq/ if `use_cache = True`.  Otherwise, the above code will checkpoint the best model after each epoch into the current directory with prefix `py_func_sum_v9_`  Evaluate Seq2Seq Model For Code SummarizationTo evaluate this model we are going to do two things:1.  Manually inspect the results of predicted docstrings for code snippets, to make sure they look sensible.2.  Calculate the [BLEU Score](https://en.wikipedia.org/wiki/BLEU) so that we can quantitately benchmark different iterations of this algorithm and to guide hyper-parameter tuning.  Manually Inspect Results (on holdout set)<jupyter_code>from seq2seq_utils import Seq2Seq_Inference
import pandas as pd

seq2seq_inf = Seq2Seq_Inference(encoder_preprocessor=enc_pp,
                                 decoder_preprocessor=dec_pp,
                                 seq2seq_model=seq2seq_Model)

demo_testdf = pd.DataFrame({'code':holdout_code, 'comment':holdout_comment, 'ref':''})
seq2seq_inf.demo_model_predictions(n=15, df=demo_testdf)<jupyter_output>==============================================
============== Example # 44697 =================

Original Input:
 def testInitialization self event_formatter task_scheduler TaskCacheEventFormatter self assertIsNotNone event_formatter
 

Original Output:
 tests the initialization .


****** Predicted Output ******:
 tests the initialization of the event


==============================================
============== Example # 151630 =================

Original Input:
 def reduceChat self cmd text attrs text cmd script_parser parse_text text val cmd script_parser _ Chat message channel get_attr cmd chat attrs channel cls get_attr cmd chat attrs cls validate_attrs cmd chat attrs text channel cls args if channel is not None args channel channel if cls is not None args cls cls self val chat text args
 

Original Output:
 % reduce chat scalar exprattrs


****** Predicted Output ******:
 parse command line arguments and return the result


==============================================
====[...]<jupyter_text>Comment on manual inspection of results:The predicted code summaries are not perfect, but we can see that the model has learned to extract some semantic meaning from the code.  That's all we need to get reasonable results in this case.    Calculate BLEU Score (on holdout set)BLEU Score is described [in this wikipedia article](https://en.wikipedia.org/wiki/BLEU), and is a way to measure the efficacy of summarization/translation such as the one we conducted here.  This metric is useful if you wish to conduct extensive hyper-parameter tuning and try to improve the seq2seq model.<jupyter_code># This will return a BLEU Score
seq2seq_inf.evaluate_model(input_strings=holdout_code, 
                           output_strings=holdout_comment, 
                           max_len=None)<jupyter_output>WARNING:root:Generating predictions.<jupyter_text>Save model to diskSave the model to disk so you can use it in Step 4 of this tutorial.<jupyter_code>seq2seq_Model.save(OUTPUT_PATH/'code_summary_seq2seq_model.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09740122543841116, 'fraction_numerical': 0.02063525600394394, 'mean_word_length': 4.68, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 14, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '83591', 'license': 'MIT', 'n_tokens_mistral': 4313, 'n_tokens_neox': 4031, 'n_words': 1201, 'path': 'colab/2_Train_Function_Summarizer_With_Keras_+_TF.ipynb', 'repo_name': 'yzhen-li/CAMEL-React-App', 'size': '14199'}