<jupyter_start><jupyter_text>Split Data and Tokenize  Load necessary modules and data:<jupyter_code>from IPython.display import display
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.model_selection import StratifiedKFold
from tqdm import tqdm_notebook
from capstone_project import preprocessor as pre
from capstone_project import utility

tqdm_notebook().pandas(desc="Progress:")
data = pd.read_csv("../data/questions.csv")
display(data.head())

print "Rows in data: {}".format(len(data))
print "Ratio of duplicates: {:.2f}".format(float(len(data[data.is_duplicate == 1]))/len(data))<jupyter_output><empty_output><jupyter_text>First we are going to split the data into a test set and a set that will be used for training and validation. The test size is 10% of the total size of the dataset. I chose relatively small set, since the test set will only be used once at the very end in order to report the final performance of the chosen estimator.<jupyter_code>labels = data.iloc[:, -1].values
train_val_data, test_data = train_test_split(data, test_size=0.1, stratify=labels, random_state=42)
display(train_val_data.head(1))
display(test_data.head(1))

print "Rows in test set: {} and train_val set: {}".format(len(test_data), len(train_val_data))
print "Ratio: {:.2f}".format(float(len(test_data))/len(train_val_data))
print "Ratio of duplicates test: {:.2f}".format(float(len(test_data[test_data.is_duplicate == 1]))/len(test_data))
print "Ratio of duplicates train: {:.2f}".format(float(len(train_val_data[train_val_data.is_duplicate == 1]))/len(train_val_data))<jupyter_output><empty_output><jupyter_text>Next we are going to split the data into a training and validation set. The validation set will be used to compare the performance of different classifiers.<jupyter_code>#split one more time to create validation set because neural nets train to long to use cross validation
labels = train_val_data.iloc[:, -1].values
train_data, val_data = train_test_split(train_val_data, test_size=0.2, stratify=labels, random_state=75293)
print "Rows in train set: {} and val set: {}".format(len(train_data), len(val_data))<jupyter_output>Rows in train set: 291132 and val set: 72783<jupyter_text>Now it is time to tokenize the data. It was initially planned to use the tokenizer within the analysis pipeline. However, the tokenizer takes a little while to run and does not need to be fit. Quite some time is saved by tokenizing the whole dataset at the beginning and savind the tokens in a Dataframe.<jupyter_code># Data is tokenized here and not in the analysis pipeline because it takes some time and only needs to be done once.
train_data = train_data.fillna(u"")  # Two questions have empty fields
train_data["q1_tokens"] = train_data["question1"].progress_apply(pre.tokenize)
train_data["q2_tokens"] = train_data["question2"].progress_apply(pre.tokenize)

val_data = val_data.fillna(u"")  # Two questions have empty fields
val_data["q1_tokens"] = val_data["question1"].progress_apply(pre.tokenize)
val_data["q2_tokens"] = val_data["question2"].progress_apply(pre.tokenize)

test_data = test_data.fillna(u"")  # Two questions have empty fields
test_data["q1_tokens"] = test_data["question1"].progress_apply(pre.tokenize)
test_data["q2_tokens"] = test_data["question2"].progress_apply(pre.tokenize)<jupyter_output><empty_output><jupyter_text>Save the three tokenized datasets:<jupyter_code>output_directory = "../output/data/"
prefix = "tokenized_"

utility.save_pickle(train_data, output_directory, prefix+"train_data.pkl") 
utility.save_pickle(val_data, output_directory, prefix+"val_data.pkl")
utility.save_pickle(test_data, output_directory, prefix+"test_data.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10623494781910624, 'fraction_numerical': 0.012576933369012578, 'mean_word_length': 7.143790849673203, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '263383', 'license': 'MIT', 'n_tokens_mistral': 1218, 'n_tokens_neox': 1181, 'n_words': 349, 'path': 'notebooks/split-data-and-tokenize-01-als.ipynb', 'repo_name': 'schorlem/capstone_project', 'size': '3737'}