<jupyter_start><jupyter_text>Word2VecGenerator  1. Import statement **Import neccessary packages and modules**<jupyter_code>import time
t = time.time()
import string
import random
import json
import math
import random
import pandas as pd
import numpy as np
from sklearn.preprocessing import LabelBinarizer
import tensorflow as tf
import matplotlib
from matplotlib import pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>**Import gensim**<jupyter_code>from gensim.models import Word2Vec<jupyter_output><empty_output><jupyter_text>**Count the number of workers**<jupyter_code>import multiprocessing
WORKERS = multiprocessing.cpu_count()
print("Number of workers:",WORKERS)<jupyter_output>Number of workers: 16<jupyter_text>2. Load and Inspect Cleaned Data **Input directory**<jupyter_code>input_dir = "../input/"<jupyter_output><empty_output><jupyter_text>**Load train and test data**<jupyter_code>train_data = pd.read_csv(input_dir+'modified_train_data.csv') 
test_data = pd.read_csv(input_dir+'modified_test_data.csv')<jupyter_output><empty_output><jupyter_text>**Inspect train and test data**<jupyter_code>train_data.head(20)
print("Shape of train data:", train_data.shape)
test_data.head(20)
print("Shape of test data:", test_data.shape)<jupyter_output>Shape of test data: (153164, 3)<jupyter_text>3. Create Word2Vec Model **Aggregrate all the comments from train and test data in a list**<jupyter_code>train_comment = train_data["comment_text"].values
test_comment = test_data["comment_text"].values
all_comments = np.concatenate((train_comment, test_comment), axis = 0)
print("Shape of all comments:",all_comments.shape)
del(train_comment, test_comment)
all_comments = all_comments.tolist()<jupyter_output>Shape of all comments: (312735,)<jupyter_text>**Break each comment into list of words**<jupyter_code>sentences = []
for comment in all_comments:
    comment = str(comment)
    sentences.append(comment.split())
del(all_comments)<jupyter_output><empty_output><jupyter_text>**Create Word2Vec model using gensim**<jupyter_code>t = time.time()
size = 300
window = 25
model = Word2Vec(sentences, sg = 1, size=size, window=window, min_count=1, workers=WORKERS)
model.train(sentences, total_examples=len(sentences), epochs=10)
print("Finished creating word2vec model.", "Time needed:", time.time()-t,"sec")<jupyter_output>Finished creating word2vec model. Time needed: 765.8476777076721 sec<jupyter_text>4. Save the model to the kernel **Save directory**<jupyter_code>save_dir = "./word2vec_models/"<jupyter_output><empty_output><jupyter_text>**Save the Word2Vec model in the designated directory**<jupyter_code>fname = "word2vec_model.mdl"
model.wv.save_word2vec_format(save_dir+fname)<jupyter_output><empty_output><jupyter_text>**Save the Word2Vec model info in a json file in the designated directory**<jupyter_code>data = {"size": size, "window":window}
with open(save_dir+'data.json', 'w') as outfile:
    json.dump(data, outfile)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11623246492985972, 'fraction_numerical': 0.020374081496325986, 'mean_word_length': 8.983333333333333, 'pattern_counts': {'":': 2, '<': 45, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '273987', 'license': 'MIT', 'n_tokens_mistral': 1059, 'n_tokens_neox': 1021, 'n_words': 223, 'path': 'toxic_comment_classification/models/word2vecgenerator.ipynb', 'repo_name': 'itratrahman/machine_learning_projects', 'size': '2994'}