<jupyter_start><jupyter_text>VectorizerThis notebook takes all preprocessings and vectorizes them, in order to be classified with the MLP. As an exploration, we used spaCy's pre-trained vectors. Note that the docuemnt vectors are obtained from the word vectors via an average.<jupyter_code>import spacy
import pandas as pd
import numpy as np
from progressbar import ProgressBar, Bar, Percentage
from os import listdir
from os.path import isfile, join<jupyter_output><empty_output><jupyter_text>Load the big model (as per [documentation](https://spacy.io/usage/vectors-similarity)<jupyter_code>nlp = spacy.load(r"Q:\anaconda\Lib\site-packages\en_core_web_lg\en_core_web_lg-2.2.5")
%%time

def_str = r"Q:\\tooBigToDrive\data-mining\kaggle\data\csv"
path = r"Q:\tooBigToDrive\data-mining\kaggle\data\csv"
files = listdir(def_str)
files = [f.replace(".csv","") for f in files if "Agg" in f]

for s in files:
    csvPath = def_str +"\\"+ s + ".csv"
    npyPath = def_str +"\\"+ s +"sSub"+ ".npy"
    train = pd.read_csv(csvPath)
    train.replace(to_replace = "empty", value = "", inplace = True)
    train["body"].fillna("",inplace = True)
    # enable this to add subreddits to body 
    train["body"] = train["subreddit"]+" "+train["body"]
    to_be_vectorized = train["body"].tolist()
    vectorsl = []
    print("doing"+" "+s+".csv ...", "len(to_be_vectorized) = ",len(to_be_vectorized) )
    pbar = ProgressBar(widgets=[Percentage(), Bar()], maxval=len(to_be_vectorized)).start()
    i = 0
    # disable parser and ner pipes to have better performance
    with nlp.disable_pipes():
        for tex in to_be_vectorized:
            vectorsl.append(nlp(tex).vector)
            i += 1
            pbar.update(i)
    pbar.finish()
    vectors = np.array(vectorsl)
    np.save(npyPath,vectors)
    print("done")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1381936887921654, 'fraction_numerical': 0.002720348204570185, 'mean_word_length': 4.094182825484765, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '520333', 'license': 'MIT', 'n_tokens_mistral': 655, 'n_tokens_neox': 631, 'n_words': 155, 'path': 'Notebooks/other-attempts/spaCy/data_preparation/vectorizer.ipynb', 'repo_name': 'daorse/RedditGenderClassification', 'size': '1838'}