<jupyter_start><jupyter_text>Step 3 Load the space separated kmers of kmers into dataframe, to be fed to Count Vectorizer once done, kill the kernel to free memory.Now your features are extracted and you can later load the dataframe from disk for training model  !Attention, for k>=8, a ~30GiB of Memory is neededMemoryError: Unable to allocate 29.3 GiB for an array with shape (60000, 65536) and data type int64To mitigate this, we can write the compact matrix X_cv directly to disk (6.5GB for k=8) like in section 2 below, and load it later transforming it into a DataFrame using slicing and use  SGDClassifier to partially train a model with each batch (slice) that fits in memory (10 batches for example).<jupyter_code>###############################################################################
#        Conversion step 3: Use Word Vectorizer to count kmer occurences      #
###############################################################################
#precise size of k from previous steps
k=4

# enable automatic garbage collector to help free unused memory
gc.enable()
# gc.disable()
start = time.time()
###############################################################################
#   load file containing space separated kmers, the result of the last step   #
###############################################################################

with open(r"D:\DataSet\TfIdf\spaced words-"+str(k), 'rb') as filehandle:
     textword = pickle.load(filehandle)
print("number of samples loaded :", len(textword))

cv = CountVectorizer() # Default ngram_range (1, 1)

###############################################################################
#in case of k=1, one character word, use this instead
###############################################################################
# cv = CountVectorizer(analyzer='char') # 

X_cv = cv.fit_transform(textword)

# Get the header, the name of columns
cols=cv.get_feature_names()

# Transform into Pandas Data Frame
X = pd.DataFrame(X_cv.toarray(), columns=cols)

#  View the converted matrix
X.head(10)

###############################################################################
#Write resulting Data Frame on disk. Feather is very fast format for loading  #
###############################################################################
X.reset_index(drop = True).to_feather(r"D:\DataSet\TfIdf\dfk"+str(k)+".feather")

# total time taken
end = time.time()
print(f"Runtime of kmer counting of size: "+ str(k)+ f" using Word Vectorizer is {end - start} secs")<jupyter_output>number of samples loaded : 60000
Runtime of kmer counting of size: 4 using Word Vectorizer is 589.4460277557373 secs
{'fraction_non_alphanumeric': 0.3019223520542782, 'fraction_numerical': 0.020354315868827744, 'mean_word_length': 6.331491712707182, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '344301', 'license': 'MIT', 'n_tokens_mistral': 715, 'n_tokens_neox': 661, 'n_words': 266, 'path': 'BoW/Step 3 CV Pre-processing .ipynb', 'repo_name': 'khansabassem/MLonGenomics', 'size': '2653'}