<jupyter_start><jupyter_text>Gensim Training Experiments- **Where:** HEIA-FR GPU-2- **Dataset:** enwiki-latest-pages-articles.xml.bz2 (16GB)- **Dictionary:** enwiki-english-wordids.txt.bz2 (16MB) What's going on- Training a Word2Vec on the full wikipedia english dataset using its full pre-extracted dictionary. Results- **Timeframe:** ~17h- **Errors:** kernel crash when *resetting layer weights*- **Comments:**    - Those 17 hours of training are lost as it was not possible to save the model before the crash<jupyter_code># Set constantes
datafile_name = "enwiki-latest-pages-articles.xml.bz2"
dictionary_name = 'enwiki-english-wordids.txt.bz2'
# Start logging process at root level
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
logging.root.setLevel(level=logging.INFO)
# Load dictionary from file
from gensim.corpora import Dictionary
dictionary = Dictionary.load_from_text(dictionary_name)
# Build WikiCorpus based on the dictionary
from gensim.corpora import WikiCorpus
wiki = WikiCorpus(datafile_name, dictionary=dictionary)
# Initialize simple sentence iterator required for the Word2Vec model
# Bypass memory errors
class SentencesIterator:
    def __init__(self, wiki):
        self.wiki = wiki

    def __iter__(self):
        for sentence in self.wiki.get_texts():
            yield list(map(lambda x: x.decode('utf-8'), sentence))

sentences = SentencesIterator(wiki)
# Train model
from gensim.models import Word2Vec
import multiprocessing

cores = multiprocessing.cpu_count()
model = Word2Vec(sentences=sentences, size=300, min_count=1, window=5, workers=cores)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10497900419916016, 'fraction_numerical': 0.013797240551889621, 'mean_word_length': 5.592885375494071, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '482283', 'license': 'MIT', 'n_tokens_mistral': 534, 'n_tokens_neox': 497, 'n_words': 149, 'path': 'notebooks/pa - heiafr-gpu-2-Word2Vec-with-full-dictionary.ipynb', 'repo_name': 'Penguinazor/mse.pa', 'size': '1667'}