<jupyter_start><jupyter_text>Parameter Selection for NMF 1 Now we look at the more advanced task of parameter selection for NMF topic modelling - namely, selecting a useful value for the number of topics *k*. First, let's load the TF-IDF normalised document-term matrix and list of terms that we stored earlier using *Joblib*:<jupyter_code>(df, data_samples, tfidf, tfidf_feature_names) = joblib.load( "df-data_samples-tfidf-tfidf_names.pkl" )
print( "Loaded %d X %d document-term matrix" % (tfidf.shape[0], tfidf.shape[1]) )<jupyter_output>Loaded 4691 X 1292 document-term matrix<jupyter_text>Create the Topic Models A common approach for parameter selection is to Measure and compare the topic coherence of models generated for different values of *k*. We need to start by pre-specifying an initial range of "sensible" values:<jupyter_code>kmin, kmax = 2, 40
def list_pas_iter(pas, iterat):
    #arrondir à la 2eme décimale
    value = round(0,2)
    values = [value]
    for k in range(iterat):
        value+= pas
        values.append(round(value,2))
    return(values)
alph_list = list_pas_iter(0.01, 31) + [0.4,0.5,0.6,0.7,0.8,0.9,1]
alph_list
l1_list = list_pas_iter(0.1,10)<jupyter_output><empty_output><jupyter_text>Apply NMF for each of these values:<jupyter_code>topic_models = []
# try each value of k
for k in range(kmin,kmax+1):
    for alph in alph_list:
        for l1 in l1_list:
            print("Applying NMF for k=%d, alpha =%.2f, l1 = =%.1f..." % (k, alph,l1) )
            # run NMF
            model = decomposition.NMF( init="nndsvd", n_components=k, alpha=alph, l1_ratio= l1, random_state = 8220) 
            W = model.fit_transform( tfidf )
            H = model.components_
            err = model.reconstruction_err_
            # store for later
            topic_models.append( (k,alph,l1,W,H,err) )
joblib.dump((topic_models), "topic_models.pkl")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10890052356020942, 'fraction_numerical': 0.028795811518324606, 'mean_word_length': 3.7775, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '324541', 'license': 'MIT', 'n_tokens_mistral': 690, 'n_tokens_neox': 661, 'n_words': 194, 'path': '3_Topic_modeling/.ipynb_checkpoints/2. Parameters_Selection_NMF1-checkpoint.ipynb', 'repo_name': 'ClaireLepault/Mapping-country-mitig-pathways', 'size': '1910'}