<jupyter_start><jupyter_text>Coherence Maximization Recipe DemoThe notebook describes how one can use the Coherence maximization recipe to build an experiment and train such a model which has high coherence value.* Recipes in TopicNet: [recipes](https://github.com/machine-intelligence-laboratory/TopicNet/tree/master/topicnet/cooking_machine/recipes)* Paper about coherence which is used in the recipe: [Intra-Text Coherence as a Measure of Topic Models' Interpretability](http://www.dialog-21.ru/media/4281/alekseevva.pdf)  Contents* [Loading Dataset](data)* [Formatting the Recipe](formatting)* [Experiment](experiment)* [Best Model](best-model)    * [Exploring Best Model](investigating)<jupyter_code>import os
import numpy as np
import shutil

import matplotlib.pyplot as plt
from matplotlib import cm

%matplotlib inline
from IPython.display import display_html
from topicnet.dataset_manager import load_dataset
from topicnet.cooking_machine import Dataset
from topicnet.cooking_machine.pretty_output import make_notebook_pretty
from topicnet.cooking_machine.recipes import IntratextCoherenceRecipe
from topicnet.viewers.top_tokens_viewer import TopTokensViewer
from topicnet.viewers.top_documents_viewer import TopDocumentsViewer
make_notebook_pretty()<jupyter_output><empty_output><jupyter_text>Loading DatasetBack to Contents Let's pick up a dataset from the [list of available for download](https://github.com/machine-intelligence-laboratory/TopicNet/blob/master/topicnet/dataset_manager/DemoDataset.ipynb)<jupyter_code>DATASET_NAME = 'postnauka'
DATASET = load_dataset(DATASET_NAME)
DATASET._data.head()
DATASET_PATH = DATASET._data_path
DATASET_PATH<jupyter_output><empty_output><jupyter_text>Let's see how big the dataset is<jupyter_code>DATASET.get_dictionary()
DATASET._data.shape<jupyter_output><empty_output><jupyter_text>Nearly 3000 documents — seems not many.  Formatting the RecipeBack to Contents Let's use the flag below to reduce memory consumption in some places<jupyter_code>LOW_MEMORY = True<jupyter_output><empty_output><jupyter_text>Below is the main part: replacement of the placeholders in the recipe string with the real values.<jupyter_code>training_pipeline = IntratextCoherenceRecipe()


num_documents_to_compute_coherence = 100  # not many — in order to speed up a bit 
total_num_documents = DATASET._data.shape[0]
documents_fraction = num_documents_to_compute_coherence / total_num_documents

recipe_config = training_pipeline.format_recipe(
    dataset_path=DATASET_PATH,
    main_modality='@word',
    modalities=['@word'],
    keep_dataset_in_memory=True,  # better try keep this True for quicker computation
    keep_dataset=not LOW_MEMORY,
    documents_fraction=documents_fraction, 
    num_specific_topics=20,
    num_background_topics=1,
    one_stage_num_iter=20,
    verbose=True,
)
print(recipe_config)<jupyter_output># The recipe mainly consists of basic cube stages,
# such as Decorrelation, Sparsing and Smoothing.
# In this way it is similar to ARTM baseline recipe.
# The core difference is that models selected based on their IntratextCoherenceScore
# (which is one of the scores included in TopicNet).
# PerplexityScore is also calculated to assure that models don't have high perplexity,
# but the main criteria is IntratextCoherenceScore.
#
# For more details about IntratextCoherence
# one may see the paper http://www.dialog-21.ru/media/4281/alekseevva.pdf
#
# Recipe usage sample:
#   file_contents_as_string = file_contents_as_string.format(
#     modality_names=modality_names,
#     main_modality=main_modality,
#     dataset_path=dataset_file_path,
#     keep_dataset_in_memory=True,
#     keep_dataset=False,
#     documents_fraction=documents_fraction,
#     specific_topics=specific_topic_names,
#     background_topics=background_topic_names,
#     one_stage_num_iter=20,
#     verbose=True,
#   )
[...]<jupyter_text>ExperimentBack to Contents Folder for future experiment<jupyter_code>EXPERIMENTS_FOLDER_PATH = os.path.join('..', 'experiments')
os.makedirs(EXPERIMENTS_FOLDER_PATH, exist_ok=True)
os.listdir(EXPERIMENTS_FOLDER_PATH)
EXPERIMENT_ID = 'Maximize_Coherence'<jupyter_output><empty_output><jupyter_text>Removing the folder with the chosen experiment ID if it exists<jupyter_code>COHERENCE_EXPERIMENT_FOLDER_PATH = os.path.join(EXPERIMENTS_FOLDER_PATH, EXPERIMENT_ID)

if os.path.isdir(COHERENCE_EXPERIMENT_FOLDER_PATH):
    shutil.rmtree(COHERENCE_EXPERIMENT_FOLDER_PATH)
%%time

experiment, dataset = training_pipeline.build_experiment_environment(
    experiment_id=EXPERIMENT_ID,
    save_path=EXPERIMENTS_FOLDER_PATH,
)
dataset.get_dictionary()<jupyter_output><empty_output><jupyter_text>There are too many words in the dictionary.Let's filter it (otherwise Phi matrix will be very huge).<jupyter_code>dictionary = dataset.get_dictionary()
dictionary.filter(min_df_rate=0.01, max_df_rate=0.9)
dataset._cached_dict = dictionary
dataset.get_dictionary()<jupyter_output><empty_output><jupyter_text>Experiment's `low_memory` mode:<jupyter_code>experiment._low_memory = LOW_MEMORY
experiment._low_memory<jupyter_output><empty_output><jupyter_text>Go!<jupyter_code>%%time

experiment.run(dataset)<jupyter_output>100%|██████████| 100/100 [00:24<00:00,  4.04it/s]
100%|██████████| 100/100 [00:25<00:00,  3.92it/s]
100%|██████████| 100/100 [00:24<00:00,  4.02it/s]
100%|██████████| 100/100 [00:25<00:00,  3.97it/s]
100%|██████████| 100/100 [00:26<00:00,  3.73it/s]
100%|██████████| 100/100 [01:03<00:00,  1.58it/s]
100%|██████████| 100/100 [00:26<00:00,  3.82it/s]<jupyter_text>Best ModelBack to Contents Let's find the best model of all (not only from the last stage of the experiment).<jupyter_code>SCORE_NAME = 'IntratextCoherenceScore'

BEST_MODEL = None
levels = range(1, len(experiment.cubes) + 1)

for level in levels:
    best_model_candidates = experiment.select(
        f'{SCORE_NAME} -> max',
        level=level
    )

    if len(best_model_candidates) == 0:
        continue

    best_model_candidate = best_model_candidates[0]

    if (BEST_MODEL is None or
            BEST_MODEL.scores[SCORE_NAME][-1] <
            best_model_candidate.scores[SCORE_NAME][-1]):

        BEST_MODEL = best_model_candidate
BEST_MODEL<jupyter_output><empty_output><jupyter_text>All other models (excluding the Root one):<jupyter_code>models_to_compare_with = [
    m for m in experiment.models.values()
    if len(m.scores[SCORE_NAME]) > 0
]
len(models_to_compare_with)
len(experiment.models)  # plus Root<jupyter_output><empty_output><jupyter_text>Best model's ID and its score value<jupyter_code>print(f'         ID: {BEST_MODEL.model_id}')
print(f'Score value: {BEST_MODEL.scores[SCORE_NAME][-1]:.5f}')<jupyter_output>ID: --10h05m21s_23d04m2020y---
Score value: 0.01554<jupyter_text>Other models' score value:<jupyter_code>for level in levels:
    print('Level:', level)
    
    current_models = experiment.select(
        '',
        level=level
    )
    
    for m in current_models:
        if len(m.scores[SCORE_NAME]) == 0:
            score_value = float('nan')
        else:
            score_value = m.scores[SCORE_NAME][-1]

        print(f'{m.model_id}: {score_value:.5f}')
    
    print()<jupyter_output>Level: 1
-----------root-----------: nan

Level: 2
--09h40m13s_23d04m2020y---: 0.00135
--09h41m07s_23d04m2020y---: 0.00068
--09h42m03s_23d04m2020y---: 0.00068
--09h43m01s_23d04m2020y---: 0.00069
--09h44m02s_23d04m2020y---: 0.00081
--09h45m06s_23d04m2020y---: 0.00054
--09h48m50s_23d04m2020y---: 0.00013

Level: 3
--09h50m34s_23d04m2020y---: 0.00180
--09h51m33s_23d04m2020y---: 0.00180
--09h52m34s_23d04m2020y---: 0.00450
--09h53m37s_23d04m2020y---: 0.00851
--09h54m41s_23d04m2020y---: 0.00988
--09h55m49s_23d04m2020y---: 0.01175

Level: 4
--09h57m16s_23d04m2020y---: 0.00991
--09h58m18s_23d04m2020y---: 0.00991
--09h59m22s_23d04m2020y---: 0.01048

Level: 5
--10h00m36s_23d04m2020y---: 0.01036
--10h01m41s_23d04m2020y---: 0.01091
--10h02m48s_23d04m2020y---: 0.01231
--10h03m57s_23d04m2020y---: 0.00000

Level: 6
--10h05m21s_23d04m2020y---: 0.01554
--10h06m33s_23d04m2020y---: 0.01554
--10h07m47s_23d04m2020y---: 0.01554<jupyter_text>Exploring Best ModelBack to ContentsMore detailed description of the best model goes here.<jupyter_code>BEST_MODEL.describe_regularizers()
BEST_MODEL.describe_scores()<jupyter_output><empty_output><jupyter_text>Let's use some viewers:<jupyter_code>toptok_viewer = TopTokensViewer(BEST_MODEL, num_top_tokens=10, method='phi')
topdoc_viewer = TopDocumentsViewer(BEST_MODEL, dataset=dataset)
top_tokens_view = toptok_viewer.view_from_jupyter(
    display_output=False,
    give_html=True,
)
display_html(' '.join(top_tokens_view), raw=True)<jupyter_output><empty_output><jupyter_text>Parts of topics' top documents:<jupyter_code>topdoc_viewer.view_from_jupyter(
    current_num_top_doc=5,
    num_view_topics=5,
)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11439530685920578, 'fraction_numerical': 0.07603790613718411, 'mean_word_length': 5.665413533834586, 'pattern_counts': {'":': 0, '<': 73, '<?xml version=': 0, '>': 67, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '249142', 'license': 'MIT', 'n_tokens_mistral': 3824, 'n_tokens_neox': 3291, 'n_words': 600, 'path': 'topicnet/demos/Coherence-Maximization-Recipe.ipynb', 'repo_name': 'machine-intelligence-laboratory/topicnet', 'size': '8864'}