<jupyter_start><jupyter_text>Demo of LDAPathwayPrediction modeling pipeline  Load all required modules<jupyter_code>from Model import Model
from Validation import Validation
import matplotlib.pyplot as plt
import pandas as pd
pd.set_option('mode.chained_assignment', None)
import numpy as np<jupyter_output><empty_output><jupyter_text>Load data files<jupyter_code>data_df=Model().df_pkl
train,test,train_df,test_df=Model().get_train_and_test(data_df.copy())<jupyter_output><empty_output><jupyter_text>Get dictionary and corpus from train set<jupyter_code>dictionary,corpus=Model().get_dict_corpus(train)
model=Model().MyLDA(corpus,dictionary,num_topics=120,random_state=1,passes=100)
model.save('lda_train_100topics_200passes.model')<jupyter_output><empty_output><jupyter_text>Get similarity between topics  Measure similarity between topics. The objective is to maximize dissimilarity between topics. Jaccard distance is a less conservative measurement, and is the default. Read gensim docs for other options.<jupyter_code>distance_list=Validation().get_topic_similarity(topics=120,model=model,distance='jensen_shannon')
#higher the value, more dissimilar the topics
print ("Mean distance is {} and standard deviation is {}".format(np.mean(distance_list),np.std(distance_list)))<jupyter_output>Mean distance is 0.5258115107117957 and standard deviation is 0.2970946580264996<jupyter_text>Test the model with test set Test 1: comparing enzymes in predicted topics with enzymes in true pathways<jupyter_code>output=Validation().solution_for_one_df(test_df,dictionary,model)
Validation().print_testing_results(output)<jupyter_output>Recall mean: 0.5558711422995022 and median 0.6058139534883721
Precision mean: 0.1220346906696905 and median 0.084375
Accuracy mean: 0.8789623100175804 and median 0.8980005795421617
F1 mean: 0.17110661073828068 and median 0.11126951054574699
specificity mean: 0.8803726132764471 and median 0.9035324111040683<jupyter_text>Run n-fold validation to check test measures<jupyter_code>#output=Validation().nfold_cv(n=10)
#Validation().print_testing_results(output)<jupyter_output>Recall mean: 0.5558711422995022 and median 0.6058139534883721
Precision mean: 0.1220346906696905 and median 0.084375
Accuracy mean: 0.8789623100175804 and median 0.8980005795421617
F1 mean: 0.17110661073828068 and median 0.11126951054574699
specificity mean: 0.8803726132764471 and median 0.9035324111040683<jupyter_text>Plot all relevant distributions<jupyter_code>#Validation().plot_a_distribution(output[0],xlabel='Recall',
#                                xlim_tuple=(0,1),
#                                ylabel='Density',title='Recall values')

#Validation().plot_a_distribution(output[1],xlabel='Precision',
#                                xlim_tuple=(0,1),
#                                ylabel='Density',title='Precision values')

#Validation().plot_a_distribution(output[2],xlabel='Accuracy',
#                                xlim_tuple=(0,1),
 #                               ylabel='Density',title='Accuracy values')

#Validation().plot_a_distribution(output[3],xlabel='F1',
#                                xlim_tuple=(0,1),
#                                ylabel='Density',title='F1 values')

#Validation().plot_a_distribution(output[4],xlabel='Specificity',
#                                xlim_tuple=(0,1),
#                                ylabel='Density',title='Specificity values')<jupyter_output><empty_output><jupyter_text>Test 2: similarity of test document with documents in the training set  Measure similarity between distribution of topics. The objective is to maximize similarity between topics. Jensen-Shannon distance is a symmetric measurement and is used in the test. [Read this, for more information on JSD](https://www.cise.ufl.edu/~anand/sp06/jensen-shannon.pdf)<jupyter_code>df_result=Validation().compare_test_train_docs(test_df,train_df,model,dictionary)
df_result.head()
Validation().print_heat_map_for_one(df_result,test_df)<jupyter_output><empty_output><jupyter_text>Validate the model with new dataset from a gut microbial strain<jupyter_code>test_df
valid=['ec:192.168.127.12','ec:2.8.3.8','ec:3.1.2.1','ec:192.168.3.11','ec:172.16.31.10','ec:2.3.1.9'];valid_dist={}
valid_dist['Map']='PF06050'
valid_dist['Name']='UNK pathway'
valid_dist['EC']=[valid]
df_validate=pd.DataFrame(valid_dist)
df_validate
df_result=Validation().compare_test_train_docs(df_validate,train_df,model,dictionary)
Validation().print_heat_map_for_one(df_result,df_validate)
import gensim
model_load= gensim.models.LdaModel.load('lda_train_90topics_200passes.model')

df_result=Validation().compare_test_train_docs(df_validate,train_df,model_load,dictionary)
Validation().print_heat_map_for_one(df_result,df_validate)
df_result.sort_values(by='JSD').head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10663063812097277, 'fraction_numerical': 0.09353564747453752, 'mean_word_length': 5.884120171673819, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 6, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '646325', 'license': 'MIT', 'n_tokens_mistral': 1880, 'n_tokens_neox': 1553, 'n_words': 274, 'path': 'src/pyext/.ipynb_checkpoints/Play-checkpoint.ipynb', 'repo_name': 'saijananiganesan/LDAPathwayPrediction', 'size': '4799'}