<jupyter_start><jupyter_text>Topic modelling on news data for 20 topics- Data is taken from kaggle.- Topic related to health start syncronising - Now increase the topic get  the more syncronised results.<jupyter_code>import pandas as pd
data=pd.read_csv('abcnews-date-text.csv',error_bad_lines=False)
data_text = data[['headline_text']]
data_text['index'] = data_text.index
documents = data_text
print(len(documents))
print(documents[:5])<jupyter_output>1103665
                                       headline_text  index
0  aba decides against community broadcasting lic...      0
1     act fire witnesses must be aware of defamation      1
2     a g calls for infrastructure protection summit      2
3           air nz staff in aust strike for pay rise      3
4      air nz strike to affect australian travellers      4<jupyter_text>Data preprocessing  Different process we do here like as Tokenization ,lemetization and stemming the data- we wanted to convert the data to their normal form for example 'stolen' would converted to steal Here nltk is used for removing different language rather than english (Hindi and urdu like that word would be removed)- In preprocessing actually we remove all the punctuation marks , exclamatory marks and commas<jupyter_code>import gensim
from gensim.utils import simple_preprocess
from gensim.parsing.preprocessing import STOPWORDS
from nltk.stem import WordNetLemmatizer, SnowballStemmer
from nltk.stem.porter import *
import numpy as np
np.random.seed(2018)
import nltk
nltk.download('wordnet')<jupyter_output>[nltk_data] Downloading package wordnet to
[nltk_data]     C:\Users\user\AppData\Roaming\nltk_data...
[nltk_data]   Package wordnet is already up-to-date!<jupyter_text>Lemmatization  Lemmatization is used for gouping of word that's contains the same meaning(synonyms,antonyms) Tokenization Tokenization is used for keeps the word having meaningfull meaning- This is used for removal of word like if,the ,a,an that word doesn't make any sense in Topic Stemming Stemming is used for convert the word into their root form<jupyter_code>def lemmatize_stemming(text):
    return WordNetLemmatizer().lemmatize(text, pos='v')
def preprocess(text):
    result = []
    for token in gensim.utils.simple_preprocess(text):
        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3:
            result.append(lemmatize_stemming(token))
    return result<jupyter_output><empty_output><jupyter_text>preview data after preprocessing - How the data will look like<jupyter_code># Select a document to preview after preprocessing
doc_sample = documents[documents['index'] == 4310].values[0][0]
print('original document: ')
words = []
for word in doc_sample.split(' '):
    words.append(word)
print(words)
print('\n\n tokenized and lemmatized document: ')
print(preprocess(doc_sample))


processed_docs = documents['headline_text'].map(preprocess)
processed_docs[:10]<jupyter_output><empty_output><jupyter_text>Dictionary is formed for<jupyter_code>dictionary = gensim.corpora.Dictionary(processed_docs)
count = 0
for k, v in dictionary.iteritems():
    print(k, v)
    count += 1
    if count > 10:
        break
dictionary.filter_extremes(no_below=15, no_above=0.5, keep_n=100000)
bow_corpus = [dictionary.doc2bow(doc) for doc in processed_docs]
bow_corpus[4310]
import pickle
pickle.dump(bow_corpus, open('bow_corpus.pkl', 'wb'))
dictionary.save('dictionary.gensim')
bow_doc_4310 = bow_corpus[4310]
for i in range(len(bow_doc_4310)):
    print("Word {} (\"{}\") appears {} time.".format(bow_doc_4310[i][0], 
                                               dictionary[bow_doc_4310[i][0]], 
bow_doc_4310[i][1]))
from gensim import corpora, models
tfidf = models.TfidfModel(bow_corpus)
corpus_tfidf = tfidf[bow_corpus]
from pprint import pprint
for doc in corpus_tfidf:
    pprint(doc)
    break
lda_model = gensim.models.LdaMulticore(bow_corpus, num_topics=20, id2word=dictionary, passes=2, workers=2)
lda_model.save('model20.gensim')
for idx, topic in lda_model.print_topics(-1):
    print('Topic: {} \nWords: {}'.format(idx, topic))<jupyter_output>Topic: 0 
Words: 0.105*"australia" + 0.040*"test" + 0.030*"time" + 0.027*"life" + 0.027*"take" + 0.024*"lead" + 0.023*"victoria" + 0.020*"industry" + 0.016*"mark" + 0.016*"fail"
Topic: 1 
Words: 0.070*"court" + 0.054*"charge" + 0.046*"face" + 0.042*"murder" + 0.038*"jail" + 0.034*"drug" + 0.034*"accuse" + 0.028*"women" + 0.028*"trial" + 0.024*"children"
Topic: 2 
Words: 0.054*"canberra" + 0.037*"fight" + 0.035*"final" + 0.033*"train" + 0.027*"price" + 0.024*"season" + 0.017*"come" + 0.016*"rail" + 0.015*"sexual" + 0.015*"march"
Topic: 3 
Words: 0.063*"sydney" + 0.050*"melbourne" + 0.049*"attack" + 0.040*"police" + 0.032*"shoot" + 0.028*"arrest" + 0.025*"win" + 0.025*"dead" + 0.023*"assault" + 0.023*"case"
Topic: 4 
Words: 0.034*"fund" + 0.033*"health" + 0.028*"world" + 0.028*"indigenous" + 0.026*"service" + 0.026*"minister" + 0.022*"need" + 0.022*"tell" + 0.021*"centre" + 0.019*"community"
Topic: 5 
Words: 0.039*"call" + 0.036*"plan" + 0.035*"council" + 0.029*"water" + 0.027*"national"[...]<jupyter_text>lda_model_tfidf = gensim.models.LdaMulticore(corpus_tfidf, num_topics=10, id2word=dictionary, passes=2, workers=4)lda_model.save('model10.gensim')for idx, topic in lda_model_tfidf.print_topics(-1):    print('Topic: {} Word: {}'.format(idx, topic))<jupyter_code>processed_docs[4310]<jupyter_output><empty_output><jupyter_text>for index, score in sorted(lda_model[bow_corpus[4310]], key=lambda tup: -1*tup[1]):    print("\nScore: {}\t \nTopic: {}".format(score, lda_model.print_topic(index, 10))) for index, score in sorted(lda_model_tfidf[bow_corpus[4310]], key=lambda tup: -1*tup[1]):    print("\nScore: {}\t \nTopic: {}".format(score, lda_model_tfidf.print_topic(index, 20)))<jupyter_code>unseen_document = 'breast cancer lead to the breakdown of women and diagnostic and medicine of cancer is very costly.'
bow_vector = dictionary.doc2bow(preprocess(unseen_document))
for index, score in sorted(lda_model[bow_vector], key=lambda tup: -1*tup[1]):
    print("Score: {}\t Topic: {}".format(score, lda_model.print_topic(index, 10)))
dictionary = gensim.corpora.Dictionary.load('dictionary.gensim')
corpus = pickle.load(open('bow_corpus.pkl', 'rb'))
lda = gensim.models.ldamodel.LdaModel.load('model20.gensim')

import pyLDAvis.gensim
lda_display = pyLDAvis.gensim.prepare(lda, corpus, dictionary, sort_topics=False)
pyLDAvis.display(lda_display)<jupyter_output>C:\ProgramData\Anaconda3\lib\site-packages\pyLDAvis\_prepare.py:257: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

  return pd.concat([default_term_info] + list(topic_dfs))
{'fraction_non_alphanumeric': 0.13127637898413622, 'fraction_numerical': 0.05122980643283365, 'mean_word_length': 5.2077687443541105, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '155860', 'license': 'MIT', 'n_tokens_mistral': 2676, 'n_tokens_neox': 2375, 'n_words': 584, 'path': 'Topic modeling on text data-20_Topic.ipynb', 'repo_name': 'saife245/HEALTH-DATA-ANALYSIS', 'size': '6871'}