<jupyter_start><jupyter_text>Sentiment Analysis using LDABusinesses often want to know how customers think about the quality of their services to improve and make more profits. Restaurant goers may want to learn from others' experience using a variety of criteria such as food quality, service, ambience, discounts and worthiness. Yelp users may post their reviews and ratings on businesses and services or simply express their thoughts on other reviews. Bad (negative) reviews from one's perspective may influence potential customers in making decisions, e.g., a potential customer may cancel a service and persuade other do the same.  Topic ModellingAs the name suggests, it is a process to automatically identify topics present in a text object and to derive hidden patterns exhibited by a text corpus. Thus, assisting better decision making.Topic Modelling is different from rule-based text mining approaches that use regular expressions or dictionary based keyword searching techniques. It is an unsupervised approach used for finding and observing the bunch of words (called “topics”) in large clusters of texts.Topics can be defined as “a repeating pattern of co-occurring terms in a corpus”Topic Models are very useful for the purpose for document clustering, organizing large blocks of textual data, information retrieval from unstructured text and feature selection.<jupyter_code>import pandas as pd
import numpy as np
import gensim
from textblob import TextBlob
import re
import nltk
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer as ps
from nltk.stem.wordnet import WordNetLemmatizer
import string
from gensim.parsing.preprocessing import STOPWORDS<jupyter_output><empty_output><jupyter_text>Reading the json file<jupyter_code>path = 'Filtered_review_10K.json'
df = pd.read_json(path, orient='records', lines=True)<jupyter_output><empty_output><jupyter_text>Preparing DocumentsCleaning is an important step before any text mining task, in this step, we will remove the punctuations, stopwords and normalize the corpus.<jupyter_code>import re

rest_review_dict = dict()
for temp in df.iterrows() :
    row = temp[1]
    business_id = row.business_id
    
    exclude = set(string.punctuation)
    
    
    review_text = row['text']
    stop_free = ' '.join([word for word in review_text.lower().split() if word not in STOPWORDS])
    stop_punc = ''.join(ch for ch in stop_free if ch not in exclude)
    text = ''.join([i for i in stop_punc if not i.isdigit()])
    
    review_stars = row['stars']
    
    if business_id in rest_review_dict :
        reviews_array = rest_review_dict[business_id]
        reviews_array.append({'review_text' : review_text, 'review_stars' : review_stars,
                              'polarity' : TextBlob(text).sentiment.polarity,
                             'stemmed_text' : text})
    else :
        reviews_array = list()
        reviews_array.append({'review_text' : review_text, 'review_stars' : review_stars,
                              'polarity' : TextBlob(text).sentiment.polarity,
                             'stemmed_text' : text})
        rest_review_dict[business_id] = reviews_array<jupyter_output><empty_output><jupyter_text>Latent Dirichlet Allocation (LDA) for Topic ModelingLatent Dirichlet Allocation is the most popular topic modeling technique. LDA assumes documents are produced from a mixture of topics. Those topics then generate words based on their probability distribution. Given a dataset of documents, LDA backtracks and tries to figure out what topics would create those documents in the first place.LDA is a matrix factorization technique. In vector space, any corpus (collection of documents) can be represented as a document-term matrix.  Preparing Document-Term MatrixAll the text documents combined is known as the corpus. To run any mathematical model on text corpus, it is a good practice to convert it into a matrix representation. LDA model looks for repeating term patterns in the entire DT matrix. Python provides many great libraries for text mining practices, “gensim” is one such clean and beautiful library to handle text data. It is scalable, robust and efficient. Following code shows how to convert a corpus into a document-term matrix. Running LDA ModelNext step is to create an object for LDA model and train it on Document-Term matrix. The training also requires few parameters as input. The gensim module allows both LDA model estimation from a training corpus and inference of topic distribution on new, unseen documents.<jupyter_code>import gensim
from gensim import corpora, models, similarities


business_corpus = dict()
for business_id, review_array in rest_review_dict.items():
    consolidated_text = [review['stemmed_text'] for review in review_array]
    texts = []
    for t in consolidated_text :
        for word in t.split(" ") :
            texts.append(word)
    texts = [texts]
    dictionary = corpora.Dictionary(texts)
    corpus = [dictionary.doc2bow(text) for text in texts]
    
    ## Creating the object for LDA model using gensim library
    lda = models.LdaModel(corpus=corpus, id2word=dictionary, num_topics=2)
    topics = dict()
    for topic in lda.top_topics(corpus) :
        b = topic[0][0:15]
        for tup in b :
            if tup[1] not in topics :
                topics[tup[1]] = tup[0]
            else :
                if topics[tup[1]] < tup[0] :
                    topics[tup[1]] = tup[0]
    
    
    business_corpus[business_id] = topics<jupyter_output>C:\Users\Monas\Anaconda3\envs\deeplearning\lib\site-packages\numpy\core\fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
C:\Users\Monas\Anaconda3\envs\deeplearning\lib\site-packages\numpy\core\_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)<jupyter_text>ResultWe can check the top topics and the top numbers from our lda model<jupyter_code>print(lda.print_topics(num_topics=10, num_words=3))
print(business_corpus['hW0Ne_HTHEAgGF1rAdmR-g'])
all_reviews = rest_review_dict['hW0Ne_HTHEAgGF1rAdmR-g']

business_reviews = []
for review in all_reviews :
    if review['polarity'] < 0 :
        print ('Negative review: ' + review['stemmed_text'])
    else :
        print ('Positive review: ' + review['stemmed_text'])<jupyter_output>Positive review: luckily didnt travel far connecting flight this thank you phoenix brief layover pleasant employees kind flight time hopefully time grace phoenix presence little longer
Positive review: beach paradise ixtapazihuatenejo awaited usunfortunately connecting flight arizona places long story short luggage arrive final destination articles clothing missing cell phone chargernot mention security forcing open sealed containers sons baby food milk litmus tests looked bright sideat didnt deported 
Positive review: plenty airports world better end easy navigate clean from experienced makes getting returning rental cars breeze airports recent travel hold security check line think want prepared check quickly possible held breath young lady came search  quart zip lock bag fearing evacuate terminal breech able flights word pet lovers insist traveling fourlegged friends allergies asthma plague large number people difficulty breathing caused pet presents unexpected glitch travel day cons[...]
{'fraction_non_alphanumeric': 0.06014935505770536, 'fraction_numerical': 0.004480651731160896, 'mean_word_length': 4.2278211497515965, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 19, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '275923', 'license': 'MIT', 'n_tokens_mistral': 1999, 'n_tokens_neox': 1915, 'n_words': 819, 'path': 'src/scripts/Sentiment Analysis using LDA.ipynb', 'repo_name': 'shrican/yelp-recommendations-sentimental-analysis', 'size': '7365'}