<jupyter_start><jupyter_text>Miquela- Automatic Music Video Comment Analysis ![Miquela](ss2.jpg)  LOAD Miquela- Automatic Music Video Comments   Import data analysis tool Pandas<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Read in data into data frame using pandas<jupyter_code>data = pd.read_json('data/Miquela_Automatic_Comments',lines=True)
## Display raw data w/emojis 
data.head(15)<jupyter_output><empty_output><jupyter_text>remove emojis from comments --unicode ascii string literal string escape<jupyter_code>for ind,val in enumerate(data['text']):
    data.loc[ind]['text'] = val.replace(u"\\",u" \\").encode('ascii','ignore')<jupyter_output><empty_output><jupyter_text>Display preprocessed data<jupyter_code>data.head(15)<jupyter_output><empty_output><jupyter_text>create array with only comments, and only comment authors<jupyter_code>train_data = (data['text'].values)
train_author = data['author'].values<jupyter_output><empty_output><jupyter_text>tokenize each comment, and append list of tokenzied commenets to a list of comments<jupyter_code>from tokenize import tokenize
import string
import gensim
token_train_data=[]
for i in train_data:
    comment = str(i).translate(None, string.punctuation)
    token_train_data.append(list(gensim.utils.tokenize(comment)))<jupyter_output><empty_output><jupyter_text>Display tokenized comment in comment array<jupyter_code>token_train_data[1]<jupyter_output><empty_output><jupyter_text>Load tokenized data into dictionary and id2words for BoW embeddings<jupyter_code>from gensim import corpora
dictionary = corpora.Dictionary(token_train_data)
corpus = [dictionary.doc2bow(text) for text in token_train_data]<jupyter_output><empty_output><jupyter_text>TRAIN LDA TOPIC MODEL ---- edit number of topics/ n - clusters<jupyter_code>NUM_TOPICS = 10
ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics = NUM_TOPICS, id2word=dictionary, passes=15)
ldamodel.save('model5.gensim')<jupyter_output><empty_output><jupyter_text>LDA TOPIC VISUALIZATION<jupyter_code>lda = gensim.models.ldamodel.LdaModel.load('lda_model/model5.gensim')
import pyLDAvis.gensim
lda_display = pyLDAvis.gensim.prepare(lda, corpus, dictionary, sort_topics=False)
pyLDAvis.display(lda_display)<jupyter_output><empty_output><jupyter_text>ANALYZE COMMENTS FOR SENTIMENT ANALYSIS<jupyter_code>from textblob import TextBlob<jupyter_output><empty_output><jupyter_text>Parts of speech tagging for each token in comment by Textblob<jupyter_code>b = TextBlob(train_data[11])
b.tags<jupyter_output><empty_output><jupyter_text>produce polarity and subjectivity metrics by iterating through comments array polarity (-1 being negative , +1 being postive) and subjectivity (0 being factual and 1 being subjective)  i.e. polarity is 0.8,  the statement is positive and 0.75 subjectivity = public opinion and not a factual information.<jupyter_code>sentiment_dict = {}
count = 0
for i in train_data:
    b= TextBlob(i)
    sentiment_dict[count]={'polarity':b.sentiment.polarity,'subj':b.sentiment.subjectivity}
    count = count+1<jupyter_output><empty_output><jupyter_text>add polarity and subjectivity metrics to initial dataframe<jupyter_code>sentiment_df = pd.DataFrame.from_dict(sentiment_dict,orient='index')
sentiment_df.columns
data['polarity']=sentiment_df[u'polarity']
data['subjectivity']=sentiment_df[u'subj']
data.head(10)<jupyter_output><empty_output><jupyter_text>Summary Statistics for Polarity and Subjectivity of Comments for  "Miquela - Automatic" music video<jupyter_code>data['polarity'].describe()
data['subjectivity'].describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10424710424710425, 'fraction_numerical': 0.00827357970215113, 'mean_word_length': 9.019337016574585, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 61, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '202485', 'license': 'MIT', 'n_tokens_mistral': 1276, 'n_tokens_neox': 1221, 'n_words': 242, 'path': 'data_preprocessing+model.ipynb', 'repo_name': 'mashmade/lilmiquela_youtube_analytics', 'size': '3626'}