<jupyter_start><jupyter_text>Natural Language Processing (NLP) Python Machine Learning Class 3We'll now turn to machine learning applied to text, a.k.a. Natural Language Processing (NLP) or computational linguistics.  We'll use the Reddit 2015 comments dataset for our investigations. Class Objectives  * Ingest and organize text data  * Create a pre-processing pipeline  * Extract features   * Create numerical representations  * Improve Python skills Resources(1) Pandas has some useful, basic support for manipulating text.          http://pandas.pydata.org/pandas-docs/stable/text.html        (2) The Python Natural Language Tookit (nltk):         http://www.nltk.org/        (3) SpaCy is a highly automated and capable NLP library        https://spacy.io/        (4) Reference and practice for Regular Expressions (regex):   * [Online Tutorial](https://www.regular-expressions.info/tutorial.html)  * [Online Practice](http://regexr.com/)        (5) [Gensim](https://radimrehurek.com/gensim/index.html)-- Python library for topic modeling which works alongside NLTK.          Scenario: Your company, a car manufacturer, wants to use social media to understand current trends in public interest and opinion.  What does the public think about the company's products?  Competitor's products?  You (a data scientist, software developer, or business analyst) have been picked as a team member due to your practical knowledge of machine-learning techniques. You decide to start with an analysis of Reddit comments.  For example: https://www.reddit.com/r/Toyota/    WARNING: WE WILL BE WORKING WITH UNCENSORED COMMENTS POSTED IN A PUBLIC DISCUSSION FORUM.  SOME OF THESE COMMENTS MAY INCLUDE OFFENSIVE CONTENT.  WE HAVE NOT SELECTED ANY OFFENSIVE COMMENTS TO SHOW IN THE CLASS.  OFFENSIVE COMMENTS WHICH MAY APPEAR ARE THE RESULT OF APPLYING ML-ALGORITHMS FOR THE PURPOSE OF TOPIC MODELING AND OPINION MINING.<jupyter_code>%matplotlib inline
# allows plotting in cells, we'll use later on.
import pylab
pylab.rcParams['figure.figsize'] = (10, 6) # set a larger figure size
!pip install matplotlib<jupyter_output>Requirement already satisfied: matplotlib in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: cycler>=0.10 in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: numpy>=1.7.1 in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: pytz in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: six>=1.10 in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: python-dateutil in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages (from matplotlib)
Requirement already satisfied: functools32 in /Users/peter/Virtualenvs/standard2/lib/python2.7/site-pack[...]<jupyter_text>Data import from the Kaggle database.  Skip this part if you are using the csv file.<jupyter_code># Ford vs. Toyota
# Topic modeling (unsupervised learning)
import sqlite3
conn = sqlite3.connect('/path/to/your/database.sqlite')
import pandas
# List the tables in this database!

query = """
SELECT name from sqlite_master WHERE type='table';
"""
c = conn.cursor()
c.execute(query)
data = c.fetchall()  # you can also iterate over c e.g. for row in c: doStuff(data)
print data
# List the fields in this table!
query = """
pragma table_info('May2015');
"""
c = conn.cursor()
c.execute(query)
data = c.fetchall()  # you can also iterate over c e.g. for row in c: doStuff(data)
print data
query = """
SELECT subreddit,
    body,
    score
    FROM May2015
    WHERE subreddit = '{0}'
"""
# On Reddit, topics are called 'subreddits'
# https://www.reddit.com/r/Toyota/
# https://www.reddit.com/r/Ford/

df_ford = pandas.read_sql(query.format('Ford'), conn)
df_toyota =  pandas.read_sql(query.format('Toyota'), conn)
df_ford.head()
df_ford.describe()
df_toyota.head()
df_toyota.describe()
# save out data frames so interested parties can skip the 30 GB database
# df_toyota.to_csv("C:/Users/peter/CM-ML-Class/3/data/toyota.csv", index=False, index_label=False, encoding='utf-8')
# df_ford.to_csv("C:/Users/peter/CM-ML-Class/3/data/ford.csv", index=False, index_label=False, encoding='utf-8')
# here's how to read csv's into a DataFrame:
#df_toyota = pandas.read_csv("C:/Users/peter/Documents/Codementor/MLClass/data/reddit/toyota.csv")
#df_ford = pandas.read_csv("C:/Users/peter/Documents/Codementor/MLClass/data/reddit/ford.csv")<jupyter_output><empty_output><jupyter_text>Start here if you are using the csv.<jupyter_code>import pandas as pd
df = pd.read_csv("allcars.csv")
df.head()
df.tail()
df.describe()
df['body'] = df.body.astype(str)
df.subreddit.unique()
df = df[df.subreddit != '1.0'].copy()
df.subreddit.unique()<jupyter_output><empty_output><jupyter_text>These will still show up as "object" types, since Pandas stores string pointers in the DataFrame.<jupyter_code>df.dtypes<jupyter_output><empty_output><jupyter_text>Sentiment analysisVADER is a sentiment analyzer included with NLTK.  It scores sentences by bouncing key words and even emoticons against its dictionary.  It has a list of negation words, e.g. "not" to handle phrases like "not good".  It also has a "booster dictionary" of words e.g. "very" to handle phrase like "very good".  VADER does its own text pre-processing, so data prep not required when using VADER.  However, you may have to make changes to the source code if it doesn't do a good job in your text domain.References:<NAME>. & <NAME>. (2014). VADER: A Parsimonious Rule-based Model forSentiment Analysis of Social Media Text. Eighth International Conference onWeblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.Source code: http://www.nltk.org/_modules/nltk/sentiment/vader.html<jupyter_code>from nltk.sentiment import vader
sia = vader.SentimentIntensityAnalyzer()<jupyter_output><empty_output><jupyter_text>Let's try a few examples:<jupyter_code>sia.polarity_scores("Worst car I ever drove.")<jupyter_output><empty_output><jupyter_text>The 'compound' is the overall sentiment score, which can range from -1 to 1.<jupyter_code>sia.polarity_scores("Worst car I ever drove. :)")
sia.polarity_scores("Best car I ever drove. :)")
result = sia.polarity_scores("Best car I ever drove. :)")
type(result)
result
result['compound']
def foo(x):
    result = sia.polarity_scores(x)['compound']
    return result<jupyter_output><empty_output><jupyter_text>Let's score the entire data set:<jupyter_code>df['sentiment'] = df.body.apply(lambda x: sia.polarity_scores(x)['compound'])
df['sentiment'] = df.body.apply(foo)
df.head()<jupyter_output><empty_output><jupyter_text>And do a quick graphical comparison:<jupyter_code>df.groupby(by='subreddit').sentiment.plot.kde(legend=True)<jupyter_output><empty_output><jupyter_text>Data prep: Tokenize, remove stop words, and stem.  TokenizeThis means breaking documents into sentences and sentences into words.What's the right tokenizer for our data?See www.nltk.org/api/nltk.tokenize.html<jupyter_code># old school
x = "A very simple example."
x.split(' ')
x = df.iloc[222]['body']
x

x.split(' ')
x = df.iloc[2224]['body']
x.split(' ')
import nltk
#nltk.download()
# http://www.nltk.org/data.html
from nltk.tokenize import word_tokenize
word_tokenize(x)
from nltk.tokenize import TreebankWordTokenizer
TreebankWordTokenizer().tokenize(x)
from nltk.tokenize import TweetTokenizer
TweetTokenizer().tokenize(x)
TweetTokenizer().tokenize("having a great time at #nlprocks :-)!!!!!!")
df.iloc[1011]['body']
x=df.iloc[1011]['body']
TweetTokenizer().tokenize(x)
TreebankWordTokenizer().tokenize(x)
word_tokenize(x)<jupyter_output><empty_output><jupyter_text>Looks like the TweetTokenizer is best for our data!  If none of the tokenizers work well for your data, you can always create a custom tokenizer with regular expressions (regex or regexp).<jupyter_code>from nltk.tokenize import RegexpTokenizer
regex_caps = '[A-Z]\w+' # capitalized words
regex_emotes = '(?::|;|=)(?:-)?(?:\)|\(|D|P)' # emoticons/smileys
tokenizer_caps = RegexpTokenizer(regex_caps)
tokenizer_emotes = RegexpTokenizer(regex_emotes)
tokenizer_caps.tokenize(x)
tokenizer_emotes.tokenize(x)<jupyter_output><empty_output><jupyter_text>Let's put both of these together in our own custom tokenizer class.  The `__init__` method will create and store two `RegexpTokenizer` objects.  The `tokenize` method will run both regexp tokenizers and return all the results.<jupyter_code>class CapsEmoteTokenizer(object):
    def __init__(self):
        regex_caps = '[A-Z]\w+' # capitalized words
        regex_emotes = '(?::|;|=)(?:-)?(?:\)|\(|D|P)' # emoticons/smileys
        self.tokenizer_caps = RegexpTokenizer(regex_caps)
        self.tokenizer_emotes = RegexpTokenizer(regex_emotes)
        return
    
    def tokenize(self, raw):
        caps = self.tokenizer_caps.tokenize(raw)
        emotes = self.tokenizer_emotes.tokenize(raw)
        answer = caps + emotes
        return answer
    
t = CapsEmoteTokenizer()
t.tokenize(x)<jupyter_output><empty_output><jupyter_text>"Stop Word" removalA "stop word" is a low-information content word such as "is", "and", "but", etc.  They occur frequently and add no value to the analysis.  Thus, it makes sense to remove them.  The exact list of stop words depends on the data and analysis.  NLTK provides a list of stopwords for starters.<jupyter_code>from nltk.corpus import stopwords as stopwordfactory
stopwords = stopwordfactory.words('english')
type(stopwordfactory)
type(stopwords)
len(stopwords)
stopwords[0:10]
x=df.iloc[1012]['body']
tokens = TweetTokenizer().tokenize(x)
tokens<jupyter_output><empty_output><jupyter_text>We'll use a **list comprehension** to remove stopwords; here's a quick refresher.See also: https://docs.python.org/2/tutorial/datastructures.htmllist-comprehensions<jupyter_code># list comprehension quick tutorial
range(10)
mylist = []
for i in range(10):
    mylist.append(2*i)
mylist
[ 2*i for i in range(10)]<jupyter_output><empty_output><jupyter_text>You can also add a condition<jupyter_code>[ 2*i for i in range(10) if i > 5]<jupyter_output><empty_output><jupyter_text>And use any data type:<jupyter_code>names = ['Curly', 'Larry', 'Moe', 'Schemp']

[ name + ' is a stooge' for name in names]<jupyter_output><empty_output><jupyter_text>Now back to our problem Recipe for removing stopwords from a comment:Use a list comprehension to cycle through our tokens (words) and create a list.  If the word is in the set of stopwords, don't add it to our list.<jupyter_code>[t for t in tokens]
[t for t in tokens if t.lower() not in stopwords] # note we don't retain lower case as we need car models caps<jupyter_output><empty_output><jupyter_text>We notice some unimport words for our application; let's add them to the stopwords list in memory.<jupyter_code>stopwords += ["i've", "...", "i'm", ".", "'"]
len(stopwords)
tokens_stopped = [t for t in tokens if t.lower() not in stopwords]
tokens_stopped<jupyter_output><empty_output><jupyter_text>StemmingOften we'll want to only use some sort of root word-- or stem-- in our analysis.  For example, in our present problem, we might want 'selling' & 'sell' to be the same.  This process is called 'stemming'.  (A related but more fundamental mapping is called 'lemmatization'.) NLTK provides us with several stemming options. http://www.nltk.org/api/nltk.stem.html<jupyter_code>from nltk.stem.porter import PorterStemmer
porter = PorterStemmer()
porter.stem('selling')
porter.stem('sell')
porter.stem('sold')
[porter.stem(t) for t in tokens_stopped] # it messed with our car model name; we'll have to separate out before stemming
from nltk.stem.lancaster import LancasterStemmer
lanc = LancasterStemmer()
lanc.stem('selling')
lanc.stem('sell')
lanc.stem('sold')
[lanc.stem(t) for t in tokens_stopped] # hungri eye or hungry ey lol, literally
from nltk.stem.snowball import EnglishStemmer
snow = EnglishStemmer()
snow.stem('sold')
[snow.stem(t) for t in tokens_stopped]<jupyter_output><empty_output><jupyter_text>Now let's process all of our Reddit comments  for word analysis!Proposed algorithm:1. TweetTokenizer2. Put caps words aside for safekeeping3. Remove stop words from both4. Porter stemmer on lowercase words5. Add caps words back inThis seems to make sense for our current application. We can do some diagnostics, see what we get, then modify this algorithm if necessary.<jupyter_code>from nltk.tokenize import TweetTokenizer
from nltk.tokenize import RegexpTokenizer
from nltk.stem import PorterStemmer
from nltk.corpus import stopwords as StopwordFactory

class CarsDataPrepper:
    def __init__(self):
        self.porter = PorterStemmer()
        regex_caps = '[A-Z]\w+' # capitalized words
        self.tokenizer_caps = RegexpTokenizer(regex_caps)
        self.tokenizer_tweet = TweetTokenizer()
        self.stopwords = StopwordFactory.words('english')
        self.stopwords += ["i've", "...", "i'm", ".", "'", "it's", "/", ")", "(", "]",
                           "[", ",", "!", "?", '"', '-', ':', '*', 'deleted', "|", "^", "#"
                          ">", "<"]
        return
    
    def _tokenize(self, raw):
        caps = self.tokenizer_caps.tokenize(raw)
        allwords = self.tokenizer_tweet.tokenize(raw)
        lowers = [w for w in allwords if w not in caps]
        return caps, lowers
    
    def _removeStopwords(self, caps, lowers):
        lowers_stopped = [w for w in lowers if w.lower() not in self.stopwords]
        caps_stopped = [w for w in caps if w.lower() not in self.stopwords]
        return caps_stopped, lowers_stopped
    
    def prep(self, raw):
        caps, lowers = self._tokenize(raw)
        caps_stopped, lowers_stopped = self._removeStopwords(caps, lowers)
        lowers_stemmed = [self.porter.stem(word) for word in lowers_stopped]
        return caps_stopped + lowers_stemmed
    
prepper = CarsDataPrepper()<jupyter_output><empty_output><jupyter_text>Let's test out our data prepper:<jupyter_code>x = df.iloc[1012].body
x
prepper.prep(x)
# note, prepper.prep is a function (method)
prepper.prep<jupyter_output><empty_output><jupyter_text>Let's apply the `prep` method to the entire DataFrame, processing all of our comments in one go.  As you can see, we get back a Series:<jupyter_code>type(df.body.apply(prepper.prep))<jupyter_output>/Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages/ipykernel_launcher.py:21: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
/Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages/ipykernel_launcher.py:26: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal<jupyter_text>We'll name this Series 'words' and tack it on to our DataFrame<jupyter_code>df['words'] = df.body.apply(prepper.prep)
df.head()<jupyter_output>/Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages/ipykernel_launcher.py:21: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
/Users/peter/Virtualenvs/standard2/lib/python2.7/site-packages/ipykernel_launcher.py:26: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal<jupyter_text>Some basic word analysisNow that our raw Reddit comments are processed, we can do some very basic but meaningful word counting.  This is a way of characterizing our data set.<jupyter_code>df_ford = df[df.subreddit=='Ford'].copy()
df_toyota = df[df.subreddit=='Toyota'].copy()
listOfordWords = [w for comment in df_ford.words for w in comment]
listOfordWords[0:10]

len(listOfordWords)<jupyter_output><empty_output><jupyter_text>Use Python's set data type to see size of the vocabulary:<jupyter_code>len(set(listOfordWords))<jupyter_output><empty_output><jupyter_text>It's possible to 'histogram' words using NLTK's frequency distribution:<jupyter_code>from nltk import FreqDist
fdist_ford = FreqDist(listOfordWords)
fdist_ford.most_common(30) # we can add to the stopwords at this point
listOtoyWords = [w for comment in df_toyota.words for w in comment]
fdist_toyota = FreqDist(listOtoyWords)

fdist_toyota.most_common(20)<jupyter_output><empty_output><jupyter_text>Digrams A digram or bigram means two words which occur together.  NLTK has some functions to find and count them.<jupyter_code>from nltk import collocations
c_ford = collocations.BigramCollocationFinder.from_words(listOfordWords)
c_ford.ngram_fd.most_common(20)
c_toyota = collocations.BigramCollocationFinder.from_words(listOtoyWords)
c_toyota.ngram_fd.most_common(20)<jupyter_output><empty_output><jupyter_text>Already, we can see sense some differences between Ford and Toyota topics.  Now, on to some more sophisticated analyses.  Clustering and TF-IDFTerm-frequency/inverse document frequency (TF-IDF): a measure of word importance in a particular document, relative to a document collection.  0 = appears in all documents. Higher = more important.We'll use the gensim library to weight our terms with TFIDF, construct a "bag of words" or "sparse" vector model of Reddit comments, then cluster the comments.  We'll be able to see different groups of comments on similar subjects.<jupyter_code>from gensim import corpora, models
import gensim<jupyter_output><empty_output><jupyter_text>First, construct a dictionary of words.  We'll use our already-prepared data to make the dictionary.  It is a list of ID numbers and words.<jupyter_code>dictionary = corpora.Dictionary(df_ford.words)
dictionary.items()<jupyter_output><empty_output><jupyter_text>Notice, it has the expected length:<jupyter_code>len(dictionary.items())<jupyter_output><empty_output><jupyter_text>Next, we'll need to build a numerical or vector representation of our Reddit comments. We start by replacing each word in a comment by its ID number, followed by the number of times it occurs in the comment.  The dictionary objectwe created has a method for this called `doc2bow`.<jupyter_code>x = df_ford.words[0]
print x
transformed_x = dictionary.doc2bow(x)
print transformed_x<jupyter_output>[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1), (13, 1), (14, 1), (15, 1), (16, 1)]<jupyter_text>Do this for *all* the comments.  A collection of documents (Reddit comments in our case) is called a corpus.<jupyter_code>corpus = [dictionary.doc2bow(comment) for comment in df_ford.words]
corpus<jupyter_output><empty_output><jupyter_text>Now we use TFIDF to weight the words.  As it stands, the number next to each word is the number of times it occurs in the comment.  All we are doing is changing this number to measure the relative importance of the word in the corpus.<jupyter_code>tfidf = models.TfidfModel(corpus) 
type(tfidf)
weighted_corpus = tfidf[corpus]<jupyter_output><empty_output><jupyter_text>Let's have a look, comparing the original word list, the encoded word list, and the tfidf-weighted word list:<jupyter_code>print df_ford.words[0]
print  "------"
print corpus[0]
print "------"
print weighted_corpus[0]<jupyter_output>[u'Yes', u'Thanks', u'bit', u'definit', u'guess', u'assum', u"I'll", u'never', u'one', u"can't", u'fulli', u'appreci', u'super', u'car', u'allur', u'lesson', u'man']
------
[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1), (13, 1), (14, 1), (15, 1), (16, 1)]
------
[(0, 0.18692854984607543), (1, 0.2101542255948645), (2, 0.3834053890178462), (3, 0.20461577431262806), (4, 0.29880260006843207), (5, 0.10541880994605857), (6, 0.19082415903929253), (7, 0.22862604176604798), (8, 0.11862957214541292), (9, 0.18692854984607543), (10, 0.267904613402525), (11, 0.2811153756018794), (12, 0.3834053890178462), (13, 0.2527821253138988), (14, 0.23447280283728636), (15, 0.19503173750623828), (16, 0.19803636488914297)]<jupyter_text>Finally, we'll fill in zeros for all the unused words in the vocabulary.  That will make each comment vector the size of the vocabulary, or,<jupyter_code>weighted_corpus[52]
len(dictionary)<jupyter_output><empty_output><jupyter_text>We can put our documents together in a big numerical matrix.  In this representation, the documents are columns:<jupyter_code>tfidf_matrix = gensim.matutils.corpus2dense(tfidf[corpus], len(dictionary)) # documents are columns
tfidf_matrix<jupyter_output><empty_output><jupyter_text>You will recognize the TFIDF weightings from our sample comment in the first (0th) column:<jupyter_code>tfidf_matrix[:,0]  # bag-of words representation<jupyter_output><empty_output><jupyter_text>Document similarity - Bag of Words model<jupyter_code>from gensim import corpora, models, similarities
import numpy as np
dictionary = corpora.Dictionary(df_ford.words)
corpus = [dictionary.doc2bow(doc) for doc in df_ford.words]
tfidf = models.TfidfModel(corpus=corpus, dictionary=dictionary) 
index = similarities.SparseMatrixSimilarity(tfidf[corpus], num_features=len(dictionary))<jupyter_output><empty_output><jupyter_text>"Seed" document, arbitrarily chosen to be doc 5:<jupyter_code>df_ford.words.iloc[5], df_ford.body.iloc[5]
doc0 = tfidf[corpus[5]]
sim0 = index[doc0]  # similarity of doc #0 to all other docs
print doc0
print 
print sim0
similars = np.argsort(-sim0)[1:10] 
# top indices not counting itself.  
#Time taken can be reduced using bottleneck.partsort
similars
df_ford.iloc[906].body
 print df_ford.iloc[[ 906,   50, 1003, 1071,  547,  550,  187,   14, 1083]]<jupyter_output>subreddit                                               body  score  \
906       Ford  Literally across the street from where I sit n...      1   
50        Ford  Modern escape isn't even actually an evolution...      5   
1003      Ford  Looking at it closer, I'm thinking it may be a...      1   
1071      Ford    I'd rather run errands in that than the Slocus       0   
547       Ford  Thanks everyone. We called the closer dealer a...      1   
550       Ford  Buy the parts. Look for same in the engine. R+...      4   
187       Ford  Looking clean.\n\n Are blue cars like a theme ...      6   
14        Ford  Aren't Kugas and Escapes the same thing?  Why ...      2   
1083      Ford  The car on the trailer? The one sitting next t...      1   

                                                  words  
906          [Literally, across, street, sit, sad, day]  
50    [Modern, Escape, European, Kuga, Kuga, Escape,...  
1003        [Looking, ZD, Fairlane, closer, think, may]  
1071   [...]<jupyter_text>Latent Dirichlet Allocation (LDA)"Latent Dirichlet allocation (LDA) is a topic model that generates topics based on word frequency from a set of documents. LDA is particularly useful for finding reasonably accurate mixtures of topics within a given document set."  Very nice information here: https://rstudio-pubs-static.s3.amazonaws.com/79360_850b2a69980c4488b1db95987a24867a.htmlNote there is another completely different LDA-- "Linear Discriminant Analysis"-- which is a classification algorithm.<jupyter_code>from gensim import corpora, models
import gensim
dictionary = corpora.Dictionary(df_ford.words)
# encode all comments = a corpus.  "Bag of Words"
corpus = [dictionary.doc2bow(comment) for comment in df_ford.words]
corpus_tfidf = tfidf[corpus]
ldamodel = gensim.models.ldamodel.LdaModel(corpus_tfidf, num_topics=10, id2word = dictionary, passes=10)
ldamodel.print_topics(num_topics=10, num_words=10)
corpus_lda = ldamodel[corpus_tfidf]
corpus_lda[0], df_ford.body[0]
# compare to Toyota
dictionary = corpora.Dictionary(df_toyota.words)
corpus = [dictionary.doc2bow(comment) for comment in df_toyota.words]
tfidf = models.TfidfModel(corpus=corpus, dictionary=dictionary) 
corpus_tfidf = tfidf[corpus]
ldamodel = gensim.models.ldamodel.LdaModel(corpus_tfidf, num_topics=10, id2word = dictionary, passes=20)
ldamodel.print_topics(num_topics=10, num_words=5)<jupyter_output><empty_output><jupyter_text>Visualization with the pyLDAvis package.  (It's also an R package)Reference: https://github.com/cpsievert/LDAvis<jupyter_code>import pyLDAvis, pyLDAvis.gensim
pyLDAvis.enable_notebook()
pyLDAvis.gensim.prepare(ldamodel, corpus_tfidf, dictionary)<jupyter_output><empty_output><jupyter_text>Latent Semantic IndexingLatent Semantic Indexing (LSI) is the equivalent of Principal Component Analysis (PCA) for documents.  Recall that PCA transformed numerical data vectors into a new set of vectors.  The new set of vectors concentrated the numerical variation in the first component.  Then, it concentrated the next level of variation in the second component, and so forth.  In our college data set, we were able to use only the first two transformed components (instead of all 4) and get the same results as using all 4 un-transformed (original) components.LSI tries to accomplish the same objective (dimensionality reduction) using two key ideas.1.  Word collocation.  Certain words tend to appear together in the data set('fast car', 'nice car').  Then, there are independent collocations in the data set ('need part', 'less fuel').  If you view the collocations as "dimensions", or, "topics", you can imagine specifying "how much" of each kind a document contains.2. Singular Value Decomposition (SVD).  This is a linear algebra technique which transforms a matrix into a new matrix of effective lower dimension.  It's the matrix analogue of fitting a line to a bunch of points.  LSI is essentially applying SVD to a document-term matrix.<jupyter_code>dictionary = corpora.Dictionary(df_ford.words)
corpus = [dictionary.doc2bow(comment) for comment in df_ford.words]
tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
corpus_tfidf[0]
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=2) # initialize an LSI transformation
corpus_lsi = lsi[corpus_tfidf] # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi
lsi.print_topics(30)<jupyter_output><empty_output><jupyter_text>Now every document is represented as a two dimensional vector.  Let's look at document 0:<jupyter_code>corpus_lsi[0]<jupyter_output><empty_output><jupyter_text>Let's make a plot<jupyter_code>data = [{'x':corpus_lsi[i][0][1], 'y':corpus_lsi[i][1][1]} for i in range(len(corpus_lsi)) if corpus_lsi[i] != []]
data
lsi_df = pandas.DataFrame(data)
lsi_df.plot.scatter(x='x', y='y')
corpus = [dictionary.doc2bow(comment) for comment in df_toyota.words]
tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=4) # initialize an LSI transformation
corpus_lsi = lsi[corpus_tfidf] # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi
lsi.print_topics(4)
for doc in corpus_lsi[0:4]: 
    print (doc)<jupyter_output>[(0, -0.065564302596982288), (1, 0.018365625681761904), (2, -0.0053696441030800235), (3, 0.055195557399478581)]
[(0, -0.039459212347970098), (1, -0.0019199817680356033), (2, 0.013970633425867622), (3, -0.0015775667385075293)]
[(0, -0.18099211850198388), (1, 0.015058364811425492), (2, -0.072614791114328087), (3, 0.17151735259700107)]
[(0, -0.12062792848167209), (1, -0.011089514165082397), (2, 0.036114455195307378), (3, 0.021770981450609253)]<jupyter_text>Gensim data structures summary1. **`dictionary`**: key/map associating numerical ID with a single word.  Create with constructor passing all words2. _**corpus**_: collection of documents, with words mapped to (id, freq) using `dictionary.doc2bow`3. **`models.TfidfModel`**: contructed with _**corpus**_ and **`dictionary`**, a normalized TF-IDF weighting function. once you `tfidf = models.TfidfModel(corpus=corpus, dictionary=dictionary)`, then you can `tfidf[corpus[i]]` to get the [ (id, tfidf), ..., (id, tfidf) ] rep of the document.  You can also input custom local - global functions to get something other than tfidf.4. _**index**_: a query engine to find the similarity of doc x with the rest of the corpus, or, all combinations (similarity matrix).  Use `similarities.SparseMatrixSimilarity(tfidf[corpus], num_features=len(dictionary))` to create an in-memory, bag-of-words similarity engine.  Query doc n with `index[tfidf[corpus[n]]]`, returns an array of similarities.  You can also query a collection of docs, in which case you'll get a matrix back.  Aside: for LDA, `num_features` can be much smaller.   From http://arxiv.org/pdf/1405.4053v2.pdfImplemented in gensimMany machine learning algorithms require theinput to be represented as a fixed-length featurevector. When it comes to texts, one of the mostcommon fixed-length features is bag-of-words.Despite their popularity, bag-of-words featureshave two major weaknesses: they lose the orderingof the words and they also ignore semanticsof the words. For example, “powerful,” “strong”and “Paris” are equally distant. In this paper, wepropose Paragraph Vector, an unsupervised algorithmthat learns fixed-length feature representationsfrom variable-length pieces of texts, such assentences, paragraphs, and documents. Our algorithmrepresents each document by a dense vectorwhich is trained to predict words in the document.Its construction gives our algorithm thepotential to overcome the weaknesses of bag-ofwordsmodels. Empirical results show that ParagraphVectors outperform bag-of-words modelsas well as other techniques for text representations.Finally, we achieve new state-of-the-art resultson several text classification and sentimentanalysis tasks.http://rare-technologies.com/doc2vec-tutorial/The tutorial is slightly outdated.The input to Doc2Vec is an iterator of TaggedDocument objects (it was formerly LabeledSentence objects).  From the source code:_A single document, made up of `words` (a list of unicode string tokens)and `tags` (a list of tokens). Tags may be one or more unicode stringtokens, but typical practice (which will also be most memory-efficient) isfor the tags list to include a unique integer id as the only tag._To construct a TaggedDocument use: `td = TaggedDocument({'words': ['i'm','a','document'], 'tags': id})`The algorithm then runs through the sentences iterator twice: once to build the vocab, and once to train the model on the input data, learning a vector representation for each word and for each label in the dataset.Doc2Vec learns representations for words and labels simultaneously. If you wish to only learn representations for words, you can use the flag train_lbls=False in your Doc2Vec class. Similarly, if you only wish to learn representations for labels and leave the word representations fixed, the model also has the flag train_words=False.The usage for Doc2Vec is the same as for gensim’s Word2Vec.<jupyter_code>from gensim.models import doc2vec

class DocIterator(object):
    """ supply a list of dictionaries of doc and unique id: [ {'doc': list of words, 'id': id}, ... ]"""
    def __init__(self, doc_dict):
        self.doc_dict = doc_dict
    def __iter__(self):
        for item in self.doc_dict:
            yield doc2vec.TaggedDocument(words = item['doc'], tags=item['id'])
x = zip(range(len(df_ford.words)), df_ford.words)
d2v_input = [{'id':[a], 'doc':b} for a,b in x]
d2v_input[0:3]
di = DocIterator(d2v_input)
model = doc2vec.Doc2Vec(size=40, window=5, min_count=5, workers=4, alpha=0.025, min_alpha=0.025)
model.build_vocab(di)
model.corpus_count, model.vector_size
for epoch in range(10):
    model.train(di)
    model.alpha -= 0.002 # decrease the learning rate
    model.min_alpha = model.alpha # fix the learning rate, no deca
    model.train(di)
model.total_train_time
model.docvecs[0]  # vector of doc 0
model.docvecs.most_similar(0)
df_ford.body[0]
print df_ford.body[758]
print df_ford.body[1037]
df_ford.body[1407]<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10625937031484257, 'fraction_numerical': 0.03548225887056472, 'mean_word_length': 5.119457186544342, 'pattern_counts': {'":': 0, '<': 177, '<?xml version=': 0, '>': 186, 'https://': 9, 'lorem ipsum': 0, 'www.': 9, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 18, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '45', 'dir': 'jupyter-structured-clean-dedup', 'id': '324471', 'license': 'MIT', 'n_tokens_mistral': 11254, 'n_tokens_neox': 10342, 'n_words': 3256, 'path': 'nlp-reddit.ipynb', 'repo_name': 'peterfig/intro2ml', 'size': '32026'}