<jupyter_start><jupyter_text>Document retrieval from wikipedia data  Fire up GraphLab Create<jupyter_code>import graphlab<jupyter_output>A newer version of GraphLab Create (v1.8.5) is available! Your current version is v1.8.2.

You can use pip to upgrade the graphlab-create package. For more information see https://dato.com/products/create/upgrade.<jupyter_text>Load some text data - from wikipedia, pages on people<jupyter_code>people = graphlab.SFrame('people_wiki.gl/')<jupyter_output>[INFO] GraphLab Create v1.8.2 started. Logging: /tmp/graphlab_server_1461192709.log<jupyter_text>Data contains:  link to wikipedia article, name of person, text of article.<jupyter_code>people.head()
len(people)<jupyter_output><empty_output><jupyter_text>Explore the dataset and checkout the text it contains Exploring the entry for president Obama<jupyter_code>obama = people[people['name'] == '<NAME>']
obama
obama['text']<jupyter_output><empty_output><jupyter_text>Exploring the entry for actor <NAME><jupyter_code>clooney = people[people['name'] == '<NAME>']
clooney['text']<jupyter_output><empty_output><jupyter_text>Get the word counts for Obama article<jupyter_code>obama['word_count'] = graphlab.text_analytics.count_words(obama['text'])
print obama['word_count']<jupyter_output>[{'operations': 1, 'represent': 1, 'office': 2, 'unemployment': 1, 'is': 2, 'doddfrank': 1, 'over': 1, 'unconstitutional': 1, 'domestic': 2, 'named': 1, 'ending': 1, 'ended': 1, 'proposition': 1, 'seats': 1, 'graduate': 1, 'worked': 1, 'before': 1, 'death': 1, '20': 2, 'taxpayer': 1, 'inaugurated': 1, 'obamacare': 1, 'civil': 1, 'mccain': 1, 'to': 14, '4': 1, 'policy': 2, '8': 1, 'has': 4, '2011': 3, '2010': 2, '2013': 1, '2012': 1, 'bin': 1, 'then': 1, 'his': 11, 'march': 1, 'gains': 1, 'cuba': 1, 'californias': 1, '1992': 1, 'new': 1, 'not': 1, 'during': 2, 'years': 1, 'continued': 1, 'presidential': 2, 'husen': 1, 'osama': 1, 'term': 3, 'equality': 1, 'prize': 1, 'lost': 1, 'stimulus': 1, 'january': 3, 'university': 2, 'rights': 1, 'gun': 1, 'republican': 2, 'rodham': 1, 'troop': 1, 'withdrawal': 1, 'involvement': 3, 'response': 3, 'where': 1, 'referred': 1, 'affordable': 1, 'attorney': 1, 'school': 3, 'senate': 3, 'house': 2, 'national': 2, 'creation': 1, 'related': 1, 'hawaii': 1,[...]<jupyter_text>Sort the word counts for the Obama article  Turning dictonary of word counts into a table<jupyter_code>obama_word_count_table = obama[['word_count']].stack('word_count', new_column_name = ['word','count'])<jupyter_output><empty_output><jupyter_text>Sorting the word counts to show most common words at the top<jupyter_code>obama_word_count_table.head()
obama_word_count_table.sort('count',ascending=False)<jupyter_output><empty_output><jupyter_text>Most common words include uninformative words like "the", "in", "and",...  Compute TF-IDF for the corpus To give more weight to informative words, we weigh them by their TF-IDF scores.<jupyter_code>people['word_count'] = graphlab.text_analytics.count_words(people['text'])
people.head()
tfidf = graphlab.text_analytics.tf_idf(people['word_count'])

# Earlier versions of GraphLab Create returned an SFrame rather than a single SArray
# This notebook was created using Graphlab Create version 1.7.1
if graphlab.version <= '1.6.1':
    tfidf = tfidf['docs']

tfidf
people['tfidf'] = tfidf<jupyter_output><empty_output><jupyter_text>Examine the TF-IDF for the Obama article<jupyter_code>obama = people[people['name'] == '<NAME>']
obama[['tfidf']].stack('tfidf',new_column_name=['word','tfidf']).sort('tfidf',ascending=False)<jupyter_output><empty_output><jupyter_text>Words with highest TF-IDF are much more informative.  Manually compute distances between a few peopleLet's manually compare the distances between the articles for a few famous people.<jupyter_code>clinton = people[people['name'] == '<NAME>']
beckham = people[people['name'] == '<NAME>']<jupyter_output><empty_output><jupyter_text>Is Obama closer to Clinton than to Beckham?We will use cosine distance, which is given by(1-cosine_similarity) and find that the article about president Obama is closer to the one about former president Clinton than that of footballer <NAME>.<jupyter_code>graphlab.distances.cosine(obama['tfidf'][0],clinton['tfidf'][0])
graphlab.distances.cosine(obama['tfidf'][0],beckham['tfidf'][0])<jupyter_output><empty_output><jupyter_text>Build a nearest neighbor model for document retrievalWe now create a nearest-neighbors model and apply it to document retrieval.<jupyter_code>knn_model = graphlab.nearest_neighbors.create(people,features=['tfidf'],label='name')<jupyter_output><empty_output><jupyter_text>Applying the nearest-neighbors model for retrieval  Who is closest to Obama?<jupyter_code>knn_model.query(obama)<jupyter_output><empty_output><jupyter_text>As we can see, president Obama's article is closest to the one about his vice-president Biden, and those of other politicians.    Other examples of document retrieval<jupyter_code>swift = people[people['name'] == '<NAME>']
knn_model.query(swift)
jolie = people[people['name'] == '<NAME>']
knn_model.query(jolie)
arnold = people[people['name'] == '<NAME>']
knn_model.query(arnold)<jupyter_output><empty_output><jupyter_text>Assignment 1: Compare top words according to word counts to TF-IDF<jupyter_code>elton = people[people['name'] == '<NAME>']
elton
elton_word_count_table = elton[['word_count']].stack('word_count', new_column_name = ['word','count'])
elton_word_count_table.sort('count',ascending = False)
elton[['tfidf']].stack('tfidf',new_column_name=['word','tfidf']).sort('tfidf',ascending=False)<jupyter_output><empty_output><jupyter_text>Assignment 2: Measuring distance<jupyter_code>victoria = people[people['name'] == '<NAME>']
Paul = people[people['name'] == '<NAME>']
graphlab.distances.cosine(elton['tfidf'][0],victoria['tfidf'][0])
graphlab.distances.cosine(elton['tfidf'][0],Paul['tfidf'][0])
graphlab.distances.cosine(elton['tfidf'][0],victoria['tfidf'][0]) - graphlab.distances.cosine(elton['tfidf'][0],Paul['tfidf'][0])<jupyter_output><empty_output><jupyter_text>Assignment 3: Building nearest neighbors models with different input features and setting the distance metric<jupyter_code>wordcount_model = graphlab.nearest_neighbors.create(people,features=['word_count'],label='name')
wordcount_model.query(elton)
knn_model.query(elton)
answer_list = ['<NAME>','<NAME>','<NAME>','<NAME>']
def n(name):
    return people[people['name'] == name]
for name in answer_list:
    print graphlab.distances.cosine(elton['word_count'][0],n(name)['word_count'][0])
answer_list2 = ['<NAME>','<NAME>','<NAME>','<NAME>']
def n(name):
    return people[people['name'] == name]
for name in answer_list2:
    print graphlab.distances.cosine(elton['tfidf'][0],n(name)['tfidf'][0])
wordcount_model.query(victoria)
answer_list = ['<NAME>','<NAME>','<NAME>','<NAME> (artist)']
def n(name):
    return people[people['name'] == name]
for name in answer_list:
    print graphlab.distances.cosine(victoria['word_count'][0],n(name)['word_count'][0])
knn_model.query(victoria)
answer_list2 = ['<NAME>','<NAME>','<NAME>','<NAME>']
def n(name):
    return people[people['name'] == name]
for name in answer_list2:
    print graphlab.distances.cosine(victoria['tfidf'][0],n(name)['tfidf'][0])<jupyter_output>0.809585523409
0.819826422919
0.548169610263
0.975395042344
{'fraction_non_alphanumeric': 0.16843537414965987, 'fraction_numerical': 0.02802721088435374, 'mean_word_length': 8.293299620733249, 'pattern_counts': {'":': 0, '<': 99, '<?xml version=': 0, '>': 98, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '18', 'dir': 'jupyter-structured-clean-dedup', 'id': '169143', 'license': 'MIT', 'n_tokens_mistral': 2792, 'n_tokens_neox': 2622, 'n_words': 590, 'path': '1. Machine-Learning-Foundations/document-retrieval/document-retrieval.ipynb', 'repo_name': 'anirudhbatra/ml-coursera-uwash', 'size': '7554'}