<jupyter_start><jupyter_text>Assignment 1 **Handin in Peergrade**: *Wednesday*, October 3, 23:59**Peergrading deadline**: *Wednesday*, October 10, 23:59**Peergrading feedback deadline**: *Sunday*, October 14, 23:59**Peergrade link**: https://app.peergrade.io/join **Peergrade code**: 3TYFF7 [**Questions**](https://github.com/ulfaslak/computational_analysis_of_big_data_2018_fall/issues) **/** [**Feedback**](https://ulfaslak.com/vent)  Problems  Week 1 >**Ex. 1.2.2**: Working with JSON files>1. Use [`requests`](https://www.google.dk/search?q=python+requests+get+json&gws_rd=cr&ei=M5OdWaewD8Ti6AS54J24Bg), or another Python module, to store **[this data](https://www.reddit.com/r/gameofthrones/.json)** in a new variable `data`. Store it as a `dict` type object (you can print its type by running `type(data)`).<jupyter_code>import requests

# header used to identify requester
headers = {'user-agent': 'my-app/0.0.1'}

r = requests.get("https://www.reddit.com/r/gameofthrones/.json", headers=headers)
data = r.json()

print(type(data))<jupyter_output><type 'dict'><jupyter_text>>**Ex 1.2.5**: Write two `for` loops (or list comprehensions for extra street credits) which:>1. Counts the number of spoilers.>2. Only prints headlines that aren't spoilers.<jupyter_code>print "Number of spoilers: " + str(len([elem['data']['spoiler'] for elem in data['data']['children'] if elem['data']['spoiler'] == True]))
print "Headlines that aren't spoilers: "
x = [elem['data']['title'] for elem in data['data']['children'] if elem['data']['spoiler'] == False]
for elem in x:
    print elem<jupyter_output>Number of spoilers: 8
Headlines that aren't spoilers: 
[SPOILERS] Stark/Targaryen jack-o-lantern with dragon projection in a craft pumpkin!
[NO SPOILERS] Just got this in at work thought you folks might like it.
[NO SPOILERS] Since these seem popular, here is my Targaryen pumpkin from 2 years ago (first attempt - ~5 hours work, or so)
[NO SPOILERS] Check Out This Drogon Model I Made for Someone
[NO SPOILERS] Why are Jon and Daenerys such controversial characters?
[NO SPOILERS] <NAME> is going to be giving a talk in London on October 13th to chat about all things Game of Thrones
[No Spoilers] Khalessi and The White Walker King image
[NO SPOILERS] Arya’s sword 🗡
[SPOILERS] Not a fake or a joke: this image is from an HBO press release announcing the new "White Walker: Limited Edition Blended Scotch Whiskey" by <NAME>! :o
[NO SPOILERS] Got this in the mail today
[SPOILERS] <NAME>'s Portrait of House Greyjoy
[Spoilers] A missing Game of Thrones character is back for[...]<jupyter_text>Week 2 >**Ex 2.1.5**: Take another list `b = list("ofcourseistillloveyou")` and1. get the `set` of characters that exist in both `a` and `b` (intersection),2. get the `set` of characters that exist in either `a` or `b` (union), and3. compute the [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index) between the distinct elements in `a` and `b`.>*Hint: use the `set` function to get a `set`-type object of distinct elements from a list*<jupyter_code>from __future__ import division

a = list("justreadtheinstructions")
b = list("ofcourseistillloveyou")

a1 = set(a)
b1 = set(b)

print a1.intersection(b1)
print a1.union(b1)
print len(a1.intersection(b1))/len(a1.union(b1))<jupyter_output>set(['c', 'e', 'i', 'o', 's', 'r', 'u', 't'])
set(['a', 'c', 'e', 'd', 'f', 'i', 'h', 'j', 'l', 'o', 'n', 's', 'r', 'u', 't', 'v', 'y'])
0.470588235294<jupyter_text>>**Ex. 2.2.4**: Looking at the scatter plots there appears to be some unevenness in the number of comments and upvotes that different posts receive.1. Plot the distributions of `x` for "gameofthrones" and "news" as histograms, side by side. My figure looks like [this](http://ulfaslak.com/computational_analysis_of_big_data/exer_figures/example_2.2c.png). It does not have to match exactly, but if there are large differences you should comment on them.2. What do these distributions say about how people comment on Reddit?<jupyter_code>import requests as rq
from collections import Counter
import seaborn as sns
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

# defines a function get_x_y with arguments subreddit, N, and count (whose default is 25)
def get_x_y(subreddit, N, count=25):
    
    # defines a function _get_data with arguments subreddit, count, and after
    def _get_data(subreddit, count, after):
        # adds in arguments to url string
        url = "https://www.reddit.com/r/%s/.json?count=%d&after=%s" % (subreddit, count, after)
        # gets data from url as json
        data = rq.get(url, headers = {'User-agent': 'sneakybot'}).json()
        # prints message detailing how many posts were retrieved from which page
        print "Retrieved %d posts from page %s" % (count, after)
        return data
    
    # defines a variable after to be an empty string
    after = ""
    
    # defines two empty lists, x and y
    x, y = [], []
    # loops through numbers between 0 and N/count
    for n in range(int(N/count)):
        # calls _get_data
        data = _get_data(subreddit, count, after)
        # loops through data
        for d in data['data']['children']:
            # appends the number of comments to x
            x.append(d['data']['num_comments'])
            # appends the score to y
            y.append(d['data']['score'])
        # updates after
        after = data['data']['after']
    # returns a list of number of comments and a list of scores
    return x, y

# calls get_x_y
x, y = get_x_y("gameofthrones", 500, count=25)
z, w = get_x_y("news", 500, count=25)

plt.subplot(1, 3, 1)
plt.hist(x, 10, color="green")
plt.title("Game of Thrones")
plt.xlabel("Number of Comments")
plt.ylabel("Bin Count")
plt.subplot(1, 3, 3)
plt.hist(z, 10)
plt.title("News")
plt.xlabel("Number of Comments")
plt.ylabel("Bin Count")
plt.show()<jupyter_output>Retrieved 25 posts from page 
Retrieved 25 posts from page t3_9kghm3
Retrieved 25 posts from page t3_9k6um0
Retrieved 25 posts from page t3_9juagf
Retrieved 25 posts from page t3_9jh2i5
Retrieved 25 posts from page t3_9iyt33
Retrieved 25 posts from page t3_9idesi
Retrieved 25 posts from page t3_9i8xn1
Retrieved 25 posts from page t3_9i52lb
Retrieved 25 posts from page t3_9hxwdn
Retrieved 25 posts from page t3_9he4cp
Retrieved 25 posts from page t3_9h19hf
Retrieved 25 posts from page t3_9gr1zw
Retrieved 25 posts from page t3_9gq55d
Retrieved 25 posts from page t3_9g4e6l
Retrieved 25 posts from page t3_9g0d3d
Retrieved 25 posts from page t3_9fp4y5
Retrieved 25 posts from page t3_9f8lwj
Retrieved 25 posts from page t3_9f093v
Retrieved 25 posts from page t3_9eerak
Retrieved 25 posts from page 
Retrieved 25 posts from page t3_9kfp6w
Retrieved 25 posts from page t3_9koctj
Retrieved 25 posts from page t3_9k5kfs
Retrieved 25 posts from page t3_9k5s2z
Retrieved 25 posts from page t3_9jrvus
Retr[...]<jupyter_text>>**Ex. 2.3.5**: Create a $5 \times 5$ array `X` (a matrix) with random numbers and $5 \times 1$ array `a` (a tall vector) with 5 random numbers.1. Compute the matrix-vector dot product between these two. Use `numpy`'s `dot` method. 2. What happens if you just use `*` instead of `np.dot`? Explain.<jupyter_code>x = np.random.rand(5, 5)
print x
y = np.random.rand(5, 1)
print y
print np.dot(x, y)
print x * y<jupyter_output>[[0.95552948 0.27888393 0.36959806 0.59986504 0.82665336]
 [0.20345199 0.92693826 0.88298096 0.77894913 0.47689831]
 [0.96515232 0.20872761 0.06487981 0.39709787 0.02736064]
 [0.41282076 0.80071899 0.36891262 0.71425428 0.02154374]
 [0.85810781 0.82769109 0.54837383 0.25016401 0.50552237]]
[[0.64367226]
 [0.6542756 ]
 [0.91325867]
 [0.78941586]
 [0.91680356]]
[[2.36647512]
 [2.59595636]
 [1.15561889]
 [1.71011985]
 [2.25563358]]
[[0.61504782 0.17950985 0.23790002 0.38611649 0.53209384]
 [0.13311367 0.60647308 0.5777129  0.50964741 0.31202292]
 [0.88143372 0.1906223  0.05925204 0.36265307 0.02498734]
 [0.32588725 0.63210027 0.29122547 0.56384366 0.01700697]
 [0.7867163  0.75883014 0.50275108 0.22935126 0.46346471]]<jupyter_text>Using * multiplies each row of x by the value in the corresponding row in y. >**Ex. 2.4.1**: Take a vector `a = [1, 3, 2, 5, 3, 1, 5, 1, 9000]`:1. Compute the mean of `a` using `numpy`.2. How is median defined? Compute the median of `a` using `numpy`.3. For `a`, why might it make sense to take the median more seriously than the mean?<jupyter_code>a = [1, 3, 2, 5, 3, 1, 5, 1, 9000]
print np.mean(a)
print np.median(a)<jupyter_output>1002.3333333333334
3.0<jupyter_text>The median is the middle-most value or the average of the two middle-most values.Since a has an outlier (9000), the mean is extremely skewed and it makes sense to take the median more seriously than the mean.  Week 3 >**Ex. 3.1.1**: From the Wikipedia API, get a list of all Marvel superheroes and another list of all Marvel supervillains. Use 'Category:Marvel_Comics_supervillains' and 'Category:Marvel_Comics_superheroes' to get the characters in each category.1. How many superheroes are there? How many supervillains?2. How many characters are both heroes and villains? What is the Jaccard similarity between the two groups?>*Hint: Google something like "get list all pages in category wikimedia api" if you're struggling with the query.*<jupyter_code>def getCategorySet(category):
    baseQuery = "https://en.wikipedia.org/w/api.php?format=json&cmtitle=Category:%s&action=query&list=categorymembers&cmlimit=500" % category
    response = rq.get(baseQuery).json()
    items = set()
    while "continue" in response:
        for item in response["query"]["categorymembers"]:
            items.add(item["title"])
        query = baseQuery + "&cmcontinue=" + response["continue"]["cmcontinue"]
        response = rq.get(query).json()
    for item in response["query"]["categorymembers"]:
        items.add(item["title"])
    return items

heroes = getCategorySet("Marvel_Comics_superheroes")
villains = getCategorySet("Marvel_Comics_supervillains")

print "Number of Superheroes: %d" % len(heroes)
print "Number of Supervillains: %d" % len(villains)

ambiguous = heroes.intersection(villains)

print "Number of Characters that are both Superheroes and Supervillains: %d" % len(ambiguous)

similarity = float(len(ambiguous))/len(heroes.union(villains))
print "Jaccard similarity: %f" % similarity<jupyter_output>Number of Superheroes: 941
Number of Supervillains: 1222
Number of Characters that are both Superheroes and Supervillains: 167
Jaccard similarity: 0.083667<jupyter_text>>**Ex. 3.2.1**: Extract the length of the page of each character, and plot the distribution of this variable for each class (heroes/villains/ambiguous). Can you say anything about the popularity of characters in the Marvel universe based on your visualization?>*Hint: The simplest thing is to make a probability mass function, i.e. a normalized histogram. Use `plt.hist` on a list of page lengths, with the argument `normed=True`. Other distribution plots are fine too, though.*<jupyter_code>import os

def getPageLengths(category):
    pageLengths = []
    path = "data/" + category + "/"
    for file in os.listdir(path):
        with open(path + file, 'r') as f:
            pageLengths.append(len(f.read()))
    return pageLengths
ambiguousPageLengths = getPageLengths("ambiguous")
heroPageLengths = getPageLengths("heroes")
villainPageLengths = getPageLengths("villains")

plt.hist(ambiguousPageLengths, density=True)
plt.xlabel("Page Length")
plt.ylabel("Normalized Bin Count")
plt.title("Ambiguous")
plt.show()

plt.hist(heroPageLengths, density=True)
plt.xlabel("Page Length")
plt.ylabel("Normalized Bin Count")
plt.title("Heroes")
plt.show()

plt.hist(villainPageLengths, density=True)
plt.xlabel("Page Length")
plt.ylabel("Normalized Bin Count")
plt.title("Villains")
plt.show()<jupyter_output><empty_output><jupyter_text>The page lengths of Wikipedia articles associated with superheroes seems to be longer than the page lengths of articles associated with supervillains and characters that are both heroes and villains. This leads us to believe that heroes may be more popular than the other two categories. >**Ex. 3.2.2**: Find the 10 characters from each class with the longest Wikipedia pages. Visualize their page lengths with bar charts. Comment on the result.<jupyter_code>import operator

def getTopTenPageLength(category):
    pageLengths = dict()
    path = "data/" + category + "/"
    for file in os.listdir(path):
        with open(path + file, 'r') as f:
            pageLengths[file.split(".txt")[0]] = len(f.read())
    sortedPageLengths = sorted(pageLengths.items(), key=operator.itemgetter(1), reverse=True)
    return ([x[0] for x in sortedPageLengths[0:10]], [x[1] for x in sortedPageLengths[0:10]])

ambiguousCharacters, ambiguousPageLengths = getTopTenPageLength("ambiguous")
heroCharacters, heroPageLengths = getTopTenPageLength("heroes")
villainCharacters, villainPageLengths = getTopTenPageLength("villains")

plt.barh(ambiguousCharacters, ambiguousPageLengths)
plt.ylabel("Character")
plt.xlabel("Page Length")
plt.title("Ambiguous")
plt.show()

plt.barh(heroCharacters, heroPageLengths)
plt.ylabel("Character")
plt.xlabel("Page Length")
plt.title("Hero")
plt.show()

plt.barh(villainCharacters, villainPageLengths)
plt.ylabel("Character")
plt.xlabel("Page Length")
plt.title("Villain")
plt.show()<jupyter_output><empty_output><jupyter_text>These plots show that page length is positively correlated with popularity, since the characters in these bar plots are, for the most part, characters that we have heard of. >**Ex. 3.2.3**: We are interested in knowing if there is a time-trend in the debut of characters.* Extract into three lists, debut years of heroes, villains, and ambiguous characters.* Do all pages have a debut year? Do some have multiple? How do you handle these inconsistencies?* For each class, visualize the amount of characters introduced over time. You choose how you want to visualize this data, but please comment on your choice. Also comment on the outcome of your analysis.>*Hint: The debut year is given on the debut row in the info table of a character's Wiki-page. There are many ways that you can extract this variable. You should try to have a go at it yourself, but if you are short on time, you can use this horribly ugly regular expression code:*>*`re.findall(r"\d{4}\)", re.findall(r"debut.+?\n", markup_text)[0])[0][:-1]`*<jupyter_code>import re

def getDebutYears(category):
    debutYears = []
    path = "data/" + category + "/"
    for file in os.listdir(path):
        with open(path + file, 'r') as f:
            markup_text = f.read()
            try:
                debutYear = re.findall(r"\d{4}\)", re.findall(r"debut.+?\n", markup_text)[0])[0][:-1]
                debutYears.append(debutYear)
            except IndexError:
                pass
    return debutYears

ambiguousDebuts = getDebutYears("ambiguous")
heroDebuts = getDebutYears("heroes")
villainDebuts = getDebutYears("villains")

plt.hist(ambiguousDebuts)
plt.xlabel("Year")
plt.ylabel("Bin Count")
plt.title("Ambiguous")
plt.show()

plt.hist(heroDebuts)
plt.xlabel("Year")
plt.ylabel("Bin Count")
plt.title("Hero")
plt.show()

plt.hist(villainDebuts)
plt.xlabel("Year")
plt.ylabel("Bin Count")
plt.title("Villain")
plt.show()<jupyter_output><empty_output><jupyter_text>We chose to visualize our data using histograms because we were interested to see the number of characters debuting in different year bins. Our plots show that, over time, less and less characters are introduced.  Week 4 *If you had trouble boulding a team alliance matrix, use mine. You can load it as a pandas.DataFrame, with pd.read_csv('affiliations.csv', index_col=0).* ***Important:*** *The rightmost column is the target array.* > **Ex. 4.2.1**: Train a classifier on all of your data and test its accuracy.>* If your team alliance matrix is `X_ta` and your target array is `y_ta` you can do this by instantiating a model like:>        from sklearn.naive_bayes import BernoulliNB        model = BernoulliNB()        model.fit(X_ta, y_ta)   <--- This is the training/fitting/learning step        > The `BernoulliNB` is a version of the Naive Bayes classifier which associates certain features with labels and asks what the probability of a label for a data point is given its features. You are free to use any other classifier if you want. Popular ones are trees, random forests, support vector machines, feed forward neural networks, logistic regression, and the list goes on. With `sklearn`, they are just as easy to employ as the `BernoulliNB` classifier.>1. Test the accuracy of your model. You can use the `.predict` method on the `model` object to get predictions for a matrix of data points. Report the accuracy of your model on the same data that you trained the model on, alongside the baseline accuracy of a "dumb" model that only guesses for the majority class.>2. Report the precision, recall and F1 scores, with respect to the minority class (heroes). `sklearn` has implementations that you can use if you are short for time. Extra credit for doing it using only basic linear algebra operations with `numpy`, though.*Hint: If you loaded the alliances/affiliations matrix that I provided, you should make sure to seperate it into feature matrix, `X_ta`, and target array, `y_ta`, before you do anything else.*<jupyter_code>import pandas as pd
from sklearn.naive_bayes import BernoulliNB
from sklearn import metrics

alliances = pd.read_csv("data/affiliations.csv", index_col = 0)
roles = alliances['faction']
non_ambiguous_alliances = alliances.copy()
ambiguous_alliances = alliances.copy()
for index, row in non_ambiguous_alliances.iterrows():
    if(row['faction'] == 2): non_ambiguous_alliances.drop(index, inplace=True)
non_ambiguous_roles = non_ambiguous_alliances['faction']
alliances_copy = non_ambiguous_alliances.copy()
non_ambiguous_alliances.drop('faction', axis = 1, inplace = True)
alliances.drop('faction', axis = 1, inplace = True)

model = BernoulliNB()
model.fit(non_ambiguous_alliances, non_ambiguous_roles)

predictions = model.predict(non_ambiguous_alliances)

print "Accuracy: %f" % model.score(non_ambiguous_alliances, non_ambiguous_roles)
print metrics.classification_report(non_ambiguous_roles, predictions, ['1'])
print "1 value represents minority class (heroes)"<jupyter_output>Accuracy: 0.857741
             precision    recall  f1-score   support

          1       0.97      0.74      0.84       118

avg / total       0.97      0.74      0.84       118

1 value represents minority class (heroes)<jupyter_text>Part 1: Feature representationIn it's raw format, the data cannot be given to a machine learning algorithm. What we must do is extract features from the data and put them into a structured format. This is the same as what we did when we looked at a dog (the data) and extracted into a matrix whether it was fluffy, sad looking, etc. (the features). The feature we will extract here is **team alliances**.We can represent the team alliances of each character as a row in a matrix where each column corresponds to a particular team. That should look something like this (numbers are made up): > **Ex. 4.3.2**: Implement cross validation. The performance of a classifier is strongly dependent on the amount of data it is trained on. In Ex. 4.3.1 you train it on only half of the data and test it on the other half. If you rerun that code multiple times, with random 50/50 partitions, you are gonna see a lot of uncertainty in performance. Cross validation solves this problem by training on a larger subset of the data and testing on a smaller one, and taking the average performance over K-folds of this process.1. Implement cross validation over $K=10$ folds. For each fold you must record the training and test accuracies. In the end, visualize the distributions of test- and training accuracy as histograms in the same plot. It's important that you comment on the result.<jupyter_code>from matplotlib import pyplot as plt
%matplotlib inline

alliances_shuffle = alliances_copy.sample(frac=1)

i = 0
k = 1
rows = len(alliances_shuffle.index)
training_accuracies = []
testing_accuracies = []
while(i < rows and k <= 10):
    print "Fold Number: %d" % k
    
    testing = alliances_shuffle.iloc[i:i + int(rows/10)]
    testing_factions = testing[['faction']]
    
    if i != 0:
        training1 = alliances_shuffle.iloc[0:i]
        training2 = alliances_shuffle.iloc[i + int(rows/10) + 1:]
        training = pd.concat([training1, training2])
    else:
        training = alliances_shuffle.iloc[i + int(rows/10) + 1:]
    training_factions = training[['faction']]
    
    training.drop('faction', axis=1, inplace=True)
    testing.drop('faction', axis=1, inplace=True)
    
    model.fit(training, training_factions)
    
    training_score = model.score(training, training_factions)
    print "Training Score: %f" % training_score
    testing_score = model.score(testing, testing_factions)
    print "Testing Score: %f" % testing_score
    print "   "
    
    training_accuracies.append(training_score)
    testing_accuracies.append(testing_score)
    
    i += int(rows/10)
    k += 1
    
plt.hist([training_accuracies, testing_accuracies])
plt.xlabel("Accuracy")
plt.ylabel("Frequency")
plt.title("Heroes/Villains Accuracies")
plt.show()
print("Blue: training")
print("Orange: testing")<jupyter_output>Fold Number: 1
Training Score: 0.865116
Testing Score: 0.826087
   
Fold Number: 2
Training Score: 0.879070
Testing Score: 0.782609
   
Fold Number: 3<jupyter_text>The training accuracy is concentrated in a small, relatively high range, as is expected because the model is fit on the same data it is testing. The testing accuracy is more widely distributed because it is tested on different data than it is trained on. >**Ex. 4.4.1**: Let's put our classifier to use!* Retrain your model on all of your data.* Create a team alliance representation of the ambiguous characters* Use the model the estimate the probability that each character is a villain (let's call this *villainness*). You can use the `.predict_proba` method on the model to get probability estimates rather than class assignments.* **Visualize the "heroness" distribution for all ambiguous characters**. Comment on the result.<jupyter_code>model.fit(alliances, roles)

for index, row in ambiguous_alliances.iterrows():
    if(row['faction'] != 2): ambiguous_alliances.drop(index, inplace=True)

ambiguous_alliances.drop('faction', axis=1, inplace=True)
probs = model.predict_proba(ambiguous_alliances)

heroness = [row[0] for row in probs]
villainness = [row[1] for row in probs]
ambiguity = [row[2] for row in probs]

plt.hist(heroness)
plt.title("Heroness of Ambiguous Characters")
plt.xlabel("Heroness")
plt.ylabel("Frequency")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10541298064290093, 'fraction_numerical': 0.04488920031531926, 'mean_word_length': 4.603680981595092, 'pattern_counts': {'":': 0, '<': 51, '<?xml version=': 0, '>': 74, 'https://': 9, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '604478', 'license': 'MIT', 'n_tokens_mistral': 8022, 'n_tokens_neox': 7202, 'n_words': 2630, 'path': '.ipynb_checkpoints/385734948479229-checkpoint.ipynb', 'repo_name': 'allengueco/marvel_project', 'size': '22859'}